Meroxa’s change data capture service works with Apache Kafka, others

Meroxa has launched a platform-as-a-service (PaaS) environment with a control plane that leverages machine learning algorithms to manage real-time data. This comes after the company raised a fresh $15 million in its series A round.

Meroxa has developed a PaaS platform through which IT organizations invoke a control plane that provides a change data capture service integrated with platforms such as Apache Kafka. That core capability is then extended through a set of rule engines that make it possible to automate repetitive engineering tasks, Meroxa CEO DeVaris Brown told VentureBeat. IT teams will be able to access that control plane via a visual interface or programmatically invoke it through a set of application programming interfaces (APIs) Meroxa has exposed.

Demand for automation

As organizations look to accelerate digital business transformation initiatives, many have discovered those projects require an ability to regularly shift massive amounts of data between the applications that enable a given process. This has necessitated hiring data engineers with the programming skills to orchestrate the movement of that data. Meroxa is making it possible for the average IT administrator or developer to now orchestrate data flows between applications. Beyond reducing the total cost of digital business initiatives, the rate at which those projects can be completed can now be significantly accelerated, Brown said.

“Anybody can be a data engineer,” he said.

Meroxa is applying many of the DevOps automation principles that were first applied to application development to the engineering of data pipelines. This has typically been viewed as an IT maintenance task that requires an individual to master the nuance of extract transform and load (ETL) tools. But people with data engineering skills are now among the most sought-after IT specialists, and the number of IT professionals with those skills is limited.

Data pipeline orchestration

The need to orchestrate data pipelines faster is becoming more acute because most digital business transformation initiatives typically require data to be processed and analyzed in near real time. Batch-oriented application processes are being replaced by applications that are capable of consuming stream data directly from platforms such as Kafka. That transition, however, will become extended if every organization needs to find, hire, and retain data engineering specialists.

The ability to automate the constriction of data pipelines will also make it feasible for a larger number of organizations to successfully re-engineer processes. Many smaller organizations simply can’t afford to hire a dedicated data engineering specialist or contract an IT services firm to provide one.

It may be a little while yet before IT teams are routinely creating data pipelines between applications and processes. However, the history of IT is littered with examples where the domain of an IT specialist has been subsumed into a function that can be handled by an IT generalist using some form of platform that automates a task. The engineering of data pipelines will ultimately be no different.

In total, Meroxa has now raised $19.2 million in funding from investors that include Drive Capital, Root, Amplify, Hustle Fund, Village Global, Meritech Capital, Sequoia, Kleiner, Addition, Menlo, and Index Ventures.

Other investors include former Heroku CEO Adam Gross; GitHub CTO Jason Warner; former Segment CTO Calvin French-Owen; and Nick Caldwell, VP of engineering at Twitter.