AI Impacting Data Engineering Faster Than Expected

We’re in the midst of a revolution thanks to generative AI. There’s no denying that. And while dbt Labs CEO Tristan Handy was expecting the AI revolution to eventually come to the world of big data and analytics engineering, it’s occurring faster than he expected.

“AI is clearly happening and it’s impacting almost all of us already,” Handy said yesterday in his remarks during a 90-minute dbt Cloud Launch Showcase. “It’s a lot faster than I would have thought.”

The big data world has been no stranger to change over the past decade, Handy said. The data center of gravity has moved to the cloud, the ETL world has transitioned to ELT, lots of “legacy tools and workflows” were eliminated, and we started applying software engineering best practices to day-to-day data work.

“And while this cloud transition is still playing out, AI is going to be the next big change in our lives as data professionals,” he said in his comments at the beginning of the event. “The changes we’ll see over the coming years will be just as dramatic as those we’ve seen play out over the past decade.”

They’ll play out in a couple of ways. For starters, the importance of having high quality data to feed AI can’t be understated. AI is a function of data–a distillation of it, really–and this puts the onus on organizations to get their data management acts in place.

“If the data you feed into a model is garbage the results are also garbage. Shocker,” Handy said. “But it might be worse than that, because with AI, hallucinations can sometimes be quite hard to spot. And this lack of trust undercuts the entire AI effort. This is why data leaders who are invested in the AI future care so much about having a robust data capability.”

But AI will also impact the data management tools that people use. In the data engineering discipline, copilots are being developed to assist analysts and engineers with the various data preparation tasks they’re responsible for.

To that end, dbt Labs unveiled its first AI-powered co-pilot yesterday. Dbt Assist, as the product is called, will allow automatically generate documentation and tests for the dbt models they write. Dbt Assist is currently in private preview.

The Philadelphia, Pennsylvania company unveiled several other enhancements, including a new “compare changes” view to the Advanced Continuous Integration (CI) component; the addition of unit testing; and dbt Cloud CLI, which lets users access dbt through a command line interface in their favorite IDE. You can read more about the latest enhancements here.

More AI-powered capabilities are undoubtedly in the future for dbt Labs, which envisions creating a data control plane that tackles many of the data tasks that organizations need to keep maintain their data edge.

“The control plane is a governance layer across all of the components of the existing data stack. It helps with things like orchestration, data cataloging, data quality, metrics definitions, lineage, and cost monitoring,” Handy says. “Our perspective is that these categories will not be separate. They’ll be integrated together into a single platform. And that is what we are building with dbt Cloud.

source: datanami.com