Most agentic AI pilots on Databricks stall on context, not capability. Here’s the fix, and the two accelerators Calfus uses.
Agentic AI is changing how data engineering gets done
Until recently, AI help with code meant autocomplete. A model suggested the next line, you accepted or rejected it. Agentic AI works differently. You give an agent an objective, and it plans the work, executes across several steps, checks its own output, and comes back when the task is done.
That shift is showing up in the platforms. Genie Code, Databricks’ coding agent for data work, builds pipelines and ingestion connectors from a plain-language description. Genie ZeroOps, in private preview, goes further. It monitors pipelines, jobs, tables, and models. When something breaks, it traces the root cause through lineage, drafts a fix, and tests that fix in a sandbox against zero-copy clones of real data, never against production. Nothing gets applied until someone approves it.
In both cases the agent proposes and a human decides. What changes is where the engineer spends time: less writing, more directing and reviewing.
This is a real change to how data engineering teams work. It’s also where a lot of enterprise pilots stall, and the reason usually has nothing to do with the model.
The problem is context, not capability
Most teams adopting AI coding agents hit the same wall within a few weeks. The agent works. It writes code, queries data, builds pipelines. What it doesn’t do is work the way your team works. It names things its own way. It picks patterns your standards rule out. It skips the validation gates your delivery process depends on.
Genie Code arrives capable. It ships with built-in skills for notebook code, Unity Catalog exploration, dashboards, pipelines, and MLflow. But it has no way of knowing your medallion conventions, your data standards, your sign-off gates, or what your team means by “done.”
So engineers explain the same context again in every session. The productivity gain leaks away into review cycles.
The usual fix is a global block of custom instructions. That works for a while. Then it stops: the instructions load on every request whether they matter or not, and one set of guidance ends up serving engineers and analysts who need different things.
What agent skills are
Databricks calls them agent skills. They package “domain-specific knowledge and workflows that Genie Code can load when relevant to perform specific tasks.” In practice, a skill is a folder with an instruction file in it. You can scope one to yourself while you prototype, or to the whole workspace once you want everyone held to the same standard.
The loading is the clever part. Skills use progressive disclosure. The agent keeps only each skill’s name and description in view, opens the full instruction set when a request matches, then pulls in referenced material as it works. Databricks is direct about why this matters: custom instructions apply globally on every request, but skills load automatically and only in the relevant context, which keeps the context window efficient.
Why this is the adoption unlock
Standards get enforced, not just published. A workspace skill is the same instruction set for every engineer on every run. You can version it, review it, promote it. That’s a different thing from a standards document nobody opens.
Cost stays predictable. Skills activate only when they’re needed, so a library of twenty costs roughly what twenty descriptions cost until one fires. At enterprise volume, where you’re paying per token, that shows up on the budget.
Your best engineer’s judgment travels. Write it down once and it applies for everyone, including whoever joined last week.
You’re not locked in. Agent skills follow an open standard that several coding agents now support, so the effort you put into writing one isn’t tied to a single tool.
What we have built
Our Data & Analytics CoE has built two accelerators. Both are reusable Calfus IP that we deploy across engagements rather than rebuild each time, and both are in use on client work now.
Medallion Test Forge writes the test cases, validation logic, entry and exit gate checklists, and sign-off reports for medallion pipelines. It covers Bronze through Gold, including CDC and incremental loads, business logic, and data quality. Two decisions did most of the work. The first was making it refuse to guess. It reads the live catalog for real schema and key relationships instead of assuming them. If it can’t verify something, it stops and asks. Where it has to infer business logic, it marks that output so a human checks it before anyone runs it.
The second was encoding our engineering standards as rules the skill executes, not comments a reviewer leaves afterwards. The standard gets applied while the output is being written.
CodeSense reviews SQL and PySpark before it runs. It applies 65 layer-aware static rules across nine categories of pipeline code, and returns findings with a severity, a fix priority, and a merge decision. It follows the same discipline. It won’t invent business context it can’t see: keys, ordering logic, access groups, intent. It asks instead. And a missing answer never softens a finding’s severity. Risk gets scored on what the code actually does.
These work as a pair because each one stops somewhere deliberate. CodeSense finds the places where a control is missing. Test Forge builds the control. Neither wanders into the other’s job, so an engineer using both never gets two opinions on the same line.
What this means for our clients
Neither accelerator needs core changes to deploy. A client’s standards go in as configuration and the skill picks them up on the first run. Nothing gets rebuilt per engagement, and there’s no custom version left behind to maintain.
Delivery gets faster. Writing a test pack for a set of pipeline objects takes two to three days of skilled manual work, depending on how much is in scope. The skill does the first pass in minutes. The engineer then reads what came out and hand-tests the paths they want to be sure about. Days of writing become hours of checking.
Quality lands at the engineering layer. Standards get applied where the code is written, so defects that used to surface late never get that far. And because the standard lives in the skill instead of in someone’s head, it survives people leaving.
Engineers can check their own work first. Code review is usually a queue. The deployment sits waiting on an approver. Here the engineer reviews first, and the approver opens a set of findings instead of a blank file, so their attention goes to whatever looks genuinely risky. Days of waiting turn into hours.
The mechanism is the same in both cases, and it’s worth being clear about it. The skill doesn’t remove the human. It changes what the human does. Instead of writing the tests, you read them. Instead of checking everything, you check what matters. That’s where the time comes from. It doesn’t come from trusting the output blind.
Two caveats
A skill that encodes a bad standard will apply that bad standard everywhere, for everyone. So the review a skill gets matters as much as the review it performs.
And someone has to own them. Skills need writing, versioning, and retiring when the standard changes. Neither point is a reason to avoid the approach. Both are reasons to treat a skill library as a real asset with a real owner.
The shift
Agents didn’t suddenly get smarter. What moved is the unit of work. Teams getting value out of this spend less time prompting and more time writing down what they know, and written-down knowledge keeps paying out in a way prompts never did.
As a Databricks partner, we think the next phase of platform adoption gets won here. Not by picking the right agent, but by working out what your organization knows that the agent doesn’t, and getting it written down somewhere the agent can use.
So if you’re piloting agentic AI on Databricks, the question isn’t “which agent.” It’s “what do we know that the agent doesn’t, and have we written it down?”
Further Reading
- Extend Genie Code with agent skills, Databricks Documentation https://docs.databricks.com/aws/en/genie-code/skills
- Agent skills for AI coding assistants, Databricks Documentation https://docs.databricks.com/aws/en/agent-skills/
- Lakeflow: A new era of agentic data engineering, Databricks Blog https://www.databricks.com/blog/lakeflow-new-era-agentic-data-engineering
- Introducing Genie ZeroOps, Databricks Blog https://www.databricks.com/blog/introducing-genie-zeroops
- The Agent Skills open standard https://agentskills.io/
Calfus is a Databricks partner. If you’re exploring agentic AI on Databricks and want to compare notes on what has worked and what hasn’t, we would love to talk!
