Agentic Development, In Practice

The mechanism, not the model

Most discussion of AI development focuses on the model. Which provider is best right now. How the new release compares to the old one. What this benchmark or that benchmark says about reasoning quality.

That discussion misses the point. In production AI delivery, the model is important but rarely the differentiator. The differentiator is the mechanism built around the model — the agents that compose its work, the skills that capture procedure, the harness that constrains and verifies its output. Without that mechanism, the model is a smart but rootless contractor. With it, the model becomes a productive member of the team.

This article is about the mechanism. What we’ve built, why it matters, and how it shaped the platform — Compass AI — that we now sell.

Skills are procedures, not prompts

The most under-leveraged construct in AI development is the custom skill. Most teams treat skills the same way they treat prompts: as a single instruction handed to the model. We treat them differently. A skill is a procedure: a named, versioned, scoped workflow that the AI follows when invoked.

The shift in framing matters. A prompt is something a developer types. A skill is something a team builds, refines, and calls deterministically. A prompt lives in one person’s head and one chat session. A skill lives in the codebase, gets reviewed, gets updated as the team learns.

Over time, a taxonomy emerges. The categories that show up:

• Lifecycle skills — drive the spec pipeline. Generate a spec from an idea, implement a phase, retire a stale spec. The discipline of the workflow gets encoded into skills that execute the workflow.
• Scaffolding skills — create new things from established patterns. New feature package, new platform app, new API module. The conventions of the codebase get baked into skills, so following them stops being a discipline and starts being the default.
• Review skills — apply domain-specific quality checks at the boundary between code and merge. Each tuned for a stack, a framework, a class of mistake the team has been burned by before.
• Audit skills — detect drift between intended and actual state. Wiring drift after a merge. Environment variable drift between services. Spec drift against the codebase. The class of slow, silent failures that cost teams disproportionately if not caught early.
• Capture skills — record information in structured formats before it’s lost. Known issues discovered mid-flight. Release notes. Post-incident learnings. Things that, in a typical team, end up in a Slack thread and stay there.

Each skill starts as a response to a real problem. Someone hit a wiring bug after a merge — an audit skill gets built to prevent the next one. Someone forgot to wire a module into the running app — a coherence-check skill gets added. The skills library accumulates as a kind of operational immune system: each past mistake teaching the system to prevent the next one.

Specialist agents change the unit of expertise

Skills are often invoked through specialist agents — narrowly scoped subagents that exist to do one kind of work well. A Java reviewer that knows the framework’s pagination quirks. A frontend auditor that knows our routing conventions. A spec adversary that argues against the spec we just generated.

Why specialist agents instead of one general-purpose model? Three reasons.

Scope produces quality. A reviewer that does one stack catches things a generalist misses. The narrower the agent’s remit, the deeper its expertise can be encoded.

Context windows stay clean. A specialist agent operates inside its own fresh context, with the relevant files, relevant patterns, relevant conventions loaded. It doesn’t carry the cognitive baggage of an unrelated prior task.

Failures stay isolated. When something goes wrong — and something always goes wrong — the failure happens in a scoped subagent invocation, not in the main session. The main session can decide what to do about it without losing its own context.

The taxonomy of agents mirrors the taxonomy of skills, but with a different orientation. Skills are what to do. Agents are who does it. Together they form the procedural layer of the development system.

The model is a capable generalist. The agent harness is what turns it into a team of specialists.

Verification is a stack, not a gate

The single biggest failure mode of AI-assisted development is over-reliance on a single point of verification. Run the tests, look at the output, ship it. This works for trivial work and fails for everything else.

We’ve found that real verification has to be layered:

Phase-level validation at every checkpoint: the structural contract. Did the implementation match what the spec required for this phase? Schema correct, endpoints return the right status codes, build passes, tests pass. This catches structural mismatch.

Multi-turn evals for AI-native components. Scripted conversation flows that probe assistants, scorers, evaluators. They catch drift in probabilistic behavior — the kind of drift that deterministic tests cannot surface, because the failure isn’t “the wrong value,” it’s “the wrong shape of answer for this kind of question.”

Human-in-the-loop testing for the experiential layer. Does the feature feel right to the people who use it? Automated systems can confirm behavior. They can’t confirm fit. Well-defined test scenarios make this phase tractable rather than open-ended; the human judgment component is real and shouldn’t be engineered away.

Each layer catches what the others can’t. Removing any one layer means a class of failure ships to production. We learned this the slow way and built the stack accordingly.

How the practice shaped the platform

We didn’t start with a platform. We started with a delivery practice. Compass AI is what fell out of that practice — the parts of our internal system that, once we’d built and refined them for our own work, turned out to be exactly what our clients needed.

This is an unusual sequencing for an AI platform. It produced a few properties:

Model-agnostic by necessity. Our delivery work spans Anthropic, OpenAI, Google, and open-source models depending on the task. The right model for extraction is rarely the right model for deep reasoning. So Compass routes across providers natively, with workspace-level governance over which models are available for which work. We didn’t build that as a feature; we built it as a necessity, because we couldn’t do our own work without it.

Governance as the membrane. Working with PE firms and their portcos, we hit the access control, audit trail, and tenant isolation requirements early — and they aren’t optional. So those properties are the membrane the platform operates within, not a layer added later. Every Compass query, API call, and background job is scoped to the correct workspace from the inside out.

Custom Assistants and Intelligent Spaces, because that’s how the practice already worked. We were already organizing AI work around specialists tuned to specific domains, operating inside specialized spaces with the right knowledge sources connected. So that became the primary user surface in Compass. It mirrors how our own delivery teams use the platform.

Solutions, not just capabilities. The biggest thing the practice taught us: customers buying a generic AI platform end up with a generic outcome. Customers buying a platform with delivered solutions on top of it — Executive Talent Search, PE Command Center, Production Planning copilots, embedded workflow assistants — get something that actually runs the work. So Compass is sold the way we deliver: the platform plus the bespoke solution build.

The compounding system is the moat

The thing we did not appreciate at first: each individual skill, each individual agent, each individual context file is small. Often a hundred lines, sometimes less. None of them is impressive on its own.

The compounding effect is what matters. A team running this kind of system for a year accumulates hundreds of these artifacts. Each one captures a piece of expertise. Each one prevents a class of mistake. Each one makes the next engagement faster than the last.

From the outside, the team looks like it has gotten radically more productive. From the inside, the team has the same number of people, but they’re operating inside a system that quietly absorbs the things they used to have to remember manually.

For us, this is the durable form of an AI delivery practice — not faster code, not cleverer prompts, but a compounding library of mechanism that gets more capable with every engagement. It’s the reason we believe AI-augmented delivery isn’t an efficiency play. It’s a structural advantage.