What You Need to Know
An AI-assisted developer workflow gives part of an engineer's work to an AI tool. An outcome measure is a property of the work itself. An adoption figure counts how much the tooling was used. A check shows you cheaply whether an output is right. Packaging changes one person's procedure into an artefact that the team shares.
This lesson uses one made-up example throughout: a university whose software team maintains its course registration system and its fee payment service. A quarter after the team adopted AI coding tools, the head of IT asks whether the tools worked.
Measure the Work, Not the Tooling
Outcome measures are properties of the work:
- cycle time from ticket to merged change
- defect escape rate (the share of defects that reach production)
- review latency (how long a change waits for review)
- how long a new joiner takes to make a safe change
- incident time-to-resolution
Each measure has a value from before. Each measure also changes for reasons that an architect can defend in a review.
Adoption figures measure how much the tooling was used. Examples are sessions per week, accept rate (the share of suggestions that developers accept), and the share of merged changes that contain assisted code. Anthropic's analytics guidance says to read adoption figures together with engineering measures, never instead of them. A productivity claim with no baseline fails in the same way as an accuracy claim without a baseline.
Name the outcome measure and its baseline before the quarter starts. A baseline reconstructed afterwards is not evidence, only a claim. At the university, daily tool use is an adoption figure. The number of payment bugs that reach production is an outcome measure. This number was counted last quarter too.
Verifiability Decides What to Automate
Before you automate a task, ask what checks its output cheaply. That question divides the work into groups. A test suite, a build result, a linter or a fixture comparison completes the loop without a human taking part. A fixture comparison compares the output with a stored expected result. In this loop, the agent works, runs the check, reads the result and iterates. Anthropic warns about an agent without a check that it can run. In that case, the docs say, "that makes you the test suite, and every mistake sits there until you happen to notice it".
Work whose correctness only an experienced human can judge, slowly, is the worst candidate for automation, not the best. Output arrives faster than people can judge it, so automation makes the review bottleneck bigger.
At the university, an agent can change the payment code and run its test suite. A choice between two designs for the registration system needs a senior engineer to judge it, so that task comes last.
The Most Troublesome Task Is Not the Task to Automate First
The tempting option automates the painful, unverifiable work. The right answer starts with work where the check is fast and objective.
Packaging Turns a Habit Into a Capability
One engineer's good procedure is a habit. Packaging changes that habit into the artefact that a team needs. The type of reusable artefact depends on what the team reuses:
- a repeatable procedure becomes a shared, version-controlled workflow that anyone can invoke by name
- a recurring specialist task becomes a defined role with its own instructions and restricted tool access
- a recurring need for context becomes shared project context, not four private copies
Outcome quality then stops depending on who ran the artefact. Improving the artefact once improves it for everyone, because it is reviewed and versioned like code. The gain also remains when someone resigns. A habit does not. At the university, one developer's release-check routine is committed as a shared workflow. It still runs if she leaves.
Review Stays Independent, Humans Keep Merge Authority
Review is the step that enforces quality. Review loses its value when the reviewer shares the author's context. Whatever produced a change still holds the assumptions under which the change was written. So that producer confirms its own reasoning, fluently and at length. A fresh context is a session that does not share the author's context. Anthropic's best-practices page says a reviewer in a fresh context "isn't swayed by the reasoning that produced the code".
Humans keep merge authority, the decision on which changes merge. Anthropic's Code Review docs say "A review never approves or blocks the pull request, so the decision to merge stays with your team." So machine review adds a signal without making the decision. An independent reviewer has its own failure mode. When you ask it to find gaps, it reports some gaps even when the work is sound. Limit the reviewer to correctness and stated requirements, not personal style preferences. At the university, a fresh context reviews the tool's changes for correctness. A named developer decides what merges.
Autonomy Follows Reversibility
Autonomy depends on how cheaply an action can be undone, not how much the team trusts the tooling. A mistake that reverts cheaply costs little. A mistake that cannot revert needs a person first. So:
- limit what automation may execute
- make irreversible actions wait for an approval
- protect the branches that matter
- let continuous integration be the objective check that no confident explanation can override
At the university, a change to the live payment records cannot be undone, so it waits for a named engineer's approval. Continuous integration blocks merges that fail the payment tests.
The same rule applies to adoption. Start where the problem is measurable. Make the gain inheritable (objective 7.1). Let evidence spread the gain, not a mandate.
In the exam
Questions in this objective all ask one thing: did the tooling work? They test what counts as an answer and which work should have been automated at all. They also test what changes one engineer's habit into a team capability, and who approves the result. Little in these questions depends on a product feature. Most depends on knowing the difference between a usage number and a result.
The usual trap option automates the painful work with no check. Packaging is the half of most right answers that is easy to miss.
Caveats
A same-period comparison shows whether the numbers changed, not why they changed. Fewer payment bugs at the university would not prove that the tooling caused the fall.
Exam Traps
Reporting an adoption percentage, an accept rate or a satisfaction survey as the return on the investment
Each counts how much the tooling was used or how it felt, not what changed in the work. With no baseline on cycle time, defect escape rate or review latency, the figure cannot separate a genuine gain from more use of the tooling.
Letting whatever produced a change review it, then recording the change as reviewed
The reviewer still holds the assumptions the change was written under and will confirm its own reasoning back fluently. Review belongs in a fresh context or with a different reviewer, and a named human keeps merge authority.
Using automation first on the most painful work, such as architecture choices, when nothing checks its output cheaply
Nothing checks an architecture choice cheaply, so it produces more to adjudicate. Automation aimed at unverifiable work sends work to a review bottleneck faster than people can review it. Start where the check is fast and objective.
Measuring the improvement honestly but leaving the procedures that produced it in individuals' memories
An unpackaged procedure cannot be reviewed or improved once for everyone, and it does not remain when someone resigns. Commit it as a shared artefact anyone can invoke by name, reviewed and versioned like the rest of the repository.
Practice Scenario
A platform team is a quarter into AI-assisted tooling and the director asks whether it worked. The only figure anyone holds is that 70% of merged changes contained assisted code. Two engineers have worked out procedures that clearly help — one for generating migration tests, one for triaging production logs — and neither has been written down. What should the architect do?
Build Exercise
Build the Workflow Inventory and Its Outcome Measure
What you'll learn
- How to separate outcome measures from adoption statistics
- How to choose the reuse unit (the form in which work is shared) for a procedure that is still a habit
- How to set a baseline before the comparison rather than after it
- How to keep review independent of whatever produced the change
- How to match autonomy to how easily the action can be undone
- List every repeatable procedure your team runs with AI assistance, and mark each one as packaged and shared, or still a habit that one person holds
Why: The inventory is the only way to see how much team capability is currently held by individuals.
You should see: One row per procedure, with the unpackaged rows outnumbering the packaged ones on a first honest pass.
- For each unpackaged procedure, name the reuse unit it should become - a shared invokable workflow, a defined role with its own instructions and restricted tools, or shared project context - and say who inherits it and where it is committed
Why: The reuse unit follows what is being reused, and inheritance is what makes the gain a team asset rather than a personal one.
You should see: Every unpackaged row assigned a unit and a location that a new joiner receives on clone.
- Choose one outcome measure per procedure, name the baseline you would compare it against, and name one adoption statistic you will deliberately not report as a result
Why: Naming the statistic you refuse to present is the fastest way to find out whether you hold a real measure.
You should see: A measure and a baseline per procedure, plus one adoption figure explicitly ruled out.
- Write the review rule: what reviews the changes the tooling produced, in which context, what it is scoped to look for, and who holds merge authority
Why: Review is the main control, and it stops working when the reviewer shares the assumptions of whatever wrote the change.
You should see: One paragraph naming an independent reviewer, a narrow scope, and a named human with merge authority.
- Audit the limits on automation. For every automated or unattended run, record what it may execute, what is gated behind an approval, what continuous integration blocks regardless, and which of those rest on a written instruction rather than an enforced control
Why: Autonomy should depend on how easily an action can be undone, and the rows resting on instruction are the findings.
You should see: A short table per automated run, with an honest count of the controls that are guidance today.
Sources
- Best practices for Claude Code — Anthropic
- Track team usage with analytics — Anthropic
- Extend Claude with skills — Anthropic
- Code Review — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 7 — Anthropic