Instruction

The Shift from Experiment to Production Practice

Configuring an agentic review job and observing its first useful comments is an amazing start. However, sustaining that value across dozens of pull requests, multiple contributors and successive model updates requires a different set of practices. The same architectural pillars from the previous lesson remain, yet each pillar must now be tuned for usage predictability, operational hygiene and continuous improvement.

Without management, three failure modes appear:

1. Unrestricted path triggers and full-repository context cause token consumption to scale linearly with repository size and pull-request frequency.
2. Review jobs that run in isolation from the existing build and test matrix can produce comments that contradict later CI failures, or can race with other jobs that also post to the conversation.
3. Triage decisions remain common knowledge; the same categories of false-positive or out-of-scope comments reappear on every subsequent pull request because nothing in the system has been updated.

The remainder of this lesson emphasizes agentic review as a production service that must be observed, constrained and iteratively refined.

Cost and Trigger Governance

Token consumption is the most visible operational constraint. A pragmatic control surface consists of four complementary mechanisms.

Path filters

• Restrict the workflow to the file types and directories that actually benefit from agentic analysis.
• For an iOS project, this typically means using Swift, Objective-C, storyboard/XIB and selected configuration files, while excluding pure documentation, localization strings, asset catalogs and generated code.
• GitHub Actions path filters are evaluated before the job starts, so excluded changes incur zero runner time and zero model tokens.

Draft pull-request exemption

• Many teams open draft pull requests early for feedback or continuous integration of incomplete work.
• Running a full agentic review on every draft push is rarely justified.
• A simple conditional if: github.event.pull_request.draft == false or an equivalent filter keeps the review step for ready-for-review changes only.

Concurrency and cancellation

• The concurrency group keyed to the pull request number, already introduced in earlier lessons, remains essential.
• When a developer pushes successive commits rapidly, only the latest review should complete; earlier runs are canceled.
• This both reduces token usage and prevents a confusing sequence of overlapping comments.

Model and turn limits

• Different agents expose different levers such as model identifiers, maximum turns, temperature, or permission profiles.
• For routine changes, a lighter, faster model is usually sufficient.
• Higher-capability models that consume more tokens can be reserved for paths that touch security-sensitive or concurrency-critical code, either by separate workflows or by conditional steps inside a single workflow.

Taken together, these controls keep the token usage of each review proportional to the risk of the change. Additionally, they make the usage visible: when a pull request does trigger a review, the team can still observe the token summary and decide whether the current configuration remains appropriate.

Integrating Review into a Multi-Job iOS CI Workflow

Most iOS repositories already possess a workflow that builds the application, runs unit tests, and perhaps executes static analysis or UI tests. The agentic review job should become one peer among these jobs rather than a separate, disconnected pipeline.

A clean multi-job structure typically looks like the following logical arrangement:

• A shared checkout and environment-setup job (or reusable composite action) that prepares the runner once.
• Parallel or sequential jobs for build, unit test, and any lint or static-analysis steps.
• An independent review job that is allowed to run in parallel with the test jobs, because its output is advisory rather than a hard gate.
• Optional downstream jobs that consume the review artifact only if further automation (for example, automatic issue creation) is desired.

Because the review job posts comments rather than setting a required status check (unless the team explicitly chooses to make it required), it can fail or time out without blocking merge. This design preserves the principle established before: human reviewers still make the decisions, but they begin from a richer starting point.

Care must be taken with permissions. The review job needs contents: read, pull-requests: write and sometimes id-token: write for certain authentication methods. It should not receive broader write access to the repository. Secrets holding provider API keys are scoped to the job or to the environment that requires them, never exposed to jobs that don’t need them.

When multiple review agents are retained from the comparative evaluation, they may share a single concurrency group or run under separate groups; the important invariant is that they don’t flood the conversation with near-identical comments. Some teams elect to run only one primary agent on every pull request and to invoke the second agent only on a path filter that matches high-risk directories, or only when a human reviewer requests it via a comment.

Least-Privilege Authentication and Secret Hygiene

Production runners process untrusted code from pull requests, including those from external branches. Any credentials provided to an agent should therefore be treated with the same care as any other CI secret.

A few good practices are:

• Prefer short-lived or federated credentials, such as workload identity federation, where supported, rather than long-lived static API keys.
• Scope repository or organization secrets so that only the jobs that need them can access them.
• Never expose secrets in logs. Official actions generally handle this automatically, but any custom scripts should be checked carefully.
• Rotate credentials regularly and immediately if there is any suspicion that they have been exposed.
• Document the required secrets in the repository’s contribution guide or CODE_REVIEW.md, so that new maintainers can set up the environment without having to guess what is needed.

Note: None of these practices are unique to AI agents; they are standard CI security practices. Using external model providers simply makes the importance of following them more apparent.

Closing the Feedback Loop

The triage policy introduced in the previous lesson—confirmed defect, discussion point, dismissible noise—produces a stream of decisions. That stream is the primary signal for improving the system.

A lightweight closed loop proceeds as follows:

1. Capture:
   • A reviewer records the categories of comments that repeatedly fall into the dismissible noise bucket and the categories that consistently yield confirmed defects.
   • These comments can be gathered into a pull request conversation or a short living document.
2. Analyze:
   • Periodically, after a fixed number of pull requests or at the end of a sprint, the team examines the captured data. Patterns emerge.
   • An agent may systematically flag a style preference that the team has already decided to ignore, or may miss a class of concurrency issue that later caused a production incident.
3. Update guidance:
   • The repository’s CODE_REVIEW.md, CLAUDE.md, AGENTS.md or equivalent file is revised.
   • New ‘do not flag’ rules are added, priorities are reordered, examples of preferred comment style are clarified.
4. Update prompts:
   • The workflow prompts that invoke each agent are adjusted to reflect the revised guidance.
   • This is often done by simply pointing the agent more explicitly at the updated file or by tightening the instruction that limits comment scope.
5. Re-measure:
   • The next set of pull requests is observed under the new configuration.
   • Token counts, comment volume and the proportion of dismissible findings are compared with the previous baseline.

Because the guidance files and prompts are ordinary version-controlled artifacts, the improvement cycle itself is auditable. A pull request that changes CODE_REVIEW.md can be reviewed by the same human process that reviews application code, ensuring that the rules enforced by the agents remain under team ownership. Over time, the false-positive rate will decline and the density of actionable findings will rise. If the opposite happens (e.g. comment volume increases while the proportion of confirmed defects falls) the team has a clear signal to re-examine model selection or the decision to keep a particular agent in the routine path.

Organizational Signals That Warrant Recalibration

Even a well-governed system requires occasional attention. The following observable signals typically indicate that recalibration is due:

• A sustained rise in the fraction of comments classified as dismissible noise.
• Review latency that begins to exceed the time developers are willing to wait before requesting human review.
• Token count that grows faster than the volume of substantive code changes.
• Divergence between agent findings and the defects that later appear in production or in human review.
• Introduction of a new major model version whose behavior on the team’s codebase hasn’t yet been measured.

When any of these signals appear, the comparative measurement technique practiced in the previous lesson can be repeated on a representative sample of recent pull requests. The resulting data informs whether to tighten filters, switch the primary agent, adjust prompts, or temporarily disable automated review for a subset of paths.

Preparing the Starter Project for the Demonstration

Before moving on to the practical section, ensure the following prerequisites are satisfied:

• The starter project already contains the multi-agent review workflows created in the previous lessons, together with a working build-and-test workflow.
• Path filters and concurrency groups are either not yet implemented or intentionally incomplete, allowing the demonstration to introduce them as part of the exercise.
• The shared CODE_REVIEW.md (or equivalent) still reflects the state left at the end of the previous lesson - it will be refined during the feedback-loop portion of the demonstration.
• At least one recent pull request with recorded triage decisions is available, so that the refinement step has concrete input.

With the governance principles and feedback loop in place, the demonstration will extend the existing workflows, integrate the review job into a realistic multi-job pipeline, add the necessary token-usage controls, and run through a complete cycle of triage and improvement. The result is a sustainable and observable agentic review process that can improve over time while fitting naturally into the wider iOS CI workflow.