> ## Content Index > Fetch the complete content index at: https://blog.elvatis.com/llms.txt > Use this file to discover other available public pages before exploring further. # Hardening an Existing Agent Fleet in Sandboxes - URL: https://blog.elvatis.com/hardening-an-existing-agent-fleet-in-sandboxes/ - Published: 2026-09-07T21:41:39.000Z - Updated: 2026-09-07T21:41:39.000Z - Description: How an established agent fleet moved into sandboxes, combining GitHub Bots, cloud workers, twenty local CI runners, durable orchestration, and budget controls. - Author: E. Kohler - Tags: ai, automation, infrastructure The agent fleet was already established. Specialized agents owned repositories, exchanged handoffs, reviewed changes, and moved work through defined governance rules. Then I moved those agents from a host-oriented runtime into isolated sandboxes. The roles stayed. Several operational assumptions did not. Credentials, paths, tools, wake-ups, and session state now had to work inside each isolation boundary. Host conveniences were no longer available. The project scope was migration and hardening: preserve the operating model, rebuild the runtime boundary, and restore autonomous flow. This guide covers that transition, including GitHub Bot identities, deterministic orchestration, review loops, retries, merge gates, and provider budget controls. ## The fleet before the sandbox migration The fleet already operated as independent roles under shared governance. Repository ownership and review separation were not introduced by this migration. Workers owned defined repository scopes. Reviewers inspected current code independently. Existing handoff rules separated implementation from approval. The migration retained that operating model. It added stronger isolation and made each runtime dependency explicit. The examples below use fictional role names. They illustrate the existing pattern without exposing the production identity map. - **Builder.** Selects one issue, creates one branch, and opens at most one pull request. - **Reviewer Alpha.** Focuses on correctness, tests, and unintended behavior. - **Reviewer Beta.** Focuses on boundaries, abuse cases, and governance rules. - **Relay.** Converts trusted GitHub activity into a scoped worker or review request. - **Merger.** Performs only the final mechanical checks and merge. - **Supervisor.** Classifies measured failures but cannot write code or change controller state. Builder and reviewer responsibilities already existed. Relay, merger, and supervisor describe the bounded operating functions separated during hardening. The controller sits outside those roles. It chooses when a model may start, not what conclusion that model must reach. ## What the sandbox migration changed Before isolation, some dependencies were inherited from the host. That included command-line authentication, filesystem layout, installed runtimes, and long-lived session context. Inside a sandbox, those assumptions disappear by design. Every dependency must be mounted, installed, resolved, or deliberately denied. Scheduled probes also execute under the owning agent's sandbox policy. A controller-side credential cannot be assumed inside that probe. The migration therefore exposed several integration boundaries: - **Authentication boundary.** Repository commands needed the agent's machine identity inside its own sandbox. - **Filesystem boundary.** Host paths had to become workspace-relative or sandbox-local paths. - **Runtime boundary.** Every agent image needed the tools required by its assigned repositories. - **Execution boundary.** A command backend had to preserve the originating sandbox instead of falling back to the host. - **Wake-up boundary.** Repository comments provided context but did not reliably start a sleeping worker. - **Session boundary.** Work, review, and relay traffic needed separate transcript lifecycles. These were migration effects, not missing agent expertise. The fleet knew what to do once the runtime delivered the right context and authority. A watcher helped measure the gaps. It could prove inactivity, but only an orchestrator could admit the next legitimate task. ## Rebinding repository work to GitHub Apps and Bots The largest identity change was binding repository authority to each sandbox. Repository actions moved away from ambient host credentials. During the migration, each fleet role received a dedicated GitHub App identity that could travel with its isolated runtime. GitHub attributes actions from an installed App to its Bot account. Comments, reviews, commits, and merges therefore have clear machine ownership. This matters for both auditability and policy. A human login should not become the invisible shared identity of an autonomous fleet. I used one App per role because each role had a distinct ownership boundary. A smaller setup could use one App per trust boundary. The migration sequence was straightforward: - **Create the App identity.** Give it a clear role name and no human credentials. - **Install it narrowly.** Grant access only to repositories assigned to that role. - **Choose repository permissions.** Allow the minimum writes needed for branches, issues, pull requests, reviews, and merges. - **Add read permissions.** Include checks, workflow runs, commit statuses, and protection metadata required for verification. - **Mount the App identity.** Make the private signing material available only inside the assigned sandbox. - **Mint short-lived tokens.** Generate one installation token for one command and discard it afterwards. - **Verify attribution.** Create and remove a harmless test reference, then confirm the expected Bot identity appears. The exact permission set depends on the role. A reviewer needs fewer write permissions than a builder or merger. Workflow write access deserves special treatment. Grant it only when that role is expected to change workflow files. An App token may receive a forbidden response from an endpoint designed for human users. That does not prove the App is broken. Validate the token against the assigned repository and the exact operation the role must perform. ```shell # Illustrative sandbox calls repo-app-run -- gh repo view example/project repo-app-run -- gh issue view 142 --repo example/project repo-app-run -- gh pr view 167 --repo example/project --json headRefOid,statusCheckRollup # The remote stays credential-free git remote set-url origin https://github.com/example/project.git ``` I never place an installation token in the remote URL. Embedded credentials can expire and override a healthy credential helper. The relay also recognizes both the App slug and the rendered Bot login. Prefix lookalikes and quoted mentions are rejected. ## Cloud workers and twenty local CI runners The execution plane combines sandboxed cloud workers with twenty local self-hosted CI runners. Both workloads share one high-core-count workstation. The machine uses a Threadripper 3960X with 24 physical cores, 48 hardware threads, and 128 GB of RAM. That provides substantial local capacity, but twenty runner registrations are not twenty independent computers. CPU, memory, storage, and container activity remain shared resources. The controller therefore manages two different capacity problems: - **Model capacity.** Cloud workers are limited by provider budgets, active sessions, and per-account concurrency. - **Local CI capacity.** Builds and tests are limited by shared CPU, memory, storage, and runner availability. - **Combined pressure.** Agent-side tests and CI jobs may compete for the same local resources. - **Admission order.** Existing pull requests and current-head reviews take priority over starting unrelated new work. I do not treat every registered runner as guaranteed parallel capacity. Heavy builds need more headroom than lint or documentation checks. The practical model is weighted admission. Short checks can fill spare slots while expensive builds receive stricter concurrency. ```text illustrative local capacity policy: runner_slots = 20 light_check_weight = 1 test_suite_weight = 2 full_build_weight = 4 total_weight_limit = measured host capacity admit job only when: runner is available weighted capacity remains memory pressure is acceptable no higher-priority pull request is waiting ``` The surprising bottleneck is not always local compute. A short build or test can finish before GitHub updates every remote status view. For fast jobs, queueing, API polling, and check propagation may take longer than the local verification itself. That difference matters. A controller should not restart finished work merely because the remote status still looks pending. I use bounded polling and always correlate the result with the exact pull request head. A later status for an older commit cannot complete the current run. ## Why I built a dedicated CI runner dashboard The standard repository view answers whether a check passed. It does not fully explain what the local runner pool is doing right now. I built a separate runner dashboard to combine local execution evidence with GitHub's job assignments. The dashboard shows: - **Runner state.** Online, idle, busy, or offline for every registered runner. - **Current work.** Repository, workflow, job name, start time, and assigned runner. - **Queue pressure.** Waiting jobs that have not received a runner yet. - **Recent activity.** Completed jobs and their results across the pool. - **Local evidence.** Runner process state and logs when remote status information is delayed. - **Capacity view.** Busy and idle counts beside the queue, rather than an isolated pass or fail icon. An operator snapshot can remain compact: ```text CI_POOL | HEALTHY runners: 20 registered, 8 busy, 12 idle, 0 offline queue: 3 jobs local: build finished remote: check status still propagating action: poll current head, do not restart ``` The values above are illustrative. The important part is the distinction between local completion, remote propagation, and actual queue pressure. This dashboard prevents three different delays from looking identical: unavailable runners, saturated local capacity, and slow remote status propagation. ## The deterministic control plane The controller is model-free. It runs a short reconciliation cycle and starts no model when the queue is idle. That is a deliberate cost and reliability decision. A model should not decide whether another model is allowed to start. ```text every reconciliation cycle: load durable controller state reconcile finished, failed, and stale jobs read the rolling budget state probe each ownership lane exclude paused and already active lanes rank eligible candidates fairly admit only within global and account limits persist state before dispatch publish a fresh heartbeat ``` A healthy cycle returns `idle` or `admitted`. It does not need a conversational answer. Persisting before dispatch prevents a crash from creating invisible work. Startup reconciliation can recover the admission by idempotency key. I also keep a slower standby selector. It becomes active only when the central heartbeat is stale. ## A complete issue-to-merge example The following example shows one normal path. The identifiers are fictional, but the state transitions are real. ```text Issue #142 opened -> controller selects Builder -> Builder verifies scope and posts a claim -> Builder creates branch issue-142-fix -> Builder opens PR #167 -> Reviewer Alpha reviews current head A1 -> Reviewer Beta requests changes on current head A1 -> Builder pushes current head A2 -> both reviewers recheck A2 -> required checks pass -> Merger re-queries A2 and merges PR #167 -> controller records completion and releases the lane ``` The worker receives only one candidate. It must not browse the backlog and substitute a more interesting issue. An illustrative controller invocation looks like this: ```shell fleetctl reconcile --once fleetctl probe --lane builder --format json fleetctl dispatch --candidate candidate.json --idempotency-key work-142-a1 ``` The names are placeholders. The important part is the separation between probe, admission, and model execution. The selected candidate is self-contained: ```json { "kind": "issue", "repository": "example/project", "number": 142, "priority": "high", "owner_role": "builder", "head_before_work": "main@abc123", "allowed_actions": ["claim", "branch", "test", "open_pr"], "forbidden_actions": ["merge_own_pr", "deploy", "change_permissions"] } ``` ## What agents should report Long narrative updates are expensive and hard to scan. I use compact completion messages with stable fields. A successful worker response can look like this: ```text WORK_COMPLETE repo: example/project issue: #142 pr: #167 head: A2 tests: 24 passed next: independent review ``` A blocked worker should report evidence and ownership, not select replacement work: ```text WORK_BLOCKED repo: example/project issue: #142 reason: required decision is outside worker authority evidence: acceptance criterion 3 requires owner selection next: owner decision ``` Human alerts should contain the repository, object number, role, classification, and next action. Normal idle cycles remain silent. Named operator sessions keep routine outcomes visible without flooding the alert channel. ## How the review loop works A review belongs to one commit, not to the pull request forever. Any new push can invalidate the earlier verdict. The reviewer fetches the canonical pull request, checks the current head, runs proportionate tests, and posts exactly one decisive review. A blocking review should be reproducible: ```text REVIEW_CHANGES_REQUESTED pr: #167 head: A1 finding: retry state is deleted before delivery acknowledgement reproduce: interrupt admission after remote fetch expected: event remains queued observed: event is missing required_fix: delete only after acknowledged admission ``` After the author pushes A2, the controller wakes every reviewer with a stale decisive review. It does not wake only the latest reviewer. The second review can then close the loop: ```text REVIEW_APPROVED pr: #167 head: A2 verified: failure reproduction now retains the event tests: retry and duplicate-delivery cases passed remaining: none ``` A later comment does not erase an earlier approval or change request. Only the latest decisive state per reviewer counts. Sensitive changes require multiple independent current-head approvals. The merger cannot replace missing review evidence. ## Retries without endless repetition Retry logic must distinguish a target failure from a broken worker lifecycle. Otherwise target rotation hides one recurring infrastructure defect. I saw this when several different issues failed through the same reused session. The issue numbers changed, but the run identity did not. Self-contained worker runs now start with a clean transcript. A malformed tool call cannot poison the next independent issue. ```text attempt 1: result = provider_tool_error action = retain candidate cooldown = bounded attempt 2: reset session lifecycle reuse idempotency key recheck repository state dispatch only if still eligible after retry limit: classify once suppress duplicate alerts wait for controller recovery or owner action ``` The controller also remembers safe skips. Without that memory, the same unsuitable issue can become a permanent queue livelock. Failed and stale jobs receive bounded cooldowns. Completed jobs disappear only after reconciliation records their outcome. ## Durable event relay Repository events should not wait synchronously for a full model response. That couples delivery reliability to model latency. I persist each event locally before admission. The relay uses an overlap lock, deterministic keys, bounded retries, and quarantine for malformed records. ```text on incoming GitHub event: fetch into durable spool validate event shape and exact Bot target reject duplicate idempotency keys check budget before admission reset the dedicated relay session enqueue without waiting for full completion remove only after admission acknowledgement ``` If the model provider pauses, the event remains queued. A later relay cycle continues without asking GitHub to resend it. Quoted comments remain untrusted pointers. The agent opens the canonical issue or pull request before interpreting the request. ## Small models as mechanical mergers A lower-cost model can merge completed work when the role is deliberately narrow. The merger re-queries the exact pull request and checks the current head. It does not fix code, resolve conflicts, or bypass protection. ```text MERGE_ELIGIBLE when all are true: pull request is not a draft merge state is clean every required check succeeded required approvals target the current head author is not the merger identity no unresolved change request remains otherwise: return MERGE_SKIPPED with the failed condition ``` Tool-heavy reviews need a more capable model when a smaller one produces unreliable structured calls. Model choice follows measured role performance. The supervisor can also use a small model because it is read-only. It classifies evidence but cannot dispatch, edit, review, or merge. ## Budget protection across subscriptions Counting agents did not balance usage. Reviews, relays, model choice, and sticky context made one subscription significantly heavier. I partitioned the fleet by role and measured load. Work, review, and relay sessions remain pinned to their assigned account. There is no silent cross-account fallback. Reaching one limit must not drain the second account immediately. A model-free guard evaluates a rolling provider window. It stages restrictions before the provider imposes a hard stop. - **Warning.** Continue running and send one state-change notification. - **Pause work.** Stop new implementation runs first. - **Pause relay.** Keep events in the spool without starting tool-heavy sessions. - **Pause review.** Stop new reviews while existing runs finish. - **Keep merge reserve.** Allow only the bounded mechanical lane while the account remains usable. An illustrative warning is intentionally short: ```text BUDGET_GUARD | WORK_PAUSED pool: account-a window: rolling provider period deferred_events: 4 active_runs: allowed to finish next: automatic recheck ``` The guard must `fail closed`. If usage cannot be measured safely, the controller admits no new expensive work. Global and per-account concurrency limits protect the provider budget and keep the interactive control plane responsive. ## Alerts need state, not noise Early alerts repeated retries and sometimes declared completed work overdue. The problem was the observation window, not the worker. A run is overdue only when wall time and the controller's latest observation both prove it remains active. The supervisor emits alerts only for new controller errors, repeated worker failures, stale heartbeats, or genuinely stale runs. Identical exceptions are suppressed during a cooldown. Healthy cycles start no supervisor model. ```text FLEET_SUPERVISOR | AUTO_RETRY role: builder target: example/project#142 class: provider tool-call failure action: candidate retained, session reset, bounded retry scheduled ``` This gives the operator a decision surface instead of a stream of internal chatter. ## The failure tests I now require A passing happy path proves very little. The important tests interrupt the system between two components. - **Controller restart.** Recover admitted work without creating a duplicate. - **Probe failure.** Start no model and select no fallback issue. - **Token mint failure.** Retain the target and emit one infrastructure classification. - **Relay interruption.** Keep the event until delivery is acknowledged. - **Malformed tool call.** Reset the session before another independent target. - **Budget pause.** Start no expensive model and preserve deferred events. - **Stale approval.** Refuse merge until approvals target the current head. - **Self-merge attempt.** Refuse even when checks and reviews are green. - **Controller outage.** Resume slower selection only after heartbeat expiry. - **Alert race.** Do not mark an already completed run as overdue. The health check validates runtime files, session routing, clean remotes, Bot access, timers, required tools, and recent scheduler failures. ## A practical rollout order I would implement the system in this order: **1\. Define ownership.** Give every role an explicit repository and action boundary. **2\. Create GitHub Apps.** Separate machine attribution from human credentials. **3\. Prove sandbox identity.** Test one real read and one harmless write per role. **4\. Build the read-only selector.** Make queue choice deterministic before model execution. **5\. Add durable controller state.** Store active work, outcomes, cooldowns, and heartbeat time. **6\. Split role sessions.** Keep work, review, relay, and merge lifecycles independent. **7\. Add idempotent dispatch.** Persist before sending and reconcile after restart. **8\. Add budget stages.** Pause expensive lanes before provider exhaustion. **9\. Add a read-only supervisor.** Classify only exceptions supported by deterministic evidence. **10\. Prove refusal paths.** Test stale approvals, self-merge, missing identity, and out-of-scope writes. ## What the migration actually improved I did not redesign the fleet's purpose. I preserved its role model, repository ownership, handoffs, and independent review structure. The improvement was operational. Host assumptions became explicit sandbox contracts with measurable health checks and refusal paths. The controller knows when to start. The worker knows what it owns. The reviewer knows which commit it judged. The relay knows when delivery is durable. The merger knows when to refuse. The budget guard knows when silence is safer. For sequential handoffs, I also keep context structured and minimal. That principle drives my public [Agent-to-Agent Handoff Protocol](https://github.com/homeofe/AAHP) work. > Sandboxing did not create the fleet. It made every dependency, identity, and control boundary impossible to leave implicit.