[Harbor 4/4] architecture docs, tutorial, and the GAIA example#6
[Harbor 4/4] architecture docs, tutorial, and the GAIA example#6varunursekar wants to merge 1 commit into
Conversation
- docs/harbor/architecture.md — what the integration is, the compiled-task topology, the two evaluation modes, the component map, and the leaderboard-integrity model. - docs/harbor/tutorial.md — build and run an optimization task end to end (both modes, the agent-side protocol), and a Harbor section in the README. - examples/gaia-optimization — a Mode-B example optimizing a GaiaAgent (a thin Terminus2 subclass with an editable prompt) on gaia/gaia via a nested harbor run on Modal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| def version(self) -> str: | ||
| return "0.1.0" |
There was a problem hiding this comment.
version() signature mismatch with name()
name() is declared as a @staticmethod but version() is an instance method. If Terminus2 defines version() as a @staticmethod (the typical pattern when name() is also static), then GaiaAgent.version() won't properly override it when called as GaiaAgent.version() on the class rather than on an instance — the base class static will shadow it. This is worth aligning with however Terminus2 declares version().
Prompt To Fix With AI
This is a comment left during a code review.
Path: vero/examples/gaia-optimization/src/gaia_agent/agent.py
Line: 30-31
Comment:
`version()` signature mismatch with `name()`
`name()` is declared as a `@staticmethod` but `version()` is an instance method. If `Terminus2` defines `version()` as a `@staticmethod` (the typical pattern when `name()` is also static), then `GaiaAgent.version()` won't properly override it when called as `GaiaAgent.version()` on the class rather than on an instance — the base class static will shadow it. This is worth aligning with however `Terminus2` declares `version()`.
How can I resolve this? If you propose a fix, please make it concise.
There was a problem hiding this comment.
This one is a non-issue, the signatures already match the base class. In Terminus2 (and its BaseAgent ABC), the two methods are intentionally declared differently: name() is a @staticmethod @abstractmethod, while version() is a plain instance method (def version(self) -> str | None:, returning "2.0.0"). GaiaAgent mirrors that exactly: static name(), instance version(). So the override is correct and there's no shadowing.
The comment's premise ("if Terminus2 defines version() as a @staticmethod") doesn't hold against the installed harbor source (harbor/agents/terminus_2/terminus_2.py, lines 354-360, and harbor/agents/base.py, lines 75-81), so no change needed here.
|
|
||
| This lets anyone optimize a coding agent with plain `harbor run`, and makes the result | ||
| leaderboard-gradeable — the optimizer cannot read hidden labels, modify the scorer, or | ||
| bypass its budget. |
There was a problem hiding this comment.
This reads as a hard guarantee ('the optimizer cannot read hidden labels, modify the scorer, or bypass its budget'), but the code makes each best-effort and the shipped GAIA example undercuts the first one (see the build.yaml comment). Suggest softening to something like: 'vero never writes per-sample labels to the agent's volume and meters every agent evaluation; OS-level mechanisms (read-only paths, a root:600 finalize token) keep the scorer and test split out of the agent's reach on a best-effort basis.'
|
|
||
| splits: | ||
| - { split: train, access: non_viewable } # optimizer sees aggregate scores only | ||
| - { split: validation, access: no_access } # hidden; never reaches the optimizer |
There was a problem hiding this comment.
'hidden; never reaches the optimizer' isn't true for this config: build.yaml is git-tracked and agent_repo is ., so vero harbor build seeds this whole file — including these validation task IDs — into /work/agent via git archive HEAD. The optimizer can read the held-out task IDs, and GAIA answers are public. Move the partition out of the agent_repo subtree, or caveat that for public benchmarks the held-out identity is visible (only per-sample scores are withheld). This is the example that backs the headline 'cannot read hidden labels' claim, so it is worth getting airtight.
|
|
||
| - [`docs/harbor/architecture.md`](docs/harbor/architecture.md) — what it is, the topology, and the leaderboard-integrity model. | ||
| - [`docs/harbor/tutorial.md`](docs/harbor/tutorial.md) — build and run a task end to end. | ||
| - [`examples/gsm8k-agent`](examples/gsm8k-agent) (Mode A) and [`examples/gaia-optimization`](examples/gaia-optimization) (Mode B). |
There was a problem hiding this comment.
examples/gsm8k-agent is cited as the Mode A example but it has no build.yaml (it's the older Policy-API example). The Harbor Mode A example that ships a build.yaml is examples/doubler-agent. Repoint here, or add a build.yaml to gsm8k-agent.
| The optimizer is untrusted. Integrity rests on a few mechanisms, all best-effort at | ||
| the OS/process level (a container escape is out of scope): | ||
|
|
||
| - **3-tier split visibility** (`SplitAccessLevel`): `visible` (aggregate + per-sample |
There was a problem hiding this comment.
Worth one explicit line here: tier_for_split defaults any split not listed to viewable (full per-sample results), so omission fails open. Tell authors to list every split explicitly. (Pairs with the protocol.py fail-open comment on #4.)
| - **Commit transfer**: the sidecar `git fetch`es the agent's commit from the mounted | ||
| repo into its *own* repo with hooks disabled and `file://` (object copy, no | ||
| alternates), so the evaluated tree is fully owned by the sidecar and tamper-evident. | ||
| - **Protected scorer / write-access**: the scorer is sidecar-only; `read_only_paths` |
There was a problem hiding this comment.
'the scorer is sidecar-only' holds for Mode B but not Mode A, where the scorer lives in the agent's editable repo, protected only by chown root:root + chmod -R a-w on read_only_paths (which isn't a real tamper control — see #5). Recommend splitting this claim by mode.
2 participants
Draft · Stack 4 of 4 — targets
harbor-3-compiler. Additive, low-risk.docs/harbor/architecture.md— what it is, the compiled-task topology, the two modes, the component map, and the leaderboard-integrity model.docs/harbor/tutorial.md— build + run end to end (both modes, the agent-side protocol); README Harbor section.examples/gaia-optimization— a Mode-B example optimizing aGaiaAgent(thinTerminus2subclass with an editable prompt) ongaia/gaiavia a nested harbor run on Modal.Start your reading here for the big picture, then dive into [1/4]–[3/4].
Stack: [1/4] core → [2/4] sidecar → [3/4] compiler → this.
🤖 Generated with Claude Code
Greptile Summary
This PR is the final stack entry (4/4) for the Harbor integration, adding architecture docs, a tutorial, and a runnable Mode-B example (
gaia-optimization) that optimizes aGaiaAgentprompt against real GAIA tasks via a nestedharbor runon Modal.docs/harbor/architecture.md,docs/harbor/tutorial.md): covers the compiled-task topology, the two evaluation modes (A = vero scores, B = nested Harbor run scores), the trust boundary / leaderboard-integrity model, and the full CLI walkthrough end to end.examples/gaia-optimization): a thinTerminus2subclass that redirects the prompt-template path to an editableprompts/directory, abuild.yamlwiring up Mode B on Modal, and the copied prompt templates that form the optimization surface.Confidence Score: 4/5
Entirely additive — new docs and an example package with no changes to vero core; safe to merge.
The changes are documentation and an example that adds no new runtime paths to vero itself. The two issues found are minor: typos in the XML prompt template (which is the optimization surface — an optimizer would fix them during a run anyway) and a potential @staticmethod vs instance-method mismatch on version() in GaiaAgent that could surface only if Harbor calls GaiaAgent.version() as a class-level static.
src/gaia_agent/agent.py and src/gaia_agent/prompts/terminus-xml-plain.txt have the two flagged issues; all other files are clean.
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant Dev as Developer participant VeroHarbor as vero harbor CLI participant Main as main container (optimizer) participant Sidecar as eval-sidecar (vero harbor serve) participant Modal as Modal (nested harbor run) participant Verifier as tests/test.sh (shared verifier) Dev->>VeroHarbor: vero harbor build -c build.yaml -o /tmp/task VeroHarbor-->>Dev: Harbor task dir (compose + Dockerfiles + instruction.md) Dev->>Main: harbor run -p /tmp/task -a claude-code -e docker activate Main activate Sidecar Note over Sidecar: vero harbor serve starts, writes per-trial admin token (root:600) Main->>Main: optimizer edits prompts/, commits Main->>Sidecar: "POST /eval?split=train" Sidecar->>Sidecar: git fetch commit (file://, hooks disabled) Sidecar->>Modal: harbor run GaiaAgent on train tasks Modal-->>Sidecar: per-task verifier rewards Sidecar-->>Main: aggregate score + remaining budget Note over Main: repeat edits + evals within budget Main->>Verifier: trial end — tests/test.sh runs Verifier->>Sidecar: POST /finalize (admin token) Sidecar->>Sidecar: select best train commit Sidecar->>Modal: harbor run on hidden validation tasks Modal-->>Sidecar: accuracy Sidecar-->>Verifier: reward.json deactivate Sidecar deactivate Main%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant Dev as Developer participant VeroHarbor as vero harbor CLI participant Main as main container (optimizer) participant Sidecar as eval-sidecar (vero harbor serve) participant Modal as Modal (nested harbor run) participant Verifier as tests/test.sh (shared verifier) Dev->>VeroHarbor: vero harbor build -c build.yaml -o /tmp/task VeroHarbor-->>Dev: Harbor task dir (compose + Dockerfiles + instruction.md) Dev->>Main: harbor run -p /tmp/task -a claude-code -e docker activate Main activate Sidecar Note over Sidecar: vero harbor serve starts, writes per-trial admin token (root:600) Main->>Main: optimizer edits prompts/, commits Main->>Sidecar: "POST /eval?split=train" Sidecar->>Sidecar: git fetch commit (file://, hooks disabled) Sidecar->>Modal: harbor run GaiaAgent on train tasks Modal-->>Sidecar: per-task verifier rewards Sidecar-->>Main: aggregate score + remaining budget Note over Main: repeat edits + evals within budget Main->>Verifier: trial end — tests/test.sh runs Verifier->>Sidecar: POST /finalize (admin token) Sidecar->>Sidecar: select best train commit Sidecar->>Modal: harbor run on hidden validation tasks Modal-->>Sidecar: accuracy Sidecar-->>Verifier: reward.json deactivate Sidecar deactivate MainPrompt To Fix All With AI
Reviews (1): Last reviewed commit: "Harbor: architecture docs, tutorial, and..." | Re-trigger Greptile