Recipe Integration

Add a Recipe when a Benchmark task needs deterministic changes to its per-task ExecutionPlan. Prefer one default Recipe per Benchmark that works with every Environment supporting the task’s requirements. Keep optional strategies, such as alternative runner images, separate from provider selection. A Recipe is planning policy. It may select an image, workspace, resources, network settings, or a specialized Benchmark or Harness plan, but it must not open a sandbox, install packages, call a model or provider, run a task, or score a result. Put those side effects in the lifecycle component that owns them.

Choose the Integration Form

Form	Use it for	Registration contract
Built-in Recipe	Public behavior maintained with AgentCompass	Decorate the class with `@RECIPES.register()` and import its module from `agentcompass.recipes`
Trusted external Recipe	Team-specific policy that should be loaded only for selected runs	Export concrete classes from the package root through a non-empty `RECIPE_CLASSES` list or tuple

External Recipe code executes in the AgentCompass process, outside the task sandbox. Only load packages you have reviewed and trust.

Implement the Base Contract

Every Recipe is a zero-argument constructible BaseRecipe with a unique id, matches(), and apply():

from copy import deepcopy

from agentcompass.runtime import BaseRecipe, ExecutionPlan, RunRequest, TaskSpec
from agentcompass.runtime.config.environment import environment_supports_task_image


class ExampleRecipe(BaseRecipe):
    id = "example_task_image"

    def matches(self, req: RunRequest, task: TaskSpec, plan: ExecutionPlan) -> bool:
        _ = task
        return (req.benchmark.id == "example_exact_match"
                and environment_supports_task_image(plan.environment.id))

    def apply(
        self,
        plan: ExecutionPlan,
        req: RunRequest,
        task: TaskSpec,
    ) -> ExecutionPlan:
        _ = req, task
        updated = deepcopy(plan)
        updated.environment.setup.image = updated.environment.setup.image or "python:3.12-slim"
        updated.environment.setup.workdir = updated.environment.setup.workdir or "/workspace"
        return updated

Use matches(req, task, plan) only to decide whether the Recipe applies. It receives the current plan, including changes made by earlier matching Recipes. It must not mutate any argument. Use apply(plan, req, task) to return a new ExecutionPlan. Deep-copy the incoming plan before changing nested Environment, Benchmark, or Harness plan fields. Preserve explicit compatible user values: fill missing fields with setdefault() or an equivalent fallback, and reject an incompatible explicit value with an actionable error instead of silently replacing it. Both methods belong to deterministic planning. Keep them free of network access, file writes, subprocesses, package installation, sandbox creation, model calls, and other externally visible side effects. Use shared fields such as environment.setup, resources, phase environment variables, and network policies. The Environment translates those requirements into native provider settings. Check capabilities rather than listing provider IDs; environment_supports_task_image() reads the registered class without constructing a provider or client. Adding a provider with the same capabilities should require no Benchmark or Recipe changes. Required capabilities are validated by the runtime before allocation; an optional image strategy can fall back to source preparation when images are unsupported.

Register a Built-in Recipe

Place a public implementation under src/agentcompass/recipes/, then register it:

from agentcompass.runtime import RECIPES


@RECIPES.register()
class ExampleRecipe(BaseRecipe):
    ...

Import the module from agentcompass.recipes so its decorator runs. The registry uses id as its key, and each registered ID must be unique. Select a Recipe by that ID; run records store the same ID. Keep native provider translation in the Environment and task-specific planning in the Recipe. Built-in Recipes must target only public providers and public infrastructure. Keep organization-specific deployment policy in a trusted external package instead.

Package a Trusted External Recipe

An external Recipe directory is a Python package, not a loose Python file:

company_recipes/
├── __init__.py
└── docker_recipe.py

Define the class without decorating it with the process-global RECIPES registry:

# company_recipes/docker_recipe.py
from copy import deepcopy

from agentcompass import BaseRecipe, ExecutionPlan, RunRequest, TaskSpec


class CompanyDockerRecipe(BaseRecipe):
    id = "company_docker"

    def matches(self, req: RunRequest, task: TaskSpec, plan: ExecutionPlan) -> bool:
        _ = task
        return req.benchmark.id == "example_exact_match" and plan.environment.id == "docker"

    def apply(self, plan: ExecutionPlan, req: RunRequest, task: TaskSpec) -> ExecutionPlan:
        _ = req, task
        updated = deepcopy(plan)
        updated.environment.setup.image = updated.environment.setup.image or "python:3.12-slim"
        updated.environment.setup.workdir = updated.environment.setup.workdir or "/workspace"
        return updated

Export the same class from the package root:

# company_recipes/__init__.py
from .docker_recipe import CompanyDockerRecipe

RECIPE_CLASSES = (CompanyDockerRecipe,)

The loader requires all of the following:

the directory exists and contains __init__.py;
RECIPE_CLASSES is a non-empty list or tuple;
every item is a concrete BaseRecipe subclass with a non-empty, unique id;
every class supports zero-argument construction.

Relative directories resolve from the current working directory. AgentCompass canonicalizes and de-duplicates paths, clones the built-in registry for the run, and appends external classes. A duplicate ID, including a collision with a built-in Recipe, fails during registration. After adding example_exact_match and example_answer from the Benchmark and Harness implementation tutorials, load and allow the Recipe for their deterministic task:

uv run agentcompass run example_exact_match example_answer unused-model \
  --env docker \
  --benchmark-params '{"sample_ids":["capital-france"]}' \
  --harness-params '{"answer":"Paris"}' \
  --recipe-dir ./company_recipes \
  --recipe company_docker \
  --task-concurrency 1 \
  --no-enable-analysis

--recipe is an allowlist, not a force-run switch. The Recipe still needs to return True from matches(). See Recipes for the CLI, SDK, configuration-file, and orchestration equivalents.

Understand Application Order

The current Planner builds one default plan for an attempt and then walks the run-local Recipe registry in insertion order:

If execution.enabled_recipes is non-empty, skip entries whose registered ID is absent from that allowlist.
Construct the Recipe with no arguments.
Call matches() with the plan produced so far.
When it matches, replace the current plan with the value returned by apply() and record the Recipe id in applied_recipes.
Continue to the next registry entry, which can inspect the updated plan.

Built-ins retain their registration insertion order. External classes are appended after built-ins, following runtime.recipe_dirs order and then each package’s RECIPE_CLASSES order. All matching Recipes are applied; the Planner does not automatically resolve overlapping writes. BaseRecipe currently declares priority and enabled_by_default, but the Planner does not consult either attribute. Do not claim that priority changes ordering or use either attribute as an enablement mechanism. Current participation is controlled by registration, execution.enabled_recipes, and matches(). Treat registration order as current execution semantics, not as a substitute for clear ownership. Avoid Recipes that write the same fields, and do not make correctness depend on an unrelated module import order.

Validate the Integration

First confirm that the package loads and the ID is present. There is currently no agentcompass list recipe command, so inspect the run-local registry directly for an external package:

uv run python -c 'from agentcompass.runtime.recipes import build_run_recipe_registry; print(build_run_recipe_registry(["./company_recipes"]).names())'

Then run one known task with --task-concurrency 1, --max-retries 0, and DEBUG logging. Verify:

matches() is false for an unrelated Benchmark or Environment;
the expected ID appears in each relevant attempt under resolved_execution_plans in run_info.json and in the terminal result’s top-level applied_recipes list;
each saved attempt’s resolved execution plan contains the intended values;
an explicit compatible image, workspace, resource, or network value survives Recipe application;
the selected Environment can build its config from the adjusted plan and clean up after both success and failure.

If two Recipes could match, validate both combinations or redesign them so their ownership does not overlap. Follow Testing and Validation for the repository checks and single-task evidence expected in a pull request.

Completion Checklist

The behavior belongs in deterministic per-task planning rather than Benchmark, Harness, Environment, or runtime mechanics.
matches() and apply() are deterministic and side-effect free.
apply() returns a copied plan and preserves explicit user intent.
The Recipe ID is unique, and the correct built-in or RECIPE_CLASSES registration path loads it.
Matching and non-matching cases, explicit overrides, resolved plans, and cleanup have been verified.
Public user behavior and supported combinations are documented in both languages.

Architecture

Extensions

Contributing

Choose the Integration Form

Implement the Base Contract

Register a Built-in Recipe

Package a Trusted External Recipe

Understand Application Order

Validate the Integration

Completion Checklist

​Choose the Integration Form

​Implement the Base Contract

​Register a Built-in Recipe

​Package a Trusted External Recipe

​Understand Application Order

​Validate the Integration

​Completion Checklist

Choose the Integration Form

Implement the Base Contract

Register a Built-in Recipe

Package a Trusted External Recipe

Understand Application Order

Validate the Integration

Completion Checklist