Skip to main content
AgentCompass records every unresolved problem in a task attempt as an issue with a severity. The severity decides whether the runtime retries the work, whether the task keeps a valid score, and whether the run succeeds. This page collects those rules; set the retry options themselves in Run Controls.

​
Read an Issue

Every attempt in a task result stores an issues list, which is [] when nothing is unresolved:
The full traceback and exception chain are saved in the same attempt’s artifacts.execution_diagnostics. Issues are independent of the score: an attempt with an ERROR or WARNING can still carry valid Benchmark metrics. The attempt status is derived from the phases of its FATAL and ERROR issues. Issues in setup, run, or collect produce run_error; issues in evaluate produce eval_error; both produce run_error_or_eval_error. An attempt with only WARNING issues stays completed. Use status to see where an attempt failed and severity to see how serious the failure is.

​
How Failures Are Classified

Each component classifies a failure where it occurs, so the same exception type can have different severities in different places. Component pages describe the classifications specific to each Benchmark or Harness, such as which Judge responses count as invalid.

​
Judges and Verifiers

A Benchmark scored by an LLM Judge cannot produce a valid observation when the Judge fails. A failed Judge request, or a response that breaks the required output format, such as invalid JSON or missing grading fields, is FATAL rather than a wrong answer. A verifier owned by the Benchmark, such as a test script, works differently. When its reward is missing or cannot be parsed, the task is scored as fail or 0 with a WARNING. A confirmed failure of trusted verifier setup, such as an evaluation Environment that cannot be prepared, remains FATAL.

​
Timeouts and Limits

A Harness receives the agent execution budget and reports its own timeout as ERROR. If the Harness does not stop within that budget plus run_timer_compensation_seconds, the runtime stops it and records ERROR run_failed. Placeholder files that the stopped Harness did not finish writing do not add a FATAL issue. A Harness that ends at its turn or step limit records ERROR harness_max_iterations, or harness_limit_reached when the limit type cannot be confirmed. In both timeout and limit cases, AgentCompass still collects the output and scores it. When evaluation_timeout_seconds expires, the runtime records ERROR. Some Benchmarks declare failing values for this case, so the task counts as failed instead of missing an observation: Other Benchmarks leave the observation missing. These values apply only when the runtime’s evaluation timeout expires; they are not used for other evaluation errors or cancellation.

​
Retry Failed Work

​
Which Issues Are Retried

max_retries sets the number of extra executions for each logical attempt; setup, execution, collection, and evaluation share this budget. It defaults to 0, so no failure is retried: a single transient failure, such as one HTTP 503 from a Judge, leaves a final FATAL that invalidates the task and fails the run. Use max_retries: 2 to tolerate transient service failures. Patterns search each issue’s message and code, not the traceback. An omitted, null, or empty retry_pattern_list retries only FATAL issues.
Earlier versions retried every error when the pattern list was omitted or null, and retried nothing when it was empty. Both now retry only FATAL issues. To retry every ERROR again, set retry_pattern_list to [".*"].
Retrying cannot fix an incorrect endpoint, a missing credential, or an invalid configuration; correct the configuration first. Cancelling a run, or reaching the overall --timeout-seconds deadline, is not retried: the runtime starts no new retry and releases the resources it opened.

​
What a Retry Repeats

A retry replaces work within the current logical attempt. It does not add a scoring sample, and completed sibling attempts are not rerun. Each retry writes its discarded result to attempt-<n>/retries/; see Retry Details. Progress files record the attempt_retry event; see progress.jsonl.

​
Wait Between Retries

Every runtime retry waits with exponential backoff. retry_min_delay_seconds defaults to 5, and retry_max_delay_seconds defaults to 60. Both must be finite positive numbers, with the maximum at least the minimum. The backoff starts at the minimum, doubles for each retry, and stops growing at the maximum: 5 → 10 → 20 → 40 → 60 → 60 … seconds by default. The actual delay is sampled uniformly from 50% to 100% of that value, so the first retry waits 2.5–5 seconds by default. Whole-attempt and evaluation-only retries share the logical attempt’s sequence; a new attempt or run starts at the minimum again. Waiting starts after required resource cleanup and supports cancellation. It counts toward the overall request deadline, but not the run or evaluation phase timeout. Whole-attempt retries release concurrency permits before waiting; evaluation-only retries retain their current permits. Progress reports the retry_wait phase, and retry diagnostics and events record the selected delay as retry_delay_seconds.

​
How Failures Affect Scores

ERROR and WARNING issues keep the Benchmark’s valid observations. The runtime never fills a missing observation with false or 0; only the Benchmark assigns failing values, as in the WARNING and evaluation timeout cases above. A FATAL issue that remains after retries in any logical attempt invalidates the whole task. The task is counted as invalidated in every metric series, category, and derived metric, and none of its observations are used. Each series satisfies evaluated + unavailable + invalidated = total; error is counted separately. See Metrics and Aggregation for how attempts are reduced.

​
When a Run Fails

A run fails when at least one task ends with a final FATAL issue. AgentCompass still writes every result file:
  • metrics.json sets evaluation_failed to true and every official value to null. reference_value is computed from tasks that were not invalidated, and is null when none can be scored.
  • summary.md states that there is no official score and labels the remaining values as references.
  • agentcompass run exits with a nonzero status. In launch, the request is marked failed, other requests continue, and the orchestration ends with partial_failure and exit status 1.
  • In the Python SDK, run_evaluation raises RuntimeError with the result paths in its message, while launch returns an OrchestrationResult with the request marked failed.
Reference values are for diagnosis and are not comparable to official scores. Fix the cause, then rerun with --reuse to repeat only the failed work. A run whose tasks have only ERROR or WARNING issues succeeds and publishes official values.

​
Reuse After Failures

--reuse applies the current retry policy to saved results:
  • Results without FATAL issues or ERROR issues that match the current patterns are reused unchanged.
  • When max_retries is greater than 0, attempts with retryable issues run again, and the rerun uses one retry from the current budget. Previous retry counts remain only as history.
  • With max_retries: 0, failed results are reused unchanged; set a retry budget to rerun them.
  • In none or fresh evaluation mode, a saved agent result is restored when only evaluation failed, so the agent does not run again.
Results from earlier formats are converted when read. A historical failure whose severity cannot be determined from the saved evidence is not reused, and its task runs again.

​
Find Failure Details

Follow Diagnose a Failed Run for the order in which to inspect these files.