> ## Documentation Index
> Fetch the complete documentation index at: https://opencompass-docs-preview-pr-354.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Error Handling

AgentCompass records every unresolved problem in a task attempt as an issue with a severity. The severity decides whether the runtime retries the work, whether the task keeps a valid score, and whether the run succeeds. This page collects those rules; set the retry options themselves in [Run Controls](/en/user_guide/using_agentcompass/run_controls#retry-only-transient-failures).

| What you want to know | Where to look |
| - | - |
| What an issue contains | [Read an Issue](#read-an-issue) |
| Why a failure is FATAL, ERROR, or WARNING | [How Failures Are Classified](#how-failures-are-classified) |
| Which failures are retried and what is repeated | [Retry Failed Work](#retry-failed-work) |
| Whether a failed task still has a score | [How Failures Affect Scores](#how-failures-affect-scores) |
| Why a run has no official score or exits nonzero | [When a Run Fails](#when-a-run-fails) |
| What `--reuse` does with failed tasks | [Reuse After Failures](#reuse-after-failures) |

## Read an Issue

Every attempt in a [task result](/en/user_guide/other_features/results/task_results#attempt-level-fields) stores an `issues` list, which is `[]` when nothing is unresolved:

```json theme={"system"}
{"severity": "fatal", "phase": "evaluate", "code": "judge_failed", "message": "HTTP 503: ..."}
```

| Field | Meaning |
| - | - |
| `severity` | `fatal`, `error`, or `warning`. |
| `phase` | Where the issue occurred: `setup`, `run`, `collect`, `evaluate`, or `cleanup`. |
| `code` | Stable identifier such as `judge_failed`, `model_timeout`, or `environment_failed`. Retry patterns can match it. |
| `message` | One-line summary with recognized credentials redacted. |

The full traceback and exception chain are saved in the same attempt's `artifacts.execution_diagnostics`. Issues are independent of the score: an attempt with an ERROR or WARNING can still carry valid Benchmark metrics.

The attempt `status` is derived from the phases of its FATAL and ERROR issues. Issues in `setup`, `run`, or `collect` produce `run_error`; issues in `evaluate` produce `eval_error`; both produce `run_error_or_eval_error`. An attempt with only WARNING issues stays `completed`. Use `status` to see where an attempt failed and `severity` to see how serious the failure is.

## How Failures Are Classified

Each component classifies a failure where it occurs, so the same exception type can have different severities in different places.

| Severity | Meaning | Typical causes |
| - | - | - |
| FATAL | The evaluation infrastructure or a required service failed, so the result does not measure the agent. | Setup failures; Environment creation, commands, file transfer, or network policy changes; external tool services such as search; model authentication, quota, or server errors (HTTP 401, 402, 403, 429, or 5xx); Judge request failures or Judge output that breaks the required format; Harness crashes other than timeouts. |
| ERROR | The agent did not finish normally, but its output may still be scored. | Model or Harness timeouts; no valid model output; a Harness reaching its turn or step limit; unexpected exceptions in a Benchmark's own inference loop or in scoring code. |
| WARNING | A diagnostic that affects neither retries nor score validity. | A verifier reward that is missing or cannot be parsed, and a missing required deliverable, both scored as fail or `0`; a failed diagnostic copy; an Environment cleanup failure. |

Component pages describe the classifications specific to each Benchmark or Harness, such as which Judge responses count as invalid.

### Judges and Verifiers

A Benchmark scored by an LLM Judge cannot produce a valid observation when the Judge fails. A failed Judge request, or a response that breaks the required output format, such as invalid JSON or missing grading fields, is FATAL rather than a wrong answer.

A verifier owned by the Benchmark, such as a test script, works differently. When its reward is missing or cannot be parsed, the task is scored as fail or `0` with a WARNING. A confirmed failure of trusted verifier setup, such as an evaluation Environment that cannot be prepared, remains FATAL.

### Timeouts and Limits

A Harness receives the agent execution budget and reports its own timeout as ERROR. If the Harness does not stop within that budget plus [`run_timer_compensation_seconds`](/en/user_guide/using_agentcompass/timeouts#allow-a-harness-to-finish-handling-its-timeout), the runtime stops it and records ERROR `run_failed`. Placeholder files that the stopped Harness did not finish writing do not add a FATAL issue.

A Harness that ends at its turn or step limit records ERROR `harness_max_iterations`, or `harness_limit_reached` when the limit type cannot be confirmed. In both timeout and limit cases, AgentCompass still collects the output and scores it.

When `evaluation_timeout_seconds` expires, the runtime records ERROR. Some Benchmarks declare failing values for this case, so the task counts as failed instead of missing an observation:

| Benchmark | Values recorded on evaluation timeout |
| - | - |
| [Terminal-Bench 2](/en/user_guide/modules/benchmarks/terminal_bench_2), [2.1](/en/user_guide/modules/benchmarks/terminal_bench_2_1), and [Verified](/en/user_guide/modules/benchmarks/terminal_bench_2_verified) | `correct=false` |
| [SWE-bench Verified](/en/user_guide/modules/benchmarks/swebench_verified), [Multilingual](/en/user_guide/modules/benchmarks/swebench_multilingual), [Pro](/en/user_guide/modules/benchmarks/swebench_pro), and [Pro Verified](/en/user_guide/modules/benchmarks/swebench_pro_verified) | `correct=false` |
| [DeepSWE](/en/user_guide/modules/benchmarks/deepswe) | `correct=false`, `reward=0` |
| [SkillsBench](/en/user_guide/modules/benchmarks/skillsbench) | `score=0`, `passed=false` |
| [Frontier-SWE](/en/user_guide/modules/benchmarks/frontier_swe) | `score=0`, `correctness=0` |

Other Benchmarks leave the observation missing. These values apply only when the runtime's evaluation timeout expires; they are not used for other evaluation errors or cancellation.

## Retry Failed Work

### Which Issues Are Retried

| Severity | Retried while budget remains |
| - | - |
| FATAL | Always. |
| ERROR | Only when a `retry_pattern_list` regular expression matches its `message` or `code`. |
| WARNING | Never. |

`max_retries` sets the number of extra executions for each logical attempt; setup, execution, collection, and evaluation share this budget. It defaults to `0`, so no failure is retried: a single transient failure, such as one HTTP 503 from a Judge, leaves a final FATAL that invalidates the task and fails the run. Use `max_retries: 2` to tolerate transient service failures.

Patterns search each issue's `message` and `code`, not the traceback. An omitted, `null`, or empty `retry_pattern_list` retries only FATAL issues.

<Note>
  Earlier versions retried every error when the pattern list was omitted or `null`, and retried nothing when it was empty. Both now retry only FATAL issues. To retry every ERROR again, set `retry_pattern_list` to `[".*"]`.
</Note>

Retrying cannot fix an incorrect endpoint, a missing credential, or an invalid configuration; correct the configuration first. Cancelling a run, or reaching the overall `--timeout-seconds` deadline, is not retried: the runtime starts no new retry and releases the resources it opened.

### What a Retry Repeats

A retry replaces work within the current logical attempt. It does not add a scoring sample, and completed sibling attempts are not rerun.

| Where the failure occurs | What the retry repeats |
| - | - |
| Setup, Environment, agent execution, or result and artifact collection | The whole attempt, in a new task Environment with the same plan. |
| Evaluation with `evaluation_environment_mode: none` | Only scoring, using the saved task input and agent result. |
| Evaluation with `evaluation_environment_mode: fresh` | Only scoring, in a newly created evaluation Environment. |
| Evaluation with `evaluation_environment_mode: reuse` | The whole attempt, because scoring shares the task Environment. |

Each retry writes its discarded result to `attempt-<n>/retries/`; see [Retry Details](/en/user_guide/other_features/results/task_results#retry-details). Progress files record the `attempt_retry` event; see [`progress.jsonl`](/en/user_guide/other_features/results/run_records#progressjsonl).

### Wait Between Retries

Every runtime retry waits with exponential backoff. `retry_min_delay_seconds` defaults to `5`, and `retry_max_delay_seconds` defaults to `60`. Both must be finite positive numbers, with the maximum at least the minimum. The backoff starts at the minimum, doubles for each retry, and stops growing at the maximum: `5 → 10 → 20 → 40 → 60 → 60 …` seconds by default. The actual delay is sampled uniformly from 50% to 100% of that value, so the first retry waits 2.5–5 seconds by default. Whole-attempt and evaluation-only retries share the logical attempt's sequence; a new attempt or run starts at the minimum again.

Waiting starts after required resource cleanup and supports cancellation. It counts toward the overall request deadline, but not the run or evaluation phase timeout. Whole-attempt retries release concurrency permits before waiting; evaluation-only retries retain their current permits. Progress reports the `retry_wait` phase, and retry diagnostics and events record the selected delay as `retry_delay_seconds`.

## How Failures Affect Scores

ERROR and WARNING issues keep the Benchmark's valid observations. The runtime never fills a missing observation with false or `0`; only the Benchmark assigns failing values, as in the WARNING and evaluation timeout cases above.

A FATAL issue that remains after retries in any logical attempt invalidates the whole task. The task is counted as `invalidated` in every metric series, category, and derived metric, and none of its observations are used. Each series satisfies `evaluated + unavailable + invalidated = total`; `error` is counted separately. See [Metrics and Aggregation](/en/user_guide/other_features/results/metrics_aggregation#treat-failed-and-missing-attempts-explicitly) for how attempts are reduced.

## When a Run Fails

A run fails when at least one task ends with a final FATAL issue. AgentCompass still writes every result file:

* `metrics.json` sets `evaluation_failed` to `true` and every official `value` to `null`. `reference_value` is computed from tasks that were not invalidated, and is `null` when none can be scored.
* `summary.md` states that there is no official score and labels the remaining values as references.
* `agentcompass run` exits with a nonzero status. In `launch`, the request is marked `failed`, other requests continue, and the orchestration ends with `partial_failure` and exit status `1`.
* In the Python SDK, `run_evaluation` raises `RuntimeError` with the result paths in its message, while `launch` returns an `OrchestrationResult` with the request marked `failed`.

Reference values are for diagnosis and are not comparable to official scores. Fix the cause, then rerun with `--reuse` to repeat only the failed work. A run whose tasks have only ERROR or WARNING issues succeeds and publishes official values.

## Reuse After Failures

[`--reuse`](/en/user_guide/using_agentcompass/run_controls#resume-an-interrupted-run) applies the current retry policy to saved results:

* Results without FATAL issues or ERROR issues that match the current patterns are reused unchanged.
* When `max_retries` is greater than `0`, attempts with retryable issues run again, and the rerun uses one retry from the current budget. Previous retry counts remain only as history.
* With `max_retries: 0`, failed results are reused unchanged; set a retry budget to rerun them.
* In `none` or `fresh` evaluation mode, a saved agent result is restored when only evaluation failed, so the agent does not run again.

Results from earlier formats are converted when read. A historical failure whose severity cannot be determined from the saved evidence is not reused, and its task runs again.

## Find Failure Details

| What you need | Where to find it |
| - | - |
| Tasks grouped by final severity | `details/fatal/`, `details/error/`, and `details/normal/` in the run directory; WARNING-only tasks are in `normal`. See [Task Results](/en/user_guide/other_features/results/task_results). |
| Issues and tracebacks | Each attempt's `issues` and `artifacts.execution_diagnostics`. |
| Discarded retries | `attempt-<n>/retries/` under the task directory. |
| Counts during the run | `failed_tasks`, `error_tasks`, and `warning_tasks` in [`progress.json`](/en/user_guide/other_features/results/run_records#progressjson). |
| Run-level validity | `evaluation_failed` and series counts in [`metrics.json`](/en/user_guide/other_features/results/summary_analysis#metricsjson). |

Follow [Diagnose a Failed Run](/en/user_guide/other_features/results/run_records#diagnose-a-failed-run) for the order in which to inspect these files.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.