Spectrum-based Failure Attribution for Multi-Agent Systems

FAMAS is a spectrum-based failure attribution framework for multi-agent systems (MASs). It attributes failures to suspicious actions by comparing one failed trajectory against multiple replayed trajectories, and then ranking action-level and agent-level responsibility.

This repository includes:

• spectrum matrix construction
• spectrum-based failure attribution
• evaluation metric computation
• optional LLM-based abstraction and clustering

This repository does not implement trajectory replay itself. It starts from prepared trajectory logs under data/trajectories/.

Pipeline Overview

The end-to-end workflow is:

1. Replay a failed task to obtain multiple rerun trajectories plus one original failed log
2. Abstract raw logs into action-label steps
3. Cluster semantically equivalent action-label steps
4. Build spectrum matrices
5. Run failure attribution
6. Compute evaluation metrics

Replay is external to this repository. For the MAS systems referenced in our workflow, see:

• Magentic-One
• CaptainAgent

Current released scripts assume the task-local trajectory set follows the FAMAS data convention used in this repository: 0.jsonl to 19.jsonl are rerun trajectories, and 20.jsonl is the original failed log.

Setup

conda env create -f environment.yml
conda activate FAMASenv

All commands in this README are expected to be run from the repository root.

Quick Start: Spectrum Only

For a first run, we recommend starting from the spectrum stage only.

The repository already includes clustered trajectories under data/trajectories/.../llm_cluster/, so you can directly build spectrum matrices and run failure attribution without replay or LLM-based abstraction.

This is the recommended first experience because abstraction and clustering are much slower:

• they require repeated external LLM calls across many trajectory files
• each task contains multiple runs, so the total number of LLM requests grows quickly
• incomplete or malformed LLM outputs may trigger automatic retries
• they require API configuration and incur latency and cost

To avoid overwriting existing outputs, run:

python code/run_full_pipeline.py \
  --matrix_root tmp/quickstart_sbfl_matrixs \
  --results_root tmp/quickstart_results \
  --metrics_root tmp/quickstart_metric \
  --rebuild-matrix

This command will:

• build spectrum matrices from the existing clustered trajectories
• run failure attribution for all benchmarks
• generate action-level and agent-level metrics

Main outputs:

• tmp/quickstart_sbfl_matrixs/
• tmp/quickstart_results/
• tmp/quickstart_metric/

Repository Data Layout

The main data directories are:

data/
├── trajectories/
│   └── <source_group>/<task_id>/
│       ├── jsonl/
│       ├── answers.txt
│       ├── llm_path/      # optional abstraction output
│       └── llm_cluster/   # clustered trajectories used by spectrum analysis
└── expected_results/

Notes:

• data/trajectories/ stores raw replay logs and task-local preprocessing outputs
• data/expected_results/ stores benchmark-level ground truth for evaluation
• spectrum matrices are generated on demand and are not tracked in the repository

Important distinction:

• trajectory data is stored by source group, such as gaia_level1, gaia_level3, assistantbench_hard, algorithmgenerated_plan, and algorithmgenerated_normal
• spectrum analysis and evaluation are reported under two benchmark groups: hand_crafted and algorithm_generated
• during matrix construction, source groups are merged into those two benchmark groups

Abstraction and Clustering

If you want to regenerate llm_path/ and llm_cluster/, use the abstraction pipeline.

Supported LLM providers:

• openai
• deepseek

You can pass API keys directly with --api-key, or use environment variables:

export OPENAI_API_KEY=YOUR_OPENAI_API_KEY
export DEEPSEEK_API_KEY=YOUR_DEEPSEEK_API_KEY

Run abstraction first and clustering second for all source groups under data/trajectories/:

python code/run_abstraction_pipeline.py \
  --trajectories-root data/trajectories \
  --base-url https://api.openai.com/v1 \
  --model gpt-4o-mini \
  --provider openai

Or with DeepSeek:

python code/run_abstraction_pipeline.py \
  --trajectories-root data/trajectories \
  --base-url https://api.deepseek.com/v1 \
  --model deepseek-chat \
  --provider deepseek

Per task, the pipeline runs:

1. abstraction over all jsonl/*.jsonl
2. clustering over the generated abstraction results

Generated task-local outputs:

• llm_path/: abstracted action-label trajectories
• llm_cluster/: clustered action-label trajectories
• abstractor.log: abstraction log
• cluster.log: clustering log

If your goal is only to reproduce the spectrum stage or inspect the final localization outputs, you can skip this stage entirely and use the precomputed llm_cluster/ directories already included in the repository.

Spectrum Matrix Construction

To build spectrum matrices from clustered trajectories:

python code/spectrum/sbfl_martix.py

This reads llm_cluster/ from each task directory, merges source groups into the two benchmark groups, and writes matrices to:

data/sbfl_matrixs/
├── hand_crafted/
└── algorithm_generated/

data/sbfl_matrixs/ is generated at runtime. It is not included as tracked repository data.

Failure Attribution

To run failure attribution manually:

python code/spectrum/failure_attribution.py -i [input_dir] -o [output_dir]

Examples:

# build matrices first if `data/sbfl_matrixs/` does not exist yet
python code/spectrum/sbfl_martix.py

python code/spectrum/failure_attribution.py \
  -i data/sbfl_matrixs/algorithm_generated \
  -o output/results/algorithm_generated

python code/spectrum/failure_attribution.py \
  -i data/sbfl_matrixs/hand_crafted \
  -o output/results/hand_crafted

Formula Configuration

The default spectrum formula is configured in:

code/spectrum/config/formula.json

Current default:

{
  "name": "kulczynski2",
  "use_lambda": true,
  "use_beta": true,
  "use_gamma": true,
  "lambda_value": 0.9
}

Supported formula names currently include:

• ochiai
• tarantula
• jaccard
• dstar2
• kulczynski2

Fields:

• name: suspiciousness formula name
• use_lambda: whether to use the lambda-weighted count variant
• use_beta: whether to apply the beta adjustment
• use_gamma: whether to apply the gamma adjustment
• lambda_value: lambda used in weighted counting

This config is read automatically by:

• python code/spectrum/failure_attribution.py ...
• python code/run_full_pipeline.py ...
• python code/run_subsample_experiment.py ...

Metric Computation

To compute action-level and agent-level rankings from localization results:

python code/evaluation/metric.py --fl_dir [fl_results_dir] --output_dir [output_dir]

Examples:

python code/evaluation/metric.py \
  --fl_dir output/results/hand_crafted \
  --output_dir output/metric

python code/evaluation/metric.py \
  --fl_dir output/results/ \
  --output_dir output/metric

One-Click Commands

Run the full spectrum-only pipeline for all benchmarks:

python code/run_full_pipeline.py --rebuild-matrix

Run the same pipeline without overwriting existing outputs:

python code/run_full_pipeline.py \
  --matrix_root tmp/sbfl_matrixs \
  --results_root tmp/results \
  --metrics_root tmp/metric \
  --rebuild-matrix

Run the k-trajectory subsampling experiment for t repeats:

python code/run_subsample_experiment.py --k 5 --times 20

Example with temporary outputs:

python code/run_subsample_experiment.py \
  --k 5 \
  --times 20 \
  --matrix_root tmp/sbfl_matrixs \
  --output_root tmp/experiments/subsample_k5_t20

This experiment samples k trajectories from the 20 rerun trajectories for each task, always keeps the original failed log, repeats the process t times, and reports min, max, and average metric totals across runs.