Skip to content
← All AI workflows

SWE-bench: evaluate code patches in Docker test environments

SWE-bench documents a Docker-based test harness for submitted code patches. Evaluators choose benchmark instances, prepare a predictions file and inspect test reports and container logs. Results depend on the chosen dataset and setup, not general software quality.

From the source

Public documentation page from SWE-bench
SWE-bench · Captured 2026-10-14Read the original

How it works

  • The original requires benchmark instances and predictions in the documented format, plus a supported host, Docker and sufficient compute.
  • Fix the dataset, split, instance IDs and harness version before comparison. Report resolved over evaluated instances for that pinned run.
  • Retain reports and logs. Separate setup failures from test failures instead of assigning all failures to agent reasoning.
  • Patches are untrusted executable inputs. Evaluation requires isolation, no secrets and limited network access.
  • No predictions were generated, containers built or patches executed.

Workflow guide

  1. Select the dataset, split and fixed instance IDs. Record the harness version and host configuration.

  2. Prepare a predictions file with each instance identifier, model identifier and proposed patch in the documented format.

  3. Have a person approve isolated Docker environments and resource settings. Exclude secrets and limit network access before executing untrusted patches.

  4. Build or obtain required environments. Run the approved harness and keep per-instance reports and container logs.

  5. Separate test failures from setup failures. Repeat questionable instances and have an evaluator review the classifications.

  6. Report resolved instances over evaluated instances for the pinned run. Compare a baseline on the same instances and record compute usage and rerun agreement.

Sources and downloads

Discuss this workflow →

Book a call

A free 30 minute call. Tell us where you are with AI, what your IT allows, and what management needs to see. We'll tell you honestly what we would do first.