From the source

How it works
- The original requires benchmark instances and predictions in the documented format, plus a supported host, Docker and sufficient compute.
- Fix the dataset, split, instance IDs and harness version before comparison. Report resolved over evaluated instances for that pinned run.
- Retain reports and logs. Separate setup failures from test failures instead of assigning all failures to agent reasoning.
- Patches are untrusted executable inputs. Evaluation requires isolation, no secrets and limited network access.
- No predictions were generated, containers built or patches executed.
Workflow guide
Select the dataset, split and fixed instance IDs. Record the harness version and host configuration.
Prepare a predictions file with each instance identifier, model identifier and proposed patch in the documented format.
Have a person approve isolated Docker environments and resource settings. Exclude secrets and limit network access before executing untrusted patches.
Build or obtain required environments. Run the approved harness and keep per-instance reports and container logs.
Separate test failures from setup failures. Repeat questionable instances and have an evaluator review the classifications.
Report resolved instances over evaluated instances for the pinned run. Compare a baseline on the same instances and record compute usage and rerun agreement.