A live benchmark of 957 performance bottlenecks mined from scientific Python repositories — pairing every task with expert patches and ~265 community workloads.
Benchmark Design
Each FormulaCode task evaluates the ability of an agent to optimize a real-world codebase under strict correctness constraints. A task begins with a baseline repository, which represents the unmodified implementation. The agent operates on the baseline and produces a modified version of the repository by making arbitrary repository-level edits.
Performance evaluation proceeds by executing the full set of workloads on both the baseline and the agent-modified code and comparing their measured outcomes. Improving performance on one workload may degrade performance on others. As a result, optimization in FormulaCode is inherently multi-objective: agents must reason about trade-offs across subsystems and deliver improvements that are broad and consistent rather than localized to a single execution path.
Dataset Construction
FormulaCode consists of multi-workload real-world code optimization problems from 70 repositories. We developed an automated four-stage pipeline that extracts these problems:
Repository Scraping
We crawl GitHub repositories with high-quality expert-defined performance workloads.
Attribute Filtering
We filter out candidate pull requests where the primary intent was not performance related, using rule-based and LLM-based filters.
Environment Synthesis
We synthesize environment building scripts using a reflexive LLM agent so that terminal interface tools function correctly.
Statistical Validation
We filter all candidate PRs that do not show statistically significant improvement in performance workloads.
Key Findings
Agents Improve Runtime but Underperform Experts
Agents generally can improve run-time performance, but perform worse than human experts.
Local vs. Global Optimization
Agents are better at local or function-level optimization, rather than repository-level optimization.
Optimization Strategy Strengths
Agents excel at using specific optimization strategies (e.g., parallelizing or batching) and struggle with others (e.g., vectorized operations).
Long-Tail Repository Performance
Agent performance relative to experts can vary dramatically by popularity of the repository, performing worst on the 4th quintile and best on the 2nd quintile.
Cost Efficiency
Despite being more expensive per call, agents using frontier LLMs are overall more cost effective than those using open weights models.
Multi-Workload Tradeoffs
Compared to human experts, agents make less favorable performance-cost trade-off decisions.
Dataset growth
Live snapshot from api.formulacode.org. 1,555 verified tasks span 111 months of real-world performance work, last refreshed Mar 26.
Problems by repository
Each tile is a repository; size is the number of verified tasks it contributes. 1,555 tasks across 133 repos.
Results
Agent advantage across the full benchmark and at each aggregation level. Positive bars beat the human expert; negative bars trail.
Overall: every agent underperforms the expert. OpenHands · Claude 4.0 Sonnet is closest at -0.0112; Terminus 2 · GPT-5 is furthest behind at -0.0504.
Compact Leaderboard
Don't see your model? Submit it!
To evaluate an agent on FormulaCode, follow the Installation instructions and run:
fceval run -d formulacode -a [your-agent-name] -m [your-model-name]