Mercor |
Multi-Domain |
SFSanta Clara |
Thiel Fellowship, Harvard, Georgetown |
  Brendan Foody F., Adarsh Hiremath H., Surya Midha M. |
250+ |
APEX |
Handshake |
Multi-Domain |
SFNYC, Bangalore, Berlin |
Palantir, Meta, Scale AI |
       Garrett Lord L., Ben Christensen C., Scott Ringwelski R.RL envs: Paco Guzmán G., Sahil Bhaiwala B., Jack Bratches B., Nishant Ranka R., Yang Zhao Z. |
250+ |
Gandalf the Grader |
Scale |
Multi-Domain Code |
SFNYC, DC, London |
Stanford, Coinbase, YC |
RL envs: Chetan Rane R. |
250+ |
SWE-bench Pro |
Turing |
Code |
SFPalo Alto, Gurugram |
Stanford, Rover, Yahoo |
 Jonathan Siddharth S., Vijay Krishnan K. |
250+ |
VLMBench |
Toloka |
Multi-Domain |
AmsterdamBelgrade |
Yandex, Microsoft, McKinsey |
Olga Megorskaya M. |
250+ |
Toloka Arena |
Surge |
Multi-Domain |
SFNYC, Seattle |
Google, Meta, Twitter |
 Edwin Chen C.RL envs: Nick Heiner H. |
101-250 |
CoreCraft |
Snorkel |
Multi-Domain Code |
SFNYC |
Stanford, UW, Meta |
      Alex Ratner R., Braden Hancock H., Christopher Ré R., Paroma Varma V., Henry Ehrenberg E.RL envs: Vincent Sunn Chen C., Frederic Sala S. |
101-250 |
Senior SWE-Bench |
micro1 |
Multi-Domain |
SF |
Stanford, Berkeley |
Ali Ansari A. |
101-250 |
Realm |
Revelo |
Code |
SFSão Paulo |
Stanford, Microsoft, Turing |
  Lucas Mendes M., Rafael Timbó T., Frédéric Monnier M. |
101-250 |
Code Index |
AfterQuery |
Multi-Domain |
SFNYC, Seattle |
Wharton, Penn, UBC |
 Carlos Georgescu G., Spencer Mateega M. |
51-100 |
AppBench |
Latch |
Science |
SF |
Berkeley, Google, Asimov |
  Alfredo Andere A., Kenny Workman W., Kyle Giffin G. |
51-100 |
benchmarks.bio |
Patronus AI |
Multi-Domain Code |
SF |
Meta AI, Google, Amazon |
 Anand Kannappan K., Rebecca Qian Q. |
51-100 |
FinanceBench, TRAIL |
Contra Labs |
Creative Code |
SFNYC |
Midjourney, Stanford, Mercor |
  Ben Huffman H., Gajus Kuizinas K.Labs: Allison Nulty N. |
26-50 |
Human Creativity Benchmark |
Datacurve |
Code |
SF |
Waterloo |
 Serena Ge G., Charley Lee L. |
26-50 |
DeepSWE |
Halluminate |
Finance Enterprise |
SF |
Capital One, Meta, Cornell |
 Jerry Wu W., Wyatt Marshall M. |
26-50 |
WebBench |
Mechanize |
Code |
SF |
Epoch AI |
   Guive Assadi A., Tamay Besiroglu B., Ege Erdil E., Matthew Barnett B. |
26-50 |
GBA Eval |
Pareto |
Multi-Domain Tool Use |
SF |
Stanford, Scale, Turing |
   Phoebe Yao Y.RL envs: Mark Whiting W., Youssef Elhady E., Joe Crawley C. |
26-50 |
Leap, Attune |
Proximal |
Code |
SFBangalore |
Prime Intellect, Cursor |
  Justus Mattern M., Navid Pour P., Calvin Chen C. |
26-50 |
FrontierSWE |
Aptura |
Finance Enterprise |
LondonSF |
Lazard, McKinsey, EF |
Chirag Modi M. |
11-25 |
SpreadsheetBench 2 |
Bespoke Labs |
Code |
Mountain ViewMenlo Park, Bangalore, SF |
Google DeepMind, UC Berkeley |
 Mahesh Sathiamoorthy S., Alex Dimakis D. |
11-25 |
OpenThoughts |
Collinear |
Security Enterprise |
Mountain ViewSunnyvale |
Hugging Face, Salesforce, Stanford |
 Nazneen Rajani R., Soumyadeep Bakshi B. |
11-25 |
CWE-bench |
Emulated |
ML Science |
SF |
AWS, DeepMind, Prime Intellect |
 Joseph Wang W., Sid Patllollu P. |
11-25 |
SWE, Autoresearch |
Quesma |
Security |
Warsaw |
Elastic, Sumo Logic, Dynatrace |
Jacek Migdal M. |
11-25 |
BinaryAudit |
ReasonCore |
Science Code |
SF |
Turing, Aera Technology, Oracle |
  Geoff Wolfe W., Anshuman Lall L., Satish Vutukuru V. |
11-25 |
Benchmarks |
Abundant |
Code |
SF |
Waymo, Google, Mercor |
  Jesse Hu H., Ke Huang H., Meji Abidoye A. |
1-10 |
SWE-Marathon |
Aviro |
Code ML |
SF |
YC, Princeton, Georgia Tech |
 Ayush Karupakula K., Aarush Kukreja K. |
1-10 |
C⁴ |
BenchFlow |
Enterprise Browser |
SF |
Berkeley, Tesla |
Xiangyi Li L. |
1-10 |
SkillsBench |
Cua |
Code Computer Use |
SF |
Microsoft, Twitter |
 Francesco Bonacci B., Dillon DuPont D. |
1-10 |
Cua-Bench |
EdotEnv |
Long Horizon ML |
SF |
G-Research, Etched, ETH Zurich |
 Rui Wang W., Michael Zhang Z. |
1-10 |
Long-Horizon Planning |
General Reasoning |
Finance |
LondonSF |
Meta, Conjecture, Aleph Alpha |
   Ross Taylor T., Chengxi Taylor T., Kip Parker P., Thomas Grady G. |
1-10 |
OpenReward |
Markov |
Computer Use Tool Use |
SF |
YC, IIT Madras |
 Dev Mandal M., Harish Ashok A. |
1-10 |
CUA data, AutoCAD-Bench |
Metaphi |
Code Enterprise |
SFNYC |
Waymo, Nuro, Netflix |
 Abhishek Chandwani C., Ishan Gupta G. |
1-10 |
CREW |
Normal |
Hardware Engineering |
SF |
Cohere, Notion, Fuse |
 Hudzah, Anson Yu Y. |
1-10 |
CAD Arena |
pre.dev |
Code Long Horizon |
Delaware |
Penn M&T, Wharton |
 Arjun Raj Jain J., Adam Elkassas E. |
1-10 |
TB3 results |
Refresh |
Code Computer Use |
SF |
Uber, Amazon |
 Christopher Settles S., Erik Quintanilla Q. |
1-10 |
BlenderBench |
Tacit Labs |
Science Long Horizon |
SF |
MosaicML, MSR, EF |
 Nicole Fitzgerald F., Anne Marie Droste D. |
1-10 |
LifeSciBench |
Ulam |
Math Science |
WarsawLondon |
Oxford |
Przemek Chojecki C. |
1-10 |
Math RL Tasks |
Vmax |
ML |
SFNYC |
UCL |
 Matthew Sargent S., Augustine Mavor-Parker M. |
1-10 |
unix-ctf |
Uber AI |
Audio |
SF |
Scale AI, Turing, Harvard |
Key people: Natesh Pillai P. |
250+ |
|
Originator |
Computer Use Code |
LondonParis |
Google, Dataform, Pareto Holdings |
 Ben Cambier C., Lewis Hemens H. |
101-250 |
|
Fleet |
Multi-Domain Enterprise |
SFNYC |
Mercor, Anthropic, MSL |
 Nicolai Ouporov O., Andrew Zhou Z. |
51-100 |
|
Parsewave |
Long Horizon |
SFRemote |
Meta, Red Hat |
  Ivan Kudryavtsev K., Louis Pan P., Louka Ewington-Pitsos E. |
51-100 |
|
Wirestock |
Creative |
SF |
Utah State, MIT |
  Mikayel Khachatryan K., Ashot Mnatsakanyan M., Hovhannes Kuloghlyan K. |
51-100 |
|
Huzzle Labs |
Long Horizon Code |
LondonBerlin, SF |
Turing |
 Ingmar Klein K., Amit Choudhary C. |
26-50 |
|
Verita AI |
Creative |
SFVancouver |
HRT, Jane Street, Mercor |
 Rithika Kacham K., Rishi Kalakuntla K. |
26-50 |
|
ARIMLABS |
Security Long Horizon |
Warsaw |
Akamai |
 Mykyta Mudryi M., Markiian Chaklosh C. |
11-25 |
|
Chakra Labs |
Computer Use Tool Use |
NYC |
Microsoft, Palantir, Snap |
 Nirmal Krishnan K., Alex Fung F. |
11-25 |
|
dmodel |
ML Alignment |
SF |
OpenAI, Google Brain, EleutherAI |
 Anish Tondwalkar T., Daniel Moon M. |
11-25 |
|
HUD |
Long Horizon Code |
SFSingapore |
METR, AWS, Hume |
  Lorenss Martinsons M., Parth Patel P., Jay Ram R. |
11-25 |
|
Idler |
Code |
SF |
Meta, Microsoft, Cornell |
  Ivan Chub C., Nalu Concepcion C., Tony Goss G. |
11-25 |
|
Incalmo |
Security |
San Mateo |
CMU, MSR, RSAC Labs |
   Brian Singer S., Mark Dong D.RL envs: Omer Akgul A., Ao Li L. |
11-25 |
|
Preference Model |
ML Code |
SFToronto, Seattle |
Anthropic, Stripe, Datology |
 Jennifer Zhou Z., Ning Cao C. |
11-25 |
|
Symbal |
Code Enterprise |
SF |
Meta AI, General Catalyst, Deeptune |
  Krishay Mukhija M., Pranay Manocha M., Akash Jain J. |
11-25 |
|
Taste Labs |
Creative |
NYC |
Exa, Palantir, Mercado Libre |
Thais Castello Branco B. |
11-25 |
|
Align AI |
Enterprise Code |
NYC |
Meta, Apple, Cornell |
 Spandana Govindgari G., Girish Kumar K. |
1-10 |
|
Andromede |
Long Horizon |
Lausanne |
MIT, EPFL |
 Alexandre Sallinen S., Guillaume Allegre A. |
1-10 |
|
Anthromind |
Medical Long Horizon |
SF |
Google, Forward Health |
 Pratik Karki K., Mannat Sandhu S. |
1-10 |
|
Diffuse Labs |
ML Long Horizon |
Palo AltoSF |
a16z, CMU |
 Daljeet Virdi V., Tejpal Singh S. |
1-10 |
|
Dissei |
Finance |
London |
Warwick, Goldman Sachs, HBS |
  Tudor Popescu P., Jai Trivedi T., Eddy B B. |
1-10 |
|
Exabite |
Code |
Remote |
Stanford, MIT, Magic |
Alana Xiang X. |
1-10 |
|
Good Start Labs |
Games Long Horizon |
BrooklynNYC, Toronto |
Waterloo, Every, Amazon Robotics |
 Alex Duffy D., Tyler Marques M. |
1-10 |
|
Hillclimb |
Math |
SF |
DeepMind, Base |
 Jun Park P., Ibrakhim Ustelbay U. |
1-10 |
|
| Ionic |
Finance Law |
SF |
Mercor, Scale AI, Citadel |
   Sid Potdar P., JW Stanly S., Gavin Gwaltney G., Austin Mann M. |
1-10 |
|
Ooak Data |
Enterprise |
Paris |
Samsung, GoPuff, YC |
  Thomas Aubry A., Pierre-Louis Vouteau V., Grégoire Lamy L. |
1-10 |
|
Phinity |
Chip Design |
SFPalo Alto |
NVIDIA, AWS, Stanford |
 Sonya Jin J., Aadi Nashikkar N. |
1-10 |
|
Plato |
Browser Enterprise |
SF |
MultiOn, Waterloo, Georgia Tech |
 Rob Farlow F., Pranav Putta P. |
1-10 |
|
Rise Data Labs |
Enterprise |
NYC |
Wharton |
Vivian Chen C. |
1-10 |
|
Theta |
Enterprise |
SF |
DeepSilicon, Cornell, MultiOn |
  Rayan Garg G., Tanmay Sharma S., Gurvir Singh S. |
1-10 |
|
Third Moment |
Code Finance |
SFNYC |
HRT, Optiver, Hebbia |
 Jeffrey Yan Y., Adam Khakhar K. |
1-10 |
|
Trajectory Labs |
Alignment |
BerkeleyToronto |
METR, GovAI, 80000 Hours |
 Peter McIntyre M., Ben West W. |
1-10 |
|
Vetto AI |
Code Computer Use |
SFSão Paulo, London |
DeepMind, Meta, ITA |
    José André Nunes N., Rodrigo Schmidt S., Lucas Smaira S., Gabriel Albuquerque A., Roberta Antunes A. |
1-10 |
|