🦖 RExBench

RExBench is a benchmark aiming to evaluate the ability of LLM agents (or other AI systems) to extend existing AI research.

The benchmark consists of 12 research experiment implementation tasks, where each task is set up as an extension to an existing research paper and codebase, accompanied by domain expert-written instructions.

End-to-end workflow of RExBench: (1) An LLM agent receives inputs consisting of the research paper(s), the original codebase, and an extension instruction; (2) the system implements the extension and a patch file is obtained; (3) the patch is applied to the original code and executed via our evaluation infrastructure; and (4) the results are evaluated using specified metrics.

Dataset

Download from 🤗 Hugging Face

Citation

Please cite the following work if you are using RExBench in your research:

  @inproceedings{rexbench2026,
   title={RExBench: Can coding agents autonomously implement AI research extensions?},
      author={Edwards, Nicholas and Lee, Yukyung and Mao, Yujun (Audrey) and Qin, Yulu and Schuster, Sebastian and Kim, Najoung},
    booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics",
    year = "2026”,
    publisher = "Association for Computational Linguistics",
      url={https://arxiv.org/abs/2506.22598}
  }
                  
            

License

We release our data under a dual license (MIT and Apache 2.0), given the mixed license of the repositories included in the full benchmark suite. Please note that this in contrast to the metadata license shown (Hugging Face currently only supports assigning one license to a dataset). Please also refer to the license agreements in the individual task repositories.

Leaderboard