SWE-bench

A benchmark that evaluates the code generation capabilities of AI models by measuring their ability to generate functionally correct code that passes predefined unit tests for a collection of programming tasks.

A benchmark for evaluating the ability of language models to generate Python code from natural language descriptions.

A contamination-aware benchmark for evaluating large language models on recent competitive programming problems.