Pass@k

A metric that measures the proportion of correct predictions or outputs produced by a model or AI system out of all evaluated cases.

The proportion of attempted tasks or requests that an AI agent or system completes successfully.

A benchmark that evaluates the code generation capabilities of AI models by measuring their ability to generate functionally correct code that passes predefined unit tests for a collection of programming tasks.

A benchmark for evaluating the ability of language models to generate Python code from natural language descriptions.

A contamination-aware benchmark for evaluating large language models on recent competitive programming problems.