Glossary

HumanEval

Also called: OpenAI HumanEval

A benchmark that evaluates the code generation capabilities of AI models by measuring their ability to generate functionally correct code that passes predefined unit tests for a collection of programming tasks.

Explore more about Benchmarks

Related terms

Code Generation

The automated creation of source code from natural language, specifications, prompts, or structured inputs using AI or software tools.

Unit Testing

A software testing method that verifies the correctness of individual isolated units or components of code.

Pass@k

A code-generation metric measuring the probability that at least one of k generated solutions correctly solves a given programming problem.

MBPPMBPP

A benchmark for evaluating the ability of language models to generate Python code from natural language descriptions.

Signal, not noise.

Focused newsletter for builders and knowledge workers tracking how AI is changing real work. We surface what matters in practice, not every headline. Curated for practitioners, not spectators.

Subscribe to our newsletter