BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Most code generation benchmarks ask for short, self-contained functions, which is not what programming looks like and is easy for a model to have memorized. BigCodeBench instead builds tasks that require invoking and composing calls across many libraries, driven by instructions with several interacting constraints.

The benchmark ships with rigorous test-based evaluation and a split that separates instruction following from raw synthesis ability. The gap it exposes between frontier models and human performance is substantially wider than on earlier benchmarks — the tasks reward actually understanding an API surface rather than recalling a snippet.

BigCodeBench has since become a standard evaluation for code LLMs: its public leaderboard covers 160+ models, it is used to evaluate Qwen2.5-Coder, CodeGeeX4 and Seed-Coder, and it is built into AI2 OLMES, NVIDIA NeMo-Skills and OpenCompass.