<ul><li>AWS AI Labs has introduced SWE-PolyBench, a multilingual, repository-level benchmark for evaluating AI coding agents.</li><li>SWE-PolyBench consists of 2,110 tasks across four programming languages - Java, JavaScript, TypeScript, and Python.</li><li>The benchmark incorporates real pull requests (PRs) and introduces Concrete Syntax Tree (CST)-based metrics for assessment.</li><li>The evaluation of agents on SWE-PolyBench demonstrates varying performance across languages and task types.</li></ul>

AWS Introduces SWE-PolyBench: A New Open-Source Multilingual Benchmark for Evaluating AI Coding Agents

Discover more