Decoding The Metrics: A 5–Year Retrospective on Evaluating AI - Generated Python Code
Tác giả
Tóm tắt
—The rise of large language models (LLMs) has revolutionized automated code generation, yet evaluating their outputs remains a complex and multifaceted challenge. This paper presents a comprehensive study of evaluation metrics used for Python code generation, grounded in an analysis of 157 peer-reviewed papers from 2020 to 2024. We propose a formal taxonomy categorizing metrics into lexical similarity, functional correctness, semantic equivalence, edit distance, and ranking-based methods. Empirical trends reveal a transition from traditional lexical metrics like BLEU toward execution-based measures such as pass@k, alongside a growing adoption of semantic metrics like CodeBLEU and CodeBERTScore. Our findings highlight that no single metric captures all facets of code quality; instead, we advocate for a multi-metric evaluation approach that balances syntactic fidelity, functional success, and structural similarity. The study concludes with actionable recommendations and anticipates future directions where metrics may evolve toward measuring explainability, robustness, and real-world utility. This work provides a principled foundation for the evaluation of AI-generated code and supports the development of more reliable and interpretable benchmarks.