Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez, Mario Linares-Vásquez, Weiyuan Ding, Bowen Xu, Massimiliano Di Penta, Antonio Mastropaolo
Large language models perform differently on identical tasks when prompted in different natural languages — a phenomenon known as language bias. We investigate how the language used to describe a programming task affects the quality of code generated by GPT-4o mini, DeepSeek, and Claude. Across 460 CoderEval tasks (Python and Java) with prompts manually curated into Chinese, Hindi, Spanish, and Italian, we evaluate functional correctness, structural quality metrics, static-analysis warnings, and the natural language used in identifiers and comments. We find that English prompts are not necessarily best, that quality varies by programming language and model, and that generated code frequently mixes English and prompt-language comments and literals.