Quantize with Confidence? An Empirical Study of Quantization for Code Generation
Saima Afrin, Md. Zahidul Haque, Antonio Mastropaolo
An empirical study of six weight-only post-training quantization techniques — GPTQ, AWQ, QuIP#, AQLM, BitsAndBytes, and GGUF — applied at 4-bit precision to the Qwen2.5-Coder and CodeLlama model families. Using the McEval and CoderEval benchmarks across Python and Java, we evaluate both functional correctness (pass@1) and code-quality attributes (maintainability, reliability, security, and structural complexity), and introduce a novel analysis of quantization robustness under varying prompt complexity, characterized by Shannon entropy and token length. We find that technique choice matters: AQLM consistently matches or exceeds the full-precision baseline, whereas QuIP# degrades most on complex prompts, while security attributes remain stable across models, benchmarks, and programming languages.