论文标题
自然语言用执行代码翻译
Natural Language to Code Translation with Execution
论文作者
论文摘要
在大型计划中预估计的代码生成模型在将自然语言转换为代码方面取得了巨大成功（Chen等，2021； Austin等，2021; Li等，2022，Inter Alia）。尽管这些模型在培训期间没有明确合并程序语义（即执行结果），但它们能够针对许多问题生成正确的解决方案。但是，从每个问题的生成集中选择一个正确的程序仍然具有挑战性。在这项工作中，我们介绍了基于执行结果的最小贝叶斯风险解码（MBR-EXEC）进行程序选择，并表明它改善了自然语言对代码任务上预审计的代码模型的少量性能。我们通过对共享相同语义的程序实现进行边缘化来从生成的候选人中选择输出程序。由于精确的等效性是棘手的，因此我们在少量的测试输入上执行每个程序以近似于语义等效。在数据集中，执行或模拟执行极大地超过了不涉及程序语义的方法。我们发现，MBR-exec始终改善所有执行 - unaware选择方法，这表明它是自然语言代码翻译的有效方法。我们在github.com/facebookresearch/mbr-exec和dl.fbaipublicfiles.com/mbr-exec/mbr-exec/mbr-exec-release.zip上打开代码。
Generative models of code, pretrained on large corpora of programs, have shown great success in translating natural language to code (Chen et al., 2021; Austin et al., 2021; Li et al., 2022, inter alia). While these models do not explicitly incorporate program semantics (i.e., execution results) during training, they are able to generate correct solutions for many problems. However, choosing a single correct program from a generated set for each problem remains challenging. In this work, we introduce execution result--based minimum Bayes risk decoding (MBR-EXEC) for program selection and show that it improves the few-shot performance of pretrained code models on natural-language-to-code tasks. We select output programs from a generated candidate set by marginalizing over program implementations that share the same semantics. Because exact equivalence is intractable, we execute each program on a small number of test inputs to approximate semantic equivalence. Across datasets, execution or simulated execution significantly outperforms the methods that do not involve program semantics. We find that MBR-EXEC consistently improves over all execution-unaware selection methods, suggesting it as an effective approach for natural language to code translation. We open-source our code at github.com/facebookresearch/mbr-exec and data at dl.fbaipublicfiles.com/mbr-exec/mbr-exec-release.zip
