AI could pass a Comp Sci course on its own, says student research paper

AI-driven tools are relatively new, and understanding their impact on education is crucial as they become more integrated into learning environments. Ben Puryear ('24, B.S. Computer Science) worked with faculty advisor Gina Sprint, Ph.D. (Computer Science) to explore the capabilities of GitHub Copilot, an AI trained on all public GitHub repositories. The results of that research has been published in the Journal of Computing Sciences in Colleges.

Puryear and Sprint collaborated last year to assess how an AI-driven development environment (AIDE) could help create original source code, and how that might impact programming education.

AIDEs are programming tools that, given comments or starter code, can generate code solution suggestions. As the accuracy of these tools continues to increase, one particular AIDE from Github, Copilot, has been gaining significant attention for its performance and ease of use. The rise of Copilot suggests that code solution generation tools will soon be commonplace in both the industry and in computer science courses, with expert and novice programmers alike benefiting from using these tools.

The study scrutinized Copilot-generated solutions to introductory computer science and data science assignments. Puryear evaluated them for correctness, style, skill level appropriateness, grade scores, and potential plagiarism. Results showed that Copilot could generate mostly-unique code, and achieving human-graded scores between 68% and 95%.

Based on these results, Puryear and Sprint provide recommendations for educators to help adapt their courses to incorporate new AIDE-based programming workflows. Those practical insights could give educators ideas for incorporating these tools into their own coursework.