Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT

The research explores students' interactions with ChatGPT-3.5 in an introductory programming course, identifying diverse patterns of engagement and the implications for teaching and learning, including the need for personalized support and guidance on using language models effectively.

Abstract

Large Language Models (LLMs) have taken the world by storm, and students are assumed to use related tools at a great scale. In this research document we aim to gain an understanding of how introductory programming students chat with LLMs and related tools, e.g., ChatGPT-3.5. To address this goal, computing students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT as part of their weekly introductory course exercises. Then students (n=213) submitted their chat protocols (with 2335 prompts in sum) as data basis for this analysis. The data was analyzed w.r.t. the prompts, frequencies, the chats' progress, contents, and other use pattern, which revealed a great variety of interactions, both potentially supportive and concerning. Learning about students' interactions with ChatGPT will help inform and align teaching practices and instructions for future introductory programming courses in higher education.

Method

The method of this document involves an exploratory study on students' interactions with ChatGPT-3.5 in the context of an introductory programming course. The study collected empirical data from 213 students who were tasked with solving programming exercises using ChatGPT-3.5 and submitting their chat protocols as part of the assignment. The data was then analyzed quantitatively and qualitatively to characterize students' use and interaction patterns with the tool. The study aimed to gain insights into the diverse interactions, both potentially supportive and concerning, to inform and align teaching practices and instructions for future introductory programming courses in higher education.

Main Finding

The main finding of this document is that students exhibit diverse patterns of interaction with GenAI tools, such as ChatGPT-3.5, in the context of introductory programming education, reflecting varying needs for support and feedback. The study emphasizes the potential for personalized support offered by GenAI tools, the importance of educators providing guidance on effective tool usage, and the need to communicate the tools' limitations and potential pitfalls to students. Additionally, the research highlights the potential for specialized AI tools for coding to enhance support for learners in the future and suggests the need for replication studies in different educational contexts to strengthen the results.

Conclusion

The conclusion of the document is that the study provides valuable insights into students' diverse interactions with GenAI tools, particularly ChatGPT-3.5, in the context of introductory programming education. It highlights the need for personalized support and guidance on effective tool usage, as well as the importance of communicating the tools' limitations and potential pitfalls to students. Additionally, the study suggests the potential for specialized AI tools for coding to enhance support for learners in the future and emphasizes the need for replication studies in different educational contexts to strengthen the results.

Keywords

ChatGPT-3.5, large language models, students, interaction pattern, application, chat protocols, introductory programming, higher education