Learning effects of permitting generative AI use on oral group quizzes
Background
In two sections of the Fall 2026 offering of Math 62: Introduction to Statistics at Mudd, I ran a randomized study to evaluate the learning effects of permitting generative AI (genAI) use on in-class oral group quizzes. Students in each section were divided randomly at the start of the semester into groups of 3-5 students. Each group had to take 8 in-class oral quizzes throughout the semester, worth a total of 15% of the grade. Whether a group was allowed to use genAI on a given quiz was randomly assigned so that each group was permitted to use AI on exactly 4 out of 8 quizzes, and exactly 3 out of 6 groups in each section were “genAI-enabled” for a given quiz. This constitutes something known as a switchback or crossover experiment (increasingly popular in big tech over the last few years) and must be analyzed carefully. The specific assignments for each section’s groups (for all 8 quizzes) were sent in an email at the start of the semester along with the group assignments.
On a “genAI-enabled” quiz, students could use Internet and genAI tools (in unrestricted ways) to assist in preparing their responses. They could not simply read off AI-generated responses when giving their final oral responses to the instructor (me).
Research questions
The main quantitative research questions are:
- To what degree, if any, did allowing genAI affect quiz performance?
- To what degree, if any, did allowing genAI affect students’ longer-term retention of the relevant concepts and problem-solving strategies?
There is also an additional qualitative research component that seeks to understand how students used genAI tools on the quizzes for which it was allowed through anonymous course surveys.
Logistics and concrete deliverables
I am looking for 1-2 students to assist with the analysis and writing for this project during the Spring 2027 semester through a ~3 credit independent study (~10 hours/week expected commitment). 2 credits will be considered in special cases. By joining this project, you will gain valuable, hands-on technical experience and knowledge in experimental design and analysis by researching state-of-the-art methods in the literature for rigorously analyzing this crossover experiment and then applying them to real (anonymized) data with student group quiz scores and final exam scores. In the process, you will hone your general research skills (literature review, writing, communication). The goal is to write a peer-reviewed publication in a scientific journal on statistics education, such as The Journal of Statistics and Data Science Education. A student who takes the initiative on this project could potentially serve as the lead author on the paper.
Essay
Please write no more than 500 words describing your interest in both the quantitative and qualitative aspects of this project. As part of this essay, I would also love to hear how this project would be useful to your future career and/or personal goals, however nebulous they may be now. I’m not looking for anything specific here, I just want to get to know what excites you!
Please also be sure to confirm you can commit to a 3 credit independent study in Spring 2027, and provide a transcript along with 1-2 references.
The Stat Methods Lab at Mudd (which is just me, Prof. Li, and any students I take on) studies interesting statistical problems directly motivated by challenges posed by real datasets. It seeks to both develop novel methods to analyze data (complete with rigorous theoretical backing and empirical/simulation evidence) and to creatively apply existing methods to new domains. A key focus of the lab is to equip students with a precise analytical toolkit and statistical judgment to understand, design, and critique their own careful data analyses.