Comparing AI and Human Performance on Psychology Data
Social scientists are increasingly using AI tools to support workflows to annotate textual data. But how well does this work? Are there situations where specific models or approaches are more or less effective? In this project, we will look at real anonymized psychology datasets from several studies focused on stress. We'll compare human annotation approaches with different levels of computational intervention (e.g. using keywords, identifying topics, training a classifier, using an LLM). In addition to analyzing the agreement of annotations between these approaches, we'll also dive deeper into the types of errors that these different systems produce in order to generate recommendations for how to evaluate, audit, and adapt automated approaches to annotation.
Name of research group, project, or lab
With Prof. Stacey Doan (CMC Psychology) and Prof. Xanda Schofield (HMC CS)
Why join this research group or lab?
This is an opportunity to practice implementing and evaluating machine learning model performance while becoming familiar with psychology transcript data and methodologies. Students will get experience running machine learning code, reasoning about how to code/annotate psychology data, and applying appropriate statistical and visualization tools to explore the outcomes.
2 sp. | 0 appl.
Time commitment
Spring - Part Time
Project categories
Humanities, Social Sciences, and the Arts(+2)
Computer ScienceHumanities, Social Sciences, and the ArtsData Science
Related ProjectRelated Projects
Countdown...
This timer uses the clock on your device and is only an estimate. The actual time remaining is determined on the website.