Comparing AI and Human Performance on Psychology Data

Social scientists are increasingly using AI tools to support workflows to annotate textual data. But how well does this work? Are there situations where specific models or approaches are more or less effective? In this project, we will look at real anonymized psychology datasets from several studies focused on stress. We'll compare human annotation approaches with different levels of computational intervention (e.g. using keywords, identifying topics, training a classifier, using an LLM). In addition to analyzing the agreement of annotations between these approaches, we'll also dive deeper into the types of errors that these different systems produce in order to generate recommendations for how to evaluate, audit, and adapt automated approaches to annotation.

Name of research group, project, or lab
With Prof. Stacey Doan (CMC Psychology) and Prof. Xanda Schofield (HMC CS)
Why join this research group or lab?

This is an opportunity to practice implementing and evaluating machine learning model performance while becoming familiar with psychology transcript data and methodologies. Students will get experience running machine learning code, reasoning about how to code/annotate psychology data, and applying appropriate statistical and visualization tools to explore the outcomes. 

Logistics Information:
Project categories
Computer Science
Humanities, Social Sciences, and the Arts
Data Science
Student ranks applicable
First-year
Sophomore
Junior
Senior
Student qualifications

Students must have:

  • Prior experience writing code using libraries in Python (e.g. CS 5)
  • Math background in statistics and data analysis (e.g. Math 62)
  • Interest in social science datasets, especially in psychology
  • Willingness to complete a human-centered research training

We appreciate the following additional experience:

  • Prior use of machine learning libraries in Python (like scikit-learn, tensorflow, pytorch, keras)
  • Prior use of data libraries in Python (numpy, scipy, matplotlib, pandas)
  • Prior experience in social science contexts of coding (as in annotating) transcripts or free response text
  • Prior coursework in psychology

Note that, while we will evaluate machine learning libraries and large language models, we will also be hand-coding some parts of our evaluation workflow, so independent ability to write, edit, and test code is a requirement.

Time commitment
Spring - Part Time
Compensation
Academic Credit
Paid Research
Number of openings
2
Project start
Spring 2027
Contact Information:
Mentor
aschofield@hmc.edu
Assistant Professor
Name of project director or principal investigator
Xanda Schofield
Email address of project director or principal investigator
xanda@cs.hmc.edu
2 sp. | 0 appl.
Time commitment
Spring - Part Time
Project categories
Humanities, Social Sciences, and the Arts (+2)
Computer ScienceHumanities, Social Sciences, and the ArtsData Science