Computer and Human Vision Research and Development

Here in the Laboratory for Cognition and Attention in Time and Space (Lab for CATS), we study visual perception in both artificial and biological agents. During Summer 2024, there will be two projects that I am seeking applicants for.

1. Attention and Feature Representation in Spatiotemporal Networks

This project is part of an NSF-funded REU site with the HMC Computer Science Department. You can learn about other REU projects here: https://www.hmc.edu/cs/research/reu/mentors/. Applications for this project will be processed on the ETAP website here: https://etap.nsf.gov/award/1783/opportunity/8020. Please submit your application there and not on the URO website! Note that due to the funding source, this project is restricted to US citizens or permanent residents.

Over the past decade, deep learning has become the predominant approach to computer vision problems. While deep neural networks have demonstrated highly impressive results in many areas, they still sometimes exhibit surprising brittleness. Often this brittleness appears to result from networks relying on unintended patterns and correlations in the training data, or optimizing over easier to learn but less reliable parts of the visual signal (e.g. Geirhos et al. (2019) demonstrated that deep networks tend to rely more heavily on local image texture rather than more global object shape). Understanding behaviour in spatiotemporal processing is less well explored, however. Our work takes advantage of techniques developed to visualize the activity of deep networks in order to better understand what visual information is driving a network’s behaviour (e.g. Grad-CAM (Selvaraju et al., 2020)) to compare action recognition network activity to human eye tracking data on the same videos. In this way, we can identify and quantify how deep networks operating on videos rely on visual information, and begin to develop novel training and data augmentation techniques to guide deep neural networks to more robust representations.

This is an ongoing project with an active code base under development. Students hired to work on this project will continue to develop our work in this area by designing new experiments to enhance our understanding of model behaviour, and/or extending previous findings to new network types or spatiotemporal applications.

2. The Psychophysics of Attention

This project is funded from several sources, all of which are directly coordinated through the Lab for CATS. Please apply for this project through the URO website following the instructions given below.

Visual cognition is complex and often requires the integration of multiple sources of external context with a crowded visual environment. Humans are remarkably adept at visual processing, and navigate visual challenges with apparent ease. Humans are not very good, however, at explaining how they performed a given visual process. Visual psychophysics is one branch of study that tries to peer into this unconscious process and understand it better; by designing carefully controlled visual stimuli and experiment protocols, the goal is to isolate particular behaviours in human cognition to provide new insight into how the human brain is processing visual information.

In our lab, we work on psychophysical experiments predominantly focused on visual attention - the process by which humans adapt their visual processing to the context of the situation, whether that is external knowledge about how the world works, or specific task information they are provided as part of the experiment. This is a highly interdisciplinary research area; students hired to this area will build skills in image processing and computer vision as they construct and manipulate experimental stimuli, as well as learn about human experiment design and data analysis. Particular areas of focus for this coming summer will be the interaction of prior experience on visual search behaviour, and conflicting cues in scene understanding.

Applying to Work in the Lab for CATS:

If you are applying to work on Project 1, please follow the link to the REU application site. Helpful information to provide in your application includes relevant experience you have to the research area and why you are specifically interested in this research topic.

If you are applying to Project 2, please write your application through the URO portal as normal. In your essay, please address the following questions:

1. Write several (e.g. 3-5) sentences on why you are interested in this specific project (e.g. personal interest or alignment with future career goals). What do you hope to learn by participating? 
2. Please also provide a few sentences describing any relevant experience you have had that will assist with getting started on that project (this can be relevant course work, past research or professional experience, or other things you think are relevant).