VisualNarrative · From Meaning to Perception: How narratives shape visual processing
Horizon Europe — Marie Skłodowska-Curie Actions
- Duration
- 2023-11-01 → 2025-10-31
- EU contribution
- €187,624
- Participants
- 1
- Scheme
- HORIZON-TMA-MSCA-PF-EF
Lines connect the coordinator with its partners.
Results in brief
From Meaning to Perception: How narratives shape visual processing
Humans constantly generate expectations of what will happen next. This allows us to process overwhelming sensory input more efficiently. However, to date most of studies examining the influence of expectations on sensory processing have used rather oversimplified paradigms which lack higher-order meaning, like simple stimuli associations. To overcome this barrier, this project aimed to assess how more cognitive, meaning-based expectations influence visual processing. To this aim, I used picture books as a more ecologically valid paradigm and to experimentally still manipulate the meaningfulness of stimuli (whether subsequent pictures from the book were presented in a coherent, i.e. correct order, or a shuffled order where the meaning was lost). I used state-of-the-art large language models to quantify expectations in such stimulus and relate it to visual processing. Together, this project bridged from more higher-order cognition to lower-level visual processing.
Data: CORDIS, © European Union
Project objective
Humans constantly generate expectations of what will happen in the future. Such expectations allow us to quickly absorb incoming data. For example, the visual system takes advantage of the world’s relative stability over space and time to anticipate upcoming input. However, most of what we currently know about predictive visual processing comes from studies using arbitrary associations between simple stimuli, such as gratings or abstract objects, which do not reflect the richness of daily life events. It is largely unknown how more complex, meaning-based expectations influence visual processing. To overcome this barrier, the proposed project will use an ecologically valid paradigm, using picture books, and investigate how such more complex, narrative-based expectations shape visual processing. I hypothesise that narratives will elicit strong expectation signals that impact even the earliest levels of sensory processing. To test this hypothesis, I will first quantify narrative-based expectations derived from the picture stories using state-of-the-art computational modelling. Then, I will assess whether these expectations shape the processing in visual cortical regions of adult human volunteers, using neuroimaging. I will inspect whether such signals become anticipatory as viewers become familiar with the narrative – in line with the idea that upon construction of a world model, the brain predicts upcoming stimuli. Moreover, I will also investigate what brain regions shape such learning of narratives. In a separate eye-tracking experiment, I will explore how narratives influence gaze behaviour, providing another assay into how meaning percolates to visual processing. This project will allow me to conduct creative, original and important research, which will propel my scientific career. More broadly, these lines of inquiry will push the envelope of understanding of human perception and learning, bridging from higher-order cognition to low-level visual processing.
Original text from CORDIS.
Participants
- STICHTING RADBOUD UNIVERSITEIT · NijmegenCoordinatorNetherlands
Links
- View on CORDIS
- DOI: 10.3030/101106569
- https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e51bc15dd2&appId=PPGMS
- https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e52348f894&appId=PPGMS
Data: CORDIS, © European Union