Toward a Multimodal Assessment of Visualization Literacy: Integrating EEG and Eye Tracking

Description

In the field of visualization comprehension, existing work focuses on evaluating viewers' cued and uncued responses to data visualizations [1 – 6]. While previous studies have examined the influence of visualization type [6 – 8], task complexity [7,8], and data quantity [9] on user performance, their methodologies often lack the control necessary to isolate and compare individual impact factors. Additionally, most assessments rely solely on correctness as a performance measure—an approach susceptible to guessing that provides limited insight into the underlying cognitive processes.

This thesis project aims to explore the impact of visual distractors on comprehension performance during comparison tasks with bar charts. Specifically, the study investigates how varying numbers and placements of distractor bars influence comprehension, using a combination of presentation times, behavioral performance (accuracy and reaction time), gaze behavior, and fixation-related potentials (FRPs). By integrating physiological measures such as EEG and eye-tracking, this work aims to uncover indicators of visual and cognitive processing linked to task success or failure.

The research is structured around three key phases: (1) programmatically generating controlled visual stimuli (bar charts) with systematic variation in distractor conditions and precisely mapped areas of interest (AOIs), (2) designing and implementing an experiment to collect EEG, gaze, and performance data, and (3) conducting a user study and analyzing the results. The insights gained from this research may enhance our understanding of how users process visual information and support the development of predictive models for visualization comprehension. Ultimately, the findings could inform the design of more effective and user-centered data visualizations.

Research Phases

The research consists of the following phases:

The project consists of the following research phases:

1. Literature review: Review existing work on visual comparison in bar charts and the role of distractors, as well as physiological measures in HCI research.
2. Stimulus generation: Programmatically generate bar chart stimuli with controlled distractor variations, and define AOIs for subsequent eye-tracking analysis.
3. Experiment design and implementation: Build the experimental setup (e.g., using PsychoPy), including response collection, synchronized EEG and eye-tracking data acquisition, and controlled stimulus presentation.
4. Data collection: Conduct a user study involving EEG, eye-tracking, and behavioral data across the designed stimulus conditions.
5. Data analysis: Preprocess and statistically analyze performance, gaze behavior, and FRPs to assess the impact of distractors on comprehension and cognitive processing.

You Will

• Conduct a literature review on bar chart comprehension, distractors, and physiological data collection in HCI.
• Design and programmatically generate controlled visual stimuli for the experiment.
• Set up and implement the experiment using PsychoPy or a similar framework, integrating EEG and eye-tracking hardware.
• Run a user study with synchronized EEG, gaze, and behavioral data collection.
• Preprocess and extract meaningful metrics from the data, such as fixation duration, ERP components, and accuracy.
• Perform statistical analysis to evaluate how visual distractors affect user performance and cognitive load.
• Summarize and present your findings in a written thesis and final presentation.
• (Optional) Collaborate on a research publication based on the outcomes of the project.

You Need

• Strong written and verbal communication skills in English.
• Proficiency in Python for experimental design and data analysis.
• (Preferred) Basic familiarity with EEG or eye-tracking data handling.

References

• [1] S. Lee, S.-H. Kim, and B. C. Kwon, “VLAT: Development of a Visualization Literacy Assessment Test,” IEEE Trans. Vis. Comput. Graph., vol. 23, no. 1, pp. 551–560, Jan. 2017, doi: 10.1109/TVCG.2016.2598920.
• [2] S. Pandey and A. Ottley, “Mini-VLAT: A Short and Effective Measure of Visualization Literacy,” Comput. Graph. Forum, vol. 42, no. 3, pp. 1–11, 2023, doi: 10.1111/cgf.14809.
• [3] L. W. Ge, Y. Cui, and M. Kay, “CALVI: Critical Thinking Assessment for Literacy in Visualizations,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, in CHI ’23. New York, NY, USA: Association for Computing Machinery, Apr. 2023, pp. 1–18. doi: 10.1145/3544548.3581406.
• [4] Y. Cui, L. W. Ge, Y. Ding, F. Yang, L. Harrison, and M. Kay, “Adaptive Assessment of Visualization Literacy,” Aug. 27, 2023, arXiv: arXiv:2308.14147. Accessed: Aug. 27, 2024. [Online]. Available: http://arxiv.org/abs/2308.14147
• [5] J. Boy, R. A. Rensink, E. Bertini, and J.-D. Fekete, “A Principled Way of Assessing Visualization Literacy,” IEEE Trans. Vis. Comput. Graph., vol. 20, no. 12, pp. 1963–1972, Dec. 2014, doi: 10.1109/TVCG.2014.2346984.
• [6] G. J. Quadri, A. Z. Wang, Z. Wang, J. Adorno, P. Rosen, and D. A. Szafir, “Do You See What I See? A Qualitative Study Eliciting High-Level Visualization Comprehension,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, in CHI ’24. New York, NY, USA: Association for Computing Machinery, May 2024, pp. 1–26. doi: 10.1145/3613904.3642813.
• [7] S. Lee, B. C. Kwon, J. Yang, B. C. Lee, and S.-H. Kim, “The Correlation between Users’ Cognitive Characteristics and Visualization Literacy,” Appl. Sci., vol. 9, no. 3, Art. no. 3, Jan. 2019, doi: 10.3390/app9030488.
• [8] C. Nobre, K. Zhu, E. Mörth, H. Pfister, and J. Beyer, “Reading Between the Pixels: Investigating the Barriers to Visualization Literacy,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, in CHI ’24. New York, NY, USA: Association for Computing Machinery, May 2024, pp. 1–17. doi: 10.1145/3613904.3642760.
• [9] J. Talbot, V. Setlur, and A. Anand, “Four Experiments on the Perception of Bar Charts,” IEEE Trans. Vis. Comput. Graph., vol. 20, no. 12, pp. 2152–2160, Dec. 2014, doi: 10.1109/TVCG.2014.2346320.