Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination
This seminar explores the complex intersection of text and data visualization, arguing that language is a critical component of effective information design. The discussion moves from established cognitive theories (like Dual Coding Theory) to modern AI architectures, detailing how Multimodal Large Language Models (MLLMs) process combined visual and textual inputs using mechanisms like cross-attention. Key findings suggest that while the optimal balance between text and visuals is highly context-dependent, MLLMs are capable of deeply integrating both modalities.
Key takeaways
-
Language is a key component of visualization
5:19
Studies show that titles and labels receive long fixations during encoding and are the most likely elements to be recalled, suggesting language significantly impacts how data visualizations are understood. The speaker notes this was historically under-explored in the visualization community.
-
Optimal design favors annotation over minimalism
21:30
Research suggests that 'more text is better' for general information displays, provided the text is relevant and properly annotated. This finding challenges traditional minimalist principles in UI/UX design.
-
MLLMs integrate modalities via cross-attention
41:20
Multimodal LLMs (MLLMs) process text and visuals by transforming inputs into embeddings. The 'cross-modality embedding architecture' uses a cross-attention mechanism, allowing information to flow between the two distinct sequences (e.g., image tokens interacting with text tokens).
-
Cognitive processing is context-dependent
30:00
The speaker notes that understanding how humans integrate text and visuals is complex, citing conflicting evidence across theories (Dual Coding vs. Cognitive Load Theory). The choice of representation depends heavily on the specific task or cognitive ability.
