Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination
Summary
This seminar explores the complex intersection of text and data visualization, arguing that language is a critical component of effective information design. The discussion moves from established cognitive theories (like Dual Coding Theory) to modern AI architectures, detailing how Multimodal Large Language Models (MLLMs) process combined visual and textual inputs using mechanisms like cross-attention. Key findings suggest that while the optimal balance between text and visuals is highly context-dependent, MLLMs are capable of deeply integrating both modalities.
Key takeaways
-
Language is a key component of visualization
5:19
Studies show that titles and labels receive long fixations during encoding and are the most likely elements to be recalled, suggesting language significantly impacts how data visualizations are understood. The speaker notes this was historically under-explored in the visualization community.
-
Optimal design favors annotation over minimalism
21:30
Research suggests that 'more text is better' for general information displays, provided the text is relevant and properly annotated. This finding challenges traditional minimalist principles in UI/UX design.
-
MLLMs integrate modalities via cross-attention
41:20
Multimodal LLMs (MLLMs) process text and visuals by transforming inputs into embeddings. The 'cross-modality embedding architecture' uses a cross-attention mechanism, allowing information to flow between the two distinct sequences (e.g., image tokens interacting with text tokens).
-
Cognitive processing is context-dependent
30:00
The speaker notes that understanding how humans integrate text and visuals is complex, citing conflicting evidence across theories (Dual Coding vs. Cognitive Load Theory). The choice of representation depends heavily on the specific task or cognitive ability.
Technical details
-
Information Visualization Principles
350s
Anscombe's quartet is cited as a famous example demonstrating how the same sufficient statistics can yield radically different insights depending on the visualization chosen.
-
MLLM Architectures (Multimodal LLMs)
2350s
Two primary architectures are discussed: 1) Unified Embedding Architecture (concatenating visual and text embeddings after dimensionality reduction), and 2) Cross-modality Embedding Architecture. The latter uses cross-attention to allow information flow between the two modalities.
-
Attention Mechanisms
2580s
Standard self-attention allows words within a sentence to interact, while cross-attention is essential for allowing information to flow and integrate between two different sequences (e.g., an image sequence and a text sequence).
Mentioned resources
- Anscombe's quartet
- Beyond Memorability
- Proust and the Squid
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.