# Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination

## Executive summary

This seminar explores the complex intersection of text and data visualization, arguing that language is a critical component of effective information design. The discussion moves from established cognitive theories (like Dual Coding Theory) to modern AI architectures, detailing how Multimodal Large Language Models (MLLMs) process combined visual and textual inputs using mechanisms like cross-attention. Key findings suggest that while the optimal balance between text and visuals is highly context-dependent, MLLMs are capable of deeply integrating both modalities.

## Key takeaways

- Language is a key component of visualization: Studies show that titles and labels receive long fixations during encoding and are the most likely elements to be recalled, suggesting language significantly impacts how data visualizations are understood. The speaker notes this was historically under-explored in the visualization community.
- Optimal design favors annotation over minimalism: Research suggests that 'more text is better' for general information displays, provided the text is relevant and properly annotated. This finding challenges traditional minimalist principles in UI/UX design.
- MLLMs integrate modalities via cross-attention: Multimodal LLMs (MLLMs) process text and visuals by transforming inputs into embeddings. The 'cross-modality embedding architecture' uses a cross-attention mechanism, allowing information to flow between the two distinct sequences (e.g., image tokens interacting with text tokens).
- Cognitive processing is context-dependent: The speaker notes that understanding how humans integrate text and visuals is complex, citing conflicting evidence across theories (Dual Coding vs. Cognitive Load Theory). The choice of representation depends heavily on the specific task or cognitive ability.

## Technical details

- Information Visualization Principles: Anscombe's quartet is cited as a famous example demonstrating how the same sufficient statistics can yield radically different insights depending on the visualization chosen.
- MLLM Architectures (Multimodal LLMs): Two primary architectures are discussed: 1) Unified Embedding Architecture (concatenating visual and text embeddings after dimensionality reduction), and 2) Cross-modality Embedding Architecture. The latter uses cross-attention to allow information flow between the two modalities.
- Attention Mechanisms: Standard self-attention allows words within a sentence to interact, while cross-attention is essential for allowing information to flow and integrate between two different sequences (e.g., an image sequence and a text sequence).

## Practical implications

- When designing dashboards or reports, prioritize annotating charts with relevant text rather than relying solely on visual elements.
- Be mindful of the context: The optimal balance between language and visualization is highly task-specific (e.g., comparing stats vs. asking for a specific value).
- For technical documentation, consider providing both a detailed textual explanation and an accompanying visualization to maximize comprehension, while being aware that excessive text can impede reading flow.
- When building AI systems that interpret data, utilize cross-attention mechanisms in the model architecture to ensure deep integration of visual and linguistic inputs.

## Topics

Human-Computer Interaction (HCI), Information Visualization, Cognitive Psychology, Multimodal AI, Natural Language Processing (NLP), Anscombe's quartet, Beyond Memorability, Proust and the Squid

Source: https://www.youtube.com/watch?v=vVeCXAFt5f4