Cognitive Tools and Visual Abstraction: Understanding How We Learn and Communicate
Introduction
Judy Fan's research explores how humans develop and use cognitive tools, particularly visual abstractions, to understand and communicate complex information. Her work bridges basic cognitive processes with higher-level thinking, examining how we learn, teach, and make sense of the world. This article summarizes her presentation, focusing on the psychological theories that explain how we discover useful abstractions and apply them to create new things.
Cognitive Tools: The Foundation of Human Understanding
What are Cognitive Tools?
Cognitive tools are material objects that encode information to impact our minds and how we think. A fundamental example is the number line, an invention that allows us to visualize and manipulate numerical concepts.
- The number line is not a natural phenomenon, but a human creation.
- It allows for deriving new mathematical discoveries.
The Evolution of Cognitive Tools
The creation of rectangular coordinates by René Descartes was a cutting-edge tool that linked algebraic expressions with geometric curves, facilitating the solution of mathematical puzzles. This illustrates how cognitive tools can revolutionize problem-solving and understanding. Over time, these tools become indispensable to education and general understanding.
The Human Mind and Innovation
Fan's research delves into how humans continually innovate, marking their environments and repurposing objects to carry meaning. This process, dating back tens of thousands of years, is intertwined with technologies for making the invisible visible.
Visual Abstraction: Making the Invisible Visible
Examples from the History of Science
Throughout history, visual abstractions have played a crucial role in scientific discovery and communication.
- Darwin's Finches: Illustrations highlighting morphological variation made differences salient.
- Galileo's Telescope: Enabled observations that challenged established views of the solar system.
- Ramón y Cajal's Drawings: Revealed the structure of the retina and its neural connections.
- Feynman Diagrams: Depicted subatomic particles and events that cannot be directly observed.
Leveraging Visual Abstraction
These examples showcase how visual abstraction, whether detailed or schematic, communicates information in a way that emphasizes relevance. This ability allows us to re-engineer the physical world according to our designs.
From Biological Insights to Bioengineering
Our ability to reformulate our understanding allows us to:
- Translate biological insights into bioengineering.
- Develop advanced physical instrumentation from physical theory.
- Create medical devices from neuroscience.
- Innovate modern electronics from quantum mechanics.
A Framework for Understanding Cognitive Processes
The Traditional Cognitive Psychology Model
This model focuses on how people process information supplied by the external world.
Enriching the Model with Social Cognition
Social cognition considers the behavior of multiple individuals and their interactions. This plays an important role in how humans learn in social contexts.
The Role of Cognitive Tools and Engineering
Fan argues that understanding how humans make discoveries requires two additional ingredients: consideration of cognitive tools and of engineering. Specifically, material objects that encode information and how people leverage their understanding to create new and useful things.
Closing the Loop
The research aims to develop psychological theories that explain how we discover useful abstractions and how we apply those abstractions to create new things.
Communicating Semantic Knowledge Through Visual Abstraction
Three Key Behaviors
Understanding visual communication involves three interconnected behaviors:
- Visual Perception: Transforming sensory inputs into meaningful experiences.
- Visual Production: Generating markings that leave a meaningful trace.
- Visual Communication: Arranging graphical elements to achieve a specific impact on other minds.
Understanding Pictorial Meaning
Initial theories suggested that drawings are meaningful because they resemble objects or because of learned conventions. Research using neural networks and computer vision algorithms shows that general-purpose vision algorithms can generalize to sparse sketches suggesting that the problem of pictorial meaning and resemblance might be resolved by building better models of the ventral stream of visual processing. However, this isn't the full story.
The Influence of Context
Context plays a crucial role in how we interpret and produce drawings. Studies show that people adjust the detail and faithfulness of their drawings based on the surrounding context, using sparser drawings when context provides sufficient information.
Evolving Graphical Conventions
New graphical conventions can emerge when shared history leads people to produce more abstract tokens over time, whose meaning is tied to that shared history.
Communicating Mechanistic Knowledge
Visual abstraction is also critical for transmitting mechanistic knowledge about how things work, moving beyond visual salience to highlight underlying mechanisms.
Distinguishing Visual Explanations from Depictions
Research explores whether visual explanations are augmented versions of depictions or if they emphasize mechanistic abstractions while de-emphasizing visual appearance. Results suggest the latter: that people think of visual explanations as images that pick out mechanistic abstractions.
The Content and Function of Visual Explanations
Studies comparing depictions and explanations of novel contraptions reveal that explanations emphasize causal parts and motion, while depictions focus on appearance. This highlights the impact of communicative context on visual communication strategies.
Developing Artificial Systems for Visual Abstraction
The Challenge of Human-Like Visual Abstraction
Developing artificial systems capable of human-like visual abstraction is crucial for creating useful scientific models of visual communication.
SEVA: A Benchmark for Sketch Understanding
A benchmark dataset, SEVA, was created to challenge vision algorithms with the complexities of sketch understanding, including variations in sparsity and semantic ambiguity.
Evaluating Machine Learning Systems
Various machine learning systems were tested on sketch categorization tasks, revealing a significant gap between model performance and human performance.
Generative Models of Sketch Production
Exploring generative models like CLIPasso reveals differences in how humans and algorithms sparsify their drawings, suggesting that even functionally similar drawings can have stylistic differences.
Multi-Modal Abstractions and Statistical Reasoning
Data Visualization as a Cognitive Tool
Data visualization is a powerful tool for understanding patterns and phenomena that are difficult to observe directly.
The Ubiquity and Importance of Plots
Plots are indispensable in various fields, enabling individuals to update and calibrate their beliefs about a complicated world.
Three Directions in Visualization Research
- Underlying Operations for Understanding Plots: Assessing machine learning systems' ability to answer questions about data visualizations and identifying gaps in alignment with human understanding.
- Selecting Appropriate Plots: Investigating how people choose the right plot to address their epistemic goals, revealing sensitivity to plot features relevant for answering specific questions.
- Measuring Visualization Understanding: Evaluating existing tests of data visualization understanding, aiming to develop improved measures that capture the underlying skills more effectively.
Benchmarking AI Systems
Comparisons between humans and AI systems on graph-based reasoning tasks reveal a gap in performance and error patterns, highlighting areas for further research.
Future Directions
By establishing the perceptual and cognitive foundations of data visualization, research aims to help learners calibrate their understanding of the world and equip them with quantitative data literacy skills.
Conclusion
Fan's work underscores the importance of understanding cognitive tools and visual abstraction in human learning and communication. By developing psychological theories and exploring the capabilities of artificial systems, this research contributes to our understanding of how we discover, create, and make sense of the world around us.