Maurizio Berta's 30% seminar

Abstract:

Computation and data are pervasive, yet the web of their actions, impact, and work remains invisible to most people. This invisibility undermines laypeople's understanding of the complexity and nuances of the digital world surrounding us, leading to issues in digital literacy, misinformation, and reduced individual agency. New methods for communicating and engaging laypeople with the digital world are therefore necessary.

Common solutions employ visualization, the predominant medium for data communication in the Western world. However, digital objects that rely on a single sensory stimulus lack perceptual depth as compared to real-world objects that can be experienced multisensorially, and often fail to provide an engaging route to sense-making. Sound can serve as an alternative or complementary medium, particularly through sonification. Used multimodally, it adds depth to visual elements, making them feel more physically present.

Although several studies have addressed multimodal representation, a gap remains in the development of interactive multimodal interfaces for data exploration. Additionally, while the Art-Science field has produced works addressing laypeople's understanding of computational phenomena, continuous engagement is essential, as only ongoing exposure can raise awareness.

To address the aforementioned issues, this seminar presents two studies. The first examines a multimodal interface for exploring data in a force-graph representation, applied to bibliometrics. The interface was evaluated through Think-Aloud sessions and interviews, and its multimodal mappings were tested via an online perceptual study. The second explores affective response through an artistic installation investigating non-human agency, realized through a symbolic feedback system between three sonic totems. Together, these studies contribute to laypeople's understanding of data and computation through multimodal interface design and Art-Science exploration.