Other Free Encyclopedias » Online Encyclopedia » Encyclopedia - Featured Articles » Contributed Topics from A-E

Experiential Media Systems - Experiential media applications, Motivating ideas, Artificial Intelligence, Distributed Cognition, Human Computer Interaction, Ubiquitous computing, Summary,  

physical world environment interfaces

Hari Sundaram and Thanassis Rikakis
Arts Media and Engineering Program
Arizona State University, Tempe, AZ, USA

Definition: Experiential media systems refer to new, complementary model of media computing, which allows us to develop a rich contextual understanding of human activity, at different scales of time and space, as well as affect human activity in a radically new way.

Our civilization is currently undergoing major changes. Traditionally, human beings acquired knowledge through experiential interactions with the physical world. That knowledge allowed them to better adapt to their reality and evolve. Today our interactions with almost every element of our lives (health, weather, economic and social policy, communication) involve computation and mediated information. However, we still lack effective ways of connecting our computational approaches with our physical experience. To achieve knowledge of our new world and significantly improve our condition we need unified experiences of the physical and computational forces that are shaping our reality.

Experiential media systems refer to new, complementary model of media computing. In the traditional multimedia computing model (e.g. the creation / consumption of a video), capture, analysis, and media consumption are not co-located, synchronous or integrated. We are, however, witnessing a rapid decline in cost of sensing, storage, computing and display. Thus, sensors (audio, video, pressure, tangible), computing, ambient visual and sound displays and other feedback devices (vibration, light, heat) can now be co-located in the same physical environment, creating a real-time feedback loop. This allows us to develop a rich contextual understanding of human activity, at different scales of time and space, as well as affect human activity in a radically new way. The goal is to achieve enhanced, user-oriented, unified physical-digital experiences. These media systems will give rise to a new set of multimedia applications grounded in human activity in the physical world.

Experiential media applications

We now briefly discuss on three application areas of societal significance – health, education and everyday living as the driving force for the development of these new media systems. In the health domain these systems can reveal new frameworks for successful interactive biofeedback for rehabilitation. In education, examples include hybrid physical digital environments that enable children to acquire complex concepts in science through natural interfaces and social interaction. In social communication and everyday living, experiential media can summarize human activity and present it as part of the physical environment to reveal the situated context in which it has occurred. We have been working on developing initial prototypes of experiential media systems. Our efforts in our application areas include:

  • Biofeedback: We have been working on developing an experiential media system that integrates task dependent physical therapy and cognitive stimuli within an interactive, multimodal environment. The environment provides a purposeful, engaging, visual and auditory scene in which patients can practice functional therapeutic reaching tasks, while receiving different types of simultaneous feedback indicating measures of both performance and results.
  • Systems for Everyday Living: We are investigating novel forms of media interaction frameworks (incorporating gestures, tangible interfaces), that use media from everyday user activities In our current work, we have built a real-time, mediated environment that uses tangible interfaces for facilitating social communication Experiential education: We are currently investigating the creation of experiential frameworks for pattern awareness in movement and in shape, for children. In our current work, fused vision and audio sensing is coupled to a system modeling gravity. This helps drive a generative model for audio-visual immersion that is cognitively consistent with the underlying physics concepts.

We note that these applications are example starting points for research in experiential media. In due course a number of new and exciting applications will emerge within these areas. They include novel interfaces for real time monitoring of patient performance by clinicians and therapists, interactive multimedia for math and teacher education, assisted home living for the elderly, and robotic assistance in navigation. Research and education activities will result in generic frameworks that can also be applied to other problem areas such as intuitive navigation of large scale data sets and new classes of interfaces for fast solutions to complex science problems (i.e. electromagnetic interference in integrated circuits). Figure 2 illustrates tangible interfaces for shared media browsing.

Motivating ideas

Experiential media analysis has been motivated by research in artificial intelligence, distributed cognition, human computer interaction / ubiquitous computing as they relate to the physical world.

Artificial Intelligence

There are two research areas in AI that are particularly relevant – robotics and bounded rationality Rodney Brooks’ robots essentially contained a set of loosely interacting behaviors (move left leg, avoid obstacles etc.), that were implemented in hardware with short paths from sensors to actuators. The behaviors were all simultaneously active, but there was a hardwired arbitration amongst them. They had no centralized reasoning unit or stored abstract representation of the world. The resulting robots seemed to be “intelligent” to human observers and were capable of complex behavior in the real world spaces. There are three crucial ideas here – situatedness, embodiment and intelligence. Situatedness is the idea that rather using an abstract stored representation of the world, “the world is its own best model”- the current state of the world is responded to by the robot (via the behaviors). Embodiment is the idea that physical grounding is crucial to intelligence and finally the last idea is that intelligence emerges though the dynamics of interaction with the world. Reactivity is another important idea the ability of the robot to be able to respond to the world in a timely manner is an important aspect of intelligence. Russell’s work on bounded rationality deals with finding programs that maximize the expected utility given the machine and the environment – i.e. the ability to generate maximally successful behavior given available information and computing resources. However, Russell’s work deals with abstract dis-embodied agents.

Distributed Cognition

There are three important ideas here – (a) cognition may be distributed across members of a social network, (b) cognition results as a consequence of coordination between external and internal structures © processes may be distributed over time to affect later events. Hutchins provides the example of a ship navigation to illustrate that different sailors have very specialized tasks that must be carefully coordinated for the ship to navigate safely. An example of how cognition is distributed in space (external) as well as internally – we will often rearrange objects on our desk until an arrangement solves a task like a search – indeed our “messy” work environments are often best suited to finding our belongings. In general, we will adapt the environment to solve the cognitive task, rather than solve it entirely in our heads.

Human Computer Interaction

Tangible interfaces are an emerging area of interest in the HCI community. They deal with using physical artifacts for both representation and control of media. These interfaces allow us to leverage our extant understanding of the physical world (ability to pick up objects, move objects etc.) to manipulate media. Dourish suggests that in an embodied system – meaning and action are tightly bound together – meaning is an emergent process and achieved through action. Lucy Suchman’s workshowed that everyday human activities did not have the detailed level of planning found in AI based approaches – indeed they revealed that while human beings may have long term goals such as “I’m going to work” etc. the details of the action are highly improvisatory in nature. For example, you decide to walk to work instead of using the car, then stop by at the coffee shop to pick up some coffee, and then stop to chat with a colleague before heading off to your physical office. She concluded situational resources rather than detailed plans, play an important role in determining human action.

Ubiquitous computing

Marc Weiser’s vision has resulted is many parallel research efforts primarily centered around paradigms of human computer interaction in moving computing away from the desktop and context awareness (limited to location and activity). Importantly, ubiquitous computing has not dealt with research issues concerned with multi-sensory analysis to help us understand the semantics of the interaction.


A central idea that emerges from these different disciplines is that human activity in the physical world is highly contextualized on the current situational resources. Embodiment allows us to distribute cognitive loads to the environment and also manipulate it to simplify tasks. Meaning can be distributed within a social network, and emerges through dynamic interaction with the environment. Finally, interactions with the environment must be reactive – they must occur in real-time.


Experimental and Avant-Garde Cinema in the 1940s - Postwar Film Culture, Conclusion [next] [back] Exo-Man

User Comments

Your email address will be altered so spam harvesting bots can't read it easily.
Hide my email completely instead?

Cancel or