HyperAudio: Location-Awareness Adaptivity

Daniela Petrelli, Elena Not, Marcello Sarini, Oliviero Stock, Carlo Strapparava, Massimo Zancanaro
IRST
Cognitive and Communication Technologies Division
I-38050 Povo - Trento - Italy
+39 461 314 444
{ petrelli | not | sarini | stock | strappa | zancana } @irst.itc.it

Statement of the Problem Addressed

HyperAudio lays where location aware systems and adaptive systems overlap. It aims at providing personalised information to a user carrying a PDA (Personal Digital Assistant) by exploiting data on the user’s position: the system follows the moving user and provides suitably tailored information at the right time.

The purpose of this demo is to show how the HyperAudio system adapts information presentations (in the context of a Museum of Natural Science) to (i) each single user, (ii) who is performing a certain task, (iii) in a physical context. Each of these three facets influences in a different way and to a different degree the process of dynamically building presentations. HyperAudio capabilities will be exemplified by comparing different sequences of interactions, pointing out how these turn into different selections of content, language style and linguistic features. Moreover, for each interaction context it will be shown how further exploration steps in the visit are proposed to the visitor (as displayed links or as explicit suggestions), and how a coherent discourse is dynamically maintained as main support to a personal path of visit.

Scenario

Figure 1 shows the hardware components of the HyperAudio system as it is in its current implementation: each visitor is equipped with a Newton MessagePad130 endowed with headphones, on which an infrared receiver is mounted. Each meaningful physical location (i.e. relevant exhibit object, entrance door, passage) has a small power-autonomous infrared emitter that continuously sends a code that uniquely identifies it. The physical space is thus partitioned into sensible zones allowing the system to identify the visitors’ movements and, to some extent, their orientation. The visitor interacts with the system (see Fig.2) both when approaching a remarkable point (implicit input) as well as when pen-pointing on the PDA screen (explicit input).

Figure 1. HyperAudio equipment

Figure 2. A user moving with the HyperAudio equipment

HyperAudio dynamically creates a presentation choosing a sequence of audio files and a list of relevant follow-ups, taking into account models of user, physical context and previous interaction.

The purpose of the demonstration is to give an idea of the HyperAudio system showing how different presentations can be built starting with the same data.

The first section of the demonstration will introduce general concepts (i.e. location-awareness and adaptivity) while overviewing HyperAudio components.

In the second part, examples will show how adaptation is performed by the system. They will describe how information coming from models of the user, the physical context and the previous interaction affects:

content selection and organization, in terms of audio files and of links selection;

language style, e.g., with the possibility of adopting simplified language for children;

linguistic choices, to allow for the inclusion of correct demonstrative expressions and spatial references;

navigation suggestions, in terms of which objects to see next or which hyperlinks further explore in the virtual information space.

Narrative

Many physical spaces, like museums or cities, may be associated with an information space. An information space is defined as a set of pieces of disparate contents related to each other to some extent which are associated to the objects in the physical environment. For example reptiles can be described by general features, by their evolutionary development, by classification, by lifecycle an so on. When visiting the reptile room in a science museum, people might wish to explore this rich information space, but, since not all people are alike, only a reduced part of the available information can be of some interest.

From this point of view, the ideal guide is the one that, following the physical movements of the visitors and reasoning on their (partly guessed) interests, selects and presents the information best suited to each person. The HyperAudio system aim is to act as a personal guide that, following the visitors in their movements, proposes personalised information at the proper time.

The basic idea of HyperAudio derives from augmented environment technology: through their movements the visitors of an "augmented museum" interact with the displayed "augmented objects", allowing the system to always know their position and making it possible to dynamically follow their current path. As soon as the visitor approaches a sensible zone (i.e. a zone equipped with an infrared emitter), the HyperAudio system catches the infrared signal, but it does not simply react to it. In fact, HyperAudio interprets visitor’s movements as an implicit input for planning a smart response. Implicit input is not limited to the action of approaching a case, but it includes standing still, going away or passing quickly from one zone to another.

Moreover, the user can directly interact with the system using the pen on the PDA (Personal Digital Assistant) screen (here called explicit input) for example to select a further elaboration on the topic.

Both implicit and explicit inputs compose the interaction history, which represents a summary of the visit (both physical and virtual) up to that moment. It plays a very important role in the adaptation process since HyperAudio refers to it when planning the next presentation, deciding, for example, to not play a piece of content already presented or to introduce a comparison with an object already seen.

The system dynamically creates a personalised presentation. This means that no fixed presentation exists in the system, but rather they are composed on the fly on the basis of a suitable subnet of contents. A presentation is composed of:

a sequence of audio files that are played through the headphones;

a set of relevant links worthy of future exploration, depicted on the PDA screen as buttons;

an image related to the object described or an oriented map displaying the user’s current position.

HyperAudio hypertextual presentations realize both link and content adaptation since only one part of the information space is accessible at a given time and the content of each presentation step is dynamically organised to best fit the present communicative purpose.

As an example of HyperAudio adaptivity, let us suppose that the visitor stops in front of a case containing a stuffed squirrel. The system may propose a description of the animal, suggesting (as clickable buttons on the screen) additional explorable elaborations. The system chooses the elaborations to suggest on the basis of the visitors’ physical location, their interests, what they have previously seen, and so on. For example, if the visitor has displayed interest in prey-predator relationships, the system may add a button to the description of the squirrel’s predators; if the visitor selects this elaboration and if the case containing a stuffed specimen of the predators is close by, the system will also augment the information about the squirrel’s predators with the suggestion to go and have a look at the specimen, explaining how to reach it.

HyperAudio organizes the information space as a hypermedia. Information related to a certain object are organized as a network; each node (called macronode) contains a unit of content plus some additional information such as the physical position of the object, the type of content (e.g. introduction, object description etc.), and the existence of coherence relations with other macronodes (e.g. elaboration on the same topic, comparison with other objects, etc.).

When the system composes the actual presentation only a few of the available macronodes are selected, depending on internal factors such as their contents or the object’s physical position, or on external ones such as the user model, the interaction history or the discourse coherence.

Figure 3 shows a fragment of the macronode net. It is worth noting that the information about a content topic, e.g. the salamander, are spread on different macronodes, each one containing a piece of information on a different aspect. In the figure, for example, the first node of the salamander subnet contains an introduction, and it is related with other two macronodes through an elaboration rhetoric link. At presentation time, the system may decide to consider as general description not only the introduction, but the legend as well, if, for example, the visitor displayed interest on curiosities.

Instead, for another visitor not interested in legends, the system will present only the introduction macronode leaving the legend one for further possible exploration, i.e. as a follow up button displayed on the screen.

Figure 3. A sample portion of macronode network

A macronode is atomic with respect to its content but it can have some optional parts that are selected only in particular discourse contexts. Figure 4 shows the internal content structure of a sample macronode, i.e. the transcription of the precanned audio files net. In general the macronode content is organized as a graph that may be traversed in different ways generating different messages: nodes are fragments of discourse that may be combined following the links. Links are labelled with conditions on context features that are used by the system at composition time to decide the current activation. The macronode audio file network in figure 4 can be instantiated in the following messages:

when the visitor is in front of the object being described (so the feature deictic is selected): "what you’re seeing is the spotted salamander <PAUSE> it gets its name from the many spots on its black or bluish black body";

when the current message has to be added to another one with the same topic (so a pronoun has to be used): "it gets its name from the many spots on its black or bluish black body";

when there are no special condition (so the standard message will be selected): "the spotted salamander gets its name from the many spots on its black or bluish black body".

From this example it is clear that whatever instantiation is choosen, the content of the final message is the same. More precisely, an audio file network encodes lexical variations of the same message.

Figure 4. The internal structure of a sample macronode.

Examples of Visits with HyperAudio

In the following examples, two different visiting paths will be presented and discussed.

At the museum entrance the visitor is provided with a Newton MessagePad130 endowed with headphones on which an infrared receiver is mounted. The visitor is asked to put on the headphones positioning the infrared receiver above their neck (this ensures that just infrared signals coming from the front are correctly received).

Then the visitor is required, interacting with the PDA, to fill in a short questionnaire. This preliminary action allows on one hand the user gets used to the new tool and on the other it allows the system to compile a user profile. The profile represents some relevant characteristics of the current user and will be used by the system as a starting point for building a dynamic user model.

The following Demo1 and Demo2 demonstrate HyperAudio adaptability potential in:

content selection and organization, according to interaction history (as in Demo1 Step3 and Demo2 Step 6 below) and visitor’s interests (as in Demo2 Step4);

language style, according to visitor’s knowledge (as in Demo1 Step3 and Demo2 Step1);

linguistic phrasing (as in Demo1 Step 2 and Demo2 Step 4 below);

navigation suggestions (as in Demo2 Step 4);

In Demo1 we suppose that the visitor has a good background in Natural Science (as specified by the visitor herself in the preliminary questionnaire filling). In Demo2 the visitor is supposed to have no specific background in Natural Science.

The texts reported in the following examples are excerpted from some pages of Britannica Online at

http://www-lj.eb.com/

Demo1

This demo presents some basic examples of how the HyperAudio system personalize information depending on visitor’s behaviour, position, and knowledge.

Step 1: The visitor enters the reptile room

HyperAudio plays the following general description

This rooms displays some members of the class Reptilia, animals that occupy an evolutionary position between amphibians, on the one hand, and birds and mammals on the other.

This introdiction might be extended depending on the visitor’s behaviour. Let us suppose that the visitor moves slowly looking around: to better support this exploration Hyperaudio adds some more general information.

Reptiles is a broad class which number some 6,000 living species very different for morphology and habits. In fact reptiles includes snakes, turtles, and lizards that succeded in adapting to deserts, forests, fresh warets, and even in air and seas.

Step 2: The visitor approaches the lizard

HyperAudio plays an object description

This is a Lizard. Lizards are by far the most diverse group of modern reptiles in body shape and size. They range in total length from geckos of three centimetres to monitor lizards of three metres, and in adult weight from less than a gram to more than 150 kilograms.

Note the use of "this" (deictic language) to focus visitors’ attention on the displayed specimen.

Step 3: The visitor then goes to the next room, to the case where a salamander is displayed

This is a Salamander. Even if it is quite similar to the lizard you’ve just seen, it is not a reptile. Salamander in fact is the common name for about 320 species of Amphibians of the order Caudata. They are distinguished from other amphibians, mainly frogs and toads, most obviously by the presence of a tail in all larvae and adults. Like the lizard you saw previously, this salamander is a cold-blooded animal.

Note the comparisons with the specimen seen before.

Note also that the system uses scientific terms (e.g., "Caudata") assuming that the user knows them or can assimilate the new information.

Step 4: The visitor selects a follow-up description from the screen

The visitor is still in front of the salamander. HyperAudio displays some ‘Additional Info’ buttons on the palmtop screen: she selects the salamander-legend follow up.

The name Salamander is derived from an old Arab\Persian word meaning "lives in fire", stemming from an old belief, false of course, that the salamander could walk through fire and remain unharmed.

Note how the visitor is allowed to explore the information space by explicit browsing.

Demo2

This demo describes an alternative, more complex, visit path and displays some further HyperAudio features.

Step 1: The visitor approaches the salamander

The system plays an object description.

This is a Salamander. Salamander is the common name for about 320 species of Amphibians with tail. They are distinguished from other amphibians, mainly frogs and toads, most obviously by the presence of a tail in all larvae and adults.

Note that this text is similar to the text in Demo1 Step3 , but the comparison with lizards is missing because it is not relevant here.

Furthermore, the scientific term "Caudata" is not mentioned here, in favour of a more straightforward description, given that the visitor declared to be a non-expert in the Natural Science field.

Step 2: The visitor selects a comparison.

The visitor is still in front of the salamander. She selects the Amphibians-Reptiles comparison on the palmtop screen.

Amphibians were the first vertebrates to move from an aquatic environment to land, and they are the ancestors of all reptiles, birds, and mammals. The big evolutionary step made by reptiles was the final emancipation from life in water. That breakthrough required two basic changes, the first of which took place in the skin and the second in the eggs.

HyperAudio displays on the screen two follow up buttons on skin and eggs.

Step 3: The visitor requests additional information

The visitors selects "skin" follow up button.

Amphibians have naked skin that lacks of any protective devices against desiccation. This susceptability to drying out forces amphibians to remain in water or in very humid places. Reptiles, instead, evolved a type of scales consisting of keratin or horn. This type of scale was in a position and of a thickness to prevent desiccation, thus allowing reptiles exploration of the terrestrial environment.

The system proposes some reptiles description as follow up.

Step 4: The visitor selects the lizard follow up

Lizards are reptiles quite similar to salamanders. Lizards are by far the most diverse group of modern reptiles in body shape and size. They range in total length from geckos of three centimetres to monitor lizards of three metres, and in adult weight from less than a gram to more than 150 kilograms. A specimen of lizard is displayed in the in the room at your right.

Note that this text is mainly the same as text Demo1 Step2, but a comparison has been introduced and deictic language is not used here, because the visitor is not in front of the object.

Given the fact that the visitor showed her interest in the lizard and a specimen is displayed near by, the system suggests where she can see a real one.

Step 5: The visitor goes towards the lizard

This is a Lizard. Even if it is quite similar to the Salamander you saw in the previous room, a lizard is not an amphibian but a reptile. Looking carefully it is possible to see the tipycal keratin scales that cover the outermost layer of the skin.

Note that although the situation is similar to that of Demo1 Step2, with the visitor in front of the lizard, the system has prepared a new description in order to present a different information with respect to that already presented in Demo2 Step4..

Moreover, because the visitor showed interest on some characteristics, suggestions about skin observation are added.

Step 6: The visitor goes back to the salamander

The visitor already listened the generic description of the salamander. Her going back to the specimen can be intended as interest on the discussion about the different skin in amphibians and in reptiles. HyperAudio prepares a deep comparison.

The most distinctive and important feature of amphibians in general and salamanders in particular is their smooth, moist skin. This organ contains mucous and poison glands as well as pigment cells and serves a major respiratory function.