What Multisensory Augmented Reality Can Bring to Dance Heritage
A traditional dance can be difficult to understand from a recording alone. A viewer may see the steps yet miss how a musical phrase guides a turn, how dancers respond to one another, or why a gesture matters to the people who perform it. Multisensory augmented reality offers ways to explore these relationships, bringing movement, sound, and selected tactile cues into an encounter with dance heritage.
SCHEDAR—Safeguarding the Cultural Heritage of Dance Through Augmented Reality—was a European research project active from 2018 to 2021. Its combination of dance documentation, 3D motion data, digital archives, and multisensory technologies provides a useful starting point for considering what interactive experiences should help people understand.
What makes an experience multisensory?
Augmented reality places digital content within a view of the physical environment. In a dance heritage experience, this might mean displaying a recorded dancer’s movement in a classroom or showing a group formation across a museum floor.
A multisensory approach connects this visual content with other channels. Music can accompany a movement phrase; spoken commentary can explain its context; an optional vibration might mark a timing cue.
Each channel should have a clear purpose. Adding more sensations does not automatically create a more meaningful encounter. The design question is whether a cue helps someone notice, interpret, or practise something relevant to the dance.
Making movement easier to examine
Showing pathways and relationships
A 3D representation can let learners examine movement from different viewpoints. Foot trajectories might highlight a travelling pattern, while a view from above could show how a circle opens, closes, or changes direction.
Consider a hypothetical teaching sequence in which a learner struggles to follow a turn. The experience could first show the whole phrase, then offer a repeated segment with a direction marker and a practitioner’s explanation. The learner could return to the complete performance once the movement makes sense.
Group relationships deserve equal attention. A formation view can reveal spacing and orientation, but it should also explain how participants coordinate: through shared rhythm, visual attention, physical contact, or other conventions identified by practitioners.
Explaining what motion data leaves out
A moving avatar is a representation of a recorded performance. Its detail depends on what was captured and how the data was processed.
A foot path, for example, does not by itself explain weight transfer. A simplified body model may omit facial expression, clothing movement, or subtle hand contact. Practitioner commentary and contextual video should accompany the visualisation where these details matter.
Designers should also identify whether a view follows the dancer’s own left and right or presents a mirrored teaching version. That small distinction can prevent considerable confusion.
Connecting music, timing, and touch
Music should be treated as part of the documented practice, with information about the performers and recording context. A generic soundtrack may accompany an animation attractively while obscuring the relationship between the dance and its music.
Useful interactions might include replaying a musical phrase alongside its movement, switching between the complete recording and a clearly labelled teaching cue, or listening to a practitioner explain an accent or transition. These options should reflect how the particular tradition is taught. Counting evenly spaced beats will not explain every rhythmic structure.
Playback changes need care. Slowing movement without adjusting the accompanying sound can separate actions from their musical cues. Even when both remain synchronised, a slowed version may alter the performance’s feel. Learners should be able to return easily to the original tempo.
Optional haptic cues, such as gentle vibrations, could offer another way to mark timing. Their meaning must be explained. A vibration cannot reproduce the sensation of a floor, a partner’s support, or the bodily experience of dancing; it provides a designed signal that users may find useful.
Keeping cultural meaning visible
Interactive cultural heritage needs interpretation alongside interaction. A visitor should be able to discover who contributed a recording, where it was made, what occasion it documents, and how practitioners describe the dance.
Community contributors should help decide which movements need explanation, which variations to include, and what material is appropriate for public presentation. Permissions should cover the intended uses of recordings and motion data, including their transformation into avatars or teaching sequences.
This also affects feedback. An application that scores every learner against one recording risks presenting that performance as the definitive version. Where practitioners recognise variation, an experience should make room for it. Feedback can address an agreed learning goal—such as following a sequence—without claiming to judge the full cultural or artistic quality of someone’s dancing.
Designing accessible dance heritage experiences
Multisensory design should offer choices. Requiring users to see, hear, and move simultaneously can create barriers. The World Wide Web Consortium’s XR Accessibility User Requirements identifies needs including alternative controls, descriptions of audio content, personalisation, and interactions that do not depend on particular bodily movements.
For dance heritage experiences, practical design options include:
- Captions for speech and text descriptions of meaningful musical cues.
- Audio descriptions that explain movement and spatial relationships.
- Seated exploration and controls that do not require stepping, turning, or precise gestures.
- Adjustable text, contrast, volume, and cue intensity.
- Easy ways to pause, repeat, or disable optional effects.
A person should be able to explore a dance without performing it. A screen-based route can also support people who cannot comfortably use a headset.
Developers should test these options with disabled participants early, including whether controls and descriptions remain usable during the experience.
Evaluating understanding and learning
An experience can be enjoyable without producing the intended learning. Evaluation should begin with a specific goal and gather evidence appropriate to it.
Match the assessment to the purpose
Different goals call for different tasks:
| Intended outcome | Possible evaluation task |
|---|---|
| Understand a formation | Describe or arrange dancers’ positions during a transition |
| Recognise a musical relationship | Identify the phrase associated with a movement change |
| Learn a sequence | Recall or attempt an agreed phrase, where appropriate |
| Understand cultural context | Explain who performs the dance and on what occasions |
| Explore independently | Find, replay, and interpret material using preferred access options |
Practitioners and educators should help define what counts as a useful response. Physical performance should be assessed only when it is relevant, voluntary, and appropriate to the participant.
Separate engagement from learning
Brief tasks before and after an encounter can reveal changes in understanding. A later follow-up can explore what participants retain once prompts are removed.
To investigate the contribution of AR, researchers could compare it with video presenting similar content over a similar learning period. To examine a tactile cue, they could compare versions with and without that cue. These comparisons help identify which features deserve further development.
Observation and interviews add essential context. Did participants understand the viewpoint? Did they mistake an instructional marker for part of the tradition? Could they explain the experience in their own words? Record comfort, access barriers, and technical interruptions alongside learning outcomes.
A thoughtfully designed encounter can end by connecting visitors with practitioner-led teaching, community events, or further archive material. That gives digital exploration a practical next step: an opportunity to continue learning through the people who sustain the dance.