Lights change. A scene is revealed. A performer enters, a sound begins, and the audience reacts. A live theatrical performance is a sequence of moments – and today each of them is usually handled by a different operator or a different system, with the connection between them living in a script and in the operators’ heads.

XRV-LTP – Live Theatrical Performance – from MPAI (Moving Picture, Audio and Data Coding by Artificial Intelligence) specifies how to describe a performance in a machine-readable language, so that AI Modules can monitor it, interpret what happens in the venue and act on it according to the script: in a physical theatre, in a virtual world, or across both at the same time.

Real and virtual, together

XRV-LTP belongs to the family of Extended Reality Venues (MPAI-XRV): venues where real and virtual worlds meet. The Real Environment is the physical venue with its performers, spectators, lights, sound and stage machinery. The Virtual Environment is a digital copy or extension of it, in which avatars, scenery and effects exist. The two can be connected, so that an event in one has consequences in the other.

The Script and the Cue List

A performance is described by a Script. The Script contains a Cue List: an ordered set of Cues. A Cue has three parts:

• the Conditions that must hold for the Cue to fire – a performer in a place, a line spoken, a time reached, a spectator’s reaction;
• the Actions to carry out, drawn from an Action Table – light, sound, video, scenery, stage machinery, avatar behaviour;
• the next Cue, so the performance can follow different paths according to what happens.

A Cue can be Real (acting on the physical venue), Virtual (acting on the virtual one) or Mixed (acting on both). The Venue Specifications describe the configuration of a specific venue – the type and positioning of lights, loudspeakers, displays – so the same Script can be staged in many venues.

The AI Modules

The Reference Model of XRV-LTP

• Environment Description builds a description of the venue or venues – real, virtual or both – from what it senses, guided by the Venue Specifications.
• Performance Status Interpretation, Participant Status Interpretation and Operator Status Interpretation interpret what the performers, the audience and the operators are doing that is relevant to the current Cue.
• Action Generation compares the interpreted status with the Conditions of the current Cue. When they are met, it hands the Actions to the Experience Generation modules and moves on to the next Cue.
• RE Experience Generation and VE Experience Generation turn the Actions into data and commands for the real and the virtual environment, as their Venue Specifications say.

As in every MPAI standard, the modules are specified by their inputs, outputs and functions, not by the technology inside them. A theatre may interpret the performers with a neural network and the audience with a simple sensor; another may do the opposite. The modules interoperate because the data they exchange – the Script, the Cue List, the Action Table, the status descriptions – is clearly specified.

Who benefits

• Theatres and venues can describe a performance once and stage it in different venues. Operators can monitor and assist the performance, or allow autonomous operation.
• Technology providers can offer their best module – a better interpretation of gestures, a better generation of lighting – and know that it will work in any XRV-LTP system.
• Audiences can attend a real or virtual environment in person, or virtually through the Virtual Environment, which can be a digital twin of the performance.

Join the public presentation

MPAI presents XRV-LTP on 17 November at 18:00 UTC, in a session open to non-members. It introduces the scope of the standard, the real and virtual environments in which any such performance takes place, and the modules that interpret what happens on stage at a particular time and then generate the next actions per the script.

Register to attend the public presentation held on 17 November, 18:00 UTC.

The specification is on the MPAI web site (mpai.community), under the MPAI-XRV page, and a one-page introduction is at xrv.mpai.community. The XRV Development Committee (XRV-DC) welcomes comments on the specification and on implementations of its modules.