Since its creation by Pierre Boulez about thirty years ago, the Institute for Music/Acoustic Research and Coordination (ircam, http://www.ircam.fr/) has been at the centre of what composer Philippe Manoury calls in an article published in September 2007 "the split that electronics provoked in the world of music".

This split immediately led the Parisian institute of research to the heart of the issue of Informed Virtual Environments [ ATLO07, HGH07 ], as many publications by research teams (http://www.ircam.fr/equipes.html) illustrate, whether they are about Instrumental acoustics or about Room acoustics [ Cau06, Cor07, VDWS06 ], about Perception and sound design [ SSM08 ], Analysis/synthesis [ VT06 ], Music representation [ DCA05 ], Analysis of musical practices [ Don06 ], or about Real-time musical interactions [ dB07 ].

The so-called Informed Virtual Environments approach is different from immersion in virtual worlds because it targets to model interactions rather than the world itself (objects, characters, simulation models). The aim is to enable users to live an experience that thus includes experiment, simulation, collaboration and creation in the virtual world.

If the ircam particularity is to associate research and musical creation within the project teams, it is the merit of the composer Philippe Manoury to have written the most synthetic recent paper about the real-time contemporary music state of the art and about the relationship between Informed Virtual Environments and musical composition support tools.

In this article, we shall build on the analysis of this composer to present the intellectual elaboration and the technical performance that make possible the live show named La Traversée de la nuit. We shall show how this work constitutes a strong scientific proposition for Informed Virtual Environments dedicated to musical and artistic contemporary stage creation.

If we have chosen to explore in more details Philippe Manoury's singular contribution, it is because it provides an exemplary overview of all the questions that ircam researchers have been asking about subjects closely related to Virtual Reality.

In his article [ Man07 ], Philippe Manoury relates his fascination and his frustration facing the first attempts to bring together experimental electronic music and instrumental music. That was in the late 1970's and the experiment was limited by the set time imposed by magnetic tape on instrumentalists who were dependent on its inexorable progress. With ircam's first successes in the 1980's, Manoury believed that real time synchronization of electronic music and instrumental playing would give the power of interpretation back to instrumentalists. He explains how as a musician himself he searched for an interactive interpretation of synthesised music to use his own instrumental playing.

At the time, the automatic tools for following scores were still missing. The interactivity between acoustic instruments and machines was only partially completed, "fragment by fragment", during limited and sporadic synchronizations between pre-composed parts of music that were unable to evolve during the performance.

According to Philippe Manoury, the conception of a reliable score follower in fact raises difficult questions: how to detect, recognize and analyze instrumental sounds and gestures in real time. Even though the situation is better for those who improvise without exactly trying to reproduce or notate their interactive works  [2], three main obstacles exist to achieve a score follower.

Philippe Manoury makes two more important points:

Since they met at ircam in 1997, the authors of this article have regularly collaborated in the field of Informed Virtual Environments [ RB07, BBR05, RB03, BR02 ]. Their goal: to layout the foundations of cooperative musical interaction between performers in a broad meaning (musicians, actors, dancers, etc.) and 'intelligent' artificial systems. Their purpose is to bind theoretical knowledge and practical stage approach in the exploration of Informed Virtual Environments for performance arts. They would like to explore all consequences of the paradigms of realtime computation in the field of performing arts, as initialized by Philippe Manoury. Contrary to a priori modelling of characters and situations implied by narrative aspects of performance, they are interested in progressive emergence of knowledge based on human-to-machine interaction to shape forms, structures or similarities.

Their collaboration is based on the common conception of stage 'scenographies' using computers (Virtualis, Norma, Alma Sola,...) addressing research issues; the final realization being achieved by Alain Bonardi (http://www.alainbonardi.net/). These productions represent a real size experiment bringing closer to reality future, more responsive stage devices. They extend previous stage direction approaches trying no longer to put the emphasis on the reification of characters and situations, for instance Reaney's or Wild's ones. Stage director Mark Reaney [ Rea99 ] has explored for years the intervention of augmented reality in theatre shows. In some of his shows, members of the audience wear 3D vizualisation glasses where elements of virtual scenographies are projected. In his staging of Orgia by Pasolini in 2001 (Belfort, France), Jean-Lambert Wild has elaborated a set from the representation of a kind of 'life game' computed from captors giving physiological information about the actors: blood pressure, temperature, etc. More generally, they try to handle computerized device and artefacts on stage to arouse common artistic knowledge in situation of interaction?

The most emblematic collaboration was La Traversée de la Nuit, created in Enghien-les-Bains in November 2003  [3] and performed several times since then. La Traversée de la Nuit involved a multimodal communication stage device that created interaction between the audience, a character on stage and a complex artificial system. The device was centred on emotional communication. The austerity of the scenography contrasted with the intensity of the dramatic situation: after fifty years of silence, a 1944 deportee from Ravensbruck, Geneviève de Gaulle Anthonioz, tells her story.

The idea was to play on duelling temporalities of sound and visual effects, working on different layers of memory to saturate the space between communication and liberty and forcing the actors and spectators to carve out, every night, a space where hope might live.

The stage directions limit the play's artistic interpretation. For example, to show the schizophrenia of the central character, she is played by two actors, one is an actress who speaks the lines, the other a dancer who moves with her and sometimes for her. And although the spoken lines themselves are to be rendered as neutrally as possible, a metaphorical representation of the narrator's mind is created in real time on a large screen at the back of the stage, in the actress' voice. The dancer and the actress both respond to the image, especially the dancer who, being freer, can vary her movements more.

Figure 1. General scheme of interaction (performers, audience, computers) in La Traversée de la Nuit.

Those dramatic choices echo Philippe Manoury's axiom about the composer's visual role in the conducting of a score. The device in La Traversée de la Nuit, for instance, involves a neural network that lets the voice of the actress be processed in real time while "recognizing" emotional states. Those emotions in turn, thanks to a multi-agent system, command the movements of mobiles arranged and projected on the wide screen at the back of the stage. Thus, in way mysterious both to the audience and the actors, the voice's emotional variation produces a visual avatar.

The interaction between the audience and the doubly portrayed character creates a complex perspective. Each player can see his or her emotions and react to them in space and time, even to the point of the actors who play with the projected mobiles. Manoury's processes of premonition, memorization, combining shapes, forming strategies, transitions, proportions and short-circuits can be achieved following narrative time, made visually concrete, then deconstructed through multiple interactions.

One thinks of Husserl in his Leçon pour une phénoménologie de la conscience intime du temps, but also of Whitehead trying to describe his naturalist cosmos [ Hus64, Whi04 ]. The entire theory of reduced listening is brought into question (Pierre Henri, Pierre Schaeffer, Michel Chion), since it must be hybridized with modern theories of visual perception. Thus we are asked to read Maurice Merleau-Ponty, but also Marie-José Mondzain and Laurence Hansen-Løve [ Mon96, HL07 ], within the new perspective of an originary multimodal theory of perception.

Facing a theatre text, every stage director would like to propose his/her interpretation. Let us remind that the interpretation is not immanent to the text, in spite of the numerous indications (playwright foreword, stage indications, etc.). A text cannot exist on stage without an exegesis completed by the stage director.

This interpretation is always an effort to create forms. Let us try to understand this approach in a computer science viewpoint. The stage director process starts by setting a synthetic ontology of the drama: it describes the characters as types (19th century light comedies work with a small number of types, mainly three: husband, wife, and lover) and instanciations, by telling the name of the character, his/her situation at the beginning of the play, his/her costume, etc. During the play, there are instanciation variations  [5]: the spectator discovers that such or such character is far different from what he/she imagined at the beginning. These instanciation variations may lead to ontology revisions. It is for instance the challenge, as well dramatic as metaphysical, in the play El Burlador de Sevilla by the Spanish playwright Tirso de Molina (1630), inaugurating the Don Juan myth: may this character be saved by acknowledging his sins in extremis before his death? May the character's ontology be modified at the very end of the play? Let us notice that this approach may be applied to other forms of dramaturgical, no more based on characters but on situations (in this case, ontologies of situations would be considered).

In this traditional approach of theatre, the concept of similarity by the ontologies is essential. The director stages every scene considering it as an example in a set of cases provided by theatre literature. Explaining a character to an actor consists in pointing in the proposed ontology and link this ontology other ontologies, either of the same play or other ones. Sometimes two stage directions by two different directors can be considered as instances of the same ontology.

In a more general point of view, this is a formal approach, where the example is represented as an instance of a general structure grasping all cases, and one looks for similarities by varying the instanciation. The obvious advantage is to provide an explication of the "similar to the example" side of the proposition, and even a distance measure — this is the way to create a recapitulate concept in intension. Ontologies enable to look for the similarities of an example by staying inside a concept, even if it means to move to the immediate more general concept when the search is unsuccessful. This kind of approach has to do with pre-organization necessities evoked by Manoury. In the field of digital arts, artistic installations as well as virtual worlds usually implement highly structured and completely predefined ontologies that model the environments, the objects, the users, etc.

In a computer science viewpoint, there is another way to deal with the question of similarity [ Per94 ]: it is the interactive data-mining approach, where the example is represented as a specialization of a set of cases. One looks for other neighbouring specializations, but without having at disposal a pre-defined ontology. The user accepts to shape an ad hoc ontology at his/her hand with the interactive help of computers. This is an approach by extension: shaping a similarity consists in shaping a list of contents of similar form by successive amending actions. It uses computing in an interpretative and interactive way, with amending actions on contents and their forms (on the user's side, who is provoked by the proposals of the computer).

In the case of the staging of La Traversée de la Nuit, we have built a dramaturgical Informed Virtual Environment system. It first means we have designed a virtual world with its multi-modal interfaces and its feedback loop: voice (actress) to image (population of kinds of sprites on screen), image to movement (dancer), image to voice (feedback to the actress), movement to voice (dancer to actress). We come to an Informed Virtual Environment, since dramaturgical knowledge is handled by the actor/machine relationship in our system. This knowledge is not a set of predefined ontologies that would "translate" the text into static structures. It is centred on a set of interactions that let the performers play with the system as an instrument but providing at the same time some surprise. If we come back to necessities expressed by Manoury:

Last but not least, we are currently working on a theatre creation named Les petites absences  [6] that is also a research in collaboration with the Real-time Musical Interactions Team at ircam, managed by Frédéric Bévilacqua, and the actors of the Théâtre du Signe in Caen. During this project, we put the emphasis on testing ideas about a future possible dramaturgical working station for performers and stage directors. We name it 'Virtual Assistant for Performers and Stage Directors'. Figure 6 shows a screenshot of the main patch: different layouts of processes are implemented as successive waves starting at the video and sound captation of a performer. After the acquisition, the first step is the analysis layout, the second one the classification level, and the last one the collection level (this level is developed in [ RB07 ]).

Figure 6. Screenshot of the main window of the virtual assistant.

Figure 7 shows a screenshot of the gesture and voice analysis module.

Figure 7. Screenshot of the gesture and voice analysis window.

We start from audio and video capture of performers (dressed in black in contrast with a white backstage wall). Their silhouettes are surrounded with bounding boxes. Descriptors about gesture (quantity of movement, surface of contraction, etc.) and about voice (pitch, loudness, brightness, etc.) are computed for each frame. Then aggregators can be computed for each descriptor: for instance, average value, variance, minimum and maximum values. This gives a 70-component vector that characterizes each sequence.

We therefore constitute a stack of vectors. The most important aspect for us is to arouse the emergence of dramaturgical knowledge thanks to man to machine dialogue. We therefore let the user (for instance a stage director) organize collections of sequences that seem quite close to him/her. From this intuitive personal collection, we infer thanks to a Principal Component Analysis the most significant aggregators (among the 70 present) for the collection. This enables to avoid pre-defined modelling of performing situations, and to shape a kind of emerging interaction.

Second step, we are developing a real-time (for Max/MSP) fuzzy logic  [7] engine able to handle uncertain reasoning [ BTA06 ]. This gives us the opportunity of thinking of semantic mapping between the inputs and the outputs of the scenography. For instance, how can we link the presence of the actor on stage (corresponding to the surface of contraction he/she occupies on the screen capture) to the level of red lightning? First idea would be to write in Max/MSP (that can handle all device for stage command) a predefined mathematical function giving the level of red according to the presence of the actor. On the contrary, our idea is to let some ambiguity in the mapping thanks to fuzzy logic, and to be able to build a feedback loop using fuzzy modifiers. For instance, let us consider a fuzzy rule like 'ifthe actor is very present, the level of red should be high'. From the first empirical observations, we can adjust the rule using fuzzy modifiers, to remain on a semantic level. This semantic level is an important thing when building up an Informed Virtual Environment and considering interaction not only as a numerical mapping.

In Les petites absences fuzzy rules were mainly used to control a generator of crowd sounds - that would fit the behavior of the main actor - or to trigger some events. Stage director M. Bataille-Testu wrote by himself the fuzzy rules.

The outcome is very positive since the play has been performed 25 times since December 2008 giving satisfying results. However, there are several interactive processes that cannot be specified using fuzzy rules: for instance, it is very difficult to program (with fuzzy rules) contrapuntic imitation of an instrumentist.

We are interested in driving scenographies thanks to fuzzy rules. As they enable to work out relationships between input linguistic variables (on the performer side) and output ones (on the electronic side), they constitute interesting ways of stating elements of interactive scenography. As the users may enter their own vocabulary, they provide original ways for artists to specify interactions for stage shows.

During our experiments one difficulty was the understanding of fuzzy rules. First of all, it is not obvious for a non-specialist that the result of a fuzzy implication will give a fuzzy subset and not a crisp value. But with the help of words, this point becomes less problematic. That is why the linguistic counterpart is so important, especially here. Second point is the implicit reciprocity in the rules stating that the reverse is true: e.g. 'if the movement is nearly still, then the frequency is high' may implicitly mean that 'if the movement is not nearly still, then the frequency is not high'.

Figure 8. Screenshot of a sketch of the real-time Max/MSP fuzzy logic engine to link an input (up: quantity of movement, elongation) and an output (down: performer status) of scenographic and dramaturgical 'systems'.

This research provides sketches of tools to implement various aspects of Informed Virtual Environment systems dealing with stage scenography and dramaturgy. Starting from the concept of similarity in extension (opposed to similarity in intension), we have looked for combinations of human practices and computerized artifacts that no longer try to model staging a priori. Our recent research and its application to such theatre shows as La Traversée de la Nuit and Les petites absences confirms the necessity according to us to continue working in the directions set by composer Philippe Manoury in the field of interactive music. Coming to semantic and macroscopic approaches of the actors's status can hopefully enable further exploration of high level human-to-machine interactions on stage.