An

— The use of cutting-edge technologies such as wearable devices to control reactive audiovisual systems are rarely applied in more conventional stage performances, such as opera performances. This work reports a cross-disciplinary approach for the research and development of the WMTSensor-Glove, a data-glove used in an opera performance to control audiovisual elements on stage through gestural movements. A system architecture of the interaction between the wireless wearable device and the different audiovisual systems is presented, taking advantage of the Open Sound Control (OSC) protocol. The developed wearable system was used as audiovisual controller in “As sete mulheres de Jeremias Epicentro”, a portuguese opera by Quarteto Contratem-pus, which was premiered in September 2017.


Introduction
The development of the first glove-based systems for hand data acquisition started about 40 years ago. These systems have been used in many computer input applications, ranging from the analysis of gestures for sign language interpretation to the biomedical sciences. The artistic and entertainment research of these devices concurred with the scientific research and since its inception they were used to explore their potential with video games, virtual reality and musical performance [1].
There is a rich ecosystem of Digital Musical Instruments based on glove systems being developed since the late 1980's that range from experimental designs to consumer products, and there is also an incredible amount of DIY projects. In the 1970's, MIT researchers had demonstrated a general-purpose computer input based on direct interpretation of hand motion [2], but the truly first data-glove for gesture acquisition only appeared in 1977, developed by Tom Defanti and Daniel Sandin: the Sayre Glove.
The first commercially available glove-based systems, the DataGlove was built by Thomas Zimmerman in 1987 and later developed in a new fibre optics version by Visual Programming Language Research Inc, the VPL DataGlove. This system quickly gained the attention of researchers of different fields and a number of devices simi-lar to it were proposed, one of those would became well known among video games players, the Nintendo PowerGlove. Some years before, in 1984, Michel Waisvisz built one of the earlier and most famous examples of a gesture input system for musical performance that would later be followed by projects such as Laetitia Sonami's "Lady's Glove", the "Bodycoder system" from Julie Wilson-Bokowiec and Mark Bokowiec and more recently the "Mi.mu gloves".
In short, during the 1980's a boom of data-gloves research and development took place and would follow for the next 30 years until today, where the renaissance of Virtual Reality emerged the research for gestural modes of interaction on virtual and augmented environments in the performing arts [3].
Opera has always been a technology pioneer in stage-based performances. Opera houses had from the start engineering departments for dealing with acoustics, stage machinery, lighting and special effects. Even today opera continues to push further in some domains like projection mapping or innovative stage machinery, and provides a great field study for the development of wearable devices and embed computer systems, since it gathers in the most basic form music and gestures [4,5].
In this paper, we present a cross-disciplinary approach for the research and development of the WTMSensorGlove, a data-glove used in an opera performance to control through gestural acquisition the sonic and visual content on stage.

Wireless Wearable Device
WTMSensorGlove is a data-glove used in the opera "As Sete Mulheres de Jeremias Epicentro", a production that moves towards the research on the development of multimedia operas using wearable devices to control the audiovisual content on stage. This device is operated by the singers on stage to control sonic and visual content, like real-time voice and instrument sound processing, and the visual elements of the digital scenography, as shown in Fig. 1. For this purpose, the implemented approach is based on the concept of Digital Musical Instruments for data acquisition and mapping.
The WMTSensorGlove is a wearable device that can be used by one or more singers/musicians on stage, to control specific sonic and visual elements. Envisaging that purpose, the system architecture from controlling the audiovisual elements is shown in Fig. 2 and comprises: • WMTSensor Glove; • USB-MiWi Dongle; • HID-to-OSC application; • Hardware dedicated for controlling the audiovisual elements. The main element of the system, the wearable sensor glove, comprises two inertial measuring sensor modules, two digital switches and a battery. The concept design of the WMTSensor Glove is shown in Fig. 3, the materialization of the first prototype components is shown in Fig. 4. Both of the sensor modules have an antenna that allows them to communicate wirelessly using the MiWi protocol with the dongle connected to a computer system that maps the data onto higher level information to be used with audiovisual software.
Wrist movements from the opera performer are one of the most important in this context of space and stage-base performance. Therefore, one of the modules is placed in posterior side of the hand and the other one is placed in the forearm between the wrist and the elbow joints. Each sensor module has an inertial sensor (6-axis low-power IMU with accelerometer and gyroscope) and data fusion routines are implemented in the microcontroller in order to output the module orientation in a quaternion representation, as well as the raw acceleration and rotation velocity of each module.

176
http://www.i-joe.org The tracking of wrist movements by the performer is possible through the comparison of the orientation of both sensor modules. An embedded electronic solution was prototyped in order to achieve the small dimension required for fitting the wearable device in the performer's figurine.
Additionally, each sensor module has a four wire JST connector that allows the use of two external digital inputs used for giving the possibility of the performer to order the execution of other commands and that can provide SPI communication, allowing the expandability of the system. Furthermore, the wireless transceivers make possible to have data transmission ranges of roughly 40 meters indoors and more than 100 meters in line of site, feasible for live performances that usually take place in large stages. The sensor modules output their orientation with an update frequency of 60Hz. Other advantages of using this wireless transceiver technology is the possibility to have multiple sensors on different performers connected to one computer, or even connect several transceivers modules as repeaters, thus having a mesh network that can increase the distance of the communication if needed.
A 5V power bank with two USB outputs is used as a replaceable battery that each of the sensor boxes trough a USB-mini-B connector, allowing the performer to move freely around the stage and have several hours of continuous power, which is important not only for the performance but also for the practice sessions.
The data transmitted from sensor modules of the WTMSensorGlove is received by the computer trough a custom developed USB-MiWi Dongle. This dongle is a custom HID device and is responsible for receiving the MiWi packages from both of the sensor modules, decode the encryption and send them to the HID-to-OSC software. In order to build this software application, code already provided by Microchip Technology for HID custom devices was used and adapted to the framework.
The HID-to-OSC application starts by interfacing with the USB Dongle using a specific device descriptor which detects the connection state and reads the raw data from the device's buffer, then creates a data packet to be sent through specific Open Sound Control (OSC) addresses and ports. The representation of the dataflow through the system is show in Fig. 5. The use of OSC protocol allows the communication between our wearable device with computers, sound synthesizers, and other multimedia devices and audiovisual elements since it's an open-ended dynamic protocol that is standard and vastly used in this art performance scenery [5,6].
The glove-based controller used in this opera, unlike conventional acoustic instruments in which a performer "excites" the physical mechanism of the instrument to produce sound [7], transforms the performer's motion energy into digital data that in turn is mapped into audiovisual outputs.
The natural movements of the arm and hand with an audiovisual system need to be compose within the musical, choreographic and narrative framework. In order to identify the gestures that control the audiovisual elements of the opera, the sensor modules' data outputs were used in two different instances of Wekinator [8]. The use of Wekinator is specifically appropriated since it also uses OSC to communicate, simplifying the communication between the HID-to-OSC, machine learning software and audiovisual software by all of them using the same protocol. Each instance of Wekinator was using a different machine learning strategy in order to identify two different types of gestures [9] in order to answer the musical and choreographic needs, as shown in Fig. 6. In total, the system uses three different computers: • Computer 1: has the USB-MiWi Dongle connected that interface with the WMTSensorGlove modules and run the HID-to-OSC software application. Two instances of Wekinator perform the gestural analysis through machine learning; • Computer 2: running Max/Msp to capture the audio inputs of the microphones. The audio inputs are modified with parameters coming from Wekinator through the OSC, enabling the use of five different performer controlled effects; • Computer 3: running Unity3D and is responsible for visual control of virtual objects and for the projection system.

Results
The developed WMTSensorGlove system was used in the opera "As Sete Mulheres de Jeremias Epicentro", performed by the Quarteto Contratempus that was premiered in September 2017.
Both the technology and the performance outcomes were put together in order to meet the initial expectations of the composer and be subdued to the rehearsal sessions and the first tour of the opera.
The gerture recognition analysis performed through machine learning by the two instances of the Wekinator allowed different performers to control and manipulate visual elements in the projection, as well as controlling different performer controlled effects: Harmonizer, Looper, Freezer, Sampler, Virtual instrument and effects section at the end of the audio route.
The developed system modules wireless communication protocol was proven to be working continuously without communication issues during the whole performance, even with a full concert room.
Until the present moment, there has been three official opera performances using the developed system. The many press publications have reported the developed wearable system to be a technology innovative concept in the use of gestural acquisition apllied to stage-based performance arts, with a total of 11 press clipping references and demonstration videos can be accessed online in [10].

Conclusion
The developed device and software presented, the WTMSensorGlove and the developed HIDtoOSC application, may contribute significantly to the evolution of musical practice using digital musical instruments and more comprehensively the use of wearable devices to control interactive audiovisual systems in into the environment of art performance, such as operas.
The adoption of a data-glove to control a computer system that represents gestural data into interactive/responsive audiovisual narratives contributes to the study of the creative solutions for interfacing human movement with computers through sonic and visual representations. The challenges on how to integrate instrumented devices with responsive systems is clarified by employing performative, choreographic and narrative practices to define gestural strategies for audiovisual outcomes. This perspective of using stage-base performances as an unconventional laboratory for the research of new technology, strengthens the idea that performance art can expand the technological advancement of instrumented devices by artistic experimentation.