Lure is an artistic research project related to machine learning in theatre and music. We have explored a variety of machine learning techniques in combination with musical instruments, voice and narration. The artistic research transfers questions on the topics of human–machine interaction and artificial intelligence to the fields of theatre and music.
I created sound and music for the "experimenta explorer", a VR/AR app for Germany’s largest science center in Heilbronn, the experimenta. The app works twofold – as a virtual stroll through the newly constructed building and as a stand-alone game. In the game mode, players get to find out more about select exhibits while assembling a robot from parts they find/win. With their robot they can play through several levels – all in augmented reality.
With the free app "INSIDE MPHIL - St. Nikolai" you can experience the Munich Philharmonic Orchestra virtually on the meadow behind the church of St. Nikolai near the Munich Gasteig as if the musicians were set up there. With the GPS-connected smartphone and headphones you enter the orchestra acoustically and hear the instruments as if you were standing next to them, just as the musicians hear themselves and their colleagues: the violins, the oboe, the timpani or whatever you want to hear. This is made possible by the sophisticated smartphone app.
The sound impression for the musician in the orchestra is clearly different from the sound impression of the audience. Not only the instruments, but the entire body of sound is designed to resonate with the concert hall and enter into a symbiotic connection with the room. As a listener in the hall, one hears the result of this complex connection. Is it possible to let a listener hear the sound of the musician within the orchestra through media? Report on a 360° Virtual Reality research project with the Munich Symphony Orchestra
The change from linearity to non-linearity associated with digitalization is accompanied by a change in the perception of art and culture. We now have the chance to shape our co-existence with the algorithms, not only socially and politically, but also artistically.
ANON: Escape the Murder is a first person adventure game series based on the world of ANON the film created by critically acclaimed and award winning director Andrew Niccol (known for Gattaca, In Time). The mobile action adventure combines the thrilling story-world of the film with cutting-edge mobile technology to create a highly immersive gaming experience.
The standard theatre situation was reconfigured to an installation- and rave-like situation by means of the MAYA augmented reality App. Alive and present as an actor was only the mezzo-soprano Martina Koppelstetter. The digital world from which she originated was only visible through smartphones and AR. Experiencing this world through own smartphones, the audience strolled through the ruins and became the counterpart for MAYA with its presence, augmenting MAYA’s loneliness: MAYA had to sing against a mass of audience members gaping through their smartphones.
VERGEHEN is an opera that you walk. Simply download the VERGEHEN app to your smartphone, take your headphones and walk at the Munich river Isar. You can walk VERGEHEN on any day at any time and in any weather.
VERGEHEN is conceived for a specific path at the Munich river Isar and only works there. The VERGEHEN VR trailer makes it possible to visit this place virtually through VR glasses, even if you can't make it physically to the river.
The singing camera by Mendel Hardeman was an experiment on how to create sound with a video camera: an image-based sound synthesizer. As the sound programmer, I helped him realizing it by taking the video output from his cameras, translating them into sound sources in my Kyma system, and processing them through matrixed cross-modulations and distortion. This is not an interpretation of the visuals; it's listening to the video signal directly as sound. Listen: