The singing camera by Mendel Hardeman was an experiment on how to create sound with a video camera: an image-based sound synthesizer. As the sound programmer, I helped him realizing it by taking the video output from his cameras, translating them into sound sources in my Kyma system, and processing them through matrixed cross-modulations and distortion. This is not an interpretation of the visuals; it’s listening to the video signal directly as sound. Listen to the result:
Starting point
Whenever a camera sees a bright picture, it produces a high-amplitude electric signal. A dark picture is translated into a low amplitude. Each second the camera scans the picture 50 times, creating a 50Hz period in the voltage. This voltage can be made audible by simply plugging the signal into an audio system. In other words a camera, when modified, can become a sound synthesizer. This idea was the starting point of Mendel’s research.
If you plug the signal of the video camera into an audio system, what you hear is mainly the 50Hz buzz covering all the other information carried by the signal. If you manage to get rid of the sync-signals, what should remain is a pure translation of LIGHT into VOLTAGE.
Listen to the sound of the pure camera output, pointing to a zebra pattern:
Cleaning the signal
The task was to clean the signal from the 50Hz burst, commonly also known as black burst. It is used to synchronize analog video technology.
The task was to cut away the 2ms blanking period in the beginning of the 20ms cycle (at 50 half frames per second) and then re-speed the remaining 18ms of the active video period to a seemless 20ms cycle.
To detect the burst the brilliant Lex van den Broek by the Technical Department of the Royal Conservatory The Hague built a box which outputted a short 22kHz burst whenever the burst started. After this conversion to an audio signal it could be detected by Kyma. Inside Kyma it would write the cycle to memory, read out milliseconds 3-20 and play it back with a speed factor of 1,11, resulting in a fully cycle. We added a live parameter to be pitch the signal even much lower for sonic effect.
Performance
After finding the way to clean the sync-information off the signal, you can hear video. It is not an interpretation of the signal through, for example, a computer – it is the video itself. If then you use the camera to scan patterns of black and white horizontal bars, each white bar will produce a pulse, and each black bar will produce silence. In this way the camera becomes a pulse generator, and the amount of black and white bars determine the frequency you hear. The less bars the camera sees, the lower the frequency it produces. The more bars it sees, the higher the frequency. The closer the camera is to the pattern, the less bars it sees. Moving the camera away from the pattern, it has a bigger overview and thus sees more bars. In this way you create a glissando passing through all the available pitches.
In this project, the players of a baroque ensemble received small Black & White surveillance cameras. The recorder player had one fixed on the end of her instrument; the harpsichordist had two, one on each wrist; the violone player had one on his forehead.
The colour of the sound (timbre) varies according to the kind of pattern filmed. It can be influenced by modifying the brighness of the white or the black. The timbre also changes according to the thickness of the white bars in relation to the black ones. The sound becomes grittier through the introduction of dirt (texture) instead of pure black and white. The sharpness of the sound is influenced by blurrint the borders between the bars. It also changes by rotation of the camera: if a bar is not completely horizontal the pitch becomes less clear. The focus of a camera can also be used to create a sharper or less sharp sound (this works as a low-pass filter).
By mixing different patterns in one image you hear an interval instead of a single-pitched sound. In this way, music can litterally be drawn – any horizontal pattern of black and white is suitable: a black text on white paper; a wall of white bricks with dark cement; a zebral a harpsichord keyboard.
A new instrument, with a huge range of possibilities, just waiting to be used.
