[ISEA2011] Paper: Steve Gibson & Justin Love – Advanced Media Control Through Drawing: Using a graphics tablet to control complex audio and video data in a live context


This paper demonstrates the results of the authors’ Wacom tablet MIDI user interface. This application enables users’ drawing actions on a graphics tablet to control audio and video parameters in real-time. The programming affords five degrees (x, y, pressure, x tilt, y tilt) of concurrent control for use in any audio or video software capable of receiving and processing MIDI data. Drawing gesture can therefore form the basis of dynamic control simultaneously in the auditory and visual realms. This creates a play of connections between parameters in both mediums, and illustrates a direct correspondence between drawing action and media transformation that is immediately apparent to viewers.

The paper considers the connection between drawing technique and media control both generally and specifically, postulating that dynamic drawing in a live context creates a performance mode not dissimilar to performing on a musical instrument or conducting with a baton. The use of a dynamic and physical real-time media interface re-inserts body actions into live media performance in a compelling manner. Performers can learn to “draw/play” the graphics tablet as a musical and visual “instrument”, creating a new and uniquely idiomatic form of electronic drawing.

The act of live drawing, though here removed from its traditional reference to a produced “drawing” (either on-screen or in print), is one that allows for dramatic gesture in a way that pressing keys on a computer keyboard or moving a mouse could never hope to achieve. In addition the fact that the graphics tablet can unite five degrees of control over live audio and video makes it an ideal tool to consolidate the roles of the DJ and the VJ under one control interface.

The paper also discusses how to practically program the application and the authors will present examples of its use as a media manipulation tool.

Audio Demo 1: telebody.ws/TRACEY/Tablet_demo1.mov
Audio Demo 2: telebody.ws/TRACEY/Tablet_demo2.mov
Audio-Video Performance Demo: telebody.ws/TRACEY/Tablet_Demo_Split.mov
Wacom MIDI software download: roguescience.org/wacom2MIDI.zip
Setup the Wacom MIDI software: telebody.ws/TRACEY/Wacom_MIDI_Setup_Demo.mov
Load your saved setup: telebody.ws/TRACEY/Wacom_MIDI_Load_Demo.mov

  • Dr. Steve Gibson is a Canadian media artist, curator, and theorist. He completed his Ph.D. at SUNY Buffalo  (USA), where he studied music composition with Louis Andriessen. He also completed postdoctoral research in media and technology with Arthur Kroker at Concordia University in Montréal. He currently serves as Reader in Interactive Media Design at Northumbria University, Newcastle, UK. He was curator for the Media Art event Interactive Futures from 2002-07. Simultaneously deeply involved with technology and deeply suspicious of its effects, Gibson’s work celebrates both the liberation and paranoia of techno-fetishism. Influenced by a diverse body of art and popular movements his work fuses electronica, immersive art, game art, montage and post-minimalism. He works in a range of media, from live electronic music to virtual reality installation. Steve Gibson’s installations and compositions have been performed in such venues as: Ars Electronica; the Whitney Museum of American Art; Banff Centre for the Arts; Digital Art Weeks; the European Media Arts Festival; ISEA; Interface3, Hamburg; the San Francisco Art Institute; 4 & 6CyberConf. His work has been published internationally by St. Martin’s Press (US), The MIT Press, New World Perspectives (Canada), Turnaround Productions (UK), Future Publications (UK), Urra Apogeo (Italy), and Passagen Verlag (Austria). He recently co-edited a volume entitled Transdisciplinary Digital Art which was published by Springer (Germany) in Spring 2008.
  •  Justin Love, University of Victoria, Canada

Full text (PDF) p. 959-964