[ISEA2011] Panel: Steve Gib­son – Sim­u­lat­ing synaes­the­sia in real-time per­for­mance

Panel Statement

Panel: Visual Effects Remixed

In this paper the au­thor will de­scribe and show ex­am­ples of his live au­dio-vi­sual work for 3D spa­tial en­vi­ron­ments. These pro­jects use mo­tion track­ing tech­nol­ogy to en­able users to in­ter­act with sound, light and video using their body move­ments in 3D space. Spe­cific video ex­am­ples of one past pro­ject (Vir­tual DJ) and one cur­rent pro­ject  (Vir­tual VJ) will be shown to il­lus­trate how sub­jec­tive and flex­i­ble user in­ter­ac­tion is en­abled through a com­plex but pre­dictable map­ping of 3D space to media con­trol. In these pro­jects au­di­ence mem­bers can in­ter­act with sound, light and video in real-time by sim­ply mov­ing around in space with a tracker in hand. Changes in sound, light and real-time vi­sual ef­fects can be syn­chro­nized with changes in sound and or light (i.e. music vol­ume = light bright­ness = video opac­ity). These changes can be dy­nam­i­cally mapped in real-time to allow the user to con­sol­i­date the roles of DJ, VJ and light de­signer in one in­ter­face. This in­ter­ac­tion model sim­u­lates the ef­fect of synaes­the­sia, in which cer­tain peo­ple ex­pe­ri­ence light or colour in re­sponse to mu­si­cal tones.

  • Dr. Steve Gib­son is a Cana­dian media artist, in­ter­face de­signer and media cu­ra­tor. He com­pleted his Ph.D. at SUNY Buf­falo, where he stud­ied music com­po­si­tion and elec­tronic music. He was also a post­doc­toral re­searcher in media and tech­nol­ogy at Con­cor­dia Uni­ver­sity in Montréal. He cur­rently serves as Se­nior Lec­turer in In­ter­ac­tive Media De­sign at Northum­bria Uni­ver­sity. He was cu­ra­tor for the Media Art event In­ter­ac­tive Fu­tures from 2002-07 and cur­rently serves as co-Di­rec­tor of Dig­i­tal Art Weeks (ETH Zurich). He cur­rently holds a Vis­it­ing Pro­fes­sor­ship at Xi’an Acad­emy of Fine Arts, China and is Artis­tic Di­rec­tor of Lim­bic Media Cor­po­ra­tion, a com­pany spe­cial­is­ing in phys­i­cal com­put­ing ap­pli­ca­tions. In­flu­enced by a di­verse body of art and pop­u­lar move­ments his work fuses elec­tron­ica, im­mer­sive art, game art, mon­tage and post-min­i­mal­ism. He works in a range of media, from live elec­tronic music to im­mer­sive and phys­i­cal in­stal­la­tion. His works have been pre­sented in such venues as: Ars Elec­tron­ica; the Whit­ney Mu­seum of Amer­i­can Art; Banff Cen­tre for the Arts; Dig­i­tal Art Weeks; the Eu­ro­pean Media Arts Fes­ti­val; ISEA; In­ter­face3, Ham­burg; the San Fran­cisco Art In­sti­tute; 4 & 6Cy­ber­Conf. He re­cently co-edited a vol­ume en­ti­tled Trans­dis­ci­pli­nary Dig­i­tal Art which was pub­lished by Springer (Ger­many) in Spring 2008.  telebody.ws  grandtheftbicycle.com  Video: “Wacoff” Tablet Demo

Full text (PDF) p. 955-958 [Longer title]