[ISEA2011] Paper: Grethe Mitchell & Andrew Clarke – Playground Gaming with Wii, Kinect and Processing


This paper is a case study in the design and development of an interactive application produced as part of an AHRC “Beyond Text” Large Grant project. It uses open source software (Processing, libfreenect, Darwiinremote, OSC) and hacked games hardware (both the Microsoft Kinect sensor and Nintendo Wiimotes) to create a low-cost motion tracking system that allows the recording, playback and analysis of children’s playground games in 3D.

By doing this, the application provides researchers in the humanities with a new and innovative way of analysing and archiving these  games. The paper covers both the technical aspects of the project and the academic/editorial challenges in making the application enjoyable to children as a game and useful to researchers as an archiving tool. It also provides insights into the potential for hacking videogames hardware (particularly motion-sensitive hardware such as the Kinect and Wiimote) for artistic purposes.

The application presents children with an attractive and intuitive interface, which they can use to record and replay the sound and moves of their game. As the child performs their clapping game, the computer tracks their hand movements and displays them as two animated hands on screen (which mimic their real hand movements realistically). Both their moves and their song/rhyme are recorded. Once they have finished recording, they can replay their own recorded movements – or those of any previous player – which are shown as two additional on-screen hands. As a result, the child can play with/against the recording, with the computer detecting when the player’s real hands “clap” with the virtual computer-controlled ones.

Every game is archived and researchers can examine and analyse this raw data. The application provides various ways doing this – ranging from simple playback to complex visualisations which show the path taken by the hands throughout the entire game.

T his 3D motion tracking system provides significant advantages over conventional archiving techniques, such as video recordings, which only allows the action to be viewed from a single angle (possibly omitting significant details). There is also the potential for the movement data to be analysed by computer (e.g. to identify similar patterns or rhythms).

  • Grethe Mitchell is Reader in Digital and New Media at the University of Lincoln, UK, and is currently Principal Investigator for an AHRC project looking at the recording, analysis and preservation of movement in the Arts and Humanities. Grethe was Co-Investigator on a major AHRC/Beyond Text investigation into children’s playground games in the new media age, where she supervised the development of an innovative computer game and application for movement preservation/analysis. She is also co-editor of “Videogames and Art” the first academic book investigating the use of videogames in artistic practice. Grethe has a background in film and TV production/post-production and her research interests include: computer games and videogames; children’s games and play in the playground and on computer; intermediality; the crossovers and connections between film and video/computer games; videogame and digital art; interactivity and interaction design.
  • Andrew Clarke is a writer, researcher and consultant on internet and new media. He has also written and presented on videogames and on videogame art. He is the co-editor (with Grethe Mitchell) of “Videogames and Art” – the first academic book on the artistic appropriation of videogame technology and iconography. This is a topic which he remains deeply interested in. He was Technical Advisor to the AHRC-funded project “Children’s Playground Games and Songs in the New Media Age” and programmed the final version of the “Game Catcher”, a low-cost motion tracking system using Kinect and Wii hardware. This is the topic of his presentation at ISEA2011.

Full text (PDF) p. 1738-1743