[ISEA2019] Paper: Lingdong Huang, Zheng Jiang, Syuan-Cheng Sun, Tong Bai, Eunsu Kang & Barnabas Poczos — Legend of Wrong Mountain: AI Generated Opera

Abstract

Keywords: Opera, Kunqu, Audiovisual, Machine Learning, Artificial Intelligence, AI Art, Total Art, Gesamtkunstwerk, LSTM, pix2pix, RNN, Markov Chain, OpenPose

As one of the oldest forms of Chinese Opera since the 16th century, Kunqu features literary virtuosity in its scripts, sophisticated vocal techniques in its singing, emotional and elegant yet rigorous bodily motions and facial expressions in its performance. Over its history, Kunqu has developed established modes and patterns, which makes it especially suitable for neural networks to learn. In order to generate Kunqu literature and performance computationally, we applied multiple machine learning and computer vision techniques. Our result, Legend of Wrong Mountain, is novel twofold: it is the first and only generated opera at the time of this writing, and it is a machine’s attempt at Gesamtkunstwerk [12], “the Total Artwork”. It explores the marriage between contemporary technologies and traditional art form. By studying historical scripts, musical notations and traditional methods for creating Kunqu, we tweaked existing algorithms and devised new ones to conform to the traditional rules and norms as closely as possible. We presented this project as a video accompanied by audio.

  • Lingdong Huang is an artist and creative coder pursuing his undergraduate degree in Computer Science and Art at Carnegie Mellon University, USA.
  • Zheng Jiang is a musician and computer scientist pursuing his master’s degree in Music and Technology at Carnegie Mellon University, USA.
  • Syuan-Cheng Sun is a theater artist and interaction designer pursuing his master degree in Video and Media Design at Carnegie Mellon University, USA.
  • Tong Bai is a computer vision engineer who recently graduated from Carnegie Mellon University, USA, and started her career at cognition group in Microsoft.
  • Dr. Eunsu Kang is a Korean media artist who creates interactive audiovisual installations and AI artworks. Her current research is focused on creative AI and artistic expressions generated by Machine Learning algorithms. Creating interdisciplinary projects, her signature has been seamless integration of art disciplines and innovative techniques. Her work has been invited to numerous places around the world including Korea, Japan, China, Switzerland, Sweden, France, Germany, and the US. All ten of her solo shows, consisting of individual or collaborative projects, were invited or awarded. She has won the Korean National Grant for Arts three times. Her researches have been presented at prestigious conferences including ACM, ICMC, ISEA, andNeurIPS. Kang earned her Ph.D. in Digital Arts and Experimental Media from DXARTS at the University of Washington. She received an MA in Media Arts and Technology from UCSB and an MFA from the Ewha Womans University. She had been a tenured art professor at the University of Akron for nine years and is currently a Visiting Professor with emphasis on Art and Machine Learning at the School of Computer Science, Carnegie Mellon University.
  • Dr. Barnabás Póczos is an associate professor in the Machine Learning Department at the School of Computer Science, Carnegie Mellon University. His research interests lie in the theoretical questions of statistics and their applications to machine learning. Currently he is developing machine learning methods for advancing automated discovery and efficient data processing in applied sciences including health sciences, neuroscience, bioinformatics, cosmology, agriculture, robotics, civil engineering, and material sciences. His results have been published in top machine learning journals and conference proceedings, and he is the co-author of 100+ peer reviewed papers. He has been a PI or co-Investigator on 15+ federal and non-federal grants. Dr. Poczos is a member of the Auton Lab in the School of Computer Science. He is a recipient of the Yahoo! ACE award. In 2001 he earned his M.Sc. in applied mathematics at Eotvos Lorand University in Budapest, Hungary. In 2007 he obtained his Ph.D. in computer science from the same university. From 2007-2010 he was a postdoctoral fellow in the RLAI group at University of Alberta, then he moved to Pittsburgh where he was a postdoctoral fellow in the Auton Lab at Carnegie Mellon from 2010-2012.

Full text (PDF) p. 255-261