Abstract

Full Paper. Humans and NonHumans / Nonhuman Creativity, and Agency

Keywords: Artificial Intelligence Creativity, AI Automated Storytelling, Human-AI Interaction, Computer Vision, AI imagination

AI.R Taletorium is a collaborative storytelling system that consists of a neural story generator and a doodler-based fairytale visualizer. We design a character-centric bidirectional connection between the story generator and the visualizer, allowing kids to creatively influence the story generation by drawing objects they want to add to the storyline.

While stories provide an effective and concise way to share experiences, visual content acts as a more comprehensive and universal communication tool that transcends barriers between user groups with different cultural backgrounds and physical preferences. However, the meaningful learning process for coherent story visualization is a challenging task in AI studies. This paper proposes AI.R Taletorium, an intelligent artistic fairytale visualizer that interacts with users in real-time. Furthermore, language and mental development are key aspects of early childhood education.

Our system is collaborative and enables common creative experiences between groups of children at remote locations. We aimed at offering an entertaining and imaginative platform for children to communicate with each other, improving their psychological health, especially under extreme circumstances such as the COVID- 19 pandemic. The system comprises two novel modules, a character-centric storyline generator and a visualizer in the form of doodlers. We connect the modules with characters and build a bidirectional link to allow kids to participate in the fairy tale creation process by simple adding their drawings.

Our key contribution is as follows:

1. – A novel multimodal fairytale co-creation interface based on interactive text-to-image transfer and vice-visa.
2. – The system presents a novel fusion method between a learning-based language model and rule-based graph update, allowing for more flexibility in story generation and visualization with limited data.
3. – Proposing a CLIP transformed story graph as an intermediate representation to transcend the barrier between digital storytelling content and insufficient illustration data (automatic AI fairy tales storytelling case).

We fuse the rule-based language model with a learning-based generation model into a unified visualization system to allow for more flexibility during generation. AI.R Taletorium can visualize complex fairy tale scenes with stylized characters and rare relationships with our fused framework. http://www.syntropic-counterpoints.com/taletorium.html

• Dr Predrag K. Nikolić is an Associate Professor at the University of The Bahamas, Faculty of Liberal and Fine Arts in Nassau. He is an interactive media designer, media artist and researcher who holds a PhD in Digital Media and an MBA. He focuses on singularity, posthumanism, AI juxtaposition, and emancipation in his creative practice. His project Syntropic Counterpoints (www.syntropic-counterpoints.com) had numerous exposures worldwide together with other artworks such as Botorikko, Robosophy Philosophy, MindCatcher, InnerBody, Ciklosol, Before & Beyond, Vrroom, Digital Lolipop, In_Visible Island. Nikolic exhibited and presented his work at Ars Electronica, SIGGRAPH, SIGGRAPH Asia, ACM SIGGRAPH Sparks, Technarte USA, Technarte Spain, Singapore Science Center, Hong Kong – Shenzhen Design Biennial, Maison Shanghai, National Museum of Applied Arts in Belgrade, National Museum of Education in Belgrade. Nikolić is Editor-In-Chief for EAI Endorsed Transactions on Creative Technologies Journal and European Alliance for Innovation (EAI) Fellow and founder of AI.R Lab.
• Ruiyang Liu is a PhD student at ShanghaiTech University (China) working on computer vision. She worked with Professor Predrag K. Nikolić on artificial intelligence for digital art and design topics. In the five years of PhD study, she has published on several CV top conferences including CVPR, ECCV etc. Besides research, she also worked for top internet companies including Microsoft and Google as software engineering.
• Giacomo Bertin is a Physic of Data student at the University of Padova (Padua, Italy) who holds a Bachelor’s degree in Physics with a thesis in computational methods for drug discovery. He is a machine learning engineer and artificial intelligence developer who is part of the Professor Nikolic’s Syntropic Counterpoints project and co-founder of AI.R Lab, focused on computer vision and natural language