"Modern Little Red Riding Hood" is our first experiment in Video-Based Interactive Storytelling using video compositing techniques.
The generation of engaging visual representations for interactive storytelling represents a key challenge for the evolution and popularization of interactive narratives. Usually, Interactive storytelling systems adopt computer graphics to represent virtual story worlds, which facilitates the dynamic generation of visual content. However, the quality of the images and motion produced by these systems is still inferior compared to the high quality experience found in live-action films. Interactive rates in photorealistic rendering for the film industry will not be possible for decades to come. A promising alternative is the replacement of 3D virtual characters with video sequences with real actors. In this work, we propose a new method for video-based interactive narratives that uses video compositing algorithms that run at truly interactive frame rates. The proposed method is consistent with plots that are generated by nondeterministic planning algorithms. Moreover, we propose a system of artificial intelligent agents that perform the same roles played by filmmaking professionals. We believe that future improvements of the techniques proposed in this work represent an important contribution to the quest for new and more immersive forms of interactive cinema.
(Watch this video in full screen to have a better view of the interactions on the smartphone)
The Princess Kidnapping
"The Princess Kidnapping" is our first experiment in Video-Based Interactive Storytelling. The project is a experiment designed specifically for Interactive Broadcast Television. It gives the viewers power to participate more actively and interact in the story. The most important characteristic of this interactive narrative is that it was made so that every viewer can choose the path they want to follow during the story, and still finish watching it at the same time (respecting the principal of air time on broadcast television). With this Video-Based Interactive Narrative Mr. Edirlei Soares de Lima won an "Honorable Mention - Innovation" from ITU (International Telecommunication Union) in the "ITU IPTV Application Challenge" competition.
Lima, E.S., Feijo, B., Furtado, A.L., Pozzer, C.T., Ciarlini, A.E.M., 2012. Automatic Video Editing For Video-Based Interactive Storytelling. In: Proceedings of the 2012 IEEE International Conference on Multimedia and Expo (ICME 2012), Melbourne, Australia, July 2012. pp. 806-811. ISBN: 978-0-7685-4711-4..
Paper and Pencil Interactive Storytelling
The Paper and Pencil Interactive Storytelling is a system able to dramatize interactive narratives in augmented reality over a conventional sheet of paper. The system allows users to freely interact with virtual characters by sketching objects on the paper. Users interacting with the system can indirectly affect the characters' decisions, even to the point of radically subverting the storyline.
Lima, E.S., Feijo, B., Barbosa, S., Furtado, A.L., Ciarlini, A.E.M., Pozzer, C.T., 2011. Draw Your Own Story: Paper and Pencil Interactive Storytelling. In: 10th International Conference on Entertainment Computing (ICEC 2011), Vancouver, Canada. pp. 1-12. ISSN: 978-3-642-24499-5.
Multimodal, Multi-User and Adaptive Interaction
The ability that users have to interact and change stories according to their own desires is what differentiates interactive narratives from conventional films. Moreover, this ability expands the boundaries of computer games towards new forms of digital entertainment. However, designing an interaction model for an interactive storytelling system involves several challenges, from the need for natural interaction interfaces to adequate multi-user settings.
Lima, E.S., Feijo, B., Barbosa, S., Silva, F.G., Furtado, A.L., Pozzer, C.T., Ciarlini, A.E.M., 2011. Multimodal, Multi-User and Adaptive Interaction for Interactive Storytelling Applications. In: X Brazilian Symposium on Games and Digital Entertainment - SBGames 2011, Salvador, Brazil.
Logtell Dramatization System
There are several questions related to the manner how the scenes of an interative narrative should be shown to the public. Unlike films, where every scene is carefully planned before being shot, in a interactive narrative this freedom does not exist; usually the storyline of the narrative results from the real-time interaction between the viewers and the virtual actors. The dramatization of stories, whose sequence of events is unknown, depends on a robust dramatization system.
Lima, E.S., 2010. Um Modelo de Dramatização Baseado em Agentes Cinematográficos Autônomos para Storytelling Interativo. Dissertação de Mestrado, Universidade Federal de Santa Maria (UFSM), Santa Maria, Brasil.
Director of Photography and Music Director
The way emotions are expressed in a film has great impact on the viewer's understanding of the narrative. Over the years, filmmakers developed several techniques to enhance the perception of these emotions, such as the photography and the audio editing of the scenes. Our interactive storytelling system incorporates two cinematography-inspired autonomous agents designed to better express the emotional aspects of interactive storytelling environments. Both the director of photography and music director use support vector machines trained with a cinematography-knowledge dataset to create and manipulate the audio and visual parameters of a runtime dramatization engine, thus increasing the immersion of the viewers in the story.
Lima, E.S., Pozzer, C.T., D'Ornellas, M., Ciarlini, A.E.M., Feijo, B., Furtado, A.L., 2009. Support Vector Machines for Cinematography Real-Time Camera Control in Storytelling Environments. In: VIII Brazilian Symposium on Games and Digital Entertainment, Rio de Janeiro, Brazil, pp. 44-51. [DOI: http://doi.ieeecomputersociety.org/10.1109/SBGAMES.2009.14]
Virtual Cinematography Director
The role of the intelligent cinematography director in our interactive storytelling system is to select in real-time the camera shots that best fit for the scenes and present the content in an interesting and coherent manner. Director's knowledge is represented with a collection of support vector machines (SVM) trained to solve cinematography problems of shot selection.
Lima, E.S., Pozzer, C.T., D'Ornellas, M., Ciarlini, A.E.M., Feijo, B., Furtado, A.L., 2009. Virtual Cinematography Director for Interactive Storytelling. In: Proceedings of the International Conference on Advances in Computer Enterntainment Technology (ACE 2009), Athens, Greece, pp. 263-270. [DOI: http://doi.acm.org/10.1145/1690388.1690432]
