Cinematic Discourse Generation

Abstract

Narrative is one of the fundamental ways in which humans organize information. 3D virtual environments provide a compelling new medium for creating and sharing nar- ratives. In pre-rendered virtual environments like animated movies, directors communicate complex narratives by carefully constructing them shot-by-shot. To do this, a film's director exploits the viewer's familiarity with narrative patterns and cinematic idioms to effectively convey a structured story. In real-time environments like games and training simulations, however, a system has much less control over the stories that need to be told, since often novel stories are constructed on demand and tailored to a specific session or user's needs. In many contexts, stories are built not solely by the system, but collaboratively with many users whose choices for action contribute to the construction of unanticipated narrative structure. In the past, intelligent cinematography systems have been developed to automat- ically record the actions of users within a virtual world and then to construct coherent visualizations that communicate these action sequences. While these systems generate co- herent visualizations, they do not attempt to address the careful construction of narrative discourse based on established and identifiable patterns of narrative communication. Cur- rent automated camera systems take into account local coherence of shots and transitions but do not address the rhetorical coherence of the communication across multiple shots. I describe an end-to-end camera planning system - Darshak - that constructs visual narrative discourse of a given story in a 3D virtual environment. Darshak uses a hierar- chical partial order causal link planning algorithm to generate narrative plans that contain both story and camera actions. Dramatic situation patterns commonly used by writers of fictional narratives and endorsed by narrative theorists are formalized as communicative plan operators that provide a basis for structuring the cinematic content of the story's vi- sualization. The dramatic patterns are realized through abstract communicative operators that represent operations on a viewer's beliefs about the story and its telling. Camera shots and transitions are defined in this plan-based framework as execution primitives. Repre- sentation of narrative discourse as a hierarchical plan structure enables us to utilize 1) the hierarchical nature of narrative patterns and film idioms through the hierarchy in decom- positional plan operators, and 2) explicit representation of causal motivation for selection of shots through causal links. I present an empirical evaluation of the algorithm, based on cognitive metrics, for three properties of cinematic discourse: Saliency, Coherence and Temporal Consistency.