Multimodal Pedagogical Planning for 3D Learning Environments - A Unified Framework

Abstract

Pedagogical planning lies at the heart of knowledge-based learning environments. In recent years, multimodality and authoring have become key issues in the creating of learning environments. The purpose of this research has been to design, implement, and evaluate a multimodal pedagogical planning system and a multimodal pedagogical authoring system.

First, we designed and implemented a multimodal pedagogical planning system for 3D learning environments. In 3D learning environments, the student can cooperate with an animated agent to solve a problem or study a new concept. The agent uses natural language narration accompanied by synchronized gestures to communicate with the student. 3D animation and camera motions are also used to offer the student a vivid and friendly learning environment. In order to control the agent's utterances, the agent's gestures and 3D multimedia presentation in the 3D learning environment, we designed a multimodal pedagogical planning system that uses a coordinated distributed planning structure. The multimodal pedagogical planning system contains a Pedagogical Planner which is a global planner, and three coordinated distributed planners: an Agent Utterance Planner, an Agent Gesture Planner, and a Cameral and Animation Planner. The planners synchronize the agent's verbal and physical behaviors on phrase boundaries. We implemented the multimodal pedagogical planning system in the PhysViz 3D learning environment for the circuit experiments.

Second, we designed and implemented a multimodal pedagogical authoring system based on the multimodal pedagogical planning architecture. One of the most difficult problems faced by ITS research is authoring. Because of their complexity, ITSs are notoriously difficult to author. However, because of the modularity of the multimodal pedagogical planning architecture we developed, we hypothesized that our model of multimodal communication designed above can serve as the basis for an ITS authoring environment. The multimodal pedagogical authoring system can be used to author the pedagogical plans, including plans that describe the agent's utterance, agent's gesture, camera motions and animations. We empirically evaluated the authoring architecture in conjunction with the PhysViz 3D learning environment by studying non-technical subjects' ability to modify learning activities for PhysViz. In the informal evaluation, 10 subjects were invited to use our multimodal pedagogical authoring system and answer a questionnaire. The results of this questionnaire suggest that most subjects without advanced knowledge of Computer Science can build 3D learning environments with the help of our authoring system.