The Q-matrix Method of Fault-Tolerant Teaching in Knowledge Assessment and Data Mining

Abstract

A major challenge for today's schools is to create individualized instruction in inexpensive, expandable ways. To achieve this goal, educational systems must adapt to each student, mining student data to determine what a student knows and does not know. Using this knowledge, teaching systems can guide students in the learning process.

This dissertation investigates the q-matrix method of fault tolerant teaching (FTT). FTT systems are adaptive teaching systems that tolerate student, teacher, and system errors in diagnosing student misconceptions. The q-matrix method is easily applied to any tutorial system data, independent of topic, and can be used to model student behavior and guide student knowledge remediation. We found that, in addition, the model is easily interpretable and can be used to understand large sets of student data, pinpointing problem areas for student learning.

In this work, we applied the q-matrix method in three computer tutorials, presenting the first experiment to use the method on a large group of students. Overall, students felt that these tutorials were beneficial. For each tutorial, the q-matrix method was used to create a student knowledge model. When compared with other data mining/knowledge discovery methods, including factor analysis and cluster analysis, the q-matrix method was superior in that it was able to fit the observed data well, while still offering the interpretability needed to devise remediation methods. Our results indicate that the q-matrix model may predict student misconceptions at least as well as students were able to predict themselves, and students generally felt that the tutorial knew which concepts each student least understood.

For our logic proofs tutorial, we devised q-matrices as data mining tools, used to extract the axioms needed to solve a proof. In this analysis, we were able to isolate sets of students using similar strategies, and to interpret the strategies of these groups using the q-matrix model.

We also compared extracted q-matrix models to expert models, and found that the extracted and expert q-matrices were not a good match, but that extracted q-matrix models were quite useful in understanding student data. This shows that expert models do not necessarily predict student behavior and more accurate student knowledge models, such as q-matrices, are needed to understand student knowledge. An extracted q-matrix can reveal student behavior that might not be predicted by an expert's understanding.

This work resulted in the construction of a fully automated, fault tolerant, intelligent tutoring system, which can diagnose and correct student misconceptions. This system also provides an interpretable model for each topic that relates each tutorial question to its underlying concepts. The experimental analysis provides valuable insight into the factors that influence the extraction and interpretability of these models, as well as their value in automatically assessing student knowledge. In addition, the q-matrix method is used as a general data mining tool in one tutorial where a traditional application of the q-matrix method would not be appropriate. This application and its favorable comparison with other data mining tools mark the q-matrix method as a viable data clustering and interpretation tool for data mining and knowledge discovery.