Assessing interactive system effectiveness with usability design heuristics and Markov models of user behavior

Abstract

The purpose of this research was to create a new measure of usability to aid researchers in determining whether the intent of system designers is realized and to objectively assess user behavior as a basis for interface design recommendations. Contemporary human-computer interaction (HCI) research has focused on methods to promote usability in interactive systems. Unfortunately, there are few quantitative, objective measures of usability, among many subjective measures. A measure that can combine both objective and subjective data may be valuable in determining user perceptions of system designs and whether performance goals are achieved. The present research utilized an existing HCI framework developed by Abowd and Beale and a mathematical model of the average number of user actions at an interface (e.g., mouse clicks) required for task performance in order to determine an overall system usability effectiveness score. Part of the score involves usability ratings by users, while considering designer interests in interface development. The components of the Abowd and Beale framework include the user, input, the system and output, which are interconnected by links that affect system effectiveness. Some links represent computational load, while others represent the complexity of articulation and system state observation for the user; that is, cognitive resources expended in performance. Four designers were asked to rank the importance of each link in the framework, based on their design intentions for the two online ordering interfaces. Pair-wise comparisons were made of the various dimensions of system effectiveness and usability represented in the Abowd and Beale framework, and the rankings were averaged across designers. Designers were expected to consider user articulation and observation as the most important aspects of the interface design because they represent cognitive load. Twenty users, divided into two equal groups, were asked to complete the task of buying a certain type of computer (ThinkPad R60) using an existing and new prototype web interface. The new prototype was designed to increase usability with consolidated pages, more pronounced buttons and a multi-level menu structure. The users then considered the links in the Abowd and Beale framework and rated the specific design alternatives to provide a subjective assessment of whether the designer?s intent was achieved. It was expected that the new interface would receive higher average ratings than the old interface, based on the design changes. Because subjective analysis of an interface is typically insufficient for identifying all serious usability problems, and for justifying design changes from a management perspective, it was necessary to include an objective component in the system usability effectiveness score. In web applications, the average number of clicks can be used as a measure of system performance efficiency. Markov Chain models can be used to predict human motor behaviors at an interface, such as the average number of mouse clicks in a task, based on small samples of actual performance data. In the present study, on-line interface state transitions elected by users were recorded by a java script and used to establish probabilities for each system state. The probabilities were included in a Markov model to predict the average number of clicks in task performance. Once the average number of clicks was predicted, the subjective usability ratings were divided by the Markov model output to determine a ratio of ?usability per interface action? for the system (i.e., the overall effectiveness score). The interface alternative resulting in the higher score (ratio of usability per interface action) can be said to have higher usability and represent a better match to the intent of the designer. The actual number of user clicks at the two web interfaces was also recorded during the study. It was expected that the new interface would reduce the number of clicks necessary to optimally reach the computer purchase goal; therefore, increasing the overall system effectiveness score. It was also expected that using Markov models would accurately predict the average number of clicks. It was found that the overall effectiveness score was, in fact, higher for the new interface, suggesting the intent of the designers was achieved and the design revision provided better usability for ordering a computer. The Markov model was also proven to accurately predict the average number of clicks. The current research developed a usability evaluation method that incorporates both subjective and objective data, which may be more effective than prior measures focusing on subjective usability ratings only. The new measure may also reduce the need for user testing experimentation, in comparison to objective usability measurement approaches.