User Interface Softbots

Abstract

A user interface softbot is a software agent that controls an interactive system through its graphical user interface, relying on visual information of the system rather than an application programming interfaces or access to source code. We have developed a prototype system that facilitates the use and extension of interface softbots, reforming the programming process and making it easier for new developers. We present the JSegMan substrate that facilitates the representation and identification of the conventional graphical user interface. The JSegMan substrate consists of sensors, effectors and framework for an agent that can control the user interface environment. Using image processing the sensor builds a representation of the desktop's visual interface. The effector module generates mouse and keyboard gestures to control the desktop environment. The sensor and effector modules work as eyes and hands of a controller application that can be tailored for a given application domain. We also present three applications - MSN Chatbots, the classical Blocks World problem solver and a GUI testing tool that use the JSegMan substrate and demonstrates applicability of JSegMan in use with off-the shelf applications, PDDL based planners and in testing environment respectively.