Crowdsourcing Performance Evaluations of User Interfaces
Can computer users be trusted to paricipate in user interface studies from the comfort of their home?
Can user interface researchers give up control over their subjects' environment? In this project we study whether we can use Amazon Mechanical Turk to conduct user interface studies reliably.
To do so, we replicated three previously known performance experiments, the "Bubble Cursor," the "Split Menus," and the "Split Interface," both in our lab and on Mechanical Turk.
We compared the lab with the online population in terms of performance metrics such as speed, accuracy, and consistency.
The results, which we share in our upcoming CHI paper, show that the Mechanical Turk participants perform just as well as the lab participants.
[Related paper]
Predicting Users' First Impressions of Website Aesthetics
Users make lasting judgments about a website's appeal within a split second of seeing it for the first time. This first impression is influential enough to later affect their opinion of a site's usability and trustworthiness. In this project, we aim to automatically adapt website aesthetics to users' various preferences in order to improve this first impression. As a first step, we are working on predicting what people find appealing, and how this is influenced by their demographic backgrounds. Although it is not yet known what exactly influences this first impression of appeal, colorfulness and visual complexity have been repeatedly found to be the most noticeable design characteristics at first sight. We have therefore developed perceptual models of perceived visual complexity and colorfulness, which we then used to predict users' perception of appeal. Our approach is based on the assumption that this first impression can be adequately captured with the help of a low-level image analysis of static website screenshots. In our upcoming CHI paper, we show that these models can account for approximately half of the variance in the observed ratings of aesthetic appeal. With that, we demonstrated that it is possible to quantify users' initial impression of appeal based on the models of perceived visual complexity and colorfulness. Our results pave the way for larger endeavors to improve the user experience on the web, because the first impression counts.
[Related paper]
SPRWeb: Preserving Subjective Responses to Website Colour Schemes through Automatic Recolouring
Colors are an important part of user experiences on the Web. Color schemes influence the aesthetics, first impressions and long-term engagement with websites. However, five percent of people perceive a subset of all colors because they have color vision deficiency (CVD), resulting in an unequal and less-rich user experience on the Web. Traditionally, people with CVD have been supported by recoloring tools that improve color differentiability, but do not consider the subjective properties of color schemes while recoloring. To address this, we developed SPRWeb, a tool that recolors websites to preserve subjective responses and improve color differentiability, thus enabling users with CVD to have similar online experiences. SPRWeb is the first tool to automatically preserve the subjective and perceptual properties of website color schemes thereby equalizing the color-based web experience for people with CVD.
[Related paper]
Cultural Differences in Time Perception and Group Decision-Making
When discussing the effect of technology on culture, people often assume that the world is slowly homogenizing into a culture of Internet users, who share similar values and behavioral norms. Our analysis of the online scheduling behavior on Doodle argues against this hypothesis. In fact, event scheduling is not simply a matter of finding a mutually agreeable time, but a process that is shaped by social norms and values. And this can highly vary between countries. To investigate the influence of national culture on people's scheduling behavior we analyzed more than 1.5 million Doodle date/time polls from 211 countries. Our findings include that people around the world steer their availabilities towards those options that have good chances to reach consensus. But people from more group-oriented collectivist countries (e.g., India, China) seem to make a larger effort to reach mutual agreement than individualists (e.g., the US). We believe that increasing the awareness of such differences can help improve intercultural scheduling and support the acceptance of cultural differences as an interesting contribution to our lives.
[Related paper]
[Data]
Lab in the Wild
Most of what we know about human-computer interaction today is based on studies conducted with Western participants, usually with American undergrads. This is despite many findings that our cultural background affects our perception and preferences. Neuroscience research has even shown that cultural exposure leads to differences in neural activity -- a finding that might affect how we interact with computers. If people around the world perceive, process, and interact with information differently, then what should their user interfaces look like in order to be most intuitive for them to use?
With Lab in the Wild we are trying to shed light on this question. Our goal is to improve the user experience and performance for computer users around the world. But Lab in the Wild doesn't just help us answer our questions. It also provides participants with personalized feedback, which lets them compare themselves and their performance to people of other countries. Try it out :)
Accurate Measurements of Pointing Performance from In Situ Observations
We present a method for obtaining lab-quality measurements of pointing performance from unobtrusive observations of natural in situ interactions. Specifically, we have developed a set of user-independent classifiers for discriminating between deliberate, targeted mouse pointer movements and those movements that were affected by any extraneous factors. Our results show that, on four distinct metrics, the data collected in-situ and filtered with our classifiers closely matches the results obtained from the formal experiment.
[Related
paper]
[Source Code and Data]
Ability-Based User Interfaces
Most of today's GUIs are designed for the typical, able-bodied user.
People with unusual abilities (due to a disability, a temporary injury, or who are just trying to operate a small device with cold fingers) have to adapt themselves to the user interfaces, perhaps using assistive technologies.
We are working to reverse this situation: we believe that user interfaces should be adapted to the invidual abilities, devices, and preferences of the people who use them.
Several invidual projects in our group contribute to this vision. Our work on the SUPPLE system, for example, demonstrated that we can automatically generate user interfaces adapted to a person's individual motor and vision abilities. The results of our studies showed that people with motor impairments were significantly faster and strongly preferred such automatically generated ability-based interfaces to the defaults provided by the software manufacturers. learn more >>
PETALS Project -- A Visual Decision Support Tool For Landmine Detection
[Related paper]
Incorporating Rich User Feedback Into Interactive Machine Learning Applications
Successful interactive machine learning systems need to generalize robustly from a very small number of examples. This poses challenges for most machine learning algorithms, which typically only solicit labels from the users while ignoring any additional rationale users might be willing to provide to explain their choices. Several projects have shown that incorporating richer feedback---that captures some of the user's rationale---leads to faster and more generalizable learning. So far, this feedback has been limited to feature relevance. Is this the best or the only type of rich feedback we can elicit from users?
The results of our preliminary study show that people naturally provide several other types of feedback to explain their decisions and that those other types of feedback have an even stronger positive impact on the predictive accuracy of machine learning algorithms than feature relevance. In this project, we study what types of explanations people can most easily provide, how to incorporate this additional information into machine learning algorithms, and how to design novel recognition-driven interactions that will help users provide such explanations with the minimum amount of additional cognitive overhead. The results of this project will impact both the algorithms and the interaction design for interactive machine learning systems.
Exploring The Design Space Of Adaptive User Interfaces
For decades, researchers have presented different adaptive user
interfaces and discussed the pros and cons of adaptation on task
performance and satisfaction. Little research, however, has been
directed at isolating and understanding those aspects of adaptive
interfaces which make some of them successful and others not. We have
conducted several laboratory studies to systematically isolate some of
the design and contextual factors that affect the impact of adaptation
on users' performance and satisfaction. The results of these studies combined with the recent work of others, provide an initial characterization of the design space of adaptive graphical user interfaces.
Our current work in this space is aimed at long-term in situ evaluations of adaptive interfaces.
[Related
papers]
PlateMate allows users to
take photos of their meals and receive estimates of food intake and
composition. Accurate awareness of this information is considered a
prerequisite to successful change of eating habits, but current
methods for food logging via self-reporting, expert observation, or
algorithmic analysis are time-consuming, expensive, or inaccurate.
PlateMate crowdsources nutritional analysis from photographs using
Amazon Mechanical Turk, automatically coordinating untrained workers
to estimate a meal's calories, fat, carbohydrates, and protein. To
make PlateMate possible, we developed the Management framework for
crowdsourcing complex tasks, which supports PlateMate's decomposition
of the nutrition analysis workflow. Two evaluations show that the
PlateMate system is nearly as accurate as a trained dietitian and
easier to use for most users than traditional self-reporting, while
remaining robust for general use across a wide variety of meal types.
Heart disease is the number one killer in the United States, and finding indicators of the disease at an early stage is critical for treatment and prevention. In this paper we evaluate visualization techniques that enable the diagnosis of coronary artery disease. A key physical quantity of medical interest is endothelial shear stress (ESS). Low ESS has been associated with sites of lesion formation and rapid progression of disease in the coronary arteries. Having effective visualizations of a patient's ESS data is vital for the quick and thorough non-invasive evaluation by a cardiologist. We present a task taxonomy for hemodynamics based on a formative user study with domain experts. Based on the results of this study we developed HemoVis, an interactive visualization application for heart disease diagnosis that uses a novel 2D tree diagram representation of coronary artery trees. We present the results of a formal quantitative user study with domain experts that evaluates the effect of 2D versus 3D artery representations and of color maps on identifying regions of low ESS. We show statistically significant results demonstrating that our 2D visualizations are more accurate and efficient than 3D representations, and that a perceptually appropriate color map leads to fewer diagnostic mistakes than a rainbow color map.
Prior work has highlighted the challenges faced by people with
motor impairments when trying to acquire on-screen targets using
a mouse or trackball. Two reasons for this are the difficulty of
positioning the mouse cursor within a confined area, and the
challenge of accurately executing a click. We hypothesize that
both of these difficulties with area pointing may be alleviated in a
different target acquisition paradigm called "goal crossing." In
goal crossing, users do not acquire a confined area, but instead
pass over a target line. Although goal crossing has been studied
for able-bodied users, its suitability for people with motor
impairments is unknown.
In our study, participants with motor impairments were faster with and preferred goal-crossing to area pointing.
This work provides the empirical
foundation from which to pursue the design of crossing-based
interfaces as accessible alternatives to pointing-based interfaces.
Recent years have revealed a trend towards
increasing use of optimization as a method for automatically designing
aspects of an interface's interaction with the user. In most cases,
this optimization may be thought of as decision-theoretic --
the objective is to minimize the expected cost of a user's
interactions or (equivalently) to maximize the user's expected
utility. While decision-theoretic optimization provides a powerful,
flexible, and principled approach for these systems, the quality of
the resulting solution is completely dependent on the accuracy of the
underlying utility or cost function. Unfortunately, determining the
correct utility function is a complex, time-consuming, and error-prone
task. While domain-specific learning techniques have been used
occasionally, most practitioners parameterize the utility function and
then engage in a laborious and unreliable process of hand-tuning.
SUPPLE is an application- and device-independent system that automatically
generates user interfaces for a wide variety of display devices.
SUPPLE uses decision-theoretic optimization to render an interface
from an abstract functional specification and an interchangeable
device model. SUPPLE can use information from the user
model to automatically adapt user interfaces to different tasks
and work styles while also prividing extensive customization
mechanisms that allow for modifications to the appearance,
organization and navigational structure of the user interface.
Child mortality is one of the most pressing health concerns almost 10
million children die worldwide each year before reaching their fifth
birthday, mostly in low-income countries. To aid overburdened and
undertrained health workers the World Health Organization (WHO) and
United Nations Children's Fund (UNICEF) have developed clinical
guidelines, such as the Integrated Management of Childhood Illness
(IMCI) to help with the classification and treatment of common
childhood illness. To help with deployment, we have developed an
electronic version (e-IMCI) that runs on a PDA. From July to September
2007, we ran a pilot of e-IMCI in southern Tanzania. The system guides
health workers step-by-step through the treatment algorithms and
automatically calculates drug doses. Our results suggest that
electronic implementations of protocols such as IMCI can reduce
training time and improve adherence to the protocol. They also
highlight several important challenges including varying levels of
education, language and expertise, which could be most adequately
addressed by implementing novel intelligent user interfaces and
systems.
Opportunity Knocks (OK) is an automated transportation routing system,
whose goal is to improve the efficiency, safety and independence of
individuals with mild cognitive disabilities. OK is
implemented on a combination of a Bluetooth sensor beacon that
broadcasts GPS data, a GPRS-enabled cell-phone, and remote activity
inference software. The system uses a novel inference engine that does
not require users to explicitly provide information about the start or
ending points of their journeys; instead this information is learned
from users' past behavior.
Alfred is an electronic butler for Intelligent Environments.
Alfred allows an end user to "program" the system by telling it the name of a new
goal, demonstrating one or more plans for achieving that goal, and finally telling
the system the conditions under which it would prefer one plan to another.
Similarly, the user can name events that arise in the environment and tell the
system what goals should be posted when those events arise. Each of these steps
can be done by simple verbal commands or other natural forms of interaction.
End users, in effect, record "macros" which, are executed adaptively and reactively.
"Look-to-talk" is a gaze-aware interface for directing a spoken
utterance to a software agent in a multiuser collaborative
environment. Through a prototype and a Wizard-of-Oz (WOz) experiment,
we showed that "look-totalk" is indeed a natural alternative to speech
and other paradigms.
FIRE is a multimodal interface for
information retrieval deployed in the Intelligent Room at the MIT AI
Lab. FIRE extracts all the category terms related to the search query
and uses entropy to generate questions that would quickly allow the
user to disambiguate her query and arrive at a small set of relevant
documents. FIRE presents information over several large displays in
the Intelligent Room and supports both speech and gesture input for
more natural interaction.
Rascal is a high-level resource management
system for the Intelligent Room Project, that addresses the problem of
the numerous applications competing for limited physical resources.
Rascal performs the service mapping and and uses constrained search
for arbitration among different requesters.