Research has shown that incorporating eye tracking in usability research can provide certain benefits compared with traditional usability testing. There are various methodologies available when conducting research using eye trackers. This paper presents the results of a study aimed to compare the outcomes from four different retrospective think aloud (RTA) methods in a web usability study: an un-cued RTA, a video cued RTA, a gaze plot cued RTA, and a gaze video cued RTA. Results indicate that using any kind of cue produces more words, comments and allows participants to identify more usability issues compared with not using any cues at all. The findings also suggest that using a gaze plot or gaze video cue stimulates participants to produce the highest number of words and comments, and mention more usability problems.
This paper reports on a preliminary study testing the use of eye tracking as a method for evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. 10 native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded noninvasively using a Tobii 1750 eye tracker. They were also asked to record retrospective protocols while watching a replay of their eye gaze reading data. The average gaze time and fixation count were found to be significantly higher for the "bad" sentences, while average fixation duration was not significantly different. Evaluative comments uttered during the retrospective protocols were also found to agree to a satisfactory degree with previous human evaluation. Overall, we found that the eye tracking method correlates reasonably well with human evaluation of MT output.
An experimental validation is presented of a novel method for usability testing that entails playback of dynamic eye-tracking data to cue the elicitation of retrospective verbal reports. Participants produced: (1) think-aloud reports during an online search task, and (2) retrospective reports during another search task cued by the playback of either the screen capture of events or their own eye-movements. Task-completion times and response rates were recorded to assess reactivity. Fewer participants completed the search task whilst thinking aloud, indicating the reactivity of this technique. Verbal transcripts were coded for instances of usability problems. The eye-cued method identified more usability problems than the think-aloud or screen-cued methods. A significant interaction between search engine type and retrospective cue-type suggests that the value of the eye-cue method may be greatest with more complex browsing environments. Our results demonstrate that when cued appropriately, retrospective reports may be less reactive and more informative than other verbalisation techniques.
Initial research to investigate users' emotional reactions to websites is presented. An Emotion Words Priming List (EWPL) was developed for UK English speakers and used to prompt users in an evaluation of 6 websites. Only half the words on the EWPL V1 were amongst the most frequently used emotion words in the retrospective verbal protocols. However a list of 16 emotion words emerged from this study that constitute version 2 of the EWPL, to be validated in a future study.
Click in to find related links.