Design researchers have used think-aloud, or protocol analysis, studies of designers in action to better understand their design processes [2, 3]. In these think-aloud studies, designers are given a problem, and then they are observed, audio-taped, and often videotaped as they work the problem. Later, the design researcher transcribes the taped record of the design session, and his or her notes, and constructs a time stamped record of the events in the session. The record typically consists of the drawings that the designer makes and his or her spoken comments; in addition it may indicate gestures such as pointing to portions of the drawing or body movement. This transcription is typically the main research material used for further analysis.

Valuable as think-aloud studies are in design research, they require a great deal of effort on the part of the researcher. One of the chief difficulties of conducting is transcribing the design session after it has been recorded on tape. Relevant events in a design session often happen every few seconds. Depending on the level of detail that the design researcher is interested in, a one-hour design session can demand tens of hours of transcription time (carefully watching a video tape frame-by-frame to locate start and end times for events, and transcribing the spoken words). The session may result in a transcript that is tens of pages long.

The Digital Design Recorder project aims to make this transcription easier, by helping the researcher capture the main spoken and drawn events of a design session, and constructing a machine-searchable transcription that serves as a pointer into the source data captured during the original design session. We have built a first, working, version of this system that captures the designer's drawings using a digitizing tablet and pen as well as the spoken think-aloud protocol. The captured audio is run through an off-the-shelf speech recognizer to generate a first pass at a text transcription of the design session. The three components: drawing, audio, and text transcript, are arrayed in a multimedia document. The design researcher can then review and annotate this document to construct a transcript of the design session. As current speech-recognition software is not entirely reliable, it is important to allow the design researcher to repair the machine-made text transcript; nevertheless, starting with an initial transcription is a significant improvement over starting from scratch. The speech recognition software associates the text transcript with the recorded audio so these repairs are somewhat easier to make than working with a simple audio or video recording. Related efforts in Human-Computer Interaction (HCI) have recently explored multi-modal interfaces [1.5]; however they have not focused on design. Complementary efforts in design research (e.g., [4]) have focused on micro-analysis of stroke data, but not on creating transcript documents linking graphics and text.

The paper reports on the design and implementation of the Digital Design Recorder, including a first round of experience using the Recorder to analyse designers in action. We review some of the specific challenges in constructing this system and how we addressed them, as well as our plans for enhancing this system in the future, to include indexed digital video as part of the record. We discuss other applications for the recorder, including its use as a note-taker for distributed design meetings that take place over the Internet.

Back of an Envelope Design
Design Amanuensis