Archive

Posts Tagged ‘mediated evaluation’

The evolution of claims

January 18, 2012 1 comment

This post seeks to trace the evolution of the claim in human-computer interaction (HCI), from its introduction in the Carroll and Kellogg (1989) paper through the appearance of three books, Carroll’s Making Use (2000), Sutcliffe’s The Domain Theory (2002), and Rosson and Carroll’s Usability Engineering (2002). (A chronological list of key papers is provided at the end of this post.) The definition and role of “claims” shifted significantly during that time period; I’m seeking to identify some of the evolutionary shifts from 1989 to 2002. This list isn’t meant to be complete, but rather it seeks to highlight the most important evolutionary points in the conceptualization of the claim.

Three phases highlight the progress in this evolution:
– Carroll and his colleagues at IBM T.J. Watson in the late 1980s and early 1990s. They were seeking ways to design not just toward creating a single design, but toward crafting a theory-based approach to design to enable designers to build on each others’ work in a meaningful, scientific way. This work continued until Carroll left for Virginia Tech, at which time his focus largely shifted to collaborative computing (save for a few papers that seemed to draw on his IBM work).
– Sutcliffe and Carroll’s collaboration, highlighted by Sutcliffe’s sabbatical time at Virginia Tech. Sutcliffe had been working for many years on knowledge abstraction in software design, and, like Carroll and his group, he was inspired by potential roles for theory in HCI.
– Three summative works led by Carroll, Sutcliffe, and Rosson. Each presented a different view of the role of claims—in the fields of design, engineering, and education, respectively.

Claims were introduced to the field of HCI in Carroll and Kellogg’s “Artifact as theory nexus” paper at CHI 1989. They seemed to base their definition on Toulmin’s 1958 use of the term, in which he established claims as a hypothesis-centered approach to crafting arguments. The Carroll and Kellogg paper seeks to move beyond the narrow focus of cognitive-based theories that were prominent in the 1980s (that focused on low-level phenomena like keystrokes) by introducing the a hermeneutic approach based on psychological claims, the effects on people of both natural and designed artifacts. Claims were the central part of a task-analysis framework, an attempt to position the design and interpretation of HCI artifacts as a central component of HCI research. This approach was intended to bridge the gap from research to innovation—reconciling the “hermeneutics vs theory-based design” conflict in the title. Several examples in the paper showed how developing an understanding of a claim—the artifact and its possible effects—can point out how much we have to learn and can encourage us to draw broader conclusions. Many of these issues, in particular the connection of claims and claims analysis to the task-artifact cycle, is elaborated in a Carroll, Kellogg, Rosson 1991, but the ideas were first presented in the 1989 paper.

A 1992 BIT paper by Carroll, Singley, and Rosson provided the first in-depth view of the tech transfer of UE results (though see the Moran and Carroll 1991 special issue and 1996 book described below). It connected the Scriven view of mediated evaluation to claims upsides and downsides, positioning claims as a contributor in the field of design rationale. In so doing, it expounded upon claims as a way to reuse knowledge, by encouraging designer consideration of specialized vs abstract claims. The expectation was that designers could use claims to “avoid throwing away thoughtful empirical work”. They avoided Grudin’s paradox, stating outright that design rationale (including claims-centric design rationale) was not an automatic mechanism, but requires additional human thought to yield a reusable knowledge unit.

A 1992 TOIS article by Carroll and Rosson opined that HCI should be an action-science “that produces ‘knowledge-in-implementation’ and views design practice as inquiry”. The paper argues that the task-artifact cycle is an action-science because designers must respond to user requirements by building artifacts with upsides and downsides—i.e., claims. This paper distinguishes the scenario/claim roles as such: “Where scenarios are a narrative account, claims are a causal account.” It argues that scenarios provided a situation narrative, but they are too rich, hard to classify, and hard to reuse (arguments brought up again and addressed to varying degrees by Sutcliffe, Chewar, and others). It is the claim that establish the link to action-science by facilitating design analysis, providing a mechanism for generalization and hypothesis, and explicitly recognizing potential tradeoffs.

A 1994 IJHCS paper by Carroll, Mack, Robertson, and Rosson provided a software-centric scenario-based design approach, with Point-of-View (POV) scenarios drawing parallels to object-centric/object-oriented development. This paper represents the most process-based, engineering-focused, and software-generative view of scenario-based design—both until this time and thereafter. Although claims play a fairly minor role in this paper (only appearing in step 4, leveraging the upsides and downsides in analysis and hillclimbing), there seemed to be opportunity for a much larger role: identifying objects, specifying interactions between objects, supporting inheritance, etc. There was also initial discussion of an education focus for POV scenarios, SBD, claims, and such—but it was not elaborated, and the 2002 Rosson and Carroll textbook described a more simplified approach to teaching design. This paper seemed to be hypothesized starting points that were not fully pursued by the authors—rich for mining by Sutcliffe, Chewar, and others in the years to come.

Moran and Carroll’s 1996 Design Rationale book (elaborated from their 1991 special issue of the HCI Journal) is pointed to as a landmark in the field of design rationale. It draws together contributions from Jintae Lee, Allan MacLean, Clayton Lewis, Simon Buckingham Shum, Gary Olson, Gerhard Fischer, Colin Potts, Jeff Conklin, Jonathan Grudin, and many others. Of relevance to the topic of claims is the introduction (by co-editors Tom Moran and Jack Carroll) and a Carroll and Rosson chapter. These chapters exhibit connections in their work to Horst Rittel (wicked problems, IBIS), Francis Bacon (deliberated evaluation), Herb Simon (environment and behavior), and Donald Schön (contexts of experience)—putting forth the most synthesized view of the position of claims within the design community. Some of the psychological themes, particularly those of Simon, are elaborated in Carroll’s 1997 journal paper in Annual Reviews of Psychology.

A 1999 Sutcliffe and Carroll IJHCS paper summarizes the joint efforts of the two authors on the use of claims as a knowledge capture and reuse mechanism. It delved into the possibility of using claims as a reuse mechanism, a concept touched upon in previous work but never described in sufficient detail. The paper introduced a formatting and classification scheme for claims (and scenarios) to enable their reuse, including a process and alternate pathways for claim evolution. Among the augmentations was the first explicit connection to its derivation history and background theory (i.e., where it came from), leading to the first claim map that can reflect parentage, original/evolving context, motivation, evidence, and possibilities for reuse. Also of great importance was the acknowledgement of work left to do: methods for indexing, tool support (hypertext links, structure matching), and the need for buy-in (and stay-in) incentives.

Sutcliffe’s 2000 TOCHI paper seeks to address the irrelevance of HCI in industry, particularly with regard to a theory-based engineering approach. The paper seeks to identify ways to deliver HCI knowledge in a tractable form that is reusable across applications—and, more importantly, across application areas. The paper argues that claims could provide a bridge if reuse scope was improved; specifically, if there were generic versions of claims and artifacts, and if there were mechanisms for matching claims to new application contexts. The bulk of the paper provides a three-step process to accomplish this: steps for creating more generic claims, mechanisms for cross-domain reuse, and approaches to recognize broader implications. Parts of these are elaborated in Sutcliffe’s book (described later) and in the dissertations of Christa Chewar and Shahtab Wahid. Other important products of this work are the notion of claim families, a claims-patterns comparison, and an explicit recognition of the importance of claims as “designer-digestible” knowledge (one of my favorite phrases).

This series of papers culminated with three books that offered very different visions of design, with very different roles for claims. I plan to elaborate on these books in a future post, but here’s a brief summary of each. Carroll’s 2000 Making Use book pulled together his vision for scenario-based design for scientists, with an eye toward the discovery process. Claims are used to augment the scenario-based design process, highlighting key aspects of the design (and leaving the generalization of claims as an exercise for the designer). Sutcliffe’s 2002 The Domain Theory provides a reuse-centric view of software engineering, extending the vision of Rittel and the design rationale literature and approaches. The role of claims is to make concrete Domain Theory’s high level of abstraction (too high, according to critics) by leveraging the high utility (but low flexibility and poor reuse) of claims. Finally, Rosson and Carroll’s 2002 Usability Engineering textbook advocates scenario-based development as a teaching tool, with claims and claims analysis a complementary and guiding technique to scenario development during each stage of design. It presents claims in a simplified, stripped-down manner (for better and worse) meant to be highly accessible for students. These books kicked off a period of scientific application, engineering refinement, and creative design that has continued in the years since they appeared.

Chronological bibliography:
== S. E. Toulmin (1958). The Uses of Argument. Cambridge Press.
== J. M. Carroll and W. A. Kellogg (1989). “Artifact as theory-nexus: Hermeneutics meets theory-based design.” In Proceedings of CHI, pp. 7-14.
== J. M. Carroll, D. A. Singley, M. B. Rosson (1992). “Integrating theory development with design evaluation.” Behaviour and Information Technology 11, pp. 247-255.
== J. M. Carroll, M. B. Rosson (1992). “Getting around the task-artifact cycle: How to make claims and design by scenario.” ACM Transactions on Information Systems 10 2, pp. 181-212.
== J. M. Carroll, Mack, S. R. Robertson, M. B. Rosson (1994). “Binding objects to scenarios of use.” International Journal of Human Computer Systems 41, pp. 243-276.
== J. M. Carroll (1997). “Human-computer interaction: Psychology as a science of design.” Annual Reviews in Psychology 48, pp. 61-83.
== A. G. Sutcliffe and J. M. Carroll (1999). “Designing claims for reuse in interactive systems design.” International Journal of Human-Computer Studies 50, pp. 213-241.
== A. G. Sutcliffe (2000). “On the effective use and reuse of HCI knowledge.” ACM Transactions on Computer-Human Interaction 7 2, pp. 197-221.
== J. M. Carroll (2000). Making Use: Scenario-based Design of Human-Computer Interactions. MIT Press.
== A. S. Sutcliffe (2002). The Domain Theory: Patterns for Knowledge and Software Reuse. Lawrence Erlbaum Associates.
== M. B. Rosson and J. M. Carroll (2002). Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Morgan Kaufman.