Archive

Posts Tagged ‘design rationale’

Celebrating Toulmin

March 25, 2013 Leave a comment

Stephen Toulmin's Wikipedia and USC photo

The late Stephen Toulmin would have turned 91 today—and he came pretty close, making it to 87.  And as an inspiration to us all, he remained very active through most of his life, released an updated version of his seminal book The Uses of Argument in 2003.  The book has never been out of print, and its ideas have influenced researchers in areas from rhetoric and communication to computer science and engineering.

At the heart of his argumentation methods is the notion of a claim, a statement that you are seeking to argue is correct.  The subtle but important part of that definition is that a claim is falsifiable, in that one can argue successfully for or against a claim.  And, the “truthiness” of a claim may vary as we learn more things—consider, for example, claims about the age of the universe or the intelligence of dinosaurs. I provide an extended look at how the notion of claims evolved in human-computer interaction in a previous post.  Or, you can read my Making Claims book for the long story about claims in HCI!

But on his birthday, we should celebrate not only his work but his his life. Toulmin was born and raised in England, and he released his seminal book in 1958, when he was still a young researcher.  But when his ideas were not well received in England, he moved to the United States.  He spent time on the faculty at Brandeis, Michigan State, Northwestern, the University of Chicago, and the University of Southern California. In the paperback version of his book, released in 1963, he was defensive of his ideas.  He certainly didn’t rest on his accomplishments though—in many ways his 1992 book Cosmopolis provides a more historically-grounded view of his philosophy (and he comes across as much more comfortable with his ideas).  The updated version of his book came out in 2003, and it, like much of his work of that time, reflected both a more confident and grounded philosophy while embracing his life position as a dissenter.

In many ways it would be hard to emulate his career track, as much of his highly-cited work was books and not papers, reflecting a different era in research.  But his career focus and ability to evolve ideas is worth studying.  And our current era has its own advantages–I can instantly post a blog entry on his birthday to initiate a small celebration and reflection!

Advertisements
Categories: Claims Tags: , ,

Thinking visually, engaging deeply

January 27, 2012 Leave a comment

Imagery provides opportunities to encourage thinking by enabling people to identify key aspects of an image and relate their own expertise to it. A well-chosen image can inspire new ideas, spark memories of prior experiences, highlight potential issues and drawbacks, and provide a point for conversation and debate. Eli Blevis has an interactions article, CHI workshop, and regular course at Indiana University that explores the impacts of digital imagery in HCI and design. In his article, he describes digital imagery as a form of visual thinking, where visual forms are used to create content and make sense of the world.

We turned to imagery as a way to inspire groups of designers to think broadly and engage meaningfully with each other during the design process. We looked for ways that images could serve as a starting point for group design activities, and as a gateway to other design knowledge. Specifically, we are interested in how imagery can be used to enhance claims during early-stage design. Claims, conceptualized by the classic Toulmin (1958) book and introduced to HCI by Carroll and Kellogg (1989), present a design artifact together with observed or hypothesized upsides (+) and downsides (-); e.g., a public display of information (+) can notify large groups of people about things of shared concern, BUT (-) often become unattractive, densely-packed discordances of data. Claims are accessible when compared to much denser knowledge capture mechanisms like papers, patterns, and cases. But it is still a daunting task for designers to look through long lists of textual claims toward finding the right ideas.

Our approach to mitigate this problem is to use imagery as a bridge to each claim. We chose to represent each claim with an image, selected not just because it captured a key aspect of the claim but also because it allowed designers who viewed it to include their own interpretation of the technology and the context.

Information exhibit

Information exhibit image used in design sessions

We have used a set of around 30 image-claim cards in design activities (e.g., brainstorming, storyboarding), using the image cards both in printed and digital form. The benefits of the images-first approach were numerous. It allowed designers to process large numbers of claims quickly, connecting the ideas to their own experiences and expertise toward solving a design problem. It supported collaboration among designers through the shared understanding centered around the images. It encouraged broad speculation down paths not captured by the claims, sometimes resulting in new and different directions. A set of papers led by Wahid at Interact, DIS, and CHI capture the lessons and tradeoffs.

All of this is in keeping with the nature of a claim, whose original intent was as a falsifiable hypothesis (Toulmin, 1958; Carroll & Kellogg, 1989). However, a purely textual claim risks narrowing the associations of the reader to the words in the claim, and thus limiting the design considerations and even alienating designers unfamiliar with the text of a claim. It is through imagery, and specifically through images as the initial shared view in a design session, that designers can make sense of a problem and create meaningful and informed content.

Book review: The Heretic’s Guide to Best Practices

January 23, 2012 5 comments

The Heretic’s Guide to Best Practices: The Reality of Managing Complex Problems in Organisations, by Paul Culmsee and Kailash Awati, examines how groups of people can work to define a complex problem and to identify possible solutions. The book is divided into three sections: the first part argues why “best practices” often fail in the face of wicked problems; the second examines how people can work together (with a focus on dialog mapping, issue-based information systems (IBIS), and Compendium), and the third provides case studies illustrating successes and lessons learned from the authors’ work experiences. I found the middle section to be the most interesting and enlightening: it included motivation and history behind dialog mapping, with lots of illustrative examples and key citations balanced by alternative approaches. Much of the book centered around Compendium use and examples, the free IBIS-based dialog mapping tool I discussed in a previous post. In case you worry that the authors don’t eat their own dog food, a great many of the figures were generated by Compendium—reflecting intermediate steps of how a manager can address wicked problems using the tool.

The book represents an interesting pairing of authors. Paul Culmsee is a consultant who probably knows more about dialog mapping and Compendium than anyone (except maybe Jeff Conklin of gIBIS fame, who wrote a glowing foreword to the book with high praise for Culmsee). Kailash Awati is an information systems manager, with a couple of Ph.D. degrees and experience at several levels in academia. Both Culmsee and Awati blog prolifically, and many of their blog posts fed nicely into this book (a trick I’m using to prepare my book). People familiar with their styles will find their key writing styles featuring irreverent humor, pop-culture references, and in-depth examples prevalent in this book. (At times, though, I feel their pop-culture irreverence would be better if rooted in fact; e.g., the real Clippy story is interesting and perhaps relevant, and people and stories behind the development are still out there.)

There were a few major weaknesses of the book (though in the spirit of “wicked-ness”, many of these drawbacks to me may be neutral (or advantages!) to you, so take them as such). The index is very weak (less than 3 pages for a book approaching 400 pages). I’d love to look up what they have to say about strong reciprocity, or whose views of claims they discuss, or their view of McCall’s PHI approach to wicked problems, or their thoughts on positions in IBIS, or numerous other topics—but such a short index just doesn’t provide adequate support for a lot of important queries. In addition, I often find that books suffer from a certain myopia when it comes to the authors’ favored approaches, though there’s somewhat less fan-dom in this book than is seen in many books of this type. They certainly show a favoritism to IBIS and Compendium, but it’s the authors’ prerogative in writing a book to choose approaches to focus on and how much to talk about the weaknesses of a favored approach. More generally, they took the “depth over breadth” approach in this book, with heavy details about a few approaches rather than touching on a more inclusive set. It’s great to see examples, but not at the exclusion of alternatives. Somewhat telling, the references list contains only 122 references—there’s no mention of the work of Schön, Toulmin, McCall, Moran, Carroll, or others who have had important (nay, foundational) things to say about the topics in this book.

So who should get this book? The book targets technology managers who are looking for a way to address complex problems, and plenty of software professionals (e.g., ones who want to “deprogram” their managers) could benefit from it as well. Certainly anyone who uses Compendium or, more generally, embraces IBIS as a design approach or wicked problems as a problem classification should read it. If you like Jeff Conklin’s book, then (dare I say it?) I bet you will like this one even more. To grossly oversimplify, this is like Conklin’s book but moreso: more motivation and framing of the problem type, lots more examples, 5 years more of experiences and Compendium advances, more history of where these ideas came from, and more positive and negative examples of Compendium’s utility. If that sounds appealing, you should get a copy of this book.

The evolution of claims

January 18, 2012 1 comment

This post seeks to trace the evolution of the claim in human-computer interaction (HCI), from its introduction in the Carroll and Kellogg (1989) paper through the appearance of three books, Carroll’s Making Use (2000), Sutcliffe’s The Domain Theory (2002), and Rosson and Carroll’s Usability Engineering (2002). (A chronological list of key papers is provided at the end of this post.) The definition and role of “claims” shifted significantly during that time period; I’m seeking to identify some of the evolutionary shifts from 1989 to 2002. This list isn’t meant to be complete, but rather it seeks to highlight the most important evolutionary points in the conceptualization of the claim.

Three phases highlight the progress in this evolution:
– Carroll and his colleagues at IBM T.J. Watson in the late 1980s and early 1990s. They were seeking ways to design not just toward creating a single design, but toward crafting a theory-based approach to design to enable designers to build on each others’ work in a meaningful, scientific way. This work continued until Carroll left for Virginia Tech, at which time his focus largely shifted to collaborative computing (save for a few papers that seemed to draw on his IBM work).
– Sutcliffe and Carroll’s collaboration, highlighted by Sutcliffe’s sabbatical time at Virginia Tech. Sutcliffe had been working for many years on knowledge abstraction in software design, and, like Carroll and his group, he was inspired by potential roles for theory in HCI.
– Three summative works led by Carroll, Sutcliffe, and Rosson. Each presented a different view of the role of claims—in the fields of design, engineering, and education, respectively.

Claims were introduced to the field of HCI in Carroll and Kellogg’s “Artifact as theory nexus” paper at CHI 1989. They seemed to base their definition on Toulmin’s 1958 use of the term, in which he established claims as a hypothesis-centered approach to crafting arguments. The Carroll and Kellogg paper seeks to move beyond the narrow focus of cognitive-based theories that were prominent in the 1980s (that focused on low-level phenomena like keystrokes) by introducing the a hermeneutic approach based on psychological claims, the effects on people of both natural and designed artifacts. Claims were the central part of a task-analysis framework, an attempt to position the design and interpretation of HCI artifacts as a central component of HCI research. This approach was intended to bridge the gap from research to innovation—reconciling the “hermeneutics vs theory-based design” conflict in the title. Several examples in the paper showed how developing an understanding of a claim—the artifact and its possible effects—can point out how much we have to learn and can encourage us to draw broader conclusions. Many of these issues, in particular the connection of claims and claims analysis to the task-artifact cycle, is elaborated in a Carroll, Kellogg, Rosson 1991, but the ideas were first presented in the 1989 paper.

A 1992 BIT paper by Carroll, Singley, and Rosson provided the first in-depth view of the tech transfer of UE results (though see the Moran and Carroll 1991 special issue and 1996 book described below). It connected the Scriven view of mediated evaluation to claims upsides and downsides, positioning claims as a contributor in the field of design rationale. In so doing, it expounded upon claims as a way to reuse knowledge, by encouraging designer consideration of specialized vs abstract claims. The expectation was that designers could use claims to “avoid throwing away thoughtful empirical work”. They avoided Grudin’s paradox, stating outright that design rationale (including claims-centric design rationale) was not an automatic mechanism, but requires additional human thought to yield a reusable knowledge unit.

A 1992 TOIS article by Carroll and Rosson opined that HCI should be an action-science “that produces ‘knowledge-in-implementation’ and views design practice as inquiry”. The paper argues that the task-artifact cycle is an action-science because designers must respond to user requirements by building artifacts with upsides and downsides—i.e., claims. This paper distinguishes the scenario/claim roles as such: “Where scenarios are a narrative account, claims are a causal account.” It argues that scenarios provided a situation narrative, but they are too rich, hard to classify, and hard to reuse (arguments brought up again and addressed to varying degrees by Sutcliffe, Chewar, and others). It is the claim that establish the link to action-science by facilitating design analysis, providing a mechanism for generalization and hypothesis, and explicitly recognizing potential tradeoffs.

A 1994 IJHCS paper by Carroll, Mack, Robertson, and Rosson provided a software-centric scenario-based design approach, with Point-of-View (POV) scenarios drawing parallels to object-centric/object-oriented development. This paper represents the most process-based, engineering-focused, and software-generative view of scenario-based design—both until this time and thereafter. Although claims play a fairly minor role in this paper (only appearing in step 4, leveraging the upsides and downsides in analysis and hillclimbing), there seemed to be opportunity for a much larger role: identifying objects, specifying interactions between objects, supporting inheritance, etc. There was also initial discussion of an education focus for POV scenarios, SBD, claims, and such—but it was not elaborated, and the 2002 Rosson and Carroll textbook described a more simplified approach to teaching design. This paper seemed to be hypothesized starting points that were not fully pursued by the authors—rich for mining by Sutcliffe, Chewar, and others in the years to come.

Moran and Carroll’s 1996 Design Rationale book (elaborated from their 1991 special issue of the HCI Journal) is pointed to as a landmark in the field of design rationale. It draws together contributions from Jintae Lee, Allan MacLean, Clayton Lewis, Simon Buckingham Shum, Gary Olson, Gerhard Fischer, Colin Potts, Jeff Conklin, Jonathan Grudin, and many others. Of relevance to the topic of claims is the introduction (by co-editors Tom Moran and Jack Carroll) and a Carroll and Rosson chapter. These chapters exhibit connections in their work to Horst Rittel (wicked problems, IBIS), Francis Bacon (deliberated evaluation), Herb Simon (environment and behavior), and Donald Schön (contexts of experience)—putting forth the most synthesized view of the position of claims within the design community. Some of the psychological themes, particularly those of Simon, are elaborated in Carroll’s 1997 journal paper in Annual Reviews of Psychology.

A 1999 Sutcliffe and Carroll IJHCS paper summarizes the joint efforts of the two authors on the use of claims as a knowledge capture and reuse mechanism. It delved into the possibility of using claims as a reuse mechanism, a concept touched upon in previous work but never described in sufficient detail. The paper introduced a formatting and classification scheme for claims (and scenarios) to enable their reuse, including a process and alternate pathways for claim evolution. Among the augmentations was the first explicit connection to its derivation history and background theory (i.e., where it came from), leading to the first claim map that can reflect parentage, original/evolving context, motivation, evidence, and possibilities for reuse. Also of great importance was the acknowledgement of work left to do: methods for indexing, tool support (hypertext links, structure matching), and the need for buy-in (and stay-in) incentives.

Sutcliffe’s 2000 TOCHI paper seeks to address the irrelevance of HCI in industry, particularly with regard to a theory-based engineering approach. The paper seeks to identify ways to deliver HCI knowledge in a tractable form that is reusable across applications—and, more importantly, across application areas. The paper argues that claims could provide a bridge if reuse scope was improved; specifically, if there were generic versions of claims and artifacts, and if there were mechanisms for matching claims to new application contexts. The bulk of the paper provides a three-step process to accomplish this: steps for creating more generic claims, mechanisms for cross-domain reuse, and approaches to recognize broader implications. Parts of these are elaborated in Sutcliffe’s book (described later) and in the dissertations of Christa Chewar and Shahtab Wahid. Other important products of this work are the notion of claim families, a claims-patterns comparison, and an explicit recognition of the importance of claims as “designer-digestible” knowledge (one of my favorite phrases).

This series of papers culminated with three books that offered very different visions of design, with very different roles for claims. I plan to elaborate on these books in a future post, but here’s a brief summary of each. Carroll’s 2000 Making Use book pulled together his vision for scenario-based design for scientists, with an eye toward the discovery process. Claims are used to augment the scenario-based design process, highlighting key aspects of the design (and leaving the generalization of claims as an exercise for the designer). Sutcliffe’s 2002 The Domain Theory provides a reuse-centric view of software engineering, extending the vision of Rittel and the design rationale literature and approaches. The role of claims is to make concrete Domain Theory’s high level of abstraction (too high, according to critics) by leveraging the high utility (but low flexibility and poor reuse) of claims. Finally, Rosson and Carroll’s 2002 Usability Engineering textbook advocates scenario-based development as a teaching tool, with claims and claims analysis a complementary and guiding technique to scenario development during each stage of design. It presents claims in a simplified, stripped-down manner (for better and worse) meant to be highly accessible for students. These books kicked off a period of scientific application, engineering refinement, and creative design that has continued in the years since they appeared.

Chronological bibliography:
== S. E. Toulmin (1958). The Uses of Argument. Cambridge Press.
== J. M. Carroll and W. A. Kellogg (1989). “Artifact as theory-nexus: Hermeneutics meets theory-based design.” In Proceedings of CHI, pp. 7-14.
== J. M. Carroll, D. A. Singley, M. B. Rosson (1992). “Integrating theory development with design evaluation.” Behaviour and Information Technology 11, pp. 247-255.
== J. M. Carroll, M. B. Rosson (1992). “Getting around the task-artifact cycle: How to make claims and design by scenario.” ACM Transactions on Information Systems 10 2, pp. 181-212.
== J. M. Carroll, Mack, S. R. Robertson, M. B. Rosson (1994). “Binding objects to scenarios of use.” International Journal of Human Computer Systems 41, pp. 243-276.
== J. M. Carroll (1997). “Human-computer interaction: Psychology as a science of design.” Annual Reviews in Psychology 48, pp. 61-83.
== A. G. Sutcliffe and J. M. Carroll (1999). “Designing claims for reuse in interactive systems design.” International Journal of Human-Computer Studies 50, pp. 213-241.
== A. G. Sutcliffe (2000). “On the effective use and reuse of HCI knowledge.” ACM Transactions on Computer-Human Interaction 7 2, pp. 197-221.
== J. M. Carroll (2000). Making Use: Scenario-based Design of Human-Computer Interactions. MIT Press.
== A. S. Sutcliffe (2002). The Domain Theory: Patterns for Knowledge and Software Reuse. Lawrence Erlbaum Associates.
== M. B. Rosson and J. M. Carroll (2002). Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Morgan Kaufman.

Rationale in SE and rationale in UE–what’s the difference?

September 20, 2011 1 comment

The question of the difference between design rationale in software engineering and usability engineering came up at my UC Boulder talk last week. It’s something that I’ve been thinking about for a while, and more recently as I’ve been reading about design rationale from a software engineering perspective (Burge, McCall, Schneider, etc.).

The big difference seems to be in the certainty embedded in the rationale. Design rationale in software engineering seeks to establish a position of truth, rooted in the functionality of the computer (compared to the nature of physical artifacts, as seen in other fields). The preface and introduction of Burge’s Rationale-based Software Engineering book do a good job of distinguishing design rationale in software engineering from design rationale in other fields.

In contrast, design rationale in usability engineering (and even more so in HCI) seeks to provide pointers toward the right directions–through methods, approaches, or lessons learned. Even a repository of web design patterns like van Duyne’s Design of Sites is lauded not because it preaches HCI truths but because it discusses how to design (e.g., how to promote e-commerce, how to settle on a page layout, how to customize for mobile devices). It includes checklists and is rooted in context, allowing the reader to decide what’s relevant.

This is why I feel claims (or something like them) are the future of design rationale in UE/HCI. They are meant to be hypothetical, subject to change based on changes in context, advances in knowledge, and retargeting of product design. They are more flexible than design rationale methods from software engineering and other fields, reflecting the flexible nature of the field of HCI. But that then makes the creation of claims libraries or other reuse repositories much more difficult–one must balance correctness with hypothesis, permanence with flexibility.

PIC-UP Mobile: For mobile by mobile

September 14, 2011 1 comment

Much of the ongoing research in my group focuses on usability tools for application developers, centered on knowledge transfer and decision-making among teams of designers. At the heart of my approach is the notion of a claim as a unit for knowledge capture, sharing, and negotiation–claims provide a falsifiable, designer-digestible chunk of knowledge that encapsulates an interface feature, its upsides, and its downsides. The small, hypothetical nature of a claim provides designers with opportunities to debate and evolve ideas to meet the needs of novel problems. Recently, Shahtab Wahid led a quality series of papers at Interact, DIS, and CHI (available from my pubs page) that summarizes our progress and describes PIC-UP for those interested in learning more about it.

An emerging focus is on tools for diverse teams, particularly teams of domain experts with little or no expertise in usability engineering. Since it’s tough for many companies to hire a large team of usability professionals to oversee interface development efforts, a suite of tools has promise to assist with the capture, sharing, and deliberation that must happen when addressing the needs of large stakeholder groups. It certainly doesn’t remove the need for a usability expert, but it can help magnify the power of experts, allowing a smaller number of experts to contribute to a larger number of projects.

As a next step, we are working to transition PIC-UP to a tool we call PIC-UP Mobile that will be useful in industrial and educational settings–specifically, to assist in the early-project development of mobile interface designs. We want the tool to be embedded with a small but high-quality set of claims that can be used in design activities. Designers will be able to browse, rate, group, and evolve claims during the design process. Design sessions can create scenarios or storyboards that incorporate key claims, with This will enable a large and diverse set of designers to have meaningful roles in the design of appropriate user interfaces for human experts–particularly domain experts with little or no knowledge about user interface design whose opinions all too often are often ignored.

Based on an established context, PIC-UP Mobile should help to answer questions like these:
– How can an interface best show multimodal information to a user within a unique context?
– How can an interface enable appropriate interaction.
– What are the tradeoffs in choosing one design technique over another?
– Where is more research needed to determine an appropriate interface approach?

There are tons of application domains where this seems well-suited: medicine, education/training, command-and-control, gas-and-oil discovery and processing. All of these domains integrate diverse situations in which stakeholders with differing backgrounds must reach decisions about acceptable (if not optimal!) approaches to addressing a problem. We expect that PIC-UP Mobile will help in reaching these decisions toward defining appropriate user interfaces.

The future of claims

September 12, 2011 Leave a comment

In preparing to write a book about claims, I have the opportunity and obligation to expound upon what I think is the future (more like present-future) of claims.

To varying degrees, all of these arguments more generally apply to design rationale–but I think claims are particularly well-suited to solve many of the traditional problems with DR (as expounded upon in the first bullet).

Here are some current thoughts:

– Fixing design rationale with claims. Are claims the perfect instantiation of design rationale? Probably not! (Though anyone who has worked with me might be convinced that I feel differently.) Even though claims seek to hypothesize rather than expound the truth, there’s always a danger that they will be read as truth. A key step: Defining claims to getting rid of upsides and downsides–they are too polarizing and can lead to bias. Replace them with bullets or (partial) ratings to allow the designer to decide what’s an upside and downside within the context of the design or to rate them based on the current project’s needs. (One person’s bug is another person’s feature.) Thus leave open for debate whether an effect is an upside or downside.

– Identifying relationships between claims. The ability to understand the alternative, complementary, and contrasting ideas is key to enabling design. It’s particularly hard to do that at the level of a paper or a product–their nuances make it hard to compare them in a meaningful way. It’s not easy to do so with claims, but at least it seems possible. Visionaries like Otlet and Nelson recommend letting people identify relationships between claims-like chunks of knowledge. At the heart of Wahid’s thesis was the notion of claims relationships. Mathematical models can refine and quantify the relationships to highlight them to designers (e.g., Chewar’s approach to index based on critical parameters then use experimental results or designer surveys to rate the claims). There seems to be progress toward this goal.

– Identifying quality claims among a sea of mediocrity. Since claims are used in brainstorming, there necessarily will be a great many claims of questionable quality: claims that are incomplete, ill-conceived, too specific to a problem/domain (who cares if the claim is specific to some other domain; perhaps quality experienced designers can understand cross-domain relevance but most can’t), too general to be of use (the “fortune cookie” problem; the claim reads like a fortune cookie meant to appeal to everyone but instead being of little use). One solution: use the “god of claims” approach to identify claims of reasonable quality. Another solution: let the masses decide through votes or other recommendations.

– Making claims more accessible. Claims are pretty accessible compared to papers, patterns, and program review–but it’s still a daunting task for designers to look through long lists of claims toward finding the right ideas. One approach to mitigate this problem is to use pictures as a bridge to each claim; that is, to represent each claim with an easy-to-interpret picture (as in my Wahid/Branham work). This allows designers to process large numbers of claims quickly, connecting the ideas to their own problems. Another idea is to limit the number of claims and focus on high-quality ones (as in Francois’ ongoing MS thesis work). This enables the designer to focus on a smaller number of ideas; a more pattern-like approach, for better or worse.

– Automatic extraction of claims from databases. The databases could include professional papers, chat/meeting logs, blogs, case studies, scenario libraries, or other such repositories. If done with high quality, it would cut down on the high costs (and low immediate benefits) of generating claims sets. Numerous people have pursued or seem to be pursuing this idea: Janet Burge is working on this as the focus of her NSF CAREER grant to use speech recognition to decompose meeting transcripts; Ray McCall expressed interest in doing it; Jintae Lee seems to have done some of this in business settings.

This is probably a post that I’ll go back and edit. Emerging from these ideas are categories for issues claims (and design rationale) need to address to be more effective: creation, quality assurance, use/reuse. But there’s lots of other work that needs to be done in this area.

Categories: Claims Tags: , ,