2014 - Design Thinking Research - Building Innovation Eco-Systems PDF
2014 - Design Thinking Research - Building Innovation Eco-Systems PDF
2014 - Design Thinking Research - Building Innovation Eco-Systems PDF
Hasso Plattner
Christoph Meinel
Larry Leifer Editors
Design Thinking
Research
Building Innovation Eco-Systems
Understanding Innovation
Series Editors
Christoph Meinel
Larry Leifer
v
ThiS is a FM Blank Page
Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Christoph Meinel and Larry Leifer
Student Teams in Search of Design Thinking . . . . . . . . . . . . . . . . . . . . . 11
Shelley Goldman, Zandile Kabayadondo, Adam Royalty, Maureen P. Carroll,
and Bernard Roth
Team Cognition and Reframing Behavior: The Impact of Team
Cognition on Problem Reframing, Team Dynamics and Design
Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Greg Kress and Joel Sadler
Early and Repeated Exposure to Examples Improves
Creative Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Chinmay Kulkarni, Steven P. Dow, and Scott R Klemmer
Part II Design Thinkers Must Preserve Ambiguity
vii
viii Contents
We have seen over the years that many individuals appreciate the power of
engineering design thinking. At the same time we witness an almost unfathomable
skepticism about the ability of established organizations (corporate, academic, and
government) to really adopt the paradigm. Some argue that the paradigm only
works in the world of “start-ups”.
Motivated by this dilemma, we ask how the fundamental rules of engineering
design thinking might be translated into design requirements for building precision
innovation eco-systems.
We now have evidence in support of several design thinking activities that have
long been considered important, but until this time we have not had an explanation
or understanding of their value. Of these, the over-arching truth lies in the fact that
every physical product delivers a service; that every service is manifest through
physical products. Our research suggests that four “rules of design thinking” are
particularly relevant. The challenge of this section is to translate these rules into
innovation eco-system design requirements.
C. Meinel (*)
Internet Technologies and -Systems, Hasso Plattner Institute for Software Systems
Engineering, Campus Griebnitzsee, Postdam 14482, Germany
e-mail: [email protected]
L. Leifer
Stanford Center for Design Research, Stanford University, Panama Mall 424, Stanford 94305-
2232, CA, USA
e-mail: [email protected]
I. The Human Rule: All Design Activity is Ultimately Social in Nature. Never Go
Hunting Alone.
There are studies that substantiate the assertion that successful innovation through
design thinking activities will always bring us back to the “human-centric point of
view”. This is the imperative to solve technical problems in ways that satisfy human
needs and acknowledge the human element in all technologies and organizations.
Design Requirements for the Human Rule
Place people at the center of all things. Cover the walls with images of your
people. Celebrate their success and failures. Build environments that celebrate the
“human scale”. Forgo monuments. Create an atmosphere of empathy-in-action; for
yourself and others.1
II. The Ambiguity Rule: Design Thinkers Must Preserve Ambiguity. Never Go
Home Empty Handed.
There is no chance for “chance discovery” if the box is closed tightly, the
constraints enumerated excessively, and the fear of failure always at hand.
Innovation demands experimentation at the limits of our knowledge, at the limits
of our ability to control events, and with freedom to see things differently.
Design Requirements for the Ambiguity Rule
Keep track of assumptions; place them boldly in your design space. For every
constraint you are coping with, list a competing opportunity. Check your thinking:
are you’re looking for the global fix, or, are you keenly aware that most everything
in design and business is context dependent. Take time to define the problem and
solutions space context.
III. The Re-Design Rule: All Design Is Re-Design. Take the Big Idea Home. It Has
Been Done Before.
The human needs that we seek to satisfy have been with us for thousands of years.
Through time and evolution there have been many successful solutions to these
problems. Because technology and social circumstances change constantly, it is
imperative to understand how these needs have been addressed in the past. Then we
can apply “foresight tools and methods” to better estimate the social and technical
conditions we will encounter 5, 10, 20 years in the future.
Design Requirements for the Re-Design Rule
Hunting is tough. Taking it home is tougher. Nothing beats a prepared mind; be
sure your team is well informed about the history of organizational change and
context. How did others effect change? How did they circumnavigate the skeptics?
Take advantage of foresight thinking and tools.2
1
Kress, 2012: http://purl.stanford.edu/hm975hz5458
2
Cockayne, 2013: http://foresight.stanford.edu/index.html
Introduction 5
IV. The Tangible Rule: Make Ideas Tangible. Facilitate Human Communication.
Curiously, this is one of our most recent findings. While conceptual prototyping has
been a central activity in design thinking during the entire period of our research, it
is only in the past few years that we have come to realize that “prototypes are
communication media”. Seen as media, we now have insights regarding their
bandwidth, granularity, time constants, and context dependencies. The “make it
tangible” rule is one of the first major findings of the design thinking research
program documented in this book.
Design Requirements for the Tangible Rule
There are more great ideas out there in the world than inside our heads. Put
differently, searching in the world tangibly is a great way to get new ideas,
unplanned associations, un-dreamed metaphors, serendipity squared. Show me,
don’t tell me.3
We have summarized and in some cases paraphrased the design requirements in
the following table. Take the framework and apply it to your project, your organi-
zation, and your team. This is not a tool of physics. Everything about it is context
dependent. Define your context.
3
Edelman, 2012: http://purl.stanford.edu/ps394dy6131
4
Kress, 2012: http://purl.stanford.edu/hm975hz5458
5
Albert Bandura: http://en.wikipedia.org/wiki/Albert_Bandura
6 C. Meinel and L. Leifer
Started in 2008, the HPI Design Thinking Research program is financed and
supported by the Hasso Plattner Foundation.
6
Cockayne, 2013: http://foresight.stanford.edu/index.html
7
Edelman, 2012: http://purl.stanford.edu/ps394dy6131
Introduction 7
differ substantially from traditional corporate structures? The overall goal of the
program is to discover metrics that determine the success of challenges approaches
with design thinking methods. A special program interest is to explore the use of
design thinking in the field of Information technology and IT systems engineering.
The organization of the material presented in this book follows the four rules: “The
Human Rule”, “The Ambiguity Rule”, “The Re-design Rule” and “The Tangible
Rule”.
Chinmay Kulkarni, Steven P. Dow and Scott R. Klemmer explore in “Early and
Repeated Exposure to Examples Improves Creative Work” online creativity exper-
iment and the outcome of the effect of sample timing on this creativity. The
between-subject experiment deals with repeated exposure to the experimental
setting and its iterative experimental results.
Examination of the issue of whether creativity can be acquired or learned by an
individual over time and how this relates to cognition, behavior and the brain is the
last article in this part. The focus is on cognitive, behavioral and neural processes
that may contribute to creative capacities. It’s entitled “Impact and Sustainability of
Creative Capacity Building”, by authors Grace Hawthorne, Eve-Marie Quintin,
Manish Saggar, Nick Bott, Eliza Keinitz, Ning Liu, Yin-Hsuan Chien, Daniel
Hong, Adam Royalty, and Allan Reiss.
The authors of the article “Acting with Creative Confidence: Developing a Creative
Agency Assessment Tool”, written by Adam Royalty, Lindsay Oishi, and Bernard
Roth, deal with the problem of enabling new technologies in learning models aimed
at developing creativity and innovation, without checking their outcome. This
article uses quality and quantity test results on behalf of new criteria models.
Captioned with the title “How Design Thinking Tools Help to Solve Wicked
Problems” the article speaks about a psychological way of thinking and a perspec-
tive on solving such problems. Julia von Thienen, Christoph Meinel and Claudia
Nicolai move in a field of design thinking tools where by their very nature these
tools offer the necessary support to answer wicked challenges.
The next article in this part continues the thoughts of the preceding. This text is
entitled “How Prototyping Helps to Solve Wicked Problems.” The authors are
Birgit Jobst and Christoph Meinel. They explore how wicked problem solutions
can be figured out by improved prototyping skills and the relations between three
major factors in this process: prototyping skills, problem solving competence and
the issue of self-efficacy.
The beginning article in this part delivers innovative companies who are adopting
design thinking methods and techniques in order to try to create products with
maximum end user value. The text focuses on small and big software companies
who combine agile development processes with design thinking and examines the
advantages of this approach. “A Research Plan for the Integration of Design
Thinking with Large Scale Software Development Projects” is by Thomas Kowark,
Franziska Häger, Ralf Gehrer, and Jens Krüger.
The authors of the article “Sharing Knowledge through Tangible Models –
Designing Kickoff Workshops for Agile Software Development Projects,” Markus
Guentert, Alexander Luebbe and Mathias Weske, look at precise descriptions of the
systems to be built. They describe that all data of a systems is compiled from
observations and follow-up discussions with the user.
Bastian Steinert and Robert Hirschfeld wrote “How to Compare Performance in
Program Design Activities: Towards an Empirical Evaluation of CoExist”. It
explores the design of an empirical experiment that compares programmers’ per-
formance in program design tasks. The target is to focus on the benefit of using
CoExist in programming and accordingly make changes from within.
The last article shows how design thinking works in large companies. Holger
Rhinow and Christoph Meinel take a closer look at management’s perspective by an
interview process in “Design Thinking Expectations from a Management
Perspective”.
3 Summary
taught and what do we expect students to learn when we teach them design
thinking? If the aim is to teach design thinking concepts, processes, practices, and
dispositions, how do we set goals for what we expect students to learn, and how do
we understand or assess their learning experiences? Should instruction focus on
individual or team experiences?
Rittel and Webber (1973) used the term “wicked” to describe design problems.
Cross (2006) extends that view to design thinking instruction, considering it as
problematic as design is itself:
It is also now widely recognized that design problems are ill-defined, ill-structured, or
‘wicked’ (Rittel and Webber 1973). They are not the same as the ‘puzzles’ that scientists,
mathematicians and other scholars set themselves. They are not problems for which all the
necessary information is, or ever can be, available to the problem-solver. They are therefore
not susceptible to exhaustive analysis, and there can never be a guarantee that ‘correct’
solutions can be found for them. In this context a solution-focused strategy is clearly
preferable to a problem-focused one: it will always be possible to go on analyzing ‘the
problem’, but the designer’s task is to produce ‘the solution’.
In design thinking classes, instructors are trying to teach the components and
process of design thinking, and also teach about the end game––transformations in
the way the newly minted design thinkers think, act and intuit as they design novel
and innovative solutions to problems. These ultimate educational aims set a high
bar with observable and documentable outcomes, and with more subtle changes
that are difficult to anticipate, define and document. von Thienen et al. (2012) liken
the process of a group learning design thinking to engaging in group therapy and
suggest that some of the processes, dynamics, and outcomes are comparable.
Teaching and learning design thinking is a complex enterprise.
In teaching design thinking, courses aim to provide students with group learning
experiences such as “radical collaborations,” interdisciplinary experiences, and
learning from deep exchanges with peers. Generally we teach students how to do
design thinking and how to do it in teams. Instructors hope students have a
productive team learning experience and, as such, rely on student teams to be
proficient enough to carry the students through the process and projects that are
assigned. The team process and practice is one of the sticky problems of design
thinking education because courses are situated in educational systems that have
emphasized and rewarded individual learning and achievement. It is not surprising
then, that the team experience is in conflict with the individual achievement
imperative, further complicating how we teach design thinking. Creating an effec-
tive and successful team learning experience is a sticky wicked problem.
As design thinking instructors and researchers, we aim to better understand the
team learning experience and to find ways to better support it. When we began this
research, we thought we would learn how to assess team learning in a design
thinking course; when we finished, we learned that the teaching process might
better address the student team experience. We were especially focused on the
experience of groups during their out-of-classroom-time experiences, so we were
able to capture how teams worked through the design process independent of their
course instructors or coaches. The research taught us about how teams handle the
Student Teams in Search of Design Thinking 13
design thinking process as they are learning and enacting it. If class time was when
students were introduced to design thinking processes and mindsets, team meetings
were times to fulfill assignment tasks in practice and production runs. They were
also the times when important components of design processes and solutions –
point of view statements, brainstorming, and design solutions – were experienced
and negotiated.
We set out to assess team learning by starting with the basics and examining
what the student teams were doing and how they were interacting. Previous studies
were of interest. Kress and Schar (2012) examined cognitive differences among
group members, and Brereton and colleagues (1996) determined that team interac-
tion affected design outcomes.
Conflicts among group members seem endemic in teamwork and surfaced in this
study. From prior research we know that a sense of belonging and togetherness, and
sharing joint goals are important to design group’s abilities to apply itself to its class
projects successfully (Mercier et al. 2009). Teams worked best at enacting and
representing what they were learning when individual members could not accom-
plish the tasks alone. Divide and conquer did not work best on design teams. Katu
contends that, “Harmonizing is about emphasizing differences together” (Katu
2012, p. 18), suggesting that the best functioning teams succeed at keeping the
team together despite members’ differences. In his account, the effective and
successful team members share passion, common goals, and commitment to
excellence.
We examined student teams using methods enabling us to capture and take an
ethnocentric view. Specifically, we followed two teams, from two courses, as they
met outside of class to work on their class-assigned projects. Each group had 2–3
weeks to work on their own.
Together, the two groups show a process of students becoming introduced to
design thinking and working to become acceptably proficient at it. Our results
indicate that teams did not necessarily stick to the tasks that corresponded to their
immediate assignments or their current stage in the design process. For example, a
team with the object of prototyping, often drifted back and forth to earlier and later
stages of the process.
We decided to focus on student teams in their independent work outside of the
classroom. We acknowledge the facilitator role instructors play in design teams. As
instructors attempt to create environments where teams can accomplish their
independent work and achieve success, they are the supporting cast to the design
thinking team ensemble. This study focused instead on students and their emergent
roles in this ensemble. Both teams accomplished moments of unease and eventual
alignment, illuminating the dynamic nature of teams in collaboration. This research
on student teams’ collaborative experiences provides a more nuanced view of how
we might design more effective courses, and seeks to answer several questions:
What is the nature of team collaboration for new design thinkers? How does what
we learn about student teams implicate how design thinking might be taught?
14 S. Goldman et al.
performance. They had nothing to confess. In other words, members talked about
enjoying the process and their team while having no sense of whether or not their
teammates might agree. Team Two shared very extensive email exchanges with the
research team. While both teams struggled to schedule times to meet, Team Two
split up tasks as indicated in the email exchange in Fig. 5.
When asked about her views on perfectionism during her exit interview, Nora
described the internal shift she had to make to adjust to her teammates’ outlook on
the design work at hand:
So I definitely sort of let go between DP1 [Design Project 1] and DP2 [Design Project 2].
Like in DP1 I was highly perfectionist and kind of freaking out and frustrated and
Just being like (to DP1 team members), ‘You guys, we have to have a perfect final
product.’ And our final presentation wasn’t perfect. There was just a lot of frustration or just
lack – just total lack of communication. It was just so frustrating because I had invested all
of this energy and then the output wasn’t what I would have wanted it to be. So how I dealt
with that going into DP2, I think, was putting less of myself into it so I had a lot less to lose.
But not in a way that I completely divorced myself from this process, but sort of in a way
that like, ‘I’m going to be more balanced in my approach to this, and rely on my team
members and if they don’t do things the way that I would want to do it then that’s okay.’
You know, so striving for my own version of perfectionism less.
[01.50] One thing that I feel like...I feel like our team’s
doing really well and we’re together, we have
productive meetings. Something that I think has been
like a little bit challenging has been coordinating
schedules and also, like, not everybody being every
meeting… the ideal would be obviously, if all of us
were there at the same time. And I just think that like
a significant amount of time is rehashing and getting
everyone on the same page. Another observation.
Umm...
similar moment of crisis and its subsequent resolution. Understandably, the team
meetings we chose for our analysis had disparate design agenda and team
imperatives. Team One’s meeting was closer to the beginning of the design process,
with members struggling over the transition from empathy work to defining a Point
of View statement (POV). The conversation and activity in this meeting occasion-
ally drifted to other stages in the process (for example, George brings up a POV that
has good implications for a future prototype), and involved very little explicit
18 S. Goldman et al.
Fig. 4 Excerpt from student Nora Smith(NS): Yeah, so if you couldn’t contribute to one
exit interview part that was fine, but then you made up for it later so…
And I think everyone walked away from the project…my
sense is that everyone walked away from the project
feeling pretty good about what they had contributed and
what everyone else contributed. And it was like a pretty
positive team dynamic in general. Like we would send
around emails…I would send something out saying
(smiling and miming typing), “I’m going to do this.” And
Steve would respond (miming typing), “Way to go, Nora!”
And it was very, like, “Go Team!”
NS: It was kind of weird. I think Steve actually was the one
who initiated all the, “Way to go!” (laughing fondly) and the
“Go us!”
ZK: How did it make you feel the first time he did it?
Hey guys,
Some updates:
I read through the booklet and made a few changes, but
since it's just a prototype, perfection isn't a requirement. I
think it's a good start.
I think it'd be great if we could photo-document our meeting
this afternoon and maybe have the preceptors (and any
other volunteers we can find around tressider?) roleplay…
this is what i have so far for the booklet. Please feel free to
make any changes and add anything that you want. Send
me your version by the end of tonight so I can have a full
prototype tomorrow before meeting the preceptors!
3 Analysis
With the videos, confessionals and interviews completed, analyses were conducted
to make use of the student provided information and directives. Analyses started
with open coding of the confessionals and interviews. We triangulated across data
sources, finding instances on the video that were reported in voice messages,
emails, or interviews. The research team watched video segments and chose one
from each team for analysis. The selection process involved trying “see” the issues
students described for us and identifying a beginning and an end to the respective
events to which they referred. Once segments of the group in a topical event were
identified, we created a content log that described what happened throughout.
The preliminary analyses and the emphasis on team interactions led us to look at
how the groups attended to and accomplished their interactions. Since we were
already rooted in the idea that talk and action were some of the building blocks of
group work and learning, it made sense for us to look at the task the group was
working on, the movements they made in talk and related actions, and how and
when their interactions were relevant to the design thinking process they were
learning. We concentrated our analyses on when “bids” were made in the group and
how they were received and acted on. Bids are a struggle for control, attention, or
for the right to speak within a group (Schegloff 1998, 2007). We considered bids to
be requests/imperatives for action, type of work, adherence to process, and attention
to relevant or irrelevant topics. They were requests for action from the team
members, and we expanded the definition to include both verbal and non-verbal
requests. A bid was as simple as requesting a turn to talk or building on another’s
idea, or as complex as entering a new topic into the conversation or suggesting the
design solution to the challenge. Because bids were invitations or requests for
interaction, we looked at bids offered by the students and subsequent responses.
When we began watching the first group, we realized that the students were having
a difficult time staying with their task to develop a POV and appeared to be all over
the process map, pulling anything they knew with respect to design thinking into
their activity. We decided to analyze the students in terms of their talk, focus, and
action, the topics they were taking up, and the design thinking skills or processes
they were invoking in the moment.
4 Coding Categories
The research team developed a series of codes by which to analyze the video. We
looked for verbal bids, non-verbal bids, and their responses in the interactional
palette. The non-verbal included movements such as changing place in the
workspace, grimacing, pointing, or writing on the board with a marker. By repeat-
edly watching the videos, we conducted an open coding process, allowing codes
with corresponding numbers to be generated and defined. The codes were not
Student Teams in Search of Design Thinking 21
Ellie Megan
Bid/Resp Bid/Respo
onse nse Megan (Objects of Megan bid
Ellie (Objects of
[Clip] Time Ellie (Verbal) Ellie (Spatial) focus/Topics) Megan (Verbal) Megan (Spatial) focu/Topics) short
[Comes back
onscreen. Wlks
[Crosses from bottom around table: from
left to the table and offscreen, along the
uncaps a marker left, to now directly
from the table and facing camera. 1
prepares to write] I places white folder
just like that idea of on the table and
like... opens it]
0:00:19 2 60 30
50
Fig. 6 Segment with bid and response codes by number by team member
exhaustive of all possible codes, but did capture the range in these particular data.
The codes established as Verbal Bid Responses are examples of this: “Acceptance”
was defined as agreeing with the previous bid; “Building up” was defined as adding
an additional action or idea to the previous bid; “Ignoring” was defined as not
responding to the previous bid or responding with an unrelated action or idea.
“Rejecting” was defined as explicitly disagreeing with the previous bid. Examples
of Spatial Bids included, “Writes,” “Proxemics moves,” “Gestures,” and “Attend-
ing to one’s own person” (fixing hair, arranging clothing). Objects of Focus codes
included “Design thinking steps,” “Team collaboration,” “Logistics,” and “Social
work.” The preliminary code list was expanded and refined during the coding
process.
From coding we realized that spatial and verbal bids and responses were being
used to affirm or dissent in similar but distributed ways. Some team members would
show agreement by leaning in and looking directly at their teammates when these
teammates had the floor to speak. At other times, they would verbalize their
agreement with “Umm, hmn” and “Sure! I think that’s great!” In both cases the
bids proposed by their teammates were accepted (Fig. 6).
In Team One, Ellie used her spatial responses for expressing dissent more often
than other teammates: walking offscreen to a different whiteboard; opening a can of
soda in response to a direct question; rifling through a stack of post-its while her
teammates were looking over a chart together; these were all examples of how she
ignored her teammates bids. These forms of ignoring and accepting were more
nuanced and passive forms of dissenting and affirming, respectively. In Team Two,
Angie was often offscreen, or documenting somewhat unrelated material on the
whiteboard, or preparing for the next stages of the design process on her laptop.
22 S. Goldman et al.
Angie’s teammates tended to overlook these dissenting spatial bids so that the bids
interrupted the flow of ideas less often than Ellie’s did. In Ellie’s team, the team
members on the receiving end of passive bids were highly attuned to them, stopping
what they were doing in response to disruption of spatial synergy, and in these
instances dissent appeared harder to resolve.
There were “idealized” and active forms of responding to bids, such as when a
teammate responded to another’s bid by elaborating on it. There were also passive
forms that could be affirming, disregarding, or dissenting. When Steve copied down
Nora’s comments on the board, this was a move to build-on her points. When
Megan responded to Ellie’s bid with a counterargument this was a move to reject
the bid made. Often, build-ons and rejections were complex and compounded, with
spatial and verbal sub-bids embedded in them. For example, George, in response to
a bid Megan makes, nods then pauses before adding,
“I think ‘why’ is more interesting. . .[Walks over to the board. Takes his time skimming it.]
Why is this the center of his life? [Approaches the circled “center of his life” and taps it with this
finger]”
In this instance, the nod is a spatial acceptance. The comment, “I think ‘why’ is
more interesting. . .” is a verbal challenge that modifies Megan’s comment and
shifts the team’s focus in a dramatic way. This bid is a verbal build-on. When
George walks over to the board, he offers a new spatial bid by demanding the
attention of his teammates, who in turn accept his bid by following him with their
eyes. He uses this to build-on his own question, “Why is this the center of his life?”
and spatially builds-on what Ellie had written on the board “center of his life.” With
this compositeness in mind, we recoded the transcripts for both teams, this time
condensing bids and responses to one of four categories: (1) accept and (2) build-
on, the two affirming activities; and (3) ignore and (4) reject, the two dissenting
activities. Figures 8 and 9 show the distribution of the four categories in Team One
and Team Two, respectively.
Figure 7 indicates a fairly evenly distributed use of talk-space by the three
members of Team One. George built-on bids more than he rejected, ignored or
accepted them. Megan, rejected more than she built-on, accepted or ignored. Ellie
rejected and built-on less than she accepted and ignored.
Figure 8 illustrates that while Nora contributed the most to the team talk space,
she built-on far less frequently than she accepted, ignored and rejected her
teammates bids. Steve on the other hand, took up the least talk-space and most of
it was accepting and building-on (his distribution is widest for these two forms of
bid response). Angie ignored far more than she accepted, but also built-on more
than she rejected.
In the following series of graphs, we show how these patterns of individual
behavior related to uptake of ideas. In our coding scheme, we cross-referenced the
bids and responses of each member. For example, for all of Georges bids, we
counted how many times either Megan or Ellie responded by elaborating, whether
Student Teams in Search of Design Thinking 23
rejection or building on. We did this for each member of both teams, and came up
with an uptake count that serves as a proxy for how visible that member was to their
teammates. The more uptake one team member had, the more visible they and their
use of talk-space were to their teammates. Was there a relationship between the
visibility of team members and their tendency to affirm or dissent? Recall how
Ellie’s voice message influenced our foci for analysis. In Ellie’s words, “So it’s a
little frustrating to me and I definitely am aware of the effect that has on me – you
know, in terms of actively contributing and continuing to when you feel like you
idea wasn’t heard.” This particular relationship between uptake (or visibility) and
bid type would help us explore Ellie’s claim about not being heard. Figures 9 and 10
show the relationship between uptake and the dissenting bids, and between uptake
and the affirming bids for Team One. Each graph shows three bars for each team
member: two bars for bid type, and the third for uptake count. In Fig. 9, Ellie, for
example, ignored her teammates bids 18 times, and rejected them 5 times. Her
teammates responded to these 23 combined dissenting bids only 5 times.
For Megan, who ignored teammates’s bids 8 times and rejected them 5 times
(a total of 13 dissenting bids), team members responded (by accepting, building on
or rejecting) 34 times. George’s total of 22 dissenting bids were responded to
26 times. Megan had, by far, the most uptake.
In fact, the difference between Megan and Ellie’s uptake counts was statistically
significant, as indicated by the Y error bars on Figs. 9 and 10. This indicates that
Ellie’s visibility was radically different and less than Megan’s and George’s during
that team meeting, while Megan’s and George’s visibility was not significantly
different from each other. Ellie was right! She wasn’t being heard as much as
Megan or George were. We mapped the relationship between uptake and type of
bids for Team Two but the results were not significant, indicating that each member
of the team was being heard (or had their bids taken up) by roughly the same
amount.
Student Teams in Search of Design Thinking 25
0:06:24 3 00:00
0:45
0:05:51 0 0:01:16
0:03:03 1:38
00:01
0:01:55
0:44:10 3 00:38:!3
0:42:05 0 0:38:22
0:41:22 0:40:27
0:41:00
individual members resemble each other or are closely aligned when members take
equal turns to lead, follow, challenge and affirm team progress. In an aligned team,
even as members disagree with each other, there is synergy. The flow of the team is
dynamic and emergent and is not easily attributed to one member but rather to how
they interact and act in concert. In contrast, moments of tension in an erratic team
are represented by scattered bidding patterns. Here, members’ contributions are
misaligned and the shifts from affirming bids (bids that accept and build on) to
dissenting bids (bids that ignore or reject) appear abrupt and disjointed. Members
appear to be working individualistically.
Figure 11 depicts Team One’s struggle to find synergy. The team member’s use
of affirmation and dissent, passive and active, verbal and spatial are radically
different from each other, indicating a struggle to come together. At time 0:00:19,
two members are building on (level 2 on radar) while one accepts (level 1 on radar).
This is “coming together” but in the next moment, time 0:00:45, one member
rejects (level 4 on radar) while the second ignores (level 3 on radar), and the third
checks out by wondering offscreen and out of the team space (indicated by 0 on
radar).
Below we include the transcript for Team One that corresponds to these eight
time segments. Each time segment comprises one bid and two subsequent responses
that (1) accept, (2) build-on, (3) ignore or (4) reject the bid in question. Time
segments 0:00:45 and 0:03:03 show mixed responses to bids, misalignment and
disarray; while time segments 0:01:16 and 0:05:51 show team alignment: some-
what strong and very strong responses to bids.
Student Teams in Search of Design Thinking 27
The goal of the Assessing Team Learning Project was to gain insights into how
independent team collaborations were accomplished as students engaged in and
learned design thinking. The ethnocentric methods we employed allowed for deep
insights into the nature of the process. Through taking an approach to data analysis
that privileged our study participants’ views of their experiences of their groups, we
were pointed to particular events for study in videotapes of team meetings. We
started the analyses by looking at what the students reported as problematic or
extremely interesting or satisfying. Central to our analysis was the idea of a “bid”
for topic introduction, change, or redirection of focus. A bid could be verbal or
non-verbal. We also identified responses, from ignoring bids, entering competing
bids, building on bids, etc., and traced the trajectories that bids and responses took
in the group interactions. Within the larger theme of team interaction, we gained
knowledge about how bids and their responses impacted on how teams functioned
to accomplish their goals. We explored the diverse interactions that students
engaged in, how they negotiated their bids to make contributions, and the effects
of their participation moves.
6 Findings
Several findings emerged from the analyses: Team One touched on many stages of
the design thinking process [empathy, define, ideate and prototype] in working to
find its way. This might have been amplified because the students had trouble
Student Teams in Search of Design Thinking 31
7 Implications
Much work has gone into designing course pedagogies at the d.school, where this
study took place, and our team’s observations of five or six courses revealed that
courses and pedagogy were in line with the standards called for by design education
researchers (Brereton 1996; Dym et al. 1999; Dym et al. 2005; Gerber 2009). These
standards included: using problem-based learning and other appropriate
pedagogies, making the investment in instructors and group coaches, considering
it a crucial investment to educate diverse students with and into design thinking.
Attention was paid to developing projects, in class activities, reflective practices
(Adams et al. 2003), and assessments. Teams had instructors and groups have
coaches. While students were in class, they were guided to engage the design
process and mindsets. They were also guided to engage in “idealized” ways,
receiving instruction and practice on different aspects of the design process. For
example, when they covered how to write a Point of View statement in class, the
very task Team One had trouble with, they received instruction and completed
activities that included: what the POV is, why it is important, what qualities and
standards a POV should meet, and ways to check their POV in order to ensure it is
an adequate one. They learned about, and practiced writing POVs in class. Still, our
video revealed Team One floundering as it worked independently to develop a POV
statement for its user. Even when instructors work extremely hard on course design
and cover so many bases, the teaching and learning of design thinking has its
wicked aspects, such as the team collaboration.
The results raise questions and suggest implications for teaching design thinking
and the need to better support independent teamwork.
First, is important to determine how to best help teams manage the design
thinking process as they move through the different stages of a project. Finding
ways to attend to team interactions in the design thinking process may pay off in
terms of groups’ overall experiences and success in generalizing solutions.
It is important to pay attention to teams’ abilities to recognize ambiguity in the
design process. In a prior studies to this one, our research group found that students
did not become design thinkers in a developmental sequence. Instead, there were
moments of significant insight that shifted one’s understanding of the mindsets and
processes that underlie design thinking (Goldman et al. 2012).
The development and handling of “teamness” is significant and worthy of extra
attention. Teams are comprised of students with different backgrounds, disciplines,
and prior design and team experiences. These differences bring both advantages
and possibilities for radical collaborations (Booker et al. 2009; Barrick et al. 1998),
and students need pointers about how to manage, massage, and capitalize on their
differences in support in the instructional process. Students may benefit from the
introduction of varied kinds of analytics in the design thinking process such as the
creation of team rubrics and specific reflections on team process. Focusing on how
teams collaborate in design thinking might benefit from a greater emphasis on
evaluating the team process rather than just the end of course design solutions.
Student Teams in Search of Design Thinking 33
Acknowledgments We would like to thank the students who participated in our research. They
have contributed to our understanding of how instruction might better impact their learning. A
grant from the Hasso Plattner Design Thinking Research Program made this work possible.
Findings and opinions presented are those of the authors and do not represent the program.
References
Adams RS, Turns J, Atman CJ (2003) Educating effective engineering designers: the role of
reflective practice. Des Stud 24(3):275–294
Bao P, Gerber E, Gergle D, Hoffman D (2010) Momentum: getting and staying on topic during a
brainstorm. In: Proceedings of CHI 2010, ACM Press
Barrick MR, Stewart GL, Neubert MJ, Mount MK (1988) Relating member ability and personality
to work-team processes and team effectiveness. J Appl Psychol 83(3):377–391
Booker A, Goldman S, Mercier E (2009) Interdisciplinarity in learning technology. In: di Giano C,
Goldman S, Chorost M (eds) Educating learning technology designers: guiding and inspiring
creators of innovative educational tools. Routledge, New York
Brereton MF, Cannon DM, Mabogunje A, Leifer L (1996) Collaboration in design teams: how
social interaction shapes the product, analyzing design activity. In: Cross NG, Christiaans
HHCM, Dorst K (eds) Analysing design activity. Wiley, Chichester, pp 319–341
Cross N (2006) Designerly ways of knowing. Springer, London
Dewey J (1916) Democracy and education: an introduction to the philosophy of education.
Macmillan, New York
Dewey J (1922) Human nature and conflict. Holt, New York
Dym CL (1999) Learning engineering: design, languages, and experiences. J Eng Educ 88
(2):145–148
Dym CL, Agogino AM, Eris O, Frey DD, Leifer L (2005) Engineering design thinking, teaching
and learning. J Eng Des 94:103–120
Gerber E (2009) Prototyping: facing uncertainty through small wins. In: Proceedings of ICED
34 S. Goldman et al.
Goldman S, Carroll MP, Kabayadondo Z, Britos Cavagnaro L, Royalty AW, Roth B, Kwek SW,
Kim J (2012) Assessing d.learning: capturing the journey of becoming a design thinker, with.
In: Meinel C, Leifer L, Plattner H (eds) Design thinking research: measuring performance in
context. Springer, London, pp 13–33
Katu (2012) Let’s spend our lives together. Bus Life 16–22
Kress GL, Schar M (2012) Teamology – the art and science of design team formation. In:
Plattner H, Meinel C, Leifer L (eds) Design thinking research: measuring performance in
context. Springer, London, pp 189–209
Mercier E, Goldman S, Booker A (2009) Focusing on process: evidence and ideas to promote
learning though the collaborative design process. In: di Giano C, Goldman S, Chorost M (eds)
Educating learning technology designers: guiding and inspiring creators of innovative educa-
tional tools. Routledge, London/New York, pp 36–61
Pimmel R (2001) Cooperative learning instructional activities in a capstone design course. J Eng
Educ 90(3):413–421
Rittel H, Webber M (1973) Dilemmas in a general theory of planning. Policy Sci 4:155–169
[Reprinted in Cross (ed) Developments in design methodology. Wiley, pp 135–144]
Rowe PG (1987) Design thinking. MIT Press, Cambridge, MA
Schegloff EA (1998) Reply to Wetherell. Discourse Soc 9:413–416
Schegloff EA (2007) Sequence organization in interaction: a primer in conversational analysis, vol
1. Cambridge University Press, New York
von Thienen J, Noweski C, Meinel C, Lang S, Nicolai C, Bartz A (2012) What can design learn
from behavior group therapy? In: Meinel C, Leifer L, Plattner H (eds) Design thinking
research: measuring performance in context. Springer, London, pp 285–302
Vygotsky LS (1976 [1934]) Thought and language. MIT Press, Cambridge, MA
Team Cognition and Reframing Behavior:
The Impact of Team Cognition on Problem
Reframing, Team Dynamics and Design
Performance
1 Introduction
An increasing amount of work takes place in a team context, and there is a need to
understand what factors can effectively diagnose, facilitate and predict team per-
formance (Skogstad et al. 2009). Prior research has shown that certain team
dynamics indicators are strong correlates with long-term innovative performance
(Jung et al. 2012; Kress and Schar 2011; Tang 1991). There is also evidence that
these dynamics may be the result of underlying intrinsic compositional
characteristics (Wilde 2008). However, current team observation techniques are
generally time-intensive (e.g. direct observation in the field) and often substantially
asynchronous (e.g. offline video analysis) or otherwise obtrusive to team function
(Tang and Leifer 1991). Another major challenge is that the definition of team
Our pilot study was conducted with the international population of ME310 students
during the 2009–2010 academic year, including 100 students and 15 globally-
distributed teams in total (Kress et al. 2012). Due to the exploratory nature of the
study, a broad data set was collected including standard ethnographic data and
responses to four separate questionnaire-based intrinsic measures. Several
promising correlations emerged at all three levels of analysis (composition, dynam-
ics and performance); an expanded study was designed to confirm the validity of the
result across contexts by incorporating measurements from the 2 years immediately
preceding and following the year of the pilot study, or a total of 234 students and
53 teams. Due to limitations on availability of archival data, the expanded data set is
analyzed at the “local-only” level (collaboration with global partner teams is not
taken into account).
On the basis of the preliminary analysis, the Wilde Type indicator emerged as the
strongest of the four measures collected in terms of apparent statistical reliability
and correlations with team-level observations (Kress et al. 2012). Archival Wilde
Team Cognition and Reframing Behavior: The Impact of Team Cognition on. . . 37
Type data were available from the ME310 teaching team for the two academic
years preceding the pilot study (2007–2008 & 2008–2009); new data were collected
for the two academic years following the pilot study (2010–2011 & 2011–2012).
The Wilde Type data are comprised of individual responses to a 20-item online
questionnaire that are combined to yield four independent “cognitive mode” pref-
erence scores (Wilde 2008). The four modes are generally distinguished by the
“dominant functions” Thinking, Feeling, Sensing and Intuition, as initially
described by Carl Jung (Jung and Hull 1971). A summary of student and team
involvement in the study by year is provided in Table 1.
Each team receives a unique project prompt from an external client; therefore,
projects range in content and difficulty. Prompts were collected and scored on a
four-factor scale to establish initial difficulty as a basis for comparison. Scoring was
conducted by crowdsourcing consensus values for each project along each of the
four factors (ambiguity, breadth of scope, technical complexity, and overall diffi-
culty) using five-point Likert-type scales in an online survey. Each prompt was
scored a minimum of 25 times by independent respondents and the group mean is
taken as the consensus score. Adding up the four factors yields a total possible
difficulty range of 5–20; bivariate correlation tests confirm that the four difficulty
factors are strongly and significantly positively correlated, suggesting that the
additive approach is appropriate (see Table 2).
The 5–20 range suggests that the most difficult project is approximately four
times as difficult as the easiest; we refer to this parameter as the difficulty ratio (α).
The raw score (Pi) can thereby be adjusted by means of a linear multiplicative
formula with a fourfold score amplification at maximum and zero amplification at
minimum (see Eq. 1). A total of 125 respondents were recruited via Amazon
Mechanical Turk to perform the ratings on a third-party survey website. Two
fake prompts were inserted as a validity check (one easy and one difficult).
Equation 1: Project Performance Difficulty Adjustment Formula
Di DMIN DMAX
PA ¼ Pi 1 þ ð/ 1Þ /¼ (1)
DMAX DMIN DMIN
The raw performance score Pi is adjusted by the specific project difficulty Di, the
difficulty limits DMIN & DMAX, and the difficulty ratio α to obtain the final score PA.
38 G. Kress and J. Sadler
Documentation, images and other media of the final project outcomes were col-
lected and archived. Because of the extensive variation between project content and
documentation layout, prior to assessment the project materials were summarized
into a standardized format providing representative information (e.g. problem
statement, written summary, images, and design requirements) in a few pages at
most. Using these summaries, a team of two graders trained in agreement assessed
the projects in random order according to a four-factor framework as follows:
• Usability – the extent to which the proposed solution would be functionally
useful by its intended target group were it to be released as an actual product.
• Feasibility – the extent to which the proposed solution is capable of being
implemented as a real product with current technology, and that the team has
demonstrated such.
• Desirability – the extent to which the product would be considered desirable by
its target group, when considered in the real market context.
• Originality – the extent to which the product represents a unique insight or
implementation, when considered in the real market context.
Each project receives a score for each of these four factors, with scores given
according to a simple three-point rating scheme where 0 represents “Definitely
Not,” 1 represents “Arguably” and 2 represents “Definitely” (e.g. a 0 for Desirabil-
ity indicates “Definitely Not Desirable”). The narrow rating scheme is designed to
reduce the ambiguity inherent in the subjective assessment scheme. The four factors
are designed such that a team will be penalized in scoring if it fails to clearly
indicate a target user group or to demonstrate how the solution would be
implemented as a real product. The graders’ initial agreement was in excess of
75 %; after discussion they reached 94 % agreement. Final scores were computed as
the mean of their two assessments. The scores for each factor are then added to yield
the total raw assessment score (0–8 range). These raw scores are then adjusted for
difficulty according to the multiplicative formula (see Eq. 1). The resulting adjusted
performance score is on a 0–32 scale that is normalized as a percentage score.
Project assessment and difficulty rating were conducted for the 53 projects in the
data set spanning five academic years (Table 3).
In the 5-year sample, raw performance scores ranged from 1 to 8 and difficulty
scores ranged from 9 to 15. The mean difficulty score was 11.7 with a standard
deviation of 1.5 points. Though the rubric allows for a maximum-to-minimum
Team Cognition and Reframing Behavior: The Impact of Team Cognition on. . . 39
difficulty ratio of 4, the consensus ratings yielded a ratio of 1.67 suggesting that the
most difficult project was only about 67 % more difficult than the easiest project. By
both measures the variation in difficulty is fairly minimal. Applying the difficulty
adjustment formula, adjusted performance scores were obtained and ranged from
2 to 24 of the total possible 0–32 range. Normalized as a percentage, the adjusted
performances scores ranged from 6.1 to 74.8 % (see Fig. 1). There is a fairly
continuous spread of scores in the 6–50 % range, with a slight gap just above the
mean line and a large gap above the 50 % line; therefore the scores can roughly be
classified as follows: Below Average, Above Average and Top Performers. The
adjusted performance assessments were significantly positively correlated with the
final grades the teams received independently from their teaching team (r ¼ 0.557,
p ¼ 0.002, n ¼ 29). No direct relationships were observed between performance
and initial project difficulty.
Only four of the teams (or less than 8 %) score as Top Performers; these were the
only teams to exceed 50 % on the performance scale. The majority of teams
(33 teams, or > 60 %) are in the Below Average group with the remainder
(16 teams, or 50 %) scoring just above average (see Table 4). The overall average
performance score was surprisingly low at 29 % of the maximum. It is apparent that
the four top performers are outliers of the otherwise continuous and nearly-linear
distribution in adjusted performance scores. Therefore, there is an indication that
something qualitatively different is happening within the top performing teams and
so they ought to be considered independently in the analysis. We believe that this is
evidence of a “dynamic transition” to a highly functional state that only a minority
of teams appear to achieve naturally. Most teams do not undergo this transition and
therefore are limited to the lower half of the performance spectrum.
The stability of performance scores over the 5 years of the expanded study, as
well as the qualitatively continuous distribution of scores taken together (excluding
top performers) offers some support for the reliability and objectivity of the
performance-rating rubric. There was no significant change observed in mean
performance year over year, though there did appear to be an expansion in the
range of scores in the penultimate year followed by a contraction and less than
significant decrease in the final year of analysis (see Fig. 2). A t-test confirms that
the null hypothesis cannot be rejected, therefore these qualitative variations are not
significant and the performance scores are comparable across years.
None of the correlations between Wilde mode composition and team perfor-
mance that were observed in the pilot study were entirely consistent over the 5-year
span (see Table 5). This result can be interpreted in two ways: either that the
40 G. Kress and J. Sadler
correlations observed in the pilot study were merely statistical artifacts, and there-
fore are not evidenced in the broader sample; or that there were considerable year-
to-year variations in the team context that introduce too much noise to draw a
meaningful comparison. It is a fact that there were substantial changes to the course
structure, curriculum and personnel during each of these 5 years, though we had no
explicit measures of these factors and were unable to account for them in the
analysis. It is likely that these year-by-year variations led to the development of
different working environments for the teams, thereby favoring different composi-
tional characteristics. The volatility of these results to changing contexts – and the
overall low performance mean – underscores the need for an improved dynamic
model for observing, explaining and influencing team effectiveness.
The scope of the expanded study, in excess of 50 unique projects spanning 5 years,
particularly highlights the need for a standardized measurement of team function
that does not rely on after-the-fact subjective assessments of the project outcome.
Ideally, this measurement would require a minimum of time, effort and equipment
on behalf of the students and researchers (or managers or employees, or whatever
the case may be). This was the inspiration to develop a short-format, in-laboratory
exercise to capture individual and team-level reframing behaviors as a proxy for
long-term innovative performance. In total, 15 trials were conducted with the final
version of the exercise as part of a pilot evaluation and validation. Further experi-
mental trials with new product development teams are ongoing at the Luleå
Institute of Technology (Division of Innovation and Design).
The instrument is an episodic case study that requires both individual and group
decision-making in the form of choice structuring. The subject of the case study was
an “urban public share bicycle” such that subjects are asked to design a bicycle
suitable for an urban environment and shared by the general public. This topic was
42 G. Kress and J. Sadler
chosen because it struck a balance of familiarity; most subjects are familiar with
bicycles, although perhaps not in this context or in great depth. The case study
involves a short text introduction describing the scenario and presenting the subject
with 10 initial design choices. The subject is asked to rank those choices from
1 (most important) to 10 (least important). This process of choice structuring within
a case study as a measure of team decision-making has been applied in prior
research (Artiz and Walker 2009). New information is introduced in a series of
three rounds, providing a number of opportunities for measuring choice
restructuring (reframing). We theorized that the multi-level measurement of
reframing behavior would give insight into the health of the team dynamic as
well as provide evidence of underlying interpersonal intrinsic differences. A com-
plete discussion of the development and application of the STDE instrument can be
found in the previous installment in this series (Kress and Schar 2012).
Prior research has suggested that the only intrinsic compositional characteristic
with a reliably positive impact on team function is the presence of individual-level
“social sensitivity” (Woolley et al. 2010). Social sensitivity is measured by the
“Reading the Mind in the Eyes” (RME) test which assesses an individual’s ability
to interpret the emotions of others by viewing a series of photographs of strangers’
faces paired with single-word answers in a multiple choice format (see Fig. 3).
There is a correct answer for each photograph, and the resulting score is simply the
individual’s total number of correct answers (Baron-Cohen et al. 2001). Elsewhere
this test has been used as a direct measure of subject empathy.
As a known correlate of project performance and a direct moderator of interper-
sonal interactions, we theorized that individual differences in social sensitivity
would correspond with variations in reframing. For example, heightened social
sensitivity could facilitate consensus-finding or empathy with the hypothetical
end-user. However, no relationships were observed between social sensitivity and
reframing behavior at either the individual or team levels. This suggests either that
reframing behavior is not reliably influenced by underlying social sensitivity, or
that the instrument is not adequately capturing interpersonal variations in predispo-
sition toward reframing. In either case, it means that the SDTE cannot be used to
understand the means in which social sensitivity positively influences team
performance.
Fig. 3 A sample item from the “Reading the Mind in the Eyes” test
Because they are traits, we expect that they are normally distributed in the popula-
tion and change either not at all or very slowly throughout a person’s lifetime.
Because the modes are differentiated by function, we expect that they at least
partially describe the functional contribution that an individual team member
could be expected to make to the team’s progress. Therefore, correlations between
individual-level reframing and Wilde mode preferences could indicate that the
tendency to reframe is also a stable individual trait with reliable impacts on
teamwork. Unfortunately, no such relationships were observed; reframing behavior
did not appear to be reliably affected by underlying Wilde cognitive mode compo-
sitional characteristics. These results suggest either that the output of the exercise is
too noisy to discern compositional effects, or that reframing behavior is not reliably
influenced by intrinsic function.
measure of team effectiveness (via the proxy of reframing behavior) were similarly
frustrated. These experiences have led us away from the traditional observational or
manually-coded dynamics measures in favor of direct measures that provide quan-
titative data coupled with real-time computer assisted analysis. We see significant
opportunities to augment team dynamics research in particular with the increasing
availably and ubiquity of low-cost sensing systems and further development of
rapid prototyping platforms for hardware and software.
How might we facilitate better teamwork through real-time monitoring and feed-
back of team dynamics factors in the workplace? Can we create quantitative metrics
for team dynamics that are reliable and meaningful predictors of team perfor-
mance? Can we create a platform that allows even non-technical users to create
and analyze their own sensing experiments? To answer these questions we propose
the “TeamSense System” – a modular platform for precision real-time data capture,
analysis and feedback. We envision that a combination of (i) unobtrusive sensing
hardware in the collaboration environment, (ii) software analysis tools for detecting
patterns of team activity and, (iii) dynamic feedback mechanisms for behavioral
intervention, may radically accelerate the collaborative design process. The general
proposed of research approach is as follows:
In order to demonstrate the feasibility of the concept we constructed and tested two
sample prototype TeamSense units at Stanford in Spring Quarter 2012 (see Fig. 4).
These sensors aim at capturing the amount of physical activity that takes places
during teamwork. Each unit was designed to be mounted on the underside of a
team’s table in the shared workspace. It registers physical activity in real time by
46 G. Kress and J. Sadler
Fig. 4 Two prototype sensing units for mounting beneath a table, showing power connection on
one unit and wireless transceiver (connected to PC)
Fig. 5 Data collected from two different ME310 teams over the same time 6-week period shows
quantitative and qualitative differences in team activity
measurements as the basis for a model of team function that can meaningfully
connect compositional characteristics and longitudinal performance.
In conclusion, our attempts to observe consistent compositional effects in the
expanded study, as well as our attempts to develop an exercise to measure team
effectiveness, were both frustrated by the heavily context-specific nature of team-
work. This research has identified the need for improved team dynamics measures
from multiple perspectives. Our aim is to develop a means of team measurement
that is meaningful across different contexts by capturing those aspects which are
most critical for successful team function. Our work to date, as well as emerging
research in the field indicates that direct quantitative measurement of the team
dynamic is the most promising avenue for development (Jung et al. 2012; Jung
et al. 2011; Sosa 1999; Woolley et al. 2010). Though the technical barriers to this
type of measurement are rapidly falling, there remains considerable work to
understand what measurements are the most meaningful, how they are best
obtained (so as not to hinder team function) and how they can most effectively be
used to improve team performance either through intervention by a mediator or
direct feedback from an automated system.
References
1 Introduction
Fig. 1 Sampling of
drawings created in our a b
experiment, with excerpts of
participant-provided
descriptions
it gets your mind stuck in one way of thinking.” This fear of conformity was echoed
by other participants, and one went on to say that he looked for inspiration only
when “facing a creative block.”
These different strategies suggest that examples may modify the creative process
differently depending on the point in the design process at which they are presented.
This leads to the practical question: what are the tradeoffs of looking at examples
earlier or later in the design process? Furthermore, even if there is an “ideal” time to
view examples, some designers feel ubiquitous information access and their own
“thirst for knowledge” bombards them constantly with examples (Herring
et al. 2009). How does this repeated exposure to examples affect the creative
process?
This article presents the results of an online creativity experiment we conducted
on Amazon Mechanical Turk. Participants in the experiment generated drawings of
alien creatures as a creative task. The pervasive use of sketches to develop and
communicate conceptual designs in the creative fields (Suwa and Tversky 1997),
and the use of similar tasks in prior work Ward (1994) inspired the choice of the
drawing task. Focusing on drawings of alien figures makes this task readily
accessible to non-designers (see Fig. 1 for a sampling of drawings created by
participants).
Participants were randomly assigned to one of four conditions: examples early,
examples late, examples early and late, or a control condition without examples.
This study’s creativity measures were the number of uncommon and novel features
in the drawings and Likert-scale ratings by condition-blind raters. Conformity was
measured by the number of critical features (features that were directly copied from
examples).
Early and Repeated Exposure to Examples Improves Creative Work 51
2 Related Work
2.1 Examples
We use Marsh et al.’s feature-based evaluation metric, and extend their work by
examining how the example timing affects creative output. In addition, we study
the effects of repeated exposure to examples in the creative process.
Our experiment uses a task (drawing sketches of alien figures) that has previously
been employed to study creativity in a context of no prior training (Marsh and
Bower 1993; Ward 1994). Drawing tasks have also been demonstrated to be
appropriate for online experiments (Yu and Nickerson 2011).
This experiment was run on Amazon Mechanical Turk (www.mturk.com), a
web-based crowd-sourcing platform. This platform has been used for experiments
on affect and creativity (Lewis et al. 2011). Mechanical Turk workers have also
been employed to provide perception responses (Heer and Bostock 2010), objective
labels (Deng et al. 2009; Snow et al. 2008), and subjective ratings (Dow et al. 2011).
3 Experiment
Our experiment had two goals. First, we wanted to see if exposure to examples at
the start of a creative process leads to a different quality of creative output in
contrast to exposure when the creative process is underway. Second, we wanted to
investigate the role of repeated exposure to examples.
Our initial hypothesis was that exposure to examples later in the creative process
would have the same creative benefits but lower conformity than exposure at the
start. This hypothesis was motivated by Weisberg (1999), who observed that
creative failures are more often explained by the absence of relevant information
than the presence of irrelevant information.
Furthermore, the presence of one’s own ideas would inhibit the adoption of
sub-optimal ideas from late exposure to examples (mirroring the intuitions of some
designers).
In the case of repeated exposure, the activation account would predict that,
similar to showing more examples at once, this would result in greater degree of
conformity due to higher activation of features present in examples.
3.1 Participants
Fig. 2 Drawing canvas with time remaining (top right), and an option to clear the canvas (bottom
left, red)
3.2 Procedure
The experiment comprised two drawing sessions, each lasting 7 min. Participants
were asked to create as many drawings as they could during the drawing session. To
encourage this (and discourage participants from spending time perfecting only a
few drawings), the experimental platform included a clear-canvas tool but no line-
eraser tool (Fig. 2).
Each session was preceded by a condition-specific task in which partic pants
were either exposed to examples, or asked to think about the aliens they planned to
draw in the next session (Table 1).
At the start of the experiment, all participants saw a Web page with instructions
adapted from Marsh et al. (1996) to account for two drawing sessions and a break
(see hci.stanford.edu/example/aliens for actual prompts used).
54 C. Kulkarni et al.
For the Example task, participants were shown three example alien drawings for
90 s (see Fig. 4). We used drawings from (Marsh et al. 1996), p. 672. Using the
prompt of Marsh et al. (1996) (and Smith et al. 1993), participants were instructed
that examples were only shown to help them create their original creations, and that
we did not want them to copy the examples in any aspect.
For the Think task, participants were asked to “think about aliens” they planned
to draw in the next session for 90 s. In the Repeated Examples condition,
participants saw the same three examples before both drawing sessions.
After the second drawing session, participants filled out a survey that covered
demographics, artistic interest and ability and the thought-process they followed
while drawing.
Participants generated a total of 543 drawings. Each drawing was labelled with the
features it incorporated from the feature set of Marsh et al. (1996) (Appendix).
Drawings were annotated on Mechanical Turk, since the features were well-
defined.
All workers were US resident and at least 18 years of age, and were compensated
US$0.50 for the task. Workers who participated in the experiment were disallowed
from the annotation task (and vice-versa). All annotators were blind to experimental
condition.
Workers were trained using a drawing from a pilot participant (Fig. 3). Then,
each worker annotated a set of seven randomly assigned drawings. Workers also
rated how creative they found the drawing on a seven-point Likert scale (each
annotator saw at least one drawing from each condition). Lastly, annotators could
flag offensive (or non-alien) drawings. Upon review, 34 flagged drawings were
discarded by the authors. Each drawing was annotated by two workers. Disputes in
annotation were resolved by the authors (Fig. 4) (Table 2).
Early and Repeated Exposure to Examples Improves Creative Work 55
Participant’s description:Basically hominid, but with four arms and two antennae.
Antennae Nose
Homs Mouth
Eyestalks Tongue
Ears Teeth
Eyes Whiskers
Beak Head/neck
Previous Continue
Fig. 3 Training interface for annotators. The training interface shows what features to label
(“click antennae”). The actual annotation is performed on an identical list of features
4 Results
Following Smith et al. (1993), conformity was measured as the number of critical
features incorporated per drawing. Without controlling for the drawing session,
examples shown at the start of the experiment increased the number of critical
56 C. Kulkarni et al.
Fig. 4 Example drawings provided to participants. All examples contain critical features – four
legs, antennae, and a tail
Early and Repeated Exposure to Examples Improves Creative Work 57
Table 2 Table of means. Means that differed from control at p < 0:05 are bold, those marginally
significant (p < 0:1) are in italics (p-values from the post-hoc analysis using mixed models, see
Sect. 5)
Condition Critical Common Uncommon Novel Total Drawing per session Likert rating
Control 0.39 4.21 0.95 0.47 6.03 4.00 3.71
Early 0.57 3.91 1.15 0.40 6.04 3.00 4.10
Late 0.52 3.82 0.78 0.45 5.57 3.68 3.43
Repeated 0.64 4.20 1.21 0.54 6.60 3.00 4.22
features that were incorporated into drawings (t(507) ¼ 2:06, p < 0:05), consistent
with results from (Smith et al. 1993; Marsh et al. 1996). Participants in the Late
Examples condition show higher conformity in the second drawing session
(i.e. post-exposure) [t(419) ¼ 1:83, p ¼ 0:07].
The number of uncommon features per drawing increased in the Early Examples
condition (t(419) ¼ 1:61; p ¼ 0:06), and in the Repeated Examples condition
(t(419) ¼ 1:72; p < 0:05), but not in the Late Examples condition (t(419) ¼ 0:45;
p ¼ 0:649) (Fig. 5). The number of novel features did not vary significantly across
condition. Participants in the Late exposure condition created drawings with mar-
ginally fewer common features(t(419) ¼ 1:33; p ¼ 0:09) and fewer total number
of features (t(419) ¼ 1:30; p ¼ 0:09).
Annotators rated drawings in the Early Examples and the Repeated Examples
conditions higher (t ¼ 2:24; p < 0:05 and t ¼ 2:65; p < 0:01, respectively).
Intra-class correlation amongst raters (average, random raters) was 0:54 (F
(508;508) ¼ 2:2; p < 0:001).
Unlike Marsh et al. (1996), participants created fewer drawings per session in all
example conditions1 [Early: t(149) ¼ 2:50; p < 0:05; Late: t(149) ¼ 2:14,
1
Since the number of drawings is not a repeated measure, analysis uses a fixed-effects model with
interaction, the experimental condition and the type of session being independent variables.
58 C. Kulkarni et al.
2.0 -
1.5 -
1.0 -
0.5 -
2.5 -
2.0 -
1.5 -
1.0 -
0.5 -
p < 0:05, Repeated: t(149) ¼ 2:63, p < 0:05] (Fig. 6). Participants in the Late
Examples condition created fewer drawings after exposure to examples (μbe f
ore ¼ 4:10, μa f ter ¼ 3:13, t(149) ¼ 1:91, pinteraction < 0:05).
Early and Repeated Exposure to Examples Improves Creative Work 59
5 Discussion
These results suggest that exposure to examples at any time increases conformity.
However, early exposure increases the number of uncommon features and subjec-
tive ratings of creativity, while late exposure provides no such benefits. This runs
counter to both our initial hypothesis and the intuitions of many designers who
delay looking at examples in an effort to reduce fixation and think “out of the box”
(Jansson and Smith 1991).
One possible explanation for these effects is that early exposure to examples aids
the designer in understanding the scope of acceptable solutions to a problem, and
helps form an initial representation of the creative concept (Heit 1992). Prototyping
results in subsequent abstraction and refinement of the initial representation (Lim
et al. 2008). Without initial exposure to examples, the refined representation may
differ widely from the one embodied in examples, which would make it harder to
map concepts from the example to one’s own representation. When exposure is
only for a short duration (90s in our experiment), it is possible that only concepts
with high enough activation, such as critical features in our experiment, are
transferred (motivated by Boroditsky 2007).
Another counter-intuitive experimental result is that repeated exposure to the
same examples led to higher creative quality. This may also be explained by a
seeding-and-transfer account. Initial exposure to examples prevents the refined
representation formed by prototyping from diverging greatly from the one embod-
ied in the examples. This refined yet similar representation would then allow the
designer to learn different concepts on re-exposure to the same example.
In essence, the crucial ingredient that allows repeated exposure to improve
creativity might be the prototyping that occurs between exposures.
Examples play a dual role in design– first, they inspire different solutions and ways
of thinking. Second, they help form expectations about what characteristics a
solution needs to have (Herring et al. 2009). The decrease in the number of
drawings created may be due to this second role. Seeing examples may have
signaled a higher threshold for “acceptable” drawings, resulting in participants
spending more time on each drawing, and creating fewer drawings overall.
Our data suggest that this expectation-setting role has a different behavior than
the inspirational role. While the number of drawings created decreased nearly
uniformly post-exposure, changes in creativity measures (uncommon features and
subjective ratings) were non-uniform. Therefore, while examples may set
60 C. Kulkarni et al.
Fig. 7 Participant-provided
description: “An ambush
predator that does move very
much, but lures prey into its
mouth using scent to make
them think there is food
there. It only occasionally
shifts using the pads on its
bottom, which can also suck
up nutrients from the ground
or water for emergencies.”
Rated highly creative by our
raters, this drawing has no
novel or uncommon visual
features, and uses a
non-visual feature (scent)
expectations any time they are presented (including late in the design process), their
inspirational value may be time-dependent.
Results from both the feature-counting measure of creativity from prior work and
the Likert-scale ratings provided by annotators are largely consistent. While the
Likert ratings are subjective, they better capture the creativity in some drawings that
combine common or critical features in a novel way, or use a non-visual feature (for
e.g. see Fig. 7). Using both together provides a better characterization of creativity.
This work demonstrates the benefits of early and repeated exposure to examples on
creative work. In addition, it suggests that conformity may be the price one pays for
these gains, regardless of when examples are seen. Hopefully, these results will
encourage designers to seek examples early and often in the design process, when
they are most useful.
This experiment also demonstrates a replication (and extension) of creativity
studies in an online crowd-sourced environment. Crowd-sourced experiments often
offer a lower cost, have a faster time to completion, and provide access to wider
populations (Heer and Bostock 2010). This paper’s experiment and the labeling
tasks took 1 week on Amazon Mechanical Turk. This was possible because the
labeling scheme from Marsh et al. (1996) provided this study with a clear taxonomy
of features that could be easily labeled by non-experts. We suggest crowd-sourcing
Early and Repeated Exposure to Examples Improves Creative Work 61
as a viable platform both for experiments that do not need participants with
specialized skills or background (or modification per participant) and for analysis/
labeling tasks that easily verifiable.
This work also raises a number of questions. First, the results of the repeated-
exposure experimental condition indicate that the processes of prototyping and
learning from examples may be intertwined in a creative task. Further empirical
studies could characterize the precise nature of this interaction. Second, this work
shows that repeated exposure to examples is beneficial. How does the frequency of
(or interval between) such exposures affect this result? Third, designers often spend
years acquiring skills and specific domain knowledge. How do such skills and
knowledge affect their interaction with examples? Furthermore, similar to cross-
cultural effects of prototyping (Kim and Hinds 2012), are effects of examples
different in different cultures? Finally, how can the results of this work inform
the design of tools that support creative work?
Acknowledgements We thank the Hasso Plattner Design Thinking Research Program for
supporting this work.
References
Baayen R, Davidson D, Bates D (2008) Mixed-effects modeling with crossed random effects for
subjects and items. J Mem Lang 59(4):390–412
Boroditsky L (2007) Comparison and the development of knowledge. Cognition 102(1):118–128
Buxton B, Buxton W (2007) Sketching user experiences: getting the design right and the right
design. Morgan Kaufmann, Amsterdam
Deng J, Dong W, Socher R, Li L, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image
database. In: Computer vision and pattern recognition (CVPR 2009). IEEE conference on,
Miami, Florida, USA, pp 248–255
Dow S, Fortuna J, Schwartz D, Altringer B, Schwartz D, Klemmer S (2011) Prototyping dynamics:
sharing multiple designs improves exploration, group rapport, and results. In: Proceedings of
CHI: ACM conference on human factors in computing systems, pp 2807–2816
Gentner D, Colhoun J (2010) Analogical processes in human thinking and learning. Towards a
theory of thinking, Springer Berlin Heidelberg, Berlin, pp 35–48
Heer J, Bostock M (2010) Crowdsourcing graphical perception: using mechanical turk to assess
visualization design. In: Proceedings of CHI: ACM conference on human factors in computing
systems, pp 203–212
Heit E (1992) Categorization using chains of examples. Cogn Psychol 24(3):341–380
Herring S, Chang C, Krantzler J, Bailey B (2009) Getting inspired!: understanding how and why
examples are used in creative design practice. In: Proceedings of CHI: ACM conference on
human factors in computing systems, pp 87–96
Jansson D, Smith S (1991) Design fixation. Des Stud 12(1):3–11
Kerne A, Koh E, Smith S, Webb A, Dworaczyk B (2008) Combinformation: mixed-initiative
composition of image and text surrogates promotes information discovery. ACM Trans Inform
Syst (TOIS) 27(1):5
Kim H, Hinds P (2012) Harmony vs. disruption: the effect of iterative prototyping on teams
creative processes and outcomes in the west and the east. In: Proceedings ICIC: international
conference on intercultural collaboration. ACM
62 C. Kulkarni et al.
Lee B, Srivastava S, Kumar R, Brafman R, Klemmer S (2010) Designing with interactive example
galleries. In: Proceedings of CHI: ACM conference on human factors in computing systems, pp
2257–2266
Lewis S, Dontcheva M, Gerber E (2011) Affective computational priming and creativity. In:
Proceedings of CHI: ACM conference on human factors in computing systems, pp 735–744
Lim Y, Stolterman E, Tenenberg J (2008) The anatomy of prototypes: prototypes as filters,
prototypes as manifestations of design ideas. ACM Trans Comput- Hum Interact (TOCHI)
15(2):7
Marsh R, Bower G (1993) Eliciting cryptomnesia: unconscious plagiarism in a puzzle task. J Exp
Psychol Learn Mem Cogn 19(3):673
Marsh R, Landau J, Hicks J (1996) How examples may (and may not) constrain creativity. Mem
Cognit 24(5):669–680
Newman M, Landay J (2000) Sitemaps, storyboards, and specifications: a sketch of web site
design practice. In: Proceedings of DIS: ACM conference on designing interactive systems, pp
263–274
Ritchie D, Kejriwal A, Klemmer S (2011) d. tour: style-based exploration of design example
galleries. In: Proceedings of UIST: ACM symposium on user interface software and technol-
ogy, pp 165–174
Schön D (1985) The design studio: an exploration of its traditions and potentials. RIBA
Publications for RIBA Building Industry Trust, London
Smith S, Ward T, Schumacher J (1993) Constraining effects of examples in a creative generation
task. Mem Cognit 21(6):837–845
Snow R, O’Connor B, Jurafsky D, Ng A (2008) Cheap and fast – but is it good?: evaluating
non-expert annotations for natural language tasks. In: Proceedings of the conference on
empirical methods in natural language processing, pp 254–263
Suwa M, Tversky B (1997) What do architects and students perceive in their design sketches? A
protocol analysis. Des Stud 18(4):385–403
Ward T (1994) Structured imagination: the role of category structure in exemplar generation. Cogn
Psychol 27(1):1–40
Weisberg R (1999) Creativity and knowledge: a challenge to theories. In: Sternberg R
(ed) Handbook of creativity. Cambridge University Press, New York, p 226
Yu L, Nickerson J (2011) Cooks or cobblers?: crowd creativity through combination. In:
Proceedings of CHI: ACM conference on human factors in computing systems, pp 1393–1402
Part II
Design Thinkers Must Preserve Ambiguity
Impact and Sustainability of Creative
Capacity Building: The Cognitive,
Behavioral, and Neural Correlates of
Increasing Creative Capacity
Abstract The impact and sustainability of creative capacity building over time is
examined using both neural and psychological approaches. Our research proposes a
unique experimental design to test whether creativity can be acquired or learned by
an individual over time and how this relates to cognition, behavior, and the brain. In
this chapter, we review the background work that focuses on specific cognitive,
behavioral, and neural processes that may contribute to creative capacity building.
We summarize key components of our experimental design, overview its imple-
mentation, and preview early outcomes of intervention research as it relates to the
creative capacity building.
1 Introduction
In the last 5 years there’s been a seismic shift in what prized, must-have skill is
needed to navigate a world of increasing complexity. IBM’s Institute for Business
Values asked 1,500 chief executives in 2010 what leadership competency they
valued above all others to face these challenges (Kern 2010). Creativity topped their
charts. In 2011, LinkedIn reported that “creative” was the most popular character-
istic people used to describe themselves for professional profiles in the United
States (Linked in 2011).
The enormous opportunity of creativity is also the underlying challenge because
studying the creative process is not easy. As the cornerstone of innovation, creativ-
ity is an elusive human characteristic that can make one individual a better design
thinker than another. Regardless if you believe that you were born with an endless
supply of creativity or not, knowing that you can acquire it through practice has
positive implications across all industries and areas of innovation.
Although there have been several recent studies focused on where creativity
originates in the brain (Abraham et al. 2012; Aziz-Zadeh et al. 2013; Fink et al.
2012; Green et al. 2012; Jung et al. 2010a; Jung et al. 2010b; Takeuchi et al. 2012),
none have focused on this construct as an acquired individual skill, in particular,
assessing a person’s capacity to become more creative over time, measuring an
individual brain’s acquisition of creativity, and determining if regular ‘exercise’ or
practice is required to maintain it.
The Hasso Plattner Institute of Design at Stanford University (Stanford d.school)
offers classes specifically aimed at enhancing creative capacity through design
thinking skill building. As is custom in the academic tradition, instructors must
come up with ways to evaluate their student’s progression. Often, students are
asked to produce deliverables that are then judged by the instructors and classmates.
Although this method has academic value, its purely qualitative nature does not allow
a formal assessment or measurement on whether the student’s creative capacity has
been enhanced. Thus, our primary question is: Can creativity be acquired or
learned by an individual over time and if so, does being more creative change an
individual’s patterns of thoughts and behaviors and even brain functioning?
Setting out to tackle this question fueled other questions that will be addressed in
this chapter. First, we defined a contemporary working definition of creativity as it is
conceptualized by the Stanford d.school. Then we asked what human characteristics
does creativity relate to on a psychological and biological level? Specifically, what
are the cognitive, behavioral, and neural correlates of creative capacity? If creativity
can be acquired or improved, does its maintenance require continuous conditioning
(like sit-ups for your body) to be retained? What new brain and behavioral features
are acquired/enhanced by an individual exposed to a creative capacity building
instructional course? What pre-existing brain, cognitive, personality and behavioral
features predict response to a creative capacity building instructional course?
In the first part of this chapter, we review literature related to these questions as
background for studying creativity. In the second part of this chapter, we propose an
experimental design to study these questions.
Impact and Sustainability of Creative Capacity Building 67
2 Creativity
Different fields define creativity in various ways. For the purpose of this study, our
definition of creativity is inspired by the philosophy of the Stanford d.school while
insuring translation to the field of cognitive neuroscience. We define creativity as “a
state of being and adaptation of personal skill sets that enables an individual to
synthesize novel connections and express meaningful outcomes”. This definition
captures the intersection of three different axes. To determine how creative a
person, deliverable or process is, these components can be rated along three
continuums from – (a) existing to new/novel, (b) linear to synthesizing, and
(c) no value/meaning to meaningful. We propose a visual illustration of these
continuums with three axes (Fig. 1a). A deliverable or process with high novelty,
meaning, and synthesis is considered highly creative and so is the person responsi-
ble for this deliverable or process. This person, the deliverable, or the process falls
within the upper right and back zone of the three-dimensional space created by
these three axes (the zone in orange in Fig. 1b). This definition of creativity focuses
attention to the person and their skills as opposed to process and outcomes as more
traditionally defined. The intention of this focus is to better align the skill of
creativity to indications that go beyond the possession of creativity into the ability
to exercise/apply it.
The Hasso Plattner Institute of Design at Stanford University (the d.school) was
founded in the School of Engineering in 2006 to prepare a generation of innovators
to tackle the complex challenges facing the world today. Solutions won’t come
from any single field, but from collaboration between creative thinkers who can see
beyond the way the world is, to the way it could be. The d.school brings together
students and faculty from radically different backgrounds to develop innovative,
human-centered solutions to real-world challenges. Design thinking is best learned
68 G. Hawthorne et al.
Research on creativity and cognitive processes has typically assessed the relation-
ship between performance on neurocognitive tasks and the performance on
measures of creativity. Unfortunately, this work gives us little information on the
developmental nature of creativity and the factors that contribute to changes in
creativity over time. Nonetheless, surveying the literature guides us in choosing
which specific aspects of cognition, personality, and the associated neural correlates
may contribute to changes in creativity through time.
Impact and Sustainability of Creative Capacity Building 69
Our definition of creativity fits with previous research that has described crea-
tivity as a human characteristic. Specific aspects of personality and cognition have
been associated with creativity. One of the most renowned ways to characterize
different personalities is to assert a value to each of five personality traits: the Big
Five personality traits (Costa and McCrae 1992). They include openness, conscien-
tiousness, extraversion, agreeableness, and neuroticism. People who obtain higher
scores on standardized assessments of creativity, also score high on measures of
openness-to-experience (Funrnham 1999; Jung et al. 2010b; Wolfradt and Pretz
2001). Using other classifications of personality, personality traits of efficacy,
independence, cognitive control, tolerance and integrity-honesty were associated
to being more creative while emotional stability, anxiety, dominance,
aggressiveness, and leadership were associated to being less creative (Sanz de
Acedo Baquedano and Sanz de Acedo Lizarraga 2012). Given that personality
traits, specifically openness, are associated to differences in creativity between
individuals, the same traits could also influence changes in creativity over time
and are thus important to consider in our intervention study.
Similarly, factors associated to creativity at the cognitive level may influence
response to creative capacity building. Cognitive inhibition, which includes the
mental ability to focus attention while inhibiting a prepotent response, plays a role
in ideational fluency or the ability to generate ideas quickly while intelligence
impacts the originality of the ideas that are generated (Benedek et al. 2012). A
cognitive strength of creative people is their ability to focus their attention in the
most efficient way to respond to task demands indicating flexibility in adjusting
focus of attention (Ansburg and Hill 2003). Vartanian et al. (2007) showed that
creative potential is associated with an increase attention when no interference is
present. For example, creative people respond faster when asked to press a button
every time a stimulus appears on a screen and distractions are minimal. However,
Vartanian et al. also showed that creative people respond slower on attention task
where interference is present such as having to make decisions about a stimulus
based on a set of conflicting rules. An oversimplification of these findings would be
that creative people have a more flexible way of using their attentional resources.
Fluency, flexibility, and originality are included within divergent thinking abilities,
which are more related to creativity than general intelligence (Kim 2008). Thus,
research has shown that attention, inhibition, fluency, flexibility are specific
neurocognitive factors that influence creative capacity building. These factors are
included within the broad concept of executive functions. Executive functions are
higher-order cognitive processes required to organize thoughts and behavior. They
include inhibition, attention, working memory, planning, organization, and verifi-
cation. They impact general cognitive ability or general intelligence and divergent
thinking, which are also important factors to consider as correlates of creativity.
Recently, a series of neuroimaging studies have focused on the neural correlates
of creativity in the brain (Abraham et al. 2012; Aziz-Zadeh et al. 2013; Fink
et al. 2012; Green et al. 2012; Jung et al. 2010a; Takeuchi et al. 2010). Although
limited to a static view of creativity at one fixed point in time, these studies provide
a starting point for our research. In one of the first studies linking neuroanatomy and
70 G. Hawthorne et al.
connectivity within the brain (via DTI techniques), or brain activation associated
with performance on divergent thinking tasks. Our study will include these
techniques to identify neural correlates of creative capacity building.
3 Experimental Design
4 Interventions
All participants underwent both training sessions during the study, what differed
between groups was the order they received the interventions.
For this study, we created a short version of the d.school “Creative Gym” class. The
adaptation of the class engaged participants in best of breed, fast-paced immersive
exercises from each of the original course’s nine themes. In an open design studio
setting, participants worked individually on unconventional, fast-paced immersive
exercises. The hands-on activities used everyday office supplies as materials.
Participants were able to reflect on their exercises through post-activity reflections
and viewing other participants’ work. Each training session was comprised of a
diverse variety of activities that were made known to the participants as they were
assigned to them.
Impact and Sustainability of Creative Capacity Building 73
TIME POINT
AREA ASSESSED NAME OF MEASURE DESCRIPTION
T1 T2 T3
This test measures verbal knowledge,
reasoning ability, concept formaon,
General intelligence Wechsler Abbreviated Scale of Intelligence (WASI) X
visuospaal problem solving, and abstract
reasoning
This test is used to measure a variety of verbal
Execuve funcon Delis-Kaplan Execuve Funcon System (D-KEFS) and nonverbal execuve funcons including X X X
aenon, inhibion, and flexibility
6 Implications
Preliminary findings have been generated, that we invite the reader to interpret with
caution until the findings are published in a peer-reviewed scientific article. The
results comparing group performance on neurocognitive and behavioral tasks
suggest that the creativity training intervention did successfully improve some
aspects of creativity and had an impact on some components of executive functioning
(Bott et al. 2013; Kienitz et al. 2013). Changes in creativity and executive functioning
do not seem to be dependent on personality types. Preliminary findings from our
imaging tasks also indicate that the creativity training intervention was associated
with changes in the neural resources associated with executive functioning and
imagination.
7 Looking Ahead
New findings on the brain basis and sustainability of creative capacity and design
thinking skills will provide completely unique information to the field. This infor-
mation has the potential to profoundly influence our understanding of human brain
and behavior links underlying design thinking during the phenomenon of
innovation.
By understanding if a person can improve their creative capacity and whether it
is retained, metrics and method of increasing creative and instructional effective-
ness can be improved. By diving deeper into the question of individual skill
building and creativity retention over time, we can discover the true value of design
thinking during the phenomenon of innovation and know better how to teach it.
References
Carson SH, Peterson JB, Higgins DM (2005) Reliability, validity, and factor structure of the
creative achievement questionnaire. Creat Res J 17(1):37–50
Costa PT, McCrae RR (1992) Manual of the revised NEO personality inventory. Psychological
Assessment Resources, Odessa
Costa PT, McCrae RR (2010) NEO five-factor inventory-3. Psychological Assessment Resources,
Lutz
de Acedo S, Baquedano MT, de Acedo S, Lizarraga ML (2012) A correlational and predictive
study of creativity and personality of college students. Span J Psychol 15(3):1081–1088
Delis DC, Kaplan E, Kramer JH (2001) Delis–Kaplan executive function system. Pearson Educa-
tion, San Antonio
Dietrich A, Kanso R (2010) A review of EEG, ERP, and neuroimaging studies of creativity and
insight. Psychol Bull 136(5):822–848
Fink A, Koschutnig K, Benedek M, Reishofer G, Ischebeck A, Weiss EM, Ebner F (2012)
Stimulating creativity via the exposure to other people’s ideas. Hum Brain Mapp 33
(11):2603–2610
Funrnham A (1999) Personality and creativity. Percept Mot Skills 88(2):407–408
Green AE, Kraemer DJM, Fugelsang JA, Gray JR, Dunbar KN (2012) Neural correlates of
creativity in analogical reasoning. J Exp Psychol Learn Mem Cogn 38(2):264–272
Jung RE, Grazioplene R, Caprihan A, Chavez RS, Haier RJ (2010a) White matter integrity,
creativity, and psychopathology: disentangling constructs with diffusion tensor imaging.
PLoS One 5(3):e9818
Jung RE, Segall JM, Jeremy Bockholt H, Flores RA, Smith SM, Chavez RS, Haier RJ (2010b)
Neuroanatomy of creativity. Hum Brain Mapp 31(3):398–409
Kern B (2010) What chief executives really want. Bloomberg businessweek. Bloomberg L.P.,
New York
Kienitz E, Bott NT, Quintin EM, Saggar M, Hawthorne G, Royalty A, Reiss AL (April 2013)
Creativity training enhances creative persistence and increased abstract connections. Poster
presentation at the Cognitive Neuroscience Society Annual Meeting. San Francisco
Kim KH (2008) Meta-analyses of the relationship of creative achievement to both IQ and
divergent thinking test scores. J Creative Behav 42(2):106–130
Linked in (2011) http://blog.linkedin.com/2011/12/13/buzzwords-redux/?trk¼corpblog_1212
Takeuchi H, Taki Y, Sassa Y, Hashizume H, Sekiguchi A, Fukushima A, Kawashima R (2010)
White matter structures associated with creativity: evidence from diffusion tensor imaging.
Neuroimage 51(1):11–18
Takeuchi H, Taki Y, Hashizume H, Sassa Y, Nagase T, Nouchi R, Kawashima R (2012) The
association between resting functional connectivity and creativity. Cerebral Cortex 22
(12):2921–2929
Torrance EP (1974) The torrance tests of creative thinking-norms-technical manual research
edition-verbal tests, forms A and B- figural tests, forms A and B. Personnel Press, Princeton
Vartanian O, Martindale C, Kwiatkowski J (2007) Creative potential, attention, and speed of
information processing. Pers Individ Dif 43:1470–1480
Wechsler D (2011) The Wechsler abbreviated scale of intelligence, 2nd edn. Pearson Education,
San Antonio
Wolfradt U, Pretz JE (2001) Individual differences in creativity: personality, story writing, and
hobbies. Eur J Pers 15(4):297–310
Acting with Creative Confidence: Developing
a Creative Agency Assessment Tool
Abstract Universities around the world are quickly introducing new learning
models aimed at developing creativity and innovation in students. A leading
model is the experiential teaching of design thinking as a creative problem solving
process aimed at enhancing students’ creative confidence. Although these programs
exist, little is known about student outcomes. Furthermore, the criteria by which we
evaluate student “success” is not well defined because these programs almost
uniformly have ambiguously stated learning objectives. This research uses qualita-
tive and quantitative data to capture and categorizes successful outcomes by
examining alumni of these programs. Based on these data is a scale that measures
creative agency, a fundamental outcome of teaching design thinking.
1 Introduction
This research focuses on a new model of teaching creativity and innovation through
a design process, often called design thinking (Cross 2007). In particular, this paper
focuses on one of the leading institutions in this field, The Hasso Plattner Institute
of Design at Stanford University (“d.school”). Stanford University and the d.school
have garnered a reputation for producing successful entrepreneurs and innovators
according to a recent Stanford Entrepreneurship Survey, and there is widespread
interest in replicating its educational methodology. For example, universities in
Japan, Chile, Malaysia and China have founded or are in the process of creating
educational programs modeled on the d.school. Corporations are also interested in
teaching design and creativity; in Europe, Germany’s design group Palomar 5 and
PICNIC in The Netherlands have created ambitious programs inspired by the
d.school’s educational methods. While the outcomes of these endeavors have not
yet been fully evaluated, previous educational trends have demonstrated that
expensive, culturally grounded practices rarely work as expected when imported
to new international contexts.
One reason that imported educational methods tend to “fail” in new contexts is
that the expected results are often imperfectly understood and specified (e.g., “math
achievement” or “innovation”). To help those interested in the d.school and design
thinking avoid this pitfall, we first sought to describe the actual outcomes of the d.
school’s educational practices using qualitative and quantitative research methods
rather than anecdotal reports or case studies alone. Such impact questions are often
motivated by goals of program evaluation, that is, identifying what is pedagogically
effective for the sake of promoting or propagating it, and what is ineffective for the
purpose of improving it. At this level, only what works is important; why it works is
unimportant. On a more theoretical level, however, we were also interested in the
psychological mechanisms underlying whatever learning happens within an educa-
tional setting and curriculum such as the d.school. Our research, therefore, occupies
the intersection of psychological research and program evaluation: we wished to
identify the psychological constructs relevant to effective learning experiences
within a particular program (the d.school), with the goal of designing evaluations
that would reflect a common learning framework that would apply across educa-
tional settings.
Naturally, describing the d.school’s impact is an impractically large research
question. Any reasonably complex educational institution has myriad impacts and
innumerable contributing variables. Furthermore, as institutions like the d.school are
non-traditional and interdisciplinary, they often do not have specific or easily mea-
surable target outcomes for students. This is quite different from a graduate program
in Physics that teaches students to do research in one of several subfields, each with a
known set of standards and criteria with which to determine alumni achievement. At
the d.school, the only concrete outcome identified in the mission statement is the
production of “future innovators” (d.school 2004). Therefore, a first step in
investigating the impact of the d.school is selecting targeted outcomes for which
research will be appropriate and useful. One outcome of the d.school, for example,
could be graduates’ romantic relationships; however, from an educational perspective
this is of relatively minor interest. Given that the prevailing motivations for studying
and imitating the d.school stem from its apparent contributions to business and the
economy (see, for example, the Design Ladder developed by the Danish Design
Centre), we chose to focus on students’ professional outcomes after graduation.
A second step in designing research for understanding the impact of the d.school
is narrowing the field of variables we examine as characteristic of the institution.
The space of potential variables is vast, including, for example, geographical
location, student population, instructor characteristics, material resources, and
individual course curricula. Because the d.school is inherently concerned with
learning and teaching (Beckman and Joyce 2012), and because its unique pedagogy
is its most salient feature, we chose to study student outcomes related to the d.
school’s signature problem-solving approach, design thinking.
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 81
2 Background
While design thinking has been defined in different ways (Buchanan 1992), (Cross
2007), d.school courses are based on a common pedagogy that focuses on an overall
process with five core constructs: Empathy, Define, Ideate, Prototype and Test. On
a more specific instructional level, nearly all d.school courses involve techniques
such as interviewing, brainstorming, and rapid prototyping. Design thinking as it is
taught at the d.school goes beyond explicit pedagogy. The goal of the d.school is to
develop future innovators, who, in addition to performing discrete observable skills,
also have a characteristic set of attitudes and dispositions that propel them toward
creative activity and achievement.
According to a framework of a d.school teaching model (Fig. 1), mindsets and
an overall sense of creative confidence are built on top of repeated practice and
success with discrete techniques (Rauth et al. 2010) such as the design thinking
process (dt.process) and its various associated methods. These mindsets include a
bias towards action, radical collaboration, and being human centered. Other
dispositions, not specified in the model but commonly promoted at the d.school,
include constant reframing and rapid iteration.
The literature about design thinking outcomes is fairly sparse. A great amount of
design methodology research focuses on engineering or design teams and how they
perform (Eris 2004), (Brereton et al. 1996). Because the d.school does not train
designers, but rather helps students from all disciplines work in a more creative
way, the existing design research is not a sufficient base.
Kolb’s learning model (Beckman and Barry 2007) connects how the design
process helps students develop a sense of integrated thinking. This is very valuable,
but we are interested in learning how deeper behavioral aspects are affected, and a
more holistic view of how individual skills and techniques learned work together to
create the overall problem-solving-approach of design thinking. There is also
research exploring teaching models that are similar to design thinking, such as
Leifer’s (1996) work evaluating Product-Based-Learning Education, Gerber’s
examination of Design-Based Learning (Gerber 2011), and the Cambridge-MIT
Institute (Lucas and Cooper 2004; Lucas and Cooper 2005). Though such work
82 A. Royalty et al.
sheds valuable light on how the d.school teaching mechanism affects students, it
does not focus particularly on student outcomes beyond the learning experience.
In deciding on a research design, we also kept in mind that the outcomes believed to
be related to design thinking as it is taught at the d.school are based on the d.school
founders’ and instructors’ a priori intuitions and beliefs. It is quite likely that
graduates’ skills, dispositions and attitudes are not fully specified by either the
traditional design thinking model or that shown in Fig. 1. Therefore, the studies
described here exhibit a tension between our a priori research focus on graduates’
professional outcomes as they relate to the d.school pedagogy, and our desire to
remain open to discovering new variables of interest, in both outcomes/impact and
contributing factors.
We chose individual alumni as the unit of analysis versus organizations that have
d.school graduates because the d.school’s mission is that of personal behavior change
and developing individual attitudes, not redefining how companies operate. That said,
knowing how individuals who apply d.school learnings within a corporate context
does inform the impact of design thinking on a company (Gerber and Carroll 2012).
As confidence to think and act creatively came up frequently in the qualitative data,
an initial psychological construct we used to characterize this phenomenon was self-
efficacy (Bandura 1977). Self-efficacy refers to an individual’s belief in his or her
abilities within a particular domain, in this case, creative problem-solving. Another
design thinking institution has done some work on how self-efficacy may be related to
design pedagogy (Jobst et al. 2012). Self-efficacy, however, is only one part of a larger
construct, agency, which Bandura (1982) defined as the means by which “people can
effect change in themselves and their situations through their own efforts,” (p. 1175).
Agency includes not only self-efficacy, but also beliefs about the world, context,
physical and emotional states, social support, and other factors. Though it is more
complex, we felt that agency better reflected the multifarious nature of the creative
competencies that many d.school graduates exhibited. We defined creative agency as
individuals’ capacity to effect change in themselves and their situations to support
successful creative problem-solving.
Given that there is little research on the outcomes of programs that involve teaching
design thinking through problem-based learning methodologies, we decided to take a
wide-to-narrow approach starting with exploratory qualitative studies (an open-ended
survey with follow-up interviews) and an initial conceptual model based on our
observations. To test the model’s key psychological construct, agency, we did a
pilot test using a new scale, revised it based on the data, and then tested a second
version of the scale across multiple populations (study 3).
Although we reported the background, methodology and initial results for Study
1 in last year’s report (Royalty et al. 2012), for this year, we analyzed additional
data and applied a new coding scheme to several questions. The preliminary goal of
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 83
this continued work on last year’s data was to generate a foundational set of
descriptive results regarding d.school alumni, so that stakeholders, prospective
students and the larger design education community can better understand the
alumni population. The second, more research-oriented purpose of these additional
analyses was to address the following new research questions:
1. To what extent does the d.school affect alumni professional outcomes (i.e. career
choice)?
2. To what extent are alumni professional activities related to d.school pedagogy?
As this study was exploratory in nature, there were no a priori hypotheses. For
the sake of completeness, we re-report the methods in the following section, but
interested readers can go straight to the results.
3.1 Method
3.1.1 Participants
We defined a d.school alumnus as any person who had taken at least one d.school
course, where a course was defined as a quarter (ten weeks) of unit-bearing
instruction. Based on student records, the participants we surveyed were a close
approximation of the entire population (approximately 670 alumni). The response
rate was 28 % for a final sample of 184. Among those reporting gender, 56 % were
men (n ¼ 73). Nearly all participants had been graduate students at Stanford
University (some had finished their degrees and others had not) and were affiliated
with various schools and programs, including Business, Law, Arts & Sciences,
Education, Medicine and Engineering. As the d.school officially offered courses
beginning in 2006, participants had been out of school and employed full-time from
zero to a maximum of 5 years. The majority (83 %) had graduated in 2008 or after.
3.1.2 Procedure
Given the large number of potential participants in this study, for ease of data
collection and analysis, we chose a survey methodology and used online survey
software (Qualtrics). Alumni were identified through student records and contacted
via personal email as well as a general Facebook announcement (it is possible that
some people received more than one notification or invitation to participate in the
study). Invitations were sent in May 2011 and the survey was open for 2 weeks.
Participants were free to take the survey at a time and place of their convenience.
There was no explicit incentive to participate; however, respondents had the
opportunity to join a mailing list that would notify them of upcoming alumni
events.
84 A. Royalty et al.
3.1.3 Materials
All of the questions on the survey were designed specifically for this research.
Participants were asked about the current and previous occupations and occupa-
tional plans, how they applied what they learned at the d.school in their professional
lives (Table 1), and demographic questions. So as not to bias respondents towards
reporting specific techniques or general dispositions, we did not use the term
“design thinking” in any questions; instead, we referred to “what you learned at
the d.school”. The survey questions also did not offer any definition or examples of
“applying” what was learned at the d.school; respondents were free to respond
based on their own interpretation of this term.
3.2 Results
Table 2 Categories and frequencies of career plans and outcomes for d.school alumni
Percent of total Mention d. Percent of category
Category Frequency (%) school mentioning d.school (%)
Career change 76 41.50 26 34.20
No career change 46 25.10 6 13.00
No original plan 59 32.20 3 5.10
No current job given 2 1.10 0 –
Total 183 100.00 35 19.10
3.3 Discussion
Re-analyzing the survey data with a particular focus on career changes and how
alumni viewed the contribution of d.school pedagogy revealed moderate evidence
for the claim that educational methods at the d.school have an impact on graduates’
professional outcomes. Across all participants, nearly twenty percent brought up
the d.school as a factor in their professional paths and current occupation. It is
important to note that these participants mentioned the d.school’s influence without
prompting – the question about why they were working in their current occupations
did not mention the d.school, and was placed before (and on a separate page from)
the question that specifically asked about the role of the d.school. Furthermore, the
results show that alumni perceive their d.school experience as a causally
contributing factor in their occupational choices.
As participants varied in what they found most significant about their experience
at the d.school, we cannot conclude that it was, in fact, d.school pedagogy rather
than other factors that drove these outcomes. The data on alumni use of
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 87
As Study 2 was described in the previous report (2011), we will just briefly review
the methodology and major findings in order to describe the larger research arc.
Sixteen d.school alumni (ten women) from Study 1 were interviewed in person or
via teleconference (Skype). Seven participants currently worked in business
(including healthcare, technology and entertainment), three were self-employed
or entrepreneurs, three worked in education, two in consulting or research, and one
was in engineering. The interviews were semi-structured and were 45–100 min.
88 A. Royalty et al.
The interviews probed how “successful” alumni viewed what they had learned at
the d.school and how that may have influenced both their career choices and their
professional activities. By “successful”, we merely meant those who said that they
chose to, and were able to, use what they learned at the d.school (which they defined
for themselves) in their working life. On a high level, we wanted to see how former
d.school students described their learning path and their subsequent behaviors in
order to better understand how the d.school may be contributing the formation of
“future innovators”. More specifically, we focused on their observable behaviors
and dispositions, and how alumni perceived the d.school as shaping these
outcomes.
The overall result of the interviews was a better understanding of how some d.
school alumni develop and demonstrate creative confidence. Beyond simply
learning basic design thinking processes and techniques, participants who reported
using what they learned at the d.school had a strong desire and confidence to
actually apply skills and methods in their workplace. This can be a daunting and
risky endeavor in some traditional work contexts, where normal or known
behaviors are expected and job responsibilities are tightly codified. In the strongest
case – seen in several interviewees – creative confidence in the professional realm
even extended to using the design thinking process to launch or change one’s entire
career. For example, one participant left her job in engineering to pursue teaching,
and explicitly stated that it was what she learned at the d.school that gave her the
confidence to “try out” or prototype a new career, even though it meant taking
significant professional risks.
Another, unexpected example of how alumni demonstrated creative confidence
that came up in multiple interviews was how they built creative environments
around themselves at work and in their personal lives. This often took the form of
manipulating physical space to allow for more creative working styles, and in
some cases, making the space similar to the interior of the d.school. Some
participants also reported teaching colleagues about design thinking as a way to
enable better collaborations and facilitate their own creativity. Across most of the
alumni we interviewed, the rich descriptions of how they used what they learned
at the d.school went far beyond simple applications of basic techniques or tools
such as brainstorming or rapid prototyping. Instead, their stories reflected an
overall approach to work and life that was informed by dispositions to solve
problems in an innovative manner, and propelled by the confidence to do so in
multiple ways.
Successful alumni seemed to share a set of behavioral patterns and dispositions
that went beyond what we had captured in Study 1. Although we could observe
those characteristics in interviews, we wanted to define a single psychological
construct that could be measured accurately and efficiently. In Study 3, we
attempted to narrow and quantify what was observed in the survey and
interviews, by creating a scale that attempted to measure the new construct of
creative agency.
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 89
5.1 Method
1. Current d.school students (General). The scale was implemented in the Autumn
quarter of 2011 with current d.school students drawn from two classes. The first
was a large, introduction to design methodology course (n ¼ 72) (95 % of class)
and the second was a more specialized course exploring the intersection of
design and society (n ¼ 13) (76 % of class). The questionnaire was given at
the beginning (September) and end (December) of the academic quarter and no
incentives were offered for participation. Of the 85 pretest participants,
68 completed the posttest.
2. Current d.school students (Education). Thirteen students in a small, Education-
focused d.school course took the questionnaire as a pretest and posttest at the
beginning and end of the Spring 2012 academic quarter. The pretest was given at
the beginning of (April) and the end (June) of the Spring quarter; eight students
took the posttest. A $15 gift card was offered for participating.
90 A. Royalty et al.
5.1.2 Materials
There were 11 to 14 Likert-scale questions on the CBCA scale; all were created for
this study. Eleven items concerned self-assessed confidence related to a set of
general competencies in the area of creative problem-solving (Table 6). The
CBCA-related Likert-scale items were presented as a group and preceded by the
question, “How confident are you that you could. . .”. Unlike the items in Study
3, the new questions were written to be sufficiently general so as to apply to nearly
any situation or professional domain. There were five response categories: “Not at
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 91
50
Women
(n=33)
35
30
Pre Post
Time
all confident” (scale point 1), “A little confident” (2), “Moderately confident” (3),
“Very confident” (4), and “Completely confident” (5). There were no “Don’t
Know” or “Not Applicable” response choices; however, participants were free to
leave any item blank.
After implementing the 11-item questionnaire with the first group of partici-
pants, all of whom were d.school students, we added three new items on topics
relatively unrelated to competency-based creative agency. These items concerned
comfort with technology, artistic ability, and goal achievement. They were added
to provide data on whether any apparent outcome of d.school study was specific
to the design thinking competencies we found in earlier studies, or a more general
positive impact.
5.2 Results
While there was no overall effect of gender, there was some evidence of a trend for
gender to interact with time, such that women started out with higher CBCA but
ended at a similar mean score, F(1, 53) ¼ 3.04, p ¼ .087 (Fig. 2). The effect size
for this possible interaction was, however, quite small, with a partial eta-squared of
.05.
There were too few participants in the non-d.school student “control” group for
statistical analysis; however, among the eight who took both pretest and posttest,
there was no apparent change in mean CBCA. At pretest, the control group mean
was 35.25 (SD ¼ 5.31); at posttest, the mean was 36.12 (SD ¼ 3.68). Figure 3 plots
the means and standard errors for the d.school students versus non-school students
at pretest and posttest.
92 A. Royalty et al.
48
Competency-based Creative
46
50
40
30
20
10
d.school students teachers/ executives non-d.school students
(n=55) (n=23) (n=8)
Pre Post
5.3 Discussion
Study 3 tested a new scale, competency-based creative agency, with three main
samples: d.school students, similar students not taking d.school classes, and
teachers and executives who took a 3-day design thinking workshop. As the scale
items were based on empirical research on the applied skills of successful creative
problem-solvers who were d.school alumni, it was hypothesized that d.school
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 93
experience would be associated with greater CBCA (see Fig. 4). The results
provided evidence for this relationship, as d.school students increased in CBCA
from the beginning to the end of the course, across two different academic quarters
and in three different d.school classes. In one of those quarters, a small group of
non-d.school students also took the scale; they did not show any significant change
in CBCA. While a larger control group would, of course, provide better support for
the hypothesis, these data do suggest that the increase in CBCA seen among d.
school students was not merely due to maturation or demand characteristics of the
scale items.
Results from the teacher/executive sample also supported the hypothesis that d.
school experience increases CBCA. Although the design thinking workshop was
only 3 days, participants reported a significant increase in CBCA from the begin-
ning to the end. Data from this sample also showed that the scale could be used with
a non-students sample and retain its internal validity as well as its usefulness in
showing change over time.
It is interesting to note that the average pre-test score of the teachers and
executives was slightly lower than the average pre-test score of the non-d.school
students, which in turn was marginally lower than the average pre-test score for d.
school students. There was, however, an apparent difference between the mean
starting CBCA for d.school students and teachers/executives, with d.school
students reporting higher CBCA at pretest. Taking a design thinking course
shows willingness to take risks by studying a topic that is relatively new and not
a regular part of any degree program; a higher pretest score among these students,
then, is not surprising.
6 Implications
The three studies reported here have led us toward a larger model of how individual
creativity develops and is subsequently expressed. One of the original concerns of
our research was to investigate how the d.school and other educational institutions
can foster the kinds of skills that lead to real-world innovations. Our initial model of
creative outcomes, as observed in our research, includes four constructs: self-
efficacy, agency, output and impact (Fig. 5).
Creative self-efficacy is belief in one’s own creative abilities, and it is developed
in many ways, such as mastery experiences (successfully completing creative
activities) and vicarious experiences (learning from others who are creative).
Self-efficacy is the most important part of creative agency, which goes beyond
mere belief in oneself and extends to real-life application of creative problem-
solving. Those who have high creative agency are capable of producing creative
output – real work, artifacts, and ideas. Creative output determines creative impact,
this is the change made due to a creative effort.
94 A. Royalty et al.
Creative Outcomes
Creative Self- Creative Agency: Creative Output: Creative Impact:
Efficacy: Applying one’s Manifestations of Effects of creative
Belief in one’s creative abilities applying creative actions
creative abilities ability
Internal External
7 Work in Progress
We are currently in the early pilot stages of three studies aimed at the creative
output node of our model, that is, the ideas, work, artifacts, procedures and other
“deliverables” that creative agents make.
problem-solving contexts (self-efficacy), and that this propels them to act to change
their own thinking and their environments in order to perform better as creative
problem-solvers (creative agency). These actions are characterized by multiple
competencies, first learned and practiced at the d.school, and then carried into the
workplace. These include beliefs (anti-perfectionism, failure as opportunity),
knowledge (of one’s creative process), dispositions (towards prototyping, radical
collaboration), and skills (teaching others, shaping one’s own environment). We
also observed that these capacities were largely domain-agnostic, and were reported
by teachers, engineers, entrepreneurs, and even chefs in equal measures.
In our quantitative research, we moved towards an efficient and accurate way to
represent and measure the outcome we observed among the alumni, by designing
and testing two scales. While the first did not function well as a singular-construct
scale, we did see interesting results, such as, more experienced d.school students
said that they were more comfortable starting a company than students with less d.
school experience. Our second scale, which built upon the lessons of the first, was
more successful at providing a valid and reliable measure of the impact of d.school
pedagogy. By testing it with several populations, we found initial support for its use
beyond the d.school. In the near future, we hope to repeat our studies with larger
and more robust control groups, and implement the scale in other educational
programs and institutions, both similar to and different from the d.school.
With more evidence for the validity and reliability of the Competency-based
Creative Agency scale, we would like to eventually expand the research to its
potential predictive validity, its neurological correlates, and ways to use it in
controlled experiments. For example, using the CBCA in the workplace may
yield insights into how workers with high creative confidence are recognized for
their creativity and the benefit they provide to their companies (predictive validity).
We are also collaborating with another research team on fMRI studies that incor-
porate the CBCA scale. Finally, to truly test the effects of the d.school pedagogy,
we would need to run experimental studies that manipulate what we believe are key
aspects of design thinking as it is learned at the d.school, and then measure how
these components affect CBCA. This will take us closer to understanding the
contribution of the d.school to educational methods, as well as how the successes
of the d.school may be not only repeated, but also enhanced, at other organizations
throughout the world.
References
Bandura A (1977) Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev
84(2):191–215
Bandura A (1982) Self-efficacy mechanism in human agency. Am Psychol 37(2):122–147
Beckman S, Barry M (2007) Innovation as a learning process: embedding design thinking. Calif
Manage Rev 50(1):25–56
Beckman S, Joyce CK (2012) Reflections on teaching design thinking to MBA students. In:
Business as an agent of world benefit conference, 2–5 June 2009
96 A. Royalty et al.
Consider mathematical problems. They may be stated once and forever. Or think
of industrial requests. These may be quite rigid, e.g., how to make a car five hp
stronger. Wicked problems, however, are so multi-faceted that many different
problem views are viable. Imagine, for instance, a case from psychology: There
is an unhappy family. The husband is mostly off at work. The wife is overstrained
by her job, the housework and restless kids. The children lack joy and warmth at
home. What is the problem? Many answers are conceivable.
“2. Wicked problems have no stopping rule” (p. 162).
When facing a mathematical problem, there may be a point when the solution is
found – or when it becomes clear that there is no solution at all. An industrial
request for making a car 5 hp stronger is fulfilled when the car is 5 hp stronger. But
at what point should an unhappy family stop trying to change things for the better?
It is often a matter of gut feeling when to give up hope completely or when to make
peace with the situation reached.
“3. Solutions to wicked problems are not true-or-false, but good-or-bad” (p. 162).
A mathematical solution is right or wrong. But how about the unhappy family?
Has the problem been “solved correctly” once the husband has given up his job,
became a stay-at-home dad and now tries to unburden his wife? What if the kids
find gratification in hobbies away from home and rarely expect anything of their
parents any more? What if the couple splits up so that their ceaseless quarrels end?
Evidently, there are many ways to counter the unhappiness-problem. And there
may well be approaches which are not exactly false, but rather bad solutions.
Thus far, it may be compelling to say that design thinkers tackle wicked
problems as described by Rittel and Webber (1973). Yet, the ten criteria they
formulated were spelled out for the field of politics. And design thinking problems
are not like political problems in all respects. Consider, for instance, the following
criteria formulated by Rittel and Webber.
“5. Every solution to a wicked problem is a ‘one-shot operation’; because there is no opportu-
nity to learn by trial-and-error, every attempt counts significantly” (p. 163).
This may be said about a government which decides over war or peace. But why
should the unhappy family not try out something new? How about a role change for
a week? How about new formats of work-sharing? How about involving the
grandparents?
“10. The planner has no right to be wrong” (p. 166).
Therefore, not all ten criteria provided by Rittel and Webber (1973) seem to be
equally pertinent to the problems found in design thinking. We shall limit ourselves
to discussing three characteristics, still obviously assimilating Rittel and Webber’s
ideas.
With the condensed three criteria in mind, let’s ask a simple question: What do
the problems common in design thinking actually demand of the people or teams
who tackle them? And how can design thinking be of help in that regard?
Consider the unhappy family. If you ask each family member what the problem is
you may hear quite diverse answers. The children may find their parents “less cool”
than their friend’s parents. The husband may complain that his efforts to earn
money for the family are not valued enough by his wife. The wife may feel she is
left alone with all responsibilities and duties; she needs more support. And the kids
are a little too loud and too messy.
Interestingly, these problem views need not be stable. Because there is no correct
or false framing of the problem, people may change their views as they encounter
new perspectives. For example, one friend of the wife might convince her that it is
all her husband’s fault. Isn’t he egoistical to head for self-fulfilment at work while
she has no choice but to struggle with laundry baskets and cleaning rags? Another
friend might convince her that the problem lies with her and not with the husband.
Why does she feel responsible for everything? She should involve the kids more
frequently and take up her old hobbies.
So, a design team working on problems of this kind needs to:
Create a stable problem view!
This is very much in line with one of the famous d.mindsets. “Craft clarity[!]
Produce a coherent vision out of messy problems. Frame it in a way to inspire others
and to fuel ideation” (Hasso Plattner Institute of Design at Stanford 2011).
Given that problem views of stakeholders need not be coherent and may actually
be quite volatile, it is an important job to reach a stable interpretation of the
problem. Otherwise, every solution you prepare may be considered promising by
the stakeholders in one moment and may be discarded in the very next. As problem
views change, people also change their understanding of what a solution should
achieve. Thus, they introduce new criteria for evaluating your proposals.
Chances for establishing a stable problem view obviously increase if you:
Integrate differing perspectives!
If your problem view ignores the perspective of a central stakeholder, he may
rebel against it. Imagine designing something delightful for the kids of the unhappy
family and then having the parents consider it an impertinence. The end result: they
will simply ban it.
100 J. von Thienen et al.
Once again, consider the unhappy family. Imagine if a therapist would appear and
act as the ultimate connoisseur of human nature. According to his analysis, the
husband just needs to exert more authority at home. Under his instruction the kids
should help their mother more often so that she will be unburdened and therefore
able to appreciate her husband again.
The family may try, but what if happiness and love fail to materialize? Maybe
the husband doesn’t enjoy ordering others around. Maybe the wife is quite an
emancipated woman who sets value on equal roles. Maybe the kids long for parents
who pay more attention to their concerns. Not even a professional like a therapist
can claim to possess an ultimate connection to “the truth”. Nothing more and
nothing less can be done than exploring people’s needs and inventing strategies
to meet them.
How Design Thinking Tools Help To Solve Wicked Problems 101
Imagine you want to help the unhappy family. Congratulations if you don’t impose
your “solution” on them right after shaking hands for the first time. A solution to
their problem will emerge only after exploring carefully what world they live in and
what they need there (defer judement).
But if you don’t have a solution ready in your mind how can you and the family
be optimistic that a good idea will pop up in a couple of days, weeks or months?
After all, the family may have tried for years to help themselves. . . without success.
Surely, individual tools may be of value which support a central step in the
problem solving process.
Merge needs and insights!
Remember, all you can do regarding wicked problems is to embrace needs. And
a fresh outlook seems essential since the old problem views were obviously
fruitless. So, you also need new insights. Design thinking has powerful tools to
ease the job.
For example, a point of view madlib or how-might-we questions push towards
generative problem statements and thereby propel towards gratifying solutions.
But there ought to be a path leading towards needs and insights. And there ought
to be a path leading from ideas to ready and tested solutions. The process of
problem solving may involve many steps, so it would be helpful to have at one’s
disposal a reliable strategy or a plan.
Yet, it is also clear what this strategy needs to avoid. Obviously it would be a bad
idea to start formalizing the problem in order to deduce “the optimal solution”. We
are not dealing with mathematics here. There are no right or wrong or objectively
optimal solutions. And every formalization would be arbitrary.
Instead, the process needs to be flexible enough to basically handle all kinds of
topics, leaving room to explore and articulate needs no one may have thought of
before.
102 J. von Thienen et al.
Thus, there are a couple of challenges to be met. As a design thinker you will
need means or heuristics to:
Structure without formalizing!
Focus without formalizing!
Explore widely and stop wisely!
Design thinking offers loads of means to channel the process of problem solving
in a productive direction without forestalling any concrete decisions.
The design thinking process model provides orientation without dictating con-
crete steps.
A game-plan allows teams to structure their approach – but not too rigidly.
Time constraints and a gong provide pragmatical stopping criteria that push the
process forwards.
Maxims like encourage wild ideas or saturate white boards, tools such as
watching out for extreme users, applying powers of ten or brainstorming and
setup features such as multidisciplinary design teams help to explore widely.
Maxims such as bias for action or visualize and tools like prototyping or voting
after brainstorming help to focus without formalizing.
Iteration helps to circle in on promising ideas.
All in all, design thinking offers ample help to solve wicked problems of a liberal
type where you may fail and experiment first to become all the more successful later
on. It does so by establishing mindsets and offering tools which save you from
the impossible task of finding “the correct problem view” or “the optimal solution”.
Instead, attention is drawn to needs which await their fulfillment. New inter-
pretations of the problem are advanced which take into account the perspectives
of different stakeholders and which help to look at the matter from a new angle,
since the old problem views turned out to be blind alleys. Finally, a lot of tools
are provided to drive the process of problem solving in a productive direction –
making sure the process remains flexible, spirited and unrestrained by arbitrary
formalizations.
References
1 Introduction
The core of design is problem solving. Most of the world’s really important
problems which must be solved are wicked ones. Because wicked problems involve
several stakeholders, with different interests this makes it difficult to solve them
(Rittel and Webber 1973). That is the reason why wicked problems, as for example
child poverty, lack of resources, and urban delinquency, are often old problems,
which have not been solved until now. There is a need to develop human-centered
and holistic solutions to soften the consequences of a globalised world with
increased complexity of a radicalized modernity for further generations (Giddens
1996) where the number of wicked problems will grow. One approach to develop
the needed innovation is the education in innovative thinking and human-centered
development of ideas and the solving of wicked problems (wicked problem solving
competence). As one possible way to solve wicked problems some authors mention
design thinking as a wicked problem solving technique (Buchanan 1992; Lindberg
B. Jobst (*)
Hasso Plattner Institute, Potsdam, Germany
e-mail: [email protected]
C. Meinel
Internet Technologies and -Systems, Hasso Plattner Institute for Software Systems
Engineering, Campus Griebnitzsee, Postdam 14482, Germany
e-mail: [email protected]
et al. 2010) Buchanan argues that design is fundamentally concerned with the
particular and there is no science of the particular.
We suppose that wicked problem solving competence could be enhanced via
good prototyping skills (Wiese and Wiese 2012) and by wicked problem self-
efficacy, both are important conditions for the development of innovative ideas.
What could effect a possible improvement of wicked problem solving compe-
tence within the setting of the schools of design thinking? This and related
questions are matters that current research has not yet widely examined. We
suppose that one way to foster wicked problem solving competence is via an
enhancement of prototyping skills and wicked problem self-efficacy, which is
addressed by schools of design thinking. In this contribution, we discuss how
factors as wicked problem self-efficacy, wicked problem solving competence and
prototyping skills interact with each other.
2 The Model
In the following, we will present the three factors of the model -wicked problem
self-efficacy, prototyping skills and wicked problem solving competence- and how
these factors interact.
How Prototyping Helps to Solve Wicked Problems 107
Rittel and Webber formulate ten criteria to differentiate and define the nature of
wicked problems. To deepen the understanding for wicked problems we will
present three criteria.
“Coexistence of problem and solution” (Rittel and Webber 1973).
Rittel and Webber mention that the problem changes while working on the
project and while bringing the problem into question. All imaginable expertise
for manifold implementation and scenarios must be checked to generate
argumentative-based variance. Scenarios must be developed and checked against
each other until one can be considered as the most viable, sustainable and superior
one (from a users perspective). To check the internalized assumption of a scenario,
prototypes will be built. These prototypes will create resonance and feedback while
being tested by users and this contributes to the further working process. For the
sake of convenience we labeled the ability to deal with wicked problems and to
generate fitting and innovative solutions that contribute something characteristic to
the world (from a user perspective). We call it wicked problem solving competence.
The Schools of Design Thinking in Stanford and Potsdam offer a process and a
toolbox for methods and techniques (see in this book, previous article v. Thienen
et al. 2012/13).
One is, for example, the reframing, the modification of a problem representation
after generating new information and insights. Rittel and Webber stress that:
“The information needed to understand the problem depends upon one’s idea for solving
it. This is to say: in order to describe a wicked problem in sufficient detail, one has to
develop an exhaustive inventory for all conceivable solutions ahead of time.” (Rittel and
Webber).
The ability to reframe a problem representation several times during the process
is central for the solution of wicked problems. A further example for the research
phase during the design thinking process is to interview users to gain insights and
108 B. Jobst and C. Meinel
empathy for their needs. The acceptance of the user and stakeholder is also
considered important in order to judge if a solution is better or worse.
“Solutions to wicked problems are not true-or-false, but better or worse.” (Rittel and
Webber).
Prototypes have two characteristic roles in the design process, on the one hand
prototypes enable interactions and feedback with the user, client or teammate and
make the generation of new information and insights possible. Via a prototype a
user can give feedback on the general idea, on a detail or a function and also on
whether the solution is a “better or worse” one. On the other hand prototyping is a
generative tool for designers that will accompany the draft and design process
(Gerber and Carroll 2012). At the same time a designer makes progress in the
process, the prototypes reflect the progress of the knowledge and assumptions.
According to Rittel and Webber, there is not only one definitive solution. There
should be manifold solution proposals more or less qualitatively developed where
one is better than the other (Rittel and Webber). The better solution is one that is able
“to improve some characteristic of the world where people live.” (Rittel and Webber)
Given that there is no ultimative test for a solution, the only possibility is a
confrontation of the user with a prototype and to test the proposed solution and to
analyse the resonance and feedback towards the prototype and the idea behind
it. Once the prototype, the service or the product is considered by users and
stakeholders as desirable, you can assume to have found one of the “better
solutions.” While prototyping, the comprehension of the problem and the solution
deepens and you can speak of the co-evolution of problem–solution (Dorst and
Cross 2001). This leads to an iterative reframing of the problem during the process.
The reframing process is closely linked to prototyping because the new frame of
the problem is only verifiable and testable via a prototype. The created new framing
for the problem will be manifested in a prototype and the “included assumption”
can be tested by user. This is the best way to generate feedback when a good
solution for a wicked problem is developed.
Furthermore, every prototype is in a way a visible reminder of a current or older
assumption and labels the station within the process. With a collection of various
prototypes, the advance in the process is seen and that there is movement even in
moments of being stuck. The visual representations of working steps helps to gain
confidence in the progress being made and is a type of visual feedback (Gerber and
Carroll 2012).
What further factors do we have to include in the solving of complex problems?
In cognitive-psychology there are relevant factors for prototyping.
Based on Dörner 1995, the human memory is a medium where all psychological
processes of a human being occur. The biggest impact for solving design problems
is the working memory, because the working memory connects information from
the long-term memory with the short-term memory.
Solving complex problems exclusively in the mind is possible only to a certain
extent. This is, because of the limitations of the working memory. For thinking,
110 B. Jobst and C. Meinel
user and how to learn more about their problems and their needs. More advanced
wicked problem solvers don’t have a fear of failure and take risks. We label a
behavior that leads to a good solution for a wicked problem as wicked problem
solving competence.
When the right prototype was chosen, the adequate technique was to incorporate the
idea and to test it with users. When the user “understands” the question that the
prototype asks, the prototyper has a feeling of success, even if the user doesn’t like
the outcome. Another positive experience of prototyping is when a prototype is
tested by a user or stakeholder and the resonance makes it possible to decide that a
solution fits a user’s vision and is a good solution for the wicked problem. This
experience leads to an enhanced wicked problem self-efficacy.
The successful prototyper receives confidence and a feeling of control in a
highly ambiguous process. According to Bandura, experiences of success lead to
mastery experience which lead to the fact that more resources will be activated to
achieve a goal, so even if there are setbacks more perseverance is executed.
112 B. Jobst and C. Meinel
“Individuals are more likely to put forth greater effort and commitment to a given task than
they are to self-doubts (Bandura 1997). ”
The user acceptance of a prototype and the included solution proposal creates
positive experiences and an enhanced self-efficacy. Students realize that their
prototyping skills are a reliable and good tool to realize and recognize complex
solutions for wicked problems.
“Perceptions of ability to control are developed through mastery experiences
(Bandura 1997).”
Prototyping seems to be important for the self-efficacy of the problem solver and
the ability to develop manifold and assumption-adequate prototypes is addressed by
the schools of design thinking toolbox.
6 Outlook
References
Bandura A (1997) Self-efficacy: the exercise of control. W.H. Freeman, New York
Buchanan R (1992) Wicked problems in design thinking. Design issues 8(2):5–21
Dörner D (1995) Problemlösen und Gedächtnis. In: Gedächtnis D (ed) Probleme-Trends-
Perspektiven. Hogrefe, Göttingen/Bern/Toronto/Seattle, pp 295–320
Dorst K, Cross N (2001) Creativity in the design process: co-evolution of problem–solution.
Design Stud 22:425–437
Ehrlenspiel K (1993) Denkfehler bei der Maschinenkonstruktion. In: Ja, mach nur einen Plan!
Huber, Bern, pp 196–207
Gerber E, Carroll M (2012) The psychological experience of prototyping. Design Stud 33(1):
64–84
Giddens A (1996) Konsequenzen der Moderne, 7th edn. Suhrkamp Verlag
Klauer KC, Hrsg. B. Strauß und Kleinmann M Grundlagen der Problemlöseforschung. In:
Computersimulierte Szenarien in der Personalarbeit. Verlag für Angewandte Psychologie,
Göttingen, pp 17–42
Lindberg T, Raja G, Birgit J, Christoph M (2010) Is there a need for a design thinking process? In:
Proceedings of design thinking research symposium 8 (Design 2010), Sydney
Rittel HWJ, Webber MM (1973) Dilemmas in a general theory of planning. Policy Sci 4(2):
155–169
Wiese E, Wiese L (2012) Designproblemlösen mit externen Repräsentationen. In: Prototyping!
Physical, virtual, hybrid, smart. Form + Zweck Verlag, Berlin, pp 160–185
Creative Collaboration in Real World
Settings
Abstract Tele-Board was designed and implemented for use in Design Thinking
teams. From the development of simple prototypes, we came up with a reliable
software system for a wide range of users. There is a growing interest in the system
by companies who seek to ease their day-to-day collaboration activities worldwide.
In this article, we present the results of a study, which we conducted with a
company and its implications on the development and adjustments on the system.
With the help of usage data, interviews, and observations, we could study the
requirements of a global company. It shows how important the flexibility of our
system is for a multitude of use cases. Different levels of knowledge, room
configurations, as well as different hardware configurations are possible and well
supported. The feedback and statistics we got from the study are evidence of a large
level of acceptance and successful adoption by the users. We reflect and abstract the
lessons we learned from that specific user group into future developments and
research opportunities.
Though we learned a lot through the studies with student teams and use in
different contexts, we now deployed Tele-Board for a longer time period and also in
an industry setting. In this setup, we could make use of Tele-Board’s advancements
in former research and had the chance to equip industry partners with the system.
This way, we conducted long-term research in a real working environment of a
digital whiteboard system that has not been possible before. In the most compre-
hensive study so far, we analyzed the system log of a 3 months usage period of a
team working in a large IT company. The team members are located in Germany,
Italy and India. In addition to the system log data, we also interviewed the users
about their experience with Tele-Board and how it had changed their collaboration
and communication behavior.
Here, we will now give a detailed description of the study and the insights we
gained. We will show which functions and properties of the system foster remote
collaboration and what could be adjusted and optimized for users in a large
company.
2 Introduction to Tele-Board
In order to understand the design and setup of Tele-Board and how it was used
during the study, we will first give a short introduction of the Tele-Board system
and how it has developed over the last years. For more details on its architecture,
please see our former publications (Gericke et al. 2012; Gumienny et al. 2011).
Tele-Board is a digital whiteboard system supporting creative teamwork, such as
design thinking, especially if team members are distributed over different locations.
All users share a digital whiteboard surface, where they can draw, create and
rearrange sticky notes, change their color, or group them in clusters that can also
be moved. All actions are synchronized to each connected whiteboard enabling
users to work on the same content simultaneously. Furthermore, users’ actions are
automatically stored while working on the whiteboard, which allows recreation of a
whiteboard’s state from any point during its course of development.
The Tele-Board software system consists of several components. These are: the
web portal, whiteboard client, digital sticky note pads, and server component,
which run on a range of hardware devices (for an example see Fig. 1).
The web portal acts as the starting point for the Tele-Board software system. Here,
users can manage projects and panels in order to organize their work. Within a
Creative Collaboration in Real World Settings 117
Fig. 1 The Tele-Board software system architecture. All connected devices are synchronized via
the central collaboration server. The whiteboard client software can be run on a wide range of
hardware devices, which connect to the server component. Sticky notes can be created with the
help of mobile devices acting as digital sticky note pads (e.g. tablets or smartphones) or in the web
portal
project, users can create various panels, which represent virtual whiteboard spaces.
Editing of a panel is facilitated by the whiteboard client, which is started directly
from the web portal. This way, installation of any additional software is not
necessary for using Tele-Board.
As a digital equivalent for paper sticky notes, different applications for writing
sticky notes on mobile devices are provided (see Fig. 1). The Java application is
best suited for tablet PCs and other pen-based input devices. Moreover, we devel-
oped apps for iOS and Android devices, which can be downloaded from the
respective app stores free of charge. Users can create a sticky note on one of
these devices and send to the whiteboard afterwards.
118 M. Wenzel et al.
The server component connects all parts of the Tele-Board system (see Fig. 1). All
events are transferred as Extensible Messaging and Presence Protocol (XMPP)
messages and distributed to other connected clients. This keeps whiteboard clients
synchronized. That is to say, all users, who open the whiteboard client of the same
panel, always see the same content and can work on the same items.
Within the last years of the HPI – Stanford Design Thinking Research Program, the
development of the Tele-Board system has resulted in a variety of different
functionalities. In Fig. 2 we outline the timeline of the Tele-Board development
from the beginning in October 2008.
In the beginning, we created an environment for teams applying creative
methods that allow them to work together efficiently across distances, without
having to change their working modes. In order to deploy a running prototype in
a short time, the first version of the whiteboard client was developed using JavaFX.
This way, it was possible to gain experience and gather feedback at an early stage.
As we learned from this prototype, it is not only important to enable synchronous
working modes for distributed design teams, but asynchronous collaborative work
as well. To address the problems of Design Thinking teams, who are working
asynchronously over distances, we developed the Tele-Board history browser: a
web-based interface making it possible to go back and forth in the timeline of a
whiteboard. It enables the design thinker to view the collected data from different
perspectives and thereby gain a deeper understanding of the project context.
Additionally, it supports teams in analyzing the overall project progress and
decision paths taken by the respective distributed sub team or by the team itself
in an earlier project phase. The team can also continue at any past state by
duplicating the whiteboard content, i.e. starting a parallel session. All data is
persisted implicitly, meaning that the user has the freedom not to think about saving
data. The JavaFX technology turned out – at that point in time – to not be fast and
flexible enough. Therefore, we decided to develop a Java Swing based version of
the whiteboard client in order to have more control of the rendering.
Based on insights we got from various user tests, our system was further refined
mostly concerning the way users interact with the system e.g. editing of sticky notes
and clustering. Moreover, to make asynchronous working modes more convenient
and enable reuse of the produced data, we wanted to find ways of better
documenting and computationally “understand” the content. Based on the data
that is already archived in our system, we evaluated different approaches in the
field of handwriting recognition and their applicability for unstructured whiteboard
notes. Applications for the recognized texts are searches on the whiteboard content
or reuse in text-applications, e.g. spreadsheets or presentations.
Creative Collaboration in Real World Settings 119
Collaborative work over distances is common practice for most employees of large
global enterprises (Koehne et al. 2012). Many meetings include participants from
multiple locations worldwide (Espinosa and Pickering 2006; Hinds and McGrath
2006). Although a variety of tools for supporting different working modes exist, the
most commonly used tools are still audio conferences (for synchronous communi-
cation) and e-mail (for asynchronous communication) because they are readily
available and easy to use (Matthews et al. 2011; Tang et al. 2011).
Still, some ways of working, such as collaborative brainstorming or sketching
ideas on a whiteboard, cannot be completely supported in remote situations using
standard tools. But this way of working is becoming more and more prevalent. It is
even seen in larger companies who seek to introduce methods such as design
thinking (Brown 2009) in order to increase their innovative potential (Martin 2009).
A lot of teams in large global enterprises are spread throughout different locations
and have to work together remotely with access only to an audio conference and
their computers for most of their meetings. For this study, we worked with a team of
120 M. Wenzel et al.
With their co-located colleagues they would just walk by the office and see if
they had time. Another difficulty for the communication with other locations is
language barriers. As already stated above, no one has English as the first language
and speaks a specific accent, which is sometimes hard to understand by the others
and can lead to misunderstandings. But not only understanding is difficult, but the
same is true for formulating ideas and thoughts in another language:
Ideas or something you want to do spontaneously, in another language you have to think:
how do I phrase this? And this is not only our problem that we have to speak English, it’s
the same for people in India and Italy. Nobody can speak their first language and I think this
is indeed a little handicap.
Especially the topic of idea generation and collecting the points of view from
people with diverse backgrounds and countries is seen as a great advantage of their
global team. They told us, that their manager some time ago, had started an exercise
where everybody was supposed to come up with challenges and opportunities for a
Creative Collaboration in Real World Settings 121
specific topic. The exercise was done simultaneously via all three locations and was
perceived differently by the participants as e.g. these two interviewees:
You have to use all the communication channels: mails, phone, audio conference options
and you have to collaborate, ideate, come up with your thoughts, collate. And then present
back in the same meeting. So it’s both interesting and challenging.
My experience was: It was a rather chaotic situation. We were all sitting in one room
and then split up into different groups. One person was on the phone and the others were
brainstorming from the side, and I was typing in everything in a word document. Others
came up with using Excel. . . so the approaches were totally different.
The mentioned tools, such as email, Word and Excel, are indeed used for remote
collaboration, but they are not ideal for idea generation or brainstorming. We also
heard that people use desktop sharing for the majority of their meetings in order to
show PowerPoint slides or other things on their desktops. However, they sometimes
apply other ways of working that are not supported by the current tools:
We had a meeting two weeks ago with another colleague in Germany. While one was
drawing something on a whiteboard, the person who was not in the room was lost because
he could not follow the discussion in that sense.
Above, we outlined the most important challenges of distributed work for the
team members we interviewed. Summing up, they have to find a way to collaborate
more closely, even if their colleagues are far away, their office hours are not the
same and everybody has to communicate in a foreign language. Additionally, they
wish for more support for new ways of working such as idea collection with the
whole team or working with whiteboards or other tools that are currently only
available for co-located teams.
Fig. 3 Distribution of user involvement throughout the hours of the day and over the whole study
period. Locations are differentiated by colors; the two involved time zones are Central European
Time (CET) and Indian Standard Time (IST)
The team mostly followed the typical working hours pattern from about 9 a.m. to
5 p.m. local time. Apparently, the overlapping synchronous work time is more
intensively used than the time spent working alone at one-location only. This is the
case, even if it implies that the Indian colleagues are working until 6:30 p.m. or
longer. In upcoming research, we want to investigate if this intense synchronous use
can also be observed in teams that have fewer overlapping working hours. We
expect to see more asynchronous work because it is likely that people prefer
working during the standard office hours at their time zone, as we can partly see
with the Indian team members here.
Figure 4 shows a selection of panels and how they were used from week 4 until
the end of the study period. The size of the circles indicates how many users were
involved each day and the colors show the location of the respective users.
The interactions on the panels reveal different usage patterns. It can be assumed,
for example, that panel 590 is used for a long-term feedback collection activity.
Panel 589 was primarily used as an idea sketchpad during a meeting on that specific
Creative Collaboration in Real World Settings 123
Fig. 4 A selection of panels and their user involvement structure over time. Some panels were
only used in a few synchronous global team meetings, while other panels were only used by a few
participants asynchronously
day. After this day, it only serves to document meeting recaps (see the interview
section as the basis for our assumptions). Looking at the different panels, we can see
that there is not one way of working with Tele-Board, people rather shift between
working alone, in small groups, and in larger groups. They use the panels during
different time periods and manipulate the content asynchronously as well as
synchronously.
Giving users the possibility for easy transitions between working together and
alone in order to support “loose and tight coupling”, which is important for
distributed groupware (Gutwin and Greenberg 2002). Our users also appreciated
that Tele-Board not only provides one way of working but several that complement
each other.
With regard to the degree of collaboration among Tele-Board users, we were
interested in how many people worked with the same content, i.e. the same sticky
notes. It turned out that almost half of all sticky notes created during the study
period were moved/modified by at least 2 people over the course of the study, with
the highest number of users editing even eight sticky notes (average: 2.08, SD:
1.44).
The most interesting fact we could derive from the log data was that the activity
of all Tele-Board users was not evenly distributed over the different locations (see
Fig. 5). For all interactions with the system – number of sticky notes created,
number of whiteboard events, number of whiteboard client sessions, and session
duration – we could find higher values for users at the company’s subsidiaries in
India and Italy. Although there were some very active users at the headquarters in
Germany, others contributed little or nothing. However, four of the five new users
who joined the team after half of the study period are located in Germany.
Because the length of service with the company in Germany is higher on
average, we were wondering if there is a correlation between the length of service
and the activity of the users (see Fig. 6). In this figure, the size of the circles shows
the number of sticky notes a user created (each circle represents a user). The y-axis
124 M. Wenzel et al.
Fig. 5 Activity of all Tele-Board users. The left y-axis indicates the number of whiteboard client
sessions (orange) and the number of created sticky notes (blue) per user. The right y-axis and the
red bars show how many hours each participant used Tele-Board within the 3 months of our study
Fig. 6 The level of activity of all Tele-Board users opposed to their length of service with the
company. Each user is represented as one circle. The number of sticky notes created is expressed
by the size of the circles. Based on their location on the y-axis, the circles also show whiteboard
activity in terms of whiteboard events/session duration
indicates the ratio between whiteboard events (¼ every interaction with the white-
board client) and session duration, i.e. whiteboard events per minute. This shows
how active the users were, after they had opened the whiteboard client. Please note
Creative Collaboration in Real World Settings 125
that three German users do not appear in this figure because they did not write any
sticky notes. Though the statistical values in the table show that the German users
generated fewer whiteboard events, opened the whiteboard client less often and
wrote fewer sticky notes (as we can also see here), a lot of them used the whiteboard
actively once they had opened it. This is indicated by the ratio of whiteboard events
per minute. However, we cannot see a correlation between activity and length of
service with the company.
There is a variation in Tele-Board usage between locations. We see the reason
for that based on the difference of potential benefits, as described by Gutwin &
Greenberg’s study on informal collaboration (Gutwin et al. 2008). As mentioned
before, there are many domain experts at the headquarters and the German users can
just informally meet them on the corridor or go to the other person’s office if they
have a question. Therefore, the benefit of using Tele-Board at the headquarters is
not as high as it is at the subsidiaries and maybe not “worth the effort of initiating
and joining into a collaborative session” (Gutwin et al. 2008). Especially with
regard to knowledge exchange, users at the subsidiaries see Tele-Board as a place
where they can ask questions and discuss them. The user saves the time normally
spent writing e-mails by implementing Tele-Board instead as the tool for this
communication. The difference in activity between locations correlates with the
opinions on the usefulness and effectiveness of Tele-Board, as the analysis of the
interviews in the next section shows.
Ten out of the sixteen active users told us that they also use Tele-Board for
asynchronous working methods. One of them for example, posted his ideas on a
whiteboard panel and afterwards he presented them in a project team meeting to the
other stakeholders. In most other cases, a meeting organizer creates a panel and
sends the link to it together with the meeting request and asks all participants to post
agenda topics or questions on sticky notes. In the meeting, they open the panel and
go through all topics on the notes. For feedback collection it may happen the other
way around, i.e. during a meeting, someone creates a panel and all participants are
asked to post their feedback after the meeting (see Fig. 4). This transition between
different ways of working is not available in other tools that the team is using and
was greatly appreciated.
We just saw that the team uses Tele-Board for idea generation and different ways
of communicating synchronously and asynchronously. But, as in the case with all
new tools a company implements, the real question is how efficiently the tool
supports employees in their daily work. Though we could find different points
of view, half of the interviewees stated that Tele-Board saves them time because
more people can work together simultaneously. Because of this, meeting
minutes and documentation can be omitted and the e-mail correspondence can be
decreased.
Earlier we used to just have an open discussion where anyone who has an idea on a
particular topic, gives his or her ideas, and the minutes are taken by one person and finally
at the end of the meeting all ideas discussed are sent out as minutes by this person; which is
time consuming, it’s additional effort for a person to capture the ideas, put them in minutes
and send it out. And not all of them speak up during a meeting if they have some ideas.
Some of them tell their ideas, some of them don’t.
Now that we are extensively using Tele-Board for idea collection, I find that the time for
those exercises has drastically been reduced. Because everyone just puts in their ideas and it
just takes 2 min. and then it’s already there. Now the person who is hosting the meeting only
needs to collect the ideas and put it in a proper grouping. The tool has improved the idea
collection phase.
Interestingly, we only heard these statements from two people who are located at
the company’s headquarters. Additionally, eight out of the nine interviewees who
work in the other subsidiaries said that Tele-Board saves time. This shows that the
benefit of Tele-Board is considered higher by the Italian and Indian users and
correlates with the usage statistics of more activity at these locations (see Figs. 5
and 6).
Most interviewees agreed that there are situations when it is not efficient or
helpful to use Tele-Board. Such situations include one-on-one phone calls, very
short meetings where to-dos are discussed, as well as document review and project
status meetings. As the tasks of all team members within their project work are very
diverse, some interviewees told us that they rarely have meetings where they can
“think out of the box” and the use of Tele-Board would make sense.
Conversely, other users see no advantage of Tele-Board when compared to
existing tools:
Creative Collaboration in Real World Settings 127
I don’t see the added value. Basically it’s another tool for making notes. I prefer having a
list and all of these sticky notes are just too much information for me if no one groups or
categorizes them but just sticks them there. You always need someone who sorts them. I
prefer having it structured. I prefer a list that I can work from. Otherwise I have to read
everything first, then structure it and that’s cumbersome.
In general, there are different points of view as to how much Tele-Board can and
should help in structuring a meeting. Some think it helps to structure a brainstorm-
ing and the grouping of ideas while another user criticizes what he sees as the
unstructured format of the meeting and yet another thinks it is good to have a
relatively flexible way of working.
Some users especially saw advantages in the asynchronous way of working as
time zone differences can be bridged and it is easy to go on working at a whiteboard
panel at any point in time, see Fig. 3.
Sometimes it helps when you are working with colleagues from other locations and you
have been doing some tasks and they have to follow up, because of the time difference it’s
always better you use the tool and you post what you have completed, so you don’t wait.
Because otherwise we have to wait until the German colleagues come in the afternoon.
Here, we can just start in the morning, based on what the German colleagues or others have
posted. That might help in going faster.
Another point where users told us that Tele-Board saves some time and effort is
with respect to the “clean desktop” policy of the company: all employees are
supposed to always wipe off whiteboard content or take away flip-chart sheets
and this is not necessary with a virtual whiteboard.
Though some interviewees already perceived the communication within the
team as very good, others saw some further improvements with the new tool. For
example, one user found it is advantageous for quieter people. The possibility to
communicate is easier when it involves posting a sticky note and explaining it
afterwards. In general, several people agreed that Tele-Board encourages commu-
nication because people are more comfortable to speak freely: They lose their
inhibitions to say something if they can post it first. Especially for asynchronous
feedback rounds, participants liked the “anonymous” appearance of the keyboard
sticky notes though it is indeed possible to track the author of the sticky note.
Interestingly, the feeling of anonymity praised by some was viewed by one partici-
pant as potentially inhibiting users from posting.
Having everybody’s input in a written format is important to many users in order
not to forget something and to improve the understanding of what people want to
say. As stated at the beginning of this paper, problems with audio connections and
different accents of non-native speakers sometimes hamper communication. In this
case, the written sticky notes can assist in communication:
When we talk over the phone, it might be that their voice is not clear and we could not
understand them properly. In that case Tele-Board was very helpful because we have a
written format and we get to see what they really want to tell us.
128 M. Wenzel et al.
Tele-Board offers a new way of working and the whiteboard and sticky note
metaphor is different than the other digital tools used in the company. It was our
goal to introduce a virtual whiteboard for remote collaboration which should be, as
one user describes it, “a perfect addition to the other tools”.
As the team formerly used MS Excel for collecting ideas and feedback, some
interviewees compared Tele-Board to this tool. One user even had the feeling that
they used it like Excel, just the clustering she considered to be easier. The team saw
the main difference and advantage as the possibility to enter data simultaneously or
“real-time.” Others thought it was easier to use than other tools because everyone
knows the analog equivalent and it is more fun to use because of its colors and
playful character.
[..] it’s very receptive, because it’s colorful, it’s sorted, you can concentrate on the visuals
and that’s easier to remember than words on an Excel sheet. We did it with Excel before and
the rows and columns don’t stick in your head. But if you remember the colorful stickies
you can say: yes, the pink topic was below the orange one, it stays in your head as a picture.
No matter if people liked working with whiteboards and sticky notes, we often
heard that they always have to transfer the content into a MS office document at
some point. Or as the manager put it: “It’s good for collecting ideas as a first step
towards a solution. But it has to go on. . .”. This could be a mind map where it is
easy to add links and documents or, as some participants suggested, a Powerpoint
slide deck.
An additional question to participants was how much they used video confer-
encing for their remote work. Most of them said that they usually do not use it for
their project work, because it is difficult to get a video conferencing room at each
location. For this reason, especially in the case of small meetings, video conferenc-
ing was seen as not worth the effort. But they also said, it is not very important to
see the faces if they see the same content on the screen. One user even thought it
could have a negative effect during brainstorming because you are distracted by the
faces, which is consistent with other research (Brubaker et al. 2012).
Several users saw it as an advantage that the use of Tele-Board is not difficult to
learn and that new users can directly start working after some features are explained
to them. These features are: creating sticky notes and adding them to the board,
changing the color of sticky notes, creating clusters, and maybe how to create a
panel and a project. We heard from the new team members that it was not too
difficult to learn how to use Tele-Board.
Members of the team think they would get feedback from other stakeholders
(e.g. product owners or consultants) more easily and on a regular basis if these
stakeholders had a Tele-Board account.
Creative Collaboration in Real World Settings 129
As the tool was only to be introduced to team members, no automatic routine for
creating other users was set up. However, Tele-Board could certainly help in
collaborating with people outside of the company as well. As the manager puts it:
In a world with less budget and possibilities to travel to a customer, a platform for quick
exchange is very valuable. It may help to get a faster understanding of the customer’s needs.
5 Lessons Learned
While working with different companies, we found out that their specific needs
differ from student teams or other groups of people that are not specifically
embedded in a very controlled environment. Therefore, we added a lot of features
to the system that are more likely to be used in a business setting. Company usage
also revealed that some concepts needed to be thought over in a more general way.
For example, the management of users and groups as well as the rights assigned to
them is crucial in a company. It is also vital to organize and structure the work
according to who is working in which team, what are certain dependencies from
other teams, and how we can ensure data security in terms of who is allowed to view
or edit which content. Bringing all this together, while keeping in mind that Tele-
Board should be a tool for creative sessions and easily accessible for almost
everybody, was a challenge.
In the following, we outline some of the features that were optimized for company
usage.
When we started with the Tele-Board project, we said: We want to make it a tool
for everybody who is used to working with a traditional whiteboard. This means
virtually everybody. We don’t want to have the next graphic editing tool, but we
wanted to have it as natural as possible. But determining if the tool feels “natural” is
difficult. In the beginning we thought, we would have to transfer everything we
know from ordinary whiteboards to the digital world. Copying everything one by
one means sticky notes are unchangeable. This limitation was one of the first things
we omitted during development. We found that people see, for example, a sticky
note text-editing function as a typical tool in the digital world. For a long time we
refused to fulfill this requirement, for basically two reasons: First, on a traditional
board, you cannot edit stickies at all. Second, we thought that the creation of
immutable sticky notes is also part of the design thinking mindset. Going for
quantity while deferring judgment – even for what the person herself wrote or
said – does not necessarily work together with an editing functionality in the tool.
As we have seen that many people are now using Tele-Board on a desktop
130 M. Wenzel et al.
computer, we decided to allow editing of sticky notes. So far, we have not identified
people misusing this functionality and spending too much time on editing single
text passages.
There were also some minor changes helping the teams to be more efficient.
Starting with an increased number of colors and shades, it becomes easier to
differentiate between clusters and sticky notes.
Our observations in larger teams with many simultaneously connected people
reflects the need for more synchronicity. In earlier versions, the bin just kept the
deleted sticky notes of each single user and people could undo only their own
deletion. As there can easily be 25 people logged in, it turned out that it was
necessary to also keep the bin global within the session. Still, when the last person
closes the panel, the bin is also emptied. This change is consequent in that the state
of a session stays global each time.
People use our system while continuing to use their traditional tools. This means
we have to integrate our system with other traditional office tools. There are
basically two major points of contact for almost every employee in a company
today: E-Mail, PowerPoint, and Excel.
As the activity feed is already an important part of the portal system, we decided
to also add e-mail notifications so that people are alerted if something has happened
on one of their panels and it would make sense to take a closer look.
The integration of PowerPoint was built to ease reuse of the content created
using Tele-Board. A whiteboard panel is exported as a slide deck. A cluster on the
whiteboard represents a single slide. The topmost sticky within a cluster is assigned
as the title of the slide. All other sticky notes are used as bullet points on the slide.
To keep readability, multiple slides are created when there are too many sticky
notes in a cluster. For having a text-centric output, we also added an Excel export
function.
This is the direction from Tele-Board to an external tool, but we also wanted to
support the other direction as well – inputting a lot of information as sticky notes to
the board. We found that uploading a file using comma-separated values fits most
needs. Each row in the file represents a sticky note text. It is also possible to set the
color of the sticky using a second attribute. People can use this to input content from
a broad range of tools, e.g. spreadsheet applications, word processing, e-mails etc.
Another very important aspect is the involvement of people that are not per se
users of the Tele-Board system, which can include any kind of external
stakeholders. There are several degrees of involvement: If people just want to
take a look at what has been done without editing anything, you can create a public
read-only link to a panel. People can use the history browser to see how the content
was created. But there is also a second option fostering direct feedback. You can
create meetings, allowing also external people to be invited to the session. Using a
temporal limited guest access, those invitees can also input into the running
sessions by writing sticky notes, while seeing in almost real-time how people are
using their input. This possibility was implemented to invite, for instance, external
consultants to give their specific expertise during meetings, which can take place
on-site or via video or telephone calls.
Creative Collaboration in Real World Settings 131
A number of computer supported collaborative work systems exist and many new
systems come up every day. Acceptable support for teams working remotely and
covering every aspect of their creative process is still missing. We want to investi-
gate why certain teams perform better than others, which leads us to basically three
dimensions of distributed work, we want to research on:
There are different factors influencing the success of distributed teams. As the
communication has to be enabled via digital tools in order to bridge the spatial
distances, these tools play an important role in team collaboration. They most
certainly have an influence on team spirit, the team’s common identity, as well as
their general well-being. We want to find out which hardware and software
combinations can support these aspects better than others. Typically, there is a
combination of telephone, videoconference, and screen sharing equipment applied
when working remotely. In a controlled experiment, we want to find out if video-
enabled equipment has significant advantages over non-video setups because it
supports gestures and facial expression during remote sessions. This could improve
the team spirit and common ground and this way enhance the overall experience of
remote meetings.
Besides the role of video and audio, we also want to investigate other hardware
equipment such as digital whiteboards or devices for creating sticky notes. We want
to understand the importance of choosing the right hardware device for a certain
situation as well as using it for the right activity. Especially in the interdisciplinary
d.school environment, it could be possible that some devices are adopted more
easily by all users and others exclude some user groups. Especially for the design of
Tele-Board we want to find out which interaction concept works best in order to
include all users. In general, we would like to know which factors influence the
adoption of different tools and why this is the case.
While investigating the tools used for remote collaboration, it is also important to
find out which kind of activity can be supported in which way. The design thinking
process model differentiates between six distinct phases (understand, observe,
define, ideate, prototype, test). We want to research how far these phases can be
supported in remote settings using Tele-Board in the current form and where it has
to be changed and in what way. From our experience during the introduction of
Tele-Board at a global software company, we know that certain activities are easier
to transfer to the digital world than others. We can say Tele-Board is well suited for
Creative Collaboration in Real World Settings 133
the ideation phase, which means for applying methods such as brainstorming. Of
course, prototyping is much harder to support in a remote setting, because it often
makes intensive use of physical material. But still, with the help of videoconfer-
encing some discussion on the prototypes could take place, and there are also other
kinds of prototypes, for example screen mockups, that could be created together. As
there are different kind of prototypes for different purposes (see the d.incorporate
project), it is interesting to find out which of them can be supported in which way.
But also for the design thinking phases as e.g. observe or define we want to
investigate how they can be supported by different tools, devices and methods. In
summary, we want to outline the factors that influence the suitability of a tool to a
working mode or activity.
References
Brown T (2009) Change by design: how design thinking transforms organizations and inspires
innovation. Harper Business. New York
Brubaker JR, Gina V, John CT (2012) Focusing on shared experiences: moving beyond the camera
in video communication. In Proceedings DIS’12. ACM Press, pp 96–105
Espinosa JA, Pickering C (2006) The effect of time separation on coordination processes and
outcomes: a case study. In: Proceedings Hawaii international conference on system sciences
(HICSS’06), IEEE, vol 1, IEEE Computer Society, Washington, DC, USA, p 25.2
Gericke L, Gumienny R, Meinel C (2012) Tele-Board: follow the traces of your design process
history. In: Plattner H, Meinel C, Leifer L (eds) Design thinking research. Springer, Berlin/
Heidelberg, pp 15–29
Gumienny R, Meinel C, Gericke L, Quasthoff M, LoBue P, Willems C (2011) Tele-Board:
enabling efficient collaboration in digital design spaces across time and distance. In:
Plattner H, Meinel C, Leifer L (eds) Design thinking, understand – improve – apply. Springer,
Berlin/Heidelberg, pp 147–164
Gumienny R, Gericke L, Wenzel M, Meinel C (2012) Tele-Board in use: applying a digital
whiteboard system in different situations and setups. In: Plattner H, Meinel C, Leifer L (eds)
Design thinking research. Springer, Berlin/Heidelberg, pp 109–125
Gutwin C, Greenberg S (2002) A descriptive framework of workspace awareness for real-time
groupware. Computer Supported Cooperative Work (CSCW) 11(3):411–446
Gutwin C, Greenberg S, Blum R, Dyck J, Tee K, McEwan G (2008) Supporting informal
collaboration in shared-workspace groupware. Journal of Universal Computer Science –
JUCS 14(9):1411–1434
Hinds P, McGrath C (2006) Structures that work: social structure, work structure and coordination
ease in geographically distributed teams. In: Proceedings CSCW’06, ACM Press, New York,
NY, USA, pp 343–352
Koehne B, Shih PC, Olson JS (2012) Remote and alone: coping with being the remote member on
the team. In: Proceedings CSCW’12, ACM Press, New York, NY, USA, pp 1257–1266
Martin RL (2009) The design of business: why design thinking is the next competitive advantage.
Harvard Business Press, Boston
Matthews T, Steve W, Thomas M, Sandra Y (2011). Collaboration personas: a new approach to
designing workplace collaboration tools. In Proceedings CHI’11, ACM Press, New York, NY,
USA, pp 2247–2256
Tang, JC, Zhao C, Cao X (2011) Your time zone or mine?: a study of globally time zone-shifted
collaboration. In Proceedings CSCW’11, ACM Press, New York, NY, USA, pp 235–244
User-Centered Innovation for the Design
and Development of Complex Products
and Systems
Abstract In this chapter, we examine user interaction for the design and develop-
ment of complex products and systems. Through a two-phase research effort, we
explore and test the influence of user involvement (i.e. novice/average and expert/
lead users) in early stage design and new product development.
In Phase I, we document the roles of users and stakeholders in early stage design,
based on a review of existing literature and inputs from practitioners in the design
and engineering fields. From this review, we develop a systematic framework for
determining which user entities designers should target for the design and develop-
ment of new products and systems. In order to increase adoption success, the
Design Stakeholder Identification, Assessment and Ranking Framework is
optimized to specifically address the needs of users and stakeholders with the
greatest degree of influence over product use and adoption.
Phase II of this research captures the effects of including expert and novice
product users at discrete phases of the product development process, with an
emphasis on early stage concept design and ideation. Through a controlled experi-
ment, novice and expert users from the healthcare setting provided inputs for the
conceptualization of an intramuscular drug delivery system. Based on inputs from
users of each group, four conceptual prototypes were developed. Using quantitative
assessment methods, we are currently examining if there are significant differences
between concepts that have been designed based on inputs from users of either
group. Factors of particular interest include the perceived usability, functionality,
efficiency, adaptability, and cost-benefit of product concepts.
1 Background
One of the core design thinking principles is user-centric design. It not only
demands thorough needs finding methods and anthropological approaches, but
also requires development teams to test iterative prototypes directly on users,
whenever possible. For the design of complex products and systems, identifying
the “real” user among multiple stakeholders is a difficult challenge. In the
healthcare field, for instance, although patients are often considered the “users”
of medical technology, so too are doctors, nurses, hospitals, and insurers. Thus, the
question remains: who is the target user? Once “the” user group is identified
(if there is only one), then which subjects does one design for within a specific
user group – a highly advanced and technologically adept lead/expert user, or the
average user?
There is an established field of research on the role of “lead users” that emerged
from studies on sources of innovation (von Hippel 1976; von Hippel 1986). Lead
users have been described as individuals or organizations who experience needs for
a given innovation earlier than the majority of the target market (von Hippel 1986),
and who are positioned to benefit from obtaining a solution to those needs. Existing
research has shown that users, as opposed to manufacturers or suppliers, have been
the first to develop new commercially successful products, with user innovation
concentrated among the lead users of those products and processes (von Hippel
1986; Urban and von Hippel 1988; Shah 1999; Lüthje 2003).
However, conflicting views regarding the role of lead users in product design
have been conveyed. Ulwick (2002) described that, “lead users can offer product
ideas, but since they are not average users, the products that spring from their
recommendations may have limited appeal.” Similarly, a retrospective analysis of
eight medical device firms (Shluzas 2011; Shluzas et al. 2011; Shluzas and Leifer
2012) illustrated that companies often prefer to work with industry thought leaders
in the conceptual design and product testing phases of development. However,
surgical procedures developed primarily based on input from lead-user surgeons
often result in the development of technology that is difficult for a broad range of
product users to embrace. To close the usability gap between first and second-
generation products, the companies studied implemented post-market revisions
based on feedback from “average users” in the clinical field.
In drawing analogies to the Technology Adoption Lifecycle (Moore 1991), lead
users often demonstrate similar characteristics to those of innovators and early
adopters. That is, they are often risk takers, have a high degree of opinion leader-
ship, and are fast to adopt new innovations (Rogers 1962). In contrast, average users
User-Centered Innovation for the Design and Development of Complex Products. . . 137
are frequently aligned with the early majority and late majority of consumers in an
adoption lifecycle. They are typically more risk averse and tend to adopt new
innovations after varying degrees of time (Rogers 1962).
A review of existing literature on user interaction for the development of
complex products and systems reveals the need for a greater understanding of the
dynamics and inter-relationships among user types across a range of industry
settings. In particular, there is a need for research that examines how lead/expert
users (e.g. early adopters and innovators) and average/novice product users (e.g. the
early majority) uniquely contribute to the design, and subsequent adoption, of
products and services.
In this chapter, we present a multi-phase research effort aimed at examining user
interaction for the design and development of complex products and systems,
across a range of high-technology industries. The first phase of research focuses
on the question: Which users and stakeholder groups should developers target for
the design of complex products and systems? The second phase of research then
centers on the questions: Within each targeted user group, which subjects does one
design with and for during the product development process – a technologically
adept expert user or the “average” user; and does designing with and for particular
users and user groups have an impact on the usability, functionality, novelty, and
cost-benefit of new products and systems?
Which users and stakeholder groups should developers target for the design of
complex products and systems?
To explore this question, we first present a literature-based comparison between
the conceptual notion of “users” and the direct inclusion of “users,” within several
design methods. After having determined that the concept of “user” varies greatly
among design methods and throughout stages of product innovation, we modify the
notion of “user” in favor of a more balanced, role-based stakeholder perspective. In
doing so, we provide a systematic framework for determining which user entities
designers should focus on throughout the design process. Contrary to current
processes for assessing and prioritizing users and stakeholders that are often
fragmented and qualitative, the framework we propose combines new and existing
tools into one systematic process. The Design Stakeholder Identification, Assess-
ment and Ranking Framework (Agrawal et al. 2012) is geared toward the design
and development of complex systems, involving an embedded network or ecosys-
tem of users and stakeholders. It intends to provide designers with a comprehensive
tool for ensuring that the needs of stakeholders with a high degree of influence over
product/system adoption are met.
138 L.A. Shluzas et al.
The range of user types shown in Table 1 illustrate that target users may vary for the
development of an individual product, based on the design method chosen. For
example, the current development of Boeing’s 787 Dreamliner has involved multi-
ple users, including the crew, passengers, airline operators, aircraft manufacturers,
and regulatory authorities. Although passengers are one of the aircraft’s final end
users, so too are pilots and the crew. But the choice of buying an aircraft will depend
on airline companies. Additionally, maintenance is a major factor in determining
the actual airtime for a plane. Thus, involving the ‘right user’ in the design process
is extremely important for the success and adoption of this product.
User-Centered Innovation for the Design and Development of Complex Products. . . 139
For simplicity sake, but also based on existing teaching dogma, users are quite often
seen as individuals; when in reality, users are part of a complex stakeholder network
with delicate and often non-obvious relationships. Per the ‘Stakeholder Network
Theory’ (Rowley 1997), each firm encounters a different set of stakeholders that
aggregate into unique patterns of influence. They respond to the interaction of multiple
influences from the entire stakeholder set. In fact, the notion of conducting participatory
observations and creating concrete user narratives and personas is gaining ground in
design practices. But, there is a danger that designers may focus on the “wrong user,”
one who may not significantly contribute to the future adoption of the product/service/
solution. We therefore aim to identify and analyze the stakeholder network, as a
foundation for the user group selection process.
Also, the roles of the user groups may vary significantly from one industry to
another. For instance, the government may be the end-user for defense products, but
it would be a legislative authority for a healthcare product. Thus, understanding the
user in terms of stakeholders and their roles in the context of different industries is
critical to developing a framework for classifying user-centric design methods. Due
to this context dependency, each product/system and industry requires an
140 L.A. Shluzas et al.
We present a systematic framework to help design teams better understand the roles
of different users and stakeholders, and to prioritize their involvement in the design
process. The four stages of the Design Stakeholder Identification, Assessment and
Ranking Framework, detailed in the paper by Agrawal et al. (2012), include the
following steps:
1. Gap analysis (or needs finding)
2. Customer value chain analysis
3a. Cycle of use analysis
3b. Monetary flow analysis
4. Final stakeholder ranking
Step 1 (pre-step) – Gap analysis
The purpose of this initial step is to identify a potential new product, or new
features of an existing product, that address unmet needs. This step is intended to
ensure that the design team has articulated a clear strategic focus such as a specific
industry, target market, or a specific need. There should be a strong match between
the expertise in the product team, the industry or market focus, and the potential of
the market for product adoption. The outcome of this initial step is a list of
requirements for the new product or product features. These could be grouped by
functionality or use cases if necessary.
Step 2 – Customer value chain analysis (CVCA)
The purpose of this analysis step is to enable the design team to comprehensively
identify pertinent stakeholders, their relationships with each other, and their role in
the product’s life cycle. The outcome of this analysis is a product value net that
includes a list of all product stakeholders, their intents and incentives, and their
inter-relationships. Typical value propositions for stakeholders include: money or
payments, complaints, regulatory influences, services or necessary information, and
tangibles such as hardware and materials.
The CVCA analysis (Donaldson et al. 2006) enables design teams to determine
stakeholders with decision-making influence, based on the number of arrows
pointing into or away from each stakeholder in the value net. Stakeholders with
peripheral influence will less likely be at a focal point in the value net. As such, the
value net diagram highlights the most important decision-making stakeholders
involved in the development and use of a new product or product feature.
User-Centered Innovation for the Design and Development of Complex Products. . . 141
The design team should note key decision makers in a tabular format, before
proceeding to the next step.
Step 3a – Cycle of use analysis
The purpose of this step is to assess different stakeholders’ value drivers and
incentives with respect to behavioral, technical, and social attributes. This step of
the analysis should be performed from the usability perspective, with functional
requirements grouped into “use cases” or procedural steps that users will perform
while using a product or feature. The output of this analysis is a list of stakeholders,
ranked according to the degree to which each is affected by a particular attribute,
from a behavioral/usability, technical/utility, and social perspective.
Behavioral/Usability attributes focus on the degree of behavior change a user
would have to make in order to use or adopt a specific product or new feature, in
comparison to existing practices or procedures. A score of 0 is assigned for no
change, one for minimum change, and five for a completely new way of
accomplishing a specific task.
Technical/Utility attributes center on a product’s technical functionality and
performance. The ranking for each feature-stakeholder combination is based on
how critical the stakeholder considers a particular feature to be. Another way to
view this is with respect to the value a stakeholder places on the technical benefit
(utility) of a new feature. A “must have” feature is scored 5, and a “nice to have”
feature is scored 1, using a tabular assessment.
Social Interests are attributes that consider social factors, such as power and
prestige, associated with using a particular product, technology, or product feature.
This ranking is based on the stakeholder’s perception of status associated with using
a specific product or feature. In scoring social interests, a perceived large increase in
social status is assigned a score of 5, and minimum or no change is assigned a score
of 1. The process of combining scores in each area is captured in detail by Agrawal
et al. (2012).
Step 3b – Monetary flow analysis
This stage of the analysis assesses stakeholders with financial interests in a
product in order to identify key decision makers who would influence product
adoption from a financial perspective. In this step, the team starts with a list of
stakeholders from step 2 (CVCA), and documents constituents that pay for products
or new product features. This involves quantifying the financial gain or loss that
each stakeholder would potentially experience, given the introduction or adoption
of a new product or feature. If a stakeholder could potentially experience a large
financial loss (relative to other stakeholders), then he/she would be assigned a score
of 5. On the other hand, if a stakeholder could potentially gain financially from the
introduction of a new product or feature, then he/she would be assigned +5. If a
stakeholder is not affected financially then the score is 0. At the end of this exercise,
the team will have a financially focused stakeholder-product features table with
scores ranging from 5 to +5 for each product.
142 L.A. Shluzas et al.
The stakeholders
Decision whom developers
Making
Influence
the design process
(highest scoring)
Stakeholders who
Stakeholders who receive low scores in two or more categories are considered
lower priority stakeholders. In terms of resource allocation, we recommend that the
design team avoid allocating a large percentage of resources toward the develop-
ment of products or product features to satisfy the needs of stakeholders in this
group (unless their needs correspond with the needs of stakeholders addressed
previously).
Following this four-step ranking and stakeholder categorization process, the
team can utilize the framework to include high priority stakeholders in the product
development cycle, in an effort to satisfy the needs of users/stakeholders with the
greatest degree of influence over product use and adoption.
The second phase of research is an ongoing study that focuses on the following
research questions:
144 L.A. Shluzas et al.
Fig. 2 Schematic
representation of early stage
design phases
Current Work
Hypothesis 1
Future Work
Hypothesis 2
Within each targeted user group, which subjects should one design with and
for during the product development process – a technologically adept expert user
or the “average” user?
Does designing with and for particular users and user groups have an impact
on the usability, functionality, novelty, adaptability, and cost-benefit of new
products and systems?
To examine these questions, we are studying the roles of expert and novice
product users at discrete phases of the product development process, with an
emphasis on early stage concept design and ideation (per Fig. 2). The goal of this
research phase is to evaluate the degree to which designs differ, in terms of factors
such as usability, functionality, and cost-benefit, when they are based on inputs
from novice versus expert users. As previously noted, this research stems from an
earlier study by Shluzas (2011), which illustrated that medical device developers
often prefer to collaborate with industry thought leaders during the conceptual
design and product testing phases of development. However, surgical procedures
developed primarily based on input from lead-user surgeons (von Hippel 1986)
frequently resulted in the development of technology that was difficult for a broad
range of users to embrace.
In the current study, we hypothesize that at the conceptual phase of product
design both expert and novice users will have a higher preference for design
concepts based on input from expert users. This is due to the ability for experts to
identify needs ahead of the target market (von Hippel 1986), and because experts
have been shown to have a better understanding of functional relationships (Chi
et al. 1981), and are better at identifying problem solving strategies (Klein 1999).
However, at the usability and functional testing phases of design we hypothesize
that both experts and novices will prefer designs that have been improved based on
inputs from novice users. This is attributed to novice designs potentially being less
reliant on user skill, more intuitive, and more adaptable to different usage scenarios.
To test these hypotheses, we chose to examine the design of intramuscular
(IM) drug delivery devices, namely syringes, since these products involve daily
interaction by users of varying skill levels. More specifically, these devices were
selected because they are small on one hand, but also complex mechanical systems.
Their usage involves multiple stakeholders with different design requirements
User-Centered Innovation for the Design and Development of Complex Products. . . 145
For each user, we conducted a 60 minute user input session at either the Stanford
Center for Design Research (CDR) or the user’s place of employment. During these
sessions, users were asked to complete an initial survey to document their injection
experience and indicate the demographics of patients to whom they have
administered injections. Users were then shown a brief video that illustrates the
injection process, in order to frame the design exercise in the context of
administering injections within a clinical setting. For about 30–45 minutes, users
described their wants and needs through sketching, written, and oral descriptions.
These sessions were videotaped to fully capture user inputs, and to provide inputs to
designers at a later time. Finally, users completed the session with a post survey, in
which they were asked to rank their top needs and document their technology
adoption profiles.
The user inputs were divided into novice and expert user groups. For each group,
three overarching themes were synthesized and each user input (i.e. comment) was
manually coded to gain a sense of the main design priorities by quantity of
comments. To supplement each designer’s empathy and understanding based on
user inputs alone, designers researched IM injection details online (google) and
watched online videos of IM injections (youtube). To generate designs, each
designer selected quotes that “stuck out” as significant and representative of the
users in each group.
146 L.A. Shluzas et al.
For our cohort of novice users, three guiding priorities emerged from the coded
data: (1) Safety, (2) Efficiency, and (3) Accuracy. Using these design priorities,
designers first generated miniature concept hand-sketches. These mini-concepts
represented small fragments of features (not fully functional designs). They next
created two to four detailed composite sketches (fully functional syringes) on letter-
sized paper that combined the most promising features. These detailed storyboard
sketches served as the basis for generating 3D CAD renderings (in Rhino3d),
corresponding technical drawings (in Solidworks), and an annotated design bro-
chure (in Powerpoint). The process was repeated for expert users, but with the new
priorities of: (1) Safety, (2) Accuracy, and (3) Efficiency.
Examples of two designs created based on inputs from the novice and expert user
groups are shown in Figs. 3 and 4, respectively.
Based on the top ranked design concepts from users of each group, we aim to map if
novice users prefer designs based on inputs from novice users or expert users, and if
expert users prefer designs based on inputs from either novices or experts, per the
diagram in Fig. 5. Selection criteria include factors such as usability/perceived ease
of use, functionality, speed and efficiency, adaptability, nurse safety, patient safety,
and cost-benefit.
The intended theoretical and practical contributions of this research phase are:
(1) to build on research in lead user innovation, by von Hippel (1976, 1986) and
others, by illustrating the unique contributions that novice and expert users make at
discrete phases of the design process, and how their contribution impacts product
adoption; (2) to contribute to literature on design adaptability and intuitive design
by illustrating which user groups facilitate the design of products for variable use
scenarios; and (3) to extend literature in participatory design, from the human
computer interaction and software development fields, to the design of tangible
products. As a final goal, we aim for the designs created in this study to be used as
future drug delivery devices in routine clinical care.
Our research shows that there is substantial dissent in the nature and concept of
‘users’ among product development and design communities. Though often seen as
individuals, users are frequently part of a complex stakeholder network that has
been integrated into the development process through a series of interconnected
relationships.
User-Centered Innovation for the Design and Development of Complex Products. . . 147
?
on inputs from novice or
expert users
Design Expert User
(Expert Input) Preference
References
1 Introduction
Fig. 1 Illustration of the collaborations between different stakeholders involved in design think-
ing projects
idea several times, the design thinking team hands over this idea to the client’s
management (3). How this handover looks depends on the client’s wishes and may
range from an informal presentation to a formal specification based on a template
provided beforehand. Afterwards, the client’s management decides about the reali-
zation of the product. If the client’s management favors the realization of the
product, the client uses in-house engineers or outsources it (4). Depending on the
setting, the client combines engineering and manufacturing. Therefore, the client’s
management passes the available information about the product idea to the engi-
neering team. Further, the engineering team realizes the desired product as possible
within engineering constraints and hands back a blueprint to the client (5). If the
client’s management is pleased with the blueprint, it is passed on to manufacturing
(6 and 7). Then, the product can be brought to market (8). The steps concerning the
blueprint (5) to bringing the product to market (8) are in general uncritical
concerning communication due to standardization, e.g. standard notations like
UML.1 Within our research project we investigate the steps concerning the impact
of the handover (3) on the subsequent realization of the product idea (4). While
practitioners have established best practices, these steps are still critical due to
non-standardization. If the outcome of these steps is insufficient, it is likely that the
product being manufactured is a less desirable product, which does not fulfill end
user needs, in contrast to products being manufactured by engineers who are well
informed.
1
http://www.uml.org
Connecting Designing and Engineering Activities 155
To summarize, innovative ideas cannot unfold their full potential if their reali-
zation does not conform to elicited requirements due to uninformed decisions made
by engineers. Engineering is the last step in the overall process before the actual
manufacturing takes place and the new innovative product is brought to market.
The involved engineers are responsible for realizing these desirable products, which
are to be feasible as well. Therefore, engineers need to be empowered to make well-
informed decisions concerning realization alternatives and tradeoffs between desir-
ability and feasibility. But typically design thinkers only present the final ideas to
the engineering team or merely to the client’s management – their journey of how
the design thinkers arrived at their final ideas is often neglected. Often, such a
presentation mainly conveys features instead of the underlying requirements or
design constraints. Besides making a final presentation, design thinkers also often
present a final design prototype that covers important aspects regarding desirability.
However, feasibility aspects can only be covered partially by design thinking team
members with a background in the involved engineering disciplines.
A design prototype can be considered a model that fulfills a certain purpose
(Gabrysiak et al. 2010). Thus, a design prototype also abstracts from properties that
are irrelevant to conveying the final idea. These abstractions might lead to misinter-
pretations during engineering, which is usually focused on feasibility, since its
result is always feasible. Still, trade-offs during engineering might lead to a less
viable or even less desirable product. Figure 2 depicts that a successful idea needs to
be desirable for the end users, feasible to realize and also economically viable.
From an engineering perspective, the dimensions of desirability and feasibility
are the most important. If desirability or feasibility are not fulfilled, then also
viability is not given – neither can you sell an undesirable product nor can you
manufacture an unfeasible product viably. Figure 3 depicts a possible tradeoff
space, which consists of the three dimensions mentioned by (Brown 2009). The
intersection of all three sets depicts all desirable, feasible and viable product
alternatives. A design prototype has a desirable design, but covers engineering
constraints only partially. Thus, only presenting and handing over a design proto-
type to engineers is insufficient. But at least, it is a good starting point, since it
provides an overview of the idea. Still, engineers require a trace of which
alternatives have been explored, which constraints have been uncovered and how
these influence the subsequent decisions.
Edelman and Leifer (2012) described two distinct modes of path determination.
Either a decision is made in-situ, based on the readily available information (way
finding), or the design thinkers plan ahead by navigating through the design space
“along the most efficient route” (navigation). Thus, the design thinkers’ journey
progresses differently depending on the available information. During this journey,
different prototypes are built. As argued by Tohidi et al. (2006), creating and
evaluating multiple prototypes in parallel yields better results compared to
employing only a single one. Consequently, by restricting the documentation on
the final prototype, much valuable information discovered from building and
evaluating the other ones are only passed on implicitly and might have to be
re-discovered later on by the engineers.
156 T. Beyhl et al.
2
Due to space limitations, we are not able to describe the large number of available traceability
approaches in detail.
158 T. Beyhl et al.
(Hayes et al. 2003), and combined approaches, e.g. (Poshyvanyk et al. 2006).
Further, traceability approaches exist capturing the hierarchy and context of trace-
ability links, e.g. (Seibel et al. 2010) and (2011).
Gotel and Morris (2009) state that agreed requirements are the result of infor-
mation content transformations between different representations, e.g. use cases
can be derived from the transcription of an interview recording. Unfortunately, such
transformations are usually not bidirectional and important information can get lost.
Therefore, Gotel et al. argue that relationships, i.e. traceability links, between
artifacts might not be as obvious as one might expect concerning syntactic and
semantic issues. They conclude that “storing, using and maintaining extensive
media-rich materials is far more costly than creating them in the first place.”
This also applies to design thinking.
As the literature conveys, traceability is important – not only in software
engineering (Gotel and Morris 2011). However, not everybody is willing to apply
traceability techniques and maintain traceability links (Arkley and Riddle 2005).
The same is true for capturing the rationale behind design decisions (Klein 1993).
3 Current Practices
When speaking about connecting design and engineering activities, the overall
process as depicted in Fig. 1 has to be considered as well. The following Figs. 4,
5 and 6 depict possibilities of how design thinking and engineering can be
connected (Beyhl et al. 2012).
In the decoupled setting in Fig. 4 design thinking (step 1 to 3 in Fig. 1) and
engineering (step 4 and 5 in Fig. 1) take place sequentially and a clear separation
between design thinking and engineering exists. In this setting, design thinkers and
engineers (or at least the management) meet rarely, mainly to present and hand over
the results (e.g. design prototype, documentation of the final result). Since the final
idea is stable it is known what kind of engineer is required. Therefore the idea has to
be conveyed to the engineering team. As for all presented settings, the engineering
team is usually larger than the design thinking team. Thus, a small set of design
thinkers has to hand over their knowledge to the larger engineering team. Of course,
the introduction of new personal is not effortless and requires documentation and
communication (Balzert 1997). Design thinking teams are multidisciplinary and,
Connecting Designing and Engineering Activities 159
therefore, also contain engineers. In this decoupled setting, these engineers are still
different from the engineers, who realize the idea later on. Thus, the engineers of
the design thinking team cannot act as knowledge carriers. Still, they are able to
provide feasibility assessments throughout the design thinking process. This
decoupled setting is the common approach for the subsequent steps of design
thinking and engineering. Therefore, the progress speed can be considered as
normal and less agile concerning the product idea, which is stable after the
handover takes place. In practice, we mainly observed the decoupled setting –
especially since the final documentation in professional settings is considered to be
sufficiently complete.
Figure 5 depicts the overlapping setting. This setting does not separate between
design thinking and engineering. Instead, design thinkers and engineers work
together before and after the handover of the final product idea. Consequently,
the engineers get insights into the design thinkers’ knowledge and the design
thinkers are available for questions later on. As for the decoupled setting, the
engineering team is also larger than the design thinking team. By introducing the
overlap between design thinking and engineering, the overall team size increases,
which requires more communication between the team members than before.
Therefore, an additional process overhead is introduced which might slow down
the overall process. Furthermore, the actual engineering should not start until the
product idea is stable. Consequently, the overlapping setting has no benefits
concerning progress speed. Besides the necessary communication between design
160 T. Beyhl et al.
thinkers and engineers, the overlapping setting allows engineers of the design
thinking team to actively contribute when engineering takes place. The overlapping
process phase can only take place if the kind (e.g. product or service) of final
outcome is considered stable and therefore it is known which engineers are
required. If the engineers start the implementation of the idea too early, these
efforts might be wasted if design thinkers come up with a different idea later
on. Or, as Alexander stated, simultaneous development and implementation of
requirements cannot always be right – “still less, while concrete is being poured
and metal is being cut” (Alexander and Beck 2007).
In the concurrent setting, design thinkers and engineers work simultaneously as
depicted in Fig. 6. While theoretically possible, this setting is unrealistic concerning
team size, progress speed and contribution. Design thinking teams are small and
additional participating engineers would increase the team size. Also if only a few
engineers participate, the knowledge they gained has to be shared with the rest of
the engineering team. Thus, the handover problem is merely shifted. Furthermore,
engineers from the engineering team are not trained in design thinking and are only
able to contribute from their perspective, which is usually focused on feasibility.
This, in turn, can be counterproductive in view of the brainstorming rule “Defer
judgment!”, if the engineers reject early ideas.
Design thinkers, on the other hand, are only able to contribute their knowledge
about the product idea during engineering. Moreover, if the engineers start too early
with the realization of an immature idea, they have to adapt their result as soon as
the design thinkers shift their direction, which introduces additional overhead.
Further, it has to be known what kind of engineers are required in advance.
However, design thinking projects are open-minded and the kind of final outcome
is not known in advance.
3
http://www.hpi.uni-potsdam.de/d_school/home.html?L¼1
Connecting Designing and Engineering Activities 161
the wiki as documentation platform, the D-School provides a file system share.
Analogue to the wiki, the information manager of the D-School only prescribes the
structure up to the project level and the students are responsible for structuring their
project folder as they like. They often use the same structure as with the wiki. The
students use the file system share to upload and share big files such as audio or video
recordings and presentation slides. Currently, the D-School is switching from the
wiki to Incom4 as the communication and documentation platform. Within Incom
the students document their findings, upload photos and add comments using a
built-in timeline feature. In general, our interview partners favor this timeline in
contrast to the wiki. The main advantages of Incom are the visualization of data and
feedback sharing. In comparison to the previous wiki, pictures are displayed, not
only stored as files. Besides that, the built-in timeline feature helps the D-School
faculty to comment on the project progress. Shortcomings of Incom for the system
administrators are the time-consuming setup and the lack of support to maintain/
adapt the platform to specific needs. Besides these software tools, the D-School also
provides templates, e.g. presentation templates, with suggestions of what to write
down. For example, the D-School provides a question template with questions to be
answered every day. During our interviews, it turned out that documentation is
created relatively close to the end of the project and before the final presentations
take place. Nonetheless, the students do not only present the final outcome, i.e. their
final prototype. Although they present up to six times their project progress, the
documentation of the project’s journey is often neglected. Depending on the project
partner, the students describe their idea in more detail afterwards, e.g. further
meetings with the project partner or final reports. We further investigated how the
students rate the provided best practices. The following statements are an excerpt
from our interviews:
• First, the problem is nobody is willing to document.
• Incom is not a blog for sure, it’s too complicated, it’s too official, it’s no fun to work
with it.
• There is no individualization in Incom.
• Incom is no collaboration instrument.
• I would like to have something like Google[Docs].
• A login screen is a barrier at the beginning.
• It’s nice to see what other are doing […], but there are too many subsections I have to
open […].
• After two weeks we start to forget where things came from.
• The communication should take place in Incom, but it’s done via e-mail.
• The best would be to document during the process, but we don’t have time for it.
• Some people document during the process and some at the end.
• At the end of a 12 week project we don’t know what we did at the beginning.
4
http://www.incom.org
162 T. Beyhl et al.
of the project partner. Among others, this collection consists of GoogleDocs5 for
collaborative editing of text documents and spreadsheets, Dropbox6 for document
sharing and easy photo upload from mobile devices and Facebook7 to arrange
meetings. Each of these choices indicates use cases and scenarios that are not yet
covered otherwise. Thus, by combining these use cases and the information they
affect into an overall tool, a documentation platform can be realized that students
are willing to use. As experience shows (Gabrysiak et al. 2012b), providing students
with a well-structured template of what to document is not sufficient as they will
only look at this template after they finished collecting all the insights and
establishing their ideas.
Based on our examination into how the software design consultancy D-LABS8
gathers and manages their insights (Gabrysiak et al. 2011), a bachelor project
investigated how these findings and results are documented (Gabrysiak
et al. 2012a). D-LABS is a start-up company located in Potsdam (Germany).
They provide expertise in design thinking for complex multi-user software products
to customers as a service (cf. 1, 2, and 3 in Fig. 1). Their projects are run based on a
catalogue of design thinking methods adapted to the domain of software.
Figure 7 depicts the Design Led Innovation (DLI) approach (Mecklenburg
2012). D-LABS covers the requirements engineering stage of a project. To do so,
they start by understanding the problem domain through a 360 view. Afterwards,
they start end user research using observations and interviews. The succeeding
synthesis phase during which the point of view is created, iterated and refined is the
most important one, since the ideation of solution concepts relies on the correctness
of these agreed-upon findings. Based on the ideation, the prototyping stage usually
starts with paper prototypes, which are iterated till a prototype of the graphical user
interface is created.
Concerning the application scenarios for design thinking, D-LABS fits into the
decoupled setting (Fig. 4). Their engineers are responsible for the creation of
interactive prototypes, which enables them to assess the corresponding feasibility.
However, these engineers are not part of the engineering team later on. In order to
maximize the reuse capabilities of their results, apart from a detailed documentation
of how they got there, D-LABS also provides re-usable software engineering
artifacts from which the final product may be generated. As in all projects,
D-LABS uses a shared folder for all documents, such as pictures, persona
5
http://docs.google.com
6
http://www.dropbox.com
7
http://www.facebook.com
8
http://www.d-labs.com/english/
Connecting Designing and Engineering Activities 163
Fig. 8 The resulting tool of the bachelor project, capable of producing specifications suitable for
multiple distinct perspectives (Gabrysiak et al. 2012a)
descriptions and digital prototypes. These folders are structured the same. Further,
file versions are managed manually.
In the bachelor project “Design Thinking meets Requirements Engineering” the
students deduced a meta-model capable of describing the overall state of a project
by systematically going through the corresponding project folder. Thus, with such
an approach, as illustrated in Fig. 8, standards for documentation can be defined and
enforced. Of course, enforcing a sophisticated documentation style introduces
overhead and might even decrease project performance. However, since the docu-
mentation being handed over is what the customer pays for, it is essential that the
overall idea is captured, documented, and represented thoroughly.
In a study conducted as part of this bachelor project, software engineering
students were asked to evaluate design thinking deliverables. They had to focus
on whether they would be able to implement the specified idea. One of the results
indicated that while it is important for design thinkers to have a process that they
can rely upon when explaining how they arrived at their results (Lyon 2011), most
of the software engineering students did not look at the attached description of the
design thinking process. Instead, they argued, that they trusted results presented by
consultants as long as their process is specified (Knolle 2011).
3.4 Summary
Current practices show that the decoupled setting dominates since design thinking
and engineering are two steps that can clearly be separated. Furthermore, it is rarely
feasible that design thinkers accompany the whole engineering process later
164 T. Beyhl et al.
on. When design thinking projects are initiated and a design challenge is issued, the
result is usually not predictable. Thus, only after the idea is presented, is it known
which engineering experts from which domain are required to successfully imple-
ment the idea. This temporal dependency restricts the application of the
overlapping or concurrent setting.
We also have to distinguish between the educational and the professional setting.
In educational settings, the design thinkers (students) perceive the documentation
effort as high and without benefits for themselves. In professional settings, on the
other hand, design thinking professionals assess the documentation effort as
medium but necessary, since they rely on their documentation to create business
value for their customer. The successful application of design thinking in profes-
sional settings is usually found in combination with structured templates and
artifacts. By customizing the documentation to the needs of engineers, the realiza-
tion of such ideas can be simplified to increase the chances of a successful
realization.
When design thinkers synthesize information from different sources, they also
re-represented it in other forms and dimensions. Consequently, the originating
sources are hard if not impossible to recover if no traceability is used (Gotel and
Morris 2009). Initially, we argued that traceability techniques may also be
9
We do not understand this documentation framework only as a framework from engineers’
perspective, but also as a framework from design thinkers’ perspective.
Connecting Designing and Engineering Activities 165
applicable for design thinking projects to hand over the traceability information
along with the documentation to the engineers later on.
Hypothesis H1
concepts are difficult to evaluate in professional settings since financial risks have
to be avoided and corporate secrets must remain secret. Therefore, we decided to
focus on educational settings first and transfer successful concepts to professional
settings later on. Consequently, applying traceability is only the second challenge,
especially in educational settings.
The first challenge of our research topic is to get design thinkers to document their
findings, insights, alternatives and decisions. They almost exclusively rely on
analog artifacts such as post-its and tangible prototypes, which are captured in
digital artifacts, for example photos and videos. They upload only the most impor-
tant artifacts to the used documentation platform, especially in educational settings.
In professional settings, common file system structures lead to a more complete and
structured set of artifacts.
Usually, the person who uploaded the artifact does not comment these digital
artifacts further since digital artifacts already carry different (meta) information.
However, the same is assumed for the analog information within the digital artifact.
Depending on the type of digital artifacts, this can be a problem later on due to
technical constraints, for example handwriting recognition is a difficult task.
Connecting Designing and Engineering Activities 167
Moreover, knowledge carriers, for example design thinkers, are not willing to
document. The reason is that someone who documents thoroughly is usually not
one of the beneficiaries of the documentation – especially in educational design
thinking settings (Fig. 11a). In the literature, different benefit problems are
described, for example the traceability benefit problem (Arkley and Riddle 2005)
and the rationale capture benefit problem (Klein 1993). Instead, they document the
way they are accustomed to which is based on their academic background. Further,
different kinds of information need to be captured and traced, e.g. process informa-
tion, artifact information and context information. By making documenting a
beneficial activity, design thinkers would intrinsically be motivated to document
appropriately (Fig. 11b). The question is how such benefits should look.
Hypothesis H2
Based on our interview results, design thinkers write down their knowledge in
natural and visual language, e.g. sketches, when it is conscious. In general, while
being the only specification language “that can be assumed to be common to all the
stakeholders”, natural language is also inherently ambiguous (Gervasi and Zowghi
2005). In contrast, engineers prefer a common standardized structure and formal
precision without a margin of misinterpretation.
Hypothesis H3
Hypothesis H4
4.4 Summary
5 Experiment Results
Fig. 14 Investigation as to which handover document most fits the concerned engineer’s needs
participated. Each document was studied by ten of the participants. For each
document, these participants were balanced concerning their experience
(e.g. academic degree, practical experience) in computer science and their field of
study (e.g. software engineering, informatics, business informatics). Each partici-
pant read the document in less than 30 minutes and answered a questionnaire
afterwards. The participants were allowed to look things up in the handover
document assigned to them while answering the questionnaire.
Going through the assigned document and answering the questionnaire took less
than one hour for all of the participants. Figure 15 depicts the most important
results. Among other questions, the participants were asked to rate their agreement
to the following statements in a Likert scale (1 means very strong disagreement,
7 means very strong agreement).
Question A: “I’m able to create a product that fulfills the requirements of the
customer.” The participants rated the software requirements specification (aver-
age of x ¼ 5:0 , variance of s2 ¼ 2:7 ) as the most suitable handover document
concerning the ability to create a product that fulfills the requirements of the
customer, because this kind of documentation fits best for software- and hardware
system. We were surprised that the participants rated the video (x ¼ 3:7, s2 ¼ 2:9)
higher than the informal document (x ¼ 2:4, s2 ¼ 1:8). A possible reason might be
that such a presentation video conveys an illusion of sophistication and customer
contact. Usually, such an implicit assumption is incorrect.
Question B: “In the document, all requirements are covered sufficiently.” For
this question, the participants rated the SRS as most sufficient concerning the
specified requirements (x ¼ 3:9, s2 ¼ 4:3), again. Although the SRS is rated best,
the Likert value is still relatively low. As initially stated, we had to restrict the
documents’ number of pages to ensure that the participants were able to read the
document within an acceptable time. In software projects, such documents can have
hundreds of pages, depending on the complexity of the product. In such a case,
we believe that engineers would rate the SRS even higher. The captured presentation
(x ¼ 2:2, s2 ¼ 2:0) and the informal document (x ¼ 2:4, s2 ¼ 4:7) are considered to
have an even lower coverage of the requirements. From an engineer’s point of view,
Connecting Designing and Engineering Activities 171
Fig. 15 Selected results (average values) of our experiment concerning the appropriateness of
handover documents (1 means very strong disagreement, 7 means very strong agreement)
this is not surprising, since these two kinds of documents are only expected to
convey the basic idea.
Question C: “I’m able to start development without having to meet the
customer first.” Also for this statement, the SRS received the highest rating (x ¼ 3:6,
s2 ¼ 3:4). The statement implies a sufficient completeness of the handover document
and, therefore, correlates with the other two statements. Again, the captured presentation
(x ¼ 2:3, s2 ¼ 2:7) is rated higher than the informal document (x ¼ 1:1, s2 ¼ 0:1).
As argued before, as opposed to the informal document, the video might convey
customer contact, which makes further meetings superfluous.
To summarize, we consider these results as a trend that showed captured
presentations and informal documents to be insufficient as handover documents.
This is because of their informal and non-standardized nature, which does not cover
all necessary insights and ideas. Nonetheless, the captured presentation and infor-
mal documents are a good starting point to pass on the basic idea and to provide a
common understanding of the idea.
H3. Since this experiment is still ongoing, we are only able to discuss preliminary
results. This experiment is based on the observation that design thinkers follow no
clear rules while documenting (Fig. 16a). One reason might be that design thinkers
often rotate the task of documenting. In contrast, engineers have common rules on
how to document (Fig. 16b). We are conducting this experiment to explore how a
common structure for both, design thinkers and engineers, might look.
We prepared a set of digital design thinking artifacts consisting of whiteboard
photos containing post-its, personas, interview transcripts, photos of prototypes and
audio/video recordings. Overall, we provided 160 digital artifacts placed into one
folder, which the students were asked to organize by using file operations (move,
delete, rename, link, sort, etc.) provided by the file system explorer of the Windows
operating system. The participants were assigned to groups of three. Each group
consisted of one experienced design thinker, one person familiar with the design
thinking method, who had not yet used it himself, and one person who had never
heard of it before.
In the first part of the experiment, each participant had to individually organize
the provided set of artifacts concerning the structure that she deemed most suitable.
Afterwards, the participants answered a questionnaire about their chosen file
structure, e.g. “How do you rate the understandability of your file structure?”,
using a seven-point Likert scale.
In the second part of the experiment, the participants presented the file structures
they created to each other. Afterwards, they were asked to discuss how a common
file structure should look and created it using the provided files. Afterwards, they
answered a subset of the questionnaire a second time, e.g. “How do you rate the
understandability of the new file structure?” and “Do you think that the under-
standability of the common file structure was increased due to the collaboration of
you and your colleagues?”, both using a Likert scale. Overall, the experiment had a
duration of two hours for each group. In the following, we summarize preliminary
observations and findings:
• D-School students recognized that the provided artifacts can be assigned to the
design thinking method (Plattner et al. 2009). Thus, they created folders for each
process step.
• Computer science students recognized parts of the software developments pro-
cess and used a suitable file system structure.
• If they were not able to assign the artifact to a distinct process step, all students
moved the corresponding artifacts into a folder named “other” or “archive”.
Connecting Designing and Engineering Activities 173
• During the discussion part of the experiment, the D-School student briefly
explained the design thinking method to the computer science students using
the created file structure. In general, the computer science students agree that this
structure is the most convenient.
• In general, different digital artifacts often capture the same analog artifact,
e.g. post-its. The students often discussed how to deal with such redundant
information. Either they only kept one single final overview image and deleted
all redundant pictures in-between or they kept all of them to be able to recover
the evolution of artifacts later on in more detail.
• As opposed to the design thinking students, the software engineering students
paid attention to details such as case sensitivity, underscores and whitespaces.
• Voting results were often encoded within file and folder names.
• Ranking of files and folders was enforced by employing prefixes such as
numbers to encode for example the chronological order of design thinking
method steps.
• The students rarely deleted, renamed, or duplicated any of the files.
• The practices we observed most frequently were:
• Moving files into folders and only duplicating them if they can be assigned to
different folders or, alternatively, creating links to single files instead.
• Sorting files based on their type (e.g., image, text, or presentation) and
recovering relationships between artifacts, afterwards.
These preliminary results indicate that design thinkers and engineers organize
artifacts in conformance to the process they are accustomed to – either by following
the design thinking method or the software development process of their choice.
Further, the results indicate that engineers agree to the file structure proposed by
design thinkers as the most suitable after having been introduced to the design
thinking method. Nonetheless, both procedures should be supported to bridge the
informal world of design thinking and the formal world of engineering.
5.3 Summary
take the major decisions for granted.” The authors conclude that “the result is a
document useful to people who know the system well, but impenetrable for
newcomers”. Therefore, documentation should be created along the design
thinkers’ idealized journey from the authors’ point of view.
6 Approach
Gotel et al. introduce the concept of sign makers (Gotel and Morris 2011). Applying
this concept to our documentation challenge, design thinkers should act as sign
makers. Later on, engineers can then rely on the created signs to trace decisions
back to their underlying rationale. In design thinking, signs about artifacts, method
steps and contexts can be made. Depending on design thinkers’ profession, also
technical signs can be created. Further, these signs have to be made permanent
otherwise they are worthless.
6.1 Architecture
Fig. 17 Envisioned
documentation framework
The presentation framework has to be easily adaptable since design thinkers are
usually among the early adopters of new tools and services. Therefore, different
software tools should be pluggable, e.g. Dropbox and GoogleDocs. Besides soft-
ware, also new hardware tools should be easily pluggable, e.g. Tele-Board
(Gumienny et al. 2012) to make use of electronic sticky notes.
To encourage design thinkers to document and make this task more exciting,
games should be provided which makes documenting a less boring task. For
176 T. Beyhl et al.
example, we currently implement a verification game (von Ahn and Dabbish 2004)
for tagging images such as photos of post-its collaboratively. Within this game,
independent players assign tags to the presented image until a first match is found.
Due to this collaborative task, a common agreement is established and the presented
Connecting Designing and Engineering Activities 177
images get a semantic based on the assigned tags. Later on, these tags can be used
by design thinkers and engineers to look up information. This game makes
non-searchable artifacts searchable. Figure 18 depicts a running game. The player
on top has already guessed the word “clock”, whereas the player below is guessing
the same word. Besides making documenting a less boring task, this verification
game exploits humans’ capabilities to interpret images and avoids difficult and
error-prone handwriting recognition. Moreover, design thinkers communicate their
ideas visually. Subsequently, post-its only contain few texts and handwriting
recognition would be inconsequential.
10
Quick Response
178 T. Beyhl et al.
After enough signs have been collected to re-establish the design thinkers’ track,
traceability techniques can be applied to follow the lifecycle of artifacts and
enhance the overall quality of the documentation. To make the gathered informa-
tion useful for engineers, an active repository combines the collected signs and
enriches the gathered information in the background. Therefore, it performs
analyses regarding the creation, validation/maintenance and deletion of traceability
links. Already established traceability links might be input for additional analyses
concerning the combination of traceability links. These analyses are performed in
the background and the results are stored in the repository and can then be queried
by engineers and design thinkers. For example, if the repository already contains
the information that the design thinkers brainstormed or voted at a certain point in
time (e.g. via position tracking) and, additionally, the design thinkers uploaded a
11
Radio-Frequency Identification
Connecting Designing and Engineering Activities 179
With the help of the reasoning framework, engineers are able to trace an artifact
back to its creation and to follow its evolution. This tracing enables engineers to get
the rationale behind design decisions within the underlying context. Therefore, they
are able to recover the reasons for design decisions and make well-informed
decisions during engineering. With such additional information at hand, engineers
have a better chance of creating the desired product within engineering constraints.
During the first year of our research project, we realized that the challenge is more
manifold than we thought at the beginning of the project. Therefore, we reframed
the problem and broadened the scope of our research. It turned out, that the first
challenge is to get design thinkers to document, especially in educational settings,
to evaluate our concepts concerning traceability in design thinking projects. The
second challenge is to apply traceability techniques and combine their results to
reduce uncertainties in the documentation. By doing so, the third challenge of
enabling engineers to create the desired products within engineering constraints
can be tackled.
In the forthcoming year, we will work on providing evidence for our hypotheses
concerning the benefits of traceability in design thinking and engineering activities.
Currently, our bachelor project “From Creative Ideas to Well-Founded Engineer-
ing” elicits requirements for the educational context and works on a first prototype
of the envisioned documentation framework with a focus on the presentation
framework. We are going to transfer approved concepts to professional settings
later on.
180 T. Beyhl et al.
Acknowledgement The authors are grateful for the input of Alexander Renneberg (D-LABS
GmbH) and our student assistants Josephine Harzmann, Christoph Kühnl, Manuel Hegner and
Lukas Pirl. The authors are also grateful for the input of Claudia Nicolai (D-School Potsdam),
Harald Gögl (D-School Potsdam) and our bachelor project team “From Creative Ideas to Well-
Founded Engineering”. The authors also thank Raja Gumienny for her support in preparing our
experiments.
References
von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: CHI’04: Proceedings of
the SIGCHI conference on human factors in computing systems. ACM Press, Vienna, Austria
Winkler S, von Pilgrim J (2010) A survey of traceability in requirements engineering and model-
driven development. Softw Syst Model 9(4):529–565
Ziv H, Richardson D, Klosch R (1997) The uncertainty principle in software engineering. In: 19th
international conference on software engineering (ICSE’97)
A Research Plan for the Integration of Design
Thinking with Large Scale Software
Development Projects
Abstract Design Thinking and agile software development processes are both
widely adopted by innovative companies that try to create products with maximum
end user value. Usually, the adopting companies work in smaller teams that tackle
projects of manageable size, but huge companies are increasingly trying to adopt
these methods in their large-scale projects as well. The main impediments for the
adoption of the aforementioned methods in such settings are the strict requirements
which, for example, enterprise software vendors have to fulfill. The need for
complying with a plethora of mandatory product standards and providing compre-
hensive documentation of the development processes is not integrated naturally into
the methodologies and, hence, tend to weaken their innovative potential. This
chapter outlines our research agenda for a process model that combines agile
software development processes, such as Scrum or Extreme Programming, with
Design Thinking activities, while trying to maintain compliance with the previously
mentioned product and process standards. Our initial process model is based on a
series of expert interviews with people that previously applied Design Thinking in
large companies, as well as on a thorough review of related work about similar
approaches. We will further outline our evaluation process. This centers on a
project-based university course that transfers the idea of Stanford’s ME310 to a
computer-science-only setting. Investigating the teams’ virtual collaboration
activities, but also using traditional research methods, such as questionnaires and
interviews, helps us to continuously improve our process model and prepare it for
future test-runs within large partner companies.
1 Introduction
Over the past years, Design Thinking has evolved into a powerful methodology to
initiate and implement successful innovation. It especially excels in generating
ideas and solutions that are beyond the incremental scope used by many companies.
To achieve such radical innovation, Design Thinking is radical in its demands on
collaboration, interdisciplinary teams, user centrism, and continuous alternation
between divergent and convergent thinking. Following the example of the Stanford
d.school,1 similar schools were initiated worldwide,2,3,4,5 to teach Design Thinking
to students and professionals. Design agencies like IDEO not only use Design
Thinking to reinvent or innovate their customer products, they also brought Design
Thinking into companies like SAP (Savvas 2012), Microsoft (Lund 2011) or Apple
(Thomke and Feinberg 2010). Furthermore, various articles in business and man-
agement journals (BusinessWeek 2004) (Beckman and Barry 2007) show that today
Design Thinking is generally accepted as an innovation method.
Today, software engineering, especially for global enterprise applications, is
becoming increasingly complex. Customers demand features faster and ideally
want them tailored to their needs. Standards for business software, e.g., for security
and accessibility, lead to a large amount of non-functional requirements, while laws
and regulations from around the world need to be considered and integrated.
Especially larger software companies introduce additional constraints on their
teams and products to ensure high quality software and high availability of the
software and its provided services. In this restricted setting, software companies are
still expected to deliver innovative applications. In the past decade the software
industry has evolved its engineering processes from strict waterfall process
frameworks to more flexible agile or lean process models including methodologies
such as Scrum (Schwaber and Beedle 2001) or Kanban (Anderson 2010). This
change allows a faster reaction to changing requirements but does not automatically
foster innovation. As Design Thinking has proven to be an effective methodology to
create innovative ideas, the software industry is starting to include it into its
processes. However, there has been little research in this area so far and the existing
research is mainly comprised of interviews with Design Thinking workshop
participants about their experience and single project case studies.
In this chapter, we introduce our research efforts to uncover useful ways of
integrating Design Thinking into software development processes and to identify at
which points in development cycles Design Thinking can excel and which tools and
methods will enable the development teams to create innovative software that
1
d.school Stanford – http://dschool.stanford.edu/.
2
Aalto Design Factory – http://www.aaltodesignfactory.fi/.
3
Aalto Tongji Design Factory – http://sfc.tongji.edu.cn.
4
HPI School of Design Thinking Potsdam – http://www.hpi.uni-potsdam.de/d_school/home.html?
5
d.school Paris-Est d.school – http://www.facebook.com/d.schoolparis
A Research Plan for the Integration of Design Thinking with Large Scale. . . 185
fulfills all necessary constraints and standards. The rest of this chapter is structured
as follows:
To get an idea where the integration of Design Thinking in software companies
currently stands, the next section provides an overview of existing integration
approaches for Design Thinking and software development from both research
and industry. Based on this overview, we explain our own research efforts and
ideas in the third section of this chapter. The fourth section provides an overview of
a Design Thinking computer science course that we are creating in order to evaluate
our ideas and findings. How this evaluation will take place is described in the fifth
section. The chapter concludes with a summary.
for children. These examples show that Design Thinking can be used to success-
fully develop software products. However, creating software with the help of
Design Thinking is not limited to design agencies like IDEO. Big software vendors
like SAP or Microsoft are making efforts to integrate Design Thinking into their
companies. During this year, SAP started creating so-called “app houses” around
the world, where teams of “developer designers” experience a startup like process,
resembling the integrated project model approach, to create small innovative apps
for SAP (Savvas 2012). Creating subdivisions for innovation and research outside
the actual company is a common approach and results in so-called innovation and
research labs (Chartron et al. 2011). While this enables the labs to make use of
processes and work flows that would not be used inside the main company, it also
hinders the adoption of innovative processes in the rest of the company (Lindberg
et al. 2011). Moreover, Brenner et al. found that this approach allows companies to
be flexible and react quickly to changing demands but also results in an “IT of two
speeds” (Brenner et al. 2012), separating the slow development in the main
company and the fast and innovative development in the company’s innovation
divisions.
This separation also reflects the two degrees of complexity inherent in today’s IT
companies. Small-scale applications and research projects do not have to comply
with all processes and restrictions that are present in large-scale enterprise software
development. The development of large and complex enterprise software systems
such as Enterprise Resource Planning (ERP), or Customer Relationship Manage-
ment (CRM) systems is subject to numerous standards, legal constraints, and
non-functional requirements, such as accessibility, availability and security
(ISO/IEC 2005). To ensure compliance with these standards, software vendors
create in-house quality assurance processes that every software product has to
follow. Furthermore, larger software systems are developed in teams of hundreds
or thousands of developers split into several smaller teams, which means that the
chosen development processes must be scaled to accommodate this situation
(cp. (Ambler 2009)). Thus the development of large enterprise applications poses
unique problems that are not addressed by current efforts to integrate Design
Thinking into software development, whereby Design Thinking is mainly
implemented by smaller size projects and teams.
The reason for this could be that the large number of additional requirements and
process restrictions, due to the complexity of the software, the standards, and the
team sizes restrict the creativity and the freedom of thinking, which is necessary in
Design Thinking. Another aspect might be that Design Thinking by default allows
wild ideas that can be far away from what can be implemented in the company.
Both are important aspects to consider when trying to integrate Design Thinking
into enterprise software development processes. The existence of constructs such as
innovation labs and app houses implies that innovation is desired by the industry
but a process framework that integrates innovation methods like Design Thinking
into large organizations is still to be found. SAP tried to bring Design Thinking into
all their divisions by adopting an approach that resembles the split project model
identified by Lindberg et al. (2012). As early as 2005, the company established a
A Research Plan for the Integration of Design Thinking with Large Scale. . . 187
so-called Design Services Team (DST) whose purpose is “to improve the design of
SAP software solutions as well as provide the organization with the means to scale
up its adoption of Design Thinking” (Holloway 2009). Hildenbrand and Meyer took
a different approach in a single project case study with SAP (Hildenbrand and
Meyer 2012). They have successfully implemented a first prototypical process that
follows the unified project model and integrates Design Thinking and Scrum into
one process. They started their project with a Design Thinking phase that was
followed by a development phase with Scrum and interleaved Design Thinking
activities as needed. Hildenbrand and Meyer found that agile or lean development
and Design Thinking have enough in common to allow a mixed mindset in their
team and a smooth transition between the two. They identified user story maps as an
especially helpful tool to bridge the gap between insights and ideas and actual
product requirements during implementation. The case study provides a glimpse of
what Design Thinking activities can achieve in software development. The pro-
posed process, however, still needs to be further refined and tested with more
project teams and different size projects. Furthermore, the process does not con-
sider the additional requirements and restrictions that apply to enterprise software,
an aspect that we believe to be critical for the success of Design Thinking in
enterprise software development.
3 Research Agenda
As shown in the previous section, large software companies are making efforts to
integrate Design Thinking into their development processes, but to a large degree
the integration is taking place in smaller scale projects in innovation labs. With our
research, we aim to identify currently existing approaches and methods for the
integration of Design Thinking into software engineering processes within real
companies and qualitatively assess their strengths and weaknesses. Based on
these findings, we aim to create a prototypical development process framework
that integrates Design Thinking activities into large-scale software development
processes, such as scaled Scrum or Lean Development, while still respecting legal
and organizational restrictions. This chapter provides an overview of our research
strategies to achieve this goal.
Design Thinking offers a huge number of creative tools and methods that can help
at different stages during the process. The following list provides a small excerpt of
some of these methods and their usage during different stages of the Design
Thinking process:
A Research Plan for the Integration of Design Thinking with Large Scale. . . 189
Similar to Hildenbrand and Meyer (Hildenbrand and Meyer 2012), we believe that
Design Thinking and Agile Development can complement each other. Together
with the Institute of Information Management at the University of St.Gallen,6 we
are currently working on a prototypical process that combines Design Thinking and
Scrum. We follow the general idea of using Design Thinking to find, define, or
refine requirements for the software in an initial project phase. In this phase, we aim
to make Design Thinking activities projectable by planning them in form of sprints,
providing a definition of done and estimating the effort of the planned tasks
beforehand. The following development phase will follow a Scrum-like process
with interwoven Design Thinking activities, such as user testing and storytelling, to
keep the team and the development user centered. As this is only a first prototype,
we will test and refine it to develop a process framework, which allows companies
to fit/scale the process to their specific needs. The process will also serve as a basis
to evaluate different team sizes and setups during software development with
Design Thinking. Possible project setups include the approaches proposed by
Lindberg et al. (2012):
6
IWI HSG – http://www.iwi.unisg.ch/
190 T. Kowark et al.
• The project will be started with a specialized Design Thinking team during the
initial phase and will later serve as the Product owner during development.
• The project will be started with a specialized Design Thinking team that will be
gradually changed into the development team by dropping Design Thinking
experts and integrating developers as needed.
• A team of developers trained in Design Thinking will work on the project the
entire time.
In addition to these setups we will investigate the effects of guidance through
Design Thinking coaches or a “Design Thinking master”.
Based on the initial insights gathered through our interviews and analysis of related
work, we aim at creating a prototypical project course that helps to assess different
approaches for the integration of Design Thinking and software development
processes. To minimize the impact of project failure within those experiments,
they will be placed in an educational setting, first, before being repeated in industry
settings.
Previous case studies have shown that teaching Design Thinking methodologies
to software engineering teams poses various problems. Domschke et al. (2009)
conclude that the common background and shared knowledge about methods and
tools, as commonly found in homogenous teams, hinder the acceptance of new
methodologies taught in Design Thinking education. They further observed that
software engineers tend to refuse usage of methods and tools that do not yield
technological outcomes but focus on intangible findings such as user needs. We
want to overcome this gap by creating a project-based software engineering course
based on the ME310 teaching approach. We henceforth name this course CS310.
Fig. 1 Phases of the project course and their relation to employed prototyping tools and methods
Conceptually, the three described phases will be kept within CS310 as well.
Prototyping assignments, however, need to be adopted – especially in the second
and third phase. Figure 1 outlines potential prototyping tools for the different
phases. During the initial phases, ME310 and CS310 teams will most likely not
differ vastly in their choice of tools. Fast and rapid prototyping with low-fidelity
tools (paper, cardboard, Lego, role playing, etc.) is the primary working mode
during this phase. Critical functions, however, potentially require some software
development effort to realistically prove that certain functionality could be
implemented given the prescribed technologies or systems.
Beginning with the second phase, the differences between projects of either
course are increasing. Instead of creating functional system prototypes out of
physical components, which at this point could still be connected by plain duct-
tape, an analogous working mode needs to be found for software teams. Potential
approaches could be the connection of computer supported mockups with briefly
prototyped backend functionality or the manual connection of two separate soft-
ware prototypes by transferring data manually between the systems instead of, for
example, already programming the data transfer.
In the final phase, prototype building in CS310 becomes a pure software devel-
opment effort. Using standard agile process models, such as Scrum, the students can
structure their development efforts and plan ahead to deliver the required prototypes
A Research Plan for the Integration of Design Thinking with Large Scale. . . 193
“X is finished”, “Penultimate”, and, in the end, the final prototype. During this phase,
it is crucial that realistic tools are being used to lower the transfer effort back to
the corporate sponsors. If, for example, the teams would create their prototype using
programming languages and frameworks that are not also used by their partner
company, the entire prototype would have to be re-implemented after the project.
If the teams are not familiar with the required tools or languages, additional training
needs to be provided. Following the rationale for ME310’s paper robot prototypes,
an equivalent exercise could be used midway through the project to get the teams
accustomed to the tools before the final phase begins.
The usage of realistic tools has another benefit: it provides a showcase for the
corporate sponsor and outlines how they could apply the ME310/CS310 approach
within their internal projects as well. This aspect of realism can be further fostered
by applying some of the requirements presented in the related work section and lets
the teams best implement them into the prototypes or at least provide concrete
concepts for required integration efforts.
The first iteration of this course is performed using the integrated project approach,
as it fits naturally into the project setup described above. An initial project was
started in October 2012 and features collaboration between SAP, HPI, and the
University of St. Gallen. This initial test project aims at evaluating both the overall
teaching approach and initial approaches for the integration of software develop-
ment processes and Design Thinking, as presented in the previous section. In future
iterations of the course, we want to integrate a bigger number of teams to be able to
compare different integration approaches with each other and/or try similar
approaches in different project settings.
5 Evaluation
In previous project years, we created a platform that has aimed to be a simple and
readily accessible way of collecting and analyzing information about artifacts of
virtual team collaboration, such as wiki pages, emails, bug tracking items, or source
code revisions (Kowark et al. 2011a; Uflacker et al. 2010). These artifacts have
proven to be effective in providing viable indicators for the adherence-to or
negligence-of both Design Thinking and software engineering principles (Kowark
and Plattner 2012; Uflacker 2010). In the following we outline how we plan to apply
this tool and its recently added features within this research project.
Fig. 2 Research Cycle used to continuously evaluate and improve the process model (Adopted
from Baskerville 1999)
Networks (TCN) (Uflacker and Zeier 2010). These networks are built upon
ontologies that represent concepts of different collaboration tools, such as wikis,
email lists, or version control systems. All nodes can be linked to each other through
relations and, thus, provide a comprehensive overview of observable virtual collab-
oration activities across tool boundaries.
Once the networks have been created, they can be analyzed in a multitude of
ways. For example, standard social network analysis functionality incorporated into
the underlying graph database7 can be used to determine how closely people
collaborate by counting the number of artifacts that they are jointly connected
to. Through this and other analyses of the graphs, reoccurring sequences of collab-
oration behavior can be identified and their impact on process or product metrics, as
captured along the collaboration activity, can be statistically evaluated.
In (Kowark et al. 2011a), we introduced a concept that allows generalizing such
findings made within single projects and also detecting occurrences of similar
behavior in other projects. In order to enable such cross-project analyses, we
introduced the “Matcher” component to the general platform architecture (see
Fig. 3). This component is able to take characteristics of the respective source
and target networks into account and adopt the data queries accordingly. If, for
example, a network pattern was initially observed in a project that uses Scrum as the
development process, role names might differ from projects that use other process
models. The Matcher component detects such differences and either uses previ-
ously defined equivalence relations or asks the users to provide new ones. Addi-
tionally, cardinalities and time constraints are translated to account for differences
in project size and duration.
With the help of this mechanism, it is possible to transfer knowledge about
potentially beneficial or detrimental virtual collaboration signatures between
projects. Each user of the system can graphically model those signatures, categorize
them to simplify later usage, and publish them in a central repository that is
7
AllegroGraph – http://franz.com/agraph/allegrograph/
196 T. Kowark et al.
Fig. 3 AnalyzeD architecture overview showing the local analysis platforms that are connected
with a central repository for virtual collaboration behavior
The following section provides a practical example for the possibilities of virtual
collaboration analysis. It is based on a software engineering course at HPI (Kowark
et al. 2011b). In this course, 6–8 student teams jointly develop an enterprise system.
They are provided with an extensive groupware infrastructure and Scrum is pre-
scribed as their development process. Being an agile software development process,
A Research Plan for the Integration of Design Thinking with Large Scale. . . 197
Scrum, however, does not focus on tools but on individuals and interactions. Hence,
while we provide them with the groupware tools, the teams are supposed to find the
most suitable solutions for their work instead of dogmatically adhering to existing
systems.
During post-hoc analysis of the first iterations of the course, we made the
following observations: when teams started to focus on physical Scrum boards
for ticket maintenance, the digital versions of the tickets were increasingly updated
in bulk by designated members of the development teams. This permits maintaining
an overview of the current progress directly within the team office. At the same time
it makes it possible to share the progress with other teams and the teaching staff
through the groupware system with minimal overhead. In order to detect this
working behavior in subsequent projects, we used the previously described
modeling approach to model two contrary graph patterns.
Figure 4 depicts a ticket being changed by its owner. The owner here is not the
person that created the ticket – which would often be the Product Owner – but the
person who was assigned to implement the described feature or perform the desired
task. We assumed that such behavior would mostly be exploited at the beginning of
the project and decrease over time when small changes are carried out exclusively
on the physical Scrum board.
The second behavior – updates by other team members that do not own the
ticket – is depicted in Fig. 5. It is assumed to increase during the project as more
changes are performed on the physical Scrum board and designated team members
start to update the digital tickets every couple of days for their teammates. Figure 6
shows the detected number of occurrences during the project for an entire develop-
ment group. It is clear that the assumptions about the occurrence rate of the
behaviors generally held true. After many developers initially edited their own
tickets on a regular basis, turnaround was reached midway through the project,
where changes by non-owners became more frequent.
Despite their seemingly strong indication towards a real change in the working
behavior of the teams, virtual collaboration analysis can only provide circumstan-
tial evidence and should be accompanied by intensive manual observations. Firstly,
digital collaboration constitutes only a fraction of overall collaboration activity.
Secondly, if the subjects under investigation know that their virtual collaboration
behavior is being analyzed, observer effects might come into play, with people (un)
consciously trying to generate “favorable” digital signatures. Hence, such analyses
should be used mainly as a project management aid that helps to detect potentially
detrimental or beneficial behavior and serve as a starting point for further, more
targeted manual investigations. We will use analyzeD in such a manner and with
the help of the “Visual Creator” for graph sequences, we are enabled to frequently
model and adapt assumptions about the students’ working behavior and test their
influences on collected process and product metrics.
198 T. Kowark et al.
Fig. 5 Behavior 2. A ticket is changed by a team member that is not the owner of the ticket
Fig. 6 Development of occurrences of the behaviors under investigation over the course of the
project
The previous examples were solely focused on the structural analysis of team
collaboration networks. In a recent master’s thesis, we also investigated possible
applications for an analysis of node contents. Specifically, the thesis used latent
Dirichlet Allocation (LDA) (Blei et al. 2003), a statistical model, to automatically
A Research Plan for the Integration of Design Thinking with Large Scale. . . 199
Fig. 7 The generative process of LDA (Adapted from Blei et al. 2003)
determine which topics are addressed by the different collaboration artifacts. The
algorithm takes a given input document (e.g., email bodies, source code revision
commit messages, wiki page contents) and detects potentially contained topics
through an analysis of word frequency (see Fig. 7). For a complete description of
the algorithm, the interested reader is referred to the original thesis (Gehrer 2012).
Building on the possibilities that topic detection in collaboration artefacts offers,
we created a prototypical topic explorer component for analyzeD (see Fig. 8 for
screenshot). This component allows filtering the plethora of available collaboration
artifacts to only those that exhibit a chosen topic. Furthermore, the implementation
is able to provide a heatmap for the chosen topics. This heatmap, which is shown on
the top left corner of the screenshot, allows the possibility of identifying during
which phases of the project certain topics were “hot” and when their importance
decreases. This could equally help to determine which ideas within a project were
carried out from initial prototyping phases to the final implementation and which
were discarded along the way of the process.
The histogram in the top right corner displays the distribution of a topic amongst
the different collaboration channels. In the example shown, it is possible to see that
implementation efforts (marked yellow) only started after initial documentation and
planning efforts (emails, tickets, wiki pages – red, green, and blue). This function-
ality further aids the structural analysis described in the previous section, as we are
not only able to see when the project members started implementations in general
but can break this down to a per-topic-level.
Finally, as the TCN link artifacts to their creators or editors, the additional
knowledge about exhibited topics makes it possible to determine potential experts
for certain topics. The “Experts” area in the lower left corner prototypically shows
200 T. Kowark et al.
how it is easily possible to determine that Person 37 and 86 potentially are most
knowledgeable about the given topic or at least might be good initial contact
persons.
6 Conclusion
References
Ambler SW (2007) Survey Says. Agile has crossed the chasm. Dr Dobbs J 32(8):59–61
Ambler SW (2009) The agile scaling model (ASM): adapting agile methods for complex
environments. Retrieved from ftp://ftp.software.ibm.com/common/ssi/sa/wh/n/
raw14204usen/RAW14204USEN.PDF
Anderson DJ (2010) Kanban: successful evolutionary change for your technology business. Blue
Hole Press, Sequim
Baskerville RL (1999) Investigating information systems with action research. Commun AIS 2
(3es):4
Beckman SL, Barry M (2007) Innovation as a learning process: embedding design thinking. Calif
Manage Rev 50(1):25–56
Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022.
Retrieved from http://dl.acm.org/citation.cfm?id¼944937
Brenner W, Uebernickel F, Wulf J, Zelt S, Györy A, Heym M, Warnke A (2012) Strategies for
application management services. Retrieved from http://www.navisco.com/download/
casestudies/navisco_strategies_ams.pdf
BusinessWeek (2004) The Power of Design, {BusinessWeek} Cover Story. BusinessWeek
202 T. Kowark et al.
Carleton T, Leifer L (2009) Stanford’s {ME310} course as an evolution of engineering design. In:
Proceedings of the 19th CIRP design conference–competitive design. Cranfield, United
Kingdom
Chartron J, Steinhoff DF, Paulssen PDM (2011) User Driven Innovation@Telekom Laboratories –
Das Innovationsforum. Ideenmanagement 1-2011, 14–17
Domschke M, Bog A, Zeier A (2009) Teaching design thinking to software engineers: two future-
oriented curriculum case studies. In: 26th ICSID World Design Congress. Singapore
Gehrer R (2012, July). Extraction of emerging topics from digital collaboration artifacts of
software engineering teams. Hasso Plattner Institute for IT Systems Engineering
Hildenbrand T, Meyer J (2012) Intertwining lean and design thinking: software product develop-
ment from empathy to shipment. In: Maedche A, Botzenhardt A, Neer L (eds) Software for
people. Springer Berlin Heidelberg, pp 217–237
Holloway M (2009) How tangible is your strategy? How design thinking can turn your strategy
into reality. J Bus Strat 30(2/3):50–56
IDEO (2009a) iPhone Applications textbar IDEO. Retrieved from http://www.ideo.com/work/
iphone-applications
IDEO (2009b) Sesame Street iPhone Apps IDEO. Retrieved from http://www.ideo.com/work/
sesame-street-iphone-apps/
ISO/IEC (2005) {ISO/IEC} 25000:2005 – Software engineering – Software product quality
requirements and evaluation (SQuaRE) – Guide to SQuaRE
Kowark T, Dobrigkeit P, Zeier A (2011a) Towards a shared repository for patterns in virtual team
collaboration. In: 5th international conference on new trends in information science and service
science. Macao, China
Kowark T, Müller J, Müller S, Zeier A (2011b) An educational testbed for the computational
analysis of collaboration in early stages of software development processes. In: Proceedings of
the 44th Hawaii international conference on system sciences (HICSS). Koloa, Kauai, HI, USA
Kowark T, Plattner H (2012) AnalyzeD: a shared tool for analyzing virtual team collaboration in
classroom software engineering projects. In: The 2012 international conference on frontiers in
education: computer science and computer engineering. Las Vegas, NV, USA
Lindberg T, Meinel C, Wagner R (2011) Design thinking: a fruitful concept for IT development?
In: Meinel C, Leifer L, Plattner H (eds) Design thinking research: studying Co-creation in
practice. Springer Berlin Heidelberg, pp 3–18
Lindberg T, Köppen E, Rauth I, Meinel C (2012) On the perception, adoption and implementation
of design thinking in the IT industry. Des Think Res 229–240
Lund A (2011) March 11: a case study of applying design thinking at Microsoft Corporation
Human Centered Design & Engineering. Retrieved from http://www.hcde.washington.edu/
nav-courses/521/win11/mar11
Plattner H, Meinel C, Weinberg U (2009) Design thinking. Mi-Verlag, Munich
Savvas A (2012, April) SAP focuses on product “desirability” through design to lure customers –
ComputerworldUK.com. Retrieved from http://www.computerworlduk.com/news/applications/
3402011/sap-focuses-on-product-desirability-through-design-lure-customers/
Schwaber K, Beedle M (2001) Agile software development with Scrum. Prentice Hall PTR, Upper
Saddle River
Thomke S, Feinberg B (2010) Design thinking and innovation at Apple. Harv Bus Sch 9-609-066
Uflacker M (2010) Monitoring virtual team collaboration: methods, applications, and experiences
in engineering design. Hasso Plattner Institute for IT Systems Engineering, Potsdam
Uflacker M, Zeier A (2010) A semantic network approach to analyzing virtual team interactions in
the early stages of conceptual design. Future Gener Comput Sys 27(1):88–99
Uflacker M, Kowark T, Zeier A (2010) An instrument for real-time design interaction capture and
analysis. In: Meinel C, Leifer L (eds) Design thinking: understand – improve – apply. Springer
(in print)
Sharing Knowledge Through Tangible
Models: Designing Kickoff Workshops for
Agile Software Development Projects
1 Introduction
There is a saying that to really build a great product, you have to build something
that you love to use yourself (Fried and Hansson 2010). This approach brought
about great innovations in consumer electronics and also software systems.
Products such as Google Maps, Twitter, Facebook or Delicious were created by
programmers to satisfy their personal needs. In these cases, the programmer has the
knowledge about what is desirable and what is technically feasible. Unfortunately,
for the vast majority of software programs this setting is not given. Business
software, written to support the operations of a company, is by definition no
consumer product. Thus, the programmer will never be the user of software that
does risk management, supply-chain management or accounting. The programmer
has the knowledge about technical feasibility, but the desirable functionality
originates from another person, the user of the system.
To enable software engineers to understand the requirements of the user, a
plethora of approaches was created in the last decades in software engineering.
As one example, the research area of requirements engineering (e.g. Alexander and
Stevens 2002; Davis et al. 2006) investigates how information is derived from the
user, documented and maintained towards the implementation of a software
program.
In the Design Thinking Research Program previous projects have investigated
paths for better requirements engineering. For example, the project by Prof. Giese
(PI) on scenario-based prototyping (Gabrysiak et al. 2009; Gabrysiak et al. 2011),
enabled the end user to simulate daily scenarios, in a role playing game to collect
information about the workflows and associated data. This information was then
used to enrich and validate requirements. The results are data models and behav-
ioral models that are checked for plausibility before they go to the further software
engineering process (Fig. 1).
In a previous project, we investigated a new approach to joint model building
together with the end users. In the case of Tangible Business Process Modeling
(TBPM) (Edelman et al. 2009; Luebbe and Weske 2011), business processes are the
activities performed in organizations. They are represented in graphical models to
enable better communication about them. They are key artifacts for the software
engineers as they also describe how people and IT systems interlink. Typically, the
end users communicate their knowledge about the process verbally. An analyst
afterwards represents this knowledge as a business process model. In the project on
TBPM the process model is co-created by the end users and the analyst acts as a
facilitator. He explains the concepts behind process modeling and the semantics of
the artifacts on the table. The resulting models are then used in the further software
engineering process. In his dissertation, Alexander Luebbe (Luebbe 2011) created a
methodology to involve domain experts more strongly in business process
modeling and has shown that it leads to more engaged end users and better validated
models.
Sharing Knowledge Through Tangible Models: Designing Kickoff Workshops for. . . 205
TBPM enables process analysts to co-create modeling artifacts together with end
users who have no modeling experience upfront. This collective modeling
facilitates and sharpens communication between the process participant and the
process analyst. As of now we have applied our methodology to the field of process
elicitation. It is our intention to also transfer the findings of the TBPM project to
other fields in software engineering, especially those which heavily rely on
modeling artifacts.
In verbal communication, information is likely to get lost or misinterpreted (Lin
et al. 2005). To ease this problem, software engineers have created modeling
languages, such as BPMN (OMG 2011) for business process modeling. It defines
a set of concepts, their relations and how they are visually represented. For
example, BPMN defines the concepts of activities that are in an ordered relationship
to each other. An activity is defined as a chunk of work that takes time to be
performed and is assigned to one responsible role. A role is another concept that
summarizes a set of abilities and, e.g. a clerk.
Typically such information is gathered by a specially trained analyst. The end
users, experts in the domain, can provide this information about their work envi-
ronment. The model is then created afterwards by the analyst. The TBPM project
has shown that it is possible to enable end users to directly create the process
models that are required for the software engineering project. Therefore, a tangible
incarnation of BPMN (OMG 2011), the process notation, was created and
evaluated. In a workshop, the analyst teaches the core concepts of process
modeling, the semantics of the shapes and facilitates the creation of the process
models. In that case, he can integrate the views of the people at the table into one
overall agreed model. The research on TBPM suggests (Luebbe 2011) that the
people learn more about the process when they model it together and they also
iterate the process model more often. Given these positive effects, one might ask:
Does this idea also work for other software engineering models?
206 M. Guentert et al.
On the most general level, the models that describe software systems can be
divided into behavioral models, data models and architectural models. Behavioral
models describe the dynamic in the system of which the software is part of. For
example, a business process is a behavioral model. It may contain human steps as
well as steps taken inside the software system. A data model describes how
information is structured. For example, an Entity-Relationship Diagrams (Chen
1976) describes the information attributes and their relationship. More concrete, an
address is an information attribute of a person. An address consists of street name,
postcode and country. Finally, architectural models describe which logical
components exist within a software system and how they interrelate.
Despite the three fundamental types of models in software engineering, there
exist a countless number of modeling languages developed in the history of
computer science to describe aspects of a software system. Each of these modeling
languages has been developed for a special purpose. Some modeling notations have
been developed by software companies as a unique selling point (e.g. Keller
et al. 1992). Others were developed by researchers to fill a particular need
(e.g. van der Aalst and ter Hofstede 2005). And again other modeling notations
have been developed by standardization committees (e.g. OMG 2011). These
languages are made for different audiences and show a different level of technical
detail. Furthermore, they embody a different set of modeling concepts even if their
core nature is of the same type. For example, the aforementioned process modeling
notation BPMN (OMG 2011) is a behavioral model, that also has the notion of data-
based decisions and system communication through systems collaborating using
message exchange. Thus, they have basic concepts from data modeling and archi-
tectural modeling but at their core they focus on behavioral models. Activity
diagrams (Dumas and ter Hofstede 2001), also a type of behavioral models, are
designed to describe internal system behavior and have no idea of interacting
systems. Thus, even within one fundamental modeling type there is a huge variety
of modeling languages. Barely any language is purely of only one type. But even
the same concepts are depicted differently, in different notations. To unify the
plethora of modeling notations, the unified modeling language (Fowler and Scott
2000) was introduced in the 1990s. It defines 13+ modeling notations for the
purpose of software engineering. Despite its adoption in teaching (Zeichick
2004), UML has not unified the scattered modeling world in the software engineer-
ing industry. Moreover, there are also alternative sets of modeling languages that
claim to cover all necessary aspects of software engineering (e.g. Gausemeier
et al. 2009) (Fig. 2).
Picking a modeling language or even a set of modeling languages is very
difficult because they are hard to compare. For business process models, a special
type of behavioral model, this was done based on their capabilities to express
frequently used modeling situations (Wohed et al. 2006; Russell et al. 2006). But
one could also argue that popularity should be the key to choosing the language.
Another option was offered by Moody (Moody 2009), who developed a metric to
measure the aesthetics of modeling languages based on the physical attributes of the
notation. It allows identifying more or less suitable languages based on objective
measures that he validated through experiments.
Sharing Knowledge Through Tangible Models: Designing Kickoff Workshops for. . . 207
Fig. 2 Fundamental
dimensions of modeling
languages. Each modeling
language focuses on one of
these dimensions but
typically is a mix of many
concepts describing
behavior, data structure and
architecture
The key point is that by means of user stories, requirements are formulated from
the perspective of the user rather than from the developer. Both this fact and the
short wording enable the customer to directly participate in the creation and
discussion of software requirements. User stories are a means to conserve the
Sharing Knowledge Through Tangible Models: Designing Kickoff Workshops for. . . 209
discussion about a requirement, but never to holistically specify it since this would
be contrary to the concept of short wording. Therefore in order to understand a user
story, a stakeholder should ideally have participated in the discussion around it.
User stories serve as primary models in the development process. As with TBPM,
they can directly be communicated with the customer and the customer is able to
express himself by way of these models. User stories can be aggregated to a common
subject, i.e., to a superordinate functionality, or decomposed if one is found to be too
general and that there are multiple stories behind it. User stories, however, only
reflect the conceptual view from the perspective of the respective user role.
Another important concept for the context of user stories is decisions. Consider
the following user stories: “As a logistics clerk I want to print a shipping label so
that the goods can be handed over to the post office.” (C) and “As a logistics clerk I
want to cancel the order if the goods are not available.” (D). It is valuable
information whether C and D are both executed or if just one is performed. The
relationship can be more complex also, e.g., first C is executed and then –
depending on the result of C – D may possibly be executed, but D is never executed
before C. This type of information is not captured in user stories.
The elaborated concepts bring in the process perspective. We see a great potential
to enrich a collection of user stories with this additional information. This results in
a sharpening of the communication between developers and customers. We propose
depicting system processes by means of user stories and their order relationship,
furthermore considering the concepts of decisions and independence. Requirements
can thereby be specified more precisely without introducing a significant overhead
of other model types.
Since user stories are formulated in natural language, their underlying concept
and mechanism can be communicated with the customer in a rather straightforward
and effortless fashion. No technical pre-knowledge is necessary to understand the
formalism and after the customer has had the chance to become familiar with the
mode of thought, he will soon gain confidence to directly participate in their
creation. The experience of agile practitioners confirms this notion.
With TBPM, referring to both the toolset and methodology, we have shown, on
the other hand, that it is also possible for end users, i.e. domain experts, to express
their knowledge by means of process models (Luebbe and Weske 2011). If the
concepts of process modeling are introduced iteratively, that is step-by-step rather
than all at once, end users with no former experience about the “process perspec-
tive” are able to follow the discussion and even create their own process models.
Seeing the potential that both model types are potentially being understood by
the customer, we believe that it is reasonable to combine user stories with concepts
from process modeling. We thereby create additional meaning and context infor-
mation to software requirements but are still able to communicate them between
developers and the customer. Baring in mind that agile software development
approaches continuously produce software prototypes as means for discussion,
we call our resulting models story prototypes. We intend to prepone the discussion
that arises by demoing a software prototype to co-created story prototypes in a stage
prior to implementation.
Sharing Knowledge Through Tangible Models: Designing Kickoff Workshops for. . . 211
How are IT projects actually started nowadays? In many cases a kickoff workshop is
organized where the key stakeholders meet (Smith and Reinertsen 1998). Also the
literature on agile software development approaches proposes holding an initial
requirements gathering workshop (Cohn 2004). But what exactly should happen
during such a workshop? The pertinent literature mentions few techniques and
provides little methodology.
212 M. Guentert et al.
Fig. 3 Kickoff workshop procedure. Macro group refers to activities carried out by all workshop
participants together and micro groups to activities distributed to smaller teams according to a
vertical decomposition of the system to be built
process context. The different system processes are thereby revealed and the
corresponding story prototypes are scoped. Furthermore, the user roles involved
in each system process become visible. The results are identified story prototypes
with an initial collection of associated requirements, together with role information.
Until now all activities have been carried out by the entire group of participants
together. We refer to this group as the macro group. Now the macro group is
divided into smaller teams – so called micro groups – to examine the identified
system processes in detail, whereby every team works out one particular story
prototype and presents it to the macro group afterwards. This approach follows the
concept of vertical decomposition. For each story prototype, if necessary,
overlapping user stories are grouped and consolidated. The resulting stories are
then transcribed to the tangible toolset (each story on one tangible shape) and laid
out on a table, like in the TBPM methodology. First their order relationship is
roughly determined. If two user stories are to be executed one after the other, they
are placed next to each other. This results in a linear sketch of the story prototype.
After a rough process layout has been prototyped, the concept of decisions is
introduced. Thereby information about exclusiveness is enriched to the linear
stream of requirements so far. OR-Gateways as found in the tangible toolset are
hence incorporated into the process and the paths of user stories rearranged
accordingly. As a next step the concept of parallel execution and synchronization
is explained. If in one and the same story prototype, user stories may be executed in
parallel, i.e., their execution order is independent, this shall be reflected by
AND-gateways of the tangible toolset. While reviewing the system process it
may be revealed that additional user stories, which have so far not been considered,
need to be reflected in the story prototype. By making the context transparent,, this
enriches the model in terms of completeness. The results of this phase are detailed
story prototypes complemented by control flow concepts.
The worked out story prototypes can now optionally be reviewed by other teams
and corrected if necessary. Each team should briefly present its results to the macro
group. As a final step for the workshop the identified requirements are prioritized
along the created models. High priority features become candidates for the first
implementation iteration to be prototyped and demoed afterwards. As a follow up to
the kickoff workshop an initial release planning is performed. Furthermore, the first
implementation sprint needs to be scoped. All relevant information should be
present as a result of the workshop. The actual planning, however, might in practice
be undertaken without the presence of the customer.
One could argue that creating models as a preparation for implementation brings
back the thought-pattern established in traditional requirements engineering,
whereby requirements were meant to be fixed in the beginning. However, consid-
ering potential need-finding iterations in agile development projects, it seems
Sharing Knowledge Through Tangible Models: Designing Kickoff Workshops for. . . 215
9 Conclusion
This, however, did not reveal much opportunity for research. We have instead
investigated other scenarios which might profit from the methodology
behind TBPM.
Agile software development approaches are gaining more and more meaning
nowadays. This led us to look at the role that models play in this context.
Requirements are primarily communicated by means of user stories and are
intended to be implemented in software prototypes that can be discussed with the
customer. An initial capturing of requirements is intentionally kept lean. After
interviewing agile practitioners, we identified the concept of need-finding
iterations, that is, initial implementation cycles which are solely carried out to
foster a better understanding of the problem, with the actual code thrown away
afterwards.
We tackle this problem by proposing a kickoff workshop in which the discussion
that arises with the demoing of a software prototype is conducted prior to
co-creation of models. We therefore introduce story prototypes which essentially
are user stories enriched with context information that reflect the process perspec-
tive behind it. A simple collection of user stories only expresses certain aspects of
the system to be built, but not their inter-relation. Rather than creating additional
models without the presence of the customer, we make use of a tangible toolset, lay
out the user stories on a table and enrich them with control flow concepts like
decisions and parallelism. Our contribution therefore is two-sided: On the one hand,
we introduce a kickoff workshop methodology which compiles different best
practices of group work techniques and the co-creation of models. The output of
the workshop, on the other hand, results in a new type of model which we call story
prototype.
In the next steps of our research we intend to develop project characteristics from
which we derive a set of metrics that allows us to quantify need-finding iterations.
We can then relate the desired productivity increase that arises by application of our
kickoff workshop technique to reference projects not making use of the technique.
We will collaborate with an industry partner currently following a suitable agile
development approach and gather data from “regular” projects. Then we incarnate
kickoff workshops in upcoming projects and evaluate the results against the data
from these reference projects.
References
Cohn M (2004) User stories applied: for agile software development. Addison-Wesley, Boston
Davis A et al (2006) Effectiveness of requirements elicitation techniques: empirical results derived
from a systematic review. In: 14th IEEE international conference requirements engineering, pp
179–188
Diehl M, Stroebe W (1987) Productivity loss in brainstorming groups: toward the solution of a
riddle. J Pers Soc Psychol 53(3):497
Dumas M, ter Hofstede A (2001) UML activity diagrams as a workflow specification language. In:
The unified modeling language. Modeling languages, concepts, and tools, S.76–90
Edelman J, Grosskopf A, Weske M (2009) Tangible business process modeling: a new approach.
In: Proceedings of the 17th international conference on engineering design, ICED’09
Fowler M, Scott K (2000) UML distilled: a brief guide to the standard object modeling language.
Addison-Wesley, Reading
Fried J, Hansson DH (2010) ReWork: change the way you work forever, Ebury
Gabrysiak G, Giese H, Seibel A (2009) Interactive visualization for elicitation and validationn of
requirements with scenario-based prototyping. In: Requirements engineering visualization
(REV), 2009 4th international workshop on, pp 41–45
Gabrysiak G, Giese H, Seibel A (2011) Towards next generation design thinking: scenario-based
prototyping for designing complex software systems with multiple users. Des Think 219–236
Gausemeier J, Plass C, Wenzelmann C (2009) Zukunftsorientierte Unternehmensgestaltung–-
Strategien, Geschäftsprozesse und IT-Systeme für die Produktion von morgen, Hanser
Fachbuch
Grosskopf A, Weske M (2010) On business process model reviews. In: Workshop proceedings of
ER-POIS: empirical research on process oriented information systems affiliated to CAiSE10,
pp 31–42
Keller G, Nüttgens M, Scheer A-W (1992) Semantische Prozessmodellierung auf der Grundlage
“Ereignisgesteuerter Prozessketten (EPK)”. Institut für Wirtschaftsinformatik, Saarbrücken
Laue R, Gadatsch A (2010) Measuring the understandability of business process models – are we
asking the right questions? In: Proceedings of the 6th international workshop on business
process design (BPD 2010)
Lin L, Geng X, Whinston AB (2005) A sender-receiver framework for knowledge transfer. MIS
Quart 29:197–219
Luebbe A (2011) Tangible business process modeling – design and evaluation of a process model
elicitation technique. Ph.D. dissertation, Hasso Plattner institute for IT systems engineering,
University of Potsdam
Luebbe A, Weske M (2011) Determining the effect of tangible business process modeling. In:
Plattner H, Meinel C, Leifer LJ (eds) Design thinking – studying co-creation in practice.
Springer, New York
Martin RC (2003) Agile software development: principles, patterns, and practices. Prentice Hall,
Upper Saddle River
Moody DL (2009) The “physics” of notations: toward a scientific basis for constructing visual
notations in software engineering. IEEE Trans Software Eng 35:756–779
Object Management Group (2011) Business process model and notation (BPMN) 2.0
Object Management Group (2012) Unified modeling language (UML) 2.5
Patig S (2008) A practical guide to testing the understandability of notations. In: Proceedings of
the 5th on Asia-Pacific conference on conceptual modelling, vol 79, pp 49–58
Pohl K (2010) Requirements engineering: fundamentals, principles, and techniques. Springer,
New York
Recker J, Dreiling A (2009) Does it really matter which process modeling grammar we use? An
experimental study on understanding process models. Inform Software Tech
Royce W (1970) Managing the development of large software systems. In: Proceedings of IEEE
WESCON, vol 26. Los Angeles
218 M. Guentert et al.
Russell N et al (2006) On the suitability of UML 2.0 activity diagrams for business process
modelling. In: Proceedings of the 3rd Asia-Pacific conference on conceptual modelling, vol
53, pp 95–104
Schwaber K (2004) Agile project management with Scrum. Microsoft Press, Redmond
Smith P, Reinertsen D (1998) Developing products in half the time: new rules, new tools. Wiley,
New York
Soni N, Soni A (2011) Agile release planning. Arete Solutions LLC
Steinberg DH, Palmer DW (2003) Extreme software engineering a hands-on approach. Prentice-
Hall, Upper Saddle River
Van Der Aalst WMP, Ter Hofstede AHM (2005) YAWL: yet another workflow language. Inf Syst
30(4):245–275
Wohed P et al (2006) On the suitability of bpmn for business process modelling. Lect Notes
Comput Sci 4102:161
Zeichick A (2004) UML adoption making strong progress. Software development times, 15 Aug
2004
How to Compare Performance in Program
Design Activities: Towards an Empirical
Evaluation of CoExist
1 Introduction
Programming involves more than continuously adding new lines of source code to a
program. It requires reasoning about already written source code and making
changes to it. Change is, for example, necessary to implement newly identified
functionality or to fix erroneous behavior. Since programmers know that tomorrow
they will have to make changes to the source code written today, they spend time on
structuring it well and making it easy to understand, so that the modifications of
tomorrow are easier to accomplish. In addition to the tasks of tomorrow,
programmers also restructure their source code to ease the implementation of
features of current interest (Beck and Andres 2004; Beck 1996; Fowler 1999).
Changing source code, however, involves risks because programmers can make
errors. A promising idea to simplify the code can suddenly turn out inappropriate
later in the process. Also, programmers can have flaws in their reasoning or can
unintentionally ignore relevant aspects. Making an error represents a risk because it
typically requires compensational activities that are time consuming and tedious to
carry out.
The recommended way to deal with the risks of change is to anticipate that errors
will be made and to continuously perform activities that keep compensation costs
low. This includes checking for errors early and often as well as maintaining a
safety net of stable development states one can fall back to (Beck and Andres 2004;
Apache Software 2009). However, such precautionary activities can only reduce
but not avoid the risk of tedious recovery work. Moreover, they can easily be
ignored and distract from the actual task at hand.
We have developed an alternative way to deal with the risks involved in
changing source code (Steinert et al. 2012). In contrast to asking for manual risk
and cost reduction, we propose the provision of tool support such as CoExist that
helps programmers deal with undesired consequences of their work. CoExist offers
dedicated support to withdraw changes, to recover knowledge from previous
development states, or to locate the cause of program failures even late in the
process. In this way, CoExist avoids that making an error implies tedious recovery.
We believe that such a tool-based approach is preferable over a manual method
based approach, which requires continuously following a set of practices. Research
findings on design and cognition suggest that externalizing ideas supports the
exploration of the problem and possible solutions. For example, creating prototypes
help pursue a line of thought and discover unforeseen implications (Goldschmidt
1991; Lim et al. 2008). Other findings suggest that the creation of external
representations inspires new associations (Kirsh 2010; Suwa and Tversky 2002).
CoExist enables programmers to make changes as they think of them, because
making errors does not imply additional costs.
We have designed an experiment to empirically examine our hypothesis that Co-
Exist better supports programmers in program design activities, particularly in
situations full of uncertainty. We have decided on a two by two mixed groups
factorial design. We repeatedly measure subject performance in two different tasks,
thereby having two groups of subjects, one using CoExist in addition to the regular
tools for the second task. We measure performance by coding the changes and
quantifying the effort for each category of change. In this report, we contribute the
design and its justification for an experiment to empirically examine programmers’
performance in program design activities.
How to Compare Performance in Program Design Activities: Towards an. . . 221
Fig. 1 Two different program snippets (in pseudo code) to fulfill the above stated need
decomposition eases creating and comprehending the individual parts, and thus
supports work on large complex systems (Parnas 1972).
However, how to decompose a program properly is not apparent from the
beginning. The qualities of the current decomposition are revealed during the
work on the program. The implementation of a particular aspect can be simple
and straight- forward or rather complicated. The resulting source code can appear
concise and right to the point or it can appear lengthy and disordered. Thus
programmers often contemplate the source code and check whether it is easy
enough to understand or still unnecessarily complex. They try to find elegant and
simple solutions to express the concerns of the problem. Thereby, they can always
introduce new modules (or programming constructs), such as selecting, or refining
existing ones to better fit the current needs.
The other reason why source code design matters is because programmers will have
to (re-) understand formerly written source code and will have to adapt it. It is
probably hard to find a piece of code that once written has never been changed
afterwards. Programmers have to work on source code written by others as well as
on source code they have written for some time. In both cases, they have to gain an
in-depth understanding of the source code, either because they have never seen it
before or because they cannot remember it in sufficient detail. In any case, the
design of the source code can either facilitate or impede gaining a proper under-
standing of its meaning and effects.
The reason that programmers revisit previously written source code is because
programming is a process of learning. It is not the same as assembling a car engine
from a set of pre-defined parts, and it is also different from constructing a house
according to a blueprint. Typically, every program is the first of its kind and the client
does not have any type of a blueprint nor a meaningful description of the problem
domain. Quite the contrary, programmers face a rather unstructured and little defined
problem domain, which is not surprising considering that clients typically do not
have the need to reason about their domain in a level of detail required for computer
programs. In addition, the set of requirements often changes faster than the entire
software solution can be accomplished.
How to Compare Performance in Program Design Activities: Towards an. . . 223
To illustrate how changing programs involves risks, we would like to report the
experience of a master’s student. The student had been working on a visualization
task using the Qt framework. At some point, he recognized that he had added
several methods that all work on the same data. He decided to extract a class
dedicated to this data structure and these methods. He created a new class called
SemanticLens as a subclass of a Qt class QEllipse.
After moving all methods in this new class and adapting his code to make proper
use of the new class, he contemplated his code and became skeptical about the
decision to subclass the Qt class. He remembered that subclassing has the drawback
of exposing the interface of the superclass to all clients who might make use of it,
thereby creating a dependency that can become difficult during maintenance tasks.
So, he decided to go for the delegation pattern instead. He was sure that delegation is
the right way to go. So, the student changed the superclass of the SemanticLense
class, added a field and accessor-methods to maintain a reference to a
QEllipse object and also added initialization code. He changed the methods in
SemanticLense class and made the required changes in the code using this class.
However, while looking at the result of making all these changes, the student
realized that his assumption had been wrong. He could now see that subclassing is
preferable over delegation in this situation because having access to the methods of
the super-class is actually useful in his program. As a consequence of this insight,
the student now faced the laborious task of manually withdrawing all the changes
previously made to replace subclassing with delegation. He had to identify the
relevant artifacts (files), and for each file to apply the undo command an undefined
number of times until reaching the desired state. Such tasks are not only time-
consuming but also tedious. Assuming that the changes made for the initial replace-
ment had taken several minutes, this meant that manually withdrawing also took a
few minutes. The required recovery work would have been even more laborious if
the student had made further changes before recognizing the error. Such situations
easily lead to irritation and are those programmers want to avoid.
One might argue that the student behaved incorrectly in this situation. He could
have finished implementing the functionality first before considering refactoring
the code. However, this could have led to more code needing adaptation later
on. One could also argue that the student could have made a local commit
(a checkpoint) before starting the refactoring, so that there would have been an
easy way back. However, the code was in an immediate state and his work was not
yet finished. He also has the very typical habit of thinking about commits only on
completion of a task.
One could also argue, that he should have thought more carefully about his ideas
before making any changes. Analyzing his idea in more depth might have been
enough to raise doubts. However, it is unclear how much deliberation is necessary
to avoid such errors. Moreover, too much thinking and questioning can easily lead
to counterproductivity.
How to Compare Performance in Program Design Activities: Towards an. . . 225
In contrast to the idea of careful upfront thinking, research findings on design and
cognition suggest that externalizing ideas supports the exploration of the problem and
possible solutions. For example, creating prototypes help pursue a line of thought
and discover unforeseen implications (Goldschmidt 1991; Lim et al. 2008). Other
findings suggest that the creation of external representations inspires new associations
(Kirsh 2010; Suwa and Tversky 2002). While these results suggest that programmers
should support their thinking by doing, this will inevitably imply the constitution
of errors.
The issues in changing source code are broader and more general than illustrated by
the story of the master’s student balancing the pros and cons of delegation and
subclassing. Writing and changing source code always involves the following risks:
• A promising idea unexpectedly turns out inappropriate and the programmer
wants to continue exploring a previous idea, but many parts of the source code
have already been modified.
• The improvement to one part of the program seems to affect the overall program
behavior in unexpected ways, but the programmer has difficulties to find out
which of the recent changes is causing the undesired behavior.
• The source code under current improvement turns out to be more complex than
the programmer expected and it is unclear how the code was previously working.
• Recent changes turn out to represent multiple independent improvements that
should be shared in separate increments.
The above listed issues are all well known. Literature describes them in detail,
teachers tell students about them, and every programmer has experienced them (and
still does) in some form or another. To reduce the risk of encountering such
situations, literature recommends following a structured and disciplined approach
and employing certain practices of work, which are, for example:
• To only work on one thing at a time, specifically a task or issue that you
understand in detail. Therefore, to break larger task items down into smaller
ones. This encourages staying on track and simplifies sharing your improvements
with others.
• Make sure you understand the items you work on, if not, consider breaking down
items into manageable parts, or consider consciously deciding on a phase of
experimenting that should be preceded by committing (and/or branching).
• Write tests and run them often and regularly, at best, after every small change.
This helps recognize bugs early in the process, and helps to pin down the cause
of the problem to a few recent changes.
• Employ a distributed version control system such as Git or Mercurial and make
frequent use of it by regularly committing small increments, which enables
going back to a stable state easily.
226 B. Steinert and R. Hirschfeld
The general pattern of these practices follows the contention that programmers
should anticipate that they will make errors and thus should perform precautionary
activities continuously and regularly in order to keep possible recovery costs low.
However, as the story about the student from above illustrates, the recommended
way of relying on best practices does not seem sufficient in all circumstances. There
are multiple reasons why this approach can fail in avoiding a need for tedious
recovery activities:
• Wrong assessment. As was the case with the student in the story above, programmers
can wrongly assess a programming situation. Applying best practices requires
interpretation and subjective judgment. While running tests often or working on
only one thing at a time are recommended, these are only guiding rules, and the
programmer has to decide what often means and what the appropriate granularity of
things is. Whenever a programmer feels confident about an idea and is unaware of
any risks, the possibility remains that the assessment is wrong. In this case, the
programmer will be unprepared for errors and will have to recover from them.
• Additional workload. Applying best practices is a continuous effort in addition to
the work on the problem domain. As such, it is easy to forget and to ignore,
in particular when one is caught up in creativity. Furthermore, mustering the
needed discipline and remembering “not to forget” require significant mental
effort (Allen 2001), even so practice helps reduce the required amount of attention.
• Upfront thinking. Following the recommended path requires programmers to
structure the work ahead of them. This is a consequence of the need for
interpretation and value judgment. For example, the practice to work on only
one thing at a time requires regular reflection about current and upcoming work
and assessing whether it should still be considered as “one thing” (a logical unit
of work). Becoming aware of potential future risks requires thinking about the
situation without working on it.
These limitations show that programmers will arrive, every now and then, in a
situation where they have to face tedious work to recover from a previously made
error, like the student in the story above.
1 2 3 4 5 6
Fig. 3 The user interface of a regular Squeak/Smalltalk programming environment (on the left),
and the version bar element added in CoExist (on the right)
Fig. 4 (From top left to top right) A programmer modifies source code which implicitly creates
items in the version bar. The programmer decides to withdraw several changes by going back to a
previous version (bottom left), and continues working and creating new changes, which will appear
on a new implicitly created branch (bottom right)
versioning and to allow for selecting previous versions, we have added a version bar
(timeline) to the user interface of the programming environment (Fig. 3).
By continuously preserving intermediate development states, CoExist makes it
easy for programmers to go back to a previous development state and to start over
as shown in Fig. 4. Starting over from a former development state will implicitly
create a new branch of versions. This preserves the changes the programmers want
to withdraw, as they might be of use later on.
228 B. Steinert and R. Hirschfeld
Fig. 5 Hovering shows which source code element has been changed (left). Holding shift in
addition shows the full difference to the previous version (right)
C Modified SemanticLense
… … …
Fig. 6 The version browser provides a tabular view on change history. Selecting a row shows
corresponding diff information in the panes to the right
55 passes
3 failures
2 errors
Go Here
Merge
Tests (55 / 5) …
...
Fig. 7 The items in the version bar are now a visualization of the results of the tests that have been
run in the background (left). A second inner environment allows the user to explore a previous
version next to the current one (right)
5 Experiment Design
Experimental Group
Figure 8 illustrates the setup of our controlled experiment. Subjects are assigned to
either of two conditions, the control group or the experimental group. Subjects in the
control group use the regular development tools for both tasks. Subjects in the
experimental group use regular development tools only for Task 1. After that, they
receive a tutorial in CoExist and have time to try it out and experience its benefits and
limitations. Thereafter, subjects in the experimental group work on Task 2 and can
therefore make use of CoExist in addition to the regular tool support. So while for
Task 1 all subjects use regular tools only, subjects in the experimental group may use
CoExist for Task 2. After receiving task instructions and material, subjects had
two hours for working on the respective task and achieving as much improvement
as possible.
At the time of writing, 20 students have already participated in the experiment.
They worked on both tasks on two different but subsequent days. Both tasks are
scheduled for the same time of the day to ensure similar working conditions
(hours past after waking up, hours already spent for work or studies, . . .). Typically,
we scheduled the task assignments after lunch so that for day 2, there was time left
to run the CoExist tutorial session upfront (before lunch time).
We try to keep subjects unaware of their assignment to the conditions, which
worked well for day/task 1. However, on day 2, subjects in the experimental group
could guess that they received special treatment because they were introduced to
CoExist and were asked to make use of it. Nevertheless, subjects in the control
group were unaware of the experimental treatment. They did not know that subjects
of the experimental group run through a tutorial and can use CoExist for task
2. Hence, subjects were not entirely blind concerning the treatment, although we
tried to be as close as possible to the blind setting.
This setup gives us two measures for every subject which will be prepared in
tables such as shown in Fig. 9. Such a data collection allows tests for statistical
differences between task 1 and task 2 as well as between the control and the
How to Compare Performance in Program Design Activities: Towards an. . . 231
Task 1 Task 2
01
Control
02
Group
03
11
Experimental
12
Group
13
experimental group. It also allows tests for the existence of an interaction effect of
the two factors, which will indicate whether or not CoExist has an effect on
programmers’ performance.
On each of the two days, subjects work on a different computer programs, but the
task is the same. Participants are requested to improve the design of the source code.
The two different programs are relatively small computer games like Tetris.
The procedure for each day is as follows. At the beginning of the assignment,
subjects are introduced to the game. After introducing the game play subjects have
a few minutes to play the game and get familiar with it. Afterwards subjects receive
the assignment. The task is to study the source code, to detect design flaws in
general and issues of unnecessary complexity in particular, and to improve the
source code as much as possible in the given time frame of 2 h. To help understand
the intent of the task, we provided sample descriptions of possible improvements,
for example:
• Extract methods to shorten and simplify overly long and complicated methods
• Replace conditional branching by polymorphism
• Detect and remove unnecessary conditions or parameters
Subjects were told to imagine that they co-authored the code and are responsible
for it, and that they now have time dedicated to improve the code in order to make
future development tasks simpler (enhancements or maintenance).
For both tasks, the given program was a relatively simple single player computer
games. Figure 10 shows screenshots of the game for Task 1 and Task 2. The game
on the left is called LaserGame. The player’s goal is to place mirrors in the field so
232 B. Steinert and R. Hirschfeld
a mirror
laser beam
destroyed wall
Fig. 10 Screenshots of the games used as experimental unit: MarbleMania (left), LaserGame (right)
that a laser is redirected properly to destroy the wall that blocks the way to the gate
to the next level. The game on the right is called MarbleMania. The user has to
switch neighbored marbles to create one or more sequences of at least 3 marbles
that have the same color, either by row or by column.
Subjects are also asked to describe their improvements. They should imagine
themselves as part of a development team. The description should help other team
members to better understand their changes and how they improve the code.
We decided to use these small games as the experimental unit primarily due to
practical reasons, but this choice turned out to be useful in unexpected ways. Such
games are developed by students in one of our undergraduate courses. Over the
years we have gained an interesting repertoire of game, all having different
characteristics and qualities. Both of these games function properly and have a
simple but still fun game play. So, only little time is required to get familiar with the
program’s functionality. Both games leave significant room for improvement in
terms of the source code, as do most of the games developed in this course.
Furthermore, both games come with a set of tests cases, which also have been
developed by the respective students. Test cases are particularly useful when
existing source code has to be changed because they are a means to automatically
check whether selected aspects of the program still work as expected. However,
while the offered test cases are useful, they are not sufficient. Manual testing of the
games is necessary, as it is often the case in other projects.
Independent of these characteristics, using the games as the subject of work has
another benefit. Participants of the study are primarily HPI students. And since they
had to take our class, they all have gained similar experiences in developing such
games. They have similar knowledge about the technologies, frameworks, and
libraries used for developing these kind of games. Consequently, experience is
not a factor that varies greatly among the participants.
How to Compare Performance in Program Design Activities: Towards an. . . 233
For our experiment, we decided to keep the time factor fixed and measure the
amount of achieved improvements. Keeping the time fixed is one of two typical
ways to examine the effects of tools or methods. The other way is to fix the amount
of work to get done by providing a clear unambiguous goal and measure the time
needed to achieve this goal (Juristo and Moreno 2010).
A fixed time setup seems preferable for experiments that focus on design tasks,
mainly because uncertainty is inherent to design tasks. The setup that measures time
to completion requires a clearly defined task without any uncertainty or ambiguity.
There has to be clear indicator when the task is finished. Also, the task description
should provoke similar thoughts, so that all subjects have the same idea what the
goal is and how it should be approached (given the defined conditions). These
criteria can hardly be met in an experiment where participants should accomplish a
design task.
Related research also shows that fixed time setup is a typical choice. In (Dow
et al. 2009, 2010), for example, the authors report on the empirical examination of
prototyping techniques. The response variable (dependent measure) has always
been some form of quality criteria of the design outcome while participant have
had a fixed amount of time to create the best possible design. However, we are
unaware of an experiment report in the software engineering field that examines an
effect in program design tasks.
Besides the decision for the fixed time setup, we have decided to rely on a repeated
measurement experiment design in contrast to other options. A repeated measure-
ment setup is preferable because programmers strongly vary in approaching such
tasks. Programmers have different working speeds, which involves code compre-
hension, code writing (typing speed), but also tool usage. Furthermore, when
programmers face the task to spend a fixed (and relatively short) amount of time
on improving source code, some programmers will have the tendency to focus on the
various small issues in the code, which are probably also easy to fix, while other
programmers will have the tendency to try to identify major flaws in the overall
program structure of the code and to improve on that while ignoring smaller issues.
While there can be significant differences, different contributions can be similarly
important for the long term success of a software project. However, the difference in
the response variable between programmers can be large in relation to the possible
difference caused by the provoked variations. For still being able to discover
statistical effects in these circumstances, literature recommends repeated measure-
ment experiment setups, in particular when having limited access to subject
candidates (Juristo and Moreno 2010).
234 B. Steinert and R. Hirschfeld
Besides a meaningful setup that allows for comparing performance, the experiment
requires a meaningful quantitative response variable (dependent variable). There-
fore, we have operationalized the notion that programmers can spend more or less
time on contemplating source code and testing their ideas mentally before actually
making the changes.
We assume that CoExist encourages participants to make changes as they think
of them because they can always recover from undesired consequence easily. We
also assume that without CoExist, participants will spend more time on thinking
about their ideas before making changes to the code in order to avoid making errors.
In the context of the experiment, we believe that CoExist enables programmers
to achieve more improvements in the given time frame. Every thought about a
possible improvement, no matter how vague and uncertain it still may be, can either
turn out appropriate or inappropriate. If the vague idea turns out to be appropriate,
programmers who directly started making changes will clearly save time because
they avoid spending time on upfront thinking. Findings in design research suggest
that given a design task full of uncertainty, programmers who directly make the
changes will save time in case that the idea turns out to be inappropriate. By making
the changes, they support their thinking by doing and explore their ideas and their
limits more efficiently. CoExist will help to recover fast and to get back to a desired
development state.
While we assume that participants using CoExist will make more changes, the pure
number of changes made, which correlates with number of versions, is not a mean-
ingful proxy for the amount of achieved improvements. This is for various reasons:
• Changes that lead to new versions strongly vary in the amount of changed code
and in the effect they have. While adding leading whitespace is a small change to
the code, which likely has no effect on the program execution, removing
statements or parameters from a method is a much larger change, which almost
always will has a effect on the program execution.
• Participants might want to withdraw changes. In an extreme case, a subject
might spend an hour of work on a particular idea to recognize later that the idea
cannot work out properly. This will create many versions in the history which
cannot be counted as improvements. Moreover, when not using CoExist,
participants have to manually withdraw their made changes, which will in turn
create additional changes.
• It might also be the case, that a series of changes was only good for inspiration
and helped the programmer to develop a better idea of how to improve the
elements of current interest. In this case, it would be unfair to double count the
made changes, the changes made for the initial idea and also changes made for
the final improvement.
Facing these constraints, a meaningful way to quantify the amount of achieved
improvements is to code the changes that persist over time, which means to identify
How to Compare Performance in Program Design Activities: Towards an. . . 235
groups of related changes and assign them to categories. Such change categories
can be either generic or rather specific to the given program.
Generic improvements are for example: to rename an instance variable; to
replace a parameter with method; to make use of cascades; to inline a temporary
expression; to replace magic string/number with method.
Improvements specific to the MarbleMania Game are, for example: to replace
the dictionary holding “exchange state” with instance variables; to replace isNil
checks in the destroyer with null objects; to remove button clicked event handling
indirections.
To perform the coding, we have analyzed the data in two steps. In a first step, we
have listed the timestamps of all versions (in a column of a spreadsheet) and
separated them according to commits, with subjects made during the task. In the
second column, we added the respective commit message (illustrated in Table 1).
The commit messages help understand the intent of the changes, which gives the
required context to understand the small changes to the individual source code
elements. In a second step, we identified improvements that we assigned to a
category. Thereby, a coded improvement can consist of only one actual change or
it can involve many changes. Sometimes, all the changes made for one commit
contribute to one coded improvement.
These change categories are assigned numbers that represent the relative effort
required to implement them. We sum up these numbers for the coded improvements
to finally get a measure of the amount of achieved improvements, which can be
used to run the statistical tests.
6 Summary
Programmers spend time on source code design to better support current and future
programming activities. While working on their code base, programmers can make
errors. Making errors represents a risk because it often requires tedious and time-
consuming recovery work. Traditionally, programmers have to anticipate such
situations to keep the costs for recovery low. However, anticipating errors is not
always possible, as we have illustrated by means of an experience report. We have
also described that precautionary activities can easily be ignored and distract from
the actual work.
We have presented CoExist as an extension to programming environments. CoExist
has been designed to encourage change. It avoids the risk of error making by providing
dedicated support to recover from undesired consequences. We believe that such a
tool-based approach is preferable because it encourages programmers to support their
thinking by doing. We have presented an experiment design to empirically examine
this claim. We measure subject performance in two different tasks. In each task,
participants have to improve the source code design of a given program. Their changes
are recorded. Afterwards, the changes are coded to identify achieved improvements
and to compute a quantitative measure of programmers’ performance.
Table 1 Spreadsheet Excerpt with coded version data
236
Version
timestamp Commit message of participants Coded improvements
...
14:01:41 In LaserBeam extracted code that is similar in all these calculate methods; LG_ExtractGenericCalculateMethod
14:02:16 improved the previously extracted, generic calculateWay: . . . method. (ReplaceSimilarStatementsWithCallToExtractedMethod)
14:02:32 (two things happened – refactoring + additional improvements) . . .
14:02:49 “Refactoring: summarized multiple similar methods into one, with
14:03:01 3 parameters. Original methods call the new one.”
14:03:05
14:03:11
14:03:16
14:06:08
14:06:42
14:06:46
14:13:06 Simplified LaserBeam>>#setStartPoint, deleted useless condition, RemoveStatements + 2 * InlineMethod
14:14:18 integrated code from called methods, and removed the other methods.
14:14:25 “Simplified method based on detected ‘invariant’, that self laser direc-
14:15:16 tion is always 1@0 at this code location. Removed 2 methods.”
14:15:28
14:15:38
14:15:42
14:17:25
14:18:39 “Removed 2 unused variables of SWA18Laser. One seemingly was not used 2*RemoveStatement + 2*RemoveUnusedMethod +
14:18:43 at all (point), the other (direction) got useless after previous commit. 2*RemoveUnusedInstVar
14:18:53 Removed all usage of the direction-variable (was only used in tests).”
14:18:53
14:18:53
14:19:41
14:20:33
B. Steinert and R. Hirschfeld
14:21:02
14:25:02 “Extracted method in level parsing.” #readNumberArrayFrom: LG_LevelLoader_ExtractSimilarStatements
14:26:00
14:26:08
14:26:26
14:26:53
14:27:15
14:27:20
14:27:37
14:27:50 TmpVarRenaming
14:29:20 TmpVarRenaming
14:29:29 “Removed useless method” – belongs to previous context InlineMethod
14:30:18 “Removed another useless method” – namely #readTimeFrom: – integrated Recategorization
14:30:59 the more meaningful one-liner in the caller InlineMethod
14:31:11
...
How to Compare Performance in Program Design Activities: Towards an. . .
237
238 B. Steinert and R. Hirschfeld
References
Allen D (2001) Getting things done: the art of stress-free productivity. Penguin, New York
Apache Software Foundation (2009) Subversion best practices
Beck K (1996) Smalltalk best practice patterns. Prentice Hall
Beck K, Andres C (2004) Extreme programming explained: embrace change. Addison-Wesley
Longman
Blackwell AF (2002) What is programming. In: 14th workshop of the Psychology of Programming
Interest Group. Citeseer, pp 204–218
Dorst K, Cross N (2001) Creativity in the design process: co-evolution of problem- solution.
Des Stud 22(5)
Dow SP, Heddleston K, Klemmer SR (2009) The efficacy of prototyping under time constraints.
In: Conference on creativity and cognition
Dow SP, Glassco A, Kass J, Schwarz M, Schwartz DL, Klemmer SR (2010) Parallel prototyping
leads to better design results, more divergence, and increased self-efficacy. ACM Trans
Comput-Hum Interact (TOCHI) 17(4):18
Fowler M (1999) Refactoring: improving the design of existing code. Addison-Wesley Professional,
Reading
Goldschmidt G (1991) The dialectics of sketching. Creativity Res J 4(2)
Juristo N, Moreno AM (2010) Basics of software engineering experimentation. Springer
Kirsh D (2010) Thinking with external representations. Ai Soc 25(4):441–454
Lim Y-K, Stolterman E, Tenenberg J (2008) The anatomy of prototypes: prototypes as filters,
prototypes as manifestations of design ideas. ACM Trans Comput-Hum Interact (TOCHI) 15(2)
Lindberg CA (2008) Oxford American writer’s thesaurus. Oxford University Press, New York
Parnas DL (1972) On the criteria to be used in decomposing systems into modules. Commun ACM
15(12):1053–1058
Steinert B, Cassou D, Hirschfeld R (2012) Coexist: overcoming aversion to change. In: Proceedings
of the 8th symposium on dynamic languages, DLS’12, New York, ACM, pp 107–118
Suwa M, Tversky B (2002) External representations contribute to the dynamic construction of
ideas. In: Diagrammatic representation and inference, vol 2317. Springer Berlin/Heidelberg
Design Thinking: Expectations from
a Management Perspective
Abstract The authors present first results from an empirical study on the integra-
tion of Design Thinking in large corporations. In this article, we examine the
managers’ expectations towards Design Thinking in one large software-developing
corporation. Empirical results from various interviews show that managers explic-
itly expect Design Thinking to contribute to existing frameworks such as Lean in
general and Scrum in particular. It can be assumed, that the success of Design
Thinking in this corporation will therefore depend on its compatibility with those
established frameworks.
1 Introduction
Over the last decade, Design Thinking stepped out of the design course and
received the global recognition of agencies and corporations. Corporate manage-
ment in various industries currently drives the embedding of Design Thinking.
We investigate the relevance of Design Thinking from a management perspec-
tive. It is based on a long-term study at a multinational software vendor that
embedded Design Thinking in its corporate structure. The study further compares
different corporate initiatives and their impacts over a longer period.
In the following article, we focus on the management’s expectations within a
specific corporation. We therefore conducted interviews with managers at different
levels of the organization, before they initiated Design Thinking projects, as well as
after the projects ended.
As a starting point for our research we present the observation that management in
various corporations embed Design Thinking with specific intentions to improve
the status quo.
Design Thinking: Expectations from a Management Perspective 241
The following case study focuses on the global embedding of Design Thinking at
a multinational software vendor.
The company is a large, multinational software vendor. It is a DAX corporation
that became highly successful over four decades by providing standardized busi-
ness software solutions for a wide range of industries.
In our research project for this case study, we conducted over 50 semi-
standardized interviews with managers, team members and Design Thinking
coaches that were all part of a large Design Thinking initiative consisting of more
than 40 projects over a period of 6 months. Most of the projects started in the spring
of 2012 and ended in the fall of 2012. A majority of projects in the initiative were
situated in Europe, others were located in Asia, the US and Canada.
The projects covered a range of innovation topics, such as new product devel-
opment, process improvement, reorganization, and the evaluation of innovative
basic technologies for the software industry.
We investigated 11 different projects, and conducted interviews with all respon-
sible project managers, the so-called product owners, with line managers, a vice
president and at least one team member and a coach of each project. The
interviewed product owners, line managers, and the vice president had not taken
part in previous Design Thinking projects, apart from smaller learning workshops.
We were particularly interested in understanding their expectations of Design
Thinking before they gained experience in this field. After the project, we
conducted further interviews in order to investigate how their previous expectations
matched the actual course of their projects and the repercussions of this. The
interviews lasted from 45 to 120 min. During our research, we constantly reviewed
and updated our questionnaires whenever new relevant observations indicated new
topics of interest. This qualitative and iterative process is compliant with the rules
of the Grounded Theory approach (Glaser and Strauss 2009).
With regard to the case study, Design Thinking did not just appear but became a
topic of interest in the last decade. During that time, the corporation was undergoing
major changes. In our interviews, stakeholders mentioned several streams of events
that had impacted the corporation significantly. Due to space constraints, we focus
on three aspects that seemed to highly influence managers’ expectations towards
Design Thinking.
One of the founders of the corporation is still a member of the advisory board and
widely regarded as a very influential person within the corporation. The founder
242 H. Rhinow and C. Meinel
Three years before the initiative took place, the corporation started to undergo a
major reorganization that is still being perpetuated today. In order to become a Lean
organization, the corporation changed not only processes and structures but also
introduced completely new positions, especially in development units, that make up
a major part of the workforce.
Lean includes a variety of frameworks that often empower teamwork for differ-
ent purposes, e.g. agile frameworks in the software development processes (Kittlaus
2012). Teams play a major role in this corporation, similar to other modern
corporations. Previous differentiations and specializations have resulted in highly
distinct work units with a perceived lack of knowledge transfer and collaboration.
While these units led to higher degrees of efficiency, it was also apparent that
corporate structures had become less flexible and agile in order to adapt to changing
markets. As a consequence, corporations began to install different team-based
approaches to establish a more creative working culture:
The Lean approach brings back the strengths of the company in its beginnings, when the
number of employees was a two or three digit number. (translated from Mackert et al. 2011,
p. 48)
A major goal anticipated with the incorporation of Lean is the banishment of all
sorts of wasteful activities and processes with the end result of becoming a more
efficient corporation. Lean activates the creative potential of employees to this end,
based on modern organization and personnel management (Stürzl 1996).
Strictly speaking, Scrum is one Lean framework among others. Nevertheless, with
regard to the case study, Scrum is of high importance. Scrum is a framework to
empower teamwork in order to efficiently manage software development processes.
The incorporation of Scrum puts a strong emphasis on cross-functional team-
work and shared responsibilities during the software development process:
Split your organization into small, cross-functional, self-organizing teams. Split your work
into a list of small, concrete deliverables. Sort the list by priority and estimate the relative
effort of each item. (Kniberg and Skarin 2010, p. 3)
Design Thinking: Expectations from a Management Perspective 243
With regard to the principles of Lean, managers expect their employees to chal-
lenge existing thoughts on user values and needs. Strictly speaking, managers
expect their employees to deliver user-valid innovations that are worth pursuing
in the light of limited resources:
For me, from my understanding, Design Thinking is a concrete tool to support innovation
and creativity. First of all, to understand what we should actually build for our customers.
And the Lean principles that we have, in my opinion, focus on how we can get what we
want to build, in a reasonable quality with a motivated team. (Interview with Product
Owner#2 March 2012)
Can anything that happens within the corporation be seen through the lenses of Lean? I
think that is what we want to achieve. (Interview with Product Owner#2 March 2012)
Any actions that do not comprehensibly contribute to the value proposition are
suspicious to be waste and may become a condition that should be banished. This
can include small actions such as holding unproductive meetings or unnecessary
delays to formal routines such as complete development processes that deliver
outdated and therefore useless outcomes:
Waste - all other moments or actions that do not add value but consume resources. Wastes
come from overburdened workers, bottlenecks, waiting, handoff, wishful thinking, and
information scatter, among many others. (Larman and Vodde 2009, p. 58)
Managers and employees perceive the given framework of Lean as highly useful
but not yet sufficient:
For me (. . .) Lean is focusing very strongly on efficiency. (Interview with Product
Owner#2 March 2012)
consider the value that people are actually willing to pay for. (Interview with Product
Owner#8 March 2012)
It is not trivial to find out clear value propositions, because e.g. different
customers provide different perspectives:
You have a customer contact people from development. We often deal with administrators
on the other hand. The customer value is quite ambiguous. (Interview with Product
Owner#8 March 2012)
survey as it is often done in market research. During the process the team redesigns
the initial project briefing as a starting point for their design challenge; they iterate
on the nature of the problem, while they approach different possible solutions, they
focus on specific user issues and synthesize them, they ideate upon self-constructed
questions, and they prototype their ideas in a way they perceive as usable for further
testing and iterations.
The teams intuitively navigate through a stream of options and make numerous
decisions even though they might not explicitly reflect their actions as decision-
making. In fact, every meaning that is abstracted from user observations, every idea
that is developed and linked to a cluster of prior ideas, and every design detail that is
built as part of a prototype is in fact a design decision that is made along the process
(Schoen 1984). The managers expect teams to work and decide autonomously to a
certain degree, similar to Lean:
I mean, we work in a Lean mode and our employees who work together on a topic organize
it themselves. It is not the case that everyone is doing stuff alone. (Interview with Product
Owner#7 April 2012)
The framework therefore gives room for creation, testing and iterating but it also
restricts the team to deliver a value proposition that is illustrated by a prototype. The
team, for example, cannot decide on their own to directly develop new software
solutions.
It is assumed that those tangible definitions of value can be used to distinguish
between value and waste. In this sense the results of Design Thinking can be used as
a prerequisite for decision-making in the Lean framework.
During the last 3 years, Scrum became the default framework to develop complex
software development processes.
Scrum as a specific framework is in an exposed position within the holistic
framework of Lean. It is incorporated in order to empower specific teams, so-called
teams of ten, to manage themselves to a certain degree in order to efficiently
develop software solutions.
Teams of ten consist of a product owner who is responsible for the solution and
the prerequisites, a crossfunctional team that designs, codes, tests, documents, and a
scrum master who facilitates the process (Grau 2012). They work in so-called
sprints in which they commit themselves to developing certain objects of the final
software solution. The teams usually meet every day for a short sprint review to
reflect on the prior tasks and define what tasks need to be done that day.
Within the corporation, Scrum is perceived as a powerful framework to facilitate
work by responsible team members independently, as they know well how much
work they can do in a certain time. The team members do not need to wait for
Design Thinking: Expectations from a Management Perspective 247
The centerpiece of every Scrum project, and its most important artifact, is the
so-called product backlog. The product owner is responsible to initially develop a
backlog, a collection of prioritized backlog items that are distinct work tasks, which
in total define a complete software solution. The later distribution of tasks is done
by the team members themselves, leading to a higher self-organization.
Depending on the accuracy of the backlog estimation, the final allocation of all
executed backlog items will inevitably lead to a valid and functional software
solution, no matter which developer is responsible for certain tasks and no matter
in which sequence they are executed.
The backlog illustrates a complete overview of all tasks that are needed to be
done in order to come up with the software solution. As we have seen with regard to
the overall Lean framework, every action, including all software developments,
needs to be based on a value proposition and therefore is legitimately creating value
for the corporation. This means that the initial backlog must be based on a value
proposition. Equally important, the backlog needs to address the value proposition
adequately, meaning that the backlog items are a valid composition in order to
deliver an evident-based, user-desired result.
It is important to realize that a value proposition itself is not a user-desired result.
The value proposition is synonymous with a statement of what a user desires and
how much she is willing to pay for it. A user-desired result is a statement of how to
address the desire with regard to a product or service solution. While a value
proposition may be ambiguous, the solution is always a concrete statement.
The backlog therefore represents a product owners’ interpretation of a concrete
solution. During the Scrum process the interpretation may change because customer
reviews indicate the necessity to do so, but it will always be expressed in a concrete
statement.
At this point, managers perceive a shortcoming with regard to Scrum. The
backlog only represents a concrete solution, if someone has done the transfer
from a rather ambiguous and complex value proposition to a concrete solution
statement. As previous design research shows, user problems and desires can be
understood as complex, so-called wicked problems (Rittel and Webber 1973).
Keeping with this terminology, a backlog can only represent so-called tame
problems, e.g. user solutions that can be calculated, e.g. broken down into smaller
parts:
‘Tamed problems,’ such as mathematical problems are causal microworlds that we, through
enormously inventive interpretive skills are at times able to use to ensure that trains run on
time, computers calculate bank balances, and bridges do not sway out of line. (Coyne 2005,
p. 9)
248 H. Rhinow and C. Meinel
Product owners, who own the product backlog, are responsible for addressing
this issue, and they therefore disassemble the overall challenge, or the overall goal:
First of all, I mean, we have a certain period of time to develop, we also have to consider to
disassemble and develop our insights that we identified while visiting our customers. This is
our operating environment, actually. (Interview with Product Owner#7 April 2012.
Especially product owners are keen to share responsibilities with teams in order
to develop operable results out of user insights:
It is not the classic situation anymore, where a product owner says: ‘I am responsible, I
make the decisions’. I believe, we, as a highly motivated team, will try to see, what we can
get out of it, this is my approach. Of course, at some point you have to make a decision, but I
believe it is the right approach to see what insights our people get from our customers and to
give everyone a voice in order to decide what we are going to do with it in the end.
(Interview with Product Owner#3 April 2012)
Managers seem to expect Design Thinking to happen before teams switch into
the Lean framework:
Design Thinking: Expectations from a Management Perspective 249
Okay, I will do Design Thinking to find out, what I can do for the end-user. And (then) I
develop a real product. That would be a consequence (. . .) effectively developing an
outcome after Design Thinking, this is part of Lean. This is suitable for me. (Interview
with Product Owner#6 May 2012)
The expectations were nurtured through the Design Thinking discourse that
described Design Thinking as a form of abductive reasoning to deal with wicked
problems, while other frameworks such as Scrum can be characterized as deductive
reasoning (Martin 2009):
The linear model of design thinking is based on determinate problems which have definite
conditions. The designer’s task is to identify those conditions precisely and then calculate a
solution. In contrast, the wicked-problems approach suggests that there is a fundamental
indeterminacy in all but the most trivial design problems -problems where, as Rittel
suggests, the ‘wickedness’ has already been taken out to yield determinate or analytic
problems. (Buchanan 1992, p. 36)
Managers expect to work on user story maps after their idea is developed and
before they develop the solution:
250 H. Rhinow and C. Meinel
I do user story maps once I am a bit further in my process, that is what I would say. Design
Thinking is more at the front end, when I want to develop my idea. (Interview with Product
Owner#5 March 2012)
5 Summary
6 Outlook
Even though our research already indicates a change of the corporate culture
beyond innovation projects, managers do not perceive Design Thinking to become
a trigger to change the overall working culture:
I see Design Thinking as a potential complement. It depends on what you want to develop. I
cannot see us starting a Design Thinking project for everything we do. It is useful,
whenever we explore open territory. (Interview with Product Owner#5 April 2012)
Beyond that, our research indicates that management will face new challenges
and opportunities if Design Thinking becomes a consistent part of the corporation.
Since teams are empowered to work independently and make design decisions on
their own, to a certain degree, management will need to position itself at the
interface between Design Thinking and Lean, Design Thinking and Scrum, as
well as other frameworks that are compatible with Design Thinking but are not
discussed in this article.
Since Design Thinking Teams may come up with a lot of potential value
propositions that may even contradict themselves, it needs a management that
makes decisions in terms of what value propositions may be useful as prerequisites
for Lean initiatives and what definitions may be inconsistent with existing value
propositions. Management in this context may act as an enabler for decentralized
teams to either transfer or withhold value propositions into other parts of the
corporations, with regard to consistency with overall strategies.
Design Thinking: Expectations from a Management Perspective 251
Management will also need to position itself at the interface between Design
Thinking and Scrum in order to couple or decouple solutions statements from the
later development process.
Also, it is still undecided how customer reviews in Scrum may lead to
reconsiderations of the validity of solutions statements coming out of Design
Thinking and, as a consequence, may demand a major change, e.g. by returning
to the framework of Design Thinking. It is assumed that this will be a highly
probable scenario in many development processes. However, it could also lead to
new challenges for management of how to deal with the contingency of iterative
frameworks in a business world of more or less fixed development cycles and
delivery dates.
Our research will further investigate how managers approach these upcoming
challenges and whether Design Thinking will match their expectations in the
long run.
Due to legal standards, the authors currently withhold the name of this specific
company and of the cited persons.
References
Badke-Schaub P, Roozenburg N, Cardoso C (eds) (2010) Design thinking: a paradigm on its way
from dilution to meaninglessness? Design thinking research symposium, p 8
Baecker D (2007) Studien zur nächsten Gesellschaft. Suhrkamp
Brown T (2008) Design thinking. Harvard Business Rev, pp 84–92
Brown T, Watt J (2010) Design thinking for social innovation. Stanford Soc Innov Rev 8(1):28–35
Buchanan R (1992) Wicked problems in design thinking. Des Issues 8(2):5–21
Bucolo S, Matthews J (eds) (2010) Using a design led disruptive innovation approach to develop
new services. In: Kobe C, Goller I (eds) Proceedings of the 11th international cinet conference:
practicing innovation in the times of discontinuity, CINet
Carlgren L, Elmquist M, Rauth I (2011) Implementing design thinking – an exploratory study of
large companies using design thinking in innovation efforts. In: Proceedings of the 2011
Tsinghua-DMI international design management symposium HK in Hong Kong, pp 1–19
Clark K, Smith R (2008) Unleashing the power of design thinking. Des Manag Rev 19(3):7–15
Cooper A (2004) The inmates are running the asylum. Sams, Indianapolis
Coyne R (2005) Wicked problems revisited. Des Stud 26(1):5–17
Dunne D, Martin R (2006) Design thinking and how it will change management education. An
interview and discussion. Acad Manage Learn Educ 5(4):512–523
Glanville R (2011) Designing complexity. Revue für Postheroisches Management 8:24–41
Glaser B, Strauss A (2009) The discovery of grounded theory: strategies for qualitative research.
Transaction Publishers, New Brunswick/London
Grau R (2012) Requirements engineering in agile software development. In: Software for people.
Springer-Verlag, Berlin/Heidelberg, pp 97–120
Hildenbrand T, Meyer J (2012) Intertwining lean and design thinking: software product develop-
ment from empathy to shipment. In: Software for people. Springer-Verlag, Berlin/Heidelberg,
pp 217–237
Kittlaus H-B (2012) Software product management and agile software development: conflicts and
solutions. In: Software for people. Springer-Verlag, Berlin/Heidelberg, pp 83–96
Kniberg H, Skarin M (2010) Kanban and scrum. Making the most of both. C4 Media, USA
252 H. Rhinow and C. Meinel
Larman C, Vodde B (2009) Scaling lean & agile development. Thinking and organizational tools
for large-scale Scrum. Addison-Wesley, Upper Saddle River
Mackert O, Hildenbrand T, Podbicanin A (2011) Von wasserfallartigen Softwareentwicklungs-
modellen hin zu “Lean software product development“. Gesellschaft für Informatik e. V.,
Fachausschuß Management der Anwendungsentwicklung und -wartung (WI-MAW) im FB
Wirtschaftsinformatik, pp 48–51
Martin RL (2009) The design of business: why design thinking is the next competitive advantage.
Harvard Business Press
Osterwalder A, Pigneur Y (2010) Business model generation: a handbook for visionaries, game
changers, and challengers. Wiley, Amsterdam (self-published)
Plattner H, Meinel C, Weinberg U (2009) Design thinking. mi-Wirtschaftsbuch, Munich
Rhinow H (2011) Translated interview with general manager on the 14th of December 2011
Rhinow H (2012a) Translated interview with product owner#2 on the 23 March 2012
Rhinow H (2012b) Translated interview with product owner#3 on the 13 April 2012
Rhinow H (2012c) Translated interview with product owner#4 on the 14 May 2012
Rhinow H (2012d) Translated interviews with product owner#5 on the 22 March, the 13 April, and
the 30 May 2012
Rhinow H (2012e) Translated interview with product owner#6 on the 14 May 2012
Rhinow H (2012f) Translated interview with product owner#7 on the 13 April 2012
Rhinow H (2012g) Translated interview with product owner#8 on the 23 March 2012
Rhinow H (2012h) Translated interview with scrum master on the 13 Sept 2012.
Rittel H, Webber M (1973) Dilemmas in a general theory of planning. Policy Sci 4(2):155–169
Rowe P (1987) Design thinking. MIT Press, Cambridge, MA
Schoen D (1984) The reflective practitioner: how professionals think in action. Basic Books,
New York
Schrage M (1999) Serious play. How the world’s best companies simulate to innovate. Harvard
Business School Press, Boston
Simon H (1999) The sciences of the artificial. MIT Press, Cambridge, MA
Stürzl W (1996) Business reengineering in der Praxis. Durch umfassende Veränderung im
Unternehmen einen Spitzenplatz im Wettbewerb erreichen. Junfermann, Paderborn
Von Foerster H (2002) Understanding understaning: essays on cybernetics and cognition.
Springer-Verlag, New York
Womack J, Jones D, Stotko E (1997) Auf dem Weg zum perfekten Unternehmen. Wilhelm Heyne
Verlag, Munich