CIFE CENTER FOR INTEGRATED FACILITY ENGINEERING

Observation, Theory, and Simulation

of Integrated Concurrent Engineering:
Risk Analysis Using Formal Models
of Radical Project Acceleration

By

John Chachere, John Kunz, and Raymond Levitt

CIFE Working Paper #WP088

August 2004

STANFORD UNIVERSITY
 COPYRIGHT © 2004 BY
Center for Integrated Facility Engineering

If you would like to contact the authors, please write to:

c/o CIFE, Civil and Environmental Engineering Dept.,

Stanford University
Terman Engineering Center
Mail Code: 4020
Stanford, CA 94305-4020
 Observation, Theory, and Simulation of Integrated Concurrent Engineering:
Risk Analysis Using Formal Models of Radical Project Acceleration

John Chachere, John Kunz, and Raymond Levitt

Stanford University

INTRODUCTION
Design Team Performance
Integrated Concurrent Engineering (ICE) uses: a singularly rapid combination of expert designers;
advanced modeling, visualization and analysis tools; social processes, and a specialized design facility;
to create preliminary designs for complex systems. When compared with a traditional parallel
engineering method, successful ICE users reduce project schedule by several orders of magnitude, while
substantially improving design cost and maintaining quality standards. Today’s pioneers of ICE are in
the aerospace and automotive industries, where several closely related methods are termed “ICE”,
“Extreme Collaboration”, ‘Concurrent Design Engineering”, or “Radical Collocation.” [ Mark, MEP,
Olsens] Whereas traditional engineering superficially resembles a government bureaucracy, ICE
performs the same work in an environment more akin to NASA’s Shuttle Mission Control operations.
Our research is based primarily on the most experienced ICE team at NASA, the Jet Propulsion
Laboratory (JPL) Advanced Project Development Team, conventionally known as Team-X1. Team-X
completes early-phase design projects in less than one-tenth the time of the previous process at JPL, and
for less than one third of the variable cost. Although there is continuing effort to improve the quality of
the Team-X designs and the generality of their method, the Team-X product is good enough that outside
investigators choose to purchase Team-X services about fifty times a year. The team is in heavy
demand in the competitive market for mission design services, and its successful plans have brought
hundreds of millions of dollars in business to JPL and its suppliers [Sercel 1998].

An Illustrative Metaphor
We find that an auto metaphor conveys our intuition that in spite of superficial differences, ICE
mechanistically differs from standard design principally in that it operates more rapidly.
Metaphorically, we conceive of ICE as analogous to the operation of high-performance race cars in that
ICE engages the same considerations as standard design teams, but like the race car, many elements of
the total system are customized for high performance. The racecar has specialized engine, transmission,
tires and even a racetrack. Analogously, ICE requires expert selection and preparation for participants,
the organization, the enabling modeling and visualization methods, and the design process the

1 With thanks, but without explicit description, we leverage observations from similar practices at the
Tactical Planning Center at Sea-Land Service Inc., and at Stanford’s Real-Time Venture Design
Laboratory, Gravity Probe B Mission Control, and Center for Integrated Facility Engineering.
Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
participants follow. For the racecar, any bump in the road, hardly noticeable at twenty miles per hour,
can be disastrous at two hundred. Therefore, before a race, the track must be cleared and leveled.
Analogously, the Team-X “pre-session” structures the tasks, and chooses the participants and the
variables of interest for the project at hand. Finally, once the race starts, the driver principally responds
by reflex in accordance with training and experience, because there is little time for deliberation. An
ICE team also must work quickly to do its design and make decisions quickly, conclusively and well.
Our intuition is that the race car and the ICE team are structurally identical to the standard car and
design team; The fundamental forces and operations in play are the same in both cases; and those
specialized, enabling adaptations of a generic design result in the radically different performance in both
cases. Thus, while operating at high speed (low latency), we are still looking at a car (or a multi-
disciplinary design project), and we can understand it by understanding the behavior of the fundamental
mechanisms.
This “Systems” perspective suggests that an ICE implementation that lacks a single critical
aspect may result in unimproved performance, or even project failure. In our analogy, an otherwise
optimized racecar with an ordinary engine cannot generate enough power to compete, and placing an
ordinary driver behind the wheel would be catastrophic. Furthermore, factors that are irrelevant under
some conditions may become important in others, and offer a key to understanding phenomena as
seemingly unprecedented as ICE. Wind resistance, for example, is of no consequence at low speeds, but
it motivates streamlining at high speeds. A truly novel enhancement, wings, converts the once
detrimental wind resistance into beneficial lift, and revolutionizes transportation.

Goals of This Research

Although this paper does not determine whether ICE is revolutionary, our observations, theories, and
simulations are likely essential to that endeavor. This paper addresses the theorists’ questions of how
and why ICE works. Most early descriptions of ICE are anecdotal, motivational, or limited in
perspective, rather than being grounded rigorously in broadly validated theory. Recent scholarly
documentation published on the behavior of ICE and similar projects [Mark 2002, Teasley et al 2000]
describes the features of ICE, namely highly concurrent design by multiple collocated multi-disciplinary
experts. The academic literature also stops short of explaining the fundamental mechanisms of ICE and
its behavior.
Our theoretical results suggest methods by which an important range of applications can adopt
ICE in its entirety. Of equal importance, they articulate reasons why most organizations may find this
move prohibitively challenging in the short term. We identify for practical organizational designers a
process performance metric that can help teams understand the limits on their performance today and a
focus of attention that can significantly improve their effectiveness in any kind of collaboration.

Our Methodology
We offer three orthogonal and complementary research elements: observations of a radically accelerated
project at JPL, formal yet intuitive theories that have face validity and offer a straightforward
comparison with established social science theories, and simulation results that show the combined
implications of foundational micro-theories on a project scale. Our claims are based on simultaneously
validating theories by comparing them with observations, verifying theories’ consistent
operationalization in a simulation model, and calibrating the results’ implications against our initial and
new observations. Our work is therefore explicitly grounded by consistencies among reality, intuition,
and formalism.

CIFE Working Paper 8/13/2004 2

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt

Observation
We visited JPL’s Team-X and ethnographically observed three design sessions of a sample project. In
several hours of on-site interviews, we collected quantitative and qualitative details about the
participating organization, process, and culture. Finally, after coding and analyzing this information, we
followed up with an online survey covering the amount of time each participant spent in direct work,
communication, and rework each week. We describe the ICE practice in detail, and propose information
response latency as a fundamental, observable process performance measure.
Theory
Our observations, interviews, and survey ground a set of factors that enable radical project acceleration.
We explain ten fundamental mechanisms that work together to keep response latency at a minimum,
and, thereby, allow projects to execute at a very high speed. Although we leverage existing literature
extensively, the work also draws on behaviors and relationships observed in practice.
Simulation
We apply three computational project models to describe and predict the performance of an ICE team.
We retrospectively calibrated the Organizational Consultant (OrgCon), Virtual Design Team (VDT),
and Interaction Value Analysis (IVA) models to accurately describe our observations at Team-X, and
found that that they are able to accurately depict the observed ICE phenomena. We conclude with
analysis using a detailed VDT model that supports our enabling factor theories.

VALIDATION OF ICE ENABLING FACTORS

Inspired by ICE’s novelty and differences from traditional structures, we explicitly assess the practice’s
amenability to previously established organizational theories. Although it does not tell the whole story,
much of mainstream theoretical research on organizations does apply to ICE. Theoretical developments
led by vonNeumann and Morgenstern, Simon and March, Thompson, and Galbraith [ all more precisely]
offer points of departure for our exploration of parallel engineering teams’ work and structure. At the
project level, we find that traditional organization theories are applicable and insightful, while at a more
detailed level, we find that important questions remain. In this section, we synthesize established
organizational theories into a foundational framework that accommodates both ICE and traditional
practice. We instigate and terminate this process by comparing observed ICE behavior against the
predictions of theory-grounded computational models.

An OrgCon Model of ICE

We began our analysis using a computer program that is firmly rooted in a range of established
organization theories. The Organizational Consultant, or OrgCon, is a rule-based expert system that
Richard Burton and BØrge Obel developed and documented in their 2004 book, Strategic Organizational
Diagnosis and Design (3rd edition). When informed of characteristics such as structure and
environment, this system predicts an organization’s potential weaknesses in terms of mismatches
between its strategy, structure, climate, management style, and other factors. OrgCon typically assesses
the strengths and weaknesses of whole companies’ organizational structures, although it is also able to
analyze subsidiaries.
We applied OrgCon to a hypothetical company that conducts the majority of its business as we
observed Team-X to do. In response, OrgCon predicted a striking range of distinctive aspects of Team-
X operations, such as:
…Coordination and control should be obtained through integrators and group meetings. The
richness of the media should be high with a large amount of information. An open organizational

CIFE Working Paper 8/13/2004 3

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
climate and team spirit must be fostered. Information must be shared among all levels.
Constructive conflict on “what to do” will be usual. Individual tolerance of ambiguity and
uncertainty will be necessary. Mutual adjustments of “give and take” will be the norm. Frequent
informal meetings and temporary task forces will be the primary coordinating devices…
- OrgCon

We provided none of the information in this diagnosis as input, and yet OrgCon returned specific
and strikingly accurate descriptions of JPL design sessions’ tools, people and process. Theory, model,
and professional practice validate one another, for example, in that the system’s “Frequent informal
meetings and temporary task forces” prediction accurately describes both Galbraith’s theoretical
recommendations [1973] and the observed Team-X sidebars (we explore the importance of this and
other features, such as “Ambiguity”, “Richness of the media”, and “Team spirit” later in this paper).
The result lends confidence in the OrgCon model’s applicability, and demonstrates that elements of
ICE’s success can be predicted by existing literature.
By predicting no misfits for the ICE approach, OrgCon raises the exciting possibility that ICE is
a new, distinct and effective organizational form. Many organizational researchers (notably Mintzberg
[], and Burton and Obel themselves) hypothesize that only a handful- typically 5 or 6- perfectly adapted
archetypal organizational styles exist. OrgCon is not single-handedly equipped to assess such a claim,
but it does provide a degree of confidence, complementary to empirical claims, that ICE is both effective
and sustainable. Because OrgCon does not offer positive, clear and compelling evidence of ICE’s
effectiveness, however, we cannot conclusively determine whether an important gap in theory, observed
practice, or model is present. We therefore turn to a selection of prominent and more operationally
explicit theories to assess in detail the extent to which social science theory encompasses ICE behavior.

Information Processing View

Galbraith (1973) indicates that organizations operate as if their primary function is the processing of
information. Shortcomings in information flow or knowledge in an organization produce “exception”
events that require managerial attention. We interpret exceptions as perceived faults or gaps in the
decision basis that disallow “Clarity of action” [ Howard]. Organizations route the information or
queries that are pertinent to these exceptions to complementary resources such as management.
Organizations, according to Galbraith, are designed primarily to route and process information and to
handle exceptions as efficiently as possible.
This “information processing view” predicts that the match between workers’ capabilities and
their tasks determines the necessary tightness of intra- and inter-organizational collaboration. For
example, Galbraith conjectures that organizations may form temporary, interdisciplinary task forces
when a large number of interdependent issues arise. Although Team-X is not a temporary organization,
it operates like a task force that is formed to address a single phase in each of several larger projects. At
a lower temporal and organizational level of abstraction the ICE sessions’ “sidebar” conversations,
which operate like internal task forces, form and dissolve continually. Our analysis of ICE shows that
the former, micro-level information processing view of exception handling motivates the formation of
Team-X, while the latter, nano-level demands of individual communications drive the group’s structure
and information technology configuration.

Direct Work
We view engineering projects as consisting of many interrelated design decisions. The institutional
branch of organizational theory indicates that people make decisions and select procedures using a sense
of personal identity and appropriateness [March 1994, Scott ibid., Powell and DeMaggio]. ICE decision
support technologies, engineering culture and public decision making processes strongly encourage

CIFE Working Paper 8/13/2004 4

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
formal and impartial evaluation of design alternatives. This conformity to a “Rational” normative
identity leads us to adopt a general rational framework model [March 1994] of decision making in the
tightly knit ICE team.
According to formal rational decision theories [vonNeumann and Morgenstern year, March
1994], individuals make decisions by combining four elements: beliefs, preferences, alternatives, and a
decision rule. Alternatives are possible actions (including passiveness) amongst which a decision-maker
must choose. Beliefs are matters of fact or expectation about history, the current world state, and the
possible future consequences of alternative actions. Preferences are personal measures of relative
desirability among specific future prospects that might result from the selection of an alternative. The
last element, a decision rule, is a method for determining which among several alternatives should be
acted upon, for a given set of beliefs and preferences.
Economics and probabilistic decision and risk analysis are based on a “Rational” ideal, which
uses the “Maximization of expected utility” decision rule [Lave and March, Howard and Raifa].
According to the “bounded rationality” thread of organizational theory, sparked by March and Simon
[Simon 1997, March and Simon 1958], individuals are cognitively and contextually unable to make fully
rational decisions. Instead, organizations accommodate and compensate for individuals’ limitations, so
that coordinated behavior may approach rationality.2

A VDT Model of Hidden Work

Project managers frequently underestimate the emergent workloads of subordinates whose work is
highly interdependent, in part because coordination efforts are not explicit in traditional planning and
schedule tracking systems. We use the term “hidden work” to describe coordination and exception
handling efforts that produce a substantial fraction of the total labor and schedule pressures in complex
projects. Overloaded workers sometimes fail to respond to communications, thereby compounding the
information supply problem and compromising others’ performance. Complexity and interdependence
thus results not simply in additional direct and communication requirements, but also triggers new
exceptions and errors. Knowledge of this phenomenon forms the basis of many experienced analysts’
skepticism toward ICE performances claims.
The Virtual Design Team simulation system (VDT) is currently the most feature-rich model of
hidden work. For a detailed description of VDT mechanics, see [ Jin and Levitt CMOT]. We have
encoded the results of our observations and interviews at JPL in a VDT model of ICE, and found that
some, but not all of the simulation’s retrospective predictions match the results of a follow-on survey.
We offer a micro-level view of information processing on the four elements of the decision basis.
Engineering actors routinely make decisions in which the amount of information, range of preferences,
work procedure (decision rule) and number of alternatives vary. Our analysis views focused, uneventful
work as consisting of sequences of straightforward decisions. In less routine work, actors occasionally
encounter decisions for which elements of the decision basis are inadequate, unavailable, or incorrect.

Exception Handling
Organizational actors are not generally aware of all the nuances of an organization’s strategic intent and
goals. Similarly, workers will sometimes find that their technical expertise is insufficient to finalize a
work element. The VDT system models perceived technical inadequacy and ignorance of organizational
preferences as exceptions (potential errors) that management must contemplate and, perhaps, order
reworked. In the model, they emerge probabilistically during work, with a frequency based on task
2
Prospect theory [Kahneman and Tversky] observes that people respond to decisions’ contexts,
even when they do not impact the traditional decision basis elements. Because we do not calculate the
engineers’ specific choices, we safely address this framing principally as part of the decision rule.

CIFE Working Paper 8/13/2004 5

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
complexity measures, as well as on the adequacy of the assigned actor’s experience and skills. The
VDT model represents project exception handling as involving an upward flow of exception handling
requests and a downward flow of rework, quick fix, or no action choices along one or two fixed
exception handling hierarchies (project and functional). Because management is the clear authority on
organizational preferences, and also –in traditional organizations- the repository of superior technical
knowledge, the hierarchical VDT exception-handling model captures a micro-organizational adaptation
to uncertainty in the decision bases’ preference and decision rule components.
Exception handling can produce a large fraction of a task’s total work volume, especially for
technically challenging or equivocal projects. Moreover, when a supervisor oversees many actors, each
of whom has complex tasks that are being performed in parallel, the supervisor’s exception handling
workload may become unmanageable. This backlog can result in a failure to properly review the
exceptions, causing a ripple effect of problems extending through all of the manager’s subordinates.

Information Exchange
Some decisions require information that does not simply reside among management, but that a previous
or parallel work task creates during the project. These facts may impact the range of available design
alternatives (as with design configuration interdependence), or they may influence the predicted results
for a given choice. Accordingly, VDT actors request information from others who are engaged in
interdependent work (at a rate that is based on actor skill, prior team experience, and task uncertainty).
In this situation, the simulator routes a virtual information request and possible reply between the actors.
Because this process supplies actors with data produced in other activities, and this data influences the
range and significance of design options, we view the VDT communications model as capturing a
micro-organizational adaptation to gaps in the belief and alternatives components of the general rational
framework’s decision basis.
When an actor performs a task that has a very large number of interdependencies, the time spent
in communications may actually exceed the amount of direct work activity. If the workload becomes
unmanageable, quality may degrade significantly- not just for the principal task, but also for others who
rely upon the activity’s output.

Rework and Design Iteration

In addition to direct, heads-down work, coordination time such as information exchange and meetings,
and decision waiting time, VDT calculates the volume and distribution of rework. Rework results from
handling exceptions conservatively, and consists of performing an activity (or subtask) a second (or
third) time. Conventional project analysis considers rework to be a measure of inefficiency, even in
contexts where its complete elimination is not feasible.
Engineering typically involves repeatedly designing a product, evaluating it against fitness
metrics, and redesigning it according to the results. For example, an initial mission design might require
a higher budget than is available. In this case, engineers proceed to construct a second design, based on
lessons learned from the first, with a greater focus on lowering costs. This “Design iteration” process
enables an engineering team to explore a range of possible designs, and is an integral part of a project
plan. An engineer who builds the same design twice because of file corruption indicates wasteful
rework, while another who considers a second way of building a component is performing valuable
design iteration.
Space mission design involves simultaneously iterating among many designs in each of many
highly interdependent engineering tasks. If a station is unaware of changes to a design element on
which it depends, it risks building an incompatible design. This constitutes rework, in that the effort is
invalid and wasteful. If the station is aware of the change, then constructive, valuable new design
iteration will ensue.

CIFE Working Paper 8/13/2004 6

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
An effective ICE session includes a large work volume of design iteration, and a low volume of
rework. VDT does not distinguish between rework and design iteration, instead using the same
exception handling simulation for both cases.

Simulating Team-X Using VDT

To build a VDT project model of Team-X, we Com parison of Average Hours per Station
created 15 virtual engineering stations, as well as
40
a facilitator and proposal manager, and we 35 M eetings

provided each with an individualized task. In a 30 Design Sessions

25
series of interviews, Team-X participants supplied 20 Decision Wait
15
the tasks’ complexities, work volumes, start 10
Coord.

5 Rework
times, and rework and information exchange 0 Work
networks. Simulating the model showed the

Intellective

Extended
Prediction

WAM
Survey
Prediction
Baseline

Survey
Total

VDT

VDT
predicted results of all the actors working,
exchanging information, and handling exceptions.
Figure 3 compares an average number of
work hours per station, by type of work activity Figure 3: Comparison of work volumes on a sample JPL Team-
and according to a number of sources. Prior to X project from various sources. The columns represent (right to
project start, all Team-X participants requested a left) reported, predicted, surveyed, simplified/simulated, and
retrospectively simulated data. At a high level, there is
time budget to do their work, which we averaged agreement among them.
as the rightmost Work Authorization Memo
(WAM) value. The Survey column reports the data participants provided after project completion. We
retrospectively calibrated the VDT simulation to predict the total work volume for each project task as
well as the direct work, coordination, rework, and time wasted waiting for exception management. The
averages of these values appear in the leftmost, Baseline VDT column.
Figure 3 illustrates that using input data collected at JPL (including work volumes reported
retrospectively by Team-X) we were able to calibrate VDT to produce emergent behavior that matches
an actual Team-X project. This suggests that at an aggregate level, a properly calibrated VDT model
can retrospectively predict the volume and distribution of work for this type of project. Because the
simulation is rooted in the information processing theory, this result coarsely cross validates the
information processing theory, ICE observations, and VDT computational model.

ANALYSIS OF ICE RISKS

Performance Analysis and Risk
VDT provides a range of product, organization, and process performance measures, including emergent
work volumes, a project schedule, and coordination rates. Our calibrated simulation of nominal Team-X
operations found outcome measures each to fall at a qualitatively acceptable level, but to vary in
theoretically significant ways. We describe the VDT outcomes as measures of risk to the mission design
product, the Team-X organization, and the conceptual design process.
As a Monte Carlo simulation, VDT converts qualitative and quantitative project design metrics
into irreducible, mathematical distributions of outcome. Specifically, the system describes outcome
measure distributions using average results and variances that characterize the model’s degree of
certainty. For example, VDT does not predict a single quantity of information exchange requests, but
instead states a simulated average and variability.
Furthermore, VDT does not attempt to model actors’ midstream monitoring of, and intervention
in, project performance. Instead, the system predicts the behavior that would develop if the organization

CIFE Working Paper 8/13/2004 7

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
and process were to proceed as it was initially directed to. For example, VDT may calculate a vastly
extended schedule, whereas real managers would likely observe the developing problems and attempt to
intervene (such as by hiring additional workers).
Because VDT measures likely propensities, rather than certain outcomes, we interpret the
system’s performance measures to characterize the degrees of risk associated with different project
aspects. For example, when a case shows high cost risk, this means that it is likely to produce large cost
overruns unless there are proportionally effective interventions.

Product Risk
The product of a Team-X ICE project is a set of complementary design choices that form the basis of a
mission. We use the term product risk to describe the likelihood that design choices are fundamentally
invalid or inconsistent. Product risk is important because it may lead to an improper decision over
whether to proceed with a mission, or to a mission that is needlessly costly, risky, or extended in
schedule.
In this paper, we do not consider the cost, quality, or schedule of planned missions, but we do
use project behavior to predict the likely accuracy and completeness of the team’s own analysis of these
factors. Team-X requires appropriate stations as well as an effective collaborative process to correctly
estimate the mission’s programmatic risk, costs and schedule. Benjamin and Pate-Cornell highlight the
need for probabilistic risk analysis in this project setting [2004], and Team-X’s new Risk Station
testifies to its perceived importance at JPL [ JPL Risk Station Paper]. Our analysis of product risk is
distinct from, and complimentary to these efforts.
Our analysis highlights the impact of organizational risk factors on process quality because they
are estimated to contribute to 50-75% of major modern catastrophes [ MEP]. For descriptions of over a
hundred organizational risk factors, and related literature reviews, see Ciaverelli [] or Cooke and
Gorman []. Important factors that VDT does not evaluate include conformity, which decreases the
likelihood that individuals will contradict peers’ public, erroneous statements [- Festinger?].
“Groupthink”, reduces the likelihood of thorough, critical evaluation of alternatives in a group setting [
Janis]. Finally, the “Risky shift” phenomenon, that leads groups to select choices that are more risky
than those which any participant would individually choose [ Bem]. Each of these organizational factors
acts principally to reduce the quality of the selected design.
VDT does calculate several measures that regard risk to the product design. Overloaded or
unqualified actors tend to ignore exceptions and information exchange requests, which contributes to
three product risk metrics. Project risk measures the rate of rework or design iteration that is ordered in
response to interdependencies among functionally related tasks. When a simulation shows high project
risk, this indicates a propensity for failures in the “System of systems” that involve more than one
station. Functional risk measures the rate of rework (or design iteration) that is ordered for individual
tasks. High functional risk at a particular station indicates that the station’s design is likely to be
independently faulty. Finally, communications risk is the fraction of information exchange requests that
stations take time to complete. High communications risk indicates that interrelated tasks are not always
sharing information appropriately, which tends to reduce integrated design quality. We can predict
overall design quality using VDT by viewing these metrics at an aggregate project level, or we may drill
down to characterize the product in detail. For example, elevated project risk at the Power station
indicates that other subsystems have not redesigned according to its needs, and a high communications
risk at the Cost station suggests that the estimates do not include relevant design details.

Organization Risk
By organization risk, we refer to the likelihood and consequences of events that degrade the operating
effectiveness of the design team (Team-X) itself. VDT measures several important pressures on the

CIFE Working Paper 8/13/2004 8

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
organization that can, especially over time, reduce its operating effectiveness. VDT tracks the amount
of work that backlogs for each actor, and this can cause the sense of time pressure that researchers have
shown to cause errors []. Compounding this is the sense of actor frustration that VDT measures as the
fraction of information exchange requests that are not granted, the fraction of exceptions that are
ignored, and the amount of time that a participant spends waiting for management decisions. People
who are under time pressure or stress are more likely to make poor decisions [], and errors of oversight.
In addition, they are more likely to burn out and leave a position, and in complex positions is another
risk factor.
In addition to understanding the baseline performance at Team-X, it is important to know
whether the ICE project design has structural stability, or whether it is sensitive to small deviations that
can be difficult to anticipate. For example, Team-X might require that a specific station be staffed by an
engineer of extraordinary skill, and might stumble when happenstance requires a more average member
to substitute. ICE should serve a routine strategic function only if it is effective both in optimal
conditions and under foreseeable organizational and other variations.
We ran an intellective VDT experiment that excludes many team-specific factors and found it to
preliminarily demonstrate stability in the project design. In the baseline case, for example, the
participants’ work volumes and experience levels vary by station (in accordance with Team-X
interviews), while those for the idealized case are uniform. Although we have not conducted a complete
sensitivity analysis, our more generic experiment shows sufficiently similar outcomes to the tailored
model that we view the project design robust within a nominal range of staffing and task variations.

Process Risk
We consider three measures of process risk that anticipate the perceived efficiency of the design study
project. These are the cost, schedule, and structural stability of the simulated design project. Our VDT
model uses the total work volume among all engineers and supervisors to represent the cost of an ICE
design project. Figure 3 shows that our calibrated Team-X model produces a similar cost structure to
that reported in surveys (with the exception of meetings, which VDT does not schedule in contingency
with project performance). Although VDT calculates detailed schedules including average start and
finish times for each station’s task, we compare alternative cases using the total project schedule, or
time between execution of the first and last work items. For structural stability, we use the same
technique as described under organization risk.

Knowledge Distribution
Persistent dynamics of change in the distribution of technical knowledge produce an important deviation
between the traditional, hierarchical information processing theories and modern, multidisciplinary
collaborative engineering behavior. As projects become more technically complex and dynamic, we
find that actors of superior knowledge or technical skill come to handle organizational deficiencies in
work procedures and alternative sets.
VDT is calibrated with a broad range of academics’ and professionals’ project study experiences,
and it has made some strikingly accurate predictions of project performance [ Lockheed, others?].
Because our VDT model is calibrated with the theory and experience of these traditional, hierarchical
projects, it offers predictions like those of an expert in “traditional” project planning. These predictions
are based on the assumption that workers route exceptions only through an authoritative management
hierarchy, and that information exchange only transpires between actors engaged in interdependent tasks
(or through manually scheduled meetings). Our Team-X model using the current, standard version of

CIFE Working Paper 8/13/2004 9

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
VDT3 matches ICE’s common, purely
Hidden Work hierarchical exception-handling processes
40 from station, to facilitator, to proposal
35
30
manager.
Hours 25
Detail As technology accelerates, it
20
15 Survey becomes increasingly difficult for
10
5
organizational managers to maintain
0 sufficient knowledge to resolve technical
Systems
Cost Estimator Structures Thermal problems. When supervisors lack the
specialized knowledge that is required to
Mission Design Propulsion Software Ground Systems
assist subordinates in technical work,
Station
organizations route technical exceptions to
domain experts who reside either
Figure 2: “Hidden work” consists of coordination and horizontally across organizational lines, or
rework activities that most software systems and less even outside the company [ Contractor].
sophisticated human planners fail to account for. This In our developing theoretical model,
chart contrasts the hidden work reported in a “Survey” organizational managers continue to
of Team X with the “Detail” hidden work that the consult on preferences, but actors locate
VDT simulator predicts. The differences between and retrieve procedural expertise from a
these values preliminarily quantify the inconsistency distributed network of actors with
between traditional theories and those required to functionally differentiated skills.
account for ICE. Planners with experience limited to Similarly, although interdependent
traditional projects are vulnerable to similar collaborators in knowledge work continue
miscalculations. to provide important beliefs about the
work in process, as complexity increases
they become less qualified to shape designers’ ever expanding alternative sets.

Comparing Traditional and Knowledge Work

Information workers’ efficiency relies upon a knowledge network that is free of expertise bottlenecks,
just as traditional projects’ success rests upon a hierarchical management that is free of decision-making
logjams. As the rate of change of technical knowledge in a field increases, the information processing
prominence of organizational managers thus diminishes while that of technically capable experts
increases [ Diane Bailey’s new paper on CEs vs. EEs]. Organizations eventually adapt to redistribute
this coordination load among specialist participants. Team-X, for example, has gradually adjusted
station definitions, added new stations, and substantially enhanced its information technology tool suite.
We compare traditional and knowledge-network based exception handling by first reviewing the
simulated (VDT) authority hierarchy results in increased detail. Our simulation does not currently
exhibit the network-based exception-handling phenomena that are becoming increasingly common in
knowledge work. We contrast this against the work volumes surveyed at JPL. These values incorporate
the hierarchy and the technical information flow among peers that characterizes an effective ICE
knowledge network.
Although we retrospectively calibrated VDT to show the same project coordination volumes
reported by Team-X participants, the simulated distribution of hidden work among individual tasks did
not match perfectly (Figure 4). This may result from the traditional, hierarchical framework’s inability
to predict bottlenecks in the participants’ knowledge network. Based on this result, we alert
organizational designers who are steeped in traditional theory to the danger of underestimating the
3
SimVision 3.11

CIFE Working Paper 8/13/2004 10

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
coordination load that technical experts will experience in decentralized knowledge-based projects such
as ICE.
Although management plays an important leadership role in ICE, at JPL we observed virtually
no project delays that were accountable to a management bottleneck. In contrast, VDT predicts that
Team-X engineers waste approximately ten percent of their time waiting for management decisions,
dramatically illustrating the insufficiency of the hierarchical structure. The ten percent figure suggests
an amount of acceleration that projects would experience by switching from a typical, bottlenecked
management hierarchy to a balanced knowledge network. Because the VDT formulation fails to capture
other qualitative features of ICE practice, however, we require additional data points to measure
precisely the importance of discrepancies between knowledge-based and traditional work theories (or to
determine whether the model can support intervention through systematic, prospective prediction). Our
current result nevertheless reinforces the assertion that planners – including human ones – who depend
on traditional methods (theory, experience, or models) to design a decentralized, collaborative
engineering structure (like ICE) are likely to overestimate the importance of management oversight.

REMARKS ON METHODOLOGY
In recent years, the computational modeling of organizations has enjoyed a popular resurgence among
researchers seeking to better understand new and established theories [March 01, and Burton 01]. By
grounding a computational model explicitly in a theoretical framework, researchers can explore complex
ramifications of a theory (or set of theories) that extend qualitatively beyond the reach of human
intuition. In addition, our team has used models to quantitatively predict the effects of theoretical and
practical changes in a baseline model. Following the tradition of mathematical proof, when a model of
theory produces a recognizable pattern of results, we interpret this and make a new claim. In a perfect
world, if the new hypothesis is shown to be false, the model’s theoretical premises are disproved (a
“proof by contradiction”).
At this time, however, model based theory generation is new to domains as complex as project
design. In this paper, we apply the technique in its most common modern form- as an engineering
method that relies in part on intuition and external observation to validate its claims. Therefore, we
accompany our model analysis with intuitive descriptions as well as observational data.
“In their anxiety to be scientific, students of psychology have often imitated the latest forms of
sciences with a long history, while ignoring the steps these sciences took when they were young”
-Psychologist Solomon Asch
The recent expansions of particularly compatible social science theories and analytic techniques
are creating an exciting time for computational organizational modelers [March, and Burton, in Lomi
and Larsen 2001]. Properly applied, the methodology facilitates practical organizational design just as
effectively as it strengthens scholarly results [Kunz et al 1998]. Our work illustrates the power of
computational organizational models to both extend and lend specificity to qualitative theory,
ethnography, and survey research.
In planning a project or adapting one midstream, sometimes alternatives may be introduced
directly to the organization. At other times, it may be more economical to test these interventions first in
a computational model. Schedule tracking systems such as Primavera are the most frequently consulted
quantitative project models, but they are not the most sophisticated. When testing interventions in the
Virtual Design Team (VDT) simulator, for example, planners can compare project participants’
predicted backlog, coordination effectiveness, schedule risk, and other results between many alternative
cases [Kunz et al 1998, ACM; Jin et al 1995, Levitt 1996, Levitt et al, Management Science]. In this
way, modelers can plan joint adaptations to organizations, processes, and culture that will meet a

CIFE Working Paper 8/13/2004 11

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
project’s goals. In time, our team believes tools like VDT will enable us to engineer projects with a
comparable methodology and confidence as is demonstrated on today’s automobiles [ Bridges text] In
VDT, for example, we can select from a list of alternative intervention scenarios, and simultaneously
compare the results of multiple cases. By weighing actor backlog and other results between alternative
cases, VDT users are able to jointly design and adapt organizations, processes, and culture in order to
meet a project’s goals.
Every model contains assumptions that limit the range of its results’ applicability. Many of
VDT’s basic assumptions are fairly well documented and understood. As a result, modelers have been
able to apply the system successfully in a very broad range of settings. For example, the authors have
personally developed VDT models of projects as diverse as aerospace engineering, facility design and
construction, and software development.
The modeling project at Team-X was noteworthy for bringing to the fore particularly many
circumstances within which VDT had not been tested. Limits of the model include:
No explicit product model
FTE allocations fixed over project life
Meetings never scheduled on-the-fly
Exceptions, communications are only 2-way
Exceptions use reporting hierarchy, not knowledge network
Work difficulty modeled as routine
One task, skill, actor per station
An unanticipated project scope extension was not included
Limited distinction between rework and design iteration

NEXT STEPS
Evolutionary organizational theorists would predict that if ICE performance were viable, it would be
widespread. The system perspective we present in the introduction suggests that this apparent conflict
may result from a careful balance of factors that aren’t ordinarily available in combination. For
example, moving to a flat hierarchy or task parallelism, alone, might be disastrous in a traditional
organization, even though they are complementary in ICE.
We are designing a computational experiment to investigate this issue by calculating the impacts
of each enabling factor from Table 1. We hope that this analysis will more clearly illuminate the
interactions between enabling factors and explain:
1. Can a single calibration of the VDT engine simultaneously demonstrate ordinary teams’
performance and that of Team-X?
2. Are intermediate states, in which some, but not all enabling factors are satisfied, better or worse
than traditional practice?
3. How precipitously does performance drop off when enabling factors are reduced in strength?
4. Is there a sequence of interventions that leads from traditional conceptual design to ICE behavior
without reducing performance at any step?
5. How do organizational risk properties change under these conditions?
6. Is there a tightrope of high performance between tradition and ICE that involves simultaneous,
gradual improvements in the enabling factors?
7. Do certain intuitive compound factors, such as collocation, match theoretical predictions, and do
they complement one another as interventions? [ Tse Tse Wong and Richard Burton, CMOT]

CIFE Working Paper 8/13/2004 12

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt

ACKNOWLEDGMENTS
The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute
of Technology, under a contract with the National Aeronautics and Space Administration. We are
grateful to NASA ARC's Engineering for Complex Systems Program for supporting the work under
Grant Number NCC21388, and for providing valuable feedback. We are especially indebted to the Jet
Propulsion Laboratory’s Team-X, including Rebecca Wheeler, Robert Oberto, Ted Sweetser, and Jason
Andringa.
We also thank the Kozmetsky Research Fellowship Program and the Stanford Media-X Center
for graciously funding this continuing research.
We further appreciate collaboration with Ingrid Erickson and Pamela Hines; The Stanford
ReVeL research team (Including Ben Shaw, Cliff Nass and Syed Shariq); The Center for Integrated
Facility Engineering (John Kunz and Martin Fischer), The Center for Design Research (Ade Mabogunje
and Larry Leifer); And the Virtual Design Team Research Group.

BIBLIOGRAPHY
Asch, S.E. 1987 (original work published 1952) “Social Psychology”. New York: Oxford University
Press
Bem, D., M. Wallach, and N. Kogan 1965 “Gropu Decision Under Risk of Aversive Consequences”
Journal of Personality and Social Psychology, 1(5), 453-460
Benjamin, J. and Pate’-Cornell, M.Elizabeth 2004 “Risk Chair for Concurrent Design Engineering:
Satellite Swarm Illustration” Journal of Spacecraft and Rockets Vol. 41 No. 1 January-February
2004
Burton R and Obel B. 2004 “Strategic Organizational Diagnosis and Design: Developing Theory for
Application 3rd Edition”. Boston: Kluwer Academic Publishers.
Carley, K. 1996 “Validating Computational Models” Working paper prepared at Carnegie Mellon
University
Chachere, J., Kunz, J., and Levitt, R. 2004, “Can You Accelerate Your Project Using Extreme
Collaboration? A Model Based Analysis” 2004 International Symposium on Collaborative
Technologies and Systems; Also available as Center for Integrated Facility Engineering Technical
Report T152, Stanford University, Stanford, CA
Ciavarelli, A. 2003 “Organizational Risk Assessment” Unpublished manuscript prepared at the Naval
Postgraduate School
Cooke, N., J. Gorman, and H. Pedersen 2002 “Toward a Model of Organizational Risk: Critical
Factors at the Team Level” Unpublished manuscript prepared at New Mexico State University and
Arizona State University
Covi, L. M., Olson, J. S., Rocco, E., Miller, W. J., Allie, P. 1998 “A Room of Your Own: What Do
We Learn about Support of Teamwork from Assessing Teams in Dedicated Project Rooms?
Cooperative Buildings : Integrating Information, Organization, and Architecture. Proceedings of
First International Workshop, CoBuild '98, Darmstadt, Germany, February 25-26, Norbert A.
Streitz, Shin’ichi Konomi, Heinz-Jürgen Burkhardt (eds.). Berlin ; New York : Springer: 53-65.
D. B. Smith, “Reengineering Space Projects”, Paris, France, March 3-5, 1997.
D. Smith, and L. Koenig, “Modeling and Project Development”, European Space & Research Contre,
Noordwijk, The Netherlands, November 3, 1998.
Eisenhardt, K. 1989 “Building Theories from Case Study Research”, Academy of Management
Review Vol. 14 No. 4 532-550

CIFE Working Paper 8/13/2004 13

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
Eisenhardt, K. 1993 “High Reliability Organizations Meet High Velocity Environments: Common
Dilemmas in Nuclear Piower Plants, Aircraft Carriers, and Microcomputer Firms” In K.H,.
Roberts (Ed.) New Challenges to Understanding Organizations New York: MacMillan, pp. 117-
136
Erickson, I. 2004 “Extreme” Unpublished manuscript prepared at Stanford University
Galbraith, J. R. (1977), Organization Design. Reading, MA: Addison-Wesley.
Garcia, A., Kunz, J., Ekstrom, M., and Kiviniemi, A. 2003 “Building a Project Ontology with Extreme
Collaboration and Virtual Design and Construction” Center for Integrated Facility Engineering
Technical Report T152, Stanford University, Stanford, CA
Glaser, B. and A. Strauss 1967 “The Discovery of Grounded Theory” Chicago: Aldine
Heath, C., Luff, P. 2000 Team Work: Collaboration and Control in London Underground Line Control
Rooms. Technology in Action, Cambridge, Cambridge University Press: 88-124
Howard, R. and Matheson, J. (eds.) 1983 Readings on the Principles and Applications of Decision
Analysis, Decision Analysis, Strategic Decisions Group, Menlo Park, CA
J. Leigh, A. Johnson, K. Park, A. Nayak, R. Singh, V. Chowdhry, “Amplified Collaboration
Environments,” VizGrid Symposium, Tokyo, November 2002
J. Smith, “Concurrent Engineering in the Jet Propulsion Laboratory Project Design Center”, Society of
Automotive Engineers, Long Beach, CA, U.S.A., June 4, 1998.
Janis, I. 1982 “Stress Attitudes, and Decisions: Selected Papers” New York, Praeger Publishers
Jin, Y.; R. Levitt; T. Christiansen; J. Kunz. 1995. "The Virtual Design Team: Modeling
Organizational Behavior of Concurrent Design Teams,” International Journal of Artificial
Intelligence for Engineering Design, Analysis and Manufacturing, Vol.9, No.2, (April) 145-158,.
Kunz, J.; T. Christiansen; G. Cohen; Y. Jin; R. Levitt. 1998. "The Virtual Design Team: A
Computational Simulation Model of Project Organizations," Communications of the Association
for Computing Machinery, (November) pp.84-92.
Levitt, R. 1996. “Organizational Analysis and Design Tools: State of the Art,” First International
Conference on Computational and Mathematical Organization Theory, Monterrey, Mexico
(October)
Lomi, A. and E. Larsen, 2001. Dynamics of Organizations, AAAI and MIT Press
Luce, R. and H. Raiffa 1990 “Utility Theory” In Moser, P.K. (Ed.) Rationality in Action:
Contemporary Approaches (pp. 19-40) Cambridge University Press: New York, NY
Marais, K., N. Dulac, and N. Leveson 2004 “Beyond Normal Accidents and High Reliability
Organizations: The Need for an Alternative Approach to Safety in Complex Systems” Working
paper prepared at the Massachusetts Institute of Technology
March, J. G., Simon, H. A. (1958) Organizations. New York, John Wiley & Sons, Inc
March, James G. A Primer on Decision Making: How Decisions Happen. New York: Free Press,
1994.
Mark, G. and DeFlorio, P. 2001 “An Experiment Using Life-size HDTV” Proceedings of the IEEE
Workshop on Advanced Collaborative Environments, San Francisco, CA
Mark, G., 2002. "Extreme Collaboration" Communications of the ACM, Volume 45, Number 6
(June), pp. 89-93.
Mark, G., Abrams, S., and Nassif, N. 2003. “Group-to-Group Distance Collaboration: Examining the
‘Space Between’”. Proceedings of the 8th European Conference of Computer-supported
Cooperative Work (ECSCW’03), 14-18. September 2003, Helsinki, Finland, pp. 99-118.
Maule, A. and A. Edland 1997 “The Effects of Time Pressure on Human Judgment and Decision

CIFE Working Paper 8/13/2004 14

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
Making” in R. Ranyard, W. Crozier, & I. Svenson (Eds.) Decision Making: Cognitive Models and
Explanations (pp. 189-204) New York: Routledge
Meshkat, L., and Oberto, R. 2004 “Towards a Systems Approach to Risk Considerations for
Concurrent Design” Working paper prepared at the Jet Propulsion Laboratory, California Institute
of Technology
Monge, P. R., & Contractor, N. S. (1999). Emergent communication networks. In L. Putnam & F.
Jablin (Eds.) Handbook of organizational communication (second edition). Newbury Park, CA:
Sage.
NASA 1995 “NASA Systems Engineering Handbook”, National Aeronautics and Space
Administration
Nasrallah, W.; R. Levitt, P. Glynn. 2003. “Interaction Value Analysis: When Structured
Communication Benefits Organizations” Organization Science forthcoming in 2003
Olson, J. S., Covi, L., Rocco, E., Miller, W. J., Allie, P. 1998 A Room of Your Own: What Would it
Take to Help Remote Groups Work as Well as Collocated Groups? Proceedings of CHI’98:
Conference on Human Factors and Computing Systems. Los Angeles, California, April 18-23,
1998. New York, ACM Press: 279-280.
Orasanu, J. “Organizational Risk Model” Working paper prepared at NASA Ames Research Center
Paté-Cornell, M.E., 1990 "Organizational Aspects of Engineering System Safety: The Case of
Offshore Platforms," Science, Vol. 250, November 1990, pp. 1210-1217.
Murphy, D.M. and M.E. Paté-Cornell 1996, "The SAM Framework: A Systems Analysis Approach to
Modeling the Effects of Management on Human Behavior in Risk Analysis", Risk Analysis,
Vol. 16, No. 4, pp.501-515.
Perrow, C. 1984 “Normal Accidents: Living with High-Risk Technologies” New York: Basic Books
Perrow, C. 1994 “The Limits of Safety: The Enhancement of a Theory of Accidents” Journal of
Contingencies and Crisis Management, 2, (4), 212-220
Powell, W. and DiMaggio, P. (Eds.) 1991 “The New Institutionalism in Organizational Analysis”
Chicago: University of Chicago Press
Reason, J. 1997 “Managing the Risks of Organizational Accidents” Brookfield: Ashgate Press
Roberts, K. 1990 “Managing High-Reliability Organizations” California Management Review, 32, (4),
101-113
Sercel, J. 1998 “ICE Heats Up Design”, Aerospace America July 1998
Teasley, S., Covi, L., Krishnan, M. S., Olson, J. S. 2000 How Does Radical Collocation Help a Team
Succeed? In Proceedings of CSCW’00: Conference on Computer-Supported Cooperative Work.
Philadelphia, Pennsylvania, December 2-6, 2000. New York, ACM Press: 339-346.
Teasley, S., Covi, L., Krishnan, M., Olson, J. 2000. “How Does Collocation Help a Team Succeed?”
ACM Conference on Computer Supported Cooperative Work.
Thompson, J., 1967. “Organizations in Action: Social Science Bases in Administrative Theory”,
McGraw-Hill, New York
Tversky, A. and D. Kahneman 1974 “Judgment Under Uncertainty: Heuristics & Biases” in D.
Kahneman, P. Slovic, &A. Tversky (Eds.) Judgment Under Uncertainty: Heuristics and Biases.
Cambridge: Cabridge University Press
Wall, S. “Design Process Enhancements for Planetary Missions”, 4th International Conference on
Lowcost Planetary Mission, Maryland, USA, May 2, 2000.
Wall, S. “Reinventing the Design Process: Teams and Models”, IAF, Specialist Symposium: Novel
Concepts for Smaller, Faster & Better Space Missions, Redondo Beach, California, USA, April
19-21, 1999

CIFE Working Paper 8/13/2004 15

Observation, Theory, and Simulation of Integrated Concurrent Engineering Chachere, Kunz and Levitt
Wall, S., D. Smith,, L. Koenig, and J. Baker, “Team Structures and Processes in the Design of Space
Missions”, MTG: 1999 IEEE Aerospace Conference, Snowmass at Aspen, CO, U.S.A., March 6-
13, 1999
Scott, W. R. 1998 “Organizations: Rational, Natural, and Open Systems (4th Edition)” New Jersey:
Prentice-Hall
Weick, K., Roberts, K. 1993 Collective Mind in Organizations: Heedful Interrelating on Flight
Weick, K.E. and K. Sutcliffe 2001 “Managing the Unexpected: Assuring High Performance in an Age
of Complexity”, Jossey-Vass, San Francisco, CA
Wheeler, R., J. Hihn, and B. Wilkinson “Distributed Collaborative Team Effectiveness: Measurement
and Process Improvement” Working paper prepared at the Jet Propulsion Laboratory / California
Institute of Technology
Witt, K., Giorcelli, R., Darrah, M., Ives, B. 2004 “DEVISE: A Collaborative Virtual Environment for
Integrated Concurrent Engineering”, 2004 International Symposium on Collaborative
Technologies and Systems
Zajonc, Robert B. “Social Facilitation” Science 149, July 16, 1965 269-74

CIFE Working Paper 8/13/2004 16