Proc. 2016 Canadian Engineering Education Association (CEEA16) Conf.

Designing Rubrics to Assess Engineering Design, Professional

Practice, and Communication Over Three Years of Study
Natasha Lanziner David S. Strong
Master’s in Mechanical Engineering Professor & NSERC Chair in Design Engineering
Queen’s University at Kingston, Ontario Queen’s University at Kingston, Ontario
[email protected] [email protected]

Abstract – When using rubric-based assessment of introductory, one-week, design/build/test project aimed
students’ understanding of design process in project- mainly at team building and reflective learning. The
based courses, it is important to provide specific feedback second (P2) phase is a 5 week, open ended,
for major design process elements while avoiding overly multidisciplinary project, with parallel teaching and active
prescriptive descriptors [8]. This paper details the learning workshops. The third (P2) Phase engages
development process of a sequence of rubrics used for student teams in a six-week discipline-related project,
assessment in successive second, third and fourth year typically with a prototyping or design-of-experiment lab
project-based courses. A major consideration in the requirement.
rubric development process was to ensure the alignment Split into APSC200, engineering design and
of assessment with course learning outcomes that can be APSC293, engineering communications, EDPS II is
easily mapped to the CEAB graduate attribute implemented faculty-wide. EDPS III includes discipline-
accreditation requirements. In the second year course, the specific courses and/or the elective multidisciplinary
rubrics are used to provide students with directed course APSC381, Advanced Design for Innovation.
feedback as they learn the basics of engineering design EDPS IV includes discipline-specific capstone courses
process. The third and fourth year rubrics progress from and/or the elective APSC480, Multidisciplinary Industry
the second year analytic rubrics by employing elements of Engineering Design Project. Together, APSC381 and
holistic assessment. The purpose of evolving these rubrics APSC480 comprise the Multidisciplinary Design Stream
year over year is to find a balance between the students’ (MDS) [3].
learning and development in design process whilst The majority of the assessment of engineering design,
accommodating variation in projects. This ultimately professional practice, and communication in
provides students with greater flexibility and encourages APSC200/293, APSC381, and APSC480 is completed by
responsibility as they progress through their program. assessing engineering design reports. In APSC200/293,
students submit a preliminary, interim, and final report for
Keywords: Rubrics; Graduate Attributes; Assessment; the P1 project and a proposal and final report for the P2
Engineering Design project. In APSC381, students submit a preliminary,
interim, and final report. In APSC480, students submit a
series of milestone “briefs”, an interim, and final report.
1. INTRODUCTION In each of the courses, the reports for a given project are
successive and build on the previous reports. This paper
Since 2011, Queen’s University has been discusses the development process of a sequence of
implementing a four-year Engineering Design and rubrics to assess engineering design, professional practice,
Practice Sequence (EDPS) as a core requirement in all and communication in the engineering design reports
programs to enhance students’ design, innovation, and submitted for APSC200/293, APSC381, and APSC480.
professionalism capabilities. Designed as a project based Although EDPS I also incorporates rubric-based
sequence to contribute to the graduate attribute assessment, the development process of the EDPS I
requirements of the Canadian Engineering Accreditation rubrics is not detailed in this paper.
Board, detailed learning outcomes governed the
implementation of the EDPS [1]. As a part of the common 2. REVIEW OF LITERATURE
first year, EDPS I, APSC100 incorporates three modules
on problem analysis and modelling, experimentation and Prior to developing the sequence of rubrics, a review
measurement, and engineering design. EDPS II of assessment literature for engineering design,
APSC200/293, engineering design/engineering professional practice, and communication was conducted.
communication is a one semester, faculty-wide course The purpose of the literature review was to establish a
comprised of three phases. The first (P0) Phase is an conception of engineering design and the engineering

CEEA16; Paper 125

Dalhousie University; June 19 – 22, 2016 – 1 of 6 –
Proc. 2016 Canadian Engineering Education Association (CEEA16) Conf.

design process, assess perspectives on assessment of next level. Some instructors who have used the ICE
engineering design, and to evaluate existing rubric-based approach for rubrics suggest that the inclusion of grade
assessment methods. values in the rubric causes students to focus less on the
learning descriptors. Because the ICE approach uses
2.1. Assessment of Engineering Design, measures associated with the learning process, it is useful
Professional Practice, and Communication in assessment of the development of process skills [10].
Prior to the development of the rubric sequence
There are multiple approaches to describing the detailed in this paper, the ICE approach was used for
engineering design process in the literature. Generally, it assessment in APSC200/293 for its applicability in
can be defined as a complex process beginning with an assessing students’ development of design process skills.
analytical phase that involves defining the problem and The ICE approach was also shown to be successful when
conducting background research and ending with an assessing engineering students’ development of
implementation phase. There often are multiple iterations professional skills [4]. Students were more capable of
of the phases within the process [Lit review]. The design grasping the concepts of the ICE learning levels than the
process taught in APSC200/293, APSC381, and nomenclature in Biggs’ and Collis’ Structure of Observed
APSC480, while unique, is most closely modelled by Learning Outcomes (SOLO) theory. The simple
Dominick et al.’s four-phase process as described in Tools vocabulary in the ICE approach also allowed for easier
and Tactics of Design [9]. The four phases are Defining communication of course learning outcomes and
the Problem, Formulating Solutions Developing Models assessment of the outcomes. When using ICE rubrics, the
and Prototypes, and Presenting and Implementing the instructors and teaching assistants (TAs) noted they were
Design [9]. able to provide more formative feedback to students with
In a review of literature on engineering design fewer frustrations in the grading process [4].
assessment, three key features of good assessment tools
were suggested to be systematic in approach, flexibility of 3. PURPOSE OF THE RUBRICS
application, and efficient in terms of time and cost [6].
Defined as guidelines that “differentiate between levels of The rubric development process for the EDPS courses
student proficiency,” rubrics are commonly used for was guided by the four main purposes of the sequential
assessment. Woodhall and Strong conducted a study of an rubrics: (i) alignment with the course learning outcomes;
Alternative Assessment Method (AAM) for use in the (ii) mapping to the graduate attributes; (iii) incorporation
MDS at Queen’s University. The design of the AAM of holistic assessment to allow for grade calibration and
included the design of rubrics to assess engineering variation in project types; and (iv) guidance and feedback
design, communication, and professional practice. When for students and teaching assistants. During development,
designing rubrics, the authors noted the importance of the major elements of the rubrics considered were the
alignment with course learning outcomes [11]. criteria, descriptors of criteria, weightings of criteria, and
levels. Because of previous successes in applications for
2.2. The ICE Assessment Approach assessment in engineering design and professional
practice courses, the ICE approach to assessment was
The ICE approach assesses students by measuring their used in the development process of the rubric sequence.
development through three stages of learning. ICE is an
acronym for these learning stages, standing for Ideas, 3.1 Alignment with Course Learning Outcomes
Connections, and Extensions. Ideas are demonstrated
when students convey the fundamental concepts, For students to improve, it is important that students
including vocabulary and definitions and the basic steps have an understanding of the learning outcomes [11].
in a process. Connections are made when students Students are presented with the detailed learning
demonstrate relationships among the elements in the outcomes for APSC200/293, APSC381, and APSC480.
Ideas stage or between what they have learned and what Oehler and Walker suggest that when learning how to
they already know. Extensions are made when students solve complex, open-ended problems, it is important for
use their learning in novel ways by finding deeper students to have knowledge of the assessment methods to
meanings and applications. When applied to rubrics, the guide their learning [5]. It is most important that the
ICE approach uses qualitative descriptors to describe the sequence of rubrics align with the course learning
characteristics of the learning stages [10]. This approach outcomes to correctly guide student learning and to
is inherently different than purely quantitative rubrics that provide assessment of the desired learning in the course.
often itemize the expected content of the work at different The focus of the EDPS courses, as driven by the
levels. By using qualitative descriptors, instructors can learning outcomes, is on the process of solving complex
provide students with guidance for how to advance to the problems as opposed to the final solution. In alignment
with the outcomes, the main purpose of the sequence of

CEEA16; Paper 125

Dalhousie University; June 19 – 22, 2016 – 2 of 6 –
Proc. 2016 Canadian Engineering Education Association (CEEA16) Conf.

rubrics is to assess the process used by students and not to process improvement. Depending on the nature of the
the final solution resulting from the process. Assessment project, the applicability of some of the factors in the
methods that focus on the final solution may detract from engineering design process varies. The incorporation of
students’ learning of process and techniques for solving holistic assessment allows for alignment with course
open-ended problems [11]. learning outcomes and consideration of the potential
variance in applicability.
3.2 Mapping to Graduate Attributes
3.4 Guidance for Students and TAs
Since 2009 when the Canadian Engineering
Accreditation Board (CEAB) began requiring assessment McMillan described three approaches to assessment as
of engineering graduate attributes, significant focus has assessment of learning, assessment for learning, and
been placed on assessment tools that can be used for both assessment as learning. Assessment for learning and as
course and graduate attribute assessment. Secondary to its learning can be considered preferable as they provide
use for course assessment, the rubric sequence is to be immediate feedback to enhance students’ motivation.
used for graduate attribute assessment to show Assessment for learning provides feedback to students
improvement within the courses and over the three-year whereas assessment as learning fosters student self-
sequence of courses. monitoring of learning [7]. An additional purpose of the
rubric sequence is to provide feedback to students on their
3.3 Incorporation of Holistic Assessment learning while providing students with guidance so they
can learn to monitor their own learning.
Engineering design is difficult to assess due to the APSC200/293 and APSC381 commonly employ both
complexity of design thinking, the nonlinear and iterative undergraduate and graduate TAs and APSC480 employs
nature of the design process, and the range of factors that graduate TAs. The sequence of rubrics also serves to
contribute to engineering design. As limiting factors in guide TAs in assessment as they become more
the assessment process, time and cost do not typically experienced. TAs feedback comments often suggest that
allow for detailed assessment criteria that consider every they learn a great deal about design as well as leadership
complexity and factor of the design process [6]. The by being involved at the mentoring and assessment level.
incorporation of holistic assessment in rubrics can reduce
the time and cost of assessment and grading calibration 4. THE RUBRIC SEQUENCE DESIGN
while ensuring the applicability of the rubrics to a range
of open-ended engineering design problems. Guided by the learning outcomes and the ICE
approach to rubric language and style, three sets of
3.3.1 Grading Calibration. The instruction teams for successive rubrics with four grading levels were created.
APSC200/293, APSC381, and APSC480 include a Each set includes rubrics for the major reports in that
coordinator, course administrator, multiple instructors course. Because the reports are meant to build on one
(one of whom may also be the coordinator), and multiple another as students work through their projects, the
teaching assistants. Grading calibration is required on two rubrics also build on one another. The final report rubrics
levels: within reports, and within instructors and teaching include the criteria in the interim report rubrics, and the
assistants. The consideration of each detailed criterion interim report rubrics include criteria in the preliminary or
combined with a holistic appraisal can improve the time proposal report rubrics. This “scaffold” format serves to
efficiency of grading calibration. guide students learning as they work through their
projects by establishing a pace and increasing quality
3.3.2 Variation in Projects. There is very little variation level for moving through the design process. In the
in projects in the APSC200/293 P1 Phase, however the interim and final report rubrics, the weightings of the
variation increases with discipline-specific projects in the rubric criteria are distributed such that the focus shifts to
P2 Phase. The projects in APSC381 are designed to have the new criteria. For some criteria, the descriptors and
relevant applications of the course material and to be levels are altered within the successive rubrics to
appropriate for multidisciplinary teams. There is a accommodate increases in the level of expectation.
maximum of 12 different projects assigned each year A minor modification was made to the ICE assessment
(typically 2 teams per project for illustration of different approach to define the four levels as Below Minimum
solutions), with some variation in the type of project, Standards (<50%), Ideas (50-65%), Connections, (65-
typically focusing on product design and/or improvement. 80%), and Extensions (80-100%). By defining the levels
The multidisciplinary projects in APSC480, sponsored by as stages in the learning process, the rubrics incorporate
industry partners, have the most variation. In the past, the assessment for learning approach as defined by
projects have included types ranging from conceptual McMillan. A grade range is provided for each level to
product and system design, to techno-economic analyses, remain practical for use in course assessment.

CEEA16; Paper 125

Dalhousie University; June 19 – 22, 2016 – 3 of 6 –
Proc. 2016 Canadian Engineering Education Association (CEEA16) Conf.

Following an approach similar to Woodhall and holistic assessment methods are further incorporated. An
Strong, the criteria in the sequence of rubrics contribute to example of the evolutions in descriptors, criteria, and
categories that include design process and technique weightings is shown in the Appendix for the problem
elements, written communication, and professional definition category in the final report rubrics.
practice elements. The design process criteria in the
rubrics were partly developed from Dominick et al.’s 4.1 APSC200/293
design process model [11]. To align with the evolving
learning outcomes through the courses, there are As the second course in the EDPS, APSC200/293
additional criteria in the APSC381 and APSC480 rubrics, covers the “fuzzy front end” of design process tools,
some of which are extracted from typical professional techniques and professional practice. The APSC200/293
engineering expectations. The criteria were partitioned in detailed descriptors of the ICE levels incorporate the
such a way that they could be mapped to the graduate assessment as learning approach discussed by McMillan.
attributes. Table 1 provides the rubric categories common The descriptors of the ICE levels characterize the stages
to the sequence and the number of criteria contributing to of learning for each criterion, allowing students to use the
the categories in the final report rubrics for each course. rubrics as a guide for their learning. Particular care was
taken in establishing the descriptors. As suggested by
Table 1: Number of criteria in major rubric categories. Rasul, an attempt was made to avoid overly prescriptive
Rubric Category 200/293 381 480 descriptions [8]. The descriptors characterize the qualities
of the given level for each criterion without fully detailing
Communication 9 7 7 how to achieve that level. The structure of the language is
Background also designed to deter the students from using the rubrics
Research and as a checklist, although achieving this in reality is an
4 5 5
Problem extremely challenging task. Minimal grade values were
Definition included in the rubrics to ensure that students focus on the
Creative Thinking 1 2 2 descriptors. The grade ranges for the levels are provided
Concept Selection 1 2 2 in the column headings and a section is provided for
Concept writing the overall report grades. To provide further
1 1 1 guidance to students, each of the levels (BICE) are
Development
Final Design 2 2 4 divided into a lower, central, and upper range for
Recommendations feedback to students. Graders highlight the appropriate
2 2 2 section (e.g. I+ or E-) on a spreadsheet, but do not assign
and Future Work
Design Process a grade value; that is calculated by a formula applied
and Integrated equally to all grade spreadsheets. The additional division
3 6 6 of the levels helps to indicate, for instance, if students are
Professional
Practice at the Connections level, but are approaching the
Project Extensions level.
0 1 1 Although EDPS II is comprised of two courses, each
Management
report assessed using the rubrics contributes to both
APSC200 and APSC293. The engineering design and
Although the rubrics in the sequence share common
communication elements were combined into single
categories of criteria and levels, the design of the
rubrics for each of the reports, giving two grades to be
APSC200/293 rubrics is distinctly different from the
calculated from each rubric.
APSC381 and APSC480 rubrics. Descriptors of each of
the four levels for each criterion are included in the
APSC200/293 rubrics. As well, weightings are assigned 4.2 APSC381 and APSC480
to each criterion. The APSC381 and APSC480 rubrics
provide less granularity, with descriptors provided for the APSC381 and APSC480 cover more advanced
main criteria but not for the levels within each criterion. concepts and techniques for engineering design and
This allows for a wide variety of project topics while still professional practice, and go further into the process for
describing competency in each assessment criterion. refinement and optimization of solutions. At a minimum,
Weightings are assigned to groupings of criteria instead of students enrolled in these courses have successfully
to each criterion. The transition in descriptors from the completed EDPS I and II. Upon enrolling in APSC381
APSC200/293 rubrics to the APSC381 and APSC480 and APSC480 students have been previously exposed to
rubrics require students to think critically when assessing the application of the ICE assessment approach in the
their learning levels. By assigning weightings to APSC200/293 rubric set and a similar approach in APSC
groupings of criteria instead of individual criterion, 100, and thus are expected to have a conceptual
understanding of the Below Minimum Standards, Ideas,

CEEA16; Paper 125

Dalhousie University; June 19 – 22, 2016 – 4 of 6 –
Proc. 2016 Canadian Engineering Education Association (CEEA16) Conf.

Connections, and Extensions levels. Guidance is provided 5. CONCLUSIONS AND FUTURE WORK
to students in the form of descriptions of each criterion
instead of descriptors of each level for each criterion. At this time, the APSC200/293 rubric set has been
Although the criteria are grouped together for weightings implemented for two academic years, while the APSC381
to calculate the overall report grades, feedback is and APSC480 rubric sets have been implemented for one
provided to students for each criterion using the four academic year. A full assessment has not yet been
levels. The evaluation of each criterion using the levels completed to determine the success of the rubric
also allows for mapping to the learning outcomes and sequence. The consideration of learning outcomes when
graduate attributes. developing the criteria, descriptions, and level descriptors
There are slight differences in the content of the ensured that each set of rubrics is aligned with their
APSC381 and APSC480 rubric sets to accommodate respective course learning outcomes. The breakdown of
differences in learning outcomes and expectations in the criteria in each rubric and collection of data for each
projects. The differences in the final report rubrics are in criterion ensure that the rubrics can be used for mapping
the descriptions of the engineering science criterion, to the graduate attributes. However, further investigation
which is within the overall design process grouping of is needed to determine the use of the rubric sequence for
criteria, and the descriptions of criteria in the concept graduate attribute assessment. The structure of the rubrics
development and final solution criteria groupings. in the sequence incorporates a degree of holistic grading,
especially for APSC381 and APSC480.
4.3 Grade Calculation Methods An exploration of how TAs and instructors use the
rubrics, and particularly the holistic capability, would be
The APSC200/293 rubric set uses a different method beneficial to further improve on the design of the
for calculating the overall report grades than the sequence. The time efficiency and reliability of the two
APSC381 and APSC480 rubric sets. To calculate the grade calculation methods could be compared to
APSC200/293 overall grades, a percentage is assigned to determine the optimal method. Assessment of the impact
the each of the lower, central, and upper ranges of each of these rubrics based on outcome quality and student
level. The overall grades result from the weighted sum of learning may lead to further evolution of this assessment
the percentages for each criterion. methodology.
Percentages were not assigned to the four levels in the
APSC381 and APSC480 rubrics. Instead, the APSC381 Acknowledgements
and APSC480 rubrics use the upper and lower bounds of
the grade ranges to calculate an upper and lower bound The authors gratefully acknowledge supportive
for the overall grade. Similar to the APSC200/293 funding from the NSERC Chair in Design Engineering.
calculation method, the upper and lower grade bounds are We would like to acknowledge and thank Gillian
calculated as a weighted sum of the criteria. However, Woodruff, Georgina Rainsford, and the past APSC381
weightings are applied to groupings of criteria as opposed and APSC480 teaching assistants and students for their
to single criteria, so a percentage is required for each valuable contributions and insights during the
criteria grouping. To simplify the calculations of the development process of the rubrics.
upper and lower grade bounds, the upper and lower grade
bounds for the level of the criteria in a grouping were References
averaged to give upper and lower percentages for the
grouping to be used in a weighted sum. The APSC381 [1] Brian M. Frank, David S. Strong, Rick Sellens, and Lynann
and APSC480 rubric structure does not require an equally Clapham, “Progress with the professional spine: A four-year
weighted average for calculating percentages for the Engineering Design and Practice Sequence,” Australasian
groupings, so depending on variation in projects, a Journal of Engineering Education 19(1), pp. 63, 2013.
weighted average can be applied. Instead of giving a
single overall grade, an upper and lower grade bound and [2] Clive L. Dym, Alice M. Agogino, Ozgur, Eris, Daniel D.
Frey, and Larry J. Leifer, “Engineering design thinking,
average grade are calculated. Holistic assessment is then
teaching, and learning,” Journal of Engineering Education,
incorporated to determine where in the overall grade pp. 103-210, 2005.
range the report actually sits. The average of the upper
and lower bounds is the starting point for the holistic [3] David S. Strong, “An approach for improving design and
approach. In practice, the final grades for all projects are innovation skills in engineering education: The
discussed and calibrated by the instructions team, with at multidisciplinary design stream,” International Journal of
least 2 readers per report. Engineering Education 28(2), pp. 339, 2012.

CEEA16; Paper 125

Dalhousie University; June 19 – 22, 2016 – 5 of 6 –
Proc. 2016 Canadian Engineering Education Association (CEEA16) Conf.

[4] David S. Strong and S. Fostaty Young. “Effectively [8] M. S. Rasul, “Employability skills assessment tool
assessing professional engineering skills,” in Proc. The development,” International Education Studies 5(5), 2012.
Fourth CDEN Design Conf., (Winnipeg, MN; 2007), 2007.
[9] P. G. Dominick, J. T. Demel, W. . Lawbaugh, R. J. Freuler,
[5] Deric Oehlers and David Walker. “Assessment of deep G. L. Kinzel, and E. Fromm, Tools and Tactics of Design.
learning ability for problem solvers,” International Journal Wiley, New York, 2001.
of Engineering Education 22(6), pp. 1261, 2006.
[10] S. Fostaty Young and R. J. Wilson, Assessment &
[6] Iman Moazzen, Terry Hansen, Mariel Miller, Peter Wild, Learning: The Ice Approach. Portage & Main Press,
Allyson Hadwin, and LillAnne Jackson, “Literature review Winnipeg, 2000.
on engineering design assessment tools,” in Proc. CEEA
Canadian Engineering Education Conf., CEEA13 [11] Thomas F.C. Woodhall and David S. Strong,
(Montreal, QC; 17-20 June 2013), 2013. “Implementing a Rubric-based Assessment Scheme into the
Multidisciplinary Design Stream at Queen’s University,” in
[7] J. H. McMillan, Classroom Assessment, Pearson Education, Proc. CEEA Canadian Engineering Education Conf.,
United States of America, 2007. CEEA08 (Halifax, NS; 27-29 July 2008), 2008.

APPENDIX
Table 2: Problem definition criteria in the APSC200/293 final report rubric.

Rubric Element B I C E

Missing or trivial
Significant stakeholders All relevant stakeholders All relevant stakeholders are
identification and
Stakeholder Needs are identified with are identified and their identified with clear descriptions,
consideration of
nominal description. needs are well defined. relevance, and impact of their needs.
stakeholder needs.
Missing or incomplete Clear and complete
Cursory description Clear, insightful, and focused
statement that leaves statement that scopes the
Problem Statement outlines key ideas, but statement elegantly distills the
ambiguity in the problem and does not
leaves some uncertainty. problem in concise language.
problem. predetermine the solution.
Simplistic criteria and Specific and measurable Comprehensive, detailed and precise
Missing, trivial, vague
Design Criteria and constraints with some criteria and constraints criteria and constraints logically
or unrelated criteria or
Specifications relevance to stakeholder developed from research developed from analysis of research
constraints.
needs. and stakeholder needs. and stakeholder needs.

Table 3: Problem definition criteria in the APSC381 final report rubric.

Report Element Subsection Level

Stakeholder Needs
- Determination, assessment, relevance of stakeholders and their needs B I C E
Analysis
- Clarity and concision
Problem Statement B I C E
- Inclusion of key “problem’ details, captured in a brief summary
Constraints and - Project scope is well defined with clear constraints and assumptions
B I C E
Assumptions - Logic, reasoning, and justification suitable for project topic
- Clear description of functional, aesthetic, and other requirements
Design Criteria and
- Comprehensiveness, detail, and precision appropriate to project topic B I C E
Specifications
- Reader clearly understands outcomes required for successful solution

Table 4: Problem definition criteria in the APSC480 final report rubric.

Report Element Subsection Level

Stakeholder
- Determination and assessment of stakeholder needs B I C E
Needs Analysis
Problem - Clarity and concision of key project details
B I C E
Statement - Establishment of a clear scope and objectives, concisely stated
Constraints and - Logic, reasoning, and justification
B I C E
Assumptions - Assessment of validity of assumptions
- Clear description of functional, aesthetic, and other requirements
Design Criteria - Value proposition in terms of monetary or non-monetary value to the client (e.g. market, productivity, cost
and reduction, customer recognition, etc.) B I C E
Specifications - Comprehensiveness, detail, and precision appropriate to project topic
- Reader clearly understands outcomes required for successful solution

CEEA16; Paper 125

Dalhousie University; June 19 – 22, 2016 – 6 of 6 –