9 2

UT Austin, EE 382M-11 2/15/2017
Assertion-Based Verification
Harry Foster
Chief Scientist Verification
[email protected] | www.verificationacademy.com
Outline
How Verification is Done Today
What Makes Verification Difficult
Observability and Controllability Challenge
Industry Case Studies
Conclusions
2 H Foster, EE 382M-11, Verification of Digital Systems, Spring 2017 Mentor Graphics Corporation, all rights reserved.
Mentor Graphics Corporation 1

UT Austin, EE 382M-11 2/15/2017
HOW VERIFICATION IS DONE TODAY
What is Verification?
Verification is a process of ensuring that a

design implementation meets its specification.
4 H Foster, EE 382M, Verification of Digital Systems, Spring 2017 Mentor Graphics Corporation, all rights reserved.

UT Austin, EE 382M-11 2/15/2017
Simulation-Based Techniques
Fundamental verification technique in use today

Generally scales well
Testing all possible states is generally incomplete
Simulation Testbench
Measure Coverage
Generate Design Check

Stimulus Model Results
Assertions can be used to check results and measure coverage

Simulation Traversal Through the State Space
// SystemVerilog Assertion
property p_comp;
initial @(posedge clk)
state E |-> (A==B);
endproperty
assert property (p_comp);

UT Austin, EE 382M-11 2/15/2017
Time Explosion Problem
How long would it take to exhaustively simulate this

example?
1000000011101011011011110111
A [31:0]
property p_comp;
@(posedge clk)
E
E |-> (A==B);
endproperty
101010001000110101110100101
B [31:0]
264 vectors X 1 vector every micro-second = 584,941 years
An extremely fast simulator by todays standards!
Simulation and the Time Explosion Problem
264 vectors X 1 vector every micro-second = 584,941 years

UT Austin, EE 382M-11 2/15/2017
Formal-Based Techniques
Does not require a testbench or input stimulus!

Automatically uses algorithms to verify the functionality
Verification can be complete
Complements simulation-based techniques
Formal Pass Yes

Done
Tool ?
No
Design
Assertions
Model
Conceptual Formal Tool
Tx x
a
Tx(a,x,y) // next state
y

UT Austin, EE 382M-11 2/15/2017
How is formal different than simulation?
initial
states
property p_comp;
@(posedge clk)
E |-> (A==B);
endproperty
Very fast!
State Space Explosion
There are more states in todays design than

there are atoms in the universe!
How many states exist in a

typical design today?

UT Austin, EE 382M-11 2/15/2017
WHAT MAKES VERIFICATION

DIFFICULT
INDUSTRY DRIVERS
Rising Design Complexity

UT Austin, EE 382M-11 2/15/2017
Rise in the Average Number of IP Blocks

90
Avergage Number of IP Blocks
Closing the Design Productivity Gap!
80
70
60
50
40
30
20
10
0
2007 2008 2009 2010 2011 2012 2013 2014 2015* 2016* 2017* 2018*
Avg. Number of 'Other' SIP Blocks Avg. Number of CPU / DSP / Controllers
Avg. Number of Embedded Memory Blocks
Source: Semico Research Corp.
Design Engineers are Being Productive

Transistors produced per electronic engineer
1,000,000,000,000,000
more than 5-orders of magnitude since 1985
100,000,000,000,000
10,000,000,000,000
1,000,000,000,000
100,000,000,000
Quantity
10,000,000,000
1,000,000,000
100,000,000
10,000,000
1,000,000
100,000
10,000
Transistors Produced Total Electronic Engineers

Source: Technology Research Group EDA Database, 1986, EDA TAM, 1989 & Gartner/Dataquest 2005 Seat Count Report,
Gary Smith EDA, 2013 Seat Count Analysis , VLSI Research, 2013 - Transistors Produced Analysis

UT Austin, EE 382M-11 2/15/2017
Design Engineers are Being Productive!
1.E+03
Growth of Transistor Volume Leads to
1.E+02
Sustained ~ 30% per Year Cost Reduction
Revenue/Transistor ($)
1.E+01
1.E+00
1.E-01
1.E-02
1.E-03
1.E-04
1.E-05
1.E-06
1.E-07 Semiconductor Learning Curve

1.E-08 1954 2012 Adjusted for Inflation
1.E-09
1.E-10
1.0E+04 1.0E+06 1.0E+08 1.0E+10 1.0E+12 1.0E+14 1.0E+16 1.0E+18 1.0E+20
Source: VLSI Research, SIA, Federal Reserve

Note: Revenue adjusted for Inflation 1954-2012
Cumulative Transistors Shipped
We are Keeping Up with Design Complexity!
Thanks to Automation and Reuse!

UT Austin, EE 382M-11 2/15/2017
Another View of Moores Law

EDA Cost per Transistor and Total IC Revenue per
Transistor Both Decrease About 30% per Year
1.00E-04 1.00E-04
1.00E-05 1.00E-05
IC Revenue/Transistor ($)
EDA Cost/Transistor ($)
1.00E-06 1.00E-06
1.00E-07 1.00E-07
1.00E-08 1.00E-08
1.00E-09 1.00E-09
1.00E+13 1.00E+14 1.00E+15 1.00E+16 1.00E+17 1.00E+18 1.00E+19 1.00E+20
Cumulative Transistors Shipped

EDA Cost/transistor
Note: EDA Cost Consists of EDA License and Maintenance revenue adjusted for Inflation IC Revenue/transistor
Source: SIA, VLSI Research, Federal Reserve
Demand for Design Engineers Grows Slowly

12
CAGR Designers 3.6%
ASIC/IC Mean Peak Number of Engineers
10
10.48
10.05
8
8.53
8.10
7.80
Design Engineers
6
0
2007 2010 2012 2014 2016

UT Austin, EE 382M-11 2/15/2017
But what about Verification Productivity?

12
CAGR Designers 3.6%
CAGR Verifiers 10.4% 11.6
ASIC/IC Mean Peak Number of Engineers
10 11.0
10.48
10.05
8
8.4
8.53
8.10
7.80 7.6
Design Engineers
6
Verification Engineers
4 4.8
0
2007 2010 2012 2014 2016
INDUSTRY DRIVERS
Rising Verification Complexity

UT Austin, EE 382M-11 2/15/2017
The Emergence of New Layers of Verification
Software
Security Domains
Verification Layers
Power Domains
Clock Domains
Functional
What Makes Verification Difficult?
Channel
TX Data Link Layer PHY
Encoder Decoder
Compressed RX
Audio
Single, sequential data streams Multiple, concurrent data streams

Floating point unit Cross bar
Graphics shading unit Bus traffic controller
DSP convolution unit DMA controller
MPEG decode Standard I/F (e.g., PCIe)
... ...
Sequential data streams Concurrent data streams

1x number of bugs 5x number of bugs
-Ted Scardamalia, internal IBM study


UT Austin, EE 382M-11 2/15/2017
Directed-Test Approach
Imagine verifying a car using a directed-test approach

Requirement: Fuse will not blow under any normal operation
Scenario 1: accelerate to 37 mph, pop in the new
Lady GaGa CD, and turn on the windshield wipers
A FEW WEEKS LATER

UT Austin, EE 382M-11 2/15/2017
Directed-Test Approach
Imagine verifying a car using a directed-test approach

Requirement: Fuse will not blow under any normal operation
Scenario 714: accelerate to 48 mph, roll down the window,
and turn on the left-turn signal
The Concurrency Challenge
A purely directed-test methodology does not scale

Imagine writing a directed test for this scenario!
Truly heroic effortbut not practical

UT Austin, EE 382M-11 2/15/2017
Finding Corner Case Bugs Due to Concurrency
Directed-test-based simulation finds

the bugs you can think of
Constrained-random simulation finds

the bugs you never anticipated!
Concurrency is Complicated to Verify
Packet-Based Design
Transaction
Tx Layer Packet
From
Fabric Reformater
To
Arbiter PHY
Retry Buffer
Data Link
Layer Packet
Reformater
Rx
From Rx
Channel

UT Austin, EE 382M-11 2/15/2017
Adoption Trends in Verification Techniques
Code coverage
Assertions 2007
2012
2014
Functional
coverage
Constrained-
Random Simulation
0% 10% 20% 30% 40% 50% 60% 70% 80%

Design Projects
Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study
OBSERVABILITY & CONTROLLABILITY

UT Austin, EE 382M-11 2/15/2017
Fundamental Challenge of Verification
DUT
A
A
0010100101010001110101001110101010100000000011101011011011110111
1. Activate
A 3. Detect
Stimulus A 2. Propagate
Checkers
A = Assertions
Observability vs. Controllability
Test didnt set up the condition to propagate the bug
bug
A
1 0
1
0
Assertions improve observability and

reduce the need to propagate bugs

UT Austin, EE 382M-11 2/15/2017
Poor Observability Misses Bugs
Code coverage measures controllability

100% code coverage does not mean all bugs
are detected [S. Devadas, A. Ghosh, and K. Keutzer. DAC 1996]
DAC paper study found cases where:
Code Coverage % of covered lines

Achieved observable
90% Covered Only 54% Observable
100% Covered Only 70% Observable
Assertions Improve Observability
Testbench
= Bugs missed due to =

poor observability
Reduce debugging up to 50% [CAV 2000, IBM FoCs paper]

Bugs detected closer to their source due to improved observability

UT Austin, EE 382M-11 2/15/2017
2014 Where Verification Engineers Spend Their Time
Test Planning
37%
Testbench Development
3%
Creating Test and Running Simulation

24% 14%
Debug
Other
22%
Designers Spend a Lot of Time in Verification & Debug
60%
Mean time design engineer spends in
Doing Design
Doing Verification
design vs. verification
55% 53%
51%
54% 53%
50%
46% 49% 47%
45%
47%
40%
2007 2010 2012 2014

UT Austin, EE 382M-11 2/15/2017
ASSERTION-BASED VERIFICATION
How can one check a large routine in the
sense of making sure that its right? In
order that the man who checks may not
have too difficult a task, the programmer
should make a number of definite assertions
which can be checked individually, and from
which the correctness of the whole program
easily flows.
Alan Turing, 1949

UT Austin, EE 382M-11 2/15/2017
Property
Property Testbench
a statement of design intent test
used to specify behavior
env
DUT
Assertion
Property Testbench
env
Assertion
A verification directive
Trace from
simulation
DUT

UT Austin, EE 382M-11 2/15/2017
High-Level Assertion
Property Testbench
env
Assertion
High-level
Architectural focused
Can be part of testbench
Trace from
simulation
DUT
Low-Level Assertion
Property
a statement of design intent
Assertion RTL
A
High-level
Architectural focused
Can be part of testbench A
Low-level
Implementation focused // Assert that the FIFO controller
Embedded in or bind to the RTL // cannot overflow nor underflow

UT Austin, EE 382M-11 2/15/2017
How Assertions Are Used Today
State Search Testbench
RTL
improved
bug rate
FPGA or
Formal
Emulation
Props
Assertions
passing tests
Formal Verification Simulation O/S Trials
[Foster, Larsen, Turpin - DVCon 2006]
Who should create the assertions?
Verification Engineer Design Engineer
High-Level Assertions Low-Level Assertions

Requirement focused Implementation focused
Black-box assertions White-box assertions
Accounted for in testplan Not accounted for in testplan
Compliance traceability Improve observability
Create reusable ABV IP Reduce debugging time

UT Austin, EE 382M-11 2/15/2017
Who should create high-level assertions?

Who should create low-level assertions?


UT Austin, EE 382M-11 2/15/2017
Specifying Design Intent
Assertions allow us to specify design intent

in a way that lends itself to automation
clk
grant0
reset_n
Arbiter grant1
req0
req1
// Assert that the grants for our simple arbiter are mutually exclusive
Identifying the Error Condition
For our arbiter example, we can write a Boolean

expression for the error condition, as follows:
clk
grant0
reset_n
Arbiter grant1
req0
req1
(grant0 & grant1) // error condition

UT Austin, EE 382M-11 2/15/2017
Checking the Error Condition before Assertions
Doesnt lend itself to automation.
module arbiter (clk, rst_n, req0, req1, grant0, grant1);

...
always @(posedge clk or negedge rst_n) begin Error
Condition
if (rst_n != 1b0) Boolean
if (grant0 & grant1) Expression
$display (ERROR: Grants not mutex);

...
endmodule
Assertion Language Adoption

80%
2007 World
70% 2012 World
60% 2014 World

Design Projects
50%
40%
30%
20%
10%
0%
Accellera Open SystemVerilog PSL Other
Verification Library Assertions (SVA)
(OVL)
Assertion Languages and Libraries * Multiple answers possible

UT Austin, EE 382M-11 2/15/2017
IEEE 1800 SystemVerilog Mutex Example
grant0 and grant1 must be mutually exclusive
clk
grant0
grant1
error
assert property ( @(posedge clk) disable iff (~rst_n) !(grant0 & grant1));
IEEE 1850 PSL Fair Arbiter Example
clk
grant0
grant1
error
assert always (!(grant1 & grant2) abort ~rst_n) @(posedge clk);

UT Austin, EE 382M-11 2/15/2017
Accellera OVL Memory Address Example
clk
grant0
grant1
error
ovl_never a_mutex (clk, rst_n, (grant1 & grant2));
INDUSTRY CASE STUDIES

UT Austin, EE 382M-11 2/15/2017
Published Data on Assertions Use
Percentage bugs found by various techniques 17% of bugs found by assertions on Cyrix M3(p1) project
Assertion Monitors 34% [Krolnik '98]
Cache Coherency Checkers 9%
Register File Trace Compare 8%
Memory State Compare 7%
End-of-Run State Compare 6% 50% of bugs found by assertions on Cyrix M3(p2) project
PC Trace Compare 4%
Self-Checking Test 11% [Krolnik 98]
Simulation Output Inspection 7%
Simulation Hang 6%
Other 8%
85% of bugs found using over 4000 assertions on an HP
Kantrowitz and Noack [DAC 1996]
server chipset project
[Foster and Coelho HDLCon 2001]
Assertion Monitors 25%

Register Miscompare 22% Thousands of assertions in Intel Pentium project
Simulation "No Progress 15%
PC Miscompare 14% [Bentley 2001]
Memory State Miscompare 8%
Manual Inspection 6%
Self-Checking Test 5%
Cache Coherency Check 3% 10,000 OVL assertion in Cisco project
SAVES Check 2%
[Sean Smith 2002]
Taylor et al. [DAC 1998]
DAC 2008 Sun paper with lots of metrics

Assertion-Based Verification of a 32 thread SPARC CMT Processor
[Turumella, Sharma, DAC 2008]
Category Unique Instantiated

Low-Level 3912 132773
Interface 5004 44756
High-Level 1930 18618
Bugs Found by Type of Assertion
Low-level
Interface
High-level

UT Austin, EE 382M-11 2/15/2017
Significant reduction in debugging time

Assertion-Based Verification of a 32 thread SPARC CMT Processor
[Turumella, Sharma, DAC 2008]
Category Unique Instantiated

Low-Level 3912 132773
Interface 5004 44756
High-Level 1930 18618
Average Debug Time

16
14
12 >50%
10 85% Formal
Hours
8 Sim + Assert
6 Sim + None
4
2
0
Formal Sim + Assert Sim + None
SUMMARY

UT Austin, EE 382M-11 2/15/2017
The process of creating assertions forces the

engineer to think. . . and in this incredible
world of automation, there is no substitute for
thinking.
Harry Foster
Chief Scientist Verification
[email protected] | www.verificationacademy.com

9 2

Uploaded by

Copyright:

Available Formats

9 2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

9 2

Uploaded by

Copyright:

Available Formats

UT Austin, EE 382M-11 2/15/2017

How Verification is Done Today

What Makes Verification Difficult

Observability and Controllability Challenge

Industry Case Studies

Mentor Graphics Corporation 1

HOW VERIFICATION IS DONE TODAY

Verification is a process of ensuring that a

Mentor Graphics Corporation 2

Fundamental verification technique in use today

Generate Design Check

Assertions can be used to check results and measure coverage

Simulation Traversal Through the State Space

Mentor Graphics Corporation 3

Time Explosion Problem

How long would it take to exhaustively simulate this

264 vectors X 1 vector every micro-second = 584,941 years

An extremely fast simulator by todays standards!

Simulation and the Time Explosion Problem

264 vectors X 1 vector every micro-second = 584,941 years

Mentor Graphics Corporation 4

Does not require a testbench or input stimulus!

Formal Pass Yes

Conceptual Formal Tool

Mentor Graphics Corporation 5

How is formal different than simulation?

State Space Explosion

There are more states in todays design than

How many states exist in a

Mentor Graphics Corporation 6

WHAT MAKES VERIFICATION

Mentor Graphics Corporation 7

Rise in the Average Number of IP Blocks

Avg. Number of Embedded Memory Blocks

Source: Semico Research Corp.

Design Engineers are Being Productive

Transistors Produced Total Electronic Engineers

Mentor Graphics Corporation 8

Design Engineers are Being Productive!

1.E-07 Semiconductor Learning Curve

Source: VLSI Research, SIA, Federal Reserve

We are Keeping Up with Design Complexity!

Thanks to Automation and Reuse!

Mentor Graphics Corporation 9

Another View of Moores Law

Cumulative Transistors Shipped

Demand for Design Engineers Grows Slowly

Mentor Graphics Corporation 10

But what about Verification Productivity?

Mentor Graphics Corporation 11

The Emergence of New Layers of Verification

What Makes Verification Difficult?

Single, sequential data streams Multiple, concurrent data streams

Sequential data streams Concurrent data streams

-Ted Scardamalia, internal IBM study

Mentor Graphics Corporation 12

Imagine verifying a car using a directed-test approach

A FEW WEEKS LATER

Mentor Graphics Corporation 13