Overview of CFD Verification and Validation 1

Overview of CFD Verification and Validation

Last Updated: Thursday, 17-Jul-2008 08:53:41 EDT

Responsible NASA Official/Curator: John W. Slater

Introduction

This page presents an overview of the process of the verification and validation of computational
fluid dynamics (CFD) simulations. The overall objective is to demonstrate the accuracy of CFD
codes so that they may be used with confidence for aerodynamic simulation and that the results be
considered credible for decision making in design.

One should first understand the distinctions between a code, simulation, and model. The formal
definitions of these terms are defined on the page entitled Glossary of Verification and Validation
Terms. Essentially, one implements a model into a computer code and then uses the code to perform
a CFD simulation which yield values used in the engineering analysis. Verification and validation
examines the errors in the code and simulation results.

Credibility is obtained by demonstrating acceptable levels of uncertainty and error. A discussion of

the uncertainties and errors in CFD simulations is provided on the page entitled Uncertainty and
Error in CFD Simulations. The levels of uncertainties and errors are determined through verification
assessment and validation assessment.

Verification assessment determines if the programming and computational implementation of the

conceptual model is correct. It examines the mathematics in the models through comparison to
exact analytical results. Verification assessment examines for computer programming errors.

Validation assessment determines if the computational simulation agrees with physical reality. It
examines the science in the models through comparison to experimental results.

There is professional disagreement on exact procedures for verification and validation of CFD
simulations. CFD is maturing, but still an emerging technology. CFD is a complex technology
involving strongly coupled non-linear partial differential equations which attempt to
computationally model theoretical and experimental models in a discrete domain of complex
geometric shape. A detailed assessment of errors and uncertainties has to concern itself with the
three roots of CFD: theory, experiment, and computation. Further, the application of CFD is rapidly
expanding with the growth in computational resources. In this work, we primarily follow the
verification and validation guidelines established by the AIAA [AIAA-G-077-1998]. Note that this
is a guide - no standards yet exist for CFD simulation verification and validation. Other ideas from
other researchers in this discipline will also be included. Their papers are referenced in the
bibliography. Notable among them is the book on verification and validation published by Roache.
 Overview of CFD Verification and Validation 2

Verification and validation are on-going activities due to the complex nature of the CFD codes and
expanding range of possible applications. Some basic verfication should be done prior to release of
a code and basic validation studies should be performed on classes of flow features prior to use of
the code for similar flows. However, as the code continues to develop, verification and validation
should continue.

Use of CFD Results

The level of accuracy required from a CFD analysis depends on the desired use of the results. A
conceptual design effort may be content with general shock structure information, whereas a
detailed design may require accurate determination of the pressure recovery. Each quantity to be
determined generally has its own accuracy requirement. Levels of credibility may vary according to
the information required.

The application of CFD for design and analysis may be catagorized into three levels according to
increased levels of required accuracy: 1) provide qualitative information, 2) provide incremental
quantities, and 3) provide absolute quantities. This discussion follows the ideas of Benek et al.

Provide qualitative information. CFD provides details on the entire flow field not possible with
experimental methods. This is useful in understanding on a qualitative level the behavior of the flow
field. Accuracy requirements are low.

Provide incremental quantities. Corrections to experimental observations can be provided by

CFD at a higher accuracy than existing with the CFD method. This is due to cancellation of part of
the error when taking the differences. For example, if there is a design change from a baseline, for
which the quanity is known, Pbaseline, the quantity P for the design change can be expressed as:

P = Pbaseline + dP

where dP is the increment in P corresponding to the design change. If two CFD simulations are
performed, the first with the baseline geometry and the second with the modified geometry, then the
increment in P due to the modified geometry can be estimated as,

dP = ( P + E )2 - ( P + E )1 = dPactual + dE.

The E is the error associated with the quantity P obtained from the CFD simulation. As can be seen,
the error to the increment is dE, which cancels out some of the error.

Provide absolute quantities. This level involves determining absolute values of the quantity P and
requires the highest level of accuracy. The accuracy required is usually stated as part of the design
process. The accuracy observed from the CFD simulation varies according to the character of the
quantity, and so, it is not possible to state an accuracy or error band that applies to all quantities
obtained from the CFD simulation. The verification methods discussed below regarding a grid
convergence study will provide the error band for the calculations.

Flow Characteristics

In applying CFD for flows typical of aerospace systems, we must first understand the characteristics
of the flow. We must understand the reality upon which we will validate the CFD code and
processes.
 Overview of CFD Verification and Validation 3

The flow is characterized primarily by the Mach number. We are interested in analyzing flows
spanning the Mach number range from Mach 0 (static conditions) to 25 (access to space).

The flow is characterized by high Reynolds numbers which result in regions of laminar flow
transitioning to turbulent flow. Flows along the body and inlet surfaces create boundary layers.
Adverse pressure gradient may be present for internal flows. At transonic, supersonic, and
hypersonic speeds, shock waves are present. Under these conditions, the boundary layer may
separate.

At hypersonic Mach numbers, real gas effects may become important. This requires use of
chemistry models for calorically and thermally perfect gases, equilibrium air, and chemically
reaction of gas mixtures.

Often the geometry of the system is complex, which has to be physically modeled.

Unsteady flow may become important.

Physical Models

There are several physical models that are commonly used within CFD codes:

Spatial Dimension. The geometry of the inlet may be modelled in some cases using two-
dimensional or axisymmetric space rather than full three-dimensional.

Temporal Dimension. One may assume steady-state flow or attempt to capture the time variations.

Navier-Stokes Equations. The Navier-Stokes equations govern the continuum flow. Viscous and
heat conduction effects are modelled. If these are removed, then inviscid flow can be used.

Turbulence Models. Various algebraic, one-equation, and two-equation turbulence models exist
with various parameters and freestream boundary conditions. The option of wall functions exists.

Thermodynamic and Transport Properties. Constants and relations for thermodynamic and
transport properties are generally constants, algebraic equations, or curve fits.

Air Chemistry Models. Inlet flows typically involve calorically perfect air adequately described by
the perfect gas equation of state. At higher temperatures (greater than 700K), modeling of thermally
perfect air, equilibrium air, and chemically reacting air (temperatures greater than 2000 K) may be
needed.

Flow Boundary Conditions. These include subsonic and supersonic freestream inflow and
outflow. Also inflow and outflow of plenum chambers.

Bleed / Blowing. These can be treated as boundary conditions as a mass flow or porous boundary.
Another option is to grid the slots and holes of the actual geometry.

Flow Control Devices. Flow control is important and several new technologies have developed.
Vortex generators are the primary flow control devices used in inlets. These can be modeled or an
approximation of the geometry can be gridded.
 Overview of CFD Verification and Validation 4

CFD Analysis Process

The general process for performing a CFD analysis is outlined below so as to provide a reference
for understanding the various aspects of a CFD simulation. The process includes:

1. Forumulate the Flow Problem

2. Model the Geometry and Flow Domain
3. Establish the Boundary and Initial Conditions
4. Generate the Grid
5. Establish the Simulation Strategy
6. Establish the Input Parameters and Files
7. Perform the Simulation
8. Monitor the Simulation for Completion
9. Post-process the Simulation to get the Results
10. Make Comparisons of the Results
11. Repeat the Process to Examine Sensitivities
12. Document

In further detail, these steps include:

1. Formulate the Flow Problem

The first step of the analysis process is to formulate the flow problem by seeking answers to
the following questions:

o what is the objective of the analysis?

o what is the easiest way to obtain those objective?
o what geometry should be included?
o what are the freestream and/or operating conditions?
o what dimensionality of the spatial model is required? (1D, quasi-1D, 2D,
axisymmetric, 3D)
o what should the flow domain look like?
o what temporal modeling is appropriate? (steady or unsteady)
o what is the nature of the viscous flow? (inviscid, laminar, turbulent)
o how should the gas be modeled?
2. Model the Geometry and Flow Domain
The body about which flow is to be analyzed requires modeling. This generally involves
modeling the geometry with a CAD software package. Approximations of the geometry and
simplifications may be required to allow an analysis with reasonable effort. Concurrently,
decisions are made as to the extent of the finite flow domain in which the flow is to be
simulated. Portions of the boundary of the flow domain conicide with the surfaces of the
body geometry. Other surfaces are free boundaries over which flow enters or leaves. The
geometry and flow domain are modeled in such a manner as to provide input for the grid
generation. Thus, the modeling often takes into account the structure and topology of the
grid generation.

3. Establish the Boundary and Initial Conditions

Overview of CFD Verification and Validation 5

Since a finite flow domain is specified, physical conditions are required on the boundaries of
the flow domain. The simulation generally starts from an initial solution and uses an
iterative method to reach a final flow field solution.

4. Generate the Grid

The flow domain is discretized into a grid. The grid generation involves defining the
structure and topology and then generating a grid on that topology. Currently all cases
involve multi-block, structured grids; however, the grid blocks may be abbuting, contiguous,
non-contiguous, and overlapping. The grid should exhibit some minimal grid quality as
defined by measures of orthogonality (especially at the boundaries), relative grid spacing
(15% to 20% stretching is considered a maximum value), grid skewness, etc... Further the
maximum spacings should be consistent with the desired resolution of important features.
The resolution of boundary layers requires the grid to be clustered in the direction normal to
the surface with the spacing of the first grid point off the wall to be well within the laminar
sublayer of the boundary layer. For turbulent flows, the first point off the wall should exhibit
a y+ value of less than 1.0.

5. Establish the Simulation Strategy

The strategy for performing the simulation involves determining such things as the use of
space-marching or time-marching, the choice of turbulence or chemistry model, and the
choice of algorithms.

6. Establish the Input Parameters and Files

A CFD codes generally requires that an input data file be created listing the values of the
input parameters consisted with the desired strategy. Further the a grid file containing the
grid and boundary condition information is generally required. The files for the grid and
initial flow solution need to be generated.

7. Perform the Simulation

The simulation is performed with various possible with options for interactive or batch
processing and distributed processing.

8. Monitor the Simulation for Completion

As the simulation proceeds, the solution is monitored to determine if a "converged" solution
has been obtained, which is iterative convergence. Further discussion can be found on the
page entitled Examining Iterative Convergence.

9. Post-Process the Simulation to get the Results

Post-Processing involves extracting the desired flow properties (thrust, lift, drag, etc...) from
the computed flowfield.

10. Make Comparisons of the Results

 Overview of CFD Verification and Validation 6

The computed flow properties are then compared to results from analytic, computational, or
experimental studies to establish the validity of the computed results.

11. Repeat the Process to Examine Sensitivities

The sensitivity of the computed results should be examined to understand the possible
differences in the accuracy of results and / or performance of the computation with respect
to such things as:

o dimensionality
o flow conditions
o initial conditions
o marching strategy
o algorithms
o grid topology and density
o turbulence model
o chemistry model
o flux model
o artificial viscosity
o boundary conditions
o computer system

Further information can be found on the pages entitled Verification Assessment and
Validation Assessment.

12. Document
Documenting the findings of an analysis involves describing each of these steps in the
process.

Uncertainty and Error in CFD Simulations

This page provides a classification of uncertainties and errors that cause CFD simulation results to
differ from their true or exact values. This discussion not only applies to the CFD code, but other
computer programs used in the analysis process such as CAD packages, grid generators, and flow
visualizers.

Defining Uncertainty and Error

Uncertainty and Error are commonly used interchangeably in everyday language. Here we follow
the definitions of the AIAA Guidlines:

Uncertainty is defined as:

"A potential deficiency in any phase or activity of the modeling process that is due to the lack of
knowledge." (AIAA G-077-1998)

Error is defined as:

 Overview of CFD Verification and Validation 7

A recoqnizable deficiency in any phase or activity of modeling and simulation that is not due to lack
of knowledge. (AIAA G-077-1998)

The key phrase differentiating the definitions of uncertainty and error is lack of knowledge.

The key word in the definition of uncertainty is potential, which indicates that deficiencies may or
may not exist. Lack of knowledge has primarily to do with lack of knowledge about the physical
processes that go into building the model. Sensitivity and uncertainty analyses can be used to better
determine uncertainty. Uncertainty applies to describing deficiencies in turbulence modeling. There
is a lot about turbulence modeling that is not understood. One approach for determining the level of
uncertainty and it effect on one's analysis is to run a number of simulations with a variety of
turbulence models and see how the modeling affects the results.

The definition for error implies that the deficiency is identifiable upon examination. Errors can also
be classified as acknowledged or unacknowledged:

Acknowledged errors (examples include round-off error and discretization error) have procedures
for identifying them and possibly removing them. Otherwise they can remain in the code with their
error estimated and listed.
Unacknowledged errors (examples include computer programming errors or usage errors) have no
set procedures for finding them and may continue within the code or simulation.

One can differentiate between local and global errors. Local errors refer to errors at a grid point or
cell, whereas global errors refer to errors over the entire flow domain. We are interested here in the
global error of the solution that accounts for the local error at each grid point but is more than just
the sum of the local errors. Local errors are transported, advected, and diffused throughout the grid.

The definition of error presented here is different than that an experimentalist may use, which is
"the difference between the measured value and the exact value". Experimentalist usually define
uncertainty as "the estimate of error". These definitions are inadequate for computational
simulations because the exact value is typically not known. Further these definitions link error with
uncertainty. The defintions provided in the above paragraphs are more definite because they
differentiate error and uncertainty according to what is known.

Classification of Errors

Here we provide a classification or taxonomy of error.

Acknowledged Error

1. Physical approximation error

o Physical modeling error
o Geometry modeling error
2. Computer round-off error
3. Iterative convergence error
4. Discretization error
o Spatial discretization error
o Temporal discretization error

Unacknowledged Error

1. Computer programming error

 Overview of CFD Verification and Validation 8

2. Usage error

Each of these types of errors are discussed below.

Physical Approximation Error

Physical modeling errors are those due to uncertainty in the formulation of the model and deliberate
simplifications of the model. These errors deal with the continuum model only. Converting the
model to discrete form for the code is discussed as part of discretization errors. Errors in the
modeling of the fluids or solids problem are concerned with the choice of the governing equations
which are solved and models for the fluid or solid properties. Further, the issue of providing a well-
posed problem can contribute to modeling errors. Often modeling is required for turbulence
quantities, transistion, and boundary conditions (bleed, time-varying flow, surface roughness).
Mehta lists sources of uncertainty in physical models as 1) the phenomenon is not thoroughly
understood; 2) parameters used in the model are known but with some degree of uncertainty; 3)
appropriate models are simplified, thus introducing uncertainty; and 4) an experimental
confirmation of the models is not possible or is incomplete. Even when a physical process is known
to a high level of accuracy, a simplified model may be used within the CFD code for the
convenience of a more efficient computation. Physical modeling errors are examined by performing
validation studies that focus on certain models (i.e. inviscid flow, turbulent boundary layers, real-
gas flows, etc...).

Computer Round-Off Error

Computer round-off errors develop with the representation of floating point numbers on the
computer and the accuracy at which numbers are stored. With advanced computer resources,
numbers are typically stored with 16, 32, or 64 bits. Round-off errors are not considered significant
when compared with other errors. If computer round-off errors are suspected of being significant,
one test is to run the code at a higher precision or on a computer known to store floating point
numbers at a higher precision. One can attempt to iterate a coarse grid solution to a residual of
machine zero; however, this may not be possible for more complex algorithms.

Iterative Convergence Error

The iterative convergence error exists because the iterative methods used in the simulation must
have a stopping point eventually. The error scales to the variation in the solution at the completion
of the simulations.

Discretization Errors

Discretization errors are those errors that occur from the representation of the governing flow
equations and other physical models as algebraic expressions in a discrete domain of space (finite-
difference, finite-volume, finite-element) and time. The discrete spatial domain is known as the grid
or mesh. The temporal discreteness is manifested through the time step taken. Discretization error is
also known as numerical error. A consistent numerical method will approach the continuum
representation of the equations and zero discretization error as the number of grid points increases
and the size of the grid spacing tends to zero. As the mesh is refined, the solution should become
less sensitive to the grid spacing and approach the continuum solution. This is grid convergence.
Such thinking also applies to the time step. The grid convergence study is a useful procedure for
determining the level of discretization error existing in a CFD solution. "Ordered" discretization
errors are those dependent on the grid size and vanish as the grid size approaches zero. These are
 Overview of CFD Verification and Validation 9

the errors that are addressed by a grid convergence study. Further details can be found on the pages
entitled Examining Spatial (Grid) Convergence and Examining Temporal Convergence.

The discretization error is of most concern to a CFD code user during an application. Discretization
errors are of major concern because they are dependent on the quality of the grid; however, it is
often difficult to precisely indicate the relationship between a quality grid and an accurate solution
prior to beginning the simulation. The level of discretization error is dependent on grid quality. The
grid should be generated with consideration of such things as resolution, density, aspect ratio,
stretching, orthogonality, grid singularities, and zonal boundary interfaces.

The level of discretization error is dependent on the features of the flow as resolved by the grid.
Errors may develop due to representation of discontinuities (shocks, slip surfaces, interfaces, ...) on
a grid. Interpolation errors come about at zonal interfaces where the solution of one zone is
approximated on the boundary of the other zone.

The truncation error is the difference between the partial differential equation (PDE) and the finite
equation. The truncation error is a function of the grid quality and flow gradients. Dispersive error
terms causes oscillations in the solution. One fix to this is adding artificial dissipation to decrease
the size of the dispersive errors. Dissipation error terms cause a smoothing of gradients. However, a
level of dissipation comparable to the actual physical viscosity may contaminate the solution.
Boundary layers may thicken. The truncation error terms are those of the expansion which are not
used in the discretized equation. If the order of the leading term of the truncation error is of second-
order, it is known as a numerical viscosity (dimensions of length2 / time), which is dimensions of
kinematic viscosity. A positive viscous term will indicate that errors will be damped whereas a
negative viscous term will indicate that errors will grow (unstable).

Included in the discretization error are errors due to not properly converging the solution with
respect to the iterations to the steady-state solution or within a time step. This is reffered to as
iterative convergence.

Computer Programming Errors

Programming errors are "bugs" and mistakes made in programming or writing the code. They are
the responsibility of the programmers. These type of errors are discovered by systematically
performing verification studies of subprograms of the code and the entire code, reviewing the lines
of code, and performing validation studies of the code. The programming errors should be removed
from the code prior to release.

Usage Errors

Usage errors are due to the application of the code in a less-than-accurate or improper manner.
Usage errors may actually show up as modeling and discretization errors. The user sets the models,
grid, algorithm, and inputs used in a simulation, which then establishes the accuracy of the
simulation. There may be blatant errors, such as attempting to compute a known turbulent flow with
an assumption of inviscid flow. A converged solution may be obtained; however, the conclusions
drawn from the simulation may be incorrect. The errors may not be as evident, such as proper
choice of turbulence model parameters for separated flows with shocks. The potential for usage
errors increases with an increased level of options available in a CFD code. Usage errors are
minimized through proper training and the accumulation of experience.
 Overview of CFD Verification and Validation 10

The user may intentionally introduce modeling and discretization error as an attempt to expedite the
simulation at the expense of accuracy. This may be proper in the conceptual stage of a design study
where more general information is needed at less accuracy. Even in the later stages, there may not
be proper computational resources to simulate at the proper grid density. One has to understand the
level of accuracy accompanying the results.

Usage errors should be controlable through proper training and analysis.

Usage errors can exist in the CAD, grid generation, and post-processing software, in addition to the
CFD code.

Verification Assessment
This page discusses Verification Assessment, which focuses on the methods for Verification of CFD
codes and simulation results.

Verification

Verification is defined as:

The process of determining that a model implementation accurately represents the developer's
conceptual description of the model and the solution to the model. (AIAA G-077-1998)

Verification assessment examines 1) if the computational models are the correct implementation of
the conceptual models, and 2) if the resulting code can be properly used for an analysis. The
strategy is to identify and quantify the errors in the model implementation and the solution. The two
aspects of verification are the verification of a code and the verification of a calculation. The
objective of verifying a code is error evaluation, that is, finding and removing errors in the code.
The objective of verifying a calculation is error estimation, that is determining the accuracy of a
calculation. Each are discussed below.

Verification has also been described as solving the equations right. It is intended to concern itself
more with mathematics rather than engineering. It is intended to look for errors in the programming
and implementation of the models.

Roache considers two aspects of verification: 1) verification of a code and 2) verification of a

calculation. These two aspects are described below:

Verification of a code involves error evaluation, which is, looking for bugs, incorrect
implementations of conceptual models, errors in inputs, and other errors in the code and usage. This
is typically done by the developers prior to release of the code. First, consistency checks are
performed which examine basic relationships expected in the solutions (i.e. mass conservation).
Then the code is used to simulate a suite of ``highly accurate'' verification cases. These cases
should be analytic or numeric solution to ordinary and partial differential equations. Verification
should not performed with experimental data. A grid refinement study should be conducted to bring
out potential errors. All the options of the code should be examined. This becomes more
complicated as the number of options available within a CFD code increase. Identifying and
quantifying each type of error is important because errors can interact and cancel each other -
leading to erroneous conclusions in the validation process. One potentially useful method of
 Overview of CFD Verification and Validation 11

verification is comparing the results of two codes. However, verification is not a democratic activity
and one should watch for comparing with an inaccurate code. The comparison is strengthened when
the two codes use differing numerical methods. The following paragraphs discuss specific checks
that can be performed as part of a code verification process.
Verification of a calculation involves error estimation, which is determining the accuracy of a
single calculation and putting an error band on the final value. The approach involves peformimg a
grid convergence study and determine the observed order of convergence, error bands, and grid
convergence indices (GCI).

Verification Assessment Process

The process for Verification Assessment of a CFD code and / or simulation can be summarized as:

1. Examine the Computer Programming of the Code.

One of the most basic tasks of verification assessment is the review of the computer programming
or coding to check for and identify computer programming errors or "bugs". This is done by
visually checking the coding and by computationally running subprograms using a test code. This is
aided by complete and clear documentation, both internal and external. This step is to directly
detect computer programming errors.

2. Examine Iterative Convergence.

Verification assessment requires that a simulation demonstrates iterative convergence. Further

details can be page entitled Examining Iterative Convergence.

3. Examine Consistency.

One should check for consistency in the CFD solution. For example, the flow in a duct should
maintain mass conservation through the duct. Further total pressure recovery in an inlet should stay
constant or decrease through the duct.

4. Examine Spatial (Grid) Convergence.

The CFD simulation results should demonstrate spatial convergence. Further details and methods
can be found on the page entitled Examining Spatial (Grid) Convergence.

5. Examine Temporal Convergence.

The CFD simulation results should demonstrate temporal convergence. Further details and methods
can be found on the page entitled Examining Temporal Convergence.

6. Compare CFD Results to Highly Accurate Solutions

The veracity of a code can be examined by comparing the CFD simulation results to highly accurate
solution to the models used within the CFD code. This can include analytical solutions, benchmark
numerical solutions to ordinary differential equations (ODEs), and benchmark numerical solutions
to partial differential equations (PDEs).
 Overview of CFD Verification and Validation 12

Validation Assessment
This page discusses Validation Assessment, which focuses on the methods for the Validation of a
CFD codes for simulation of a certain type of flows.

Validation

Validation is defined as:

The process of determining the degree to which a model is an accurate representation of the real
world from the perspective of the intended uses of the model. (AIAA G-077-1998)

Validation has also been described as "solving the right equations". It is not possible to validate the
entire CFD code. One can only validate the code for a specific range of applications for which there
is experimental data. Thus one validates a model or simulation. Applying the code to flows beyond
the region of validity is termed prediction.

Validation examines if the conceptual models, computational models as implemented into the CFD
code, and computational simulation agree with real world observations. The strategy is to indentify
and quantify error and uncertainty through comparison of simulation results with experimental data.
The experiment data sets themselves will contain bias errors and random errors which must be
properly quantified and documented as part of the data set. The accuracy required in the validation
activities is dependent on the application, and so, the validation should be flexible to allow various
levels of accuracy.

The approach to Validation Assessment is to perform a systematic comparison of CFD simulation

results to experimental data from a set increasingly complex cases.

Each CFD simulation requires verification of the calculation as specified in the discussion of
Verification Assessment.

Validation Assessment Process

The process for Validation Assessment of a CFD simulation can be summarized as:

1. Examine Iterative Convergence.

Validation assessment requires that a simulation demonstrates iterative convergence. Further details
can be page entitled Examining Iterative Convergence.

2. Examine Consistency.

One should check for consistency in the CFD solution. For example, the flow in a duct should
maintain mass conservation through the duct. Further total pressure recovery in an inlet should stay
constant or decrease through the duct.

3. Examine Spatial (Grid) Convergence.

The CFD simulation results should demonstrate spatial convergence. Further details and methods
can be found on the page entitled Examining Spatial (Grid) Convergence.
 Overview of CFD Verification and Validation 13

4. Examine Temporal Convergence.

The CFD simulation results should demonstrate temporal convergence. Further details and methods
can be found on the page entitled Examining Temporal Convergence.

5. Compare CFD Results to Experimental Data.

Experimental data is the observation of the "real world" in some controlled manner. By comparing
the CFD results to experimental data, one hopes that there is a good agreement, which inreases
confidence that the physical models and the code represents the "real world" for this class of
simulations. However, the experimental data contains some level of error. This is usually related to
the complexity of the experiment. Validation assessment calls for a "building block" approach of
experiments which sets a hierarchy of experiment complexity.

6. Examine Model Uncertainties.

The physical models in the CFD code contain uncertainties due to a lack of complete understanding
or knowledge of the physical processes. One of the models with the most uncertainty is the
turbulence models. The uncertainty can be examined by running a number of simulations with the
various turbulence models and examine the affect on the results.

Building-Block Approach for Experiments

A building-block approach is followed in performing the validation assessment for a complex

system such as an aircraft inlet. The approach consists of phases involving successively more
complex flow physics, geometry, and interactions. These phases include:

Unit Problems involve simple geometry, one element of the complex flow physics, and one
relevant flow feature. An example is the measurement of a turbulent boundary layer over a flat
plate. The experiment data set contains detailed data collected with high accuracy. The boundary
conditions and initial conditions are accurately measured.
Benchmark Cases involve fairly simple hardware representing a key feature of the system. The
flow field contains only two separate flow features of the flow physics which are likely coupled. An
example is a shock / boundary layer interaction. The experiment data set is extensive in scope and
uncertainties are low; however, some measurements, such as, initial and boundary conditions, may
not have been collected.
Subsystem Cases involve geometry of a component of the complete system which may have been
simplified. The flow physics of the complete system may be well represented; but the level of
coupling between flow phenomena is typically reduced. An example is a test of a subsonic diffuser
for a supersonic inlet. The exact inflow conditions may not be matched. The quality and quantity of
the experiment data set may not be as extensive as the benchmark cases.
Complete System Cases involve actual hardware and the complete flow physics. All of the
relevant flow features are present. An example is a test of a mixed-compression inlet in the 10x10
wind tunnel at NASA Glenn. Less detailed data is collected since the emphasis is on system
evaluation. Uncertainties on initial and boundary conditions may be large.

Requirements for Experimental Data

The experimental data likely has uncertainties and error associated with it. In comparing the CFD
simulation results to experimental data, one should discuss the experimental errors. Plots comparing
 Overview of CFD Verification and Validation 14

CFD results and experimental data should include a visual display of the error bars on the
experimental data.

Examining Iterative Convergence

Generally, CFD methods involve some iterative scheme to arrive at the simulation results. Here it is
assumed that the iteration is with respect to time or a pseudo-temporal quantity and some type of
time step is taken at each iteration. A steady-state flow simulation involves starting with a uniform
or fabricated flow field and iterating in time until the steady-state flow field is obtained. This is
termed iterative convergence, but requires some criteria for determining convergence. Criteria
include:

Residuals. The residuals of the equations are the change in the equations over an iteration. These
are usually scaled or normalized. One usually looks for the residuals to reach a certain level and
then level-off as an indication of iterative convergence. For a time-marching, steady-state strategy,
this involves examining whether the residual has been reduced a certain number (usually 3-4) of
orders of magnitude.

Results. The CFD simulation has the objective of determining some quantity such as lift, drag,
recovery, etc... One can track the values of such engineering quantities with respect to iteration and
define iterative convergence when these quantities converge. The convergence criteria is usually
defined by acceptable error in these values. It is often the case that certain quantities may reach
convergence at a different rate than other quantities. One can check that a monitored flow value
(such as thrust, drag, or boundary layer profile) has remained unchanged with respect to the number
of iterations.

Time-Accurate Simulations. For a time-marching, time-accurate strategy, this involves examining

whether the final time has been reached with proper convergence at each time step.

Space-Marching Simulation For a space-marching strategy, this involves examining whether the
end of the marching segment has been reached with proper convergence at each marching step.

Examining Solution Consistency

One can evaluate convergence by checking for consistency in the flow field. The conservation
relations require a balance of fluxes through a control surface. For a closed duct, the flow through
the duct should be conserved. Low-speed flow over a closed body should have zero drag. Other
such consistency relations can be defined for specific flow fields. These provide verification of the
code since the consistency relations are usually a statement of some analytic result.
 Overview of CFD Verification and Validation 15

Examining Spatial (Grid) Convergence

Introduction

The examination of the spatial convergence of a simulation is a straight-forward method for

determining the ordered discretization error in a CFD simulation. The method involves performing
the simulation on two or more successively finer grids. The term grid convergence study is
equivalent to the commonly used term grid refinement study.

As the grid is refined (grid cells become smaller and the number of cells in the flow domain
increase) and the time step is refined (reduced) the spatial and temporal discretization errors,
respectively, should asymptotically approaches zero, excluding computer round-off error.

Methods for examining the spatial and temporal convergence of CFD simulations are presented in
the book by Roache. They are based on use of Richardson's extrapolation. A summary of the
method is presented here.

A general discussion of errors in CFD computations is available for background.

We will mostly likely want to determine the error band for the engineering quantities obtained from
the finest grid solution. However, if the CFD simulations are part of a design study that may require
tens or hundreds of simulations, we may want to use one of the coarser grids. Thus we may also
want to be able to determine the error on the coarser grid.

Grid Considerations for a Grid Convergence Study

The easiest approach for generating the series of grids is to generate a grid with what one would
consider fine grid spacing, perhaps reaching the upper limit of one's tolerance for generating a grid
or waiting for the computation on that grid to converge. Then coarser grids can be obtained by
removing every other grid line in each coordinate direction. This can be continued to create
additional levels of coarser grids. In generating the fine grid, one can build in the n levels of coarser
grids by making sure that the number of grid points in each coordinate direction satisfies the
relation

N = 2n m + 1

where m is an integer. For example, if two levels of coarser grids are desired (i.e. fine, medium, and
coarse grids) then the number of grid points in each coordinate direction must equal 4 m + 1. The m
may be different for each coordinate direction.

The WIND code has a grid sequencing control that will solve the solution on the coarser grid
without having to change the grid input file, boundary condition settings, or the input data file.
Further, the converged solution on the coarser grid then can be used directly as the initial solution
on the finer grid. This option was originally used to speed up convergence of solutions; however,
can be used effectively for a grid convergence study.

It is not necessary to halve the number of grid points in each coordinate direction to obtain the
coarser grid. Non-integer grid refinement or coarsening can be used. This may be desired since
halfing a grid may put the solution out of the asymptotic range. Non-integer grid refinement or
coarsening will require the generation of a new grid. It is important to maintain the same grid
 Overview of CFD Verification and Validation 16

generation parameters as the original grid. One approach is to select several grid spacings as
reference grid spacings. One should be the grid spacing normal to the walls. Others may be spacings
at flow boundaries, at junctures in the geometry, or at zonal interfaces. Upon picking the ratio as
which the grid is to be refined or coarsened, this same ratio is applied to these spacings. The number
of grid points are then adjusted according to grid quality parameters, such as grid spacing ratio
limits. The surface and volume grids are then generated using the same methods as the original grid.
The grid refinement ratio should be a minimum of r >= 1.1 to allow the discretization error to be
differentiated from other error sources (iterative convergence errors, computer round-off, etc...).

Order of Grid Convergence

The order of grid convergence involves the behavior of the solution error defined as the difference
between the discrete solution and the exact solution,

where C is a constant, h is some measure of grid spacing, and p is the order of convergence. A
"second-order" solution would have p = 2.

A CFD code uses a numerical algorithm that will provide a theoretical order of convergence;
however, the boundary conditions, numerical models, and grid will reduce this order so that the
observed order of convergence will likely be lower.

Neglecting higher-order terms and taking the logarithm of both sides of the above equation results
in:

The order of convergence p can be obtained from the slope of the curve of log(E) versus log(h). If
such data points are available, the slope can be read from the graph or the slope can be computed
from a least-squares fit of the data. The least-squares will likely be inaccurate if there are only a few
data points.

A more direct evaluation of p can be obtained from three solutions using a constant grid refinement
ratio r,

The order of accuracy is determined by the order of the leading term of the truncation error and is
represented with respect to the scale of the discretization, h. The local order of accuracy is the
order for the stencil representing the discretization of the equation at one location in the grid. The
global order of accuracy considers the propagation and accumulation of errors outside the stencil.
This propagation causes the global order of accuracy to be, in general, one degree less than the local
order of accuracy. The order of accuracy of the boundary conditions can be one order of accuracy
lower than the interior order of accuracy without degrading the overall global accuracy.

Asymptotic Range of Convergence

Assessing the accuracy of code and caluculations requires that the grid is sufficiently refined such
that the solution is in the asymptotic range of convergence. The asymptotic range of convergence is
 Overview of CFD Verification and Validation 17

obtained when the grid spacing is such that the various grid spacings h and errors E result in the
constancy of C,

C = E / hp

Another check of the asymptotic range will be discussed in the section on the grid convergence
index.

Richardson Extrapolation

Richardson extrapolation is a method for obtaining a higher-order estimate of the continuum value
(value at zero grid spacing) from a series of lower-order discrete values.

A simulation will yield a quantity f that can be expressed in a general form by the series expansion:

f = fh=0 + g1 h + g2 h2 + g3 h3 + ...

where h is the grid spacing and the functions g1, g2, and g3 are independent of the grid spacing. The
quantity f is considered "second-order" if g1 = 0.0. The fh=0 is the continuum value at zero grid
spacing.

If one assumes a second-order solution and has computed f on two grids of spacing h1 and h2 with
h1 being the finer (smaller) spacing, then one can write two equations for the above expansion,
neglect third-order and higher terms, and solve for fh=0 to estimate the continuum value,

where the grid refinement ratio is:

r = h2 / h1

The Richardson extrapolation can be generalized for a p-th order methods and r-value of grid ratio
(which does not have to be an integer) as:

Traditionally, Richardson extrapolation has been used with grid refinement ratios of r = 2. Thus, the
above equation simplifies to:

In theory, the above equations for the Richardson extrapolation will provide a fourth-order estimate
of fh=0, if f1 and f2 were computed using exactly second-order methods. Otherwise, it will be a third-
order estimate. In general, we will consider fh=0 to be p+1 order accurate. Richardson extrapolation
can be applied for the solution at each grid point, or to solution functionals, such as pressure
recovery or drag. This assumes that the solution is globally second-order in addition to locally
second-order and that the solution functionals were computed using consistent second-order
methods. Other cautions with using Richardson extrapolation (non-conservative, amplification of
round-off error, etc...) are discussed in the book by Roache.
 Overview of CFD Verification and Validation 18

For our purposes we will assume f is a solution functional (i.e. pressure recovery). The fh=0 is then
as an estimate of f in the limit as the grid spacing goes to zero. One use of fh=0 is to report the value
as the an improved estimate of f1 from the CFD study; however, one has to understand the caveats
mentioned above that go along with that value.

The other use of fh=0 is to obtain an estimate of the discretization error that bands f obtained from
the CFD. This use will now be examined.

The difference between f1 and fh=0 is one error estimator; however, this requires consideration of the
caveats attached to fh=0.

We will focus on using f1 and f2 to obtain an error estimate. Examining the generalized Richardson
extrapolation equation above, the second term on the right-hand side can be considered to be an an
error estimator of f1. The equation can be expressed as:

A1 = E1 + O( hp+1, E12)

where A1 is the actual fractional error defined as:

A1 = ( f1 - fh=0 ) / fh=0

and E1 is the estimated fractional error for f1 defined as:

where the relative error is defined as:

This quantity should not be used as an error estimator since it does not take into account r or p. This
may lead to an underestimation or overestimation of the error. One could make this quantity
artificially small by simply using a grid refinement ratio r close to 1.0.

The estimated fractional error E1 is an ordered error estimator and a good approximation of the
discretization error on the fine grid if f1 and f2 were obtained with good accuracy (i.e. E1<1). The
value of E1 may be meaningless if f1 and fh=0 are zero or very small relative to f2-f1. If such is the
case, then another normalizing value should be used in place of f1.

If a large number of CFD computations are to be performed (i.e for a DOE study), one may wish to
use the coarser grid with h2. We will then want to estimate the error on the coarser grid. The
Richardson extrapolation can be expressed as:

The estimated fractional error for f2 is defined as:

 Overview of CFD Verification and Validation 19

Richardson extrapolation is based on a Taylor series representation as indicated in Eqn.

\ref{eq:series}. If there are shocks and other discontinuities present, the Richardson extrapolation is
invalid in the region of the discontinuity. It is still felt that it applies to solution functionals
computed from the entire flow field.

Grid Convergence Index (GCI)

Roache suggests a grid convergence index GCI to provide a consistent manner in reporting the
results of grid convergence studies and perhaps provide an error band on the grid convergence of
the solution. The GCI can be computed using two levels of grid; however, three levels are
recommended in order to accurately estimate the order of convergence and to check that the
solutions are within the asymptotic range of convergence.

A consistent numerical analysis will provide a result which approaches the actual result as the grid
resolution approaches zero. Thus, the discretized equations will approach the solution of the actual
equations. One significant issue in numerical computations is what level of grid resolution is
appropriate. This is a function of the flow conditions, type of analysis, geometry, and other
variables. One is often left to start with a grid resolution and then conduct a series of grid
refinements to assess the effect of grid resolution. This is known as a grid refinement study.

One must recognize the distinction between a numerical result which approaches an asymptotic
numerical value and one which approaches the true solution. It is hoped that as the grid is refined
and resolution improves that the computed solution will not change much and approach an
asymptotic value (i.e. the true numerical solution). There still may be error between this asymptotic
value and the true physical solution to the equations.

Roache has provided a methodology for the uniform reporting of grid refinement studies. "The
basic idea is to approximately relate the results from any grid refinement test to the expected results
from a grid doubling using a second-order method. The GCI is based upon a grid refinement error
estimator derived from the theory of generalized Richardson Extrapolation. It is recommended for
use whether or not Richardson Extrapolation is actually used to improve the accuracy, and in some
cases even if the conditions for the theory do not strictly hold." The object is to provide a measure
of uncertainty of the grid convergence.

The GCI is a measure of the percentage the computed value is away from the value of the
asymptotic numerical value. It indicates an error band on how far the solution is from the
asymptotic value. It indicates how much the solution would change with a further refinement of the
grid. A small value of GCI indicates that the computation is within the asymptotic range.

The GCI on the fine grid is defined as:

where Fs is a factor of safety. The refinement may be spatial or in time. The factor of safety is
recommended to be Fs=3.0 for comparisons of two grids and Fs=1.25 for comparisons over three or
more grids. The higher factor of safety is recommended for reporting purposes and is quite
conservative of the actual errors.

When a design or analysis activity will involve many CFD simulations (i.e. DOE study), one may
want to use the coarser grid h2. It is then necessary to quantify the error for the coarser grid. The
GCI for the coarser grid is defined as:
 Overview of CFD Verification and Validation 20

It is important that each grid level yield solutions that are in the asymptotic range of convergence
for the computed solution. This can be checked by observing two GCI values as computed over
three grids,

GCI23 = rp GCI12

Required Grid Resolution

If a desired accuracy level is known and results from the grid resolution study are available, one can
then estimate the grid resolution required to obtain the desired level of accuracy,

Independent Coordinate Refinement and Mixed Order Methods

The grid refinement ratio assumes that the refinement ratio r applies equally in all coordinate
directions (i,j,k) for steady-state solutions and also time t for time-dependent solutions. If this is not
the case, then the grid convergence indices can be computed for each direction independently and
then added to give the overall grid convergence index,

Effective Grid Refinement Ratio

If one generates a finer or coarser grid and is unsure of the value of grid refinement ratio to use, one
can compute an effective grid refinement ratio as:

where N is the total number of grid points used for the grid and D is the dimension of the flow
domain. This effective grid refinement ratio can also be used for unstructured grids.

Example Grid Convergence Study

The following example demonstrates the application of the above procedures in conducting a grid
convergence study. The objective of the CFD analysis was to determine the pressure recovery for an
inlet. The flow field is computed on three grids, each with twice the number of grid points in the i
and j coordinate directions as the previous grid. The number of grid points in the k direction
remains the same. Since the flow is axisymmetric in the k direction, we consider the finer grid to be
double the next coarser grid. The table below indicates the grid information and the resulting
pressure recovery computed from the solutions. Each solution was properly converged with respect
to iterations. The column indicated by "spacing" is the spacing normalized by the spacing of the
finest grid.
 Overview of CFD Verification and Validation 21

Grid Normalized Grid Spacing Pressure Recovery, Pr

1 1 0.97050
2 2 0.96854
3 4 0.96178

The figure below shows the plot of pressure recoveries with varying grid spacings. As the grid
spacing reduces, the pressure recoveries approach an asymptotic zero-grid spacing value.

We determine the order of convergence observed from these results,

p = ln[ ( 0.96178 - 0.96854 ) / ( 0.96854 - 0.97050 ) ] / ln(2) = 1.786170

The theoretical order of convergence is p=2.0. The difference is most likely due to grid stretching,
grid quality, non-linearities in the solution, presence of shocks, turbulence modeling, and perhaps
other factors.

We now can apply Richardson extrapolation using the two finest grids to obtain an estimate of the
value of the pressure recovery at zero grid spacing,

Prh=0 = 0.97050 + ( 0.97050 - 0.96854 ) / ( 21.786170 - 1 )

= 0.97050 + 0.00080 = 0.97130

This value is also plotted on the figure below.

The grid convergence index for the fine grid solution can now be computed. A factor of safety of
FS=1.25 is used since three grids were used to estimage p. The GCI for grids 1 and 2 is:

GCI12 = 1.25 | ( 0.97050 - 0.96854 ) / 0.97050 | / ( 21.786170 - 1 ) 100% = 0.103083%

The GCI for grids 2 and 3 is:

GCI23 = 1.25 | ( 0.96854 - 0.96178 ) / 0.96854 | / ( 21.786170 - 1 ) 100% = 0.356249%

We can now check that the solutions were in the asymptotic range of convergence,

0.356249 / ( 21.786170 0.103083 ) = 1.002019

which is approximately one and indicates that the solutions are well within the asymptotic range of
convergence.

Based on this study we could say that the pressure recovery for the supersonic ramp is estimated to
be Pr = 0.97130 with an error band of 0.103% or 0.001.
 Overview of CFD Verification and Validation 22

VERIFY: A Fortran program to Perform Calculations Associated with a Grid

Convergence Study

The Fortran 90 program verify.f90 was written to carry out the calculations associated with a grid
convergence study involving 3 or more grids. The program is compiled on a unix system through
the commands:

f90 verify.f90 -o verify

It reads in an ASCII file (prD.do) through the standard input unit (5) that contains a list of pairs of
grid size and value of the observed quantity f.

verify < prD.do > prD.out

It assumes the values from the finest grid are listed first. The output is then written to the standard
output unit (6) prD.out. The output from the of {\tt verify} for the results of Appendix A are:

--- VERIFY: Performs verification calculations ---

Number of data sets read = 3

Grid Size Quantity

1.000000 0.970500
2.000000 0.968540
4.000000 0.961780

Order of convergence using first three finest grid

and assuming constant grid refinement (Eqn. 5.10.6.1)
Order of Convergence, p = 1.78618479

Richardson Extrapolation: Use above order of convergence

 Overview of CFD Verification and Validation 23

and first and second finest grids (Eqn. 5.4.1)

Estimate to zero grid value, f_exact = 0.971300304

Grid Convergence Index on fine grids. Uses p from above.

Factor of Safety = 1.25

Grid Refinement
Step Ratio, r GCI(%)
1 2 2.000000 0.103080
2 3 2.000000 0.356244

Checking for asymptotic range using Eqn. 5.10.5.2.

A ratio of 1.0 indicates asymptotic range.

Grid Range Ratio

12 23 0.997980

--- End of VERIFY ---

Examples of Grid Converence Studies in the Archive

A grid convergence study is performed in the Supersonic Wedge case.

Examples of Grid Converence Studies in Literature

Other examples of grid convergence studies that use the procedures outlined above can be found in
the book by Roache and the paper by Steffen et al..

NPARC Alliance Policy with Respect to Grid Converence Studies

For the WIND verification and validation effort, it is suggested that the above procedures be used
when conducting and reporting results from a grid convergence study.

Examining Temporal Convergence

Time-accurate simulations involve taking discrete time steps. One must examine the sensitivity of
the simulation results to the magnitude of the time step. The effects and possible errors are usually
related to the time filtering of various time scales existing in the unsteady flow field.

Glossary of Verification and Validation Terms

The following is a glossary for verification and validation. The objective is to clarify meanings and
indicate how some of the terms have specific definitions that may be different than common usage
of those terms. For example, verification and validation share the same meaning in most
dictionaries and even in common usage among technical personnel; however, here we are
concerned with attaching to them specific technical definitions that are associated with different
aspects of CFD. Most of the definitions come directly from AIAA G-077-1998, "Guide for the
Verification and Validation of Computational Fluid Dynamics Simulations". Other definitions and
discussions come from other literature cited in the bibliography. The terms are listed alphabetically.
 Overview of CFD Verification and Validation 24

Calibration

Calibration in the context of CFD simulations is defined as

The process of adjusting numerical or physical modeling parameters in the computational model
for the purpose of improving agreement with experimental data. (AIAA G-077-1998)

Calibration has more to do with improving agreement of computational results with experimental
results rather than assessing error and uncertainty. Calibration arises from uncertainty in modeling
complex physical processes. It adjusts unmeasured or poorly characterized experimental
parameters. One use of calibration may be the adjustment of emperical constants found within a
turbulence model. While calibration improves agreement for a class of problems, it may make the
code less general.

Certification.

Certification of a code encompases verification and validation, but also includes such things as
documentation, quality assurance, and version control. Certification is concerned with managerial
and programmatic concerns on the use of the CFD code for design activities. Certification is
intended to put a "seal" of credibility on a production code for use in a design project. Certification
can mean that a certain standard has been satisfied (i.e. ISO, NASA). NASA Glenn has a center
level procedure LeR-P2.10.2 entitled "Software Product Assurance" which offers some high level
guidance on verification and validation of software.

Code.

A code is a set of computer instructions and data inputs and definitions. This is the CFD code
(WIND, NPARC, OVERFLOW, USM3D, ...) which may include ancillary codes and
documentation as part of the software package. A code typically has three stages related to the level
of validation completed: research, pilot, and production. The production code has been fully
validated for the intended design applications, including a system-level validation (full inlet, inlet-
aircraft integration).

Credibility.

An improvement of credibility is considered to be the same as confidence building or providing

quality to the customer.

Error.

Error is defined as
A recoqnizable deficiency in any phase or activity of modeling and simulation that is not due to lack
of knowledge. (AIAA G-077-1998)

Further discussion of error can be found at the page entitled Uncertainty and Error in CFD
Simulations.

Grid Convergence.

Grid convergence indicates that as the grid spacing is reduced, the computed simulation results
approach the continum result. Here "grid spacing" can refer to both spatial spacing, as well as, time
 Overview of CFD Verification and Validation 25

step for the case of unsteady, time-accurate simulations. Further discussion can be found on the
page entitled Examining Spatial (Grid) Convergence.

Iterative Convergence.

Iterative convergence indicates that as the discrete equations are iterated, the computed simulation
results approach a fixed value. Further discussion can be found on the page entitled Examining
Iterative Convergence.

Model.

A model is defined as:

A representation of a physical system or process intended to enhance our ability to understand,
predict, or control its behavior. (AIAA G-077-1998)

A conceptual model for CFD consists of the observations, mathematical modeling data, and
mathematical (partial differential) equations that describe the physical system. It also includes
initial and boundary conditions.

The computational model is the computer program or code that implements the conceptual model.
This may be finite-difference, finite-volume, finite-element, or other type of discretization. It
includes the algorithms and iterative strategies. Parameters for the computational model include the
number of grid points, algorithm inputs, and similar parameters.

Modeling.

Modeling is defined as
The process of construction or modification of a model. (AIAA G-077-1998)

Prediction

. Prediction is defined as
Use of a CFD model to foretell the state of a physical system under conditions for which the CFD
model has not been validated. (AIAA G-077-1998)

Prediction is going beyond the validation database and performing simulations of untested systems.
This of course, is a tremendous power of CFD and of use in design studies. However, we would like
to be able to estimate the accuracy of the predictions. Unfortunately, the verification and validation
processes can not formally provide these estimates. The best one can do is point to the verification
results and the most similar validation case.

Robustness

. Robustness defines the ability of the numerical method to provide a solution despite variabilities in
the initial solution and control parameters. This incorporates issues of fault tolerance. Generally,
robustness is achieved at the expense of accuracy.

Simulation.

A simulation is defined as
The exercise or use of a model. (That is, a model is used in a simulation). (AIAA G-077-1998)
 Overview of CFD Verification and Validation 26

For a CFD analysis the application or run of the CFD code is a simulation.

Uncertainty

. Uncertainty is defined as
"A potential deficiency in any phase or activity of the modeling process that is due to the lack of
knowledge." (AIAA G-077-1998)

Further discussion of uncertainty can be found at the page entitled Uncertainty and Error in CFD
Simulations.

Validation

. Validation is defined as
The process of determining the degree to which a model is an accurate representation of the real
world from the perspective of the intended uses of the model. (AIAA G-077-1998)

Further discussion of validation can be found at the page entitled Validation Assessment.

Verification.

Verification is defined as:

The process of determining that a model implementation accurately represents the developer's
conceptual description of the model and the solution to the model. (AIAA G-077-1998)

Further discussion of verification can be found at the page entitled Verification Assessment.
 Overview of CFD Verification and Validation 27

Bibliography of CFD Verification and Validation

The following is a list of references related to CFD Verification and Validation of CFD codes. The
references are grouped according to whether they are:

• Policy Statements
• Books
• Compilations and Proceedings
• Individual Papers

If anyone has any additional references they would like to have listed, please contact the
Webmaster.

Policy Statements

Roache, P.J., K. Ghia, and F. White, "Editorial Policy Statement on the Control of Numerical
Accuracy," ASME Journal of Fluids Engineering, Vol. 108,No. 1., March 1986, p. 2.

AIAA, "Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty," AIAA
Journal, Vol. 32, January 1994, p. 3.

AIAA, "Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,"
AIAA G-077-1998, 1998.

ASME Editorial Board, "Journal of Heat Transfer Editorial Policy Statement on Numerical
Accuracy," ASME Journal of Heat Transfer, Vol. 116, November 1994. pp. 797-798.

Books

Anderson, D.A., Tannehill, J.C., and Pletcher, R.H. , Computational Fluid Mechanics and Heat
Transfer, McGraw-Hill Book Company, New York, 1984.

Briggs, W.L., A Multigrid Tutorial, SIAM, Philidelphia, PA, 1987.

ERCOFTAC, Best Practices Guidelines for Industrial Computational Fluid Dynamics, Version 1.0,
January 2000.

Hirsch, C. Numerical Computation of Internal and External Flows, Volume I: Fundamentals of

Numerical Discretization. New York: John Wiley & Sons, 1988.

Hirsch, C. Numerical Computation of Internal and External Flows, Volume II: Computational
Methods for Inviscid and Viscous Flows. New York: John Wiley & Sons, 1990.

Roache, P.J., Verification and Validation in Computational Science and Engineering, Hermosa
Publishers, Albuquerque, New Mexico, 1998.
 Overview of CFD Verification and Validation 28

Roache, P.J., Fundamentals of Computational Fluid Dynamics, Hermosa Publishers, Albuquerque,

New Mexico, 1998.

Shyy, W., Computational Modeling for Fluid Flow and Interfacial Transport, New York: Elsevier,
1994.

Compilations and Proceedings

AGARD, Validation of Computational Fluid Dynamics, Lisbon, Portugal, May 2-5, 1988, NATO
Advisory Group for Aeronautical Research and Development, AGARD CP 437, December 1988.

AIAA, AIAA Journal of Spacecraft and Rockets, Vol. 27, No. 2, March-April 1990, pp. 97-215.
This issue contains 5 papers on CFD Code Validation / Verification / Certification with the
emphasis on hypersonic flight. Another section entitled CFD Code Applications has 8 papers on
applying CFD codes for hypersonic flight and includes some discussion on validation. Some of
these 13 papers are listed individually below.

AIAA, AIAA Journal, Vol. 36, No. 5, May 1998, pp. 665-764. This issue contains 12 papers in a
special section entitled Credible Computational Fluid Dynamics Simulations. Some of these 12
papers are listed individually below.

AGARD, Experimental Data Base for Computer Program Assessment, Report of the Fluid
Dynamics Panel Working Group 04, AGARD-AR-138, May 1979. This is the report that contains
the papers and data for the RAE2822 airfoil and ONERA M6 wing.

Individual Papers

Aeschliman, D.P., W.L. Oberkampf, and F.G. Blottner, "A Proposed Methodology for
Computational Fluid Dynamics Code Verification, Calibration, and Validation," Paper presented at
the 16th International Congress on Instrumentation in Aerospace Simulation Facilities (ICIASF),
July 18-21, 1995, Wright-Patterson AFB, OH 45433.

Aeschliman, D.P. and W.L. Oberkampf, "Experimental Methodology for Computational Fluid
Dynamics Code Validation," AIAA Journal, Vol. 36, No. 5, pp. 733-741.

Barber, T.J., "Role of Code Validation and Certification in the Design Environment," AIAA
Journal, Vol. 36, No. 5, pp. 752-758.

Bardina, J.E., P.G. Huang, and T.J. Coakley, "Turbulence Modeling Validation, Testing, and
Development," NASA TM 110446, April 1997.

Benek, J.A., E.M. Kraft, and R.F. Lauer, "Validation Issues for Engine - Airframe Integration,"
AIAA Journal, Vol. 36, No. 5, pp. 759-764.

Blottner, F.G., "Accurate Navier-Stokes Results for the Hypersonic Flow over a Spherical Nosetip,"
AIAA Journal of Spacecraft and Rockets, Vol. 27, No. 2, pp. 113-122.
 Overview of CFD Verification and Validation 29

Bobbitt, P.J., "The Pros and Cons of Code Validation," AIAA Paper 88-2535 (NASA TM 100657),
July 1988.

Coleman, H.W. and F. Stern, "Uncertainties and CFD Validation," ASME Journal of Fluids
Engineering, Vol. 119, December 1997, pp. 795-803.

Cosner, R.R., "Issues in Aerospace Application of CFD Analysis," AIAA Paper 94-0464, January
1994.

Cosner, R.R., "CFD Validation Requirements for Technology Transition," AIAA Paper 95-2227,
June 1995.

Dolling, D.S, "High-Speed Turbulent Separated Flows: Consistency of Mathematical Models and
Flow Physics,"

AIAA Journal, Vol. 36, No. 5, pp. 725-732.

Dudek, J.C., N.J. Georgiadis, and D.A. Yoder, "Calculation of Turbulent Subsonic Diffuser Flows
Using the NPARC Navier-Stokes Code," AIAA Paper 96-0497, January 1996.

Dudek, J.C., "NPARC Validation - Subsonic Turbulent Diffusing Pipe Flow," The NPARC
Alliance, April 1996.

Dudek, J.C., "Testing Guidelines for NPARC Alliance Software Development," The NPARC
Alliance, April 1997.

Dudek, J.C., D.O. Davis, and J.W. Slater, "Validation and Verificaiton of the WIND Code for
Supersonic Diffuser Flows," AIAA Paper 2001-0224, January 2001.

Gnoffo, P.A., "CFD Validation Studies for Hypersonic Flow Prediction," AIAA Paper 2001-1025,
January 2001.

Habashi, W.G., J. Dompierre, Y. Bourgault, M. Fortin, and M.-G. Vallet, "Certifiable

Computational Fluid Dynamics Through Mesh Optimization," AIAA Journal, Vol. 36, No. 5, pp.
703-711.

Jameson, A. and L. Martinelli, "Mesh Refinement and Modeling Errors in Flow Simulations,"
AIAA Journal, Vol. 36, No. 5, May 1998, pp. 676-686.

Lewis, C.H., "Comments on the Need for CFD Code Validation," AIAA Journal of Spacecraft and
Rockets, Vol. 27, No. 2, pp. 97.

Marvin, J.G., "Perspective on Computational Fluid Dynamics Validation," AIAA Journal, Vol. 33,
No. 10, October 1995, pp. 1778-1787.

Mehta, U.B., "Computational Requirements for Hypersonic Flight Performance Estimates," AIAA
Journal of Spacecraft and Rockets, Vol. 27, No. 2, pp. 103-112.

Mehta, U.B., "Some Aspects of Uncertainty in Computational Fluid Dynamics Results,"

Transactions of the ASME, Vol. 113, December 1991, pp. 538-543.
 Overview of CFD Verification and Validation 30

Mehta, U.B., "Guide to Credible Computer Simulations of Fluid Flows," AIAA Journal of
Propulsion and Power, Vol. 12, No. 5, September-October 1996, pp. 940-948. (Also AIAA Paper
95-2225).

Mehta, U.B., "Credible Computational Fluid Dynamics Simulations," AIAA Journal, Vol. 36, No.
5, May 1998, pp. 665-667.

NPARC Alliance, "NPARC Alliance Policies and Plans", August, 1999.

Oberkampf, W.L., "A Proposed Framework for Computational Fluid Dynamics Code Calibration /
Validation," AIAA Paper 94-2540, June 1994.

Oberkampf, W.L. and F.G. Blottner, "Issues in Computational Fluid Dynamics Code Verification
and Validation," AIAA Journal, Vol. 36, No. 5, May 1998, pp. 687-695.

Oberkampf, W.L. and T.G. Trucano, "Validation Methodology in Computational Fluid Dynamics"
AIAA Paper 2000-2549, June 2000.

Paynter, G.C. and E. Tjonneland, "Accuracy Issues in the Prediction of Supersonic Inlet Flows,"
ASME 92-GT-400.

Reed, H.L., T.S. Haynes, and W.S. Saric, "Computational Fluid Dynamics Validation Issues in
Transition Modeling," AIAA Journal, Vol. 36, No. 5, pp. 742-751.

Rizzi, A. and J. Vos, "Toward Establishing Credibility in Computational Fluid Dynamic

Simulations," AIAA Journal, Vol. 36, No. 5, May 1998, pp. 668-675.

Roache, P.J., "Need for Control of Numerical Accuracy," AIAA Journal of Spacecraft and Rockets,
Vol. 27, No. 2, pp. 98-102.

Roache, P.J., "Perspective: A Method for Uniform Reporting of Grid Refinement Studies", ASME
Journal of Fluids Engineering, Vol. 116, September 1994.

Roache, P.J., "Quantification of Uncertainty in Computational Fluid Dynamics," Annual Review of

Fluid Mechanics, Vol. 29, 1997, pp. 123-160.

Roache, P.J., "Verification of Codes and Calculations," AIAA Journal, Vol. 36, No. 5, May 1998,
pp. 696-702.

Slater, J.W., J.C. Dudek, and K.E. Tatum, "The NPARC Verification and Validation Archive",
ASME Paper 2000-FED-11233, June 2000.

Slater, J.W., "Verification Assessment of Flow Boundary Conditions for CFD Analysis of
Supersonic Inlet Flows", AIAA Paper 2001-3882, July 2001.

Slater, J.W., J.M. Abbott, and R.H. Cavicchi, "Validation of WIND for a Series of Inlet Flows",
AIAA Paper 2002-0669, January 2002.

Steffen, C.J. Jr., Reddy, D.R., and K.B.M.Q. Zaman, "Analysis of Flowfield from a Rectangular
Nozzle with Delta Tabs", AIAA 95-2146, June, 1995.
 Overview of CFD Verification and Validation 31

Tatum, K.E. and J.W. Slater, "The Validation Archive of the NPARC Alliance", AIAA Paper 99-
0747, January, 1999.

Towne, C.E. and R.R. Jones, "Results and Current Status of the NPARC Alliance Validation
Effort", AIAA Paper 96-0387, January, 1996.

van Wie, D.M. and T. Rice, "Quantification of Data Uncertainties and Validation of CFD Results
in the Development of Hypersonic Airbreathing Engines," AIAA Paper 96-2028, June 1996.

Yee, H.C. and P.K. Sweby, "Aspects of Numerical Uncertainties in Time Marching to Steady-State
Numerical Solutions," AIAA Journal, Vol. 36, No. 5, pp. 712-724.

Links to Web Sites for CFD Verification and Validation

This is a list of web sites that have information related to CFD Verification and Validation:

NPARC Alliance Verification and Validation Archive

CFD Resources at NASA Glenn

CFD Resources at NASA Langley

MADIC/NASA Code Certification Project

CFD Online

ERCOFTAC Databases

UIUC Airfoil Data Site

NASA NAS Data Sets

FLOWnet Test-Cases Database