SMS Module5 Verification and Validation Notes
SMS Module5 Verification and Validation Notes
SMS Module5 Verification and Validation Notes
One of the most important and difficult tasks facing a model developer is the Verification
and validation of the simulation model.
It is the job of the model developer to work closely with the end users Throughout
the period (development and validation to reduce this skepticism
And to increase the credibility.
1: To produce a model that represents true system behavior closely enough for the model to be
used as a substitute for the actual system for the purpose of experimenting with system.
2: To increase an acceptable, level the credibility of the model ,so that the model will be used by
managers and other decision makers. |
2: Validation is concerned with building the right model. It is utilized to determine that a model
is an accurate representation of the real system. It is usually achieved through the calibration of
the model
Model Building, Verification, and Validation
The first step in model building consists of observing the real system and the interactions
among its various components and collecting data on its behavior. Operators, technicians ,repair
and maintenance personnel, engineers, supervisors, and managers under certain aspects of the
system which may be unfamiliar to others. As model development proceeds, new questions may
arise, and the model developers will return, to this step of learning true system structure and
behavior.
The second step in model building is the construction of a conceptual model – a collection of
assumptions on the components and the structure of the system, plus hypotheses on the values of
model input parameters, illustrated by the following figure.
The third step is the translation of the operational model into a computer recognizable
form- the
computerized model
Verification of Simulation Models
The purpose of model verification is to assure that the conceptual model is reflected
accurately in the computerized representation.
The conceptual model quite often involves some degree of abstraction about system
operations, or some amount of simplification of actual operations.
Many common-sense suggestions can be given for use in the verification process:-
Have the computerized representation checked by someone other than its developer.
Make a flow diagram which includes each logically possible action a system can take
when an event occurs, and follow the model logic for each a for each action for each
event type.
Closely examine the model output for reasonableness under a variety of settings of Input
parameters.
Have the computerized representation print the input parameters at the end of
the
Simulation to be sure that these parameter values have not been changed inadvertently.
Make the computerized representation of self-documenting as possible.
If the computerized representation is animated, verify that what is seen in the animation
imitates the actual system.
The interactive run controller (IRC) or debugger is an essential component of Successful
simulation model building. Even the best of simulation analysts makes mistakes or
commits logical errors when building a model.
The IRC assists in finding and correcting those errors in the follow ways:
(a) The simulation can be monitored as it progresses.
(b) Attention can be focused on a particular line of logic or multiple lines of logic
that constitute a procedure or a particular entity.
(c) Values of selected model components can be observed. When the simulation
has paused, the current value or status of variables, attributes, queues, resources,
counters, etc., can be observed
(d) The simulation can be temporarily suspended, or paused, not only to view
information but also to reassign values or redirect entities.
• Example: A model of a complex network of queues consisting many service centers.
– Response time is the primary interest, however, it is important to collect and print
out many statistics in addition to response time.
• Two statistics that give a quick indication of model reasonableness are
current contents and total counts, for example:
– If the current content grows in a more or less linear fashion as the
simulation run time increases, it is likely that a queue is unstable
– If the total count for some subsystem is zero, indicates no items
entered that subsystem, a highly suspect occurrence
– If the total and current count are equal to one, can indicate that an
entity has captured a resource but never freed that resource.
• Compute certain long-run measures of performance, e.g. compute the
long-run server utilization and compare to simulation results
Another advantage of user involvement is the increase in the models perceived validity or
credibility without which manager will not be willing to trust simulation results as the
basis for decision making.
Sensitivity analysis can also be used to check model's face validity.
The model user is asked if the model behaves in the expected way when one or more
input variables is changed.
Based on experience and observations on the real system the model user and model
builder would probably have some notion at least of the direction of change in model
output when an input variable is increased or decreased.
The model builder must attempt to choose the most critical input variables for testing if it
is too expensive or time consuming to: vary all input variables
The number of tellers may be fixed or variable. These structural assumptions should be
verified by actual observation during appropriate time periods together with discussions
with managers and tellers regarding bank policies and actual implementation of these
policies.
Data assumptions should be based on the collection of reliable data and correct statistical
analysis of the data.data were collected on:
In this phase of validation process the model is viewed as input –output transformation.
That is, the model accepts the values of input parameters and transforms these inputs
into output measure of performance. It is this correspondence that is being validated.
Instead of validating the model input-output transformation by predicting the future ,the
modeler may use past historical data which has been served for validation purposes, if
one set has been used to develop calibrate the model, its recommended that a separate
data test be used as final validation test.
Thus accurate “ prediction of the past” may replace prediction of the future for purpose
of validating the future.
A necessary condition for input-output transformation is that some version of the
system under study exists so that the system data under at least one set of input
condition can be collected to compare to model prediction.
If the system is in planning stage and no system operating data can be collected,
complete input-output validation is not possible.
Validation increases modeler’s confidence that the model of existing system is accurate.
Changes in the computerized representation of the system, ranging from
relatively minor to relatively major include :
If the change to the computerized representation of the system is minor such as in items
one or two these change can be carefully verified and output from new model can be
accepted with considerable confidence.
• Example: One drive-in window serviced by one teller, only one or two transactions are
allowed.
– Data collection: 90 customers during 11 am to 1 pm.
• Observed service times {Si, i = 1,2, …, 90}.
• Observed interarrival times {Ai, i = 1,2, …, 90}.
– Data analysis let to the conclusion that:
• Interarrival times: exponentially distributed with rate l = 45
• Service times: N(1.1, 0.22)
•
The Black Box [Bank Example: Validate I-O Transformation]
• A model was developed in close consultation with bank management and employees
• Model assumptions were validated
• Resulting model is now viewed as a “black box”:
Input Variables
Possion arrivals
l = 45/hr: X11, X12, …
Services times,
N(D2, 0.22): X21, X22, …
D1 = 1 (one teller)
D2 = 1.1 min
(mean service time)
D3 = 1 (one line)
Model
“black box”
f(X,D) = Y
• Confidence interval testing: evaluate whether the simulation and the real system are close
enough.
• If Y is the simulation output, and m = E(Y), the confidence interval (C.I.) for µ is:
Thus,in the bank model, artificial input data {X\n, X2n, n = 1,2, , .} for inter arrival and service
times were generated and replicates of the output data Y2 were compared to what was
observed in the real system
An alternative to generating input data is to use the actual historical record, {An, Sn, n =
1,2,...}, to drive simulation model and then to compare model output to system data.
To implement this technique for the bank model, the data Ai, A2,..., S1 S2 would have to
be entered into the model into arrays, or stored on a file to be read as the need arose.
To conduct a validation test using historical input data, it is important that all input data
(An, Sn,...) and all the system response data, such as average delay(Z2), be collected
during the same time period.
Otherwise, comparison of model responses to system responses, such as the comparison
of average delay in the model (Y2) to that in the system (Z2), could be misleading.
responses (Y2 and 22) depend on the inputs (An and Sn) as well as on the structure of the
system, or model.
Implementation of this technique could be difficult for a large system because of the need
for simultaneous data collection of all input variables and those response variables of
primary interest.
Input-Output Validation: Using a Turing Test
For example, suppose that five reports of system performance over five different days are
prepared, and simulation output are used to produce five "fake" reports. The 10 reports
should all be in exactly in the same format and should contain information of the type
that manager and engineer have previously seen on the system.
The ten reports are randomly shuffled and given to the engineers, who is asked to decide
which report are fake and which are real.
If engineer identifies substantial number of fake reports the model builder questions the
engineer and uses the information gained to improve the model.
If the engineer cannot distinguish between fake and real reports with any consistency, the
modeler will conclude that this test provides no evidence of model inadequacy .