Propensity Score Weighting with Double Samples:

A Simulation Study

Boris Lorenc
Stockholm University
Department of Statistics
SE-106 91 Stockholm
E-mail: [email protected]

Report nr. 22 from the project “Modern statistical survey methods”

January 2004

Keywords: double samples procedure, propensity score weighting, web surveys

AMS 2000 subject classification: 62D05

Abstract: The propensity score adjustment technique for web surveys, introduced by Terhanian, is
theoretically known to reduce bias caused by nonrepresentativeness of web panel respondents with
respect to the general population, on provision that the assumptions that pertain to the technique hold.
In practical applications though, the technique implies making choices whose implications for the
weighted point and variance estimates are too complex to express in analytic terms.

The propensity for participation in a web survey is not known and thus needs to be estimated. Choice
of the model and estimation of the model parameters introduce uncertainties that do not exist in a
purely theoretical analysis. Further, covariance structure of the variables chosen to be analysed cannot
easily be estimated as the study variables are not observed in the unrestricted sample. Even
characteristics specific to the propensity score technique, like number of strata or ratio of the sample
sizes, may play a role when making inference from a finite set of data. Finally, the assumptions
required by the technique may or may not actually be fulfilled.

As analytic expressions relating these factors to the point estimates and the estimates of their variance
are lacking, a simulation was run to study their effects. A known probability distribution was used to
generate artificial populations whose parameters were then estimated while varying the factors under
study in a setup of a designed experiment. The results in general confirmed the bias reducing effect of
the propensity score weighting. They also indicated that, under a particular covariance structure, the
adjustment could actually increase the estimator's bias if a relevant variable was left out of the model
by which the propensity score was estimated.

The support from the Bank of Sweden Tercentenary Foundation (Grant no 2000-5063) is gratefully
acknowledged.
 Propensity Score Weighting with
Double Samples: A Simulation Study
Boris Lorenc
Department of Statistics, Stockholm University
SE-106 91 Stockholm, Sweden
E-mail: [email protected]

     

                   ! " # # ! $ % " & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ = A B C @ ; D

E F G H F G I H J K E L M N L K H O F P P Q R S T U S V T W X Y Z [ X \ ] ^ _ [ ^ Z _ X Y \ ` S T S W X a W X _ X S V ^ V ] b X S X _ _ T c

U X \ a ^ S X d W X _ a T S Y X S V _ U ] V e W X _ a X [ V V T V e X f g h g i j k l m l n k j o p m h q m h l i m r p s p m h o t j o o t g

j s s n u l o p m h s o t j o l g i o j p h o m o t g o g v t h p w n g t m k x y z { | } ~  €   ~ ‚ ~ | | ‚   ~ €  ƒ { „ € ƒ † ‡ ˆ

€ ‰ € ‰  {  Š † ‰  ‹ | ‚  ‰ „ ‹ ~ Œ  { ‡  ƒ   ‰ „  ƒ „ ‰  ‹ | ‚   ~ €  ƒ { „ Ž ƒ } € ‰  ‰  ‡ € ‰  | ƒ  { € ~ { 

~ }  ~ {  ‰ ‰ „ €  ‹ ~ € ‰ „ ~ } ‰ € ƒ ƒ  ƒ ‹ | ‚ ‰ ‘ ’ “ ” ‘ • – ” — — ˜ ™ š ™ š › œ ’ ˜  ’ ” – ž — Ÿ

” • – “ • ” ™ — ˜ ’ œ ¢ “ – • š – ’ ˜  ˜ • š ’ ˜ “ ™ ˜ ™ š £ ” ¤ — ¥ – ¦ ” œ ˜ — ™ “ ’ § ™ “ £ ™ š ™ ¨ ’ ¥ — ™ ” ” ¨ —

¡ ¡

’ “ ¤ ” ” — ’ ˜ ž š ’ ” ¨ Ÿ © ª « ¬ ­ ® « ¯ ° ª ® ± « ² ® ³ ´ µ ² ® ¶ ° ¬ ± ´ ° ¬ « µ « ¯ ° ª ® ± « ² ® ³ · ´ ¸ ´ ± ® ° ® ¸ ¶ ¬ µ ¹

° ¸ « ² º ­ ® º µ ­ ® ¸ ° ´ ¬ µ ° ¬ ® ¶ ° ª ´ ° ² « µ « ° ® » ¬ ¶ ° ¬ µ ´ · º ¸ ® ³ ¼ ° ª ® « ¸ ® ° ¬ ­ ´ ³ ´ µ ´ ³ ¼ ¶ ¬ ¶ ½ ¾ ¿ À Á Â Ã À Ä

À É Ã Ë Á À ¿ Á ¿ À Ã Á Â Ã À É Ã Ë Â Ë Ã Á Ã Ë Ã Ð Á Ã Ë É Ã Ã Ë Á É Ñ

Å Æ Ç È È Ê Å Å Æ Ì Ç È È Í Î Å Æ Ê Æ Í È Ê È Î Ï Å È Ê Ê Æ È Î Ï Í

Á Ã Ð Ë Á Â Ã Ë Á ¿ Ð À É Ã Ë À Ã Á Ë Ã À Ã Ð É Á Â Ã ¿ À Ã Ë Á À É Á Ã Ð Ë Ã Ô Õ Ö × Ø

Ò È È Ï Ç È È Í Î È Ê Æ Æ Í Ç Ê Ê Å È Ò Ó Î

× ×

Ù Ú Û Ü Û Ù Ý Ü Þ ß Ý Þ Ù ß ß à Ù Þ á â ã ä ã å æ ç è ä ç æ é ê ë ã ì ê â ä è æ ã æ â å é ë í î æ ï ð ë ñ æ é î ò ó æ è ä ô ê ã è õ ã õ ä è

è õ ã ë ä ä ô ã å æ ê õ ò ç ð æ ê ë ö ÷ ø ù ú û ü ý þ û ü û ÿ þ ÷
 ÷  ú û        ÷ ÿ ÷   ÷  ÿ ú û   ÷ ø ÷ 

û û   û þ þ ü ù  ÷ û ø ø  ú ý   ø ÿ ÷    ÿ ÷ ü  ÷ ÷       ÷ ú û ü ÿ ú û ü  û   û þ þ ü

÷   þ þ þ ÷

                                 !                    "       #  

    "      !     $              %    &    " '       (    '      ' #       ) * + , - .

/ 0 1 2 3 1 4 5 2 6 7 6 8 9 8 , : ; 8 - , 5 8 6 < , 8 2 1 3 7 - < - * ; , 2 = * 1 * 5 7 , * 7 5 , 8 > + 8 7 9 4 2 4 < 9 7 , 8 2 1 - 3 ? 2 - *

4 7 5 7 @ * , * 5 - 3 * 5 * , ? * 1 * - , 8 @ 7 , * ; 3 ? 8 9 * A 7 5 : 8 1 = , ? * B 7 + , 2 5 - < 1 ; * 5 - , < ; : 8 1 7 - * , < 4

2 B 7 ; * - 8 = 1 * ; * C 4 * 5 8 @ * 1 , . D ? * 5 * - < 9 , - 8 1 = * 1 * 5 7 9 + 2 1 > 5 @ * ; , ? * 6 8 7 - 5 * ; < + 8 1 = * ) * + ,

2 B , ? * 4 5 2 4 * 1 - 8 , : - + 2 5 * 3 * 8 = ? , 8 1 = . D ? * : 7 9 - 2 8 1 ; 8 + 7 , * ; , ? 7 , E < 1 ; * 5 7 4 7 5 , 8 + < 9 7 5

+ 2 A 7 5 8 7 1 + * - , 5 < + , < 5 * E , ? * 7 ; F < - , @ * 1 , + 2 < 9 ; 7 + , < 7 9 9 : 8 1 + 5 * 7 - * , ? * * - , 8 @ 7 , 2 5 G H I J K H

J L K M N O N P K Q R P K M J K I O N S K H O N L R T U R T L R V N W T X N O I Y S V J Z V R V N [ M T [ N Q H J R Y H Z T M N S K H

N H R J W K R N X \

Introduction
To reduce the bias of the estimates of population parameters that arises when web panels
are surveyed instead of random samples from the population, Terhanian (e.g. Terhanian,
Marcus, Bremer, and Smith, 2001) suggested a new application of the propensity score

1
weighting technique. The technique was originally developed by Rosenbaum and Rubin
(1983a) to obtain a proper estimate of a treatment di¤erence between two populations.
The new in Terhanian’s approach was to apply, with minor changes, this method to two
samples from the same population—the name “double samples procedure” seems thus
appropriate.
Situations where the method might be suitable arise when (i) access to and data
collection from one of the samples is much cheaper than from the other one, and (ii)
the inclusion probabilities into the former (cheaper to collect data from) sample are not
explicitly known. It is often, with respect to ii, further suspected that the unknown
inclusion probabilities are related to auxiliary information or to variables under study,
that is, this sample is presumably gathered from some speci c subset of the general pop-
ulation. Additionally, self-selection into this sample might be present. (The researcher,
when modelling this situation, may focus on estimating the inclusion probabilities or on
estimating the di¤erence in the distribution of the variables in the population and in the
subset. The propensity score technique is geared towards the latter goal.)
In Terhanian’s practical application of the technique, a web panel takes the role of
the restricted sample as it consists of web users presumably di¤erent from the general
population on important properties like age, income, and the like. In order to produce
appropriate weights, web panel data are augmented with incomplete data (only the aux-
iliary variables are collected) from an unrestricted sample—a sample from the general
population with the element inclusion probabilities known.
Terhanian and the colleagues reported some very accurate predictions of elections
outcomes obtained by the propensity score adjustment technique (Terhanian, Marcus, et
al., 2001, Terhanian, Taylor, Siegel, Bremer, and Smith, 2001). Their presentations were,
unfortunately, given in descriptive and rather vague terms, with formal expressions and
technical details omitted. In a study related to this paper (Lorenc, 2003), the present
author gave a simple analytic demonstration of the e¤ectiveness of the propensity score
weighting in a double samples setting. Akin to the study of Cochran (1968) who investi-
gated the case of sampling from two populations, this new study showed that Terhanian’s
method performed as indicated by its creator, reducing the bias of the unadjusted estima-
tor by about 90% in the situations studied. The technique gives theoretically an unbiased
estimator, but in practical applications the necessity to estimate the parameters from data
hinders the accomplishment of this goal.
The technique may be said to consist of these steps:
1. collecting complete data—the auxiliary variables and the variables under study—
from a web panel (the restricted sample) and collecting the auxiliary variables from
a random sample of the population (the unrestricted sample),
2. given the whole set of auxiliary information (from the unrestricted and the restricted
samples) but not the sample membership indicator, estimating for each unit the
probability of being a panel member (this magnitude is known as the estimated
propensity score); a common way of estimating this probability is by building a
logistic regression model,

                               ! " # $ % & ' ( ) * + ) , - . / , - 0 1 2 3 4 5 6 7 8 9 : ; < = > ? = @ A B C D E B F G

E B H I J K K L M N O N O P Q K L K I J R S T T S U V W X Y Z [ \ ] ^ _ ` a _ ` b c d e f d g h i j k l m n o p q l r s p t l u v w x y z y v x u { v z | } ~  €   ‚ ƒ „ † ‡ ˆ ‰ ‡

‡ ‰ ‰ † † “ ” • – — ˜ ™ ˜ • – ˜ – š › † ‰ † œ “ 

Š ‹ Œ Œ  Ž Ž  Œ Š   ‘ Š Œ  Š ’  Œ  ‘ Œ

2
 3. estimating the distribution of the propensity score in the population by considering
the distribution of the estimated propensity score in the unrestricted sample only;
in particular, identifying cuto¤ points for strati cation: usually equidistant cuto¤
points are chosen and 5 intervals are used, in which case the cuto¤ points would be
the 20 , 40 , 60 , and 80 percentile of the estimated propensity score distribution

     

in the population,
4. classifying the units in the restricted sample (panel) into appropriate strata based
on their individual estimated propensity score values,
5. for each stratum, building a mean of the study variable values of the panelists in that
stratum; then, weighting the strata means appropriately together to produce the
nal, adjusted estimate for that study variable; in the case of equidistant intervals
the weighting amounts to calculating the arithmetic mean of the strata means.

Justi cation for the procedure and its details were given in (Lorenc, 2003).
Despite its theoretical clarity, the method nevertheless leaves some questions open. For
one, the propensity score is in real applications not known and thus needs to be estimated.
This uncertainty comes atop of the usual one, that of the unrestricted sample being a
sample rather than a census. The most common method for estimating the propensity
score is logistic regression, but discriminant analysis is an alternative suggested in the
literature.
An expression for variance of the propensity score adjusted point estimate is lacking.
The applied variance estimate (e.g. Rosenbaum and Rubin, 1984) is conditional on the
chosen model, not taking into account the uncertainty regarding the model itself. Yet
another issue is that of choosing the correct cuto¤ points for strati cation. This is still
an open research issue.
In addition, there are other factors that might in‡uence the point estimate and the
estimate of its variance. They are: absolute and relative sample sizes, covariance structure
of the data, whether all the variables relevant to the study were observed or only a part
of them, and so on. The e¤ects of these factors are not clear because explicit expressions
for the variance estimates are lacking. In such a situation, an alternative way to address
these issues is through a simulation study. This path was chosen for the present study.
The present study is a simulation study that investigates the e¤ect of the above men-
tioned factors on performance of the propensity score adjusted estimator in the situation
of taking double samples from the same population. The study is performed in a manner
of a designed experiment. In Section 1, goals of the study and motivation for investigat-
ing the chosen factors are presented. Details of the method are provided in Section 2,
while Section 3 gives the results. Some general conclusions arising from the study and
comments are given in the nal Section 4.

1 Goal and studied factors

1.1 Goal
The present study investigates the in‡uence of a number of factors on the e¤ectiveness—
here understood as the ability to reduce bias—of the propensity score technique under

3
certain conditions (given in more detail below), with the main aim of preparation for the
future analyses of real data. The setup of the study is of an experiment with a number
of factors.

1.2 Studied factors

1.2.1 Preliminaries
Let X denote the auxiliary variables, let Y be the variable under study, and let Z be an
indicator of inclusion into a subset (to be speci ed) of the population. Let two samples
be drawn by the simple random sampling mechanism: an unrestricted sample, s, of
size n from the population, and a restricted sample, r, of size k from the subset of the
population.
The propensity score, denoted e (x), is de ned as e (x) = Pr (Z = 1jX = x). Let the
following two conditions be ful lled:

² a positive probability at every level of the propensity score for every unit in the
population to be assigned to any of the samples , 0 < e (x) < 1, and

² independence of the study variable and the subset membership indicator conditional
on the auxiliary information , Y ? Z j X .

The two assumptions are jointly referred to as strongly ignorable treatment assignment
(SITA). Then, as proved by Rosenbaum and Rubin (1983a),

² conditioning on the propensity score establishes a (conditional) independence of the

auxiliary information and the subset membership indicator, X ? Z j e (X)—which
is sometimes referred to as balancing—and,

² conditioning on the propensity score removes, in expectation, the bias of the un-
adjusted mean of the study variable between the population and the subset ,


E fY j e (X) ; Z = 1g = E fY j e (X) ; Z = 0g.

The propensity score is in practice not known and must be estimated, all relevant
auxiliary information might have been gathered or some of it left out, and the assumptions
may hold or may not hold. These, and some further issues, are now discussed in more
detail.
                                   ! " # $ % & ' ( % $ ) * ! $ ! # " " + , - . / 0 1 2 3 4 5 4 5 6 4 7 1 8 2 9 : 5 ; <

2 = > 8 4 9 7 4 1 4 = > 1 4 > > 2 = ? @ A B @ 0 6 / 4 1 ; 5 9 / 4 1 A 7 @ 5 > 8 4 1 ; 5 4 9 @ 9 0 / ; > 2 @ = 6 4 7 4 / ; - 4 B > @ C D E F G H I J K L M L N O P

L L T U P V W X Y Z [ \ ] Y ^ X _ Y ` a ^ _ \ b \ c \ _ Y Z ^ d d Y X b b ^ e f d X Y X g \ e ] f h ^ h X b X Y ` Y f ^ e e \ ^ ] X _ i j k l m n n o k p n q r p s t

Q R R S

k l u v o p s w u x m p w u v u s k y q o z y n p k p { u | l o s | u q y w u o | l u } u v u s k k y o z z u o w p s o s r y q k l u n o v z } u n ~

 €  ‚ ƒ „ † ‡ † ƒ „ ˆ ‰ ƒ „ ˆ „ Š ‹ ˆ ‰ ˆ „ ‚ † „ Œ  ‚ ƒ  Ž  ƒ  ‘ ’ † ‚ ’ ‡ ’ Ž “  ƒ “ Ž „ †  † ‡ Š  ‚ ƒ  Ž †  ‡ ’ Ž ‚ ƒ ˆ   Ž  ‡ ” • – ’ Ž “  ƒ ƒ  † 

Œ † — Ž „ † „ ˜ ™ š › œ  ž Ÿ ž œ ¡ ˜ Ÿ  ¢ œ £ ¤ ¥ ¦ § ¨ © ¨ ª « ¬ ­ ® ¯ ° ° ± ° ± ² ¬ ­ ° ¯ ­ ³ § ´ µ

¶ · ¸ ¹ º » ¼ ½ ¾ ¿ À Á Â ½ ¾ ¿ Â Ã º Ä ¾ ¹ » Ã ¾ ¿ Å Æ º Ã Ç È É ¿ Ç º È º ½ ¹ É Ç É ¿ À ¹ ¹ É Ç É º ¸ À ½ ½ Ê Ã ¾ Ë » É Ã ¾ ¹ Ç È À Ç Ç È ¾ Ã ¾ É ¿ ¸ º ¾ Ì ¾ Ä Ç º Æ

¹ À Ç À Ä º ½ ½ ¾ Ä Ç É º ¸ Á ¾ Ç È º ¹ º ¸ Á ¾ À ¿ » Ã ¾ Á ¾ ¸ Ç Í ¿ ¾ ¾ Î Ï Ð Ñ Ò Ó Ô Õ Ö Ö × Ø Ù

4
1.2.2 Estimation of the propensity score
Methods
The propensity score is in practice most often estimated by the use of a logistic regres-
sion with the logit link. Instead of the logit link, other links like the probit link might be
considered, but in the present study, with a relatively simple population model used and
a simple subpopulation inclusion rule (cf. Section 2) these give the same results, so this
comparison was omitted.
Instead, discriminant analysis was used as an alternative for estimating the propensity
score. Rosenbaum and Rubin (1983a) point out though that with multivariate normal
distributions having common covariance in both treatment groups (which is the case in
the present study), the propensity score is a monotone function of the discriminant score,
why the eventual di¤erence between the results of applying the two methods would stem
from the estimation algorithms rather than from theoretical di¤erences.

Available amount of data

With limited amount of data on which to base the estimation, a problem of insu¢-
cient overlap of the distributions of the propensity score in the two samples may arise.
Practically, it may happen that, when classi ed in step 4 of the procedure outlined in the
Introduction, no unit in the restricted sample falls into one of the strata. No contribution
to the adjusted average response can thus come from this stratum, with the consequence
that if an adjusted estimate is produced, it will then be biased. Occurrence of this is
clearly related to the sample sizes, why the sample size factor was included in the study.
Theoretically, the larger the number of strata into which the sample is strati ed based
on the estimated propensity score, the lesser the bias of the adjusted estimate (Section 2
of Lorenc (2003) gives the details). In practical situations with a limited amount of data,
though, a ner-grained strati cation yields greater risk for empty strata which in turn
might lead to a biased estimate. Coarser strati cation lowers the risk for empty strata
but might preserve a larger intrinsic bias compared to a ner-grained strati cation. The
number of strata was thus varied in the study.
Additionally, proportions in which the two samples partake in the joint sample might
be of signi cance to the propensity score estimation methods. This was also included in
the study, as a factor.

Variance estimation
While conditionally on the regression model variance estimation of the propensity score
adjusted estimate is not di¢cult to obtain (see p. 10 under heading “Procedure”, be-
low), the model selection process includes sources of variance that are harder to express
analytically. A simulation study provided the opportunity to estimate the variance of
the propensity score adjusted estimate without access to analytic expressions, by simply
obtaining the adjusted point estimates in each repeated drawing of the population and
of the samples, and then calculating the variance of these values and comparing them to
the variances calculated conditional on the regression model.

5
1.2.3 Assumption violations
Treating the same problem as in Cochran (1968) and Cochran and Rubin (1973), namely
that of estimating treatment e¤ects, Rosenbaum and Rubin (1983a) showed in a situation
with multivariate covariates that conditioning the response on the propensity score (or
on any balancing score, of which the propensity score is the coarsest) eliminates bias due
to nonrandom treatment assignment provided that the assumptions given in SITA hold.
But, what would the consequences be if these assumptions did not hold? While
analytical results might be derived in simpler cases, there was an interest in checking the
behaviour of the estimator in more complex situations that in a sense resemble those that
might be met in real applications.

SITA violations
SITA assumptions might be violated in two ways: some units may have a zero chance
of being included in the restricted sample, 9i : e (x ) = 0 where i indexes units in the
population, and a dependence between the study variable and the subset membership
indicator may remain even after conditioning on the auxiliary information, here symbol-
ically denoted as (Y \Z) j X. The e¤ects of both violations were investigated in the
simulation study.

Unobserved covariates
In addition, violation of the latter of the SITA assumptions occurs when an important
auxiliary variable is left unobserved. In practical applications, the di¢culty precisely it
that there is an uncertainty whether all the covariates causally related to the response,
Y , and the subset membership indicator, Z, are observed.
This factor is a variant of the preceding one, the only di¤erence being a conceptual
one: here, there was a variable that we ought to have observed but failed to do so while,
in the previous case ((Y \Z) j X), the nature of the phenomenon was such that Y and Z
are tangled and could not be untangled by conditioning.
When investigating sensitivity of the propensity score technique to not recording a
covariate, Rosenbaum and Rubin (1983b) considered an unobserved binary covariate. In
the general multivariate normal setup of the present study (see subsection “The general
setup” in next section), it seemed natural to investigate the e¤ect of not including a
continuous covariate, while varying the degree of correlation between this covariate and
the other variables.

2 Method
Point of departure for the double samples application of the propensity score technique,
as noted in Introduction, is that the values of the study variable Y are not observed for
the units in the unrestricted sample from the population, s. So, given the information
available in the two samples—s with only auxiliary variables and the restricted sample
from the subset, r, with the complete information—the goal was to estimate the mean of
the study variable Y for the whole population by applying the propensity score weighting
technique. For the model presented in (1), p.7 below, this mean of Y in the population
is zero.

6
 The factors from the preceding section were included in a simulation experiment
which consisted of repeatedly drawing s and r samples from populations with known
characteristics de ned by levels of the factors pertaining to the population properties
and the samples’ properties, and producing point and variance estimates based on levels
of the factors pertaining to the estimator properties.
A summary of the factors is given after a presentation of the general setup.

2.1 The general setup

The following multivariate normal model was used:

(X ; X ; Y; V ) » N (0; §) ,

where
2 3
1 ½ ½ ½
6 ½ 1 ½ ½ 7
§=6 7.
     

4 ½ (1)
½ 1 ½ 5

  

½ ½ ½ 1

 

  

This model de ned a nite population, U = f1; 2; ; :::; i; :::; N g, while inclusion into the
subset was de ned through either I or I . The variables in the model were
given the following meanings, not uncommon in the survey literature:
           

X , an auxiliary variable,
X , another auxiliary variable, also involved in de ning


the subset—“the participation variable”,



Y , the study variable,

V , another variable involved in de ning the subset.

The present model di¤ered from the one studied in (Lorenc, 2003) in that here there
are two auxiliary variables instead of one, each with its speci c correlation coe¢cients
with the other variables, and also that V may here even have a nonzero correlation with
the other variables. In order not to in‡ate the number of factors in the experiment, V ’s
correlation coe¢cient with the other variables was kept the same, ½ , across the variables.
Setting V aside for the moment, the covariance matrix in (1) produces 8 di¤erent
 

models when each of ½ , ½ , and ½ is held on one of the two positive levels, “high” and
“low”. Varying the covariance structure in this way gave the opportunity to investigate
   

the e¢ciency of the propensity score adjustment under the “high” and “low” levels of
correlation between each of the covariates and the response (Table 1).
The values of “low” and “high” for the ½’s in this reduced, 3 £ 3, covariance matrix
were set to :22 and :78, respectively—in any combination producing a positive de nite
matrix, as required by the model. The value of ½ was set to :175, producing with both
signs (¡ and +) positive de nite matrices in all the 8 models. Three of the matrices,
! "

those pertaining to the structures 5, 6, and 7, became thereby almost singular (i.e., the
third variable an almost deterministic function of the other two).

7
 Table 1: Denotations for the covariance structures investigated in the study.
Covariance structure ½ ½ ½
1 low low low




2 low low high

3 low high low
4 high low low
5 low high high
6 high low high
7 high high low
8 high high high

Table 2: Expected values of the study variable Y in the subset for the chosen values of
½ and ½ , which also are the biases of the corresponding sample means of Y
in r if they were to estimate the expected value of Y in the population (the

  

latter being zero).

½ = :22 ½ = :78
Z =I Z=I Z =I Z=I




½ = ¡:175 :206 :244 :497 :719

       
           

½ =0 :124 :201 :440 :708

 

½ = :175 :028 :141 :376 :692

 

 

Assignment to the subset was set either by Z = I , conforming to the SITA

assumption that 8i 2 U : 0 < e (x ) 6 1, or by Z = I , violating this assumption
   

through 9i 2 U : x < 0 ¡! e (x ) = 0. For this latter case the reference “SITA

          !

violation #2” is used below.

" # $ %

The coe¢cient ½ —determining the correlation between X and V —may here, in

contrast to the setup in (Lorenc, 2003), have taken on values di¤erent from 0. When
& ' (

½ 6= 0, assignment to the subset was not strongly ignorable: the assumption (Y ? Z) j

X was violated. For this condition the reference “SITA violation #1” is used in what
& '

follows.
)

With no SITA assumption violations, the expected value of the study variable Y in
the subset was biased with respect to the expected value of Y in the population, with
the bias expressible as ½ £ ¼ = ½ £ :564 (ibid.), giving for the case ½ = :22 the
,

bias of :124, and for the case ½ = :78 the bias of :440. The expected values of Y in the
+ . / . /

) *

subset, both with and without the SITA violations, are given in Table 2.
. /

Two samples were drawn using simple random sampling, one from the complete pop-
ulation, denoted by s and of size n, and another from the subset, denoted by r and of
size k.
The variable X , which took part in de ning the subset from which the restricted sam-
ple r was drawn (“the web users”), is sometimes referred to as “the participation variable”
.

meaning participation in the restricted sample, the only sample providing information on
Y.

8
2.2 Summary of the studied factors
The following factors were included in the study:

1. Covariance structure [denoted covstr in the Tables and Figures]: 8 levels (the 8
models presented in Table 1),

2. Sample sizes [ssize]: 2 levels (“low”, n = 1000, and “high”, n = 5000, for the
sample s),

    

3. Ratio of k, the size of the sample r, to n, the size of the sample s [knratio]: 3
levels ( , , and , giving the restricted sample’s sizes k = f500; 1000; 1500g
 

for the “ssize low” condition and k = f2500; 5000; 7500g for the “ssize high”
  

condition ),
 

4. Method [method]: 2 levels (logistic regression with logit link and discriminant
analysis),

5. Observed covariates [observed]: 4 levels (an “analytic” level (a), where the known
propensity score is used (see the heading “Recorded statistics”, below), and three
levels where the propensity score is estimated after only X is observed, only X is
observed, and both covariates are observed and forced into the model),
 

6. Number of strata into which the empirical distribution of the propensity score is
strati ed [nstrata]: 2 levels (5 and 7 strata),

7. SITA violation #1 [sitavio1]: 3 levels (“yes, negative correlation”, ½       

= ¡:175, “no”, ½      = 0, and “yes, positive correlation”, ½   ! " # $ = :175),

8. SITA violation #2 [sitavio2]: 2 levels (“no”, 8i : 0 < e (x ) < 1, and “yes”,

9i : e (x ) = 0),
%

2.3 Procedure
For each of the level combinations of the all the factors except method and observed,
b = 1000 independent trials were run, where a trial consisted of generating a simulated
population of size N = 50000 with the required properties, taking an unrestricted sample
s and a restricted sample r, and calculating the required statistics (see next subsection)
from them. As comparisons between the two propensity score estimation techniques, and
between the e¤ects of observing di¤ering amount of information, were of interest, the
required statistics for the levels of the factors method and observed were calculated
on the same sets of data.
For every point estimate, a corresponding estimate of its variance was calculated. For
this, the method of Mosteller and Tukey was used. This particular formulation, “the
method of Mosteller and Tukey”, is the consequence of Rosenbaum and Rubin’s (1984)
special mention of it; they namely say: “standard errors for the adjusted proportions
were calculated following Mosteller and Tukey (1977, Chap. 11c).”
Chapter 11c of Mosteller and Tukey (1977) considers primarily the choice of the
standard population when comparing samples from two populations. All the examples in

9
11c concern proportions, and the usual variance estimators for the estimated proportions,
p^, are applied within each stratum l, V^ (^p ) = p^ (1 ¡ p^ ), the only speci c issue being

the weight given to each stratum’s variance in building the overall variance.
 

 

In their example in the reference above, Rosenbaum and Rubin seem to give equal
weights to each stratum; this could not be fully con rmed though, as using the data
supplied in their Table 1 did not give an exact replicate of their reported standard errors
(equal weights gave though the values closest to those in the article, about 0:01 below, of
the several considered weightings; the di¤erence seems though to be too large for a result
of rounding errors, why this remains an open issue).
In the present application, each stratum’s standard deviation was given equal weight
as the strata cuto¤ points were equidistant:
³ ¡ ¢ ´ X µN ¶ s Xµ N¶ s 1 Xs
\
V^ E Y¹
  

= = = , (2)





    

N k N k L k
  

 

  

  

where k denotes the number of units in r falling into stratum l of the quantized distri-
bution of the estimated propensity score in the population, and s denotes the variance


of the observed values on Y for these units.



The simulation was performed in Matlab.

2.4 Recorded statistics

The following statistics were recorded for every simulated population:

1. the propensity score adjusted estimates of the population mean for Y based on the
restricted sample r:

² one estimate based on the true propensity score, Yb¹ , (i.e., it was taken as
known that e (X) = © (X ), the cumulative distribution function of X , as in
     

the demonstration example of (Lorenc, 2003), subsection “Reducing the bias

 

of Y¹ as an estimator of E (Y ) by strati cation on the propensity score”) ; the

cuto¤ points were estimated from the empirical distribution of the propensity
!

score in the unrestricted sample s rather than found in the table of © (¢); this
estimate corresponds to the level a of the factor observed,
² three estimates based on the estimated propensity score obtained by the cur-
rent method (either logistic regression or discriminant analysis), where the
information observed and available for estimation of the propensity score var-
ied: it could be only X , which gave the point estimator Yb ¹
f g , only X ,
which gave Yb
& ' ( )

" *

¹
g , and (X , X ) forcing both variables in the model, which
# $ %

f
. / ( 0

gave Yb
1 *

¹
g ; the cuto¤ points were estimated from the empirical distri-
+ , -

f
bution of PcS = e[
. / ( ) ( 0

(X) in the unrestricted sample s, with X replaced by the

-
+ ,

currently observed X-values (X , X , or X and X ).

2

3 4 3 4

5 6 7 8 9 : ; < = : > ? @ A B C D E F G H I J K L M L N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b a c d e f g h i j g k j i l m n f o m p k g i q r s t u v w x y z

{ | } } ~  € {  | ‚ ƒ „ | † ~ † ƒ | ‚ € ‡ ˆ ‰ } € Š ‹

10
 2. the bias of the estimates in item 1 with respect to ¹ , the mean of the variable Y
in the current population,

3. variance estimates for the estimates in item 1 calculated using (2),

4. whether the population mean of the current population, ¹ , was contained in the
estimated nominal 95% con dence interval built around the point estimates in item
1 by subtracting and adding 1:96 times the root of the corresponding variance
estimates in item 3.

As b = 1000 trials were performed at each combination of the factor levels, some
summary statistics could be generated:

I. mean bias across the b trials (MeanBias in the reported Figures and Tables),

II. empirical con dence level: proportion of con dence interval “hits”—the mean of
the statistics in item 4 above across the b trials (Clevel),

III. di¤erence between the mean across the b trials of the root of the variance estimate
in item 3 above and the observed simulation standard deviation across the b trials
of the corresponding point estimate in item 1 above (StDi¤ ).

The statistics I-III are presented as results in the next section. The statistics were
analysed using the design-of-experiments module in the statistical package Minitab.
When percentage reduction in bias for the compound simulation statistics MeanBias
is presented, it was calculated using
0 ¯ P ¯1
³ ´ ¯ ^µ ¯
¯ ¡ µ¯


prb ^µ


= 100 @1 ¡ ¯ ¯ A,


 

¯^ ¯
¯µ ¡ µ ¯

 

where ^µ , ^µ , and µ are the estimator adjusted using the technique f¢g, the unadjusted
estimator (based on the r sample only), and the parameter they aim to estimate, respec-
   

tively. Thus, prb was calculated from the summary data, and not for each generated
population separately.

3 Results
The results of the simulation are presented in tabular and graphical form. The main
table of results consists of percentages reduction in bias and empirical con dence levels
of the propensity score weighted estimator under conformance and the deviations from


the assumptions. Second-order interaction plots of the studied factors are added with the
aim to give the reader an impression about the individual contributions of the studied
factors on the simulation statistics, as well as about the contributions of their interactions.
Two additional kinds of tables, containing more detailed information, also exist: ANOVA
tables for each of the summary statistics, up to second order e¤ects, and tables of means
            !

11
 Table 3: The eight cases of assumption violations.
Case Only X observed ½ 6= 0 Z = I
0 no no no

      

1 yes no no
2 no yes no
3 no no yes
4 yes yes no
5 yes no yes
6 no yes yes
7 yes yes yes

of the rst and second order e¤ects, across all the levels partaking in the current analysis.
These two kinds of Tables—too large and detailed to constitute a part of the text—are
given in the Appendix.
The factors that proved to have a dominating e¤ect on the observed simulation statis-
tics were those related to violations of the assumptions for the propensity score technique.
In order to give a clear picture of the contributions of all the factors investigated, rst
presented is the case where all the assumptions held, followed by the cases where they
were violated in various ways. There are 8 such cases all in all (including the one where
all the assumptions held), as the Table 3 illustrates. The results are presented in this
order.
Within cases, the results are presented rst for the point estimation (i.e., the sim-
ulation statistic MeanBias), followed by those regarding variance estimation (i.e., the
statistics StDi¤ and Clevel). But, as variance estimates are of little use for the produc-
tion of correct con dence intervals if the point estimates are seriously biased, which in
general turned out to be the case when the SITA assumptions did not hold, the variance
estimates are not always presented for such cases (but can always be found in the Tables
in the Appendix).

3.1 Case 0: SITA assumptions hold

Besides the two factors explicitly named so, sitavio1 and sitavio2, even not observing an
important covariate constitutes a violation of a SITA assumption. Speci cally, observing
only X would amount to observing incomplete information as it does not hod that
(Y ? Z) j X . So, in presenting here the results where the SITA assumptions held,


the estimates obtained using the true propensity score, the information in X , and the


information in X and X together are included. Thus, the e¤ects of the factors covstr,


ssize, knratio, method, nstrata, and observed, excluding observed=x1, are

 

analysed here.

3.1.1 Case 0: Point estimation

When SITA assumptions held, major factors that in‡uenced the remaining bias in the
propensity score adjusted point estimates of the mean of Y in the population were cov-
str and nstrata (Figure 1). Inspection of the ANOVA table (Table I in the Appendix)

12
Figure 1: Case 0, interaction plot for MeanBias. (Identi cation of the covstr levels
given in the text.)

showed that all the main e¤ects except method had a statistically signi cant contribu-
tion (p 6 :001), but that the dominating ones were those two mentioned. In addition,
all the two-way interactions involving covstr except that with method plus the in-
teraction knratio£nstrata were the signi cant ones among the two-way interactions
(p 6 :005).
The covariance structures whose point estimates had the higher mean level of the bias
in the rst row of plates in Figure 1 (about :044 on the average) were those denoted by
the numbers 2, 5, 6, and 8, that is, those where the correlation between the participation
variable and the study variable was high (i.e. ½ = :78). This con rmed the analytically
derived values of the adjusted point estimates for the same situation (Lorenc, 2003): the

values of the bias of these point estimates falling under “5” and “7” in the pane for
the interaction covstr£nstrata in Figure 1—the means across the structures 2, 5, 6,
and 8—were .0537 and .0352 respectively, deviating only in the last decimal from the
theoretically derived ones, .0532 and .0351 (also, see Table IV in the Appendix).

3.1.2 Case 0: Variance estimation

When the point estimates were based on the true propensity scores, that is, when the
estimator Yb
¹ was used, the corresponding variance estimates, calculated applying the
suggested method (”the method of Mosteller and Tukey”), were approximately correct
     

irrespective of the covariance structure (Figure 2; the estimator Yb

¹ is represented
as method=a). But, when the point estimates were based on the estimated propensity
     

13
Figure 2: Case 0, interaction plot for StDi¤. (Identi cation of the covstr levels given
in the text.)

scores instead, resulting in the estimators Yb

¹
f


g , the accuracy of the variance estimates



 

depended on the covariance structure: less underestimating or approximately correct were

those where the original bias was low (that is, those with the low correlation between
X and Y , denoted by 1, 3, 4, and 7), while more underestimating were those where the
original bias was high (that is, those where the correlation between X and Y was high,


denoted by 2, 5, 6, and 8).



But, that the variance estimates of the point estimates were eventually correct did not
necessarily result in correct con dence intervals, those where the parameter of interest
would be found in the interval in a prespeci ed proportion of trials. The reason for this
could be found in the biasedness of the point estimators, where intervals of the eventually
correct lengths were built around wrong point estimates, thus performing below the
required coverage level. Thus, for the estimator Yb ¹ , whose variance was estimated
approximately correctly irrespective of the covariance structure, the attained con dence


level was nevertheless much below the required one when the correlation between the
participation variable X and the study variable Y was high (i.e., the covariance structures
2, 5, 6, and 8, that gave rise to the more biased point estimates—Figure 3).


It can be noted that the levels corresponding to the smaller amounts of data (ssize=1000
and knratio= ) led to con dence intervals with somewhat better coverages than those
with the larger amounts of data.


14
Figure 3: Case 0, interaction plot for Clevel. (Identi cation of the covstr levels given
in the text.)

3.2 Case 1: SITA violated, not all the relevant information ob-
served
The results presented thus far had not included the estimates that were based on just the
auxiliary variable X being observed (i.e., observed=x1). Observing only X would
amount to observing incomplete information: the participation variable X is required

instead in order to ‘explain’ Z. Only insofar as X and X would be correlated would



the observation of X help when X ought to have been observed instead. So, here even
 

the condition observed=x1 is added into the analysis besides the 6 factors and levels
 

explored previously.

3.2.1 Case 1: Point estimation

The estimates of the mean of Y in the population based on only X were considerably
more biased than those derived using the true propensity score or when X was amongst


the variables observed (Figure 4). And, an interesting pattern arose: a comparison of the


unadjusted estimates (Y¹ , the means of Y in the restricted sample r) with those adjusted
on the estimated propensity scores after observing only X , across all the other factors, is


given in Table 4 together with the corresponding correlation coe¢cients in the covariance


matrix.
The bias of the unadjusted estimator Y¹ was a function of ½ , which is the correlation
between the participation variable X and the study variable Y . When the participation
variable X was observed, the bias was reduced to approximately 10% of its original value,

15
Figure 4: Case 1, interaction plot for MeanBias. (Identi cation of the covstr levels
given in the text and Table 4.)

as reported in the preceding section of the results.

When X was not observed but X was, adjustment for the bias could be attempted
using the information contained in the correlation between X and X (i.e., ½ ) and the

correlation between X and Y (i.e., ½ ). When information in both ½ and ½ was low,

then the adjustment attempts in general resulted in small percentage reduction in bias,





though smaller when ½ was high then when it was low (rows 1 and 2 corresponding to
the estimator Yb
¹ b
¹
g —denoted for short Y — in Table 4). When the original bias


f


  

was low (i.e., ½ was low), it took one of ½ and ½ to be high in order to produce
 

a considerable reduction in bias (rows 3 and 4 corresponding to Yb ¹ in Table 4). But,

 

when the original bias was high (i.e., ½ was high), it did not su¢ce that just one of
½ and ½ was high to produce a large reduction in bias (rows 5 and 6 corresponding
 

to Yb
¹ in Table 4). Both needed to be high, and yet the percentage reduction in bias
 

could only be moderate (row 8 corresponding to Yb ¹ in Table 4). But, when both the
information contained in ½ and in ½ was at the high level but when the original bias
was low, then the estimator “overadjusted” for the bias, actually almost doubling it (row
 

7 corresponding to Yb
¹ in Table 4).
The factor observed (including the level observed=x1) was by far the strongest


among those included (Table V in the Appendix).

16
Table 4:
            ! " # # $ % % & ' ( ) * + , + - . / 0 1 , 2 3 4 5 6 7 8 9 : ; 6 8 < = > 9 ; 8

              

; > 9 ? @ 6 9 A B > E C D F G H I J K L M L M N K O P Q O O N R S Q T U K T V W X Y Z X [ \ ] V X ^ Y X _ ` Z \ a b [ a [ c d e f g h i j k l m n o p q r s r t l u

t v m w x y z { y | y } ~  €  ‚ ƒ „ † ‡ ˆ ‰ † Š ‹ Œ  Ž   ‘ ’ “ ” ‘  ” • ‘ – ” — ˜ ™ š › œ  ž Ÿ ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® « ­ ¯ ° ® ± ­ ±

® ² ¯ ³ ´ µ « ¶ µ ² ­ ® ° ¶ ³ · ¸ ¬ « ¶ ­ ª ° ¸ ¹ ® ¬ µ ® ² ° ­ ± « ¬ º ° « º ¬ ­ ± ® ° ¬ ¸ ± ± ® » » « ¶ ­ » ­ ¹ ­ » ± ² ¸ « ¯ ­ ¼ ² µ ² ½ ® ° ® ± ­ ¾

¿ À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Ü Þ ß Ý à á â â ã ä å æ ç è é ê ë é ì í î í ï ð ñ í ò ó ô õ ö õ ÷ ø ù ú ù û ü ý þ ÿ
ý
  û          

           

                      ! " # $ % & ' % ( # ( ! ! ) * + # , % - . / 0 1 2 3 4 5 6 4 7 8 9 9 : ; < 8 = > 4 8

= : ? 6 4 @ A : A 6 B C : ; 8 D E F G H I J K I L K M I N O J K P O M L Q K R S T K N H K O T O L U O V K P W X Y Z [ \ ] ^ _ ` a b c a d e f d g ` d c h i ` j k l m n o p q m p

r s t u s v p v q q w x o p y q m w q r s z z p y u s v { q s u s y n q n | p | w o } p y s ~  €  { s v s q m w | p q m n y y n ‚ v ƒ n v q m p } u u p z u w z q s ~ q m p

q w x o p j „ † ‡ ˆ ‡ ‰ Š ‹ Œ  Ž Œ  ‡  Ž Œ  ‡ ‘ ’ “ ” • – – — ˜ ™ š › ” “ œ › ”  • š › ” “ – ž › – ™  • Ÿ ¡ › “ š ¢ ˜ › ¡ ¡  ” £ š ¢ š • ¤  ’ ¥ — – š ¤  ” ¦

š ¢ Ÿ ” ˜ ™ ” “ “ š š • ¤  – § £ § › “ ¨ š ” ¥ — – š © ª « ¬ ­ ® ¯ ° ± ² ® ¯ ³ ´ µ ¶ · ¸ ¹ ¸ º » ¼ ½ ¹ ½ ¾ ¿ À Á  à Ä

Ô Õ Ø Ö × Ù Ú Û Ü Ý Þ ß à á â å ã ä æ ç è é ê ë ì í ì ë

Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ Ò Ð Ó

î ï ð ð ï ð ð ï ð ð ï ñ ò ó ô õ ö ÷ ø ù ú û ÷ ú ü ü ÷ ü ý þ ÿ





            

                  

                

                 

                    

                                

                           

% & % ' " ( ) * + , - . - , % ' / ( ) * + , - . - ,

      ! " ! # " # $ $ $

                                  

                            

                                  

                                     

                            

                             

                                   

                               

' '
% & % # ( ) * + , - . - , % 0 1 2 3 4 5 6 7 6 5

      ! " ! # " # $ $ $

8 9 : : 9 : : 9 : : 9 : ; ; 9 8 < = : : ; 9 8 = > 8 ? ;

: 9 : : 9 : : 9 @ ? 9 @ ; ? 9 < < A : : ; 9 = ? = A ;

A 9 : : 9 @ ? 9 : : 9 : ; ; 9 8 < = : : ; 9 ; < ? @ 8 9 8 ?

> 9 @ ? 9 : : 9 : : 9 : ; ; 9 8 < = : : ; 9 ; B A < > 9 A >

< 9 : : 9 @ ? 9 @ ? 9 @ ; ? 9 < < A : : ; 9 < ? 8 8 ? ;

= 9 @ ? 9 : : 9 @ ? 9 @ ; ? 9 < < A : : ; 9 B @ : C A @ ;

@ 9 @ ? 9 @ ? 9 : : 9 : ; ; 9 8 < = : : ; C 9 < < A C 8 @ @ ;

? 9 @ ? 9 @ ? 9 @ ? 9 @ ; ? 9 < < A : : ; 9 A : @ < > ;

O P O Q R S T U V W X Y X W \ Z [ ] S T U V W X Y X W

D E F G H I J K L J K M J L M N N

^ _ ` ` _ ` ` _ ` ` _ ^ ` a _ b ^ c d b _ d ` _ ^ b ^ ^ d _ b d

` _ ` ` _ ` ` _ e f _ a a b _ b a g d b _ a ^ _ a ` ^ a b

c _ ` ` _ e f _ ` ` _ ^ ` a _ b ^ ` d b _ d ` _ b c h e ^ _ c e

a _ e f _ ` ` _ ` ` _ ^ ` a _ b ^ c d b _ d ^ _ b a c h g _ g `

g _ ` ` _ e f _ e f _ a a b _ b a a d b _ a ^ _ c g e ^ d b

h _ e f _ ` ` _ e f _ a a b _ b a a d b _ a b _ a c g ^ b

e _ e f _ e f _ ` ` _ ^ ` a _ b ^ c d b _ d ^ i _ ` a b i d c b

f _ e f _ e f _ e f _ a a b _ b a a d b _ a b _ ^ g ` h h b

Z
\ [ s \ [ t u v w x y z { | { z  } ~ t € v w x y z { | { z

j k l m n o p ] q p ] r p q r

 ‚ ƒ ƒ ‚ ƒ ƒ ‚ ƒ ƒ ‚ ƒ „ ‚  ƒ † † ‡ ‚  ˆ ‚  ‡   ‡ „

ƒ ‚ ƒ ƒ ‚ ƒ ƒ ‚ ‡ ‰ ‚ ˆ Š ‡ ‚ „ ‰ † ‰ † ‚  ‡ ‚ ˆ ‡ ˆ ‹ „

† ‚ ƒ ƒ ‚ ‡ ‰ ‚ ƒ ƒ ‚ ƒ „ ‚  „ ‡ ˆ ‰ ‚ ƒ ‰ ‚ „  ‡ „ ‚ 

ˆ ‚ ‡ ‰ ‚ ƒ ƒ ‚ ƒ ƒ ‚ ƒ „ ‚  † ˆ † ‹ ‚  † ‚  † Š † ƒ ‚ „ ‹

‹ ‚ ƒ ƒ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ˆ Š ‡ ‚ „ „ ‰ ‰ ‚ † ‹ ‚ † ˆ ƒ ‡ „

‚ ‡ ‰ ‚ ƒ ƒ ‚ ‡ ‰ ‚ ˆ Š ‡ ‚ „ Š ‰  ‚  ˆ ‚ ‹ †  Œ ‡ „

‡ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ƒ ƒ ‚ ƒ „ ‚  ƒ „ ˆ ƒ ‚  ‰ Œ ‚ ƒ „ ‡ Œ  „

‰ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ˆ Š ‡ ‚ „ ‰ „ ‰ ˆ ‚  ‰ ‚  ‰ ˆ † „

 ~ –  ~ t — v w x y z { | { z  ~ t ˜ v w x y z { | { z
} }

 Ž   ‘ ’ “ ” u “ ” • “ u •

 ‚ ƒ ƒ ‚ ƒ ƒ ‚ ƒ ƒ ‚ ƒ ˆ ‹ ‚ ƒ „ ‡  ‹ „ ‚ ƒ „ †  ‡ „

ƒ ‚ ƒ ƒ ‚ ƒ ƒ ‚ ‡ ‰ ‚ ‡  Š ‚ ‹ ‹ ƒ † „ ‚ Š ‡ † „

† ‚ ƒ ƒ ‚ ‡ ‰ ‚ ƒ ƒ ‚ ƒ ˆ ‹ ‚  Š ˆ ƒ  „ ‚ „ ‡ ƒ ‡ „ ‚  

ˆ ‚ ‡ ‰ ‚ ƒ ƒ ‚ ƒ ƒ ‚ ƒ ˆ ‹ ‚ ƒ  ƒ  ˆ „ ‚  ‹ ƒ † ‰ ‚  ‹

‹ ‚ ƒ ƒ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ‡  Š ‚ ‹ ˆ † ƒ ‹ „ ‚ ‹ ƒ  „

‚ ‡ ‰ ‚ ƒ ƒ ‚ ‡ ‰ ‚ ‡  Š ‚ ‹ ˆ ƒ ƒ „  ‚ „ „ ‹ Œ ˆ „ „

‡ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ƒ ƒ ‚ ƒ ˆ ‹ ‚ ƒ „ ˆ  ‡ „ Œ ‚ ‹  Œ   „ „

‰ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ‡ ‰ ‚ ‡  Š ‚ ‹ ‹ ‡ ƒ † „ ‚ † † ‡ ‹ † „

17
Figure 5: Case 1, interaction plot for StDi¤. (Identi cation of the covstr levels given
in the text.)

3.2.2 Case 1: Variance estimation

The accuracy of the variance estimation for the estimator involving only X followed in a
sense the same pattern as for those reported for the corresponding results for Case 0. It
was less biased or practically unbiased for some covariance structures, here those where
the correlation between X and Y was low (½ = :22): that is, the structures denoted by
the numbers 1, 2, 4, and 6. And it was more biased for the covariance structures where

the correlation between X and Y was high (½ = :78): that is, those denoted by the
numbers 3, 5, 7, and 8 (Figure 5, the pane covstr £ observed). But, as previously,


 

that the variance estimate was correct did not help to achieve the desired signi cance
level of the con dence intervals: the biased point estimate prevented this (Figure 6).

3.3 Case 2: SITA violated, Y and Z correlated after all the

relevant information observed
A dependence between the study variable Y and the indicator of the subset membership
Z that remains after conditioning their joint distribution on the auxiliary information,
symbolically represented as (Y \Z) j X, violates one of the assumptions of SITA. In such
a situation, in words, there exists information in the subset membership Z about Y that
is not available for adjustment by the propensity score technique. This factor, named
sitavio1, was now added to the 6 previously analysed ones: covstr, ssize, knratio,
method, nstrata, and observed (excluding observed=x1).

18
Figure 6: Case 1, interaction plot for Clevel. (Identi cation of the covstr levels given
in the text.)

3.3.1 Case 2: Point estimation

The direction of in‡uence of a remaining conditional dependence, (Y \Z) j X, on the
estimate of the mean of Y in the population was a function of the sign of the correlation
coe¢cient ½ between X and Y . Negative correlations increased the point estimate
and positive ones decreased it (Figure 7). If with ½ = 0 the point estimate was an

overestimate, then a change in the covariance structure to a negative ½ would increase

 

the bias, while a change to a positive ½ , for an appropriate range of small values of ½ ,
 

would decrease the bias but for a large value of ½ again increase it. The reverse would
   

hold if, with ½ = 0, the point estimate was an underestimate.

 

The factor sitavio1 was by far the strongest among those included (Table IX in the
 

Appendix).
The covariance structures di¤ered in how much they were a¤ected by the departures of
½ from zero. More resistant were those where the correlation between the participation
variable X and the study variable Y was high (the structures 2, 5, 6, and 8), while less
 

resistant were those where the correlation between X and Y was low (the structures 1,


3, 4, and 7).


3.3.2 Case 2: Variance estimation

The simulation statistics related to variance estimation and con dence levels, StDi¤ and
Clevel, followed in general the patterns detected for the corresponding results for the
Cases 0 and 1.

19
Figure 7: Case 2, interaction plot for MeanBias. (Identi cation of the covstr levels
given in the text.)

For the level method=a, StDi¤ was almost correct irrespective of the covariance
structure and of the ½ , but the con dence level was approximately correct in only two
of the six cases: it was approximately correct when ½ = 0 for the covariance structures

with low correlation between the participation variable X and the study variable Y

(the structures 1, 3, 4, and 7), and it was approximately correct when ½ = :175 for


the covariance structures with high correlation of the two variables (the structures 2, 5,

6, and 8). The latter is result due to the point estimate having been “drawn” towards
the correct value by the positive ½ , as mentioned under the preceding heading, “Point
estimation”. For all the other combinations of ½ and ½ the con dence level was quite
 

low, between :1 and :4.

   

When the propensity score was estimated (i.e. method6=a), the variance estimates
for the covariance structures 1, 3, 4, and 7 (those with low correlation ½ between the
participation variable X and the study variable Y ) were approximately correct for all
the three levels of ½ , while the variance estimates for the structures 2, 5, 6, and 8 (those
with high ½ ) were considerably underestimating the true variances for all the three
levels of ½ . The e¤ect on con dence level was even here that the positive ½ “drew”
the point estimate towards the correct value, enabling even underestimated con dence
intervals to cover the true parameter in a somewhat higher proportion of outcomes then
when ½ = 0.

20
Figure 8: Case 3, interaction plot for MeanBias. (Identi cation of the covstr levels
given in the text.)

3.4 Case 3: SITA violated, not all units given a positive prob-
ability to appear in r
When the determining property of the subset was Z = I , a unit in the subset may
have taken on any value of X , it was only more probable that it would have taken on

 

a higher X value then what a unit in the population would, on the average. When,


on the other hand, the determining property of the subset was Z = I —as in


the alternative termed “SITA violation #2”—no unit in the subset may have taken on a
     

negative value of X . In other words, units with the negative X had no chance to appear
in the sample from the subset, r. While far from the only such rule, in what follows the
 

e¤ects of just the rule I were investigated.

          

3.4.1 Case 3: Point estimation

That only the units with positive X values could appear in the restricted sample r had
a big e¤ect on the adjusted point estimates of the mean of Y in the population, which


was due to the correlation between X and Y . Accordingly, the covariance structures
with high ½ (i.e., the structures 2, 5, 6, and 8) were more biased than those with low


½ .(Figure 8).
 

It is interesting to note that not much of the unadjusted bias could in this case of SITA
 

violation be corrected by the propensity score technique. For the covariance structures
with high ½ , the reduction was from :622 to :553, that is, 11%, and for the covariance
structures with low ½ , it was from :175 to :156, again 11%. This held irrespective of
 

 

21
 Figure 9: Case 4, interaction plot for MeanBias.

whether the true or the estimated propensity score was used.

The factor sitavio2 was by far the dominating one in the ANOVA decomposition for
MeanBias (Table XIII in the Appendix).

3.4.2 Case 3: Variance estimation

With the current violation of SITA, variance estimates underestimated the true variance
of the point estimators: the mean of StDi¤ for sitavio2=yes, across all the other
factors, was ¡:0145 (Table XVI in the Appendix). This, coupled to grossly biased point
estimates, yielded empirical con dence levels that were o¤ the mark practically all the
time.

3.5 Case 4–Case 6: Pairwise combinations of the SITA viola-

tions
The cases 4-6 present pairwise combinations of the SITA violations investigated under
Case 1 – Case 3. The results are reported only for MeanBias, in Figures 9–11 (but the
details concerning all the simulation statistics can be found in the Appendix).
The main conclusion could be drawn that also with combinations of the SITA viola-
tions, the adjusted point estimates of the mean of Y in the population were in general
badly biased. But, in some cases of interaction of the factor levels, these estimates were
less biased than without such an interaction. Two identi ed sources for this were the
level rho14 of the factor sitavio1 and the level observed=x1. Regarding the former,
the impact of rho14 was to“draw” the adjusted point estimates in the negative direction

22
Figure 10: Case 5, interaction plot for MeanBias.

Figure 11: Case 6, interaction plot for MeanBias.

23
Figure 12: Case 7, interaction plot for MeanBias. (Identi cation of the covstr levels
given in the text.)

thus having the potential to reduce the remaining bias of any overestimating adjusted
point estimate. Regarding the latter, observed=x1 coupled with covariance structure
7 led to major underestimation and thus it too had the potential to reduce the bias of an
overestimating point estimate.
How bene cial would the e¤ect of these sources together be is something predictable
only provided all the relevant information on the exact nature of the SITA violations is
known in advance—which is di¢cult to achieve in practice. In other words, while under
some fortuitous occasions violation of the SITA assumptions could actually reduce the
bias, it is di¢cult to determine, without the complete knowledge, for some situation at
hand whether it is such an occasion or not.

3.6 Case 7: All the violations at the same time

The preceding remarks concerning the in‡uence of the pairwise violations of the SITA
assumptions on the point estimates were applicable also when all the violations were
introduced into the same experiment (Figure 12). Of interest remained to compare the
relative contributions of the studied factors in this case when they all were present.
The comparison was done using the ANOVA decomposition for MeanBias (Table
XXIX in the Appendix). In Table 5, the 11 most important e¤ects from this table
are presented in the descending order of their F-statistics values; together, these e¤ects
accounted for the 98% of the observed sums of squares.
The two factors with the greatest impact on the bias of the adjusted point estimate
were sitavio2 and covstr, in this order, while their interaction had the third largest

24
Table 5: Case 7, partial reproduction of the ANOVA decomposition for MeanBias (Table
XXIX in the Appendix): the e¤ects with P<.05 presented in an approximate
increasing order of magnitude (the rst 9 P-values were evaluated as 0 by the
F cdf function of a mathematical program with precise calculation (Maple),
why they are given in the order where main e¤ects come rst, followed by their
interactions).

              

            ! " # $ # % $ & " # $ # % $ & " # # & ' ' ' '

( ) * + , - . / / 0 1 0 0 2 3 4 4 5 6 5 5 7 3 4 3 6 8 4 9 : ; ; < < < <

= > ? @ A > B ; C D E D F G G D E D F G G G E G G D D C G : F E C G <

B H I J K L J M N O P Q R S T O P Q R S T Q P N U Q R S T R P O T Q

V W L X Y K Z [ \ ] ^ _ \ ` a b c d e f f g h i d e f f g h j e k l m m l l d e m m l

n o p q r s q t u p v w x s v n f i g e i f m m g e i f m m f e y y m g h d g j e f k l

z n s p w r u n o p q r s q t f h j f e g k j g j f e g k j g f e j h y m h g l h e h h l

p v w x s v n h u p v w x s v n f f h e f f l k h e f f l k l e k h l i m i k e k j l

z n s p w r u p v w x s v n h h m f e j m d m f e j m d m l e h g f h h i l e f g l

n o p q r s q t u p v w x s v n h k l e f k i y l e f k i y l e l m m i h e m j l

{ | } ~  }  € |  }  ‚  ƒ „ † ‡ † ˆ ‰ „ † ‡ † ˆ ‰ „ † ‡ † ˆ ‰ „ Š „ ‡ ‹ Œ †

 Ž }   ‘ ’ “ ” • – — ”  ˜ ™ š › š ˜ œ  š › š ˜ œ  š › š š œ œ  › š ž š › š š ˜

Ÿ “ • – • – ˜ š › š š  ¡ š › š š  ¡ š › š š  ¡ ¢ › ¢  š › š ˜ ¡

impact. sitavio1 had the next largest impact, followed by some of the interactions of the
factor observed and then this factor by itself together with some other of the pairwise
interactions.
It is noteworthy that two of the “regular” factors—not involving violations of the
assumptions for the propensity score technique—showed a large in‡uence on the adjusted
point estimate even when the assumptions were not ful lled. These were covstr and
observed, emphasizing the two facts: (a) that the true (and not known beforehand)
covariance structure of the data at hand may have a profound e¤ect on the adjusted
point estimate, and (b) that failure to observe all the relevant information may too have
a profound e¤ect on this estimate.

4 Conclusions
This simulation of the propensity score adjustment technique partially demonstrated the
practical viability of the approach and partially investigated the e¤ects of certain factors
whose in‡uence it was not possible to express in a closed statement. An important
aspect concerning the latter goal was the behaviour of the propensity score adjusted
point estimate under violations of the assumptions required for the technique to work
optimally. The following summary of the results starts with the situation when the
assumptions held.

25
4.1 The factors conforming to the SITA assumptions
4.1.1 Method
The only factor that in the present study had no signi cant e¤ect on the remaining
bias of the adjusted point estimate was the method for estimation of the propensity
score: applying logistic regression versus discriminant analysis resulted in only negligible
di¤erences between the point estimates. It was already noted that the two methods yield
theoretically the same results under the models chosen for this study: only the di¤erences
related to the estimation algorithms were eventually expected to show up—these turned
out to be practically nonexistent. With other models, the outcome might have been
di¤erent.

4.1.2 Observed variables

The rest of the factors did have a signi cant e¤ect. Whether the distribution of the true
propensity score was known, or estimated using the variable that “really” determined the
participation, or using this variable plus another variable, did have an increasing e¤ect
on the remaining bias of the resulting point estimate.
Even interaction of this factor with the sample size was signi cant: the aforementioned
e¤ect was noticeable with the smaller sample size but had practically vanished with the
larger sample size.

4.1.3 Balance of the samples

Next, the ratio of the sizes of the samples s and r did have an e¤ect on the remaining
bias of the adjusted point estimate. Increasing this factor—from the restricted sample r
being half the size of s, to being as large as s, to being one and a half the size of s—led to
the reduction of the bias of the point estimate. An increase in this factor always implied
access to more data, which resulted in more accurate point estimates. Another candidate
explanation, needing further exploration, would be that with more data available, the
chances for a stratum being empty in the restricted sample were smaller, leading to a
smaller bias of the resulting estimate: the signi cant interaction between this factor and
the sample size, with much smaller e¤ect on the high sample size level, possibly gives
support to this interpretation.

4.1.4 Sizes of the samples

The sample size was a factor whose increase led to a smaller bias of the resulting point
estimates, with the straightforward interpretation that with more data the estimates
became more precise, as well as that the chance of a stratum being empty in the restricted
sample was smaller with larger sample sizes.

4.1.5 Number of strata

Finally, the two factors with the largest impact on the remaining bias of the adjusted point
estimate were the number of strata into which to classify the observations and the covari-
ance structure of the data at hand. Both were theoretically expected, as it was demon-

26
strated in for instance (Lorenc, 2003). Interestingly, the interaction nstrata£ssize
proved to be insigni cant contrary to the expectation that with the small sample size the
larger number of strata would more frequently lead to empty strata in the restricted sam-
ple and thus to larger bias of the resulting point estimates, compared to the large sample
size. Possibly, with a further reduction of the sample size, the e¤ect would eventually
show up.

4.1.6 Covariance structure

The remaining bias of the point estimate for the 8 covariance structures was dependent
on the correlation between the “participation” variable and the study variable. But, this
correlation also determined the original bias: it was higher for the structures with the
higher correlation, and lower for those with the lower correlation. So, the propensity
score adjustment did in fact have in both cases (i.e., with both high and low ½ ) the
same percentage reduction in bias.

4.2 The factors violating the SITA assumptions

The remaining three factors—whether all the relevant auxiliary information was observed,
whether the study variable Y could be “untangled" from the subset indicator Z, and
whether all the units in the population had a positive chance to appear in the restricted
sample r—had substantial e¤ects on the remaining bias of the adjusted point estimator.
As the levels of particularly the latter two of the factors (sitavio1 and sitavio2, respec-
tively) were set rather arbitrarily, their impact in the present study should be taken as
an illustration of their potential rather than as something to directly generalize to other
situations.

4.2.1 Not all relevant information observed

The e¤ect of not observing all relevant auxiliary information—more precisely, of missing
the participation variable—proved particularly interesting in the case when the available
auxiliary information was highly correlated with both the participation variable and the
study variable but the correlation of the latter two was poor. In this case, a strong
overadjustment occurred, resulting in almost doubling the original bias.

4.2.2 Y and Z correlated after all relevant information observed

Peculiar to the case of a remaining correlation between the study variable and the subset
indicator after conditioning on the auxiliary information was that, dependent on the sign
of the correlation, this could increase or decrease the remaining bias of the adjusted
point estimator. If there was a remaining bias in this estimator, as in general is the case
with the propensity score adjusted estimates, then under “lucky” occasions the remaining
correlation may have reduced the bias of the adjusted estimator, compared to the case
with no remaining correlation. Unfortunately, the researcher in real applications has no
such deep knowledge of the variables of relevance (and in particular of the study variables)
to be able to count on this: had this knowledge existed, no study would have been needed

27
in the rst place. Thus, counting on this e¤ect would in general be like “shooting in the
dark”.

4.2.3 Not all units had a positive chance to appear in r

That the factor sitavio2 had a profound e¤ect in the study depended to a large degree
on the rough alternative inclusion function chosen, Z = I . There are in nitely
many other functions that could be conceived, with a varying e¤ect on the remaining

      

bias of the point estimate.

4.3 Summary
This study demonstrated the e¤ectiveness of the propensity score weighting technique in
the situation with known properties of the underlying population, and investigated the
e¤ect of some of the factors plausibly in e¤ect even in real-life studies, thus providing some
impression of what might be expected in practical applications of this same technique.
The study proved practically that the propensity score weighting works, reducing most
of the bias in the situation when the assumptions that pertain to the technique hold.
It further demonstrated the relative sensitivity of the resulting estimates to variation of
both the factors internal to the technique (like sample sizes, ratio of the sample sizes,
number of strata, etc.) and the factors that determine the nature of data to which the
technique is applied (like covariance structure and the violations of the assumptions).

Acknowledgement
Support for this study from the Bank of Sweden Tercentenary Foundation, Grant no.
2000-5063, is gratefully acknowledged.

References
[1] Cochran, W.G. (1968). “The e¤ectiveness of adjustment by subclassi cation in re-
moving bias in observational studies”. Biometrics, 24:205-13.

[2] Cochran, W.G. and Rubin, D.B. (1973). “Controlling bias in observational studies: a
review”. Sankya, ser. A, 35:417-46.

[3] Lorenc, B. (2003). “E¤ectiveness of weighting by strati cation on the propensity score
using double samples”. Research report 2003:10. Department of statistics, Stockholm
university.

[4] Mosteller, F. and Tukey, J.W. (1977). Data Analysis and Regression: A Second Course
in Statistics. Reading, MA: Addison-Wesley.

[5] Rosenbaum, P.R. and Rubin, D.B. (1983a). ”The central role of the propensity score
in observational studies for causal e¤ects”. Biometrika, 70:41-55.

28
[6] small Rosenbaum, P.R. and Rubin, D.B. (1983b). ”Assessing sensitivity to an unob-
served binary covariate in an observational study with binary outcome”. Journal of
the Royal Statistical Society, ser. B, 45:212-18.

[7] Rosenbaum, P.R. and Rubin, D.B. (1984). ”Reducing bias in observational studies
using subclassi cation on the propensity score.” Journal of the American Statistical
Association, 79:516-24.

[8] Terhanian, G., Marcus, S., Bremer, J., and Smith, R. (2001). ”Reducing error associ-
ated with non-probability sampling through propensity scores: evidence from election
2000”. Joint Statistical Meeting 2001, August 5-9, 2001, Atlanta, Georgia, USA.

[9] Terhanian, G., Taylor, H., Siegel, J., Bremer, J., and Smith, R. (2001): “The Accuracy
of Harris Interactive’s Pre-Election Polls of 2000”. AAPOR 2001 Annual Conference,
17-20 May 2001, Montreal, Quebec.

29
Appendix to: “Propensity Score Weighting with
Double Samples: a Simulation Study”

Table I: Case 0: Analysis of Variance for MeanBias, using Adjusted SS for Tests.

                

          ! " " # $ % & ! " " # $ # ' ( ) * + , , - - - -

. . / 0 1 2 2 3 4 5 6 7 8 8 9 : ; < = > 7 8 8 9 : ; < = > 7 8 8 9 ? ? ; @ < 8

A B C D E F G H I J K L M N O O L P Q K L M N O O L K Q R M N O O S T U V U W X

Y Z [ \ ] ^ _ ` ` ` ` a ` _ ` a b c d

e f g h i g i _ ` a ` _ b j k k ` a ` _ b j k k ` a ` _ b j k k c l ` ` ` `

] m n o p q o r s t u v w x y y z t u v w x y y z v u { w x y y z { u v v y u y y |

} ~
q n  p €   ‚ ƒ „ † ‡ ˆ ‰ Š ‹ Œ Œ  † ‡ ˆ ‰ Š ‹ Œ Œ  ˆ ‡ Ž Š ‹ Œ Œ ‰  ‡ ˆ Œ

  ‘  — —
 ’ “ € ” • “ – ’ ‚ Ž Ž ‡  ‰ Š ‹ Œ Œ  Ž ‡  ‰ Š ‹ Œ Œ  ˆ ‡ ˆ Š ‹ Œ Œ ‰  ‡ Œ

  ‘  š —

 ’ “ € ˜ „ ’ ™ Œ Œ Œ Œ

  ‘ — —

 ’ “ € •  ’ “ – ’ – Œ ‡ Œ Œ ‰ ‰ ‰ Ž ˆ Œ ‡ Œ Œ ‰ ‰ ‰ Ž ˆ Œ ‡ Œ Œ Œ ›  †  ˆ ‡ ‰  Œ

  ‘  œ ‘ š — — — — — —
 ’ “ €  „ “ „ Ž ‡ ˆ Š ‹ Œ Œ  ‡ ˆ Š ‹ Œ Œ  › Š ‹ Œ Œ ‡ Ž  Œ ‡ ‰

 —

  ‚ ƒ „ € ” • “ – ’ ‚ †  ‡ Š ‹ Œ Œ ‰  ‡ Š ‹ Œ Œ ‰ † ‡  Š ‹ Œ Œ ‰ Ž ‡  Œ ‡ Œ †

 š —

  ‚ ƒ „ € ˜ „ ’ ™ Œ Œ Œ Œ Œ ‡ ›  ‰

— — — — —

  ‚ ƒ „ € •  ’ “ – ’ – Š ‹ Œ Œ Š ‹ Œ Œ Š ‹ Œ Œ Œ ‡ Œ ‡ Ž ›

 œ ‘ š —

  ‚ ƒ „ €  „ “ „ † Ž Š ‹ Œ Œ ‰ Ž Š ‹ Œ Œ ‰ † Š ‹ Œ Œ ‰ ˆ ‡ ‰ Œ ‡ Œ Ž ˆ

  š

” • “ – ’ ‚ € ˜ „ ’ ™ † Œ Œ Œ Œ Œ ‡ › › ›

” • “ – ’ ‚ € •  ’ “ – ’ – † ‰ ‡  Š ‹ Œ Œ ‰ ‰ ‡  Š ‹ Œ Œ ‰ ˆ ‡ Ž Š ‹ Œ Œ ‰  ‡ ˆ Œ ‡ Œ Œ 

  œ ‘ š — —

” • “ – ’ ‚ €  „ “ „ Ž ‡ Ž Š ‹ Œ Œ ‰ ‡ Ž Š ‹ Œ Œ ‰ Ž Š ‹ Œ Œ Œ ‡  Œ ‡ ‰  

 š — —

˜ „ ’ ™ € •  ’ “ – ’ – Œ Œ Œ Œ Œ ‡ › 

 š  œ ‘ š —

˜ „ ’ ™ €  „ “ „ † Œ Œ Œ Œ ‡ Œ Œ ‡ › › ˆ

 œ ‘ š

•  ’ “ – ’ – €  „ “ „ † › Š ‹ Œ Œ › Š ‹ Œ Œ  Š ‹ Œ Œ Œ ‡ ˆ Œ ‡ Ž 

— — — —
„ ž ž Ÿ ž Ž › ˆ Œ ‡ Œ Œ Œ ˆ Œ Œ ‡ Œ Œ Œ ˆ Œ ‰ Š ‹ Œ Œ

’ Ÿ ¡ ¢ £ ¤ £ ¥ ¦ § ¤ ¨ ¤ ¤

Table II: Case 0: Analysis of Variance for Clevel, using Adjusted SS for Tests.
© ª « ¬ ­ ® ¯ ° © ® ± © © ² ³ ´ © © ² ³ ´ µ © ° ¶

· ¸ ¹ © º » ¼ ½ ¼ ¾ ¿ À Á ¿ ½ ¼ ¾ ¿ À Á ¿ À ¾ ½ ¿ ¿ Â Â Ã Â ¿ ¾ Ä ¿ Ã

© © Å Æ Ç È Á ¾ ¿ É À À Á ¾ ¿ É À À Á ¾ ¿ É À À Ä ½ Ä ¿ ¾ ½ ¼ Ã

» ² º Å ¸ ¿ ¿ ¾ Ã ½ ¼ È ¿ ¾ Ã ½ ¼ È È ¾ Ã È Á Â È È Â Ã ¾ ¿ É Ã

Ê Ë

µ Ç º Ì ¸ ¯ È Ã Ã Ã Ã Ã ¾ Ä É É

© º » ² º ² È ¿ ¾ È Á Ã À ¿ ¾ È Á Ã À ¿ ¾ È Á Ã À ¿ É Á ½ ¾ Á ½ Ã

¸ Í © Ç » ¹ Ç ¯ ¿ Ã ¾ Ã Ã É À Á Ã ¾ Ã Ã É À Á Ã ¾ Ã Ã ¿ ¿ Ä ¿ ¾ Â È Ã ¾ Ã ¼ À

· ¸ ¹ © º » Î © © Å Æ Ç ¼ À ¾ Ä ¿ È Ä À ¾ Ä ¿ È Ä Ã ¾ Á É À Ä Á Ä Â ½ ¾ Â À Ã

· ¸ ¹ © º » Î » ² º Å ¸ È É È ¾ ¿ È Â È ¾ ¿ È Â Ã ¾ Ã Á Â Á Â Ä Á ¾ Ä É Ã
Ê Ë

· ¸ ¹ © º » Î µ Ç º Ì ¸ ¯ ¼ ¿ ® Ï Ã Ã À ¿ ® Ï Ã Ã À Ã Ã È

· ¸ ¹ © º » Î © º » ² º ² ¼ È ¾ ½ È À ¼ È ¾ ½ È À ¼ Ã ¾ È Á ¼ Ä Â ¿ È É ¾ È Ã

· ¸ ¹ © º » Î ¸ Í © Ç » ¹ Ç ¯ È É Ã ¾ Ã Ã ¼ À Â Ã ¾ Ã Ã ¼ À Â Ã ¾ Ã Ã Ã À É Ã ¾ Â ¿ Ã ¾ Á À ½

© © Å Æ Ç Î » ² º Å ¸ ¿ Ã ¾ Ã È Ä ½ Â Ã ¾ Ã È Ä ½ Â Ã ¾ Ã Ã Ä Â Á È È ¾ Ã ¿ Ã

Ê Ë

© © Å Æ Ç Î µ Ç º Ì ¸ ¯ È Ã Ã Ã Ã Ã ¾ Ä Ä È

© © Å Æ Ç Î © º » ² º ² È Ã ¾ Ã ½ Á ¼ Á Ã ¾ Ã ½ Á ¼ Á Ã ¾ Ã ½ Á ¼ Á É É ¾ È ¼ Ã

© © Å Æ Ç Î ¸ Í © Ç » ¹ Ç ¯ ¿ Ã ¾ Ã ½ ¿ É À Ã ¾ Ã ½ ¿ É À Ã ¾ Ã È Â ¿ ¿ È Á ¾ É Á Ã

» ² º Å ¸ Î µ Ç º Ì ¸ ¯ ¿ Ã Ã Ã Ã È

Ê Ë

» ² º Å ¸ Î © º » ² º ² ¿ Ã ¾ Ã Ã É ¼ Â Ã ¾ Ã Ã É ¼ Â Ã ¾ Ã Ã ¿ ½ Á ¿ ¾ ¼ È Ã ¾ Ã Â Á
Ê Ë Ë

» ² º Å ¸ Î ¸ Í © Ç » ¹ Ç ¯ É Ã ¾ Ã Ã À É Â Ã ¾ Ã Ã À É Â Ã ¾ Ã Ã È ½ Â È ¾ À À Ã ¾ È Á Â

Ê Ë

µ Ç º Ì ¸ ¯ Î © º » ² º ² È Ã Ã Ã Ã Ã ¾ Ä ¼ Á
Ë

µ Ç º Ì ¸ ¯ Î ¸ Í © Ç » ¹ Ç ¯ ¿ È ® Ï Ã Ã À È ® Ï Ã Ã À Ã Ã Ã ¾ Ä Ä À

© º » ² º ² Î ¸ Í © Ç » ¹ Ç ¯ ¿ Ã ¾ Ã Ã ¿ È ¿ Ã ¾ Ã Ã ¿ È ¿ Ã ¾ Ã Ã È Ã Â È ¾ ¿ È Ã ¾ ½

Ç ¬ ¬ ª ¬ É Ä ½ Ã ¾ É ½ ¿ Á Ã ¾ É ½ ¿ Á Ã ¾ Ã Ã Ã Á Á

º ª Ð Ñ Ò À ¼ À À Á ¾ ¼ ¿ ¿ Ä

1
Table III: Case 0: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                       

                                   

                                  

! "

  #              

        $        $        $           
"

 %                                    

     &                         $       

     &                              $         

! "

     &   #      $      $         

     &       $      $       $           

"

     &  %                               $         

   &           $         $         $        

! "

   &   #            

   &         $        $        $            

"

   &  %                                   

    &   #        

! "

    &        $       $        $           
! " "

    &  %                              $        
! "

  #   &             
"

  #   &  %         $      $            

   &  %           $        $      $           

"

  
                         $   


' ( )            

2
Table IV: Case 0: Means and standard errors (SE) of the 1st and 2nd order e¤ects on
MeanBias, Clevel and StDi¤ across the other factors.

                                        

 !  " #

 , ,  ,
$ % & % $ ' ( % ) * & + % % ( % & * $ - ) ' % & % % + ) * ' ' & - % % ( % & % % % $ +

 , ,
' % & % ) ) ( ( ' * & + % % ( % & ) % - ) - % & % % + ) * ' % & % % ) ' ( . % & % % % $ +

 , ,
+ % & % $ ' ) ' ' * & + % % ( % & * $ ( / ( % & % % + ) * ' % & % % % + - / % & % % % $ +

 , ,
) % & % $ ' . ) ( * & + % % ( % & * $ $ $ * % & % % + ) * ' % & % % % * $ * % & % % % $ +

 , ,
( % & % ) ) + * - * & + % % ( % & ) % / ( % & % % + ) * ' % & % % ) ' $ ( % & % % % $ +

 , ,
/ % & % ) ) ) * * & + % % ( % & ) % ' $ . % & % % + ) * ' % & % % ) + ) / % & % % % $ +

 , ,
- % & % $ ' / * ( * & + % % ( % & * $ % . / % & % % + ) * ' % & % % $ $ + ' % & % % % $ +

 , ,
. % & % ) ) ' . . * & + % % ( % & ) % ) ) ' % & % % + ) * ' % & % % ) $ * - % & % % % $ +

  0 1 

 , ,  ,
$ % % % % & % ' . / . ' ) & - % % ( % & - - * $ $ % & % % $ - ) / % & % % + + - * / & ( % % (

 , ,  ,
( % % % % & % ' . + / - ) & - % % ( % & ( + * . ' % & % % $ - ) / % & % % $ ) . * / & ( % % (

2 3 # 4 " 0

$ 5 6 7 8 7 6 9 : 9 ; < 8 : = > 7 7 < 7 8 : ? : : @ 7 8 7 7 6 A ? 9 > 7 8 7 7 A @ < : 9 = > 7 7 <

6 5 6 7 8 7 6 9 ? B @ < 8 : = > 7 7 < 7 8 @ ; @ B 6 7 8 7 7 6 A ? 9 > 7 8 7 7 6 < ; 6 9 = > 7 7 <

? 5 6 7 8 7 6 9 ? B 6 < 8 : = > 7 7 < 7 8 < B ? : A 7 8 7 7 6 A ? 9 > 7 8 7 7 ? A 7 6 9 = > 7 7 <

C D E F G H

I J G I
7 8 7 6 9 < 6 A ; 8 : = > 7 7 < 7 8 @ < B < < 7 8 7 7 A : ; @ > 7 8 7 7 6 ; 6 @ @ 8 < = > 7 7 <

H K
7 8 7 6 9 < 6 : ; 8 : = > 7 7 < 7 8 @ < B ? 9 7 8 7 7 A : ; @ > 7 8 7 7 6 ; ; 6 @ 8 < = > 7 7 <

L M E J K E K

< 7 8 7 ? ; ? B B ; 8 : = > 7 7 < 7 8 < B : B ? 7 8 7 7 A : ; @ > 7 8 7 7 6 ? A 9 @ 8 < = > 7 7 <

: 7 8 7 6 6 @ < ; 8 : = > 7 7 < 7 8 : 6 7 B B 7 8 7 7 A : ; @ > 7 8 7 7 6 < < @ 8 < = > 7 7 <

G N M D J O D H

K
7 8 7 6 9 ? : @ < 8 : = > 7 7 < 7 8 @ @ 6 < 7 8 7 7 6 A ? 9 > 7 8 7 7 7 A ? 9 9 = > 7 7 <

P Q R S R Q T U R V U S W X Y R R U R S Z Z R [ T R S R R Q [ \ T Y R S R R \ \ ] V T X Y R R U

P [ P Q R S R Q T Z T W U S W X Y R R U R S Z U U W [ R S R R Q [ \ T Y R S R R \ T [ Z T X Y R R U

^ _ ` a b c d a a e f g

[ [ R R R R S R [ Q ] Q ] R S R R R [ \ Q R S V \ T R \ R S R R ] V \ T R S R R R Q R [ R S R R R [ T ]

[ U R R R R S R [ Q U T ] R S R R R [ \ Q R S T V Z T [ R S R R ] V \ T Y R S R R R Q U U R S R R R [ T ]

Q [ R R R R S R ] ] V [ W R S R R R [ \ Q R S Z Q W Q Q R S R R ] V \ T Y R S R R U T ] Z R S R R R [ T ]

Q U R R R R S R ] ] [ T T R S R R R [ \ Q R S [ T W W Q R S R R ] V \ T Y R S R R Q Z W [ R S R R R [ T ]

\ [ R R R R S R [ Q Q [ V R S R R R [ \ Q R S V \ W W U R S R R ] V \ T Y R S R R R \ U U R S R R R [ T ]

\ U R R R R S R [ Q Z Q U R S R R R [ \ Q R S T V \ U Z R S R R ] V \ T Y R S R R R \ V T R S R R R [ T ]

] [ R R R R S R [ \ Q Q ] R S R R R [ \ Q R S V Q T Q U R S R R ] V \ T Y R S R R [ ] T W R S R R R [ T ]

] U R R R R S R [ Q ] Z U R S R R R [ \ Q R S T V ] [ ] R S R R ] V \ T Y R S R R R \ U [ R S R R R [ T ]

U [ R R R R S R ] ] Z Q R S R R R [ \ Q R S Z Q W Z [ R S R R ] V \ T Y R S R R U W [ R S R R R [ T ]

U U R R R R S R ] ] [ W U R S R R R [ \ Q R S [ T U \ V R S R R ] V \ T Y R S R R Q W Q [ R S R R R [ T ]

Z [ R R R R S R ] ] U W V R S R R R [ \ Q R S Z Q ] W T R S R R ] V \ T Y R S R R Z R V ] R S R R R [ T ]

Z U R R R R S R ] ] ] R [ R S R R R [ \ Q R S [ W V U T R S R R ] V \ T Y R S R R Q U V V R S R R R [ T ]

W [ R R R R S R [ Q T Q Z R S R R R [ \ Q R S V Q ] Q Q R S R R ] V \ T Y R S R R [ V ] Q R S R R R [ T ]

W U R R R R S R [ Q U Z ] R S R R R [ \ Q R S T V W U R S R R ] V \ T Y R S R R R \ Q \ R S R R R [ T ]

T [ R R R R S R ] ] Z ] ] R S R R R [ \ Q R S Z Q U R S R R ] V \ T Y R S R R U W V W R S R R R [ T ]

T U R R R R S R ] \ V \ [ R S R R R [ \ Q R S [ T \ T \ R S R R ] V \ T Y R S R R Q U V W R S R R R [ T ]

^ _ ` a b c d h i c j b e _

[ [ k Q R S R [ Q U Q T R S R R R [ Z Q R S V \ [ Q U R S R R Z R ] T R S R R R Q U V R S R R R Q Q U

[ Q k Q R S R [ Q W \ V R S R R R [ Z Q R S V [ \ Q [ R S R R Z R ] T Y R S R R R U Q [ R S R R R Q Q U

[ \ k Q R S R [ Q Q ] ] R S R R R [ Z Q R S V R W W V R S R R Z R ] T R S R R R [ T [ R S R R R Q Q U

Q [ k Q R S R ] U R V ] R S R R R [ Z Q R S U ] \ Q U R S R R Z R ] T Y R S R R \ U Z ] R S R R R Q Q U

Q Q k Q R S R ] ] Q U \ R S R R R [ Z Q R S \ T U U ] R S R R Z R ] T Y R S R R \ T V U R S R R R Q Q U

Q \ k Q R S R ] ] \ [ R S R R R [ Z Q R S Q V \ Z \ R S R R Z R ] T Y R S R R U \ [ Z R S R R R Q Q U

\ [ k Q R S R [ \ R \ U R S R R R [ Z Q R S V \ \ U T R S R R Z R ] T R S R R R ] W T R S R R R Q Q U

\ Q k Q R S R [ Q R T [ R S R R R [ Z Q R S V [ V R T R S R R Z R ] T Y R S R R R U Q [ R S R R R Q Q U

\ \ k Q R S R [ Q [ U R S R R R [ Z Q R S T V ] Q V R S R R Z R ] T Y R S R R [ R T Z R S R R R Q Q U

] [ k Q R S R [ \ [ W Q R S R R R [ Z Q R S V Q U W V R S R R Z R ] T Y R S R R [ Z R S R R R Q Q U

] Q k Q R S R [ Q [ T Z R S R R R [ Z Q R S V [ U ] Q R S R R Z R ] T Y R S R R R Z [ V R S R R R Q Q U

] \ k Q R S R [ \ [ W U R S R R R [ Z Q R S T V Q \ T R S R R Z R ] T Y R S R R R U \ W R S R R R Q Q U

U [ k Q R S R ] ] \ ] Q R S R R R [ Z Q R S U U R \ \ R S R R Z R ] T Y R S R R Q Q W U R S R R R Q Q U

U Q k Q R S R ] ] W T Z R S R R R [ Z Q R S \ Z V ] Z R S R R Z R ] T Y R S R R ] W W ] R S R R R Q Q U

U \ k Q R S R ] ] R Z ] R S R R R [ Z Q R S Q V V W [ R S R R Z R ] T Y R S R R U U V W R S R R R Q Q U

Z [ k Q R S R ] ] \ ] V R S R R R [ Z Q R S U ] U \ T R S R R Z R ] T Y R S R R Q T U U R S R R R Q Q U

Z Q k Q R S R ] ] ] W T R S R R R [ Z Q R S \ W Q T T R S R R Z R ] T Y R S R R ] Z R Z R S R R R Q Q U

Z \ k Q R S R ] ] Z ] ] R S R R R [ Z Q R S Q T T Q V R S R R Z R ] T Y R S R R U U W T R S R R R Q Q U

W [ k Q R S R [ Q V Z ] R S R R R [ Z Q R S V \ [ Q V R S R R Z R ] T Y R S R R R Q Q T R S R R R Q Q U

W Q k Q R S R [ Q W Z ] R S R R R [ Z Q R S V [ Q U ] R S R R Z R ] T Y R S R R R V ] Q R S R R R Q Q U

W \ k Q R S R [ Q \ U Z R S R R R [ Z Q R S T T T W U R S R R Z R ] T Y R S R R Q Q Q T R S R R R Q Q U

T [ k Q R S R ] ] W V [ R S R R R [ Z Q R S U ] [ [ W R S R R Z R ] T Y R S R R \ ] W ] R S R R R Q Q U

T Q k Q R S R ] \ T T [ R S R R R [ Z Q R S \ T W Q [ R S R R Z R ] T Y R S R R ] ] U V R S R R R Q Q U

T \ k Q R S R ] ] [ V [ R S R R R [ Z Q R S Q T ] T W R S R R Z R ] T Y R S R R ] Z U T R S R R R Q Q U

3

                      
 

         

  
                                 !   !          

 " 
                               !   # !          

 
                         #        !               

 "
                              !               

 
          #                       !       #       

 "
                              !              

 
                                !               

 "
                             !               

 
                       #          !               

 "
                   #         !               

 
#                               !               

 "
#                               !       #        

 
        #                         !               

 "
      #  #                      !       #        

 
                      #         !               

 "
                             !               

     $   "  "


                                !    !          


                                !          

       #  #                       !               

                                !        #       

         #                      !       #       

                             !              

                     #         !               

        #               #         !      #        

        #                       !              

                                !               

#                              !              

#                                !               

                     #  #        !               

                                !               

        #                       !               

                      #         !              

         

"
       #        #            #                  

 %               #      #     #   !               


 %  %                #      #       #   ! #   !           

" 
               #            #   !   !           

 %                #             #   !     #           

 %  %                #      #       #   !     #          

"
              #            #                

 %                #            #   !                

 %  %                #            #   !       #         

"
       # #       #       #      #   !      #          

 %               #          #   !                

 %  %               #             #   !                

"
               #             #   !        #        

 %              #       #      #   !      #        

 %  %               #       #      #   !     # #          

"
#             #             #   !       #         

# %                #           #   !     #           

# %  %       #       #             #   !     #          

"
      #        #             #   !                

 %                #            #   !                

 %  %       #         #          #   !               

"
              #            #   !                

%                #       #      #   !     #   #        

%  %                #            #   !     #          

& '  ( $  "  & 


     )            !                 !                


     )           !                   !        #        


     )           !                   !                


     )           !      #  #           !                


     )          !                   !                


     )          !                   !                

4

                      
 

       

     
                        !        "  !     # #   !    

 $     
                     !        "  !     # #  ! !    

     
           #          # !  !      "  !      "  # !    

 $     
        #          # !  "      "  !      " ! !    

   %  $  $


    
       # "             !      "  !     #    !    

    
           !          #  " #      "  !     # "  !    

    
      # "  #        "          "  !      "  " !    

    
          #          !      "  !      " !    

     
 &

$   
         " "          !   !     #   "               #

  
    '        "            #     #   "     "           #

  
    '  '       !           !       #   "     # #        #

$   
        #          #  #      #   "        !        #

  
   '       #          " "  #     #   "                #

  
   '  '       "          "  "      #   "                #

( % $         


   
 )                    #   !     #   "               #

 $   
 )         !         #        #   "        "        #

   
 )          # ! "         "  !      #   "      # #        #

 $   
 )       # !          "       #   "              #

   
# )          # !        ! #       #   "     #  ! #        #

 $   
# )       # ! #        ! #      #   "     #           #

( % $     %  $  $
 

  
 )     # "             !      #   "      # !        #

  
 )        !           ! ! #     #   "                #

  
 )     # " "                #   "      # #         #

  
 )        #             #     #   "               #

  
# )     # "           #       #   "     #           #

  
# )          #               #   "     #   #        #

( % $       
  &

$   
 )         !  !      "        #   "      #  #       # 

  
 )  '        #  !  !      # # #     #   "              # 

  
 )  '  '      !  " !  !      #        #   "      ! # !       # 

$   
 )       #   !  !       #      #   "              # 

  
 )  '       #  ! !  !      "  " "     #   "     # " " #       # 

  
 )  '  '       " ! # !  !      "  ! "     #   "     "   "       # 

$    
# )        "  !  !     ! "     #   " !          # 

  
# )  '       "   !  !     ! !      #   "     #   #       # 

  
# )  '  '        ! !  !             #   "     " # #       # 

      %  $  $


     
      # " # !         !  ! !      "  !      #   !    

     
          "                   "  !      " # !    

 $     
   # " "        !         "  !      #   !    

 $     
       #                  "  !       !    

        
 &

 $   
        #                 #   "       #         #

   
   '        !                 #   "     # # " !        #

   
   '  '         !         !      #   "     #  !         #

 $ $   
     #                 #   "       #         #

 $   
'        !                 #   "     # # "         #

 $   
'  '        !         "     #   "     #  "        #

%  $  $    
  &

$     
   # "          !   !     #   "   "          #

  
'     # " " #           #     #   "     #           #

  
'  '     # "         !       #   "     #           #

$   
      "                  #   "                #

  
 '        #              #   "     # " #        #

  
 '  '         !                 #   "     "   !        #

5
Table V: Case 1: Analysis of Variance for MeanBias, using Adjusted SS for Tests.

        

                              

                            

                     #    

! "

  $          #  

        #  #     #  #     #  #        
"

 %                      #        

     &                        

     &                            

! "

     &   $        

     &           #       #          #    

"

     &  %         #      #             

   &                              

! "

   &   $          #  

   &              #  

"

   &  %                           

    &   $       

! "

    &                       
! " "

    &  %                # # #
! "

  $   &           # # 
"

  $   &  %           

   &  %                          #    

"

  
                    


' ( )          

Table VI: Case 1: Analysis of Variance for Clevel, using Adjusted SS for Tests.

        

           #       #            

                         

                #      

! "

  $          #  

                     
"

 %                              

     &      #    #          #    

     &                             

! "

     &   $        

     &       #     #           

"

     &  %                       

   &                             

! "

   &   $          # # 

   &                           

"

   &  %          #     #       #     

    &   $       

! "

    &                              

! " "

    &  %                               
! "

  $   &           # # 

"

  $   &  %           

   &  %           #      #           

"

  
       #      #       


' ( )           

6
Table VII: Case 1: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                       

                                    

                                  

  "               

      #      #      #       $     
!

 %      $                         $      

     &          $         $         #         

     &               $         $      #       $ 

     &   "        

     &         #        #        #            $

     &  %            $         $             $       

   &      $     #    $     #         #         

   &   "            

   &         #        #        #       $   $ $ 

   &  %      $                        #         

    &   "        

    &         #        #        #            
! !

    &  %                              #         
!

  "   &             
!

  "   &  %      $   #      #             

   &  %      $     #        #      #      $      

  
                          #   


' ( )           

7
Table VIII: Case 1: Means and standard errors (SE) of the 1st and 2nd order e¤ects
on MeanBias, Clevel and StDi¤ across the other factors.

                      
 

   

                                           

                                           

                                           

                                           

                                          

                                           

                                            

                                          


                                             


                                            

" #  $  


 %                                        


 %                                       


 %                                        

  &  

'   ' 
                                         

 $ 
                                        

#   $  $


                                          


                                          

    

$ 
                                         

( 
                                        

( 
                                          

( ( 
                                           

    ) !

                                        

                                              

                                              

                                              

                                            

                                              

                                              

                                             

                                              

                                             

                                              

                                              

                                               

                                               

                                              

                                              

8

                      
 

           

                   !        "  !              #

         !           !     "  ! $      #        #

                    # "      "  !              #

        %              !      "  ! $      !        #

        "             " %      "  ! $             #

        "                    "  ! $              #

        " "           "  #      "  ! $             #

        ! %           ! !  %     "  ! $              #

        "           !  "     "  ! $             #

         "           " #  !     "  ! $              #

        % "           "   #     "  ! $      "        #

         !           ! #  "     "  ! $      #        #

        #              !     "  ! $              #

        "            ! !      "  ! $            #

                      "     "  ! $     #         #

#                     %     "  ! $             #

#                    ! % !     "  ! $             #

#                      #      "  ! $              #

!    $                # % "     "  ! $              #

!    $                # "       "  ! $              #

!    $     !           # # # #     "  ! $             #

"       !              ! !     "  ! $             #

"       !  %            %  !     "  ! $            #

"       !                !     "  ! $             #

       &  

 
 ' '               !  %       "  $             "

 
               !  %       "  $             "

 
 ' '     " !                 "  $             "

 
     " !            !      "  $             "

 
 ' '     "           ! ! %       "  $             "

 
     "           ! ! " %      "  $             "

 
 ' '                "   !      "  $      %       "

 
                "   !      "  $      %       "

 
' '                  "      "  $           "

 
                 %      "  $           "

 
# ' '                   #      "  $             "

 
#                   #      "  $             "

 
! ' ' $              # "        "  $             "

 
! $              # "  %      "  $             "

 
" ' '    !                     "  $      #       "

 
"    !               "      "  $      #       "

          

     # #          # % "       "  $             "

 !               !        "  $             "

     #                 "  $             "

 !      !           " #       "  $             "

                ! #        "  $            "

 !     #           ! %  "      "  $             "

      #          ! % "       "  $            "

 !     "           "  !       "  $      %       "

    % %             "      "  $      %       "

!               " ! %      "  $            "

#     %             %       "  $            "

# !               "       "  $             "

! $     #           # !        "  $             "

! ! $     #          # %  %      "  $             "

"    ! % !             #      "  $             "

" !    #  #           " #       "  $      %       "

9

                      
 

         

                                           

                              !              

                             !              

                                 !              

       "                       !              

                          !      "        

                               !      "        

                        "         !              

"                                 "        

"      "            "   "         !      "        

"                              !              

"          "                       !      "        

                              !              

       "              "         !             

                             !             

         "                       !              

      "                      !              

    "                    !             

                              !             

                               !              

                              !              

      "                                 

                             !              

                                 !              

                              !              

   !                      !              

                     "          !              

                                 !             

                             !              

                              !             

                               !              

                                 !              

# $  % &   # 


     '                     "         !              "

     '                   "         !              "

    " '                   "         !              "

    '                              !              "

    '                             !              "

   " '                  "           !              "

# $    (  

  
    ) )      "           "  "        " !     "     !  

 
          "           "          " !     "     !  

  
   ) )                         " !          !  

 
                             " !          !  

# $  &   
 


         "            "        " !     "     !  


                     "         " !          !  


                  "          " !          !  


                            " !          !  

# $      

                 " "              " !             " 

          "       " "             " !           " 

                  " "             " !             " 

                   " "             " !            " 

         "       " "   "  "        " !             " 

         "       " "             " !            " 

                 " "             " !             " 

                   " "             " !      "       " 

% &   #     (  


 
 '  ) )                   "         !             "


 '                              !             "

 
 '  ) )                 "          !     "        "


 '                   "         !     "        "

 
" '  ) )                           !     "        "


" '                    "         !     "        "

10

                      
 

             

                    !      !   ! "      !        #

                    #     !   ! "              #

                     # #     !   ! "      $        #

         $             # $     !   ! "     #         #

#                         !   ! "     #        #

#                          !   ! "     #         #

           % 


       $        $            $ # ! "      #       

   &      #        $             $ # ! "     #       

   &      $        $     # #       $ # ! "             

   &  &      !       $     # $       $ # ! "      !       


       $ #       $            $ # ! "             

   &       $       $      #      $ # ! "     #       

   &      $       $           $ # ! "     #       

   &  &      $        $      !      $ # ! "            


#       $        $     !       $ # !        

#   &       !       $     $       $ # ! "            

#   &      $       $     ! !       $ # ! "     # !       

#   &  &      $        $     $        $ # ! "            

  '         

(   ( 
                          # "       !   "   

(   ( 
                           # "      ! !   "   

  
                          # "       !   "   

  
                          # "      ! !   "   

  '       % 

(   ( 
    $       # #            $ # "             # 

(   (
&      #       # #      $      $ # "           # 

(   (
&      $        # #            $ # "     # #       # 

(   (
&  &      $        # #           $ # "     # $       # 

  
    $       # #            $ # "             # 

 
&      #       # #      $      $ # "           # 

 
&      $        # #            $ # "     # #       # 

 
&  &      $        # #           $ # "     # $       # 

          % 


    #        # #    ! $ #      $ # "             # 

 &              # #            $ # "     # !       # 

 &     #       # #           $ # "     # #       # 

 &  &     #        # #    !        $ # "     #        # 


             # #            $ # "             # 

 &             # #           $ # "            # 

 &              # #          $ # "     #       # 

 &  &       $       # #      #      $ # "           # 

11
Table IX: Case 2: Analysis of Variance for MeanBias, using Adjusted SS for Tests.

        

                                 

       !       !       !            

       !      !      !         

" #

  $       !       !       !          

                               
#

 %                           !            

                             

     &        !       !      !        

     &                         !        

" #

     &   $        !        !     !        

     &                               

     &  %                            !        

     &                                 

   &        !      !     !            

" #

   &   $            

   &       !      !      !            

   &  %           !        !      !           

   &            !        !      !           

    &   $        
" #

    &         !        !      !           

" # #

    &  %         !      !      !        

" #

    &            !        !      !            

" #

  $   &       !      !      !          

  $   &  %                         !            

  $   &           !       !        !           

   &  %           

   &           !       !        !            

 %      &                                    

  
                   


' ( )          

Table X: Case 2: Analysis of Variance for Clevel, using Adjusted SS for Tests.

        

                          

                          

                        

" #

  $                             

                         

 %                           

                              

     &                        

     &                              

" #

     &   $                          

     &                          
#

     &  %                              

     &                                  

   &                         

" #

   &   $                              

   &                               

   &  %                                 

   &                           

    &   $                           

" #

    &                             

" # #

    &  %                             

" #

    &                                 

" #

  $   &                            

  $   &  %                                

  $   &                               

   &  %                               

   &                             

 %      &                          

  
                    


' ( )         

12
Table XI: Case 2: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                       

                                 

                                   

! "

  #      $      $      $            

        $        $        $         
"

 %                                       

           $         $         $      

     &                            $         

     &                              $        

! "

     &   #      $      $        

     &          $         $       $         

"

     &  %                                   

     &                              $        

   &          $         $        $            

! "

   &   #      $      $      $           

   &        $       $       $         

"

   &  %                                    

   &            $         $       $         

    &   #    $     $      $           
! "

    &         $         $        $            

! " "

    &  %                                    

! "

    &             $         $       $        

! "

  #   &      $     $     $            

"

  #   &  %        $      $     $            

  #   &          $       $       $          

   &  %          $        $        $          

"

   &           $        $        $           

"

 %      &                               $        

  
                         $   


' ( )          

13
Table XII: Case 2: Means and standard errors (SE) of the 1st and 2nd order e¤ects on
MeanBias, Clevel and StDi¤ across the other factors.

                      
 

   


                                          


                                        


                                          


                                          


                                          


                                          


                                         


                                          


                                             


                                             

" #  $  


 %                                           


 %                                           


 %                                           

  &  

'   ' 
                                         

 $ 
                                         

#   $  $


                                          


                                          

    

$ 
                                         

( 
                                         

( ( 
                                           

 $  


 &  
                                            


                                          

 &  
                                            

    ) !

                                               

                                              

                                               

                                               

                                               

                                               

                                               

                                              

                                               

                                               

                                              

                                               

                                               

                                               

                                               

                                               

14

                      
 

           

                   ! "       # $ " " %      !       $ 

        ! !           "      # $ " " %      !       $ 

        !           !       # $ " " %             $ 

       ! $           $ #      # $ " " %             $ 

       ! $           ! $ ! !     # $ " " %     !       $ 

       ! ! "                # $ " " %     $       $ 

         #          #  $     # $ " " %      !       $ 

                  $   #     # $ " " %      #       $ 

         #          ! $       # $ " " %             $ 

!        !           !        # $ " " %      $       $ 

!         !                 # $ " " %      #       $ 

!        ! $            #      # $ " " %             $ 

$       ! ! $          $ #      # $ " " %      #       $ 

$       ! ! $          ! !  "     # $ " " %     !        $ 

$       ! ! !                # $ " " %     $        $ 

#       ! $           $   #     # $ " " %             $ 

#       ! $           !        # $ " " %     !        $ 

#       ! $ $           ! " !     # $ " " %     $ $       $ 

         $          $  ! "     # $ " "        $ 

        !           !   $     # $ " " %             $ 

        !              !     # $ " " %             $ 

      ! $          $ #       # $ " " %             $ 

      ! ! #          ! !       # $ " " %     !        $ 

      ! !                   # $ " " %     $       $ 

       &  

 
 ' '           !     !         $ $  %      !        !

 
     ! !      !     !  " #      $ $  %      !        !

 
 ' '    ! $       !     ! #      $ $  %     !         !

 
    ! $       !     !         $ $  %     !         !

 
 ' '      !      !     $  !       $ $  %              !

 
     !      !     $   !      $ $  %              !

 
! ' '           !      " #       $ $  %             !

 
!     ! !      !      " $ !      $ $  %             !

 
$ ' '    ! ! !      !     ! #  $      $ $  %     !         !

 
$    ! ! $      !     ! # $ #      $ $  %     !         !

 
# ' '    ! $       !     !   $      $ $  %     ! !        !

 
#    ! $ !      !     !  $       $ $  %     ! $        !

 
 ' '            !     ! !        $ $  %             !

 
     !       !     ! !       $ $  %              !

 
' '    ! !       !     ! #  "      $ $  %     !         !

 
   ! ! "      !     ! # $ "      $ $  %     !         !

          

 $     # #      !     !   "      $ $  %              !

       #      !     !        $ $  %      $        !

 $    $ !       !      "  "      $ $  %     !         !

      #      !     $ !        $ $  %     !         !

 $     #      !     $         $ $  %      $        !

       !      !     $  $ $      $ $  %             !

! $     #       !      #      $ $  %              !

!       "      !     !  !       $ $  %      "        !

$ $    $        !      "  $      $ $  %     !        !

$      $       !     $  $ #      $ $  %     !         !

# $    $ ! !      !      # #      $ $  %     !         !

#      #       !     !  #      $ $  %     ! #        !

 $     #       !     !   #      $ $  %             !

             !     ! $        $ $  %              !

$    $        !      " $      $ $  %             !

     $      !     $  !      $ $  %     ! $        !

15

                      
 

         

                                           

 !     " "                                    

 !  !     "           " "                        

    " "            "                        

!    "             "  "                      

!  !    "             "                        

                                     

 !     "                                     

 !  !                                         

"                                           

" !     " "                                   

" !  !     " "           "                        

     " "            "                         

 !    " "            "                         

 !  !    " "            " "                        

     " "            "                        

 !    "            "                       

 !  !    "                                     

                                           

 !     "                                    

 !  !                  "                      

     " "            "                        

 !    " "            "                        

 !  !    "            " "                       

     #   # 
 

 $ 
   "                 "                        

                    "                

 $ 
  "                    "                      

 $ 
  "                                "       

    " "            "                 "        

 $ 
 "                                  "        

 $ 
   "                                  "       

      "                             "       

 $ 
  "       "             "                "       

 $ 
"   "     "                                    

"                                         

 $ 
"  "      "             "                       

 $ 
   "                                  "        

     " " "           "                 "       

 $ 
  "                                  "       

 $ 
   "                                  "       

     " "            "               "        

 $ 
  "      "                            "        

 $ 
   "                   "                       

                                          

 $ 
  "                                   "       

 $ 
   "                                         

     " "            "  " "              "       

 $ 
  "                   "               " "       

# %  & '   # 



     (                                     


    (                 "                      


     (                "               "        


     (                                      


    (     "                                 


     (     "            "                     

# %    $  

  
    ) )     "                                   

 
                                            

  
    ) )                                "      

 
                                     "      

# %  '   
 


                                             


                                            


                                     "      


                                           

16

                      
 

       


                                       !    


    "                   #              #   !    


    "  "                 !   !             !  !    


                       #          !    


    "       !             !               !    


    "  "                   #                !    

    $   
 

 %  
      #      !             !             #  !    


                                  #  !    

 %  
     #     #  !              !               !    

 %  
      #                                #  !    


          #                             !    

 %  
     #     #                             #  !    

& '  $     $ %  


  
 (  ) )                  #                 !    

 
 (        !            #                 !    

  
 (  ) )                 #                  !    

 
 (        !           #  # #               !    

  
 (  ) )                   #               !    

 
 (                   #               !    

& '  $    ' $  $

  


 (                                   !  !    


 (       !              !               !    


 (                                   #  !    


 (                  #                !    


 (                                    !    


 (                  #                  !    

& '  $        



 (                             !        #       


 (  "       !                     !              


 (  "  "       !             !       !               


 (                   #          !               


 (  "                  #          !        #       


 (  "  "       #           # ! !        !       #        


 (                    !         !         


 (  "                            !               


 (  "  "                            !        #       

& '  $     $   
  

 %  
 (    #                          !               


 (                           !              

 %  
 (   #     #              !        !               

 %  
 (    #      #                    !        !       


 (        #           ! # !        !               

 %  
 (   #     #             #         !        #       

 %  
 (    #                           !              


 (        #                    !               

 %  
 (   #     #             #         !               

 $ %    ' $  $
 

  
) )                #   #     !        !      

  
) )                #  !      !       ! !      

 
                 #        !        !      

 
                #        !       !      

 $ %        

  
) )                  #  #                !    

  
) ) "                 #               #  !    

  
) ) "  "                  #  #                !    

 
                 # #                 !    

 
 "                 #               #  !    

 
 "  "                  #  # !               !    

 $ %     $   
 

   %  
) )   #                                !  !    

  
) )                  !                #  !    

   %  
) )  #     #            #                #  !    

  %  
   #                  #                !    

 
                  !   #             #  !    

  %  
  #     #             #   #               !    

17

                      
 

           

                               !    
     

                              !    
 " #     

                           !  !    
 " " #  #   

               !               !    
#   # 

             ! !              !    
" # #    

                           !    
" " # #    # #  # 

       $    $  

  %                  !                !    
   # 

                            !    
      # 

 %         !                        !    
    # 

  %       !           !              !    
    # 

    !                     !  !    
# #   # # 

 %                                !    
    #  

      $    $  

   %                         !             
  # #   

               ! !       !             
#  # #    

  %                  !       !             
  #  #   

  %                         !             
" #   #    

             ! !        !             
" # #   # #   

 %                         !            
" #  #  #    

   %                           !       !      
" " #  #    

              !       !             
" " # #  #     

  %         !                   !             
" " # #  #   

Table XIII: Case 3: Analysis of Variance for MeanBias, using Adjusted SS for Tests.
& ' (    )  *  + ,  + ,  ) -

                         
# #  # 

$ .                        !
   

/     $           !
#  

  %          


                          
 #

                        
# #  #    

$    $      !      !      !          
#   

     $ .        

     /     $                     !    
    

       %       

                              
  

                          
 

     $    $   !   !  !   !           
#   

$ .  /     $                    
#      

$ .    %          ! 


$ .                           
#  #  #   #

$ .                        
# #  #   # #

$ .  $    $   !     !     !     !       
#   

/     $     %         !
#  

/     $                        !  
#   

/     $                         
  

/     $   $    $                  !    
# #   # 

  %                
 

  %              
#   

  %    $    $        
#  

                                
# #  #   

       $    $                      !    
# # # # #

      $    $                      
# #     # #

( ( & (                      
 

 &          
  

18
Table XIV: Case 3: Analysis of Variance for Clevel, using Adjusted SS for Tests.

        

                                

                           

                             

! "

  #            

                           
"

 $                                

                              

     %                        

     %                          

! "

     %   #        

     %                          

"

     %  $                              

     %                                 

   %                                

! "

   %   #            

   %                               

"

   %  $                                 

   %                            

    %   #        
! "

    %                               

! " "

    %  $                                 

! "

    %                          

! "

  #   %             

"

  #   %  $           

  #   %                

   %  $                               

"

   %                              

"

 $      %                               

  
                    


& ' (           

Table XV: Case 3: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                   

                                 

                                    

! "

  #        )        )        )            

                                    

"

 $                                    

                                  

     %                                   

     %                              )          

! "

     %   #         )         )       )            

     %                           )          
"

     %  $                                 

     %                                    

   %                                  

! "

   %   #        )        )        )          

   %                                  

"

   %  $                                      

   %                                     

    %   #      )      )      )            

! "

    %          )         )         )            

! " "

    %  $                          )            

! "

    %                                    

! "

  #   %         )        )        )            

"

  #   %  $           )        )        )           

  #   %           )       )       )            

   %  $                                    

"

   %                                       

"

 $      %                                        

  
                          )   


& ' (         

19
Table XVI: Case 3: Means and standard errors (SE) of the 1st and 2nd order e¤ects on
MeanBias, Clevel and StDi¤ across the other factors.

                      
 

   

                                             

                                              

                                              

                                              

                                              

                                              

                                             

                                             

                                                 

                                                 

" #  $  

 %                                            

 %                                            

 %                                            

  &  

'   '
                                             

 $
                                             

#   $  $

                                              

                                              

    

$
                                          

(
                                           

( (
                                            

 $  


#
                                             

) * + , - . - / * + * * * / , / * + * * 0 . 1 * + * * 0 2 - 2 3 * + * / . . 1 * + * * * / 2 2

4 5 6 7 8 9 : 7 7 ; < =

/ / * * * * + * 2 . 0 / * + * * * , > 1 * + . ? , 2 > * + * * 2 * 2 . 3 * + * * 2 > - * + * * * - , ,

/ - * * * * + * 2 . , 1 * + * * * , > 1 * + . . 2 . * + * * 2 * 2 . 3 * + * * , 2 0 * + * * * - , ,

0 / * * * * + 0 1 2 > , * + * * * , > 1 * + , / , > / * + * * 2 * 2 . 3 * + * / / 1 ? * + * * * - , ,

0 - * * * * + 0 1 2 2 . * + * * * , > 1 * + * 1 , 2 > * + * * 2 * 2 . 3 * + * * - * 2 * + * * * - , ,

, / * * * * + * 2 . 0 - * + * * * , > 1 * + . ? , 0 1 * + * * 2 * 2 . 3 * + * / 0 ? / * + * * * - , ,

, - * * * * + * 2 . , ? * + * * * , > 1 * + . . > ? 2 * + * * 2 * 2 . 3 * + * * , > , * + * * * - , ,

. / * * * * + * 2 . 0 0 * + * * * , > 1 * + . > 1 2 0 * + * * 2 * 2 . 3 * + * * 1 2 . * + * * * - , ,

. - * * * * + * 2 . 0 > * + * * * , > 1 * + . . ? * ? * + * * 2 * 2 . 3 * + * * , > 2 * + * * * - , ,

- / * * * * + 0 1 2 - 2 * + * * * , > 1 * + , / , 1 0 * + * * 2 * 2 . 3 * + * / . , 0 * + * * * - , ,

- - * * * * + 0 1 2 ? / * + * * * , > 1 * + * 1 0 > 1 * + * * 2 * 2 . 3 * + * * - / * + * * * - , ,

> / * * * * + 0 1 2 ? - * + * * * , > 1 * + , / 0 - ? * + * * 2 * 2 . 3 * + * / . - . * + * * * - , ,

> - * * * * + 0 1 1 * / * + * * * , > 1 * + * 2 1 ? 1 * + * * 2 * 2 . 3 * + * * . 1 2 * + * * * - , ,

? / * * * * + * 2 . , . * + * * * , > 1 * + . > > ? 1 * + * * 2 * 2 . 3 * + * / - > 0 * + * * * - , ,

? - * * * * + * 2 . - , * + * * * , > 1 * + . . 2 ? - * + * * 2 * 2 . 3 * + * * , 1 2 * + * * * - , ,

2 / * * * * + 0 1 2 > * + * * * , > 1 * + , / 0 - * + * * 2 * 2 . 3 * + * / 0 . > * + * * * - , ,

2 - * * * * + 0 1 2 > * + * * * , > 1 * + * 1 / 1 0 * + * * 2 * 2 . 3 * + * * - * - * + * * * - , ,

20

                      
 

           

                     !   "     # #   $            % 

         !           !       # #   $     %        % 

 "                          # #   $      % "      % 

       #  !           !  %      # #   $     #  !      % 

       #  % %           #  ! !     # #   $     ! # "      % 

 "      #              %       # #   $             % 

"         %           !  #      # #   $             % 

"          %           # # %     # #   $       !      % 

" "                     !       # #   $     %       % 

                    !  %     # #   $     #  "      % 

        " !                  # #   $     % "      % 

 "        % "            %  #     # #   $      # %      % 

      #   %           ! " "     # #   $             % 

      # #               ! "     # #   $     #       % 

"      #             #      # #   $     # " !      % 

%       #              !   "     # #   $     # # !      % 

%       #  % !            % %     # #   $             % 

% "      # #                    # #   $      #      % 

!                   !        # #   $       "      % 

!                    % ! #     # #   $             % 

! "                          # #   $       #      % 

       #             !       # #   $     #  !      % 

       #              # " %     # #   $      !      % 

 "      #  "                   # #   $      "       % 

       &  

 
 ' '       #      " % #    %           $     %       " "

 
      "       " % #    %   !         $     %  !      " "

 
 ' '    #  ! %      " % #     " %          $           " "

 
    #  !       " % #     " ! #         $           " "

 
" ' '       #      " % #    %            $     !  !      " "

 
"      " "      " % #    # #          $       %      " "

 
 ' '             " % #     " !         $     % % #      " "

 
       "      " % #              $     %        " "

 
' '    #  %       " % #     "          $     #  !      " "

 
   #  %       " % #     " " %         $     # #      " "

 
% ' '    #         " % #                $     # %      " "

 
%    #   !      " % #                $     # #       " "

 
! ' '       "      " % #    !  !         $     # "      " "

 
!       "      " % #    ! % !         $      "      " "

 
 ' '    #  %       " % #      " "         $      %      " "

 
    #  #      " % #                $       %      " "

          

      ! #      " % #    "  %         $           " "

 !     "        " % #    % #         $     % %       " "

   "         " % #     # # #         $             " "

 !    # !        " % #    !  #         $      # "      " "

"      #      " % #    "           $     % % #      " "

" !       "      " % #    % % # #         $     # %      " "

       %      " % #     %          $     % "       " "

 !             " % #    %   %         $     !        " "

  "         " % #              $            " "

!    # !  #      " % #     %          $      # !      " "

%   "   " %      " % #     %           $      %      " "

% !    # !       " % #    %           $       %      " "

!      #       " % #     % #         $       %      " "

! !     " #       " % #    %           $             " "

   "          " % #     ! %         $     ! #       " "

 !    # !        " % #    %           $     # %      " "

21

                      
 

         

                                              

 !       "                                      

 !  !                                     " "        

        "                            "          

 !                     "                         

 !  !                     "                         

"                     "                         

" !                                            

" !  !      "                              "          

                      "                        

 !                                     "         

 !  !                                              

                                    "          

 !                       "                       

 !  !                                    "         

                      "              "  "        

 !                                             

 !  !                                              

                                              

 !                                             

 !  !       "                                      

                      "              "          

 !        "               "                       

 !  !       "                                       

     #   # 
 


 $            "                  "          " "

 %            "                               " "

 $             "                               " "

 %             "                         " "

" $             "                       "        " "

" %            "                               " "

 $             "                               " "

 %       "      "                               " "

 $            "                              " "

 %             "                               " "

 $             "                       "        " "

 %     "        "                               " "

 $             "                        "       " "

 %             "                               " "

 $             "                              " "

 %             "                " "        " "

# &  '   # 
$ 

     (       "             "                       "  

     (       "            "                        "  

    " (                  "                        "  

     (                   "  "                     "  

     (                                    "      "  

    " (                   "                       "  

# &    )  

 
    * *                  "    "                      


                      "                           

 
    * *                                    "         


                                                 

# &    
$  

                      "   " "                       

                                                

                      "                          

                      "                           

# &      

                     "   "                     "  

    !                  "   "                    "  

    !  !                   "                        "  

                                              "  

    !                     "                     "  

    !  !                                         "  

22

                      
 

         

       "    
                        !  !     # #     

      "      
    $   # !            !     !  !         

         "   
       #          #     !  !      !     

      "   
   $   # !             !  !      # !     

%        '  
 &

 )  )          "   
 ( &            #      !     #      #

           "   
 (     #        #      !            #

  )  )      "   
( &     !         # #  !     !          #

           "    
(     !        # #      !     #      #

 )  )        "    
# ( &     !        !     !           #

           "   
# (                 !            #

%         
 &  &

          "    
 (           # !  #     !          #

       "   
 (             !  #      !            #

          "   
(    !         #     !            #

          "  
(      #        # #     !     # #      #

         "   
# (    #          !     !      !      #

       "  
# (       #        #      !           #

%         
 & &

       " 
 (     !          #  #       #     #      #

 *       " 
 (      #          #        #           #

 * *         "
 (               #        #     ! #      #

           " 
(              #      #     ! !      #

  *       "
(              # # !      #           #

  * *       " 
(      #          #  !       #            #

          " 
# (                    #     # !      #

 *         "  
# (                    #           #

 * *       "
# (      !          #  !      #       !      #

%            
 &

         "    
 (              #       !           #

        "    
 ( $   # !               !           #

           "    
(     !        !     !     !      #

          "   
( $   # ! #               !     # #      #

        "  
# (     #        #       !     #       #

      "   
# ( $   # !              !     !      #      #

  '       
 &

)  )        "   
&    !         #       !  !      ! #     

)  )       "   
&              #       !  !      !     

        "   
   !         #         !  !            

         "     
             #       !  !         

  '       
&

)  )          "   
&            # #      !     !       #

)  ) *        "    
&            # #  # #     !           #

)  ) * *       "  
&      # #        # #      !      !      #

           "   
           # #      !     !       #

  *        "    
           # #  # #     !           #

  * *        "   
     #         #     !     #       #

  '          

)  )            "    
&                 !  !     ! #     

)  )        "   
& $   # !           ! #     !  !     !  !     

           "    
    #         #     !  !     ! !     

          "     
$   # !               !  !     !     

        
 & &

        "    
   #        #       !     !      #

 *         "   
   !        #        !      !      #

 * *            "  
    !            !      #       #

      "  
              # ! # #     !     ! !      #

*         "   
             #       !     !      #

* *          "    
             #      !     !      #

           
 &

     "     
    # ! !          #     !  !     #     

       "      
$   #            ! !     !  !         

        "      
                   !  !         

          "    
 $   #               !  !           

23

                      
 

  
         

            !   
                      "      

        !     

 #    $                        

           !      

%     "         "              

        !     

% #    $                   "     

           !      
% " %            $ "              

           !     

% " % #                    " "     

Table XVII: Case 4: Analysis of Variance for MeanBias, using Adjusted SS for Tests.
& ' (    )  *  + ,  + ,  ) -

              

   $   "   "   

 !   !   !   

 . "            "   $

 

/           

  !   !   !  

  0  "            "    $ 

     

      "     $     $     $ "  

         
    $  $  " $  $  "  $ $ $ $  

         
     "  $ $  $ $  "     

 !  !  !  

     . $         "       "

  !  

    /      "       "      " "       "

  !  ! 
      0  $ "    "        " "

       

          $    "      "         

                 

        " "   "     $ "  $ 

          

         " "  "  $  "  $    " " " $  " 

  !  !  !   

 .  /      "    "    "         

 

 .    0  "      

 .        "       "    

   !   !  !  
 .            "         $

  !  !   !   

 .       "                 

  

/         0      "

  !  !  

/             "    "        "  

   !  ! 

/           "    "      "

  !  !  !   
/            "          "        

 

  0         "       "

    !  

  0            "       "       "    $

    !   !  !  

  0        "       "          $

     
                $      $     " $   $     

   !    

            "      "      "      "   $

           

          "   $   $     "   

     

( ( & ( " $   $ " "   $ " "     

      
 &      "

24
Table XVIII: Case 4: Analysis of Variance for Clevel, using Adjusted SS for Tests.

        

                             

                              

                           

! "

  #                              

                       
"

 $                          

                             

     %                             

     %                             

! "

     %   #                          

     %                          

"

     %  $                               

     %                           

   %                            

! "

   %   #                             

   %                               

"

   %  $                              

   %                                 

    %   #                           
! "

    %                               

! " "

    %  $                              

! "

    %                                 

! "

  #   %                               

"

  #   %  $                                

  #   %                                

   %  $                              

"

   %                                  

"

 $      %                               

  
                


& ' (            

Table XIX: Case 4: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                    

                                     

                                      

! "

  #      )      )      )          

       )      )      )       

"

 $                                   

           )       )      )            

     %                                   

     %                           )        

! "

     %   #      )      )         

     %         )       )       )           
"

     %  $                                   

     %                              )       

   %           )        )        )        

! "

   %   #      )      )      )            

   %          )        )        )           

"

   %  $                                    

   %            )       )       )      

    %   #      )      )      )            

! "

    %         )        )      )           

! " "

    %  $                                    

! "

    %             )        )        )        

! "

  #   %       )      )      )            

"

  #   %  $         )      )      )           

  #   %            )        )        )           

   %  $            )        )       )           

"

   %             )        )      )           

"

 $      %                               )        

  
                       )   


& ' (            

25
Table XX: Case 4: Means and standard errors (SE) of the 1st and 2nd order e¤ects on
MeanBias, Clevel and StDi¤ across the other factors.

                      
 

   


                                         

  
                                        

  
                                       

 
                                        

 
                                        

  
                                        

   
                                        


                                          

   
                                         

   
                                          

" #  $  

    
 %                                       

    
%                                        

  
 %                                         

  &  

'   '    
                                      

 $   
                                      

#   $  $

  
                                        

 
                                        

    

$   
                                      

(  
                                       

(    
                                     

( (   
                                        

 $  


 &    
                                          

   
                                       

 &   
                                           

    ) !

                                               


                                             


                                              

  
                                           

                                               


                                              

 
                                             

  
                                           

 
                                             

   
                                           

 
                                             

   
                                           


                                             


                                               

                                               

 
                                             

26

                      
 

           

              ! " #    " "     #  !  $      "        

        "       ! " #    %       #  !  $              

        "      ! " #     " %     #  !  $              

        # "      ! " #     " !     #  !  $      !        

        #       ! " #      !     #  !  $              

        #       ! " #    #  #     #  !  $      #        

        ! #      ! " #   " #  #     #  !  $             

        #       ! " #    ! !     #  !  $              

        ! #      ! " #     #      #  !  $              

               ! " #    % #     #  !  $              

         "      ! " #    " !      #  !  $      %        

               ! " #    #      #  !  $              

"               ! " #     " !     #  !  $      "        

"               ! " #      %     #  !  $     "         

"              ! " #    ! % "     #  !  $     %         

%               ! " #           #  !  $              

%               ! " #           #  !  $      %        

%              ! " #    %      #  !  $              

   $    "        ! " #           #  !  $      #        

   $    "      ! " #         #  !  $              

   $    "        ! " #    !     #  !  $              

!        #      ! " #     #      #  !  $      %        

!        "      ! " #     %     #  !  $      %        

!             ! " #    !  !     #  !  $     " "        

       &  

 
 ' '      !      "  !    #     " " # $              !

 
     "       "  !    ! %     " " # $              !

 
 ' '     #       "  !    "      " " # $              !

 
     #       "  !    "      " " # $              !

 
 ' '     ! !      "  !   "        " " # $              !

 
     #       "  !   "        " " # $              !

 
 ' '      !      "  !          " " # $             !

 
            "  !     !     " " # $      !        !

 
" ' '            "  !     %      " " # $     "        !

 
"            "  !     #      " " # $     "         !

 
% ' '            "  !      %     " " # $              !

 
%      "      "  !     % !     " " # $              !

 
' ' $    "        "  !           " " # $              !

 
$    "      "  !           " " # $              !

 
! ' '     %      "  !     #     " " # $      %        !

 
!     !      "  !    "  !     " " # $      %        !

          

 "          "  !     "     " " # $              !

           "  !    ! "      " " # $      "        !

 "     %       "  !    # ! "     " " # $              !

      "      "  !     % %     " " # $              !

 "            "  !    #  !     " " # $              !

     % #      "  !   "        " " # $      %        !

 "            "  !    % %      " " # $             !

     #       "  !    !  #     " " # $      !        !

" "      "      "  !    #  %     " " # $      #        !

"     " !      "  !          " " # $     "         !

% "    "        "  !    ! # #     " " # $              !

%     %      "  !    %  "     " " # $      "        !

" $     % "      "  !      "     " " # $      #        !

$    "  #      "  !     # #     " " # $              !

! "    # #      "  !    #       " " # $              !

!    %  "      "  !     " %     " " # $      #        !

27

                      
 

         

                                          

 !                   "                       

 !                                          

 !  !                                         

       "            "                       

 !                                    

 !                                        

 !  !                                       

                                    

 !                    "                      

 !                                        

 !  !       "             "                      

                                         

 !                                        

 !                                          

 !  !                                         

                   "  "                     

 !                                    

 !       "                                 

 !  !                                         

                    "                     

!      "                        

!                                      

!  !                                       

                                          

 !                                          

 !                    "               "      

 !  !                                          

"                   "                       

" !                                          

" !       "                                

" !  !                   "                     

     #   # 
 

 $ 
               "               "                

             "               "                

 $ 
             "               "                

 $ 
       "       "               "               

      "       "              "                

 $ 
       "       "             "                

 $ 
              "               "                

      "       "               "                

 $ 
              "               "                

 $ 
               "       " "       "        "        

             "     "          "                

 $ 
               "              "                

 $ 
              "              "                

             "               "               

 $ 
              "        "       "               

 $ 
             "        "       "               

            "              "                

 $ 
     "        "               "                

 $ 
        "       "       "        "                

              "     "         "                

 $ 
        "       "              "               

 $ 
"              "               "                

"             "              "               

 $ 
"        "      "             "                

# %  & '   # 



     (                 "                      


     (                                         


     (                  "  "              "       


     (                  "                      


     (                                        


     (      "                                  

28

                      
 

       

 
                      !          "     !   "   

 # 
        $           !  !        "     !    "   

 
                                "         "   

 # 
                             "         "   

   %  #  #



                    ! ! ! !       "     $    "   


                              "     ! $   "   


                     $        "         "   


                             "         "   

     
 &

# 
                $               "      $   $ "   


    '      $        $              "      !   $ "   


    '             $    !           "     !    $ "   


    '  '              $               "         $ "   

# 
                $   $  !             $ "   


    '              $               "      !   $ "   


    '              $   $           "        $ "   


    '  '              $   $ !          "         $ "   

     #  
& 

  
    "   !    $                  ! "     ! $    "   


          $          $        ! "     $    "   

  
      ! "     $ !                 ! "     $    "   

  
    "   !     !          ! ! $      ! "          "   


                  !        ! "          "   

  
      ! "     $ $         $ $      ! "          "   

( % #         


 
 )                   ! ! !      ! "          "   

 # 
 )      $         !         ! "          "   

 
 )                   $  !       ! "         "   

 # 
 )               $   !      ! "         "   

 
$ )                   $  !       ! "     $     "   

 # 
$ )      $         $         ! "     $     "   

( % #     %  #  #
 


 )                 !         ! "         "   


 )                 !        ! "      !    "   


 )                 $   $      ! "          "   


 )                $        ! "     $    "   


$ )                          ! "     $ !    "   


$ )                $ $ ! !      ! "     $     "   

( % #       
  &

#
 )            $            $ !  "      !        

 )  '      $      $        $     $ !  "     $         

 )  '            $            $ !  "     $        

 )  '  '            $             $ !  "              

#
 )            $     !       $ !  "              

 )  '            $             $ !  "     !        

 )  '            $     !        $ !  "     $ !        

 )  '  '      !      $     !       $ !  "     !         

#
$ )            $     $       $ !          

$ )  '      $      $            $ !  "     !         

$ )  '            $     $        $ !  "     $        

$ )  '  '            $     $      $ !  "      !        

( % #       #  
 & 

  
 )  "   !            $     ! ! $         "          "   


 )        !       $               "         "   

  
 )    ! "     $ !       $                "         "   

  
 )  "   !            $     $ !         "         "   


 )              $      $          "     $    "   

  
 )    ! "     $ $       $    !            "         "   

  
$ )  "   !            $                "     $ $    "   


$ )              $    !   !         "     $     "   

  
$ )    ! "     $ $       $    $  $          "     $     "   

      %  #  #


 
                   $ !         "         "   

 
                   $         "     $   "   

 # 
                $ !         "         "   

 # 
                !          "     $   "   

29

                      
 

          

                      !        " #           #   

    $     "              !         " #      !     #   

    $      !             !        " #          #   

    $  $      !             !        " #      !     #   

       !                     " #           #   

  $     "              !         " #      !     #   

  $      !             !        " #          #   

  $  $      !             "        " #      !     #   

      %    %  

    #         !      !  !    ! !     !  #      !     #   

        "        !  !           !  #           #   

        #            !  !        !  #           #   

  #              !  !     ! !     !  #          #   

      "        !  !          !  #           #   

      #            !  !     !     !  #           #   

&           

                  !  !        " #           #   

 $     "                       " #      !     #   

 $                  !         " #           #   

 $  $                          " #      "     #   

                  "         " #           #   

 $     "  "                     " #         #   

 $                " "         " #           #   

 $  $                             " #          #   

&       %    %  

 #               !  !     "     !  #           #   

     "        !  !     "     !  #           #   

                !  !    !     !  #           #   

 #               !  !           !  #           #   

             !  !           !  #      !     #   

     #           !  !        !  #      !     #   

      %    %  

 #         !                    !  #              

                 " "        !  #              

     #                   "      !  #              

$  #                             !  #              

$      "                     !  #            

$                              !  #             

$  #          !                   !  #              

$                   " "        !  #              

$      #                   !      !  #              

$  $  #                             !  #     "        

$  $                   "         !  #              

$  $      #      "                  !  #              

30
Table XXI: Case 5: Analysis of Variance for MeanBias, using Adjusted SS for Tests.

        

                                 

            

             

  "            #

          #       #       #   # #    
!

 $                                 # 

                                 

     %       &      &      

     %       #  &    #  &      &        

     %   "        

     %                                 

     %  $                        #           

     %                        #       #  #  

   %        &      &      &         

   %   "            

   %       &      &      &            

   %  $         &      &      &            

   %          &      &      &          #  

    %   "        
!

    %       &      &          

! !

    %  $         &      &      

    %          &      &      &           # 

  "   %             

  "   %  $           

  "   %                

   %  $                               #     

   %                                  

 $      %          #      #           #      

  
                     


' ( )            

Table XXII: Case 5: Analysis of Variance for Clevel, using Adjusted SS for Tests.

        

                                

                   

        #     #            # 

  "            

         #      #      # #     

 $                         # #    

        #       #       #      #       

     %                            

     %                                

     %   "        

     %                         
!

     %  $                              

     %           #       #         #     #  

   %                          #   #  #

   %   "            

   %                               

   %  $                                

   %           #  #    #  #    #  #      

    %   "        

    %          #      #          #     

! !

    %  $                             

    %          #      #         #     

  "   %             

  "   %  $           

  "   %               # 

   %  $          #      #                 

   %          #      #      #         

 $      %                      #    #  #     

  
                        


' ( )           

31
Table XXIII: Case 5: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                      

                                  

                                    

! "

  #        $        $        $            

                                   
"

 %                                     

                                      

     &                                 

     &                             $            

! "

     &   #        $        $        $          

     &                           $            

"

     &  %                                       

     &                                       

   &                                   

! "

   &   #         $         $         $           

   &                                     

"

   &  %                                      

   &                                     

    &   #      $      $      $           
! "

    &         $        $         $           

! " "

    &  %                                     

! "

    &                                      

! "

  #   &         $        $        $            

"

  #   &  %            $         $        $            

  #   &             $         $         $           

   &  %                                       

"

   &                                        

"

 %      &                                    

  
                          $   


' ( )          

32
Table XXIV: Case 5: Means and standard errors (SE) of the 1st and 2nd order e¤ects
on MeanBias, Clevel and StDi¤ across the other factors.

                      
 

   

                                           

                                           

                                          

                                           

                                           

                                           

                                            

                                          

                                             

                                             

" #  $  

 %                                           

 %                                            

 %                                            

  &  

'   '
                                          

 $
                                          

#   $  $

                                           

                                           

    

$
                                          

(
                                          

(
                                           

( (
                                            

 $  


#
                                          

)
                                          

    * !

                                               

                                               

                                              

                                               

                                              

                                               

                                               

                                               

                                               

                                               

                                               

                                               

                                               

                                                

                                               

                                               

33

                      
 

           

                !   " !       "   #            $ " $

         $       !   " $  "     "   #           $ " $

 "        "       !   "       "   #     " $      $ " $

      "         !     " !     "   #            $ " $

      "         !         "   #           $ " $

 "     "          !           "   #           $ " $

"       ! $ "       !   $  "     "   #            $ " $

"       ! %       !          "   #     !       $ " $

" "      ! $       !   " %       "   #      %      $ " $

      %  $       !    % !     "   #     !      $ " $

      !  $       !    $     "   #     $       $ " $

"      %  $       !    %      "   #     "       $ " $

$      "        !      $     "   #            $ " $

$      "         !    " % $     "   #            $ " $

$ "     "         !          "   #      "      $ " $

            !          "   #     !       $ " $

      "          !    "       "   #     %       $ " $

 "       "       !     %      "   #     !       $ " $

!    #    " $ %       !   " $     "   #           $ " $

!    #    " $ !       !   "       "   #           $ " $

! "   #    "        !   " " "     "   #      $      $ " $

%       % "        !         "   #     %       $ " $

%       % "        !    $ "     "   #     !       $ " $

% "      % " %       !      %     "   #     ! !      $ " $

       &  

 
 ' '      "     "   " $ !      ! "  #     !      " !

 
      "     "   " $ ! $      ! "  #     %      " !

 
 ' '   "   $     "    $  %      ! "  #           " !

 
   "   $     "    $  %      ! "  #      $      " !

 
" ' '    ! $     "    "       ! "  #     ! !      " !

 
"    ! $     "    "       ! "  #     %       " !

 
' '    %       "   $  "      ! "  #     $      " !

 
   %       "   $       ! "  #     $ $      " !

 
$ ' '   "  "     "    $       ! "  #           " !

 
$   "  "     "    $  $      ! "  #           " !

 
 ' '          "    $        ! "  #     !       " !

 
       "    $        ! "  #     !       " !

 
! ' ' #    " $ %     "   " "      ! "  #          " !

 
! #    " $ %     "   " "       ! "  #      %      " !

 
% ' '    % " %     "    $        ! "  #     ! %      " !

 
%    % " %     "    $        ! "  #     !       " !

          

 $      %     "   " $       ! "  #          " !

 !     $ !     "   "  " $      ! "  #     $       " !

 $   "  " !     "      $      ! "  #            " !

 !   "        "     "       ! "  #      %      " !

" $    ! $      "    "       ! "  #      %      " !

" !    !      "    "       ! "  #           " !

$    %  !     "    $      ! "  #     $      " !

!    !  !     "           ! "  #           " !

$ $   "       "           ! "  #     %       " !

$ !   " "       "          ! "  #            " !

 $          "      !      ! "  #      %      " !

 !       "            ! "  #     %       " !

! $ #    "      "   " " %      ! "  #      "      " !

! ! #    "       "   " %       ! "  #            " !

% $    %  %     "      %      ! "  #     !       " !

% !    %  !     "     " "      ! "  #     %       " !

34

                      
 

         

                                           

      ! !                                    

 !                                  "        

  !      !                                    

!    ! "            !                          

!                              !        

! !   ! "  "          !                 !        

!  !   ! "            !   !                       

                                           

                 !  !                !        

 !                  ! !                      

  !      "                                   

       !                            "        

                                          

 !                                          

  !      !            "                       

    ! "            !  !                        

      "                                

 !   ! "            !                          

  !   ! "            !                          

    ! "  "          !                          

                                      

 !   ! " "           !                        

  !   ! "            !                          

                   !                       

      "                        !         

 !                  "                        

  !                   "                       

    ! "            !                          

    !  "                             !        

 !   ! "            !                  !        

  !   ! "            !                 !         

     #   # 
 !

 $                   "                        

 %                                 "         

! $                                           

! %                             "         

 $                   "               !         

 %                    !                       

 $    !               !                "        

 %     " "            "                       

 $    ! !               "                      

 %                                         

 $     !                              !        

 %                                 ! !        

 $                                    !        

 %     !  !                                     

 $      !                                    

 %    "                                       

# &  '   # 
$ 

     ( !   !        !   "                    !       !  

    ! ( !   !        !   "    !  "                     !  

     ( !   !        !   "   ! "                "       !  

     ( !   !        !   "   !                  !      !  

    ! ( !   !        !   "   !                  "      !  

     ( !   !        !   "     "                     !  

# &    )  

 
    * *   !          ! !     !                      !  "


       !          ! !     !                !      !  "

 
    * *   !          ! !   !                  "      !  "


       !          ! !   !                       !  "

# &    
$  

       !   "       ! !                    "       !  "

        "         ! !                    !       !  "

       !          ! !      !                     !  "

        " "        ! !   !                  !      !  "

35

                      
 

       

                            !    "              

    #      !                    !    "              

    #       !                   !    "              

    #  #                  !      !    "      !        

                             !    "              

    #      !                   !    "      !        

    #                   !       !    "              

    #  #                   !       !    "              

    $   
 

    %            !                "              

    &           !               "             

    %           !       !         "      !        

    &           !                "              

'  $     $ (  
% 

 
 )  * *                          !  "            


 )                            !  "            

 
 )  * *                           !  "     !        


 )                             !  "     !        

 
 )  * *                   !       !  "      !       


 )                     !       !  "             

'  $    $  $
%  %  

 )                          !  "     !        

 )  !                  !        !  "             

 )                           !  "             

 )  !                          !  "            

 )                            !  "             

 )  !                          !  "     !        

'  $        
% 

 )                   !  !        ! "             !

 )  #      !                     ! "             !

 )  #       !           !          ! "            !

 )  #  #                  !         ! "             !

 )                            ! "             !

 )  #      !                    ! "             !

 )  #                            ! "           !

 )  #  #                            ! "             !

 )                    !         ! "             !

 )  #      !                      ! "             !

 )  #                           ! "     !        !

 )  #  #                    !        ! "            !

'  $     $   
%   

 )  %                           !  "            

 )  &                   !       !  "            

 )  %                          !  "            

 )  &     !                     !  "             

 )  %                          !  "             

 )  &                          !  "           

 $ (    $  $
%  

 
* *             !       !         "              

 
* * !           !                "             


             !       !         "             


 !           !                "             

 $ (        

 
* *                          !    "      !        

 
* * #      !                  !    "              

 
* * #       !                   !    "             

 
* * #  #                          !    "              


                          !    "      !        


 #      !                  !    "              


 #       !                   !    "             


 #  #                          !    "             

 $ (     $   
 

 
* * %            !                "             

 
* * &     !      !               "              


 %            !               "             


 &     !      !               "              

36

                      
 

           

                                !        
  

                 # #                     
 "  

                              #        
 "    

                   # #                     
 " "  

       #           !                       
 

                   !                     
"  

       #           !              #        
"   

                    #                     
" "   

       $    $  

    !                !      ! #              
 

    !             !       ! #      #        
 %  

                        ! #              
   

      #                  ! #             
%  

      $    $  

      #           ! !                      
  

     #                                 
%   

     !              #                    
"  

                 ! #                    
" %    

      #          ! !                       
"   

     #                                
" %    

       #           !              #        
" "    

                                      
" " %   

Table XXV: Case 6: Analysis of Variance for MeanBias, using Adjusted SS for Tests.
& ' (    )  *  + ,  + ,  ) -

                             


$ .                          !  #

/     $             ! 

  0                        #     ! 

                      !      


                                !

$    $     !   #  !   #   #  #    !    ! 
 

$    $        #       #       #         

     $ .                   

     /     $                    


       0           

                 !      !             !  !

                                      

     $    $        !     !     ! # # !     
  

     $    $          !       !            

$ .  /     $             ! 

$ .    0           # 

$ .                    

$ .                             #   ! # !

$ .  $    $                    !   # 


$ .  $    $                             

/     $     0            

/     $                #     

/     $                

/     $   $    $                    !    


/     $   $    $              # 

  0                   

  0                            #      
 

  0    $    $        !      !           # # 

  0    $    $                              

                      
 

       $    $         !      !               

       $    $       #  #    #  #    #  #  #    

      $    $                  #   !   #  

      $    $                   #   !     
  

$    $    $    $                !  #  #     # 
  

( ( & (    !                  

 &          
   

37
Table XXVI: Case 6: Analysis of Variance for Clevel, using Adjusted SS for Tests.

        

                            

                              

                             

! "

  #                              

                           
"

 $                           

                                 

                                  

     %                               

     %                             

! "

     %   #        

     %                        

"

     %  $                               

     %                                 

     %                           

   %                            

! "

   %   #                         

   %                              
"

   %  $                               

   %                                 

   %                                  

    %   #                          

! "

    %                               

! " "

    %  $                              

! "

    %                               

! "

    %                            

! "

  #   %                              

"

  #   %  $                               

  #   %                                

  #   %                               

   %  $                               

"

   %                                  

"

   %                        
"

 $      %                             

 $      %                            

      %                                  

  
                      


& ' (            

38
Table XXVII: Case 6: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                    

                                 

                                     

! "

 #                                      

                                     
"

 $                                

                                       

                                   

     %                                   

     %                              &            

! "

     %  #                          &         

     %                                    

"

     %  $                                    

     %                                        

     %                                     

  %                                         

! "

  %  #        &        &        &            

  %         &        &        &            

"

  %  $                                    

  %                                        

  %                                      

    %  #        &        &      &            

! "

    %          &         &         &          

! " "

    %  $                                        

! "

    %                           &            

! "

    %                                        

! "

 #   %       &      &      &            

"

 #   %  $                                    

 #   %             &         &         &            

 #   %                                           

   %  $                                    

"

   %                                       

"

   %                                         
"

 $    %                                      

 $    %                                     

      %                                       

 
                        &   


' ( )           

39
Table XXVIII: Case 6: Means and standard errors (SE) of the 1st and 2nd order e¤ects
on MeanBias, Clevel and StDi¤ across the other factors.

                      
 

   

                                              

                                              

                                              

                                              

                                            

                                              

                                              

                                              

                                                

                                                 

" #  $  

 %                                              

 %                                             

 %                                              

  &  

'   '
                                             

 $
                                         

#   $  $

                                            

                                              

    

$
                                           

(
                                             

( (
                                              

 $  


 & 
                                              

                                             

 & 
                                             

 $  


#
                                            

)
                                             

    * !

                                                  

                                                  

                                                 

                                                  

                                                  

                                                

                                                  

                                                 

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                 

40

                      
 

           

                    !  "         #      ! !      ! $ !

         % "             !        #       !      ! $ !

         %           % $ !        #     %        ! $ !

       "   %            "        #      "       ! $ !

       "             %  $        #     " !      ! $ !

       "  $ !          "  "         #     " $ $      ! $ !

                    !        #     ! $      ! $ !

                   $ " "         #            ! $ !

                              #       $      ! $ !

        " $          !           #     "  "      ! $ !

         "          " "         #     ! ! %      ! $ !

        % !          ! $ $         #     $        ! $ !

$       "  "            %        #     "      ! $ !

$       "  " !            !         #      "      ! $ !

$       "            "          #             ! $ !

!       "  $           ! "  $        #      " "      ! $ !

!       "  % "           %          #       !      ! $ !

!       " "  %          % %        #     " $       ! $ !

%         $ !                    #       $      ! $ !

%                   ! ! "        #            ! $ !

%         !           %          #     "       ! $ !

       " "  "            !         #     " ! !      ! $ !

       "  %            !         #             ! $ !

       "  %             $         #      !      ! $ !

       &  

 
 ' '      %        %      "      " $  #            $  $

 
       %       %      %        " $  #      "       $  $

 
 ' '    "         %           " $  #     "        $  $

 
    " % " !       %      !  !      " $  #       %      $  $

 
 ' '      "       %     !  ! $      " $  #      $ "      $  $

 
      $       %     %  $      " $  #     "       $  $

 
' '      %        %      % $ %      " $  #     ! "       $  $

 
            %      " $       " $  #     %       $  $

 
$ ' '    " "        %       "       " $  #      !       $  $

 
$    "        %        %      " $  #     !       $  $

 
! ' '    " "         %      $  !      " $  #      ! $      $  $

 
!    "  $       %      % "       " $  #            $  $

 
% ' '      "        %             " $  #      $      $  $

 
%      !        %       ! !      " $  #            $  $

 
 ' '    " "         %       " %      " $  #      "       $  $

 
    "  $        %       " !      " $  #     "  !      $  $

          

 $      " !       %              " $  #     %        $  $

 %              %             " $  #            $  $

 $              %     " "      " $  #      " !      $  $

 %    " ! % $       %     %   $      " $  #             $  $

 $      "       %     ! $       " $  #             $  $

 %     !       %     % !      " $  #     % !       $  $

$      !        %      %       " $  #     !  %      $  $

%             %              " $  #     % !       $  $

$ $              %     " $ " %      " $  #      $ !      $  $

$ %    " %  %       %     !         " $  #     ! ! "      $  $

! $       "       %     "   %      " $  #     " $      $  $

! %    " %        %      %      " $  #       "      $  $

% $              %      $ $      " $  #     "  "      $  $

% %      %        %       $ $      " $  #       "      $  $

 $       $       %     " % $      " $  #             $  $

 %    " % $ "       %     ! "        " $  #     " " %      $  $

41

                      
 

         

                                      !      ! !

 "                                     !        ! !

 "  "                                          ! !

                                            ! !

 "                                            ! !

 "  "      ! !                                   ! !

                     !                !       ! !

 "                                    !        ! !

 "  "                                     !      ! !

                                            ! !

 "                       !              !        ! !

 "  "                                            ! !

                                         ! !

"                                            ! !

"  "                                      !      ! !

!                                      !      ! !

! "                                             ! !

! "  "        !                                     ! !

                      ! !                    ! !

 "        !                             !        ! !

 "  "                                           ! !

                                            ! !

 "       !                                !      ! !

 "  "                                            ! !

     #   # 
 

 $ 
       !  !                                      ! !

                   !                 !        ! !

 $ 
                                              ! !

 $ 
                       !                        ! !

                                            ! !

 $ 
       !                                    ! !

 $ 
        !                             !        ! !

                    !                         ! !

 $ 
         !             !                  !      ! !

 $ 
                      !                         ! !

                                    !  !      ! !

 $ 
        !                                      ! !

 $ 
                      !                       ! !

      !                                      ! !

 $ 
                    !                      ! !

 $ 
!                      !                       ! !

!                                       !      ! !

 $ 
!      !                                     ! !

 $ 
       !  !               !                      ! !

                                           ! !

 $ 
                                      !       ! !

 $ 
          !               !               !      ! !

       !                                    ! !

 $ 
                                             ! !

     #   # 
 

 %                                           

 &                                !        

 %                                         


 &    !                                

 %      !                                    

 &      !                                   

 %                                         

 &                                         

%                  !                      

&                                 !      

! %                                         

! &                    !               !      

 %       !                              !      

 &                  !                       

 %       !            !                  !      


 &                                      

42

                      
 

         

                       !   "        #           !  "

             $           "          #     $        !  "

    !        !            ! ! "        #             !  "

%            ! %            $         #     "      !  "

%            $ !           % " % "       #      %       !  "

%    !        !            $   %       #     %       !  "

     &  

 
    ' '      ! "      % "            % $  #     $          "

 
         " !      % "       %     % $  #     %  !        "

 
%    ' '       $      % "      %       % $  #      !         "

 
%                % "      $ " $     % $  #       "        "

        

    %             % "      "      % $  #               "

          %      % "    !  !       % $  #      ! $        "

%    %       $      % "     $ %       % $  #     %  $        "

%         !      % "     " $      % $  #     " ! !        "

      ( 


         "            " !        #           !  "

    )        $           "  %       #      $ %      !  "

    )  )       !           !    $       #     % !      !  "


%         " !                     #             !  "

%    )                   % " %        #     %      !  "

%    )  )       !            $ ! %       #       $      !  "

      (  


 & 
    #  $    $                        #       $      !  "

          $ %          !    %       #      %       !  "

 & 
     $    $  %          !  ! %       #     % !       !  "

 & 
%    #  $    $  " %             % "       #           !  "

%          %                      #     $ $       !  "

 & 
%     $    $             % $ %       #     " $ !      !  "

      (  



         % "      % "    %   "     % $  #     ! % "        "

    *   ! %   !      % "      " %      % $  #     %         "


%         !      % "    !    $     % $  #      $         "

%    *   ! % ! $      % "       !      % $  #       %        "

          &  

 
   ' '      $ $           "  !        #            !  "

 
        "           " !       #      %      !  "

 
   ' '      %             " !       #       %      !  "

 
                     "        #            !  "

 
!   ' '      %            "        #     "        !  "

 
!                    " " $ !       #            !  "

             

   %       "            $ %       #      % %      !  "

         !             $ $       #     !  !      !  "

   %                           #     "       !  "

        %           !  !        #            !  "

!   %                   " "        #     "      !  "

!                      %       #             !  "

           ( 


        " !      " " $      $     "   " #      !       $  

   )              " " $      %     "   " #       !      $  

   )  )              " " $   !   $      "   " #             $  


        "       " " $      %     "   " #     $ $       $  

   )             " " $      ! !     "   " #      %      $  

   )  )       "       " " $    !  $      "   " #     " $      $  


!        "       " " $    " ! %      "   " #     ! %       $  

!   )        !      " " $     ! !     "   " #           $  

!   )  )        "      " " $      !      "   " #      "      $  

43

                      
 

              

           ! " " #     $ $    % % %    $   $     ! % !       

      "  %      $ $   ! #   #    $   $     " $        

            " %     $ $   !  $  %    $   $     ! ! %       

             % &     $ $    $ #      $   $             

      "   &     $ $   !  ! # !    $   $     $  "       

            & "     $ $     $ & !    $   $      # &       

!             "      $ $    % % ! &    $   $       !       

!      "  % %     $ $    " & "     $   $     # ! #       

!            %      $ $      "     $   $      % !       

              

       "      #    %   !     & #       $ #     !  $

   '  ! %  "      #      " & #    & #       &     !  $

       "       #     ! ! ! !    & #       % !     !  $

   '  ! % !  !     #      &  #    & #      $       !  $

!       "  #     #    ! # ! #     & #      !      !  $

!   '  ! % !       #        !    & #      % % "     !  $

           

(   ( %   "   %     % $ "     % $    %  # "     $ & #      & $

(   ( #   "  # #     % $ "    #  &    %  # "      "       & $

  %   "  # %     % $ "     "     %  # "     "  #      & $

  #   "       % $ "   %   #    %  # "      # !      & $

          

(   (    "  & &     #      " % $    & #       $     !  $

(   ( )    "   #     #       %     & #      $  !     !  $

(   ( )  )    "  $     #      & &     & #      # $ "     !  $

     "        #      % &     & #       $     !  $

  )    "   #     #       %     & #      $  !     !  $

  )  )    "  $      #      & $     & #      " " $     !  $

            

(   (         ! # %      #     " % "     & #             !  $

(   (    "  %      #    ! !  " "    & #      $  $     !  $

(   (         % %     #     % " $ %    & #             !  $

           !  &     #       & #    & #             !  $

     "  %      #    ! !  " &    & #      $ &      !  $

         ! # " %     #     % " ! %    & #       !      !  $

            

(   (     "  %     % $ "    " &     %  # "       $      & $

(   ( '  ! % ! # $     % $ "    $  !    %  # "     $  !      & $

      " & !     % $ "   %  !     %  # "      %       & $

  '  ! %  !      % $ "     & "    %  # "     "  $      & $

           

%    "  %     #      ! $     & #       &     !  $

% )    "  $      #       ! #    & #      # $ $     !  $

% )  )    "  & "     #      # %     & #             !  $

#    "   "     #       !     & #      %     !  $

# )    "  !      #     ! $ #     & #      $ % $     !  $

# )  )    $ " " !     #     & % " %    & #      !  !     !  $

             

%                 #     $       & #      " & #     !  $

%    "   %     #    !    "    & #      # % #     !  $

%          !      #     % !      & #      " & $     !  $

#         ! $ # &     #      $  &    & #      ! % #     !  $

#    "  % "     #    ! &  # #    & #      " ! &     !  $

#                #     & &  $    & #        #     !  $

             

%    ! %  %     % $ "          %  # "      !       & $

% '  !  $ $ %     % $ "      &    %  # "     % &       & $

#     ! # !     % $ "   $ $  $    %  # "      & #      & $

# '  ! % #  %     % $ "    $ $ %    %  # "           & $

44

                      
 

  
         

                   !   "           #          
    

      #         " "                #        

 

                    "             !         

   

       #              ! #                     

$    

       ! #         " "  " "            !        

$ 

       "   "          # " !                    

$   

         #           #                       

$ $    

        "          "   " "                    

$ $ 

        #  #          #  "            #         

$ $   

  

         

                 "  #      !               " 

 % 

  " #  !              !     !               " 

 & 

       "                    !       "       " 

$ % 

   " #                  "     !       "  #      " 

$ & 

                     #     !       "        " 

$ $ % 

    " #   "                   !      " "  !      " 

$ $ & 

   
          

        " "                    !              " 

   % 

     "   !                "     !        #       " 

   & 

     #            ! #   !     !         "      " 

% 

   " #  #                     !               " 

& 

        #                #     !        " !      " 

  %  

     "  #               #     !        "       " 

  & 

Table XXIX: Case 7: Analysis of Variance for MeanBias, using Adjusted SS for Tests.
' ( )    *  +  , -  , -  * .

        " #       " #  #          

  

          

 /

          !

0 %    

                       "     #

   

                #  #     
%     

 "     #     #   " !   #   # 

   

   !  !  " " !  !  " " "  " " ! !  "    " 

    

   !    ! !    ! !    ! !     

    

                   
     /

                     

    0 %    

       

       

      !      !          ! #     #

    %     

    #   ! # #   ! #   #          
       

       #      #          "    

        

    "      "      #  !          

        

             

 /  0 %    

          "

 /     

           !
 /  %     

 "      "      "              

 /     

                        
 /      

                        "     

 /      

       
0 %         

            

0 %      %     

  !     
0 %         

                         

0 %          

            
0 %          

          

     %     

  "      !      !              " 

        

                         !     

         

         #      #      #   "    # " 

         

 "                        

%          

                       #     

%           

      #      #      #      " ! 

%           

   !    ! "     ! "       "    # 

         

   "  "     "           #   ! 

         

          !      !   !   "  " !  ! # 

          

  !  !   " ! ! !   " ! !      

) ) ' )

 !   "  #  "   #

 '  

45
Table XXX: Case 7: Analysis of Variance for Clevel, using Adjusted SS for Tests.

        

                             

                            

    "    " "     " "  "     "      

  #        "      "      "          

        " "     " "     " "     " 

!

 $                         "      "  

      "  "       "                   

     "      "        "        "           

     %                               

     %                      "          

     %   #        

     %                 "  "       " 

     %  $      "  " "    " " "    "            

     %         "       "          "       

     %      "  "      "         "        

   %     "                        

   %   #                         "     

   %                          "     
!

   %  $                          "     

   %       "     "     "              

   %      "       "      "      " "      

    %   #   "                           

    %    "     "      "                

! !

    %  $                                 

    %            "      "               

    %      " " "      "                 

  #   %                               

  #   %  $                       "         

  #   %       "                           "

  #   %      "                         

   %  $            "      "       "        

   %       "   "  " "   "  " "               

   %      "     "      "      "   "     

!

 $      %          "      "      "        " 

 $      %      "       "       "      "         

      %      " " "       "        "    "     

  
    "     "        "        


& ' (       "     

46
Table XXXI: Case 7: Analysis of Variance for StDi¤, using Adjusted SS for Tests.

        

                                    

                                

                                     

! "

  #                                      

                                  
"

 $                                 

                                   

                                  

     %                                   

     %                            &            

! "

     %   #                           &           

     %                                

"

     %  $                                    

     %                                      

     %                                      

   %                                        

! "

   %   #       &      &      &           

   %          &         &         &            

"

   %  $                                    

   %                                         

   %                                  

    %   #        &       &    &            

! "

    %          &         &         &           

! " "

    %  $                                     

! "

    %                               &            

! "

    %                                      

! "

  #   %      &    &    &           

"

  #   %  $                                     

  #   %             &         &         &           

  #   %                                        

   %  $                                   

"

   %                                       

"

   %                                  
"

 $      %                                     

 $      %                                  

      %                                     

  
                     &   


' ( )          

47
Table XXXII: Case 7: Means and standard errors (SE) of the 1st and 2nd order e¤ects
on MeanBias, Clevel and StDi¤ across the other factors.

                      
 

   

                                        

                                         

                                         

                                         

                                        

                                         

                                          

                                         

                                         

                                            

" #  $  

 %                                            

 %                                            

 %                                            

  &  

'   '
                                         

 $
                                         

#   $  $

                                         

                                          

    

$
                                          

(
                                           

(
                                           

( (
                                            

 $  


 & 
                                             

                                           

 & 
                                            

 $  


#
                                         

)
                                         

    * !

                                               

                                               

                                               

                                              

                                               

                                               

                                               

                                               

                                               

                                              

                                               

                                               

                                                

                                                

                                               

                                            

48

                      
 

           

              !  "    #  #        $     "       #  

        #       !  "    " "         $     ! %      #  

 %       #       !  "    !  !        $     # #      #  

      % !        !  "              $     "       #  

      % !  %      !  "             $     %      #  

 %     % !  #      !  "     #         $           #  

%        "      !  "   %   #        $     % "      #  

%             !  "    ! " !        $            #  

% %       "      !  "    %          $      %      #  

        !      !  "   %  " !        $     !      #  

       " !      !  "    %         $     # %      #  

 %       #      !  "              $     %       #  

#      %   !      !  "     %         $      "      #  

#      %         !  "    ! # #        $      !      #  

# %     %   "      !  "     % %        $            #  

!            !  "      "        $            #  

!         %      !  "    # # %        $     " "      #  

! %        !      !  "    %         $     !      #  

   $    % ! "      !  "     % !        $     % %      #  

   $    % ! !      !  "    !  #        $      !      #  

%   $    %      !  "      %        $      %      #  

"       "  %      !  "      "        $     " #      #  

"       " %       !  "    ! "        $           #  

" %      "       !  "              $           #  

       &  

 
 ' '     #         %      %      %  $     !         

 
              %            %  $     ! "        

 
 ' '   % !  !        %    # #      %  $     #        

 
   % !         %         %  $            

 
% ' '     "        %    "        %  $             

 
%     !        %    "  "      %  $     %        

 
 ' '             %      "      %  $     # #        

 
             %    #  %      %  $     #        

 
# ' '   %          %    %       %  $              

 
#   %   #        %     "      %  $              

 
! ' '      !        %    !        %  $     "         

 
!           %    ! % #      %  $     " "        

 
' ' $    % ! !        %     %      %  $              

 
$    % !         %    %      %  $              

 
" ' '    "  %        %          %  $     "        

 
"    " % "        %    # #      %  $     "         

          

 #     # #        %     ! "      %  $     # !        

      #        %     %       %  $     !        

 #   % ! % !        %      %      %  $     ! "        

   % !         %     % %      %  $     "         

% #    #         %    %       %  $              

%     %        %    " " %      %  $     #         

 #            %            %  $              

    " "        %    # !       %  $     ! %        

# #   %  % %        %          %  $      !        

#   %           %           %  $      #        

! #              %     #      %  $     #        

!              %    "        %  $      "        

# $    % % %        %    !        %  $             

$              %    #       %  $     %         

" #    " ! "        %     ! !      %  $             

"    "  %        %            %  $     "         

49

                      
 

         

                                           

 !                                           

 !                     "                      

 !  !                                           

                                           

 !                                      

 !       "                                    

 !  !       "              "                      

                                   "        

 !                                  "         

 !                                           

 !  !                                          

                                           

 !                   "                        

 !                                           

 !  !                                           

                                           

 !     "                        " "        

 !                                           

 !  !                                           

                                           

 !    "   "                               

 !                                           

 !  !      "             "                       

"                    "                      

" !                                      

" !                                           

" !  !                   "                       

       "                                    

 !       "                                    

 !       "                                    

 !  !                                          

     #   # 
 

 $ 
      "                      "       "         

                    "       "                

 $ 
                            "       "         

 $ 
                      "      "       "         

                           "                

 $ 
                            "                

 $ 
        "              "      "                

     "                     "       "         

 $ 
                            "                

 $ 
                    "        "        "        

                           "                

 $ 
                            "                

 $ 
                            "        "        

                           "                

 $ 
                     "       "        "        

 $ 
                            "       "         

                           "       " "        

 $ 
                            "                

 $ 
"                           "               

"                           "                

 $ 
"                      "      "                

 $ 
                      "      "                

                          "       "         

 $ 
                            "       "         

50

                      
 

           

  
                                !      ! !

    
 "                !                   ! !

    
     # !                               ! !

    
"                          #      ! !

   $  
      #                       !      ! !

  $   $ 
 "                 #                 ! !

   
!                 !                    ! !

   
! "                                     ! !

   
                !                     ! !

$    
 "     !                             ! !

$     
    !                                ! !

$ $    
"                              #      ! !

     
                                    ! !

    
 "                             #       ! !

  $ 
                 ! # !             !      ! !

$   $ 
 "   ! #                                ! !

 %  &     


   $ $ 
     '         ! #                !        #

   $ $  
    '    # # #       ! #   !                    #

     $ 
     '           ! #    !             !       #

  $ $ $ $ $ 
     '            ! #               !       #

     $ $ 
    '           ! #     #                   #

   $ 
     '           ! #      !                    #

 %    (  

       
    ) )                                    

   $ $   
       # #                         #      

    
    ) )     !             #               !      

    
                   #              #      

 %      


    
         !           ! !                    

$  $  
        #                          !      

  $  
         !                         ! !      

$  
        #  #             !              #      

 %      

  $
       #  #                                

  $ $
    *                              !      

   $
    *    #                           !      

 
    *  *    #               #                    

 $  
       #                           #      

  $ $  $
    *                 #                  


    *    #                                  

 
    *  *    #               ! !                   

 %       


 (   $ $ $ 
     !   !        ! #    !  !                   #

  $ $ 
         !       ! #                       #

 (   $ 
     !      !       ! #     #                    #

 (   $   $ $ 
     !   !        ! #    !                  #

  $ 
                ! #                 !       #

 (  $ 
     !             ! #                         #

 %        

$    
        #           !               !      

  
    "                  ! !                    

$   $  
                    !                    

$ $  
    "                               #       

&         (  


     $ 
 ' ) )            ! #   !                    #

    $ $ 
 '    # #        ! #   !                     #

     $ 
' ) )            ! #      #                    #

    $  
'    # # #       ! #    #              #       #

    $ $ 
 ' ) )            ! #                        #

    $ $ 
 '         ! #                         #

&           
 

    $ 
 '             ! #                        #

 $  $ $ $ 
 '     #        ! #                       #

   $ 
'             ! #     !                    #

   $ 
'     #         ! #                         #

  $ 
 '      !       ! #    !                     #

 $ $ 
 '     #  #       ! #                 # !       #

51

                      
 

            

                       $
            !     " ! # !

         $              $
   %    !      &     " #  !

                      $
   %              "   !

          $              $
   %  %    !     #     "    !

                       $
             !     " # & !

          $              $
   %    ! &          " # & !

          $              $
   %               "  !

          $             $
   %  %         ! &     "   !

$           $         $      $
           !     " ! !

$                        $
  %    ! !         " #  !

$          $              $
  %               " ! !

$           $             $
  %  %              "  !  !

             


 '                      $
   "  #  # !   ! &       "     

           $ $              $
    #  ! &    "  #  

 '    $                   $
    #  & &  ! &    ! #  "    

 '                        $
   "  #  # ! !  ! &     "  &  

     $       $         $      $
     ! &   ! #  "  

 '    $ $       $ $             $
    #  &  ! &   &  "   

$  '                        $
  "  #  # !   ! &  # #  "    

$            $              $
   #  ! &     " !   

$  '    $                    $
   #  &   ! &    "    

             


        $    $     $         
   !    #  # & ! &  "    & 

  $ $ $     $        $        
   (  #  #  # ! &  "     & 

        $   $     $         
   !    #  & & ! ! &  "    & 

  $ $ $     $    $     $        
   (  #   ! &  "  #   & 

$         $   $      $     $     
  !    #  #  ! &  " &  & 

$   $ $     $        $    $     
  ( &  #   ! ! &  "   & 

  '         

)   )           $             
&   !      & &   & " !   

)   )                      
          &   & "     

                        
&         # & &   & "    

                       
   !      & &   & "     

  '        

)   )              $ $ $          $
    & &     ! " # 

)   )              $ $ $          $
%    ! !  & &   &  " # &

)   )              $ $ $          $
%       & &    & "  

)   )              $ $ $         $
%  %      & &   # ! ! "  

               $ $ $          $
    & &    & ! " # 

               $ $ $          $
%    ! !  & &   &  " # &

               $ $ $          $
%       & &    & "  

               $ $ $         $
%  %      & &   # !  " 

  '         


)   )  '    $     $        $         
"  #  # #  #   # ! &  "  &  & 

)   )          $       $     $     
 #  #    ! &  "  & 

)   )  '        $   $     $         
 #  & !   #   # & ! &  "   & 

   '        $        $         
"  #  #    #   # # ! &  "    & 

           $       $         
 #  #    ! &  " !  & 

   '         $   $     $         
 #  & &  #   #  ! &  "  #  & 

  '         


)   )            $             
!     !  &   & "     

)   )   $ $                    
(      ! &   & "  #    

      $        $              
!     &   & "     

    $ $                    
( !       & &   & "  &    

           

          $     $ $ $          $
&    &  & &    " # !

             $ $ $          $
& %     !  & &     " # 

             $ $ $          $
& %       & &    # " 

            $ $ $         $
& %  %      & &    & "  # #

         $     $ $ $          $
     & &   #  " &

  $            $ $ $          $
%     & &    " # 

   $       $     $ $ $          $
%     & &    "  !

            $ $ $    $      $
%  %       & &   ! &  "  #

52

                      
 

             

         !      "  #    ! ! #     ! "      "       #

    

      #      "  #     $      ! "      ! $       #
   

        " $      "  #    " $ #     ! "             #

    

$                "  #    # ! #     ! "              #
    

$     # #      "  #    # !      ! "             #
  

$         "      "  #     $     ! "             #

   

             

    ! $ $           "              $        
   

  " "              !                  
 %    

$     $ #           " # #          "        
    

$   " " # !                       $ #        
%   

            

                #     !       #           " ! $
  

     #  !      #    " "  "     #       $      " ! $



                 #     $ " "     #       !      " ! $



&          !  $      #      # $     #             " ! $
 

&       $      #     # "     #            " ! $

 

&                #     #  "     #       !      " ! $



&                 #     $ ! !     #           " ! $
  

&      #  $      #    " "  "     #            " ! $



&          "        #     " $     #           " ! $

  

&  &                #     #      #            " ! $

   

&  &      #  "      #    "  # "     #            " ! $

 

&  &           !      #          #            " ! $

   

            

      #          " $ !     $ " " "            "

  

   "  !          $ !     $ " " "     #       "

%    

&      ! "          #     $ " " "     " #      "

  

&     #  "               $ " " "           "

%    

&       # $          #     $ " " "     "       "

  

&    "                 $ " " "     "      "

%    

&  &       #          $     $ " " "     " #      "

   

&  &    "            $     $ " " "    "       "

%    

              

          # $      "  #     #     ! "       #       #
    

       " ! "       "  #     #     ! "      ! $       #

 %   

     !        "  #         ! "             #
    

   " " $      "  #     "     ! "              #

%   

          "       "  #    "  #     ! "             #

   

       "       "  #    $     ! "      $ "       #

%     

53