CONSECUTIVE PAIRWISE ANALYSIS OF TRIALS FOR RELIABILITY

Hover cursor for citation:

Hover cursor for updates: 2015 2012 2010 2009

This spreadsheet calculates reliability statistics for consecutive pairs of trials and for the means of these statistics wh
For trials spaced at equal intervals, these means are better estimates of test-retest reliability than the more usual A
But see the 1-way and 2-way spreadsheets in this workbook for such analyses.
The reliability statistics are calculated for raw data and after log transformation.
I generated the data shown with http://www.sportsci.org/2007/SimulateSamples.xls. See the accompanying article
I deliberately chose data with large errors that are clearly better analyzed via log transformation to give percent or
Unless your data are times to exhaustion or gene transcription scores, they won't be so obviously in need of log tra
Delete and replace the numbers in blue. Stats you might want are in red. Don't change any cells with a colored bac
Missing values must be blanks or the graphs will display incorrectly. (The X values will plot as consecutive integers
Clear all the #NUM! corresponding to missing raw values from the log-transformed panel. Restore the formula (by co
Check the graphs (especially of the change scores) for outliers and non-uniformity of scatter. The log-transformed v
For more than 20 subjects, COPY any number of rows after the first data row and INSERT in the same place witho
Do NOT copy and insert the first data row.
Double-click on any of the red mean or SD cells to check that you have done this operation properly.
Colored boxes should enclose all rows of your data.
For less than 20 subjects, delete or clear any unwanted or empty rows.
To remove a trial, clear its data, or set the corrresponding "Include which trials?" cell to 0 or blank.
For more than 6 trials, copy and insert the entire column(s) for Trials 5 and/or 6 and their change scores for the RAW
And follow the instructions in the cells indicated by "Hover cursor for >6 trials". If such editing is too difficult for you,

Choose smallest effect: 0.2

Choose conf. limits (%): 90 Hover cursor for >6 trials: Read me
RAW Data
Trial: 1 2 3 4 5 6
Alex 0.7 4.77 7.58 2.83
Ariel 2.83 1.37 1.41 3.86
Ashley 0.71 3.29 4.49
Bernie 14.81 2.15 14.83
Casey 0.29 0.61 0.94
Chris 3.31 2.07 2.25
Corey 1.76 4.41 2.94
Courtney 8.41 9.01 11.53 14.28
Devon 3.67 5.67 10.91 7.81
Drew 1.92 3.74 4.4 6.71
Dylan 0.53 0.47 1.68 2.39
Frances 0.06 0.17 0.46 0.54
Gene 2.75 2.74 7.03 4.79
Jaimie 1.73 8.8 2.93 7.25
Jean 2.17 2.36 1 1.6
Jesse 1.01 4 4.52 11.29
Jo 1 1.39 1.11 1.36
Jody 1.74 2.02 2.49 1.86
Jordan 23.09 18.56 26.65 27.12
Kade 1.49 8.5 4.46 8.85
Mean 3.7 4.3 5.7 6.8
SD 5.7 4.3 6.3 6.9
Total no. of subjects 20 20 20 15 0 0
Include which trials? 1 1 1 1 0 0
 DF 19 19 19 14 0 0

For your data, you may have to set max and min values of X and Y axes to appropriate values.

20 10
18

Trial 2-1
Trial 2

16 5
14
12
0
10
0 5 10 15 20 25
8
6 -5
4
2 -10
0 Trial 1
0 5 10Trial15
1 20 25 -15

30 15
Trial 3-2
Trial 3

25
10
20
5
15

10 0
0 2 4 6 8 10 12 14 16 18 20
5
-5
0 Trial 2
Trial
0 2 4 6 8 10 212 14 16 18 20 -10

30 8
Trial 4-3
Trial 4

25 6

20 4

15 2

10 0

5 -2 0 5 10 15 20 25 30

0 -4
Trial 3
0 5 10 Trial
15 3 20 25 30 -6

12 12
Trial 5-4
Trial 5

10 10

8 8

6 6

4 4

2 2

0 0
0 5 10 Trial
15 420 25 30 0 5 10 Trial
15 4 20 25 30

12 12
Trial 6-5
Trial 6

10 10

8 8

6 6

4 4

2 2

0 0
0 2 4 Trial
6 58 10 12 0 2 4 Trial
6 5 8 10 12
 12 12

Trial 6-5
Trial 6
10 10

8 8

6 6

4 4

2 2

0 0
0 2 4 Trial
6 58 10 12 0 2 4 Trial
6 5 8 10 12
 e means of these statistics when there are >2 trials.
eliability than the more usual ANOVA-based reliablity statistics.

See the accompanying article at http://www.sportsci.org/2007/wghstats.htm.

nsformation to give percent or factor effects and errors.
e so obviously in need of log transformaiton, but log transformaiton may still be better.
ge any cells with a colored background.
will plot as consecutive integers.)
nel. Restore the formula (by copying an adjacent cell) if you restore a missing value or for new data.
scatter. The log-transformed variables may show less non-uniformity, as evident in these data.
SERT in the same place without deselecting.
Use analysis of RAW data in one or more of the following situations:
peration properly. the data can be zero or negative;
the data were measured from an arbitrary reference value;
the magnitude of the error is expected to be similar for all subjects.
o 0 or blank.
heir change scores for the RAW and the LOG data. Missing values produce delete in the panel of change scores.
h editing is too difficult for you, contact me. Clear delete from these cells to stop them plotting as zeros.
Copy-paste or -drag to restore formulae to those cells if you enter new da

Hover cursor for >6 trials: Read me Read me Do not

Change scores of raw data modify
Mean 2-1 3-2 4-3 5-4 6-5 this col.
4.07 2.81 -4.75 delete delete
-1.46 0.04 2.45 delete delete
2.58 1.2 delete delete delete
-12.66 12.68 delete delete delete
0.32 0.33 delete delete delete
-1.24 0.18 delete delete delete
2.65 -1.47 delete delete delete
0.6 2.52 2.75 delete delete
2 5.24 -3.1 delete delete
1.82 0.66 2.31 delete delete
-0.06 1.21 0.71 delete delete
0.11 0.29 0.08 delete delete
-0.01 4.29 -2.24 delete delete
7.07 -5.87 4.32 delete delete
0.19 -1.36 0.6 delete delete
2.99 0.52 6.77 delete delete
0.39 -0.28 0.25 delete delete
0.28 0.47 -0.63 delete delete
-4.53 8.09 0.47 delete delete
7.01 -4.04 4.39 delete delete
5.0 Mean of change 0.6 1.4 1.0 Mean
5.8 SD of change 4.1 4.0 3.0 3.82
20 Total no. of subjects 20 20 15 0 0 20
 19

Measures of reliability via the RAW variable

Raw Trials 2-1 3-2 4-3 5-4 6-5 Mean
Change in mean 0.61 1.38 0.96
Lower conf. limit -0.99 -0.18 -0.41
Upper conf. limit 2.20 2.93 2.33
Conf. limits as value 1.60 1.56 1.37
Typical error 2.92 2.84 2.13 2.70
Lower conf. limit 2.32 2.26 1.64 2.30
Upper conf. limit 4.00 3.90 3.12 3.32
Conf. limits as approx. /factor 1.31 1.31 1.38 1.20
Bias correction factor 1.01 1.01 1.02 1.01
Smallest effect from pure SD 0.82 0.92 1.24 1.03
Lower conf. limit 0.40 0.50 0.79 0.58
Upper conf. limit 1.09 1.20 1.57 1.33
Smallest effect from observed SD 1.01 1.08 1.31 1.16
Lower conf. limit 0.80 0.86 1.04 0.92
Upper conf. limit 1.38 1.48 1.80 1.59
Degrees of freedom 19 19 14 0 0 41
Standardized (when ICC>0)
Change in mean 0.15 0.30 0.15
Lower conf. limit -0.24 -0.04 -0.07
Upper conf. limit 0.54 0.64 0.38
Conf. limits as value 0.39 0.34 0.22
Typical error 0.71 0.62 0.34 0.47
Lower conf. limit 0.56 0.49 0.26 0.40
Upper conf. limit 0.97 0.85 0.50 0.57
Conf. limits as approx. /factor 1.31 1.31 1.38 1.20
Bias correction factor 1.01 1.01 1.02 1.01
Correlations
Pearson correlation 0.70 0.79 0.91 0.80
Lower conf. limit 0.44 0.58 0.78
Upper conf. limit 0.85 0.90 0.96
Bias correction factor 1.02 1.01 1.01
Intraclass correlation (ICC) 0.69 0.74 0.90 0.80
Lower conf. limit 0.42 0.52 0.78 0.67
Upper conf. limit 0.84 0.87 0.96 0.89
Effective no. of trials 2.0 2.0 1.7 3.2
F 5.4 6.8 17.5 13.7
Numerator DF 19 19 19 19
Denominator DF 19 19 14 0 0 41
F lower 2.5 3.1 7.3 7.4
F upper 11.6 14.7 39.5 27.8
Fisher r 0.87 1.06 1.52 1.11
Fisher SE^-2 17 17 12
Pure variance 16.8 21.1 38.6 26.3
SE of pure variance 7.7 9.1 13.9 10.8
LCL 4.1 6.1 15.7 8.5
UCL 29.6 36.1 61.4 44.1
e of the following situations: Use LOG transformation in one or more of the following situat
the data can only be positive and non-zero;
y reference value; the within-subject, between-subject or between-trial SD is more t
be similar for all subjects. the magnitude of the percent or factor error is expected to be sim

l of change scores. Missing values produce #NUM! in the panel of log-transformed dat
plotting as zeros. Clear #NUM! from these cells to get complete analyses.
those cells if you enter new data. Copy-paste or -drag to restore formulae to those cells if you enter n

Hover cursor for >6 trials: Read me Read

Any 100*LOG-Transformed Data
subject? 1 2 3 4 5
1 Alex -35.7 156.2 202.6 104.0 Err:502
1 Ariel 104.0 31.5 34.4 135.1 Err:502
1 Ashley -34.2 119.1 150.2 Err:502 Err:502
1 Bernie 269.5 76.5 269.7 Err:502 Err:502
1 Casey -123.8 -49.4 -6.2 Err:502 Err:502
1 Chris 119.7 72.8 81.1 Err:502 Err:502
1 Corey 56.5 148.4 107.8 Err:502 Err:502
1 Courtney 212.9 219.8 244.5 265.9 Err:502
1 Devon 130.0 173.5 239.0 205.5 Err:502
1 Drew 65.2 131.9 148.2 190.4 Err:502
1 Dylan -63.5 -75.5 51.9 87.1 Err:502
1 Frances -281.3 -177.2 -77.7 -61.6 Err:502
1 Gene 101.2 100.8 195.0 156.7 Err:502
1 Jaimie 54.8 217.5 107.5 198.1 Err:502
1 Jean 77.5 85.9 0.0 47.0 Err:502
1 Jesse 1.0 138.6 150.9 242.4 Err:502
1 Jo 0.0 32.9 10.4 30.7 Err:502
1 Jody 55.4 70.3 91.2 62.1 Err:502
1 Jordan 313.9 292.1 328.3 330.0 Err:502
1 Kade 39.9 214.0 149.5 218.0 Err:502
20 Mean 53.15 98.99 123.91 Err:502
SD 132.66 110.82 104.41 Err:502
N 20 20 20 15 0
Include which trials? 1 1 1 1 0
 DF 19 19 19 14 0
Back-transformed mean 1.7 2.7 3.5
SD as a / factor 3.77 3.03 2.84
SD as a CV (%) 276.8 202.9 184.1
Total no. of subjects 20 20 20 15 0

Clear #NUM! and "delete" from the log-transformed cells to stop them plotting as zeros.
Restore the log-transformation formula to those cells if you enter new data.

400.0 250
200

Trial 2-1
300.0
Trial 2

150
200.0 100
100.0 50
0
0.0
-400.0 -200.0-50 0.0 200.0 400.0
-400.0 -200.0 0.0 200.0 400.0
-100.0 -100
-150
-200.0
-200
Trial 1
Trial 1
-300.0 -250

350.0 250
Trial 3-2

300.0 200
Trial 3

250.0 150
200.0
100
150.0
50
100.0
0
50.0
-400.0 -200.0-50 0.0 200.0 400.0
0.0
-400.0 -200.0
-50.0 0.0 200.0 400.0 -100
Trial 2 Trial 2
-100.0 -150

400.0 150
Trial 4-3

350.0
100
Trial 4

300.0
250.0 50
200.0
150.0 0
100.0 -100.0 0.0 100.0 200.0 300.0 400.0
-50
50.0
0.0 -100
-50.0 0.0
-100.0 100.0
Trial200.0 300.0 400.0 Trial 3
3
-100.0 -150

12.0 12
Trial 5-4
Trial 5

10.0 10

8.0 8

6.0 6

4.0 4

2.0 2

0.0 0
-100.0 0.0 Trial200.0
100.0 4 300.0 400.0 -100.0 0.0 100.0 200.0 300.0 400.0
Trial 4

12.0 12
Trial 6-5
Trial 6

10.0 10

8.0 8

6.0 6

4.0 4
 12.0 12

Trial 6-5
Trial 6
10.0 10

8.0 8

6.0 6

4.0 4

2.0 2

0.0 0
0.0 5.0 10.0
Trial15.0
5 20.0 25.0 0.0 5.0 10.0 15.0
Trial 5 20.0 25.0
ne or more of the following situations:
and non-zero;
ubject or between-trial SD is more than ~20% of the mean;
or factor error is expected to be similar for all subjects.

in the panel of log-transformed data. Missing values produce delete in the panel of change scores.
o get complete analyses. Clear delete from these cells to stop them plotting as zeros.
ormulae to those cells if you enter new data. Copy-paste or -drag to restore formulae to those cells if you enter new

me Do not Hover cursor for >6 trials: Read meRead me

modify Change scores of Log-transformed data
6 this col. 2-1 3-2 4-3 5-4 6-5
Err:502 191.9021249 46.31669 -98.5236 delete delete
Err:502 -72.5465972 2.877896 100.7077 delete delete
Err:502 153.3377874 31.09651 delete delete delete
Err:502 -192.983479 193.1184 delete delete delete
Err:502 74.35780342 43.24209 delete delete delete
Err:502 -46.9399582 8.338161 delete delete delete
Err:502 91.85608804 -40.5465 delete delete delete
Err:502 6.891359764 24.66173 21.39076 delete delete
Err:502 43.49974557 65.44907 -33.4275 delete delete
Err:502 66.67604254 16.25189 42.19944 delete delete
Err:502 -12.0144312 127.3816 35.24996 delete delete
Err:502 104.1453875 99.54281 16.03427 delete delete
Err:502 -0.36429913 94.22288 -38.3656 delete delete
Err:502 162.6630313 -109.975 90.5999 delete delete
Err:502 8.393445149 -85.8662 47.00036 delete delete
Err:502 137.634403 12.22176 91.54054 delete delete
Err:502 32.93037471 -22.4944 20.31247 delete delete
Err:502 14.92123982 20.91852 -29.1706 delete delete
Err:502 -21.8390896 36.17804 1.748231 delete delete
Err:502 174.1290044 -64.4917 68.52687 delete delete
Err:502 Mean 45.8 24.9 22.4
Err:502 SD 95.1 71.7 55.7
0 N 20 20 15 0 0
0
 0 Mean Factor change in mean 1.6 1.3 1.3
SD of change as a factor 2.59 2.05 1.75
Change in mean (%) 58.1 28.3 25.1
SD of change as CV (%) 158.9 104.9 74.5
0 Total no. of subjects 20 20 15 0 0
Measures of reliability via the LOG-TRANSFORMED variable
m plotting as zeros. Factors Trials 2-1 3-2 4-3 5-4 6-5
Factor change in mean 1.58 1.28
Lower conf. limit 1.09 0.97
Upper conf. limit 2.28 1.69
Conf. limits as /factor 1.44 1.32
Factor typical error 1.96 1.66
Lower conf. limit 1.71 1.50
Upper conf. limit 2.51 2.00
Conf. limits as approx. /factor 1.21 1.16
Bias correction factor 1.01 1.01
Smallest effect from pure SD 1.23 1.21
Lower conf. limit 1.10 1.11
Upper conf. limit 1.31 1.28
Smallest effect from observed SD 1.28 1.24
Lower conf. limit 1.21 1.19
Upper conf. limit 1.40 1.34
Degrees of freedom 19 19 0 0 0
Percents
Change in mean (%) 58.1 28.3
Lower conf. limit 9.5 -2.8
Upper conf. limit 128.5 69.3
Conf. limits as approx. value 59.5 36.0
Typical error as a CV (%) 96.0 66.1
Lower conf. limit 70.6 49.6
Upper conf. limit 151.4 100.4
Conf. limits as approx. /factor 1.46 1.42
Bias correction factor 1.02 1.02
Smallest effect from pure SD 22.6 20.9
Lower conf. limit 10.4 11.2
Upper conf. limit 31.2 28.0
Smallest effect from observed SD 27.7 24.0
Lower conf. limit 21.4 18.6
Upper conf. limit 39.8 34.3
Standardized (when ICC>0)
Change in mean 0.45 0.26
Lower conf. limit 0.09 -0.03
Upper conf. limit 0.81 0.55
Conf. limits as value 0.36 0.29
Typical error 0.66 0.53
Lower conf. limit 0.52 0.42
Upper conf. limit 0.90 0.73
Conf. limits as approx. /factor 1.31 1.31
Bias correction factor 1.01 1.01
Correlations
Pearson correlation 0.72 0.79
 Lower conf. limit 0.47 0.58
Upper conf. limit 0.86 0.90
Bias correction factor 1.01 1.01
Intraclass correlation 0.72 0.80
Lower conf. limit 0.48 0.60
Upper conf. limit 0.86 0.90

Measures of reliability of log-transformed variable

2-1 3-2 4-3 5-4 6-5
Change in mean 45.8 24.9
Lower conf. limit 9.0 -2.8
Upper conf. limit 82.6 52.7
Typical error 67.3 50.7
Lower conf. limit 53.4 40.3
Upper conf. limit 92.2 69.5
Bias correction factor 1.01 1.01
Smallest effect from pure SD 20.4 19.0
Lower conf. limit 9.9 10.6
Upper conf. limit 27.1 24.7
Smallest effect from observed SD 24.4 21.5
Lower conf. limit 19.4 17.1
Upper conf. limit 33.5 29.5
DF 19 19 0 0 0

Stdized change in mean 0.45 0.26

Lower conf. limit 0.09 -0.03
Upper conf. limit 0.81 0.55
Stdized typical error 0.66 0.53
Lower conf. limit 0.52 0.42
Upper conf. limit 0.90 0.73

Pearson r 0.72 0.79

Lower conf. limit 0.47 0.58
Upper conf. limit 0.86 0.90
Bias correction factor 1.01 1.01
Intraclass r 0.72 0.80
Lower conf. limit 0.48 0.60
Upper conf. limit 0.86 0.90
Effective no. of trials 2.0 2.0
F 6.1 8.8
Numerator DF 19 19
Denominator DF 19 19 0 0 0
F lower 2.8 4.1
F upper 13.2 19.1
Fisher r 0.91 1.07
Fisher SE^-2 17 17
Pure variance 10416 9018
SE of pure variance 4846 3759
LCL 2445 2835
UCL 18387 15201
 change scores.
tting as zeros.
ose cells if you enter new data.

Do not
modify Any
this col. subject?
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
Err:502
0
#DIV/0!
Mean

Mean
 Mean

#DIV/0!
#DIV/0!
#DIV/0!

#DIV/0!
#DIV/0!
#DIV/0!
Err:502
#DIV/0!
#DIV/0!
#DIV/0!

0.76

#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
0.99

#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
SIMULATION AND ANALYSIS FOR 1-WAY RELIABILITY
Hover cursor for citation:
This spreadsheet simulates and analyzes data for a 1-way reliability study. It was adapted from a spreadsheet in the ar
Sequence of development of the simulation:
The data are 10 repeated measurements on each of 20 subjects. The data could be something like distance of high-sp
or values of repeated tests on athletes, when you are confident that the tests are in some sense all similar (no habitu
Such data are unusual, but I have provided this spreadsheet to help you understand reliability, even if you don't use it fo
All subjects have been given the same variability from measurement to measurement, the within-subject random error.
For this 1-way analysis of reliability, the measurements are not associated with particular games.
The data are simulated by assigning percent standard deviations (coefficients of variation) to the differences between s
The data therefore need log transformation before analysis, the aim of which is to estimate the two CVs and to express
An analysis is also shown for the raw data, and for the mean of a chosen number of repeated measurements.
Change the values in bold blue and see what happens to the outcomes in red.
To generate a whole new set of data, click in a blank cell away from other cells and hit Ctrl-D.
This spreadsheet works with missing data. Try deleting some. Copy-drag or copy-paste from adjacent cells to reinst
To analyze your own data, replace the main block of data shown in plum color.
You can also delete or copy-and-insert whole rows and whole columns.
Don't delete and don't copy-and-insert the first data row or the first five data columns (or you will corrupt the spreadsh
For data requiring log transformation, whatever deletions and insertions you do to the block of raw data
will have to be done to the block of log-transformed data. Check that the right cells have been transformed.
Read the article for an explanation of the analyses.
Do not use log transformation in one or more of the following
Choose confidence level (%): 90 the data can be zero or negative;
Simulation to create raw observed values the data were measured from an arbitrary reference value;
Depedent variable the magnitude of the error is expected to be similar for all subjec
True
values
Mean 400
Between-subject CV (%) 30 < Insert or delete who
Equivalent factor SD 1.3
Within-subject random error (%) 10 10 10 10 10 10

Do not copy/insert this row 390 Alex 341 368 401 344 466 382
556 Ariel 600 554 469 537 587 534
307 Ashley 297 329 292 272 377 284
281 Bernie 258 260 319 309 290 263
890 Casey 844 953 1012 796 820 851
752 Chris 819 743 744 718 732 701
510 Corey 462 457 490 492 568 462
767 Courtney 821 726 746 729 788 838
372 Devon 373 460 331 451 360 423
333 Drew 302 335 344 292 324 313
340 Dylan 411 388 345 363 386 361
315 Frances 340 323 299 361 321 342
329 Gene 373 335 325 370 344 329
470 Jaimie 474 484 485 378 375 505
324 Jean 315 281 328 364 305 320
316 Jesse 383 239 314 289 302 397
377 Jo 338 336 306 386 368 404
408 Jody 438 464 424 350 390 359
 518 Jordan 470 509 569 514 530 476
343 Kade 336 322 389 332 377 311

Mean 449.7 443.2 446.6 432.4 450.4 442.8

Analysis of raw data LCL UCL CL CL

Observed subject SD 183.5 145.7 251.5 1.31
True subject SD 177.4 120.6 219.9 49.6
Error SD 47.2 43.5 51.7 1.09
ICC 0.93 0.90 0.96 0.03
Error and ICC for mean of 2 trials
LCL UCL CL CL
Error SD 33.4 30.7 36.6 1.09
ICC 0.97 0.94 0.98 0.02
Subjects
(rows) total error
n 20 200
DegFree 19 199 180
SumSquares 6E+06 6420991 401332
SD^2 31459 2230
SE^2 of SD^2 1E+08 55235.7
SE of SD^2 10279 235
LCL UCL
F ratio 142.1 86.4 271.9
Obsvd subj variance 33689 21235 63269
True subj variance 31459 14552 48367
 from a spreadsheet in the article Understanding Stats via Simulations.

ething like distance of high-speed running in games

me sense all similar (no habituation or order effects).
ility, even if you don't use it for your own data.
within-subject random error.

to the differences between subjects and to the within-subject random error.

the two CVs and to express them as intraclass correlations. coefficient.
ted measurements.

e from adjacent cells to reinstate those cells.

you will corrupt the spreadsheet).

k of raw data
been transformed.

ne or more of the following situations: Use log transformation in one or more of the followin
the data can only be positive and non-zero;
rbitrary reference value; the within-subject, between-subject or between-trial SD
ted to be similar for all subjects. the magnitude of the percent or factor error is expected

< Insert or delete whole columns here >

10 10 10 10

For sums of
Mean squares
371 402 303 381 376 51186 10
578 593 508 610 557 119356 10
295 325 277 323 307 197243 10
302 268 261 250 278 287703 10
1048 920 1203 860 931 2E+06 10
762 748 809 654 743 871601 10
417 489 469 526 483 12683 10
782 750 861 739 778 1E+06 10
369 368 373 413 392 30782 10
303 331 363 327 323 154151 10
381 386 345 331 370 60744 10
355 405 313 336 339 116872 10
342 377 286 330 341 113531 10
404 446 564 475 459 1294.34 10
341 344 329 348 328 143983 10
268 331 314 344 318 167858 10
363 385 321 316 352 90799 10
393 380 447 389 403 19537 10
541 637 511 482 524 57937 10
353 364 347 363 349 96558 10

Mean 447.6 SD 6E+06 10

448.4 462.4 460.2 439.8 447.6 447.6 9.0 Mean
SD 178.0 179.6
Analysis of log-transformed data
Observed subject factor SD
True subject factor SD
Error factor SD
Observed subject CV (%)
True subject CV (%)
Error CV (%)
ICC
Error and ICC for mean of

Error factor SD
Error CV (%)
ICC
Subjects

n
DegFree
SumSquares
SD^2
SE^2 of SD^2
SE of SD^2

F ratio
Obsvd subj variance
True subj variance

Observed subject SD
True subject SD
Error SD
Error for mean of 2 trials
n in one or more of the following situations:
positive and non-zero;
tween-subject or between-trial SD is more than ~20% of the mean;
percent or factor error is expected to be similar for all subjects.

< Insert or delete whole columns here >

100*log(observed values)
If you have missing raw data, you will have to delete #NUM! from the corresponding cells here.

583.3 590.7 599.5 584.1 614.3 594.6 591.7 599.7 571.4 594.4
639.6 631.7 615.0 628.6 637.4 628.1 636.0 638.6 623.0 641.3
569.3 579.5 567.8 560.5 593.2 565.1 568.6 578.5 562.6 577.7
555.4 556.1 576.5 573.4 566.9 557.3 571.0 559.0 556.3 552.3
673.8 686.0 692.0 668.0 670.9 674.7 695.4 682.4 709.3 675.6
670.8 661.1 661.2 657.6 659.5 655.3 663.5 661.7 669.5 648.3
613.5 612.4 619.5 619.9 634.1 613.6 603.3 619.2 615.1 626.5
671.1 658.8 661.4 659.2 667.0 673.1 666.2 662.0 675.8 660.5
592.2 613.1 580.2 611.1 588.7 604.7 591.0 590.9 592.2 602.3
570.9 581.5 584.1 567.8 578.0 574.8 571.3 580.3 589.4 579.0
601.9 596.1 584.4 589.5 595.6 588.8 594.3 595.6 584.2 580.1
582.8 577.7 570.0 588.9 577.2 583.4 587.1 600.5 574.7 581.7
592.2 581.3 578.3 591.2 584.1 579.6 583.5 593.2 565.6 579.9
616.1 618.1 618.3 593.4 592.7 622.4 600.2 610.0 633.6 616.4
575.2 563.7 579.4 589.8 571.9 576.9 583.3 584.1 579.7 585.1
594.7 547.7 574.9 566.5 571.1 598.4 559.2 580.2 574.9 584.0
582.3 581.9 572.4 595.5 590.7 600.1 589.4 595.3 577.2 575.7
608.2 613.9 604.9 585.9 596.6 588.2 597.5 594.0 610.2 596.4
 615.2 623.2 634.3 624.2 627.3 616.6 629.3 645.6 623.6 617.7
581.8 577.5 596.5 580.5 593.1 573.9 586.5 589.7 584.8 589.4

604.5 602.6 603.5 601.8 605.5 603.5 603.4 608.0 603.7 603.2

og-transformed data LCL UCL CL CL

1.42 1.32 1.61 1.11
1.40 1.26 1.52 1.10
1.10 1.09 1.11 1.01
41.8 31.9 61.3 1.39
39.9 25.6 51.6 13.0
10.0 9.2 11.0 1.10
0.93 0.88 0.96 0.04
r and ICC for mean of 2 trials
LCL UCL CL CL
1.07 1.06 1.08 1.01
7.0 6.4 7.7 1.09
0.96 0.94 0.98 0.02
Subjects
(rows) total error
20 200
19 199 180
215772 232099 16327
1126.6 90.7
135757 91.415
368.45 9.56
LCL UCL
125.2 76.1 239.5
1217.3 767 2286.08
1126.6 520.522 1732.62
LCL UCL
34.9 27.7 47.8
33.6 22.8 41.6
9.5 8.8 10.4
6.7 6.2 7.4
 For sums of
Mean squares
592 1345.914 10
632 7808.221 10
572 10047.44 10
562 17280.82 10
683 62147.06 10
661 32351.9 10
618 1890.114 10
666 37866.46 10
597 537.9423 10
578 6903.506 10
591 1670.742 10
582 4651.323 10
583 4441.522 10
612 664.8976 10
579 6272.664 10
575 8303.041 10
586 3215.379 10
600 192.1129 10
 626 4721.19 10
585 3459.969 10

Mean 604.0 SD 215772.2 10

604.0 604.0 1.7
SD 33.7 34.2
Back-transformed:
Mean 419.8 CV (%)
419.8 419.8 1.8
CV (%) 40.1 40.7
 SIMULATION AND ANALYSIS FOR 2-WAY RELIABILITY
Hover cursor for citation:
See the previous spreadsheet (1-way reliability) for an explanation of the generation of the data and for detailed instruc
Most straightforward reliability analyses can be done with this spreadsheet or the earlier one for consecutive pairwise a
When familiarization or other habituation or order effects occur with only a few repeated measurements, use the cons
Use this spreadsheet when there are no such effects, and when you can assume that the errors on each measurement
The repeated measurements are nevertheless associated with known measuring occasions, such as multiple games or
Here the subjects are players in a team. They all experience a given increase or decrease in their scores from game to
depending on whether the team lost, tied, or won the game.
The effects of losing, tieing or winning are shown as percents. Losing and winning have been given the same factor eff
The aim of the analysis is the same as before, but now the mean score in each game can be adjusted for, and the differ
To analyze your own data, replace the main block of data shown in plum color.
You can also delete or copy-and-insert whole rows and whole columns. Don't delete and don't copy-and-insert the
This spreadsheet does NOT work with missing data. Use mixed modeling instead, with data in "long" format.
Effects of losing, tieing or winning (and any other within- or between-subject factors) can be estimated by including ap
Do not use log transformation in one or more of the following
Choose confidence level (%): 90 the data can be zero or negative;
Simulation to create raw observed values the data were measured from an arbitrary reference value;
Depedent variable the magnitude of the error is expected to be similar for all subject
True
values
Mean 400
Between-subject CV (%) 30 < Insert or delete whole
Equivalent factor SD 1.3
Within-subject random error (%) 10 10 10 10 10 10
Effect of lose/tie/win (%) 5 0 -4.8 5 0 -4.8
Lose/tie/win L T W L T W
Game1 Game2 Game3 Game4 Game5 Game6
Do not copy/insert this row 237 Alex 223 294 226 255 265 240
426 Ariel 458 385 400 423 424 439
275 Ashley 222 298 241 271 244 283
254 Bernie 313 234 239 308 259 256
417 Casey 443 406 383 386 458 386
252 Chris 245 217 255 255 285 234
359 Corey 401 348 382 385 370 321
582 Courtney 666 537 534 573 589 545
338 Devon 346 347 329 354 343 332
335 Drew 335 348 317 354 354 370
463 Dylan 505 460 479 568 452 352
309 Frances 350 347 287 296 327 307
278 Gene 346 260 285 313 286 250
309 Jaimie 346 276 345 365 301 293
262 Jean 244 266 252 249 258 280
638 Jesse 628 586 640 726 548 492
484 Jo 513 500 464 426 489 447
344 Jody 384 304 295 350 365 332
325 Jordan 373 350 294 356 316 272
399 Kade 419 365 417 555 482 352

Mean 388.0 356.4 353.2 388.4 370.8 339.1

 Analysis of raw data LCL UCL CL CL
Observed subject SD 112.0 89.0 153.6 1.31
True subject SD 105.8 71.7 131.3 29.8
Observed game SD 18.8 13.7 30.9 1.50
True game SD 16.9 3.3 23.7 10.2
Error SD (same raters) 36.9 33.9 40.5 1.09
Error SD different raters) 40.6 not yet available
ICC (same raters) 0.89 0.83 0.94 0.05
ICC (different raters) 0.87 not yet available
Error and ICC for mean of 2 trials Read comment.
LCL UCL CL CL
Error SD (same raters) 26.1 24.0 28.7 1.09
Error SD (different raters) 28.7 not yet available
ICC (same raters) 0.94 0.91 0.97 0.03
ICC (different raters) 0.93 not yet available
Subjects Games
(rows) (columns) total error
n 20 10 200
DegFree 19 9 199 171
SumSquares 2152693 63623.2 2449072 232756
SD^2 11194 285 1361
SE^2 of SD^2 1.4E+07 27817.6 21669.2
SE of SD^2 3676 167 147
LCL UCL
F ratio 83.2 50.5 159.4
Obsvd subj variance 12555 7914 23579
True subject variance 11194 5147 17240
Obsvd game variance 353 188 957
True game variance 285 11 560
 e data and for detailed instructions.
one for consecutive pairwise analysis of repeated measurements.
measurements, use the consecutive-pairwise spreadsheet.
e errors on each measurement are equal and independent of each other.
ns, such as multiple games or items in a psychometric inventory.
e in their scores from game to game,

been given the same factor effect (1.05 and 1.05, which equal 5% and -4.5%).
n be adjusted for, and the differences in the means can be expressed as an SD.

d don't copy-and-insert the first data row or the first five data columns (or you will corrupt the spreadsheet).
th data in "long" format.
n be estimated by including appropriate variables in the mixed model.
one or more of the following situations: Use log transformation in one or more of the following
the data can only be positive and non-zero;
rbitrary reference value; the within-subject, between-subject or between-trial SD
cted to be similar for all subjects. the magnitude of the percent or factor error is expected

< Insert or delete whole columns here >

10 10 10 10
5 0 -4.8 0
L T W T
Game7 Game8 Game9 Game10 Mean
269 208 198 285 246
464 439 458 465 435
249 272 273 283 264
268 221 197 253 255
474 434 418 414 420
259 228 263 242 248
407 332 316 418 368
557 642 566 464 567
401 323 279 274 333
375 415 351 299 352
504 507 468 547 484
364 366 269 284 320
270 259 309 322 290
366 319 272 290 317
295 272 247 239 260
753 596 556 593 612
499 536 507 440 482
382 339 333 295 338
330 317 309 324 324
380 421 356 434 418

Mean 366.7 SD
393.3 372.3 347.2 358.2 366.7 366.7 18.8 Mean
SD 106.4 110.9
 Analysis of log-transformed data
Observed subject factor SD
True subject factor SD
Observed game factor SD
True game factor SD
Error factor SD (same raters)
Error factor SD (different raters)
Observed subject CV (%)
True subject CV (%)
Observed game CV (%)
True game CV (%)
Error CV (%) (same raters)
Error CV (%) different raters)
ICC (same raters)
ICC (different raters)
Error and ICC for mean of

Error factor SD (same raters)

Error factor SD (different raters)
Error CV (%) (same raters)
Error CV (%) (different raters)
ICC (same raters)
ICC (different raters)
Subjects

n
DegFree
SumSquares
SD^2
SE^2 of SD^2
SE of SD^2

F ratio
Observed subject variance
True subject variance
Observed game variance
True game variance

Observed subject SD
True subject SD
Observed game SD
True game SD
Error SD same raters
Error SD different raters
Error for mean of 2 trials same raters
Error for mean of 2 trials different raters
n in one or more of the following situations:
positive and non-zero;
tween-subject or between-trial SD is more than ~20% of the mean;
percent or factor error is expected to be similar for all subjects.

< Insert or delete whole columns here >

100*log(observed values)
If you have missing raw data, you will have to delete #NUM! from the corresponding cells here.

Game1 Game2 Game3 Game4 Game5 Game6 Game7 Game8 Game9 Game10
540.9 568.4 541.9 554.2 558.1 547.9 559.4 534.0 528.6 565.1
612.7 595.4 599.1 604.7 605.1 608.4 613.9 608.4 612.7 614.2
540.4 569.6 548.7 560.3 549.6 564.6 551.9 560.5 561.0 564.7
574.7 545.4 547.6 573.1 555.7 554.5 559.0 539.6 528.3 553.2
609.4 600.7 594.9 595.7 612.6 595.6 616.2 607.3 603.5 602.6
550.3 537.8 554.0 554.3 565.1 545.4 555.6 542.7 557.3 548.8
599.3 585.3 594.4 595.2 591.2 577.1 600.9 580.5 575.7 603.6
650.1 628.7 628.0 635.0 637.9 630.1 632.2 646.5 633.8 613.9
584.6 584.9 579.7 586.8 583.9 580.5 599.4 577.7 563.3 561.1
581.3 585.3 575.9 587.0 587.0 591.3 592.7 602.9 586.0 569.9
622.5 613.0 617.2 634.3 611.3 586.5 622.3 622.9 614.8 630.5
585.8 584.8 566.0 569.1 579.1 572.7 589.8 590.2 559.4 564.9
584.6 556.1 565.2 574.5 565.6 552.0 560.0 555.7 573.2 577.3
584.6 562.2 584.3 589.9 570.7 568.0 590.1 576.4 560.5 567.1
549.7 558.4 552.8 551.9 555.3 563.3 568.6 560.5 551.1 547.5
644.3 637.2 646.1 658.7 630.5 619.8 662.4 638.9 632.0 638.6
624.0 621.4 614.0 605.5 619.2 610.4 621.2 628.4 622.9 608.8
595.0 571.8 568.8 585.8 590.0 580.5 594.6 582.5 580.7 568.7
592.2 585.9 568.4 587.5 575.6 560.7 580.0 575.8 573.3 578.0
603.7 589.9 603.4 632.0 617.9 586.4 594.0 604.4 587.5 607.4

591.5 584.1 582.5 591.8 588.1 579.8 593.2 586.8 580.3 584.3
og-transformed data LCL UCL CL CL
1.34 1.26 1.49 1.09
1.31 1.20 1.40 1.08
1.05 1.04 1.08 1.02
1.04 1.01 1.06 1.03
1.10 1.09 1.11 1.01
1.11 not yet available
33.6 25.9 48.8 1.37
31.5 20.4 40.4 10.0
4.9 3.6 8.3 1.52
4.4 0.8 6.3 2.7
10.1 9.2 11.1 1.10
11.1 not yet available
0.89 0.83 0.94 0.05
0.87 not yet available
r and ICC for mean of 2 trials Read comment.
LCL UCL CL CL
1.07 1.06 1.08 1.01
1.08 not yet available
7.0 6.4 7.8 1.10
7.7 not yet available
0.94 0.91 0.97 0.03
0.93 not yet available
Subjects Games
(rows) (columns) total error
20 10 200
19 9 199 171
143986 4194.25 163989 15809
749 19 92
60453 121 100
246 11 10
LCL UCL
82.0 49.7 157.0
841.0 530 1579.47
749 344 1153
23 12 63
18.7 0.6 36.8
LCL UCL
29.0 23.0 39.7
27.4 18.6 34.0
4.8 3.5 7.9
4.3 0.8 6.1
9.6 8.8 10.6
10.5
6.8 6.2 7.5
7.5
 Mean
550
607
557
553
604
551
590
634
580
586
618
576
566
575
556
641
618
582
578
603

Mean 586.2 SD
586.2 586.2 4.83
SD 27.53 28.71
Back-transformed:
Mean 351.6 CV (%)
351.6 351.6 4.9
CV (%) 31.7 33.3