Variance Outlier Test
Thursday, February 14, 2013
Note on browser
The appearance of my blog (especially the manuscript) depends on your browser. The format of my manuscript is a compromise to obtain acceptable appearance with the "Mozilla Firefox" and the "Internet Explorer" browser. If you use another browser and you are not satisfied with the appearance of my blog, consider to visit my blog with either Mozilla Firefox or Internet Explorer.
Sunday, November 7, 2010
1. Introduction
1.1 Purpose of this blog
February 2010, I published a variance outlier test, which I called the "G test", in Analytica Chimica Acta. This journal is well appreciated by analytical chemists. However, the G test is relevant to other disciplines as well. The G test is an extension of Cochran's C test [23] and is of potential interest to anyone who needs to decide if one or more variance (or standard deviation) estimates are extreme in comparison with a group with which they are supposed to be comparable. I am aiming for a broader audience by posting a personal manuscript version of my article in this blog. The publisher of the original article [33] allows such practice provided that the posting includes a link to the article's Digital Object Identifier (DOI) as well as a complete citation for the original article [34]. Please find the link and the citation immediately below:
doi:10.1016/j.aca.2009.11.032 R.U.E. 't Lam, Analytica Chimica Acta 659 (2010) 68–84.
Alternatively, you may visit my manuscript. In the manuscript I have corrected some errata and inconsistencies. I also have extended the appendices of the original article to supply critical values for a broader selection of designs. Besides, I have added explanatory notes on performing a G test as Chapters 2 to 4 of the Home page. In priciple, a blog is not intended to be permanent. For reference purposes you are therefore advised to include a citation of my original article. If you have questions, suggestions or concerns regarding the original article, the blog or the G test in general, you are encouraged to leave me a message, contact me at rtlam@zeelandnet.nl or contact me through my page at LinkedIn.
1.2 Abstract
The abstract of the article in Analytica Chimica Acta [34] reads:
ISO Standard 5725 “Accuracy (trueness and precision) of measurement methods and results” recommends Cochran’s C test to numerically verify if three or more normally distributed data sets show “homogeneity of variances” or “homoscedasticity”. The C test is a onesided outlier test that will identify deviant standard deviations. It can be run on summary data using a pocket calculator. However, the C test has limitations. It only applies to data sets of equal size. It uses critical values that are only available for the upper tail of the variance distribution, at selected numbers of data sets, selected numbers of replicates per set and only at two significance levels. Cochran’s C test will not identify an outlying low variance, but may mistake a high variance for an outlier instead. We transform the C test into a more general “G test”.
Expressions are derived to calculate upper limit as well as lower limit critical values for data sets of equal and unequal size at any significance level. The expressions are validated against literature values and through simulations in Excel. Representative critical values are tabulated for those who prefer to work from tables. The power of the G test is verified for data sets of equal and unequal size. The G test appears superior to the C test in detecting effects from low variances. The G test allows positive identification of exceptionally low variances. The application of the G test is illustrated with a numerical example.
1.3 Errata
1.3 Errata
1. In § 3.3 equation (18) was corrected to read:
"ζ_{LL}(1) = 1_{ }− ζ_{UL}(1) = 1_{ }− α_{1}/L" in stead of "ζ_{LL}(1) = 1_{ }− ζ_{UL}(1) = (1_{ }− α_{1})/L"
2. In § 3.5 the order of equations (30) and (31) was reversed to match the remainder of the manuscript.
3. Some typo's and rounding errors in Tables 68 were corrected.
4. The lower left corner of Table 8 was changed to read "L= 6", instead of "L= 7".
1.4 Application Areas
Variance outlier testing is routinely applied in interlaboratory studies for determining the precision of analytical test methods according to ISO 5725. However, variance outlier testing is also practiced outside (analytical) chemistry. A search in Google Scholar on "Cochran's C test" + additional keyword, resulted in the numbers of hits listed below. Key words with less than ten hits are not shown.
As Cochran's C test is fully comprised in the G test, the G test is at least as widely applicable as Cochran's C test. Wikipedia provides a short background article on Cochran's C test [36].
Saturday, November 6, 2010
2. General summary of G test
2.1 Calculating a G ratio
2.4 Evaluating balanced and unbalanced designs
The G test can be run as a onesided test upperlimit test, as a onesided test lowerlimit test and as a twosided test at any significance level α. When running a G test, it is checked for each data set j if the corresponding inequality is met:
Onesided upper limit test: G_{j} < G_{UL}(α,ν_{j},ν_{total},L)
Onesided lower limit test: G_{LL}(α,ν_{j},ν_{total},L) < G_{j}
Twosided test: G_{LL}(α/2,ν_{j},ν_{total},L) < G_{j }< G_{UL}(α/2,ν_{j},ν_{total},L)
2.3 Computing critical G values
The critical G values G_{UL} and G_{LL} are calculated from:
The equations to calculate G_{UL} and G_{LL} are exact. The accuracy of G_{UL} and G_{LL} only depends on the accuracy of the input value F_{c}. F_{c} can be read from tables, but can also be obtained using the FINV function in Excel: F_{c} = FINV(probability, degrees of freedom 1, degrees of freedom 2).
2.4 Evaluating balanced and unbalanced designs
The G test can handle both balanced and unbalanced designs. In case of balanced designs, all data sets have the same number of degrees of freedom. Balanced designs are covered in § 3, unbalanced designs in § 4.
3. Balanced designs
3.1 Cochran's C ratio
In a balanced design, all data sets have the same number of observations. The G ratio (General summary of G test: § 2.1) then reduces to the wellknown Cochran's C ratio [23]:
Where:
G_{j} = G test statistic for data set j
C_{j} = Cochran's C test statistic for data set j
C_{j} = Cochran's C test statistic for data set j
3.2 Computing critical C values
For balanced designs the critical values C_{UL} and C_{LL }take the reduced forms:
Below you find a simple example spreadsheet to calculate critical values for balanced designs:
Cell contents:
B11: "=1/(1+(B51)/FINV(1B4/B5,B61,(B51)*(B61)))"
B12: "=1/(1+(B51)/FINV(1B4/2/B5,B61,(B51)*(B61)))"
C10: "=1/(1+(B51)/FINV(B4/B5,B61,(B51)*(B61)))"
C12: "=1/(1+(B51)/FINV(B4/2/B5,B61,(B51)*(B61)))"Current practitioners of Cochran's test may prefer to continue working from tables. For their convenience, Appendices A to F provide extensive lists with critical values for balanced designs:
Appendix

Limit

ζ

Onesided α

Twosided α

Upper Limit

0.01

0.01

0.02
 
Upper Limit

0.025

0.025

0.05
 
Upper Limit

0.05

0.05

0.1
 
Lower Limit

0.01

0.01

0.02
 
Lower Limit

0.025

0.025

0.05
 
Lower Limit

0.05

0.05

0.1

3.3 Onesided upper limit G test
When conducting a onesided upper limit G test on balanced data, it is not necessary to evaluate G_{j} for all data sets j. It is sufficient to evaluate:
Where:
G_{max} = G test statistic for the data set with the highest variance
C_{max} = Cochran's C test statistic for the data set with the highest variance
C_{max} = Cochran's C test statistic for the data set with the highest variance
Running a onesided upper limit G test on balanced data is similar to running a traditional Cochran's C test:
Determine the upper limit value C_{UL} according to § 3.2. If G_{max} exceeds C_{UL}, label the corresponding variance value as "exceptionally large", remove the value from the variance data, and repeat the test on the remaining variance values. Continue the process until you have identified all remaining variance outliers.
Several text books [1,2,6] and ISO 5725 [19] provide guidance on conducting a C test. The main advantages of conducting a G test in stead are:
Determine the upper limit value C_{UL} according to § 3.2. If G_{max} exceeds C_{UL}, label the corresponding variance value as "exceptionally large", remove the value from the variance data, and repeat the test on the remaining variance values. Continue the process until you have identified all remaining variance outliers.
Several text books [1,2,6] and ISO 5725 [19] provide guidance on conducting a C test. The main advantages of conducting a G test in stead are:
 The G test is not restricted to the significance levels α = 0.01 and α = 0.05 but will run at any significance level.
 The upper limit values for the G test need not be obtained from tables, but can be computed (§ 3.2).
 The limit values for Cochran's C test have only been tabulated for selected numbers of data sets (L) and selected numbers of observations (n) per data set; whereas the G test allows limit values to be calculated for any combination of L and n.
3.4 Onesided lower limit G test
When conducting a onesided lower limit G test on balanced data, it is not necessary to evaluate G_{j} for all data sets j. It is sufficient to evaluate:
Where:
Where:
G_{min} = G test statistic for the data set with the lowest variance
Cochran's C test does not allow for lower limit tests. A onesided lower limit G test on balanced data is conducted as follows:
Determine the lower limit value C_{LL} according to § 3.2. If G_{min} is less than C_{LL}, label the corresponding variance value as "exceptionally small", remove the value from the variance data, and repeat the test on the remaining variance values. Continue the process until you have identified all remaining variance outliers.
3.5 Twosided G test
Determine the lower limit value C_{LL} according to § 3.2. If G_{min} is less than C_{LL}, label the corresponding variance value as "exceptionally small", remove the value from the variance data, and repeat the test on the remaining variance values. Continue the process until you have identified all remaining variance outliers.
3.5 Twosided G test
When conducting a twosided G test on balanced data, it is not necessary to evaluate G_{j} for all data sets j. It is sufficient to evaluate:
Where:
Where:
G_{max} = G test statistic for the data set with the highest variance
G_{min} = G test statistic for the data set with the lowest variance
Usually a twosided G test on balanced data can be conducted as follows:
Determine the upper limit value C_{UL} and the lower limit value C_{LL} according to § 3.2, taking ζ = α/2. If G_{max} exceeds C_{UL}, label the highest variance value as "deviant" and remove this value from the variance data. Likewise, if G_{min} is less than C_{LL}, label the lowest variance value as "deviant", remove that value from the variance data, and repeat the test on the remaining variance values. Continue the process continues until you have identified all remaining variance outliers.
Occasionally both the highest variance and the lowest variance value may appear "deviant". In that case you should only remove the variance value that has the highest probability to be deviant indeed, and continue the process from there. There are two equivalent ways to identify the variance value that is most likely to be deviant. Conceptually the simplest approach is:
Lower the initial significance level of the twosided test such that only one of the variance values will still be marked "deviant". Remove only that value from the variance data, and repeat the test at the lower significance level on the remaining variance values. Continue the process until you have identified and removed all variance outliers at the lower significance level. Then run the twosided test on the remaining variance data; let the significance level gradually increase until you reach your initial significance level.
While conceptually simple, the above approach has the drawback that it will require more calculation work than is strictly necessary. If you plan to apply the G test routinely, you may prefer to directly calculate the probability that a specific variance is truly deviant according to § 3.6 of the manuscript.
4. Unbalanced designs
The approaches described below are conceptually simplest, but require more calculation work than necessary. If you plan to apply the G test routinely, you are referred to § 3.6 of the manuscript.
4.1 Onesided upper limit G test
For each data set j, calculate the ratio G_{j} according to § 2.1. For each data set j, also calculate the upper limit critical value G_{UL} at the desired significance level α according to § 2.3, and check if G_{j} ≤ G_{UL}.
 If G_{j} ≤ G_{UL} for all data sets, there is no reason to assume that one of the data sets has an exceptionally large variance value in comparison with the other data sets.
 If G_{j} > G_{UL} for one data set, label the corresponding variance value as "exceptionally large", remove the value from the variance data, and repeat the test on the remaining variance values.
 If G_{j} > G_{UL} for several data sets, lower the significance level of the test until only one of the data sets still will show G_{j} > G_{UL}. Label the corresponding variance value as "exceptionally large", remove the value from the variance data, and repeat the test on the remaining variance values at the lowered significance level. Continue the process until you have identified and removed all deviant data sets at the lowered significance level. Then run the test on the remaining variance data; let the significance level gradually increase until you reach your initial significance level.
For each data set j, calculate the ratio G_{j} according to § 2.1. For each data set j, also calculate the lower limit critical value G_{LL} at the desired significance level α according to § 2.3, and check if G_{j} ≥ G_{LL}.
 If G_{j} ≥ G_{LL} for all data sets, there is no reason to assume that one of the data sets has an exceptionally small variance value in comparison with the other data sets.
 If G_{j} < G_{LL} for one data set, label the corresponding variance value as "exceptionally small", remove the value from the variance data, and repeat the test on the remaining variance values.
 If G_{j} < G_{LL} for several data sets, lower the significance level of the test until only one of the data sets still will show G_{j} < G_{LL}. Label the corresponding variance value as "exceptionally small", remove the value from the variance data, and repeat the test on the remaining variance values at the lowered significance level. Continue the process until you have identified and removed all deviant data sets at the lowered significance level. Then run the test on the remaining variance data; let the significance level gradually increase until you reach your initial significance level.
4.3 Twosided G test
For each data set j, calculate the ratio G_{j} according to § 2.1. For each data set j, also calculate the upper limit critical value G_{UL} and the lower limit critical value G_{LL} at the desired significance level α according to § 2.3, and check if G_{LL} ≤ G_{j} ≤ G_{UL}.
 If G_{LL} ≤ G_{j} ≤ G_{UL} for all data sets, there is no reason to assume that one of the data sets has a deviant variance value in comparison with the other data sets.
 If G_{LL} ≤ G_{j} ≤ G_{UL} is not met for one particular data set, label the corresponding variance value as "deviant", remove the value from the variance data, and repeat the test on the remaining variance values.
 If G_{LL} ≤ G_{j} ≤ G_{UL} is not met for several data sets, lower the significance level of the test until only one of the data sets still will fail G_{LL} ≤ G_{j} ≤ G_{UL}. Label the corresponding variance value as "deviant", remove the value from the variance data, and repeat the test on the remaining variance values at the lowered significance level. Continue the process until you have identified and removed all deviant data sets at the lowered significance level. Then run the test on the remaining variance data; let the significance level gradually increase until you reach your initial significance level.
4.4 Example case of twosided G test
Cell contents:
D5: "=B51"
E5: "=D5*C5^2"
G5: "=1/(1+(D13/D51)/FINV(1G3/2/COUNT(C5:C12),D5,D13D5))"
H5: "=E5/E13"
I5: "=1/(1+(D13/D51)/FINV(G3/2/COUNT(C5:C12),D5,D13D5))"
(...)
D13: "=SUM(D5:D12)"
D13: "=SUM(D5:D12)"
D12: "=B121"
E12: "=D12*C12^2"
G12: "=1/(1+(D13/D121)/FINV(1G3/2/COUNT(C5:C12),D12,D13D12))"
H12: "=E12/E13"
I12: "=1/(1+(D13/D121)/FINV(G3/2/COUNT(C5:C12),D12,D13D12))"
E13: "=SUM(E5:E12)"
Subscribe to:
Posts (Atom)