Detecting Accuracy Differences
We sometimes wish to know if one primer or powder charge or bullet produces smaller groups than another. Or we might wish to test one sight or bench rest or shooting technique against another for accuracy. For simplicity, we'll refer to each set of conditions tested as a "load".
In the course of experimenting I've never been sure how to detect the accuracy differences between one load and another.
For example, if Remington 2 1/2 primers shoot into 1.2" for five 5-shot groups, and if Winchester WLP primers shoot into 1.1" for five 5-shot groups-all other things being equal-do I know that the WLR primers will produce smaller groups? Sometimes the same test on a different day yields opposite results. How should I go about determining that there is a difference in accuracy between two loads?
Estimating Average Group Size
A given gun and load and set of conditions will shoot groups that follow the "Normal" or bell-shaped distribution. The two and only numbers that describe one of these distributions are the average (arithmetic mean) and the standard deviation. The standard deviation is a measure of variation, or width of the distribution.
Yesterday, 26 January, 2005, with a M54 Winchester in 30 WCF, five 5-shot groups averaged 1.265 inches What do I know after shooting these groups? Is the 1.265 the true average? If I shot another twenty groups, would they average 1.265"? How about another hundred groups-what would they average? How many groups would I have to shoot to know the true average?
Group Size Variation
When I go to the range and shoot-for example-five 5-shot groups at 100 yards, I see a wide variation in the size of these groups, from smallest to largest. Others report shooting "one inch five shot groups at 100 yards" for example, with little or no detail given. My results are much messier. See the example above, where the M54 averaged 1.265 inch groups; the groups ranged in size from .9" to 1.875". I didn't know if this kind of variation was normal, or was my fault, or a problem with the gun or load.
I went back to the statistics books, collected a lot of group-size data, got a lot of help from my friends; and answered these questions.
Five or ten shot group sizes are popular measures of gun accuracy, with 100/200 yards for rifles and 50 yards for pistols the most frequently reported ranges. The almost universal use of the five shot or ten shot group size-distance between centers of the furthest apart two shots-in measuring rifle accuracy at matches also contributes to the popularity of the measure.
Some have argued for more complicated statistical measures, but while increasing information and perhaps precision, the difficulties in measurement and calculation seem to have kept these measures, such as Mean Radius, from becoming more popular.
At matches, group sizes are measured with precision calipers equipped with optical magnifiers. John Alexander, Vice President of the Cast Bullet Association, suggested that informal group size measurement could be made using a transparent six-inch plastic rule graduated in tenths of an inch. I use one of these rules that he gave me, and can easily measure group sizes to within twenty-five thousandths of an inch. It is easy to tell if a group measures, for example, .900" or .925" or .950" or .975" or 1.00".
A caveat: We cannot make any valid statements about groups unless the groups are fairly round. Round groups are distributed "normally" (bell curve), and the shooting-loading process is said to be "in control". Vertically strung or horizontally strung groups, or groups made up of 2 or more sub groups are not in control. If the groups exhibit consistent patterns other than round, the following doesn't apply.
Detecting Accuracy Differences
How can we tell if one load shoots smaller groups than another?
(My favorite Statistics book is "Facts from Figures", by M. J. Moroney. M. J. doesn't take it too seriously; his chapter title most appropriate to this topic is "How to be precise though vague".)
If one load shoots five shots into half an inch at 100 yards, and another load shoots into eight inches; there's no problem. We might want to shoot another pair of groups to check, but if the difference between group sizes is very large, the decision is easy. Difficulties arise when two loads shoot into groups that aren't very different. Then deciding if one load shoots smaller groups than another is tougher.
We can never be absolutely sure that one load shoots smaller groups than another. We can define how unsure we are willing to be, thus: "We're 90% sure that load A shoots smaller groups than load B".
The difference between average group size with load A and load B, the number of groups shot with each load, and how sure we want to be about our decision are the three factors.
First decide how sure we must be that we're right. The most frequently used percentages are 95% and 90%, we're 95% sure or we're 90% sure that we're right.
Then, shoot groups with load A and with load B, measure the groups, calculate the average group size for each load and divide the larger average into the smaller. Get the percent difference in averages.
For example, five shot groups at 100 yards were shot so:
| Group #1 | Group #2 | Average Group Size | |
| Load A | 1.00" | 1.5" | 1.25" |
| Load B | .75" | 1.25" | 1.00" |
The Load B average is 1" divided by 1.25" = 80% of the Load A average, and the difference is 20%, Load B shot a 20% smaller average groups size than Load A.
The tables tell us how many groups must be shot, at various % differences in group averages, to be either 90% or 95% sure that we're right when we say "Load X shoots smaller groups than Load Y".
If we want to be 95% sure that we're right, and the difference between load A and load B group size averages is 20%, we must shoot 9 groups with each load. (See the bold "9" in the NUMBER OF 5 SHOT GROUPS NEEDED table, in the 95% SURE column.)
We've shot 2 groups, so 7 more groups with each load are needed.
As we shoot the other 7 groups with each load, the difference between group averages might/will change.
Let's say that the difference between the averages (after all 9 groups are shot with load A and load B) falls to 10%.
Now, to be 95% sure that we're right; we must shoot a total of 38 groups with each load.
So adjustment must go on throughout the testing
NUMBER OF 5 SHOT GROUPS NEEDED |
NUMBER OF 10 SHOT GROUPS NEEDED |
|||||
% DIFFERENCE |
% DIFFERENCE |
|||||
IN GROUP |
90% |
95% |
IN GROUP |
90% |
95% |
|
AVERAGES |
SURE |
SURE |
AVERAGES |
SURE |
SURE |
|
1% |
2452 |
4050 |
1% |
1039 |
1716 |
|
2% |
607 |
1003 |
2% |
258 |
425 |
|
3% |
267 |
442 |
3% |
114 |
187 |
|
4% |
149 |
246 |
4% |
63 |
105 |
|
5% |
95 |
156 |
5% |
40 |
66 |
|
6% |
65 |
108 |
6% |
28 |
46 |
|
7% |
48 |
78 |
7% |
20 |
33 |
|
8% |
36 |
60 |
8% |
16 |
25 |
|
9% |
28 |
47 |
9% |
12 |
20 |
|
10% |
23 |
38 |
10% |
10 |
16 |
|
11% |
19 |
31 |
11% |
8 |
13 |
|
12% |
16 |
26 |
12% |
7 |
11 |
|
13% |
13 |
22 |
13% |
6 |
9 |
|
14% |
11 |
19 |
14% |
5 |
8 |
|
15% |
10 |
16 |
15% |
4 |
7 |
|
16% |
9 |
14 |
16% |
4 |
6 |
|
17% |
8 |
12 |
17% |
4 |
6 |
|
18% |
7 |
11 |
18% |
3 |
5 |
|
19% |
6 |
10 |
19% |
3 |
4 |
|
20% |
5 |
9 |
20% |
3 |
4 |
|
25% |
4 |
6 |
|
25% |
2 |
3 |
30% |
3 |
4 |
|
30% |
1 |
2 |
35% |
2 |
3 |
|
35% |
1 |
1 |
40% |
1 |
2 |
|
40% |
1 |
1 |
45% |
1 |
2 |
|
45% |
1 |
1 |
50% |
1 |
2 |
|
50% |
1 |
1 |
Statistics and this table enable us to make sets of statements. For instance, "If I shoot 9 five shot groups with load A and 9 five shot groups with load B, and if the difference between the averages is 20%, then I am 95% sure that the loads produce different group sizes".
Now some of these answers are impractical. For instance, if we want to be 95% sure that one load is more accurate than another, with five shot group averages differing by 1%, the table shows that we must shoot 4050 groups with each load. This would be a total of 40,500 shots, which would take quite a while. But, think about 1%. If the larger average is 1", and the smaller is .990", that's a difference of ten thousandths of an inch or 1%. If the larger average is .500" and the smaller is .495", that's 1%.
We're in "Who Cares" territory here.
Detecting small accuracy differences is probably not worth doing in many cases.
Detecting small accuracy differences and being very sure about these differences is difficult and expensive, and may sometimes be impossible in practical terms. Keep in mind that this is not my fault!
For the statistically inclined- (normal people may (should) ignore this.)
"Average" or "Mean" below = arithmetic mean
The theory supporting the table above is that the standard deviation of group sizes is a function of the number of shots in the group, and that this estimator of the population standard deviation is "better" than calculated estimates of small sample standard deviations. Certainly, this estimator is easier to calculate.
The standard deviation of the group size distributions is taken as .275 (for 5 shot groups) and .179 (for 10 shot groups) times the mean. Thus, the distribution of five shot groups sizes has a standard deviation of .275 times average group size and the distribution of 10 shot group sizes has a standard deviation of .179 times average group size.
The table is the product of an EXCEL spreadsheet program for the "Z" statistic for the differences between means when the variances are known.
Z = ((X1-X2)- (m1-m2))/Ö((s 12/n1)-(s 2 2/n2))
The hypothesis is that the means are equal, that the difference between means is 0.
Then (m1-m2) = 0 in the above formula.
The alternative hypothesis is that they ain't equal, the difference ain't 0.
The data needed are sample averages (group size averages), sample sizes or n's (number of groups shot with each load), and the variance of the populations.
An EXCEL spreadsheet was constructed for the above formula and n was varied for an array of differences between sample averages until Z equaled 1.645 (a= .05) and 1.28 (a = .1). (Note that since the variance and sample mean are related, Z is a function of the DIFFERENCE between sample means as described above.)
Where do the .275 and .179 come from?
I used the 2002 and 2004 Cast Bullet Association (CBA) National Match results for groups, 45ACP 50 yard pistol groups from "Cast Bullets" by E. H. Harrison and my records of shooting a Savage Model 12BVSS in 223 and a M54 Winchester in 30/30 at 100 yards.
Average group size, standard deviation of group size, and standard deviation of group size over average group size were calculated for many shots.
Note the consistency of the Stdev/Avg numbers in both tables.
Average |
Average |
||||||
Groups |
Averages |
Group |
Group |
Average |
|||
Year |
Range |
Shot |
Average |
Stdev |
Stdev/Avg |
||
2002 |
100 |
196 |
49 |
1.117 |
0.314 |
0.275 |
|
2002 |
200 |
183 |
46 |
1.957 |
0.631 |
0.297 |
|
2004 |
100 |
156 |
39 |
0.874 |
0.212 |
0.249 |
|
2004 |
200 |
145 |
37 |
2.006 |
0.592 |
0.286 |
|
223 Sav |
100 |
436 |
102 |
1.960 |
0.587 |
0.303 |
|
M54 |
100 |
78 |
19 |
1.375 |
0.351 |
0.238 |
|
|
|
|
|
|
Avg.>>> |
0.275 |
|
Explaining the tables:
In 2002, at 100 yards, there were 196 five shot groups shot, with 49 averages of four groups shot Each of 49 shooters shot 4 groups. Each set of four groups was averaged, the standard deviation was calculated, and the statistic "standard deviation/average" was calculated. The average of all 49 averages was 1.117". The average of all 49 standard deviations was .314". The average of all "standard deviation/average" statistics was .275. Then for these 196 groups, on the average, the standard deviation was 27.5% of the average or arithmetic mean, of the group size.
Average |
Average |
||||||
Groups |
Averages |
Group |
Group |
Average |
|||
Year |
Range |
Shot |
Average |
Stdev |
Stdev/Avg |
||
2002 |
100 |
92 |
46 |
1.235 |
0.257 |
0.197 |
|
2002 |
200 |
82 |
41 |
2.778 |
0.543 |
0.193 |
|
2004 |
100 |
76 |
38 |
1.075 |
0.197 |
0.177 |
|
2004 |
200 |
72 |
36 |
2.913 |
0.440 |
0.144 |
|
NRA |
50 |
51 |
17 |
2.573 |
0.468 |
0.182 |
|
|
|
|
|
|
Avg.>>> |
0.179 |
Why is the standard deviation of group size a fraction of average group size?
Well, first off, because it is. The standard deviation of any distribution is a fraction of the mean. What I'm proposing here is that the standard deviation of group size is a CONSTANT fraction of the mean, and that that fraction varies with n. Then for any n = number of shots in the groups, the distributions are scale models of each other, varying with group size. I'm a little uncomfortable with this, because, as an example, the standard deviation of bullet weights seems fairly independent of mean bullet weight. Four hundred grain bullets vary +/- half a grain; sixty-grain bullets vary +/- half a grain. My weighing experiences tell me that the standard deviations of cast bullet weights is independent of bullet weight. However, for close average group sizes, it doesn't matter.
Then there's the fact that the data are remarkably similar. My experience with Economic and gun-related data is that data bounces around furiously. This doesn't. Looking at the rightmost column on each table shows lovely consistency.
And again, there's the fact that somebody else came to the conclusion long before I did.
"Shot Group Statistics", "Matching Ammo To The Rifle" and "Estimating Ammo Quality From Shot Group Diameters" by Jeroen Hogema explained the notion that the standard deviation of group size is a constant fraction of group size, that fraction varying with n. These and more are to be found on the Internet at http://home-2.worldonline.nl/~jhogema/ballist.htm
I sent Jeroen a copy of this article for his review and comment; He did some simulation to estimate the relationship between the standard deviation and arithmetic mean of shot groups.
His result is that the standard deviation of five shot groups is .2691 times group diameter, and the standard deviation of ten shot groups is .1947 times group diameter. These multipliers are in substantial agreement with the multipliers I derived from real-world data.
Here is a letter from Jeroen Hogema, explaining his method:
From: Jeroen Hogema
To: Joe Brennan
Date: 9 November 2004
The following assumptions were used.
*The location of the X-co-ordinates of the shots (Left-Right) vary according to a normal distribution.
*The location of the Y-co-ordinates of the shots (High-Low) vary according to a normal distribution.
*The x- and y-distributions are independent (i.e. no stringing of the shot group).
*The standard deviations of Left-Right and High-Low deviations are equal (i.e. a circular shot group, not an elliptical one). This common sd is referred to as the ‘shot group sd’. Not to be confused with the sd of group diameters as discussed below.
*The rifle is accurately zeroed, i.e. the means of the X and Y distributions are zero. Thus, the center of a shot group will, on average, be in the ‘10’.
For a given value of this common shot group sd, I did the following.
*‘Fired’ (simulated) a number of shots (either 5 or 10),
*from the resulting shot group, measured the resulting shot group diameter (center-center) and kept that as the observations-to-be-analyzed;
*repeated this many times, thus collecting my observations.
*Then calculated the resulting mean and sd of the observed shot group diameters.
*Repeated this process for a wide range of shot group sd’s.
*Results are shown in Fig. 1.
Fig. 1 Relationship between standard deviation and mean of shot group diameters as a function of the number of shots in a group (results from simulations and fitted functions).
Clearly, there is a linear relationship between the sd and the mean of the shot group diameter. The slopes that I found, compared to what you wrote:
Nr of shots |
Jeroen H. |
Joe B. |
5 |
0.2691 |
0.275 |
10 |
0.1947 |
0.179 |
I’d say that we agree…
Total number of shots simulated to generate Fig. 1: 6,600,000
I believe that this method presents a reasonable method of estimating the number of groups to be shot to detect differences in loads.
All of the original data are available in EXCEL, from joeb33050@yahoo.com.
Note: I suspected that small-group-shooters had small standard deviations; that their groups didn't vary much. Regression analysis for average group size vs. standard deviation for the CBA data failed to show any relationship.
Note: I did not choose:
There may be a better and easier test than that employing the Z statistic for (m1-m2) with variances known, but I can't find it.
Estimating Average Group Size
Everything below has to do with five shot groups measured center to center of the widest shots.
I've got a target with a .693" five shot 100-yard group. I keep it in my C. Sharps 1875 45/70 rifle logbook. This is the smallest group ever shot with this rifle at this range. I like the gun and I like the group, but I'd never say: "the rifle shoots into seven tenths of an inch".
We read and hear statements about accuracy, and sometimes suspect that the statements are not 100% true.
So the question I had is: "How do we estimate the inherent or true accuracy of a gun and load and equipment combination?".
We assume that for a given combination, that any gun will make groups that have an average, and that the individual groups will vary somewhat from this average. And we assume that these group sizes are distributed normally, that old "bell-shaped" curve. These are not unreasonable assumptions.
If we shoot one group of 1" for five shots at 100 yards we don't know much about the average group size to be expected, but we can make some statements. One is:" I'm very sure that the average size of groups shot with this combination is between 0 inches and 5 inches". Another is: "I'm very slightly certain that this combination will group between .5 inch and 1.5 inches".
We can make a range of statements that have two kinds of uncertainty- the "how sure am I that the statement I'm making is true" uncertainty and the "the average lies between this number and that number" kind of uncertainty.
As the number of groups shot increases the certainty increases and the bounds decrease.
If we shoot three groups that average one inch, we are 90% certain that the average group size lies between 1.535" and .465". We are also 95% sure that the average group size lies between 1.789" and .211". So the more sure we must be, the wider the bounds within which the average lies.
Now we shoot three additional groups, and the six groups shot average one inch.
We are 90% certain that average group size lies between 1.185" and, 815", and we're 95% sure that the average is between 1.236" and .764". Note that the certainty increases and the bounds decrease as the number of groups shot increases.
Since average group size lower than the lower bound is not "bad", we can reasonably rearrange the statements as follows:
With the three groups in the example above, we are 95% sure that average group size is below 1.535", and 97.5% sure that the average is less than 1.789"
Using the six group example, we are 95% sure that the average is below 1.185", and we're 97.5% sure that the average is below 1.236".
So we have two kinds of statements, the "between this number and that number" kind, and the "less than this number" kind. There are many statements that can be made, we can be 90% sure or 95% sure or 99% sure or 83% sure or any percent sure. We use 90%, 95% and 97.5%, these are commonly used. To be 99% sure takes a lot of groups; to be 75% sure isn't sure enough for many people.
Fortunately for us, the statistics works out that we can express these numbers as percentages of the average group size in the tables below. For an example, let's say that we shot five 5-shot groups at 100 yards, and have an average of 1.25 inches. In the 90% Certainty Table below at five groups shot, we see the upper bound at 123.4%, and the lower bound at 76.5%. These numbers are shown bold. Multiply the average of 1.2" by 123.4% and get 1.481". Multiply the average of 1.2" by 76.5% and get .918". We're 90% sure that the average group size is between 1.481" and .918".
Looking at the bold numbers in the lower table, we see that the average of 1.2" times 1.234% or 1.481" that we got just above is the upper limit, and we can say that we are 95% sure that the average group size is less than 1.481".
Imagine that we shoot another five groups, calculate the average for the ten groups as 1.15". Doing the multiplication, 1.15" *110.10% = 1.266", 1.15"*89.9% = 1.034", we're 90% sure that the average lies between 1.266" and 1.034", and we're 95% sure that the average is less than 1.266".
Here is a table for the "between this number and that number" kind:
90%
Certainty Table, 5 shot groups 95%
Certainty Table, 5 shot groups #
Of Groups Upper Lower #
Of Groups Upper Lower Shot Bound Bound Shot Bound Bound 2 273.60% -73.60% 2 449.40% -249.40% 3 153.50% 46.50% 3 178.90% 21.10% 4 132.40% 67.60% 4 143.80% 56.20% 5 123.40% 76.50% 5 130.50% 69.50% 6 118.50% 81.50% 6 123.60% 76.40% 7 115.30% 84.70% 7 119.20% 80.80% 8 113.00% 87.00% 8 116.30% 83.70% 9 111.20% 88.80% 9 114.10% 85.90% 10 110.10% 89.90% 10 112.40% 87.60% 15 106.50% 93.50% 15 107.90% 92.10% 20 104.80% 95.20% 20 105.70% 94.30% 30 103.20% 96.80% 30 103.90% 96.10% Note that for 2 groups, the lower bounds in each table are minus quantities. I don't know what that means, but I'd like to shoot one group of -1" or so. Maybe we can forget about these negative numbers.
Here's the table for the "less than this number" kind.
95%
Certainty Table, 5 shot groups 97.5%
Certainty Table, 5 shot groups Average
Average
Group
size Group
size #
Of Groups Is
less #
Of Groups Is
less Shot Than Shot Than 2 273.60% 2 449.40% 3 153.50% 3 178.90% 4 132.40% 4 143.80% 5 123.40% 5 130.50% 6 118.50% 6
the Z statistic for (m1-m2) with variances unknown because n must be 30+ and the variance must be calculated.
the t statistic for (m1-m2) because the variances must be equal.
ANOVA because the variances must be equal.
the Wilcoxon rank sum test because the variances must be equal and tables stop at n=10.