Uncertainty & Confidence in Measurements

An Introduction for Beginners

This is one of the most challenging aspects of calibration -- but are we sure??? If you are also uncertain (or lack confidence!) about some of the special language used in metrology, you may wish to review our terminology guide before continuing with this more detailed explanation.

• Terminology Guide

But before we scare ourselves with the mathematics involved, let's consider the fundamental questions... who cares about uncertainty and why?

Significance of Uncertainty to User and Calibration Lab

Specifications define any product's performance capability so that its adequacy for a certain task may be determined. In order to have confidence that outgoing product meets its specification, good practice is for the "standard" to be several times more accurate than the item being tested. A rule-of-thumb for this so-called test accuracy ratio (TAR and also known as test uncertainty ratio, TUR) is for it to be greater or equal to 4:1 (as indicated in ANSI/NCSL-Z540).

Where a measurement involves more than simple comparison which means that the overall accuracy of the test is less evident, perhaps because several items of test equipment are involved or environmental factors (including test method) influence the result, an uncertainty budget should be developed. This is also termed an error budget but this is not encouraged since, by definition, errors are known and can usually be taken into account by correcting measured values, whereas uncertainty merely defines the limits of potential inaccuracy.

Until 1993 and the publication of the ISO Guide to Expression of Uncertainty in Measurement, there was no international consensus on the method for calculating uncertainty. This is also a reason that manufacturers' specifications lack consistent definition. Of course, compliance isn't mandatory but at least standardization may be encouraged by it. A statistical approach is recommended, including the combination of contributions by quadratic summation and reporting the final value at a 95% confidence level with Gaussian distribution. Where the confidence level or distribution of a contributor is unknown, such as is often the case with instrument specifications, the "worst-case" rectangular distribution form is assumed and the equivalent 95% confidence, normal distribution error-limit calculated. The attributes of the item under test must also be considered in the budget.

Uncertainty, Test Limits & Risk

It is also common industry practice to use the specification as the acceptance (or test) limit when deciding compliance of the tested item. Except that uncertainty shall be taken into account when the TAR falls below the prescribed minimum, the referenced standards do not stipulate how it should be done.

Acceptance of this practice establishes a maximum consumer risk for the tested item being incorrectly determined as within tolerance. Assuming both the specification and uncertainty have Gaussian distribution at 95% confidence and that the TAR is 4:1, a test result at the specification limit means that there would be a 0.8% chance that it was, in reality, out-of-tolerance. There is an associated chance of conforming product being falsely declared non-compliant, resulting in unnecessary corrective action. In this example, the producer risk is 1.5%. As the TAR reduces, these risks increase but by setting a test limit which is tighter than the specification, the possibility of incorrect acceptance can be maintained at the same level as the de facto standard 4:1. Rather complex mathematics is necessary to determine the actual test limit for a particular TAR but this guardbanding provides a mechanism to comply with the standards' need to account for uncertainty, thus protecting the customer from undue risk of unknowingly using nonconforming product, while limiting the supplier's commercial exposure.

An alternative method, often demanded under accreditation schemes based on ISO/IEC17025, particularly in Europe, is to guardband to the full extent of the uncertainty, whatever the TAR. That is:-

Test Limit = Specification - Uncertainty

This provides attractive minimal risk to the consumer, for instance 0.02% at 4:1 and only 0.03% at 2:1, but significant producer risk of 10% at 4:1 (33% at 2:1). The economic impact to the supplier (and inevitably the customer) may therefore outweigh the benefit of this practice.
Opponents argue that such a conservative approach is unnecessarily pessimistic and is inconsistent with the established statistical basis for error (uncertainty) propagation. To explain, the specification will contribute to the user's uncertainty budget by quadratic summation and, therefore, the setting of the acceptance limit by simple arithmetic is inappropriate. The calculation should be a quadratic difference:-

Test Limit = Squareroot [Spec ² - Uncertainty ²]

This provides a fairly constant chance of under 0.7% of false acceptance for TARs from 4:1 to 1.5:1 although the chance of incorrect rejection is 2% at 4:1 rising to 8.2% at 2:1.

This latter method is useful because of its simplicity in application, generally acceptable consumer-risk and commercial viability. It's argued that a statement on a calibration certificate, such as follows, could succinctly address  the uncertainty and measurement adequacy criteria of standards such as ISO9001 and ANSI-Z540.

"Our calibration procedures are designed to provide a measurement uncertainty of less than a quarter of the specification of the unit-under-test. In these conditions and at 95% confidence level for specification and uncertainty, the chance of incorrect declaration of conformance to specification is 0.8%. Where the objective cannot be achieved, tightened test limits are used to maintain equivalent confidence in the product's compliance to specification."