Accreditation for Complex Electronic Instruments
About this Article
This article, written by Dave Abell of Agilent
Technologies, Santa Clara, was presented at the May 2001 Simposio
de Metrología by Sergio Lopez-Carmona. The conference was organized
by the Mexican national metrology institute, CENAM.
Summary
Accreditation for calibration has been long sought after by suppliers
and users as a way to reduce multiple independent audits. With the
adoption of ISO/IEC 17025 and the rapid development of mutual recognition
through organizations such as ILAC and NACLA, the groundwork has
been laid for a robust, internationally recognized system for calibration
laboratory accreditation. As the old axiom goes, caveat emptor,
"let the buyer beware". Purchasing calibrations from an
accredited laboratory for a complex electronic instrument doesn't
assure that you will necessarily receive a better or even more complete
calibration than from a non-accredited laboratory. This presentation
explores the questions to ask when having an instrument like a spectrum
analyzer or precision digital multimeter calibrated.
Choosing the Right Standard
Several standards address the calibration of test equipment, but
one has achieved almost universal adoption as the basis for accreditation:
ISO/IEC 17025. The general quality standard, ISO 9000:1994, has
a section titled 4.11 Inspection, Measuring, and Test Equipment
that lays out the basic requirements for calibration. Section 4.11
also points out the need for adequate measurement uncertainties
in the related procedures. Typically an assessor to ISO 9000 doesn't
have the technical background to make a judgement on the adequacy
of calibration procedures. ISO 9000:2000 7.6 Control of Measuring
and Monitoring Devices references ISO 10012 for guidance, but
this standard is not normally utilized by laboratory accreditation
agencies.
The use of an accredited laboratory is meant to reduce multiple
audits to a supplier. Accreditation, unlike registration of a quality
system, adds the dimension of technical proficiency to the assessment.
An accredited laboratory has been evaluated not only on the robustness
of their quality system, but also their technical proficiency to
determine if they can actually make the measurements to the degree
of accuracy claimed. If a laboratory is accredited in this manner,
it eliminates the need for second party audits since the purchaser
can rely on the results from the third party accreditation assessor.
It is for this reason - technical proficiency and assurance - that
some industries and customers rely on accreditation as the hallmark
of a capable laboratory. QS9000 developed by the big three automotive
companies - Ford, General Motors and Daimler Chrysler - is moving
to require accredited calibrations for suppliers. One of the difficulties
they are encountering, however, is the scarcity of accredited labs
for calibration. Until there are sufficient laboratories available,
the Automotive Industry Action Group (AIAG) recommends some alternatives
such as second party audits. See the AIAG web site for details.
ISO 17025 General requirements for the competence of testing
and calibration laboratories applies both to companies that
calibrate testing equipment as well as those that do materials and
other kinds of testing. Also, ISO 17025 is based on ISO 9000:1994,
not ISO 9000:2000. This may require some suppliers to carry dual
elements in their quality systems - one for the '94 version and
one for the '00 release of ISO 9000.
Calibration Quality System Basics
Figure 1 shows at a conceptual level the components of a
quality system required by the ISO 17025 standard. The assessment
done for accreditation to ISO 17025 only reviews the calibration
operation of a business. Other services, such as repair provided
by a calibration laboratory, are not included in the scope of ISO
17025 assessment. 17025 review is not a substitute for the broader
evaluation done by an ISO 9000 auditor. The elements of the quality
system are essentially the same, however, and there is a great deal
of leverage for a supplier between these two quality systems.
Figure
1 -- Basic requirements of ISO/IEC17025
The unique difference for the evaluation by ISO 17025 is the assessment
of the technical capability of the laboratory to perform the work
required by the customer. For the calibration of electronic equipment,
this is highly dependent on the laboratory's traceability and measurement
uncertainty as indicated by stars in Figure 1. As defined by the
International Vocabulary of Basic and General Terms in Metrology
(VIM), traceability is the "property of the result of a
measurement or the value of a standard whereby it can be related
to stated references, usually national or international standards,
through an unbroken chain of comparisons all having stated uncertainties."
Traceability assures that the measurements made by the laboratory
are comparable to those made by others, a fundamental requirement
for fairness in trade. Knowing measurement uncertainties associated
with the procedures allows an assessor to judge the adequacy of
the laboratory's calibration methods.
Checklist for Calibration Services
Philip Stein in "Calibration Buyers, Beware!: What to look
for when considering a commercial calibration laboratory"
(American Society for Quality's Quality Progress magazine,
Sept. 2000) suggests these questions be addressed:
- Is the laboratory …accredited?
- Is the body that accredited this laboratory a signatory to
one of the lab accreditation agreements?
- Are the measurement parameters you wish to have calibrated
listed on the laboratory's scope of accreditation? Are the ranges
of the parameters you have chosen within the scope?
- Have you specified accredited service on purchase order to
the laboratory?
- Do all the certificates you received from the laboratory have
a logo from the accreditation body, and are no exceptions taken
for specific results?
This is good advice, especially item three. If the device to be
calibrated is simple, such as a gage block, then it is straight
forward to compare the parameters to be calibrated against the scope
of the laboratory's capability. In the case of a complex electronic
instrument, this comparison may not be so simple, however.
Scope versus Equipment Specifications
Most users of an electronic instrument simply want to know if it
meets the specifications as advertised by the manufacturer. The
accreditation systems in use today are the product of measurement
professionals at various national laboratories, with minimal input
from manufacturers of electronic instruments. Calibration services
provided by national laboratories don't reference an artifact's
specifications, but provide a detailed and precise report on the
characteristics of the device being tested. This is one of the problems
with calibrations for electronic test equipment -- the accreditation
system is based upon providing an objective measure of characteristics,
not stating that the device operates within a series of specifications.
Accredited calibration laboratories can, of course, provide a calibration
within their scope of capabilities that reports whether or not an
electronic instrument meets its specification. The challenge for
the purchaser of calibration services is to choose the laboratory
which has the necessary capabilities to provide a statement of conformance
to specifications. An instrument specification is typically written
by a manufacturer in a different form than the way in which an accrediting
agency writes a scope of accreditation. This means that the purchaser
must make a translation between these terms and formats to determine
the adequacy of a laboratory's capabilities. The accredited calibration
laboratory may have already done this if it has calibrated the specific
model in question before. If not, the laboratory may have to do
some research before it can state whether or not it can fully calibrate
an instrument to manufacturer's specifications.
Figure
2 -- Scope versus equipment specification
Specifications for an electronic instrument are derived from the
design characteristics of the instrument (see Figure 2).
In many cases, the instrument is not actually 'calibrated' on a
test system at the end of the line, but the specifications are assured
through the manufacturing process. Constant measurement of this
process makes it possible to be confident that the instrument fully
meets its specification when it reaches the end of the production
line. The specifications may not be directly tested, but inferred
from other operating parameters of the instrument. Users also like
to see a number of 'typical' specifications to better understand
the operating characteristics of the instrument. These typical specifications
are usually not guaranteed, but given as an example of nominal performance.
Reputable companies will stand behind their products, but matching
up an obscure 'typical' specification with a scope of accreditation
may be difficult.
The scope of accreditation is written for a specific International
System (SI) unit at a particular level of measurement, with its
corresponding uncertainty. The manufacturer's calibration procedure
is written to insure all of the advertised specifications are within
the limits called out in the data sheet. In some cases, the actual
capability of the instrument may exceed those specified in the data
sheet. The best combination is having the scope of accreditation
cover all of the stated specifications with the measurement uncertainty
needed to assure the data sheeted performance. For simple instruments
such as DC voltmeters, this may be possible. For more complex instruments
such as microwave signal analyzers, this may be very difficult to
accomplish.
The purchaser of calibration services should also be aware that
assigned uncertainties by the laboratory may be larger than the
published scope and may also be larger than the specification precluding
the inclusion of a compliance statement. The laboratory's scope
for an RF power measurement may have been done using type-N connectors,
but the instrument in question uses a different kind. In this case,
the recipient of the calibration report must determine if the measured
characteristics are adequate for the intended use. To further complicate
the matter, an instrument specification may be related to SI units
through more than one parameter. Resolution bandwidth is one such
example.
An experienced lab should communicate clearly to a customer what
they can and cannot provide in the way of service. An informed customer
will ask the question "can you provide a fully accredited calibration
on this instrument?" If the answer is 'no' the lab must tell
which specifications must be eliminated or done with greater measurement
uncertainty.
Three fundamental questions should be asked when determining if
an instrument can be fully calibrated:
- Is the range of parameters available?
- Is the required cal method available?
- Is the measurement uncertainty of the laboratory's accredited
scope sufficient?
Simple example: DC Voltage
The top table in Figure 3, is a excerpt from the data sheet
specifications for the instrument. The bottom table is an excerpt
from the scope of accreditation for a laboratory when generating
DC volts. Notice that the data sheet specifications are in '% of
reading + % of range' whereas the scope document is for generating
a voltage with 95% confidence '6 ppm + 0.4uV' at up to 0.22V. The
scope also includes the equipment that is used to meet this specification.
Figure
3 -- Comparing specifications for an Agilent 34401A voltmeter
against a calibration laboratory's accredited best measurement uncertainties
Without any other information, we must assume that the instrument
specification is at least 95% confidence (k=2). Many electronic
instruments were designed before the wide acceptance of the Guide
to Expression of Uncertainty in Measurement (GUM), so the design
teams may not have used the specific GUM language and methodology.
Most instrument manufacturers are, however, conservative in nature
and it is safe generally to assume at least 95% confidence.
In this case, the comparison between the manufacturer's specifications
and the laboratory's scope of accreditation is quite simple, requiring
only one intermediate step (see Figure 4).
Figure
4
First, choose the voltmeter's data sheet table for one year specifications
at 100mV. The accuracy as a percent of reading and as a percent
of range is listed at 0.0050 and 0.0035 respectively. Using simple
multiplication, this means that the indicated voltmeter reading
is ± (5 mV + 3.5mV) or a total of ± 8.5mV maximum.
Looking at the laboratory accreditation scope, 100mV is covered
by the laboratory's capability at the 0.22V range. The measurement
is listed as the ability to generate a voltage with a specific accuracy
at this range. In this case, the accuracy is listed in ppm, or 'parts
per million' which is 1/1000000 or 0.0001%. Thus, 100mV generated
on this range has an accuracy of ±(6ppm*100mV + 0.4mV ) or
±(0.6mV + 0.4mV ) or a maximum total of ±1.0mV . This
is well within the specification of the voltmeter. As long as the
laboratory uses the method that was prescribed by the assessor for
this voltage range, the specification of the voltmeter can be assured.
Complex example: Resolution Bandwidth
In the next example there is not a one-to-one correspondence between
the instrument specification and the scope of accreditation. Figure
5 is a section of the catalogue specifications for a series
of spectrum analyzers made by Agilent Technologies which extend
up to 26.5GHz.
Figure
5 -- Extract from a typical spectrum analyzer specification
One of the first specifications is Resolution Bandwidth (RBW).
For the Agilent ESA series of signal (spectrum) analysers, the accuracy
is stated as ±15% for 1kHz to 3MHz and ±30% for 5MHz
settings. What does it mean in relation to the typical scope of
best measurement capability published by an accredited laboratory?
RBW is an example where the specified function doesn't appear in
the capability listing, although the performance is stated in terms
that are listed such as Hz or frequency.
Is the uncertainty for this test defined by the lab's frequency
capability? No, unfortunately it is more complicated than that.
To determine the uncertainty it's necessary to investigate the test
procedure. This may vary from lab-to-lab or even model-to-model.
The RBW uncertainty is actually a function of amplitude accuracy
(y-axis) as attenuation or power and knowledge of the input filter
shape, specifically its roll-off characteristic, translated into
equivalent frequency (x-axis).
Figure
6 -- Relationship between amplitude and frequency uncertainty
The diagram in Figure 6 shows the frequency uncertainty
only on the left in the expanded view. Essentially the same figure
must be added for the right side of the signal curve. The typical
uncertainty for this product is 1% -- nothing like the frequency
uncertainties stated in the schedule that might initially be thought
to relate to the RBW specification. For example, the accreditation
schedule for an Agilent Technologies laboratory in the UK states
frequency accuracy of 2 x 10-10 to 4 x 10-9
in the 0.01 Hz to 100 MHz range. This is considerably smaller than
the actual capability for RBW at 1%. The exact calculation of accuracy
available from the laboratory depends upon the error equation of
the method used, which includes both frequency accuracy and RF power
amplitude accuracy.
The customer alone can't determine the extent of calibration or
the possible uncertainties from the published accreditation scope
schedule. They will need to rely on the deeper analysis done by
the supplying laboratory. This may also cause delays in ordering
calibration since the laboratory may not have calculated the error
equations for the specific model being requested, and may have to
do considerable research.
Again, the most direct route is for the customer to ask if the
laboratory has ever calibrated this model instrument before and
if so, what limitations arose.
Conclusion
All accredited calibrations are not equal. A laboratory could have
a simple scope of accreditation for DC/AC volts based upon a single
commercial calibrator, or an extensive scope including arcane procedures
for difficult optical or microwave parameters. The buyer must be
aware of the parameters that are critical for their use when choosing
a laboratory.
In any case, a buyer must be flexible when purchasing accredited
calibrations. Even the best of laboratories may not have the accredited
capabilities in optical power, phase noise measurements or other
state of the art parameters. In fact, some leading edge instruments
are difficult to calibrate, and some can only be done to full accuracy
at the manufacturer's production facility.
However, there is value in purchasing from an accredited laboratory.
Even if their accredited uncertainties are too large to verify an
instrument's performance against its published specification, the
fact that they have undergone a rigorous evaluation emphasizes their
commitment to quality and technical proficiency of their measurement
services.
Like most transactions, it is important to establish a level of
trust and evaluate the reputation of the calibration provider. It
also is good to work with a laboratory that regularly calibrates
the type of complex electronic instrument you own since they will
more likely have done the detailed analysis need to connect the
instrument specifications with the scope of accreditation. Accreditation
goes a long way towards improving this selection process but, one
must always keep in mind, however, it is not a comprehensive solution
in this measurement arena.
Acknowledgements
Michael Hutchins of Agilent Technologies, United Kingdom contributed
the example for resolution bandwidth. Greg Burnett of Agilent Technologies,
United States contributed the example for DC volts. Both helped
greatly with ideas and proofreading.
Bibliography
- Examples of Schedules of Accreditation
- Hysert, Gary, The CLAS Way to Cal Lab Accreditation
(Boulder Colorado: NCSL International Proceedings NCSL 2000).
- Stein, Philip. Calibration Buyers, Beware !
- Automotive Industry Action Group AIAG, QS9000 requirements
- International Vocabulary of Basic and General Terms in Metrology
(Geneva, Switzerland International Organization for Standardization,
1993).
- ISO/IEC 17025. General requirements for the competence of
testing and calibration laboratories (Geneva, Switzerland
International Organization for Standardization, 1999).
|
Search | Site Map | Contact Us | Login | Home 
