Viability of Accreditation in Production

Challenges of an ISO/IEC Guide 25 Accreditation System in a High-volume, Production-oriented Environment

Formal criteria for the assessment and recognition of the technical element in calibration service provision has existed in some countries for over a quarter of a century. By contrast, independent appraisal of quality management practices is a relatively recent development. Yet the success of ISO9000, as defined by wide-scale pursuit of registration and its adoption in vendor selection criteria, has not been mirrored in "accreditation". However, validation of the product/service actually delivered is beyond the scope of the standard. For testing (including calibration) this is addressed by the ISO/IEC Guide 25 / EN45001 standard and it is, perhaps, surprising that equal emphasis does not seem to have been given by firms to development of such accredited facilities adequate to meet the total demand of, the otherwise, quality-conscious industry.

The extent to which the operating requirements of national accreditation bodies, such as the UK's NAMAS, may have contributed to this situation is explored from the perspective of the practices of a large, commercial accredited service organization.

Accreditation in Britain

The National Measurement Accreditation Service was formed in 1985 as a result of merging the British Calibration Service (BCS) with the National Testing Laboratory Accreditation Scheme (NATLAS) and is currently a division of the measurement standards bureau, the National Physical Laboratory. By March 1995 the total number of NAMAS accreditations exceeded 1700, of which 512 were for calibration. Established in 1968, the BCS was for many years targeted at high-level laboratories in government departments and industry. It is not the oldest such agency -- an honor which belongs to the Australian agency, NATA, dating back to 1947 -- but has established a respected leadership role not least because of the illustrated "business results". Figure 1 shows the number of calibration certificates issued annually, in total and for the electrical field while Figure 2 illustrates the continuing growth in electrical accreditations. Recently, the emphasis in this field has been in accreditation for on-site calibrations although permanent facilities continue to increase in number too.

Many countries have sought to copy the NAMAS model when establishing their own accreditation agencies. Its position is supported by a useful collection of guidance and regulatory documentation, the creation of which owes much to the contributions of committees including industry, academia and other government department representatives.

Of particular note in this regard is a recently revised practical guide to uncertainty determination ⁽¹⁾ which also now addresses non-electrical measurements. The primary criteria for accreditation ^(2,3) is based upon ISO Guide 25 with substitution for subjective language, while further prescriptive detail is provided by several other documents ^(4-8) which enlarge upon particular aspects.

So What's Wrong ?

An indication from a survey made by NAMAS last year supports the widely-held belief that accredited calibrations account for a fractional part of the total UK calibration business. This is especially the case in the electrical field where, even in the accredited laboratories, NAMAS certifications account only for an estimated 10-20 percent of calibrations performed. Interestingly, in the dominant mechanical (dimensional) field, the majority of accredited labs. deliver NAMAS certification as their standard practice. A possible explanation for this disparity is discussed later.

Despite NAMAS' published assertion ⁽⁹⁾ that calibration integrity is only assured through the service, the majority of customers seem to be satisfied with company-proprietary calibration certificates. Although not attesting to the technical validity of the work performed, most calibration suppliers' quality management systems are ISO9000 registered and this "evidence" of acceptable process control is typically prominent in the vendor selection criteria of other ISO9000 compliant companies (over 30 thousand such UK registrations).

This ratio of accredited to non-accredited certifications is certainly not dissimilar to other countries with recognized accreditation schemes. The operation of traditional standards laboratories has provided the basis for the regulations and expectations of assessors. Whilst untrue for NAMAS, agencies in some countries have preferred to focus on high-echelon industrial labs., declining requests to assess general-purpose/lower grade calibration areas or for work done on customer sites. However, to address the mass-market for calibration driven by a general trend towards "out-sourcing" and ISO9000 etc., the overall capability and capacity of the service represented by accredited facilities must increase.

Considering NAMAS as a typical ISO/IEC Guide 25 agency, which accreditation criteria or expectations can present implementation difficulty in a large, production-oriented service operation that may be less significant to a lower volume standards room? The issues can be divided into two groups -- the technical, relating to validation of test methods and logistical, concerning the actual deliverable that is evident to the customer.

Technical

Probably the single biggest challenge for anyone seeking calibration accreditation is the requirement to evaluate the measurement uncertainty of all testpoints in accordance with an internationally recognized "recipe" such as the ISO Guide to Uncertainty in Measurement, NIST publication 1297 or, particularly in Britain, NIS3003 ⁽¹⁾.

Unfortunately, this scientific technique infers a necessity to collect multiple data-sets by repeated measurement in order to assess the type-A or "random" component of the total uncertainty. The mean for each testpoint data-set would be the actual value reported in the calibration certificate.

This rigorous approach may be practicable for traditional laboratory artifacts such as resistors, voltage references or attenuators having one or two parameters and few points to characterize. However, in the case of instruments, customers usually expect the calibration to effectively assess the product for its full capability as indicated by a specification.

This will often involve measurements at a multitude of points for different ranges and functions which can, even with a high-degree of automation, extend to several hours -- days if done manually. Few customers are willing to pay 3 , 5 or 10 times the cost of a single calibration that would be asked because of the extra time required just to perform these repeated measurements. In addition, longer equipment downtime may be an issue if it's from a production area. Clearly, a more viable method needs to be devised.

The practice adopted by HP's UK service center and accepted by NAMAS relies upon a "type-test" assessment carried-out on the first example of the model encountered. The uncertainties thus calculated are then attributed to all subsequent calibrations on that model-type. Figures 3 and 4 present summary extracts from a paper ⁽¹⁰⁾ justifying the method.

There is an acknowledged (unquantifiable) decision-risk concerning the significance of variation between only two data-sets or whether more extensive testing is necessary to calculate the real "random" element. Consequently, consideration should be given to any trends evident in the series of result-differences; the poorest repeatability value being attributed to the range/parameter rather than to individual testpoints. But even using this minimalized method, the time/cost-burden of performing two calibrations cannot be regimentally followed and a further short-cut may be sought.

A knowledge of the unit-under-test's operation and construction can justify assignment of repeatability figures determined for a similar model/family or even generic category/class, providing the same procedures and equipment are used. When taking this option, some attempt at verification at selected testpoints is advisable; manufacturers sometimes improve their product during its life-cycle by circuit modification, better component performance, etc., which may not be apparent by change to the model number. For instance, earlier "A" versions may not be as stable as the later-released "B".

To provide confidence that the "type-test" remains valid for the product population and hence uncertainty assignment in later calibrations, a sample audit process is necessary which includes determination of differences between repeated measurements at selected points and comparison with the "random" allowances. Obviously, excursion suggests that a full reassessment of the uncertainties may be required. So far, concentration has been on the "random" uncertainty -- but what of the type-B or "systematic" component ?

To meet the requirement of the standard it's necessary to produce an uncertainty budget which numerically quantifies the possible error-sources. At HP, like many others, the general practice to-date has been to base calibration services on the manufacturers recommended procedures, equipment and testpoints. Owing to the many permutations that this policy creates, a huge man-power investment is needed to properly evaluate the measurement accuracies in a manner consistent with modern-thinking. Having embarked upon this route in 1991 as part of a strategy to provide "NAMAS calibration" from our high-production service center (rather than the standards labs.), documented budgets for several hundred models have been produced. In volume terms, the top 150 models account for 40 percent of our 30 thousand item calibration business with the remainder falling into the less than 10 per year category. Continuation with this model-by-model approach is therefore inappropriate for the thousands of items where low volumes are expected. This realization has been supported by a study made by our US colleagues which indicated the task to require 70 man-years, even for only their existing input. Of course with new products appearing on the market every day, the goal would be unattainable with available resources.

Consequently, we are attempting to rationalize our testing methods and, ideally, develop generic uncertainty budgets based on a more select test gear list. A downside of this, especially for HP manufactured instruments, is the potential customer concern that we are deviating from the recommended performance test as published in a product's service manual. Another is that existing software developed by various HP operations worldwide may be unusable for this calibration option; serious since 70 percent of our input volume is ATE calibrated. It's clear that with limited resources in any one country, a coherent, company-wide strategy is needed.

Logistical

Calibration certificates must be in a NAMAS-defined format which extends to all pages of the report, not simply the first page which usually carries only item identification, environmental and other "legal" details which may include a specification status statement -- more on this later. Measurement data must be supplemented with uncertainties, set-up conditions and summary description or reference to the method used.

By tradition, this "full" certificate has become most customer's expectation but NAMAS does allow an abbreviated or "short-form" certificate. This is applicable only to equipment having performance specifications since test results are not presented. An extract from an example issued for an HP8574 EMI Spectrum Analyzer system is shown in Figure 5.

Figure 5 -- Extract from a "Short-form" Certificate for an HP8574A EMC System. The status of 25 tests are reported altogether on only 6 pages - condensed from about 60 pages of test data. Click image to enlarge.

It is particularly useful for complex products involving an extensive calibration routine and where test data is recorded by hand, or automatically printed in a form which is unacceptable to NAMAS, so would need to be transcribed. Even with a "short-form" presentation the certificate design is unique for each model. This contrasts with the common-look of our single-page "commercial" certificate, which is rather similar to the cover-page of a NAMAS certificate described earlier, augmented by a copy of the results in original format. This practice is common and evidently meets the needs of the majority of the market; the inference is that accredited certificate demands are out of step with customer requirements.

For the relatively simple devices often found at standards laboratory level, it is normal to use calibration data for "corrections" or in SPC programs. With general-purpose M&TE such use is cautioned against since complex relationships can exist and insufficient data is supplied in the calibration for reliable predictions/corrections to be made. Confirmation, or otherwise, of specification compliance is the principal need. Most users recognize this and have no interest in the intricacies of the certified results, preferring a simple status statement. Its absence means a failure to meet expectations.

However, NAMAS does not require these status remarks to be included in a "full" certificate. Since "certification" is often (incorrectly) assumed to imply specification conformity, there may be a case for recognizing and protecting against misinterpretation by making status advice mandatory or, where a specification is unknown, provision of a calibration report rather than certificate.

Where measured values are compared with specifications, NAMAS requires that acceptance limits are determined by simple arithmetic difference of specification and uncertainty -- graphically shown in Figure 6.

Although widely adopted, particularly in the US, NAMAS rejects the producer/consumer risk management approach so this conservative guardband must be established whatever test accuracy ratio (TAR) exists. Whilst it assures the customer that there is very little chance that an out-of-tolerance parameter is reported as in-tolerance, it can lead to a significant likelihood that unnecessary adjustment and re-testing will be incurred by the calibration lab., especially when TARs are low. These costs must be passed-on to the customer. Further, the desire for a specification compliance statement cannot be satisfied when measured values fall into the "gray area". Neither are labs. allowed to quantify the consumer-risk which might assist the customer meet his own quality system criteria. Again, selection of accredited certification seems to make life harder for the purchaser.

Similar to the matter of specification status, NAMAS feels it has no mandate to define the extent of service provided for any particular generic equipment-type. Many users are unaware of this, assuming that the validity of measurements performed (for which the agency does acknowledge responsibility) also implies "policing" of the calibration's scope. From the customer's viewpoint it is another reason to be wary in comparing the offerings from competing service-providers.

Several documents were published by BCS which attempted to provide guidance on the extent of calibration for a few instruments but these were not compulsory. The diversity of modern electronic equipment makes it difficult to devise a suitable testing procedure applicable to a generic-type. There are no British Standards prescribing the performance characteristics and verification of general-purpose electronic instrumentation which is not used in safety-critical, consumer or legal metrology situations. In other measurement fields, such standards exist and are well-established in the user-community. Thus, "tested in accordance with CISPR 16" (EMC) or "calibrated against BS 870" (micrometers) concisely describes the extent of service. These standards often also define the precise equipment to be used, together with acceptance limits (which allow for uncertainty in some way accepted in the relevant industry-sector and which supersedes NAMAS' own policy). Consequently, the ease and commercial viability with which NAMAS' requirements can be complied with is enhanced.

Having finally produced the calibration certificate, a NAMAS authorized person must sign it to provide "authentication". This is intended to be a 100 percent final inspection to confirm correctness which is manageable for low volumes and comparatively short certificates issued for traditional laboratory references. Impose that same practice on multipage certificates for products with many functions/ranges/testpoints and the task becomes significant and adds delay if a limited number of authorized personnel are available. Modern quality management recognizes that "quality" cannot be inspected-in at the end of the process which seems to be the intent. Acceptance by accreditation bodies of properly implemented sampling audit (a cornerstone of most quality management systems) as a means of protecting certificate integrity rather than signature, would eliminate this associated cost.

Added Assurance or Caveat Emptor * ?

The foregoing gives an indication why the critical technical assurance conferred by accredited calibration, in its presently practiced format, often incurs additional delivery costs. Inevitably, while these practices are not "standard" in a laboratory, this extra cost is passed-on to those customers demanding such a service. This position will become less tenable if a much larger proportion of the user-community, which is currently indifferent to accredited calibration because of the cost-premium, alters its purchasing behavior in appreciation of ISO/IEC Guide 25's purpose and limitations of ISO9000 registration alone. Commercial pressure to remove any premium will intensify which, if not effectively policed, may lead to a reduction in overall product integrity through less extensive testing as a cost-cutting measure. The "buyer-beware *" caution would remain and the value of accreditation diminished.

The possibility of this scenario could be averted by developing a simplified form of accreditation more practically applicable to general-purpose equipment verification which, while remaining consistent with the general principles of ISO/IEC Guide 25, allowed more flexibility in implementation. Such a proposal ⁽¹¹⁾ was produced last year in order to stimulate debate, especially at national level.

Conclusion

The purpose was to summarize operational aspects of a representative accreditation program which are felt to be factors in the restricted provision of accredited calibration, particularly in the electrical field.

Clearly, a balance must be achieved between metrological ideology and commercial practice to encourage new or extended accredited facilities necessary to satisfy the requirements of the mass-market. Most importantly, such consensus must lead to actual use of these resources in delivery of all calibrations. The readiness of the authorities to respond to the sensitivities of today's business environment will be key to progress, through the co-operation of the service providers and not by dictate.

If this can be done, it will be to the mutual benefit of assessors, cal.labs. and equipment users. We will indeed have risen to the challenge of producing a viable framework to attain the perceived intent of ISO9000 - real "quality" in the product, not just the process.

References

1. NAMAS NIS3003 (1995, Edition 8) - Uncertainty and Confidence in Measurement
2. NAMAS M10 & Supplement - General criteria of Competence
3. NAMAS M11 - Regulations
4. NAMAS M18 - Accreditation for Site Calibration & Testing
5. NAMAS M13 - Conditions for Use of NAMAS Logo [includes cal label design spec.]
6. NAMAS M25 - Certificates of Calibration
7. NAMAS M16 - Quality Manual Preparation Guide
8. NAMAS M51 - Quality Audit & Quality System Review
9. Rogers, J. (NAMAS), "Validity of Calibration & Test Data" : Paper published by NAMAS contrasting ISO9000 with EN-45001 and summarized in their Newsletter (Spring 1994).
10. Instone, I. (Hewlett-Packard Ltd., Winnersh, UK), "Calculating the Uncertainty of a Single Measurement" : Institution of Electrical Engineers (London), digest number 1993/109.
11. Abell, D. (Hewlett-Packard Co., Mountain View, Ca.), "Proposal for Simplified M&TE Accreditation", February 1994: Public discussion paper drawing upon ideas and work of the NCSL TQM Committee on Calibration System Requirements in relation to ANSI/NCSL Z-540-1 development and submitted to senior management of organizations including NAMAS, NIST, NAVLAP, Standards Council of Canada and EAL.