Performance Evaluation Laboratories

Why participate?

Independent laboratory performance evaluation of in vitro diagnostics (IVDs) submitted for prequalification assessment enables WHO to verify certain standards of performance that are considered essential for resource-limited settings.

Manufacturers who wish to submit a product for prequalification have two options with respect to meeting the performance evaluation requirement of prequalification assessment: performance evaluation coordinated and financed by WHO (option 1), or performance evaluation coordinated and paid for by themselves (option 2). But whichever option a manufacturer chooses, the evaluation must be performed by a designated Performance Evaluation Laboratory (PEL).

Laboratories that successfully apply to become a performance evaluation laboratory contribute to WHO's aim to increase access to IVDs for accurate diagnosis and monitoring of priority disease and conditions. In brief, such laboratories are a key partner for WHO prequalification.

Who can apply for listing as a performance evaluation laboratory?

Applications are open to laboratories with experience in conducting independent performance evaluations of IVDs, including:

WHO Collaborating Centres.
National laboratories or laboratories that provide testing services to the government in their respective country.
Laboratories located in geographical areas corresponding to the intended setting of use of the IVDs submitted for assessment.

Due to a sufficient number of PELs for the evaluation of IVDs for the diagnosis and/or monitoring of infection for HIV (including CD4), syphilis, hepatitis B, hepatitis C, applications for these analytes are currently closed.

Currently, applications are open for the following types of tests:

Human papilloma virus molecular tests
G6PD deficiency
TB molecular tests.

Requirements for WHO listing

WHO requirements for listing of a performance evaluation laboratory are specifically related to performance evaluations. An assessment will ascertain the potential for a laboratory to undertake a performance evaluation according to the relevant WHO protocol, as well as the laboratory’s compliance with the principles of ISO/IEC 17025:2005 and ISO 15189:2012. Additional references relating to good practice for laboratories performing IVD evaluations, including other ISO standards, will be referred to during the audit. The auditing process is based on the principles outlined in ISO 19011:2018.

More information about audit Stages 1 and 2, WHO and laboratory responsibilities (with respect to the on-site audit), organization and output of the on-site audit, and the principles that audits adhere to, are given below.

Assessment process

Assessment of a candidate laboratory consists of:

Submission by the laboratory of an EOI.
Stage 1 audit: assessment by WHO of the EOI and the specific quality management system (QMS) documentation submitted by the manufacturer.
Stage 2 audit: on-site audit by WHO of the laboratory to assess compliance with WHO requirements.
If the laboratory meets all requirements, listing of the laboratory as a WHO Performance Evaluation Laboratory.

Laboratories that meet assessment requirements will be informed by letter of their listing as a WHO Prequalification Evaluating Laboratory.

Maintaining compliance

Performance Evaluation Laboratories must ensure ongoing compliance with WHO requirements following listing.

Mandatory re-assessment, to verify ongoing compliance with WHO requirements and relevant standards, takes place every three to five years after laboratory listing. It takes the form of a partial or full audit. However, re-assessment may be carried out sooner than this if deemed necessary. In addition, WHO may decide to delist laboratories should they fail to deliver the agreed work (compliance with the protocol, agreed timeline) or to communicate adequately with WHO.

Responsibilities of WHO

It is generally the WHO lead auditor who undertakes to:

plan and prepare the audit
communicate with the laboratory
define the scope of the audit
prepare the audit plan, working documents, briefing documents and supervise the travel arrangements for the auditors
communicate any challenges regarding the audit to the laboratory management prior to or during the audit
prepare and present the draft audit report to the laboratory management at the closing meeting on the final day of the audit
submit the final report to the laboratory
follow-up on nonconformities with a request for a detailed corrective action plan from the laboratory.

Note: The WHO audit team will have experience and knowledge in laboratory QMS, performance evaluations and the specific IVDs for which the laboratory conducts performance evaluations. A WHO audit team usually consists of two members. The laboratory will be informed of the identity of the proposed audit team members before the site audit and receive their curriculum vitae to ensure that any conflict of interest, or other issues that may compromise the audit, are avoided. The laboratory will have the opportunity to express any concerns about the audit team before the audit takes place. If concerns about an individual auditor cannot be resolved in discussion with WHO, the laboratory should notify WHO in writing of its objection to that individual's member's participation in the site visit within 10 days of receipt of the proposed audit team composition. All members of the audit team are obliged to act according to the requirements of the International Civil Service Commission's Standards of Conduct for the International Civil Service.

Responsiblities of laboratory

Responsibilities of the laboratory will be communicated to the laboratory before the audit. In general, the laboratory's responsibilities will be to:

agree upon the objectives and the scope of the audit with the WHO team
inform WHO of any major changes to the QMS since the document review was performed
inform the WHO team of any issues that may affect an effective and efficient audit process
cooperate with the auditors to ensure that the audit objectives are achieved
identify a person responsible for coordinating and facilitating the audit process
inform relevant employees about the objectives and the scope of the audit
appoint responsible members of staff to accompany members of the audit team and to ensure auditors are aware of health, safety and other applicable requirements
provide on-site resources, for the audit team, to ensure an effective and efficient audit process, such as meeting rooms
provide access to the laboratory facilities, documents and records and other evidence as requested by the auditors in a timely manner to ensure an effective and efficient audit process and so that the audit timetable can be met.

Logistics for the on-site audit

Dates and time allocated for the audit

The dates and time allocated for the audit are to be agreed upon by all participants. It is the responsibility of the laboratory to ensure that during the audit the laboratory is performing routine testing and that the key personnel for the QMS and laboratory activities are present.

WHO will provide the audit plan at least two weeks before the audit and will include details of the audit to be conducted using information from the submitted documentation on the QMS, which includes the quality manual. The plan will be a guide only and flexible, to permit changes in emphasis based on information gathered during the audit.

The audit plan will:

provide information about the scope and purpose of the audit
identify audit team members
give the date and place of the audit, and the expected timing and duration of each audit activity, including meetings to be held with the laboratory's management team.

Language for the on-site audit

The audit will be conducted in English. To enable a smooth and effective audit, the relevant higher-level quality management documents must be available in English. Any translation needs will be discussed with the laboratory.

Travel and accommodation arrangements

WHO is responsible for the travel and accommodation arrangements of the audit team and all costs associated with the audit team. However, the laboratory will generally be asked to provide advice for making of these arrangements.

Meetings during the audit

Opening meeting

The opening meeting (20 minutes for the audit team and 40 minutes for the laboratory, approximately 1 hour in total) is held to exchange information between the audit team and the laboratory team on the audit process, to confirm the audit's scope, objectives and plan, and the availability of responsible persons on-site.

The lead auditor of the audit team (20 minutes approximately) will:

introduce the audit team
explain the functions and responsibilities of the WHO audit team
make sure that the attendance register is completed
explain the purpose, and objectives of audit
provide a short summary of the audit process as part of the WHO prequalification
confirm the audit plan including:
- the areas/activities of facility to be covered
- requirement for access to selected documents, records, reports
- the timetable for the audit (see example below)
introduce the methods to be used for the auditing, checklists and classification of findings
establish official communications links between the laboratory and the audit team
confirm that the resources and facilities needed by the audit team are available
allow the laboratory personnel to ask questions regarding the process
allow the audit team to seek clarification regarding the laboratory.

The laboratory (40 minutes approximately) will:

introduce principle staff
provide an organigram (with photographs if possible) and a written list with contact details for key personnel during the audit process
provide a brief overview of the QMS
provide a brief overview of the laboratory, particularly with respect to the evaluation of testing of the IVDs to be evaluated
inform the audit team of any changes since the submission of information about the laboratory to WHO
provide an activity schedule including shifts (if applicable) for the audit days.

Meeting of auditors

Auditors will meet as necessary throughout the audit, with the following meetings taking place as a minimum:

Discussion of findings: during the course of the audit, the auditors will confer regularly and informally regarding the progress of the audit.
Daily summary: At the conclusion of each day (or at the beginning of the following day), the lead auditor will present a brief summary of the day's activities and findings.
Audit summary: At the conclusion of the audit process, on the last day of the audit, the auditors will meet to discuss their findings in detail.
Preparation of on-site draft audit report: the auditors will then assist the lead auditor in summarizing any nonconformities found at the audit.

Daily wrap-up meeting

The laboratory staff will be invited to discuss the outcomes and any issues of concern (potential nonconformities) at the end of each day of the audit. This meeting allows the laboratory to discuss/clarify any issues of concern. If the laboratory contests a nonconformity it must provide a rationale in writing to WHO, including supporting evidence.

Closing meeting

The closing meeting concludes the audit and will be held in the presence of the audit team and the management team of the laboratory. Other staff members may be invited by the management, as appropriate.

The outcome of the audit, including areas covered and not covered, and limitations to the audit, will be presented. The audit team will summarize the findings and issues of concern in the order of significance, and present the list of nonconformities that will be rated for severity, in the format of a draft audit report.

Example of audit timetable*
	Audit activity – WHO lead auditor	Audit activity – technical auditor
Day 1
8.30–09.00	Team briefing meeting and audit Preparation with team.
9.00–10.00	Opening meeting as per opening meeting agenda. Introduction of personnel and overview of audit by WHO; overview of laboratory processes, principle staff and QMS by the laboratory.
10.00–11.30	Facility tour.
11.30–13.00	Horizontal audits (includes 15-minute break) QMS: Management responsibility including interviewing of senior management about: - organization and management system - document control/record control - external services and supplies - service to the customer and complaints - control of nonconforming work - improvement - correction actioon plan - internal audits - managements reviews.	Assessment of technical competence, including observations of performance of tests/vertical audits and interviews. Technical audit including but not limited to: - personnel - accommodation and environment - pre-examination (sourcing and handling of panels) - methods and method validation - ensuring Quality of examinations - equipment - reporting the results.
13.00–13.45	Lunch break (on-site).
13.45–17.00	Audit (continued).
16.45–17.00	Daily wrap-up meeting: auditors report briefly to laboratory personnel on day's findings.
Days 2–3
All day	Short opening meeting to schedule activities. Audit continued as above and as required, including breaks. Daily wrap-up meeting.
Day 4
9.00–12.30	Audit (continued).
12.30–13.15	Lunch break (on-site).
13.15–16.00	Auditors' meeting: discuss findings, prepare draft on-site report.
16.00–17.00	Closing meeting: present audit outcome and draft report and discuss findings with laboratory.

* Times may need to be modified to better comply with daily routine and the length of the site visit may need to adjusted to meet audit requirements.

Final audit report

The final audit report will be issued by WHO within 60 days of the on-site visit. If an audit report cannot be issued within this time, the laboratory will be notified regarding the cause of the delay.

Report content

The final report will describe the main findings, issues of concern and summarize the general outcome of the audit. It will contain an annex with substantial, specific information on the nonconformities observed by the auditors.

Reports with no requirements

If no nonconformities were recorded, WHO will notify the laboratory by letter of successful completion of the on-site audit. Notification of the laboratory listing will be provided in a separate letter.

Reports with requirements (relating to nonconformities)

The report will include a description of the nonconformities found during the audit, their severity, findings that contributed to this nonconformity and the relevant internal QMS document or specific requirement in the standard (individual clause or subclause). The severity of nonconformities will be classified into major, minor findings or observations. The laboratory will be expected to take appropriate action to address the nonconformities before listing.

Laboratory corrective action plan (CAP)

A proposed CAP must be submitted by the laboratory within 10 days of receipt of the final audit report. For each identified nonconformity, the CAP must include:

root cause analysis;
immediate action taken where applicable;
corrective action;
timeline to correct the nonconformity
person(s) responsible.

When documentary evidence exists that an identified nonconformity has been rectified, this should accompany the CAP. The laboratory should provide the CAP to WHO in an editable format such as Excel spreadsheet. The laboratory will have a maximum of two opportunities to supply the necessary information to address nonconformities in a timely manner, usually within 10 working days of the request. However, the nature of some nonconformities is such that an extended time period in which to rectify them may be needed. Justifiable requests for an extension of time in which to respond may be granted. If the laboratory fails to respond adequately to requests for submissions relating to nonconformities in the requested timeframe, the application for WHO laboratory listing may be terminated.

WHO review of corrective action plan

Before finalizing the audit, all nonconformities must have been appropriately addressed by the laboratory. Following WHO review of the initial CAP and associated documentation, one of the following may occur:

If the submissions are acceptable, WHO will notify the laboratory by letter that the audit and follow-up are complete. Notification of the laboratory listing will be provided by a separate letter.
If the submissions are not acceptable, the lead auditor will request an improved CAP and may ask for further evidence that must be presented no later than 10 days after the first review report on the CAP was sent to the laboratory. The laboratory will have a maximum of two opportunities to submit a CAP.

If the laboratory is unable to properly address the nonconformities, the application will/may be closed according to the severity and number of the nonconformities.

Performance Evaluation Laboratories

Why participate?

Who can apply for listing as a performance evaluation laboratory?

Requirements for WHO listing

Assessment process

Maintaining compliance

EVALUATING LABORATORIES LIST, SUBMISSION FORM, GUIDANCE DOCUMENTS & WHO PUBLIC INSPECTION REPORTS

List of WHO prequalification evaluating laboratories

Submission form

WHO guidance

International Organization for Standardization guidance

STAGE 1 AUDIT: REVIEW OF QUALITY MANAGEMENT SYSTEM DOCUMENTATION (QMS)

STAGE 2 AUDIT: ON-SITE AUDIT OF THE LABORATORY

WHO & LABORATORY RESPONSIBILITIES DURING ON-SITE AUDIT

Responsibilities of WHO

Responsiblities of laboratory

PRINCIPLES TO WHICH WHO AUDITS MUST ADHERE

Independence

Audit objectives and scope

Roles and responsibilities

Resources

Competence of the audit team

Consistency of procedures

Confidentiality and standard of conduct

Audit results and conclusions

Quality system

ORGANIZATION OF ON-SITE AUDIT