Four Steps to Conducting a Successful Data Integrity Audit
Auditing is a critical aspect of an overall data integrity assurance plan. To successfully audit the integrity of data, auditors must possess specific knowledge, insights, and skills.
A solid audit strategy and plan are critical to providing the highest degree of assurance that no data integrity breaches have occurred, short of 100% verification (which is unfeasible in an industry that generates large volumes of data).
But first, an auditor must understand what to look for before embarking on a data integrity audit. It is important to focus on high-risk areas within a dataset, since these provide the greatest opportunities to identify issues in the limited time available.
The following are four steps an auditor should follow to conduct a successful data integrity audit.
1. Preparing for the Audit
The target for data integrity audits is to find, should it exist, data or relevant metadata that may not be apparent to those involved in the final product disposition decision. This includes data that has been deleted, reprocessed, injected as test/trial samples, or resides outside of the review during the final batch disposition. Using a term borrowed from the technology sector, such data is referred to as “orphan data.”
A good strategy in preparing for the audit is to identify the intersection of “opportunity” and “motivation.” Opportunities that can allow for breaches in data integrity should be identified review of existing system controls and requirements designed to prevent and preclude such issues. Examples of these types of controls include access control, data management across the data lifecycle, management controls, and definition of file structure. The latter, plus file naming conventions, are central to data integrity and must be understood for each system prior to initiation of the audit.
Motivations to breach data integrity tenets come in a variety of forms. Broad motivations can result from the firm’s culture, e.g., pressure from management for fast results or competitive pressure from coworkers. It’s also not uncommon for the basic foundation of product manufacturing to lack vigor. For example, many data integrity issues have been found rooted in inadequate methods that often trace back to poor development and method transfer.
Identifying and recognizing these potential inadequacies allows auditors to target the audit on a high risk dataset. Good starting places for high risk targets are OOS and stability systems. Look for products with high rates of OOS, particularly those that go to phase II investigations and those which do not conclude in definitive lab error. Also, recognize highly restrictive stability specifications which may prove to be a challenge for the expiry period established. Once the opportunities have been identified, select a broad dataset over a predetermined timeframe, and subsequently, trace all data generated from the beginning to final reporting. If data integrity breaches exist, this strategy provides the highest probability of finding the breach.
2. Executing the Audit
Execution of the audit is the most important aspect of the process. Conduct the audit in the most professional manner possible but remember to facilitate a comfortable environment for those being audited. Otherwise, they may not feel comfortable providing full disclosure.
The first step after entering the department or area under audit is to identify the employee hierarchy. Although management should be involved in the audit, the majority of time should be spent with frontline employees. Look out for managers attempting to provide responses to certain inspectional questions on behalf of frontline employees. If this becomes an issue during an inspection, politely request that the employees in question be given appropriate time to answer the questions pertaining to their work responsibilities. By relying on management to provide critical data, the goal of obtaining the full story may not be achieved since managers may be several layers removed from the specific process being reviewed.
An auditor must receive complete information in the limited time provided so that a significant amount of data can be collected and multiple systems covered. One way to maximize the amount of data collected? By facilitating an environment of openness during the audit. Another strategy involves focusing on those processes which could most affect the quality of the product or results. This can be identified relatively quickly through reviewing process flowcharts, identifying critical process parameters, listing test methods, and/or examining SOPs. Additionally, the flexibility to change course is critical. It’s not unheard of during the initial walk-through for an audit team to get a feeling something is not quite right with a particular operation.
Summary reports are critical to identifying potential trends; however, they should not be solely relied upon to assess the acceptability of any given system. Still, it is critical to review the raw data whenever possible because this is where the most critical cGMP operations take place. For example, a review of a summary report for all OOS investigations should be used to identify trends and select individual investigations for further review. Once the individual investigations have been received, it is critical to follow the investigation to its source—such as a certain sample set within a specific HPLC system. It is important to understand the chain of events beginning with the running of a sample set to initiation of an OOS investigation, and, finally, the compilation of a summary report. If any piece of this chain is missing, it introduces a gap in the ability to draw conclusions regarding the department’s adherence to cGMP requirements.
When significant discrepancies are uncovered, do not criticize the responsible employees or managers, instead gather as much information as possible without placing judgment. If excessive workloads led to timesaving non-cGMP practices, empathize with the frontline employees. After all, this additional root cause information provides upper management not only with an outline of the problems, but also potential solutions.
A key component to an audit includes the collection of evidence when objectionable conditions are identified. Often, further investigations will be initiated by persons not directly involved in the audit. The quality of any resulting investigations, therefore, will often depend on the evidence collected when the issue was first uncovered, e.g., photographs, photocopies, printouts, or electronic copies on removable media. Evidence also includes information provided verbally by responsible employees. As in a court of law, information provided directly from responsible employees is considered more reliable than hearsay. Again, interviews should not be conducted in a manner that blames on any one person or group of persons responsible for certain objectionable actions or processes, as this limits the ability of the inspection team to gather relevant information and hinders subsequent investigations.
3. Concluding the Audit
When concluding the audit, document objectionable conditions found in data integrity controls, oversight and governance, and outcomes. As previously mentioned, when data integrity breaches are not found, it cannot be definitively concluded that such breaches do not exist due the fact that 100% verification is logically impossible. If conditions exist that show potential opportunities for data integrtiy breahes due to inadequate controls and/or inappropriate motivation, then these conditions should be identified based upon the facts observed. The existence of such conditions could also be a basis for extending or refocusing the audit to ensure the evaluation is adequate.
When it comes to orphan data, annotate all findings related to its discovery, keeping in mind that as the saying goes “absence of evidence is not evidence of absence.” If orphan data is found, indicate the type and conditions that allowed for the existence of such orphan data to occur such as “deleted data as a result of unrestricted authorizations to modify data,” or “data residing in electronic system, unreported and undiscovered do to lack of consistent file naming conventions and review of data in electronic system.”
4. Rectifying the Issues
The existence and impact of orphan data and impact thereof must be addressed to include the evaluation of such data for OOS results. Additionally, further action may include reporting through the Field Alert Reporting (FAR) system or Biological Process Deviation Report (BPDR) and/or recall assessment. Retrospective assessment of broader datasets may be appropriate when conditions are found that allow for additional orphan data; any new orphan data will need to be evaluated as described above.
Once the data assessment is complete, corrective and preventative action planning should commence to include consideration for interim controls. These may be necessary when the longer term technological solutions leave areas for substantial risk. Interim controls can include activities such as additional periodic review of datasets, “four” eyes principals for systems without adequate audit trails, or review of all reprocessing and/or manual integration when integration parameters are not adequately controlled by the system. CAPAs to remediate issues and improve the overall controls of the systems, interfaces, and oversight should be carefully planned. Verification of CAPAs is recommended to evaluate effectiveness post implementation. Mangers should also govern and monitor CAPA plans to ensure adequate progress and risk management.
A data integrity improvement map is also a useful tool to illustrate the progression of improvements and controls over the course of time. Such a tool is helpful to ensure that the full landscape is considered and carefully managed, and can include the following information for different points on the timeline: description of controls, limitations, immediate mitigations, interim controls, planned next steps, and timing.
While data integrity auditing is not easy, it is essential in the industry’s regulated environment. Every decision in the GxP space rests on the reliability of data generated. Auditing of this high risk asset is critical to business viability, continuity, and competitive advantage.
About the Author
Crystal Mersh is an Executive Partner with Quality Executive Partners, a firm which specializes in compliance consulting for FDA-regulated industries.