The Risk Evaluation Method

Aseptic processing of sterile drug products can and should be improved. The same challenges, problems and issues seem to appear, reappear, or never really disappear from year to year. These problems persist despite more awareness of the issues due to increased training, conference sessions on the topic, guidance documents, quality system management approaches and metrics. Each year, regulatory audit observations, 483s, and warning letters continue to cite the same problems and issues over and over. Admittedly, aseptic processing is challenging, and there are obstacles to improvement, but it is the job of those working in this area to resolve these challenges.

Numerous documents stress the need for science and risk-based decision making in aseptic process design and performance. These include PDA Technical Report No. 22 (1), PDA Technical Report No. 44 (2), PDA Points to Consider for Aseptic Processing (Parts 1 and 2) (3–4), as well as the U.S. FDA aseptic processing guidance and the planned revision to EU’s Annex 1. As stated in Part I of the PDA Points to Consider for Aseptic Processing, decisions based on product risk can, and should, be used to improve aseptic process. Risk assessments should provide the information needed to make informed decisions throughout the process lifecycle. The objective is not merely to identify risk but to improve the process. Yet, the industry still struggles with developing effective risk assessment methods for aseptic processing.

Aseptic process risk assessments have some unique challenges. The severity of sterility-related issues, primarily the loss of sterility, is high and certainly worthy of consideration. Yet, the causes of aseptic process failures, which are directly correlated with the loss of sterility, are relatively rare; hence, very low occurrence rates are noted. Detection of process failures is not always reliable, and the correlation between what we can detect and the desired or undesired outcome may not be quantifiable. An issue with many aseptic processing risk assessments is the lack of objectivity. Since it is not easy to measure or quantify people or environmental risk factors, risk assessments tend to be subjective and therefore of limited benefit.

This subjectivity comes from the lack of data that can be correlated to sterility or aseptic process failure. For example, interventions can add microbiological contamination to the environment. However, we may not know when, or to what extent, those interventions can result in a level of contamination necessary to contaminate the product. In common practice, boundaries and limits are set, then the results are observed. Too often this leads to approaches based largely on the assessor’s experience and bias rather than science. An objective method for considering the correlation and evaluation of risk to aseptic processes is needed to facilitate process improvement.

Objective Defines the Method

The Risk Evaluation Method or REM described in this article was initially developed to satisfy a need for an objective, simple, and more accurate method to rank aseptic process interventions, in order to better plan media fills (5). This first article will present the basic objective and steps for the REM. Subsequent articles will explore its specific use for aseptic process improvement.

For the aseptic processing REM to be effective and useful, it should be:

Objective: Objectivity reduces the potential for bias. Bias leads to disagreement, loss of confidence in the method, and risk assessments without benefit. Objectivity is achieved if different people with similar knowledge about the process and access to information are able to use the REM and conclude a similar risk rating. For example, if the Quality lead and the line mechanic use the REM, they should obtain similar rankings. For this to happen, meaningful, measurable data or information must be available and accessible. It may be that different people have different perspectives and varying levels of knowledge. This knowledge becomes valuable input into setting the ranking criteria. Once the criteria are set, the evaluation of information should be objective.

Simple: Simplicity allows for the use of the REM by a larger group. This leads to better acceptance of the REM outcome. The REM should be performed by the people engaged in the process itself. The REM should not be burdensome to use. The people using the method should be able to explain the basis of the method and be able to perform the assessment.

Robust: A robust method is one that is applicable to most relevant applications or processes. It also avoids the need for multiple or changing methods which may add confusion, bias and complexity.

Logical: The REM must be based on sound science in order to be viewed as truly objective, to gain user acceptance of results and convince regulators that the method and its results are credible.

The REM relies on two principles to achieve effectiveness. It uses a Line of Sight approach. Line of Sight focuses and links actions to the overall objective of the assessment. That objective is defined and stated in a problem statement. The use of a problem statement linked to the objective assures that all actions are directly beneficial and useful to accomplishing the objective and that actions of little value are omitted.

For this to work, the definition of the objective as reflected in the problem statement must be well thought out. For example, is the objective of the environmental monitoring program to find particles, achieve a particle free environment, meet regulatory requirements, or achieve an environment suitable for aseptic processing? Is the objective of the aseptic process simulation to qualify the operators, the interventions or the aseptic process? As the objective varies, so will the focus and use of the assessment.

The second principle is the use of a Key Word approach, a tactic in which those assessing process risk identify the words and values most relevant and meaningful to them. Rather than use general terms or risk tables, the participants decide which words best describe or assess their situation and tie these to risk scales.

Aseptic processes may seem similar throughout the industry, but companies have unique cultures and individuals have different backgrounds, biases, perspectives, and experience. Trying to match one set of criteria to all aseptic processes would prove difficult. The Key Word approach applies the company’s and the involved individuals’ knowledge to the problem at hand. Also, by asking the involved individuals to select the risk criteria, it better assures ownership, respect and use of the method. It reduces the chance of the model becoming a checklist exercise.

Because the Key Word approach encourages individual assertion of criteria and factors, one company/process may not be completely applicable to other companies/processes. The temptation to develop a one-size-fits-all template should be resisted, as it is the process of developing the factors and criteria for the assessment which is most valuable in ensuring its effectiveness.

While not necessarily an exhaustive list, those factors should fit the four basic requirements of the REM—objectivity, simplicity, robustness and logic. All of the elements should be objectively measured. No one factor alone fully defines risk. It is the combination of the factors that presents risk.

Taking these principles into consideration, the REM comprises eight steps:

• Step 1: Problem Statement. The team develops a problem statement that defines the objective and boundaries of the REM. This helps the team stay focused. The objective should be meaningful to those performing and analyzing the assessment and should not be broad beyond reasonable usefulness, yet it should not be too narrow as to be limiting.
• Step 2: Team Selection. The REM team should consist of subject matter experts in the field defined by the problem statement. They should represent multiple departments and levels of authority. The team does not have to include all affected stakeholders. In this phase, the objective is to define risk elements and criteria, rather than inform stakeholders. Team size is important. While it should include all relevant stakeholders, large groups can stifle participation. It should include diverse participation, thus reducing the risk of bias or preconceived outcome. A facilitator may be of benefit.
• Step 3: Risk Factors Determination. The team identifies the parameters, actions, events, conditions or items that affect the objective or problem. These should have measurable criteria. The data for those criteria should be accessible and understandable by anyone doing the assessment. Brainstorming may be used to select risk elements.
• Step 4: Criteria Setting. The criteria, limits or ranges used to rank the parameters or elements should be set by the team prior to the assessment. The criteria should be meaningful, logical, attainable, useful, verifiable and measurable. The data should be accessible. Rules should be set for accepting levels of risk. For instance, are high levels of risk acceptable or must steps be taken to reduce those risks? Are less-than-low levels of risk accepted without attempts to further reduce them?
• Step 5: Assessment Tool Development. The team should choose a tool that evaluates the parameters and meets the objective, both from a product quality and process performance perspective. One-size-fit-all tools may not work for all applications. Customizing a tool or model for a particular use may be useful.
• Step 6: Risk Evaluation. The team performs the assessment and obtains the risk rankings. The meeting in which the evaluation is performed should have a duration target to help keep things on track. All should have a voice at the meeting.
• Step 7: Mitigation Actions. The team or additional stakeholders should determine, evaluate and implement actions and steps based on the ranking in an effort to improve the process and/or meet the objective, considering the rules set out in Step 4. Remember that the purpose of the evaluation is to improve the process, not merely identify risk. Where actions or steps are to be taken, consideration should be given to the addition of risk from any unintended consequences of the action or change.
• Step 8: Review. Follow up on actions to confirm that they a) were implemented properly, b) were effective, and c) did not add any unintended consequences or additional risk. Communicate results to affected departments. Perform periodic review of the REM to confirm accuracy and effectiveness of results.

Now, that a case for aseptic process improvement has been made and the REM introduced, Parts II and III of this article series will show how the method is used to improve aseptic process interventions and cleanroom environmental monitoring through examples and case studies.

References

1. Agalloco, J. et al. PDA Technical Report No. 22: Process Simulation of Aseptically Filled Products. Bethesda: PDA, 2011.
2. Ahmed, R., et al. PDA Technical Report No. 44: Risk Management for Aseptic Processes. Bethesda: PDA, 2008.
3. PDA Points to Consider for Aseptic Processing: Part 1. Bethesda: PDA, 2015
4. PDA Points to Consider for Aseptic Processing: Part 2. Bethesda: PDA, 2016
5. Baseman, H., and Long, M. “Risk Management of Microbial Contamination Control in Aseptic Processing and Interventions Risk Assessment Model (IREM): The Use of Critical Thinking to Make Informed Decisions.” In Contamination Control in Healthcare Product Manufacturing, Vol. 3, eds. Russell Madsen and Jeanne Moldenhauer, 341-404. Bethesda: PDA/DHI, 2014.