Chapter 2

How to define and adequately measure peri-operative patient safety (Haller)

Since the early development of anaesthesia, manipulating powerful drugs, thereby altering physiological functions, taking control of the circulation as well as the airway in unconscious patients have been recognised as intrinsically challenging to patient safety. As a consequence, the speciality has been at the forefront of many developments, particularly in-patient safety measurement, according to the rule: ‘One can only address what one can see’. However, before patient safety is reliably assessed, it first needs to be accurately defined.

Definition of patient safety

Many patient safety definitions exist but all include the presence of adverse outcomes/injuries stemming from the processes of anaesthesia care and are related to errors, or deviations from expected care. Cooper et al.⁵⁹ defined the concept of patient safety as ‘the avoidance, prevention, amelioration of adverse outcomes or injuries stemming from the processes of health care (i.e. anaesthesia care). Patient safety should address events that span the continuum from what may be called errors and deviations to accidents’. This concept can be modelled as a triangle integrating all these three dimensions (Fig. 2).

 

Several tools have been developed to measure adverse outcomes related to anaesthesia care and human errors and/or deviations. These tools can be divided into traditional safety measurement methods, developed by clinicians, and alternative methods, developed by IT specialists or quality assurance organisations.

Traditional patient safety measurement methods

Anaesthesia-related mortality

Anaesthesia-related mortality refers to any death occurring during or following the care from an anaesthetist. For analysis, cases are extracted from coroners’ registries, voluntary reports, surveys and malpractice reports/autopsies.^60–63 Information is usually forwarded to peer review committees of expert practitioners, usually senior anaesthetists. Experts then determine whether adverse outcomes are related to anaesthesia or not, and whether any errors have occurred.

The peer review process is largely based on implicit criteria: individual reviewers determine the standard of care, including their personal opinions regarding what should be defined as an error.

This has been a preferred method since the beginning of the speciality. Some examples include the National Confidential Enquiry into Peri-Operative Deaths in the United Kingdom, the survey of anaesthesia-related death in France, the study of deaths associated with anaesthesia in Taiwan, the review of anaesthesia-related mortality reporting in Australia and New Zealand, and the study of anaesthesia-related mortality in the United States.^6,64–66 In all these studies, reviewers assess whether adverse outcomes are related to anaesthesia or not. They identify which factors contributed to the death of the patient, including human errors and case mismanagement. The latest figures report a mortality rate related solely to anaesthesia of 2.96 per million population per year. Depending on countries and studies, the contribution of error and mismanagement to the number of patients dying as a consequence of anaesthesia has been found to vary between 77 and 97% of cases.^65,67

Although used over decades, mortality reviews have a number of limitations. The first is a lack of a standardised definition for anaesthesia-related mortality. In a number of studies, mortality includes only cases of intra-operative or immediate postoperative death to which human error of the anaesthesia provider has contributed, while for others anaesthesia-related mortality refers to all potential causes of deaths occurring during or following anaesthesia.^61,62,68 The second limitation relates to peer review as a method to assess the contribution of human error. There is often variability among peer reviewers’ opinion on ‘preventability’ of adverse outcomes and their level of agreement is sometimes only slightly better than chance.⁶⁹ The third limitation relates to the absence of a valid denominator. Because most studies use coroners’ registries, voluntary reports, surveys and malpractice claims, these data sources only include cases, while the denominator (number of patients undergoing anaesthesia care) is unknown. Approximations are often used, based on population registries or estimates of discharges from hospitals. Finally, death under or following anaesthesia is a rare event, and therefore patient safety cannot be determined from the sole analysis of anaesthesia-related mortality. Other types of adverse outcomes have to be used. The most popular is anaesthesia-related morbidity.

Anaesthesia-related morbidity

Anaesthesia-related morbidity includes any complication, excluding death, occurring during the peri-operative period.70 Complications include a whole range of undesirable outcomes, from postoperative nausea and vomiting (PONV) to dental injury, cardiac arrest and permanent neurological disability. Methods to assess safety of care from morbidity data are similar to those used for mortality studies. Data provided by voluntary reports, post anaesthetic follow-up programmes and malpractice reports are analysed by peer review committees which determine whether anaesthesia is implicated, and, if so, whether the anaesthesia-related morbidity is due to an accident or a preventable error.71–73 Cardiac arrest and coma are the most widely analysed adverse outcomes. The current prevalence of anaesthesia-related cardiac arrest in high-income countries is between 0.7 and 5.8/10 000 procedures and the prevalence of anaesthesia-related brain injuries ranges between 0.02 and 0.05/10 000.74–78 Less severe complications such as dental injuries or PONV occur more frequently in 0.2 to 1.3/1000 patients for dental injuries, and 10 to 45/100 patients for PONV.79–82 These figures show that the incidence of different morbidities varies significantly and conclusions regarding the level of patient safety differ accordingly. In addition, there is only a limited number of studies looking at anaesthesia-related morbidity which include both, the identification of anaesthesia-related injuries and a formal peer review assessment that can be considered as formally measuring patient safety.73,83,84 This is partly based on a lack of consensus in the literature as to what precisely defines anaesthesia-related morbidity. Some studies report events occurring mainly during surgery while others look at events occurring in the recovery room, or both.85,86 Some studies analyse any event while others consider mainly serious injuries.87,88 Finally, most large studies analysing anaesthesia-related morbidity take a larger perspective on peri-operative outcomes and do not include a formal assessment of the preventability of complications. These studies mainly focus on patient characteristics and associations with pre-existing risk factors.89–91 Despite these limitations, anaesthesia-related morbidity is a useful outcome with which to assess the safety of anaesthesia care.

Closed claim studies

While based on mortality and morbidity, closed claims studies represent a distinct category of patient safety measurement tools. This approach is on the basis of the standardised collection and analysis of anaesthetic complications in closed claim files from professional liability insurances or peer review committees from professional organisations. The most well known initiative is the closed claims project of the Committee on Professional Liability of the American Society of Anesthesiologists (ASA) in the United States.⁹² Initiated in 1984, this project explores, through a range of successive analyses, more than 4000 claims collected from 35 insurance organisations throughout the United States. Analysed complications include cardiac arrest, major respiratory system events, difficult intubations, nerve injuries, awareness, claims associated with chronic pain management, injuries associated with regional anaesthesia and events during obstetrical anaesthesia.^93–98 A closed claim file typically includes hospital and anaesthesia records, narrative statements of the healthcare personnel involved, chart summaries, experts’ opinions, outcome reports, cost of settlements and jury awards. At the start, a practicing anaesthetist reviews the files using a standardised data collection form to assess the cause of injury and appropriateness of care. A second, third and sometimes fourth practitioner is then involved in the file review process to ensure expert agreement on the level of preventability of the injuries.⁹⁹

This method is a comprehensive tool to measure patient safety in anaesthesia. It is highly specific to anaesthesia care. It also systematically includes injuries and a formal assessment of physicians’ liability.¹⁰⁰ However, despite its high specificity for patient safety issues, this method cannot be used to measure the true incidence of safety issues in anaesthesia. First, because not all injuries related to anaesthesia care are followed by claims for compensation from insurance companies. Second, because the overall number of anaesthetic procedures performed (denominator) is unknown, the true rate of injuries cannot be calculated. Third, this method is retrospective, and the validity of the performed analysis largely depends on the amount and quality of the available information. Despite these limitations, analysis of these cases can provide interesting information on the trends of the main injuries in anaesthesia care resulting in claims for compensation. For instance, when comparing the period between 1970 to 1989 and 1990 to 2007, analyses showed that oesophageal intubation had nearly disappeared (probably due to the more systematic use of end-tidal capnography) while in contrast inadequate oxygenation and ventilation have risen in nonoperating room locations, mainly due to the development of monitored anaesthesia care. Difficult intubation still represents 27% of adverse respiratory events reported in 1990 to 2007, increasing over the two time periods.¹⁰¹ This is why closed claim studies are still performed on a regular basis in many countries.^{100,102–105}

Incident reporting

Since the first structured questionnaires measuring critical incidents as part of the Australian Incident Monitoring Study, incident reporting systems have progressively evolved into large and worldwide initiatives of voluntarily reported incidents.^106–109 These are defined as ‘unintended or unexpected events which could have or did lead to harm’.¹¹⁰ As such the definition of incidents includes a wide range of events, from transfusion accidents or near misses, to drug reactions or nosocomial infections.

Staff members or risk managers analyse and classify reported incidents to assess causality and preventability. The analysis can sometimes take the format of an extensive inquiry including staff and patient interviews, which has been formalised and named root cause analysis (RCA).¹¹¹ Because this extensive process requires much time and resources, it is mainly used for severe incidents, often part of so-called sentinel events.^108,112 Other incidents are recorded, and crude analysis are performed to spot trends and identify rare or alarming events.

Although the incident reporting is relatively specific to patient safety it still has some limitations. Incident reporting is qualitative by nature,¹¹³ and qualitative methods emphasise description and interpretation rather than quantification.¹¹⁴ They cannot be used to assess the level of safety of anaesthetic practice since only voluntarily reported incidents are available for analysis. Underreporting is as common, and several studies suggest its magnitude as high as 77 to 94%.^115,116 Incident reporting is also prone to selection and hindsight bias. Clinicians can for instance select the type of incident they will record: these tend to be the most severe ones, those in accordance with individual perspectives of safety, or those likely to carry a message to the health organisation’s management.¹¹⁷ To overcome these limitations, reporting systems are increasingly becoming more standardised and are nowadays accessible through electronic solutions. Some systems use predefined categories of events or are integrated into the electronic patient records routinely used in operating theatres.¹¹⁸ This integration can improve accuracy and acceptance of the system with a very high level of reporting and with up to 85.1% of incidents documented,¹¹⁹ thereby reducing its limitation as a patient safety measurement method. Even so, incident reporting systems should not be used to measure patient safety or identify unsafe hospitals or unsafe professionals.

Adverse events

Adverse events refer specifically to iatrogenic injuries resulting from the process of care.¹²⁰ They comprise all types of injuries including those occurring during the peri-operative period. Adverse events analysis is the cornerstone of hospital safety assessment. It has been used in many well known studies on healthcare safety measurement such as the Harvard Medical Practice Study, the Quality in Australian Healthcare Study, a total of the Canadian Adverse Events Study, the Adverse Events in British Hospitals study, the ENEIS study and more recently similar studies in Switzerland, Sweden and Norway.^121–129 All except one study on adverse events have used the investigation approach.¹²¹ This method is based on a two-stage analysis process of medical records selected among hospitalised patients. During the first stage, records are selected using predefined screening criteria or trigger tools. These criteria usually involve the presence of identifiable events likely to be associated with adverse events and negligence, such as readmission to the operating theatre, hospital-acquired infection/sepsis or adverse drug reaction. In a second stage, expert reviewers examine the records that screen positive for one of these criteria and confirm the presence of an adverse event and its possible preventability. These studies, in which large samples of hospitalised patients are examined over a period of time, have provided the most comprehensive overview of hospital care-related adverse events and safety issues, including the field of anaesthesia. In the Quality in Australian Healthcare Study, a total of 16.6% of patients included in the study had an adverse event, with 2.2% of the total being anaesthesia related.¹²⁴ One third of these events were considered preventable.^122,123,130

There are also some limitations to the user adverse events for patient safety measurement: they represent mainly iatrogenic injuries associated with permanent or prolonged sequelae. This excludes a range of transitory complications from analysis, which could also be related to safety issues. The final decision as to whether an injury or complication is a real adverse event is left to the peer reviewer. The overall sensitivity of this measurement method largely relies on the quality and availability of information, and missing information can significantly bias the analysis. And finally, although this measurement method captures adverse events related to peri-operative care, it was not specifically designed to capture anaesthesia-related adverse events. In addition, only earlier studies published before 2000 have systematically assessed anaesthesia, and this limits the validity of this approach as a global measure of patient safety in anaesthesia.

Alternative methods of safety measurement

Clinical indicators

Developed initially in the manufacturing industry, indicators have been increasingly implemented in health care to be used as quantifiable surrogates of patient safety.¹³¹ A good example is an unplanned admission to the Intensive Care Unit within 24 h of a procedure with an anaesthetist in attendance. This indicator is considered as a valid measure of patient safety because it has been found to be associated with complication caused by anaesthesia and/or surgery in 87 to 92% of the patients. Furthermore, 74 to 92% of these complications can be considered as being preventable.^132–134 However, like other patient safety measurement methods, clinical indicators have limitations. First, there is a lack of consensus definitions for clinical indicators across healthcare systems and the same clinical indicator can be defined and used in different ways, depending on countries. For example, peri-operative anaesthesia-related mortality can be measured by three different indicators: death within 48 h of a procedure involving anaesthesia [Agency for Healthcare Research and Quality (AHRQ) patient safety indicators programme – United States], death rate associated with procedures involving anaesthesia (Veterans administration quality indicators programme – United States) or deaths within 30 days of surgery (NHS clinical indicators programme – United Kingdom).^134–136 Second, clinical indicators developed for anaesthesia care focus largely on complications but less on preventability and errors. Finally, there appears to be limited academic interest in clinical indicators. As a result, these measurement tools are more often viewed as ‘quality improvement tools’ and are disregarded as measures of patient safety in anaesthesia. Their validity is often limited and largely based on expert opinion. Even when process indicators which are based on sound scientific evidence are used, it still needs to be demonstrated that compliance with evidence-based best practice systematically results in better patient outcome.^137–139

Despite these limitations, clinical indicators offer promising perspectives as handy and useful metrics to quantify patient safety on a day-to-day basis. Indicators such as an unplanned admission to the ICU, surgical site infections, wrong side procedures, death within 30 days of coronary artery bypass surgery, and readmission to hospital following complications are some examples of straightforward and valid measures of patient safety since, beyond complications, they also exemplify events that are largely preventable through targeted evidence-based practices.¹⁴⁰

Information technologies

With the development of computing technologies and artificial intelligence, a number of new methods have become available. They are based on electronic tracking, and examples include recognition of nosocomial infections, hazards associated with drugs or wrong patient orders and medical device dysfunctions.¹⁴¹ To detect errors and complications, methods based on computing technologies use electronic algorithms to analyse hospital administrative databases, patient computerised records or monitoring devices.¹⁴² These algorithms are designed to flag specific adverse events-related codes, for example, iatrogenic pneumothorax, Methicillin-resistant Staphylococcus aureus (MRSA) nosocomial infections or drug antidote administration (e.g. Naloxone).^143,144 More advanced algorithms have been designed recently to detect out-of-range signals from monitoring systems, or wrong drug prescriptions. These systems are also able to support decision making on correction strategies to be put in place.^145–148 These computerised systems should not be confused with portable computerised data collection and automatic monitoring systems for physiological data monitoring (i.e. ECG, pulse oximetry, invasive BP) which are not direct patient safety measurement tools.

Information technologies have some limitations: they require advanced and fully implemented information systems at a hospital level, a technology that many hospitals cannot afford. In addition, they are only highly effective in detecting adverse events if these events generate computable and identifiable markers (e.g. a positive MRSA blood culture and absence of antibiotic administration). When adverse events are not documented or simply inappropriately coded, they cannot be captured by these technologies. Finally, no information regarding association with patient care and preventability is provided by these computerised systems. Therefore, further developments are still needed before this method can be used for routine measurement of patient safety in anaesthesia.

Safety culture questionnaires

Largely influenced by analyses of large-scale accidents such as the capsizing of the Herald of Free Enterprise in 1982 or the combustion of the Chernobyl nuclear power plant reactors in 1985, psychologists have developed organisational models of accident analysis.^149,150 All these models consider that accidents occur because of a succession of breeches in the organisation’s defence system. Consequently, safety questionnaires are designed to perform a global assessment of the defence system of an organisation. The best defence is considered to be the safety culture of the organisation itself. As a result, safety culture is assessed through formal questionnaires analysing the different dimensions of safety culture. These include levels of staffing, communication and interaction between management and staff members, and overall perception of the level of safety of the organisation. Some examples of organisational safety questionnaires in anaesthesia include the anaesthetists’ attitude to teamwork and safety questionnaire, the safety attitude questionnaire and the Hospital Survey on Patient Safety Culture.^33,151,152

Although promising, the model used to build these questionnaires is based on the assumption that management is at the top of the organisation, defines policies and provides resources that will largely influence work procedures and outcomes at the lower level of the organisation. Tasks and procedures are clearly defined, roles readily identifiable and co-ordination ensured by policies and written rules. Organisational culture is safe whenever there is a harmonious balance between policies, co-ordination of teams and environment. Whether this model really applies to healthcare organisations remains unclear. A dual line of control characterises healthcare organisations: professional (physicians and other allied health staff) and managerial (hospital administrative management). The influence of management decisions and policies on professionals, particularly physicians, is limited. There are few written rules to coordinate work among different professionals and co-ordination is largely ensured by skills and knowledge. The collaboration between surgeons and anaesthetists offers a good example of such a co-ordination.¹⁵³ This makes healthcare organisation a very complex and ambiguous environment and the validity of safety culture questionnaires based on the model of a normalised organisation is still to be demonstrated. Furthermore, evidence of the validity of safety climate questionnaires in all various hospital settings and cultural contexts is largely challenged.¹⁵⁴ These major limitations make the use of this method unsuitable for routine measurement of patient safety in anaesthesia.

Summary of findings and future directions

The current chapter provides an overview of current patient safety measurement tools in anaesthesia. Strengths, weaknesses and prevalence of injuries according to the chosen patient safety measurement method are summarised in Table 4. There is as yet no gold standard for patient safety measurement in anaesthesia. Various methods with different strengths and weaknesses co-exist. Analysis of safety levels of anaesthesia care will depend on the type of measurement method used. If anaesthesia-related mortality is used, it can be stated that anaesthesia has reached the ‘six sigma’ level of reliability (99.99966% free of defects) and is the safest speciality. If morbidity or incident figures are analysed, there is clearly a need for improvement, with a rate of undesirable outcomes ranging between 13.6 and 79%. Furthermore, most measurement methods rely on peer review opinion regarding preventability of recorded complications and unsafe events. This opinion can vary significantly from one reviewer to another. Even when complications that are related more specifically to patient safety issues, such as those defined in the critical incident method by Cooper et al.,³ are analysed, there still is a significant reliance on the interpretative process of reporters.⁶⁹

 

Despite these limitations, all the different patient safety measurement methods provide a comprehensive picture of anaesthesia-related adverse outcomes and errors. This is particularly true for studies analysing anaesthesia-related mortality and adverse events. These studies include all the different areas of the patient safety definition: adverse outcomes/injuries related to anaesthesia care and associated with errors or deviations. They also all use relatively similar methods for injury collection and analysis, allowing quantitative assessment and comparison over time. These methods can provide a broad overview of the level of safety in anaesthesia throughout different time periods and countries.

The described methods are factual metrics of clinical practice from a physician’s perspective. Patients’ perceptions are not incorporated into these tools. In a recent systematic review analysing critically ill patients, authors found that only 24% of the primary and 22% of the secondary outcomes used in the analysed reports included patient-centred outcomes.¹⁵⁵ These outcomes are defined as ‘reflecting how a patient feels, functions or survives’.^156,157 This is why several initiatives have focused on the development of patient-centred outcomes to be used in clinical trials. These include the EuroQol five D, the WHO Disability Assessment Schedule version 2.0 and the one life-impact measure (days alive and out of hospital at 30 days after surgery).¹⁵⁸

This approach has not yet translated into the area of patient safety measurement. While there are initiatives to include patients through interviews and conversations for safety monitoring in the area of care, these approaches are still largely physician centred (i.e. preventing drug errors, identity confusion).^159,160 Future developments in this area should focus on the development of measures of patients’ personal experience of care, looking at the way anaesthetists answer concerns regarding the overall anaesthetic care process and how they ensure continuity of care through drug reconciliation and postoperative follow-up, particularly when complications have occurred.¹⁶¹ This could be summarised as the ability to provide safe and professional care with empathy and respect for patients’ needs.

Please visit the following link to see this article in full on the EJA website