Journal Club by CanadiEM - E04 Approach to Systematic Reviews and Meta Analyses

CanadiEM Journal Club E04 Systematic reviews and meta analyses show notes

Welcome back to Journal Club by CanadiEM! In this episode we go over an approach to systematic reviews and meta analyses based on Oxford centre of EBM, and learn about diagnosing pneumothorax with ultrasound vs X-ray

Using the Oxford centre of EBM tool, we will ask:

1. What question(s) did the systematic review address?
2. Is it likely that important, relevant studies were missed?
3. Were the criteria used to select articles for inclusion appropriate?
4. Were the included studies sufficiently valid for the type of question asked?
5. Were the results similar from study to study?
6. What were the results?
7. What is the clinical significance of the results?

and then a clinical pearl on pneumothorax!!

Hosts: 

• Dakoda Herman
• Jayneel Limbachia
• Jake Domm

Paper: “Chest ultrasonography versus supine chest radiography for diagnosis of pneumothorax in trauma patients in the emergency department” Cochrane Database of Systematic Reviews by Chan KK, Joo DA, McRae AD, Takwoingi Y, Premji ZA, Lang E, Wakai A

What question(s) did the systematic review address?

P: Trauma patients in the ER

I: chest ultrasonography by non rad physicians

C: Chest xray

O: diagnosis of pneumothorax, improved patient safety 

• Secondary: investigate potential sources of hetero such as type of CUS operator, type of trauma, type of US probe on test accuracy

T: inception to 10 April 2020

Is it unlikely that important, relevant studies were missed?

• This study included prospective, paired comparative  accuracy studies in which patients were suspected of having pneumothorax. Patients must have undergone both CUS by a frontline non-radiologist and CXR, as well as CT of the chest or tube thoracostomy as the reference standard.
• The authors carried out systematic searches in the following electronic databases: Cochrane Database of Systematic Reviews; Cochrane Central Register of Controlled Trials; MEDLINE; Embase; Cumulative Index to Nursing and Allied Health Literature (CINAHL) Plus; Database of Abstracts of Reviews of EIects; Web of Science core collection (which includes: Science Citation Index Expanded; Social Sciences Citation Index; Arts & Humanities Citation Index; Conference Proceedings Citation Index - Science; Conference Proceedings Citation Index - Social Sciences & Humanities; and Emerging Sources Citation Index; and Clinicaltrials.gov from database creation to April 2020.
• The authors also handsearched reference lists of included articles and reviews, retrieved via electronic searching, for potentially eligible studies. Additionally, they carried out forward citation searching of relevant articles in Google Scholar and looked at the “Related articles” on PubMed.
• They did not limit the search to Englsih language only and included articles published in all languages.
• Their search strategy in volved the use of MeSH terms such as Pneumothorax, Radiography, Ultrasonography, and focused assessment with ultrasonography for trauma. They also used many text words.
• Using this search strategy 3473 records were identified. 1180 duplicated records were removed, leaving 2293 records to be screened. These records were screened by two of the authors for their relevance, when there was a discrepancy a third author decided whether to include the record or not. 2268 records were excluded, leaving 25 full-text articles that were assessed for eligibility. 12 studies were excluded - 5 missing CUS/CXR/CT chest/chest tube, 4 CUS not performed by frontline non-radiologist physicians, 2 wrong patient population, 1 wrong study design. A total of 13 studies were included in qualitative and quantitative analysis. 9 studies using patients as units of analysis included in primary analysis. 4 studies using lung fields as units of analysis included in secondary analysis.
• Authors provide a nice figure depicting this.
• Authors did not contact experts for unpublished data but were very thorough and transparent in their search strategy. I think it is unlikely that important, relevant studies were missed.

Were the criteria used to select articles for inclusion appropriate?

• The authors of this study clearly outlined their study inclusion and exclusion criteria.
• They included prospective, comparative accuracy studies in which patients were suspected of having pneumothorax. 
• They included trauma patients in the emergency department setting. Patients must have undergone both CUS by frontline non-radiologist physicians and CXR as index tests, as well as CT of the chest or tube thoracostomy as the reference standard. The two main index tests were CUS completed by a frontline nonradiologist physician and CXR, both being performed in the supine position. If data on specific CUS findings (such as the absence of lung sliding, absence of B-lines or comet-tail artefact, presence of lung point, and absence of lung pulse) were available, they planned to assess the diagnostic accuracy of these individual CUS findings. The target condition was traumatic pneumothorax of any severity. They defined a pneumothorax identified on CT scan of the chest or via clinical findings of a rush of air or bubbling in a chest drain after tube thoracostomy as the reference standard.
• The authors excluded studies involving participants with already diagnosed pneumothorax (i.e. case-control studies); studies involving participants with nontraumatic pneumothorax; studies involving participants who had already been treated with tube thoracostomy; and studies in which a frontline non-radiologist physician did not perform CUS.
• These criteria were appropriate for inclusion. However, many studies did differ in their units of measurement - lung fields vs. patients, which could introduce significant bias into the results of individual studies.

Were the included studies sufficiently valid for the type of question asked?

• The authors used the QUADAS-2 tool to assess risk of bias and the applicability of each included study. 
• This tool assesses risk of bias in four domains: patient selection; index tests; reference standard; and flow and timing. In addition, they examined concerns about applicability in the first three domains. They tailored the tool to their review question - one of the signalling questions in the patient selection domain was not applicable because they excluded case-control studies; therefore they deleted this question from the tool. 
• Two review authors performed the assessments independently. They discussed and resolved any disagreements that arose through consultation with a third review author.
• They included a figure describing their assessments of study quality.
• Of the nine studies that we included in the primary analysis:
• • One had a low risk of bias, two had an unclear risk, and six had high risk of bias in the patient selection domain, mostly due to convenience sampling or inappropriate exclusion criteria, such as excluding haemodynamically unstable patients, lack of access to CUS, chest wall injuries precluding CUS, or if CT was not indicated. 
  • The risk of bias in the interpretation of CUS results was low in five studies, unclear in two studies, and high in two studies; this was related to unclear blinding methodology of outcome assessors interpreting CUS and CXR results. 
  • The risk of bias in the interpretation of CXR results was low in two studies, unclear in six studies, and high in one study; this was largely due to unclear blinding methodology of outcome assessors interpreting CXR and CT results, as in some studies it was not clear whether radiologists had access to both imaging results or not. 
  • The risk of bias introduced in interpretation of the reference standard results was low in three studies but unclear in the remainder for similar concerns regarding blinding methodology. 
  • The risk of bias in the flow and timing domain was low in two studies, unclear in four studies, and high in three studies; this was due to the exclusion of patients based on missing CT data or unclear/inappropriate time intervals between CUS, CXR, and CT.
  • They judged applicability concerns regarding patient selection as low for six studies but high for three studies; this was due to the exclusion of haemodynamically unstable patients or lack of access to CUS despite the study focusing on comparing CUS. 
  • They judged one included study to have unclear concern regarding applicability of the reference standard used as there was insuIicient reporting of the method of assessment. 
  • They deemed all other domains for applicability concerns as low risk for all studies.
• Studies that used lung fields as their unit of analysis had several limitations including missing CUS data for some lung fields and using two CUS tests (one for each lung field) compared to one CXR (for both lungs) on the same patient. Inherently, there would be an inability to blind the CUS operator during collection of CUS data while performing the two CUS tests (one on each side of the patient), as well as during the interpretation of the CXR and CT results between the two lung fields. By analysing lung fields separately, it is diIicult to ascertain whether patient characteristics, past medical history, or traumatic injury pattern could have affected one or both lungs and may have confounded the diagnostic accuracy. Out of the four studies included in the secondary analysis: 
• • The risk of bias in the patient selection domain was low in one study, unclear in two studies, and high in one study; this was due to inappropriate exclusion criteria, such as excluding haemodynamically unstable patients or chest wall injuries precluding CUS, or due to unclear sampling technique.
  • The risk of bias introduced in interpreting CUS results was low in two studies and unclear in two studies due to lack of clarity about blinding of the outcome assessors interpreting CUS and CXR results. 
  • The risk of bias introduced in interpreting CXR and CT results to be low in one study, unclear in two studies, and high in one study; this was again due to definite unblinded interpretation of test results or unclear blinding methodology of outcome assessors interpreting CXR and CT results.
  • The risk of bias in the flow and timing domain was low in two studies and high in two studies due to missing patient data.
  • They judged applicability concerns in the patient selection domain as low for two studies, unclear for one study, and high for one study, due to unclear patient selection methods and exclusion of haemodynamically unstable patients or chest wall injuries precluding CUS. 
  • They judged one study to have unclear concern regarding applicability of the reference standard, as blinding of the outcome assessor interpreting the results of CUS, CXR, and CT was unclear. We deemed all other domains as 'low concern' for all studies.
• The authors were thorough in their assessment of each study's quality and discussed how this may have impacted the results. The most common area of bias seemed to be in patient selection where 11/13 studies were judged to be at unclear or high risk of bias.

Were the results similar from study to study?

• Substantial heterogeneity in sensitivity analysis of supine CXR
• • Based on Figure 4, Forest plot: 
  • • Sensitivity ranged from 0.09 to 0.75
    • Wide and non overlapping confidence intervals are suggestive of high variability between studies 
    • Limits the evidence 
  • But they do not list the reasons for why such heterogeneity exists
  • • Some possibilities for the heterogeneity include the high or unclear risk of bias of included studies 
    • There is no I squared statistic shown - a statistical measure that confirms the presence of significant heterogeneity in a meta-analysis

Results of the study: 

• Included 13 studies, 1271 trauma patients with 410 who had a pneumothorax
• • 9 studies used patients as unit of analysis
  • 4 studies used lung field as unit of analysis
  • Most studies were high or unclear risk of bias - 11/13 
• CUS sensitivity and specificity: 0.91 (95% CI: 0.85 - 0.94) and 0.99 (95% CI: 0.97 - 1.00)
• CXR sensitivity and specificity: 0.47 (95% CI: 0.31 - 0.63) and 1.00 (95% CI: 0.97 - 1.00) 
• • Difference in sensitivity: 0.44 (95% CI: 0.27 - 0.61) 

Practise Changing?

This is an excellently executed Systematic Review that makes the most of the limited evidence available and presents such in a transparent way. The results of this study suggest that CUS has better sensitivity in detecting pneumothorax in the emergency department than CXR, and comparable specificity. This publication adds to the body of research that is building in support of US in the primary care setting, and promotes a change in culture as US adoption rates grow. 

Clinical pearl: 

Pneumothorax, or, air in the pleural space, is a fairly common complication of thoracic traumas, occurring 15-50% of the time.

• History: If you’re able to get a history, and often you won’t be able to, you’ll hear about a Sudden, severe, chest pain, sometimes referred to shoulder and maybe associated pleuritic pain or SOB.
• On Exam: Exam findings range from nothing to respiratory distress and shock. May have decreased chest wall motion, subcutaneous emphysema, or poprock skin, hyperresonance, or decreased or absent breath sounds on the affected side.
• Imaging: traditionally via x-ray or US depending on physician.
• • On xray: Absence of lung markings beyond the edge of the lung. The edge of the scapula or upper ribs are often mistaken for a pneumothorax.
  • On US: Scan over the most upright part of the patient's chest. if supine go right over the top of their chest (third or fourth intercostal space in the midclavicular line and is repeated on both sides) and if upright scan the apices. Looking for lung sliding and the classic “seashore sign” in M-mode in normal lungs, or absence of lung sliding and “barcode sign” seen in pneumothorax. There are tons of good videos online to take a look at. 
  • CT is gold standard, but rarely necessary

- Rush of air on thoracostomy is also diagnostic.

- Treatment: varies based on severity, from no treatment if patient tolerating well, to drains and chest tubes, to needle or finger thoracostomy for immediate decompression if pneumothorax clinically indicated and hypotension.