Article Text
PDF
Abstract
Objective To determine whether updating a diagnostic prediction model by adding a combination assay (tumour necrosis factor-related apoptosis-inducing ligand, interferon γ induced protein-10 and C reactive protein (CRP)) can accurately identify children with pneumonia or other serious bacterial infections (SBIs).
Design Observational double-blind diagnostic study.
Setting Two hospitals in Israel and four hospitals in the Netherlands.
Patients 591 children, aged 1–60 months, presenting with lower respiratory tract infections or fever without source. 96 of them had SBIs. The original Feverkidstool, a polytomous logistic regression model including clinical variables and CRP, was recalibrated and thereafter updated by using the assay.
Main outcome measures Pneumonia, other SBIs or no SBI.
Results The recalibrated original Feverkidstool discriminated well between SBIs and viral infections, with a c-statistic for pneumonia of 0.84 (95% CI 0.77 to 0.92) and 0.82 (95% CI 0.77 to 0.86) for other SBIs. The discriminatory ability increased when CRP was replaced by the combination assay; c-statistic for pneumonia increased to 0.89 (95% CI 0.82 to 0.96) and for other SBIs to 0.91 (95% CI 0.87 to 0.94). This updated Feverkidstool improved diagnosis of SBIs mainly in children with low–moderate risk estimates of SBIs.
Conclusion We improved the diagnostic accuracy of the Feverkidstool by replacing CRP with a combination assay to predict pneumonia or other SBIs in febrile children. The updated Feverkidstool has the largest potential to rule out bacterial infections and thus to decrease unnecessary antibiotic prescription in children with low-to-moderate predicted risk of SBIs.
- infectious diseases
- accident & emergency
- epidemiology
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
What is known about the subject?
Identifying serious bacterial infections (SBIs) is a major diagnostic challenge.
A combination assay (tumour necrosis factor-related apoptosis-inducing ligand, interferon γ induced protein-10 and C reactive protein) can differentiate accurately between viral and bacterial infections.
Clinical prediction models that also include biomarkers, such as the Feverkidstool, are helpful in predicting SBIs.
What this study adds?
The diagnostic accuracy of the Feverkidstool increases when updated with the combination assay to predict SBIs.
The Feverkidstool with the combination assay benefits children with low–moderate risk estimates for pneumonia or other SBIs in particular.
Combining a clinical risk profile (like the Feverkidstool) with a new biomarker contributes to positioning that biomarker in the diagnostic workup.
Introduction
Suspicion of infectious disease is one of the most common causes of paediatric emergency department (ED) visits.1 The proportion of bacterial infections in children with fever without source (FWS) and acute respiratory tract infections (RTIs) is however low (respectively 0.02%–13% and 26%–28%).2–4 Next, identifying patients benefiting from antibiotic treatment remains a major diagnostic challenge. Consequentially, children with acute RTI receive antibiotics almost twice as often as the estimated prevalence.2 5 6 Antibiotic overuse is associated with increased antibiotic resistance, causing 25 000 deaths in Europe annually.7 8 This underlines the need to better differentiate between viral and bacterial infections. Therefore, several prediction models have been developed.9 10 The Feverkidstool, a clinical prediction model including both clinical parameters and C reactive protein (CRP), is a validated tool for supporting clinical decision-making on, for example, whether or not to start antibiotics.11–13 However, further improvement of this diagnostic tool is warranted as it does not provide an accurate diagnosis for all patients.
We recently showed that a novel blood assay, combining concentrations of CRP with tumour necrosis factor-related apoptosis-inducing ligand (TRAIL) and interferon γ induced protein-10 (IP-10), could diagnose bacterial infections more accurately than CRP alone.14–16
The dynamics of TRAIL are complementary to traditionally studied bacteria-induced proteins; TRAIL concentrations decrease in bacterial infection and increase in viral infections.15 The aim of this study is to investigate whether updating the Feverkidstool by replacing CRP with the combination assay can improve the diagnosis of serious bacterial infections (SBIs) in preschool children.
Patients and methods
The current study builds on the prospective observational OPPORTUNITY Study performed in four hospitals in the Netherlands and two hospitals in Israel between 16 October 2013 and 28 January 2015. For detailed methods, we refer to the original publication.14 In short, this study included clinical data, a host-protein-based assay, nasal swab PCR and 28-day follow-up data from children aged 1–60 months with lower RTI or FWS (n=777). The study was an observational double-blind diagnostic study. The analysis of the serum samples for the assay (index test) was done in the absence of any clinical or other patient-related information, and the expert panel (reference standard) was blinded to the decisions of their peers and to the results of the index test.
General inclusion and exclusion criteria can be found in the original publication that had 577 children in the primary analysis. We added to our current study data of children aged 1–2 months that in the original OPPORTUNITY Study were included for a subanalysis (n=28), as this study showed accurate results of the assay in this population. Children admitted to the intensive care (all referred from the wards of other hospitals, n=14) were excluded as the Feverkidstool was developed for febrile patients presenting at the ED (online supplementary figure 1). Parents gave written informed consent prior to sampling. This study was designed and analysed without patient involvement. Patients were not invited to contribute to the writing or editing of this document for readability or accuracy. This manuscript follows the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis)(online supplementary file 3) reporting guidelines.
Supplemental material
Supplemental material
Prediction model
The Feverkidstool is a polytomous prediction model that predicts the risk of pneumonia or other SBIs, based on the following variables: age, body temperature, heart rate, respiratory rate, oxygen saturation, ill-appearance, peripheral capillary refill, chest wall retractions and CRP (definitions presented in the statistical analysis plan, online supplementary file 1).12
Supplemental material
Combined host-protein-based assay
The assay is currently ELISA-based (a point-of-care test is being developed) and combines the concentrations of TRAIL, IP-10 and CRP using a predetermined logistic regression formula to compute the likelihood score (scale 0–100) for a bacterial infection.14 15 The assay was performed on coded serum samples in the absence of any patient-related information.
Reference standard diagnosis
Currently, no single reference standard test exists for determining the aetiology of an infection.17 Therefore, in the OPPORTUNITY Study, England’s National Health Service standard for evaluating diagnostic tests was followed and an expert panel reference standard was composed.14 18 Every recruited patient was diagnosed by three panel members affiliated to the country of recruitment using all available electronic Case Record Form (eCRF) information (clinical and laboratory information, including a 28-day follow-up), but blinded to the assay and Feverkidstool results and to the labels of their peers. Each expert assigned one of the following aetiologies to each patient: bacterial infection, viral infection, mixed infection (ie, bacterial and viral co-infection), non-infectious disease or indeterminate. Patients assigned as mixed infection were later classified as bacterial because they are clinically managed similarly. Patients with a bacterial reference standard diagnosis were divided into pneumonia or other SBIs (eg, meningitis, urinary tract infections, bacteraemia) based on the diagnosis at hospital discharge assigned by the attending physician.
Statistical analysis
General approach
We compared the diagnostic accuracy to predict SBIs of the original Feverkidstool with the accuracy of the Feverkidstool updated with the assay (hereafter called updated Feverkidstool). Statistical analyses were performed in SPSS V.21.0 for Windows and R V.3.2.2. We used the Mann-Whitney U test for comparison of continuous variables. Categorical outcomes were analysed using the χ2 test or Fisher’s exact test where expected cell counts were less than 5.
Model development and performance
First, we recalibrated the original Feverkidstool on our data using two separate logistic regressions, one for pneumonia and one for other SBIs (recalibrated Feverkidstool: logit (pneumonia or other SBI)= β0 + β1 (linear predictor Feverkidstool)). Then, we updated the Feverkidstool by adding the combination assay (updated Feverkidstool: logit (pneumonia or other SBI) = β0 + β2a (linear predictor Feverkidstool) + β2b (score assay)). One element of the assay, CRP, was also a predictor in the linear predictor Feverkidstool. To be able to compare the effect of using the assay (updated Feverkidstool) instead of the CRP only (original Feverkidstool), we standardised the effect of the CRP coefficient (as part of the linear predictor Feverkidstool) in this updated Feverkidstool. Therefore the linear predictor Feverkidstool was based on the median CRP value for all participants. The actual CRP value of patients was used to compute the score assay. Detailed models are presented in the statistical analysis plan (online supplementary file 1). The discriminative ability of the recalibrated original and updated models was expressed using pairwise c-statistics.19
Predicted risk thresholds
To help interpret the potential benefit of the different models along the range of predicted probabilities, a decision curve analysis (DCA) was performed. The DCA assesses the relative harm of false positives and false negatives for different probability thresholds if the models were used to guide antibiotic prescription.20 For predefined risk thresholds, we calculated sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios (LR+, LR−) for the original and updated models.
Reclassification
To assess the potential added value of the updated Feverkidstool in correctly classifying SBIs and viral infections using defined thresholds, we did a head-to-head comparison for the updated Feverkidstool and the recalibrated original Feverkidstool in a reclassification table.21
Missing values
Multiple imputation techniques enabled analysing all available data. Missing values in the variables, needed for the Feverkidstool, were imputed 10 times using the multivariate imputation by chained equations (MICE) algorithm in R statistical software.22 Variables used in the imputation model are presented in the statistical analyses plan (online supplementary file 1). Analyses were performed separately in the 10 imputed data sets and combined using Rubin’s rules.23 The Feverkidstool variable peripheral capillary refill was not recorded in the OPPORTUNITY Study for any patient. Because we had no information on the distribution of values for capillary refill, we could not include this variable in the model for multiple imputation. Leaving the coefficient for capillary refill out of the Feverkidstool would have led to systematic lower predictions (biassed calibration), so we replaced this systematic missing variable by the mean prevalence of prolonged capillary refill in the initial Feverkidstool derivation cohort (=0.039, mean imputation).24 Inconclusive diagnoses (n=71) were also imputed; a sensitivity analysis was performed leaving out patients with an inconclusive diagnosis.
Results
Population characteristics
A total of 591 patients was available for analysis: 30 pneumonia, 66 other SBIs and 495 viral infections (online supplementary figure 1). Children with pneumonia were older than children with other SBIs or with viral infections (median age of 24.5 vs 15 months) and children with pneumonia or other SBIs were hospitalised more often (73% and 77%) than children with viral infections (52%, table 1). Children with an inconclusive reference standard diagnosis differed from children with a conclusive diagnosis on age, biomarker values and antibiotic prescription (online supplementary table 1).
Characteristics of patients included in the primary analysis
Model performance
The recalibrated original Feverkidstool discriminated well between pneumonia and other infections (c-statistic 0.84, 95% CI 0.77 to 0.92), and between other SBIs and other infections (c-statistic 0.82, 95% CI 0.77 to 0.86, online supplementary figure 2). This performance is similar to previous Feverkidstool validations.11 12 Updating the Feverkidstool with the assay improved discrimination between bacterial and other infections, reflected by an improved c-statistic for pneumonia to 0.89 (95% CI 0.82 to 0.96), and for other SBIs to 0.91 (95% CI 0.87 to 0.94) (online supplementary figure 2). A sensitivity analysis of the cohort without imputed reference standard diagnoses showed similar results, with improved prediction of pneumonia and other SBIs (online supplementary table 2).
Predicted risk thresholds
The DCA shows the net benefit of starting antibiotics using predictions of the updated instead of the original Feverkidstool, depending on the choice of probability threshold. Children with low–moderate predicted risks ≤40% for pneumonia or other SBIs had the most benefit from the updated Feverkidstool (figure 1). For example, if all children with a predicted risk of >20% would be treated with antibiotics, the net benefit of the updated Feverkidstool increases with 0.2 from the 0.2 net benefit in the original feverkidstool for predicting pneumonia to 0.4 net benefit. For predicting other SBI the net benefit increases from 0.1 (original Feverkidstool) to 0.5 by the updated Feverkidstool. In absolute numbers this would mean that by using the updated Feverkidstool at a threshold of 20%, we achieve 20 or 50 more correct treatment decisions (out of 100 patients) than when using the original Feverkidstool. Table 2 gives more detailed insight on the effects of the updated model in diagnostic value using several thresholds. Using a rule of thumb of LR+ of 5 and LR− of 0.2,25 thresholds of 10% and 2.5% using the updated model seem better applicable for ruling in and out of both pneumonia and other SBIs.
Decision curve analysis with the net benefit of starting antibiotics to none of the patients (black line), to all patients (grey line), the original Feverkidstool (red line) and the updated Feverkidstool (blue line), depending on the choice of probability threshold for starting antibiotics for pneumonia (a) and other SBIs (b). FKT, Feverkidstool; SBI, serious bacterial infection.
Diagnostic performance of prediction models including clinical signs and symptoms and either C reactive protein or the assay
Reclassification
Table 3 shows the clinical consequences if the updated Feverkidstool was used instead of the original Feverkidstool. In total, the updated Feverkidstool for pneumonia reduced the number of children with a falsely predicted high or intermediate risk from 234 to 143 (39%) compared with the original Feverkidstool, and from 427 to 197 (54%) for children with other SBIs.
Diagnostic reclassification by the updated Feverkidstool compared with the original Feverkidstool
Discussion
In this study, we showed that when a clinical prediction model including CRP was updated with a combined host-protein-based assay, SBIs were predicted more accurately in children presenting with a lower RTI or FWS at the hospital. We showed that children with low-to-moderate predicted risk benefit most from this updated model.
Strengths and weaknesses of the study
A strength of the present study is the used combination of variables. To our knowledge, this is the first study combining clinical parameters with both bacterial and viral biomarkers. A second strength is the use of an expert panel reference standard, which has the advantage of capturing a wide spectrum of illness severities, including difficult to diagnose cases, thereby reflecting the diagnostic process in clinical practice.26 Another strength is the prospective patient recruitment. The ethics committees approved venous blood sampling even if blood sampling was not indicated for routine care. Therefore, a wide spectrum of illness severities was captured, including difficult to diagnose cases. In addition, a strength of the Feverkidstool is its polytomous character. This enables the discrimination between different diseases: pneumonia and other SBIs versus no SBI.27 Finally, in clinical practice different thresholds are needed depending on the setting. Therefore we performed DCA to help interpret the differences between the models along the wide range of predicted probabilities.20
Limitations of our study should also be addressed. First, the number of bacterial cases was relatively low. Therefore, it was not possible to refit the individual coefficients for all Feverkidstool variables. The aim of the current study was to see whether the new assay had additional value to the original well validated Feverkidstool rather than to build an optimal diagnostic model. Second, one of the Feverkidstool variables, capillary refill, was not available for any of the participants. Therefore, multiple imputation was impossible. Leaving capillary refill out by entering 0 for all capillary values would have resulted in an unfair reduction of model calibration. We think imputation of the mean was the best available option.24 Even though this may have limited the discriminative performance of the model, we believe the influence is limited, because the dichotomous variable capillary refill has little diagnostic value within the Feverkidstool. In addition, the aim of the current study was to compare the original Feverkidstool with the updated model. Imputation of capillary refill values was performed similarly for both models, so this had no influence on the comparison of the models. Third, the reclassification is based on arbitrarily chosen thresholds (2.5% and 10%). These thresholds, however, mostly correspond with LR that have been reported to be meaningful in decisions for febrile children: LR+>5.0 for ruling in SBIs and LR−<0.2 for ruling out SBIs.25 Fourth, the Feverkidstool includes important clinical variables that are used by every physician when deciding whether to start antibiotics or not. Following clinical care, during the expert panel reference standard process, the panel was provided with a wide range of clinical information, including the clinical Feverkidstool variables, but they were not informed about the algorithm. We, however, cannot exclude that incorporation bias—in which part of the test being assessed is included in the reference standard—has resulted in some degree of overestimation of the outcome. Fifth, due to the relatively low number of bacterial cases, some diseases, such as bacterial meningitis, were not observed in our study cohort. Information to discriminate sepsis from bacteraemia cases was not available. Sixth, in the current study, none of the ED patients were hospitalised at the intensive care unit. Further studies with higher numbers of patients with a more severe clinical presentation are warranted as especially in these patients early detection of a SBI can be critical. Seventh, in this study we wanted to study the added value of the assay on top of the model that had also undergone impact analysis28 (ie, the original Feverkidstool (including CRP)), rather than developing a new model. As both the assay and the original Feverkidstool contain CRP, we had to solve this. Refitting the Feverkidstool’s model without CRP will not provide the answer to our research question on the added value of the assay to the Feverkidstool. In addition, the number of bacterial cases was too low to refit the original Feverkidstool. We also did not want to simply add the assay as a predictor in the Feverkidstool model, as this would downwardly bias the assay’s added value. Therefore, we essentially removed the effect of the CRP in the Feverkidstool by assigning the median value of CRP to the CRP single predictor in the Feverkidstool to all patients. To our knowledge, using the median CRP value was the best possible approach to avoid double counting of CRP. Finally, there were 71 patients for whom the expert panel could not assign a final diagnosis. Such inconclusive cases are inherent to studies using outcomes lacking a gold standard. To make optimal use of the data from all recruited patients, we have imputed these reference standard diagnoses. As the imputed diagnoses are used for both the original and the updated Feverkidstool, we do not expect this to have influenced the results of updating the Feverkidstool. In addition, a sensitivity analysis in which all cases with imputed diagnoses were excluded showed comparable results.
Comparison with existing literature
The study in which the Feverkidstool was developed and the OPPORTUNITY Study had comparable inclusion criteria. Most importantly, both studies included children suspected of infection based on increased temperature.12 14 Differences in inclusion criteria should also be discussed. The Feverkidstool derivation cohort included Dutch children aged 1 month to 15 years, whereas the OPPORTUNITY Study included Dutch and Israeli children aged 1 month to 5 years. In addition, for the Feverkidstool development study, children who received antibiotics before the ED visit were excluded; for the OPPORTUNITY Study antibiotic use was no exclusion criterion.12 14 As mentioned/argued in the methods, we additionally excluded children admitted to intensive care (all referred from wards of other hospitals, n=14), and added 28 children aged 1–2 months from the OPPORTUNITY subanalysis.
We recently confirmed the external validity of the Feverkidstool, but when procalcitonin (PCT) was added to the prediction model or when CRP was replaced by PCT the accuracy for predicting SBIs in febrile children did not improve.11 Another study updated the Feverkidstool by adding PCT and resistin, resulting in minimal changes in LR+ and LR− for different risk thresholds.13 In contrast to the two abovementioned Feverkidstool updates, our current study has shown that updating the Feverkidstool with the assay does meaningfully improve the accuracy of the model. This further confirms our previous observation that the combination of CRP, TRAIL and IP-10 had higher diagnostic value in differentiating between bacterial and viral infections compared with PCT or CRP alone.14
Implications for clinical practice
Clinical signs and symptoms play an important role when physicians diagnose febrile children, but do not sufficiently differentiate between viral and bacterial infections. Therefore, the use of diagnostic prediction models that include both clinical parameters and biomarkers is intuitive and helpful. The Feverkidstool provides two risk percentages; the risk for having pneumonia and the risk for having other SBIs. After further validation, a digital calculator should be constructed to facilitate potential clinical use. Both the assay and the Feverkidstool are designed to predict the outcome ‘bacterial infection’, and not the general level of illness of the child. Therefore, the outcome can be used to guide decisions on starting antibiotics or not, but cannot be used to guide decisions on how to administrate antibiotics or what type. We showed that the updated Feverkidstool has the most added value for patients in the low–moderate risk group, with predicted risk for SBIs below 40%. At thresholds of 2.5% and 10%, the reclassification table showed substantial improvement in diagnosis. The cases with predicted risks between low to moderate (2.5 vs 40%) may be characterised by having intermediate values of CRP with higher diagnostic uncertainty. Adding two viral biomarkers to the prediction rule will provide an extra dimension to the model and therefore improve diagnosis especially for those cases. As a point-of-care test is under development, the manufacturer has not given an indication to the eventual cost yet. To optimise cost-effective use of the combination test, our results suggest that the added value of the assay is the highest in children with a predicted risk <40% as predicted by the original Feverkidstool. Utility studies are needed to determine the cut-off for the best clinical utility and cost-effectiveness.
Implications for future research
Since we have proven the accuracy of the updated Feverkidstool, the next step is to perform a prospective external validation and to evaluate its impact on resource use and antibiotic treatment. An important aspect is to define risk cut-offs for different settings in clinical practice, for example, young children and children with comorbidities. In addition, in order to optimise resource use, new biomarkers may benefit selected patient subgroups in particular (eg, selected on a set of clinical characteristics/predicted risk) rather than in all febrile children. This targeted risk approach may also be applied to position the role of, for example, myxovirus resistance protein A and CRP.29
Conclusion
In conclusion, a new blood assay including viral and bacterial biomarkers, combined with a clinical prediction model is, in this study cohort, superior to the model with CRP only for predicting SBIs in preschool children. In children with low-to-moderate predicted risk of SBIs in particular, the updated Feverkidstool with the assay has the potential to optimise targeted antibiotic prescription and to prevent unnecessary use of antibiotics.
References
Footnotes
Contributors CvH was responsible for protocol development, data analyses and was the main author of the paper; JSvdM was responsible for protocol development and writing of the manuscript; CN was responsible for data analyses and revision of the manuscript; LB supervised protocol development, analyses and writing of the manuscript; RO was responsible for protocol development and supervised the analyses and writing of the manuscript.
Funding The OPPORTUNITY Study was supported by an unrestricted grant from MeMed Diagnostics to the University Medical Centre Utrecht. The sponsor of the OPPORTUNITY Study had no role in study design, data collection, data analysis, data interpretation, writing of the manuscript or the decision to submit the manuscript for publication.
Competing interests None declared.
Patient consent for publication Parents gave written informed consent.
Ethics approval The OPPORTUNITY Study was approved by the ethics committees in the participating countries.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information.