Multicentre external validation of the Neonatal Healthcare-associated infectiOn Prediction (NeoHoP) score: a retrospective case–control study

Study design and participants

To validate the NeoHoP score, we used prospectively collected clinical and laboratory data from three large South African neonatal units that participated in a global neonatal sepsis observational cohort study (NeoOBS),17 including 200 neonates hospitalised in each centre (Chris Hani Baragwanath Hospital, Charlotte Maxeke Johannesburg Academic Hospital and Tygerberg Hospital; NeoOBS-SA) between August 2018 and February 2020.

The NeoOBS study enrolled hospitalised infants of any birth weight or gestation, with postnatal age of <60 days, with suspected sepsis episodes, collecting daily observations of clinical signs and symptoms of possible infection, vital signs, supportive care, antibiotic treatment, laboratory indices (complete blood count, CRP and microbiology specimens) and 28-day mortality.17 Infants were eligible for inclusion if the local physician commenced antibiotic treatment for a new episode of neonatal sepsis meeting the enrolment criteria. These criteria included a minimum of two clinical, or one clinical and one laboratory sepsis criteria based on the WHO18 and EMA19 criteria.17 If the same pathogen was isolated on a repeat culture while the patient was still receiving appropriate antibiotic treatment, it was considered a single infection episode. Multiple sepsis episodes were included if a new antibiotic regimen was started after a blood culture for a new episode of sepsis occurring during the 28-day follow-up period. Infants were excluded if an alternative primary diagnosis was suspected by the treating physician. Data were collected by research and clinical staff based on clinical observation and routine source documentation and entered using REDCap electronic data capture tools.20 21

To derive the patient population for the external validation of the NeoHoP score, the NeoOBS-SA data were interrogated, and the following exclusion criteria were applied to identify patients to be included in the external validation cohort. The exclusion criteria were sepsis investigations performed <72 hours after birth/hospital admission; positive cultures from sites other than blood, cerebrospinal fluid (CSF); positive cultures from the tip of a central venous catheter; repeat investigations within a single infection episode; positive cultures (blood and/or CSF) classified as contaminants by the site clinicians and microbiologists (using site-specific definitions); absence of a CRP result; and incomplete clinical notes and/or absence of any of the score parameters.

Study definitions

The eligible NeoOBS-SA HAI episodes, occurring after 72 hours of life/admission, were classified into two categories based on definitions adapted from those used in the NeoHoP study (table 1).16 Central line-associated bloodstream infection was defined as a primary bloodstream infection in a patient who had a central line within 48 hours before the development of infection; infection must not be related to an alternative cause.22 Laboratory investigations were considered as related to a single episode of infection if they were performed within 48–72 hours of the original investigation.

NeoHoP score

The NeoHoP score is a novel infection prediction score developed for the evaluation of suspected neonatal HAI episodes using clinical, management and laboratory variables in a retrospective cohort of VLBW infants in a low-resource setting.16 It is an easy-to-use score, consisting of five variables (capillary refill time >3 s, lethargy, abdominal distention, presence of a central venous catheter currently or in the preceding 48 hours and laboratory CRP ≥10 mg/L. A score of ≥2 is well positioned to be used as a ‘rule-in’ test.

Statistical analysis and evaluation of NeoHop score

Statistical analysis was performed using IBM SPSS Statistics for MacIntosh, V.27.0, using an α level of 0.05 with a corresponding 95% CI for descriptive statistics. For normally distributed continuous variables, means and SD were calculated. Medians and IQRs were used for non-normally distributed continuous data.

We used a standard 2×2 contingency table to classify the disease status of all cases. Using the reference standard of any HAI, we classified the disease status as ‘true positive’ when the NeoHoP score correctly identified the presence of any HAI (table 1). ‘True negative’ indicates a neonate where the score correctly identified the absence of any HAI. ‘False positive’ refers to neonates where the score falsely identified the presence of any HAI, when the neonate did not have any HAI. ‘False negative’ indicates neonates where the score failed to identify the presence of any HAI.

Diagnostic test evaluation was performed using MedCalc Software, V.20.0.5. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and likelihood ratios (positive likelihood ratio, PLR; negative likelihood ratio, NLR) were calculated for any HAI episode in the external validation cohort (NeoOBS-SA), as well as for a subset of the NeoOBS-SA data that included all VLBW infants (NeoOBS-SA-VLBW). A good screening test should have a low false-negative rate and thus a high sensitivity.23 A test with a PLR of >10 or conversely NLR <0.1, is considered a good screening test.23 24 The discriminative performance of the score was evaluated by assessing the receiver operating characteristic (ROC) curves and area under ROC curves. The Youden index (J), a summary measure of the ROC curve, was also calculated. J=1 represents a perfect diagnostic test, and J=0 indicates a test that is not effective in determining disease status.25

All findings were reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE-NI) criteria.26

Patient and public involvement

Patients and/or the public were not involved in this research’s design, conduct, reporting or dissemination plans.

Derivation of the study population for the external validation of the NeoHoP score

In the NeoOBS-SA cohort (n=600 patients) at three tertiary hospital neonatal units, 1766 possible sepsis episodes were investigated (figure 1). A total of 1193 (68.6%) episodes were excluded, leaving 573 sepsis investigation episodes performed in 346 infants suitable for external validation of the NeoHoP score (figure 1). Of the 573 suspected HAI episodes, the majority were confirmed as any HAI (468, 81.7%) including presumed HAI (282, 49.2%) and proven HAI (186, 32.5 %).

Request permission

Neonatal demographic profile

Neonates in the external validation cohort (NeoOBS-SA) had higher median birth weight (1270 g vs 1010 g) and gestational age (30 weeks vs 28 weeks) than those in the internal validation cohort, with both differences being statistically significant (p<0.001) (see table 2). Neonates in the external validation cohort (NeoOBS-SA) were more likely to be born before arrival at the healthcare facility when compared with the internal validation cohort. The proportion of neonates undergoing sepsis evaluation, where any HAI was diagnosed, was significantly higher in the external validation cohort than the internal validation cohort. Pathogen distribution was similar in the cohorts with Gram-negative infections predominating; fungal infections were more common in the external validation cohort (online supplemental table 1).

Performance of the NeoHoP score

When applied to the external validation cohort (NeoOBS-SA), the NeoHoP score achieved a sensitivity of 65%, specificity of 92%, a PLR of 7.73 and a Youden index of 0.57 for a score of ≥2. The area under the ROC curve was 0.874 (95% CI: 0.838 to 0.910) for the prediction of HAI (table 3). There were 1.6% (9/573) false positive results and 28% (163/573) false negative results. Similar results were achieved when the NeoHoP score was applied to a subgroup of the external validation cohort, which included only VLBW infants (NeoOBS-SA-VLBW). In this subgroup, a NeoHoP score of ≥2 achieved diagnostic performance of sensitivity (66%), specificity (90%), PLR (6.44) and Youden index (0.56), with an area under the ROC curve of 0.861 (95% CI: 0.815 to 0.907) (table 3). When applying the score to only proven HAI, the area under the curve was 0.627 (95% CI: 0.579 to 0.627) (figure 2).

Request permission

Discussion

We conducted a multicentre external validation of the NeoHoP score in a prospectively enrolled cohort of 600 South African neonates participating in a global neonatal sepsis observational (NeoOBS) study. The NeoHoP infection prediction score demonstrated comparable diagnostic accuracy with high specificity and high PLRs (>85%) across all weight categories. This confirms that a NeoHoP score ≥2 is effectively positioned as a ‘rule-in’ test for HAI in settings where CRP is used as a marker for neonatal sepsis.

The NeoHoP score comprises five variables making it easy to use and incorporating parameters available in most low-resource settings. Compared with the original study the score achieved higher sensitivity, slightly lower specificity, higher PPV and a higher Youden index in both birth weight categories. Although the PLRs of 7.74 and 6.44 for all birth weights and VLBW infants, respectively, were lower than the ideal PLR of >10 required to ‘rule-in’ disease, the high specificity of the NeoHoP score still retains valuable clinical utility.23 24

Existing infection prediction scores, such as the NOSEP1,27 NOSEP-NEW1,28 Singh et al,29 Rosenberg et al30 and Bekhof et al31 scores, were all developed in settings outside of Africa and did not perform well when applied to a South African cohort of VLBW infants. In an evaluation of these scores in an LMIC setting, the ROC curves for diagnostic accuracy for the prediction of proven and/or presumed infection were 0.898, 0.820, 0.550, 0.566 and 0.620, respectively.15 The present external validation of the NeoHoP score achieved diagnostic accuracy similar to the NOSEP1 score (fever ≥38.2°C, CRP ≥14 mg/L, neutrophil percentage >50%, platelets <1 50 000/mm³, total parenteral nutrition ≥14 days),27 with an area under ROC curve of 0.874 and 0.861 for all birthweights and VLBW infants, respectively.

Infection prediction scores developed for use in LMIC should ideally not include parameters such as blood gas analysis, blood pressure monitoring and continuous heart rate monitoring as these are not frequently available in resource-limited settings. The NeoHoP score was developed specifically for such settings, and purposively included parameters that are readily available in most LMIC neonatal units. Lethargy and abdominal distention are subjective clinical parameters included in the NeoHoP score, and this may be concerning as it is subject to individual interpretation. However, it should be highlighted that the presence of two or more features is required for the NeoHoP score to achieve clinical relevance. The NeoOBS-SA data were collected prospectively for a global neonatal sepsis study, thus recording the parameters required for the NeoHoP score was not mandated. Due to this, 19% of HAI episodes were excluded due to incomplete clinical records, and 28% were excluded as no CRP was done. Despite the exclusion of 46% of the HAI episodes investigated, the NeoHoP score performed well in the NeoOBS-SA cohort. Of note, the NeoOBS study targeted infants with highly suspected sepsis, in comparison to the internal validation cohort of the NeoHoP score which included any baby that received a sepsis work-up. This may explain the higher proportion of confirmed HAI in the external validation cohort (NeoOBS-SA), as well as the difference in the specificity and NPV between the internal validation cohort and the NeoOBS-SA cohort.

The inclusion of CRP in the score may be considered controversial. The purpose of an infection prediction score is to assist clinicians at the bedside in their decision-making process. A laboratory CRP is not always immediately available in many resource-limited neonatal unit settings. Using a point-of-care (POC) CRP test can mitigate the delay in results, thus enhancing the utility of the score as a bedside diagnostic aid.13 POC CRP has been evaluated in another South African neonatal unit and was found to be a quick and reliable method to determine CRP, that can be used to rationalise antibiotic use and reduce hospital expenditure.32 In a recent systematic review, it was demonstrated that POC tests, including POC CRP, are cost-effective in reducing antimicrobial prescribing in LMIC settings,33 highlighting the need for further studies evaluating the use of POC CRP, especially looking at cost-effectiveness and feasibility in low-resource settings. Additionally, there is no consensus in the literature regarding the appropriate time to perform a CRP. Performing a CRP at the onset of symptoms may result in a false-negative result as serum CRP concentrations rise within 10–12 hours, and peak after 36–48 hours.34 In this study, conducted in three large tertiary hospitals in South Africa, CRP testing was not universally performed in all investigations, despite all of these facilities having access to an on-site laboratory. This may be due to several reasons, including the uncertainty regarding the correct time to perform the test and the interpretation of these test results, as well as internal institutional cost-saving measures that may limit laboratory investigations.

A strength of this study is that the external validation cohort included a more diverse neonatal population, as evidenced by the statistically significant differences in the demographics of the study population when compared with the internal validation cohort. Although the NeoHoP score was initially developed for use in VLBW infants, it demonstrated strong performance across all weight categories, thus increasing its generalisability for diagnosing HAI in all newborns.

The predominance of Gram-negative organisms in the external validation cohort is in keeping with reports from other LMICs.7 35 In a recent South African national study on neonatal bloodstream infections, 57% of the identified isolates were Gram-negative bacteria, and 7% were fungal.36 The study also found that these infections were more likely to occur in national central and provincial tertiary facilities, with the highest incidence occurring in the Gauteng province, where two of the study sites are located.36 In the external validation cohort, Gram-negative bacteria were isolated in 53.8% and fungal infections in 12.4% of cases, confirming that the study population is representative of the broader South African neonatal population. It is concerning that the diagnostic performance of the score was less optimal when only including proven HAI, however, this may be indicative of the low yield of blood cultures in paucibacillary bacteraemia, as well as the important role of presumed sepsis and non-bacterial infections in the neonate.37 It must be noted that in the Newborn Essential Solutions and Technologies (NEST360) study, it was demonstrated that in some African countries blood cultures are under-used, with 40% of the hospitals evaluated not performing any blood cultures for newborns.38

The major strength of this study is the use of a large, well-curated, prospectively collected dataset (NeoOBS-SA) including all birth weight categories, demonstrating the utility of the NeoHoP score in neonatal populations other than VLBW infants. As the score performed well in this expanded patient cohort, it speaks to its generalisability in a wider neonatal population where CRP testing is included in the management of neonatal sepsis.

A major limitation of this study was the retrospective study design and the application of the score to neonates from central academic hospitals in a single country. The NeoOBS study targeted infants with highly suspected sepsis, which may have introduced some patient selection bias.

Conclusion

The NeoHoP score has demonstrated remarkable performance in a multicentre validation, establishing its efficacy as a rule-in test for HAI across all birth weight categories. This signifies a significant advancement, suggesting that healthcare practitioners in LMIC settings could use the NeoHoP score as a valuable sepsis diagnostic tool to ‘rule-in’ HAI, complementing existing laboratory evaluations. Future research efforts should focus on prospectively evaluating the NeoHoP score as a bedside test in non-tertiary low-resource settings, possibly combined with POC CRP testing to improve the practicality, and to evaluate the time point at which the use of the score would be most valuable. This synergistic approach holds considerable potential to enhance the diagnostic accuracy and practical application of the NeoHoP score within LMIC neonatal units.