Objective To determine the accuracy of the National Institute of Child Health and Human Development (NICHD) calculator in predicting death and neurodevelopmental impairment in Australian and New Zealand infants.
Design Population-based cohort study.
Setting Australia and New Zealand.
Patients Preterm infants 22–25 completed weeks gestation.
Interventions Comparison of NICHD calculator predicted rates of death and death or neurodevelopmental impairment, with actual rates recorded in the Australian and New Zealand Neonatal Network cohort.
Main outcome measures Infant death and death or neurodevelopmental impairment rates.
Results A total of 714 infants were included in the study. Of these infants, 100 (14.0%) were <24 weeks, 389 (54.5%) male, 529 (74.1%) were singletons, 42 (5.9%) had intrauterine growth restriction, 563 (78.9%) received antenatal steroids and 625 (87.5 %) were born in a tertiary hospital. There were 288 deaths (40.3%), 75 infants (10.5%) with neurodevelopment impairment and 363 (50.8%) with death or neurodevelopmental impairment. The area under the curve (AUC) for prediction of death and the composite death or neurodevelopmental impairment by the NICHD calculator in our population was 0.65(95% CI 0.61 to 0.69) and 0.65 (95% CI 0.61 to 0.69), respectively. When stratified and compared with gestational age outcomes, the AUC did not change substantially for the outcomes investigated. The calculator was less accurate with outcome predictions at the extreme categories of predicted outcomes—underestimation of outcomes for those predicted to have the lowest risk (<20%) and overestimation for those in the highest risk category (>80%).
Conclusion In our recent cohort of extremely preterm infants, the NICHD model does not accurately predict outcomes and is marginally better than gestational age based outcomes.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
What is known about this topic
Tools to predict death and neurodevelopmental outcomes of extremely preterm newborns are limited and are typically based on gestational age in completed weeks.
Prediction models such as the National Institute of Child Health and Human Development (NICHD) calculator have been validated in several cohorts and are routinely used in clinical practice.
What this study adds
The NICHD model does not provide an accurate prediction of outcomes for extremely preterm infants born between 22 weeks and 25 weeks in Australia and New Zealand.
The NICHD model showed inaccuracies at the highest and lowest predicted probabilities of outcomes for extremely preterm infants in the Australian New Zealand cohort.
The initiation of resuscitation and provision of intensive care for extremely preterm infants at the margins of viability between 22 weeks and 25 weeks gestational age is an area of major controversy and raises many ethical questions for parents and medical staff. Even with improvements in intensive care, these infants continue to be at high risk of death, major morbidities and neurodevelopmental impairment.1–3 Clinicians in the neonatal intensive care unit (NICU) routinely provide antenatal counselling to parents and guide management decisions based on available estimates of survival and neurodevelopmental outcomes of these infants.4
Tools to predict death and neurodevelopmental outcomes of extremely low gestational age newborns are limited and are based on best estimates of gestational age in completed weeks.5–9 The US National Institute of Child Health and Human Development (NICHD) Neonatal Research Network developed a model for predicting outcomes for extremely low gestational age newborns (22–25 weeks) based on five risk factors including gestational age in completed weeks, gender, estimated birth weight, prenatal steroids and plurality.6 This study demonstrated that each of these factors had an impact on survival and neurodevelopmental outcome similar to that of a single week of gestational age. This prediction model has since been validated by several other groups and is routinely used in clinical practice with the creation of an internet-based calculator that estimates survival and survival without neurodevelopmental impairment (https://www.nichd.nih.gov/about/org/der/branches/ppb/programs/epbo/pages/epbo_case.aspx).10 11
The generalisability of the NICHD model to non-academic and non-US setting deserves further study as it is based on data collected from inborns at 19 US tertiary academic institutions.10 11 Survival rates and outcomes based on single and selected clusters of NICUs may be biased by local patient variations and heterogeneous medical practices. The objective of our study was to determine if the NICHD calculator published in 2009 is valid in predicting death and neurodevelopmental impairment for a contemporary population cohort of extremely low gestational age newborns admitted for neonatal intensive care during 2009 to 2010 in Australia and New Zealand.
The data for this study were obtained from the Australian and New Zealand Neonatal Network (ANZNN) database that began in January 1995. All ANZNN registrants born between 2009 and 2010, 22–25 weeks gestation and admitted to one of the 29 tertiary NICUs in Australia and New Zealand were included in this study. The Network monitors the care of high-risk newborns by pooling data for all infants admitted to tertiary level NICUs during the neonatal period (≤28 days) for any of the following indications: gestational age of <32 completed weeks, birth weight of <1500 grams, need for assisted ventilation (mechanical ventilator or continuous positive airway pressure device (CPAP)) for ≥4 consecutive hours, received major surgery or received therapeutic hypothermia. The care of extremely preterm infants has been regionalised since the early 1990s, where the ANZNN contains 99% of all live births <29 weeks in both countries. Neonatal, maternal and perinatal data are collected from each NICU and compiled into a central ANZNN database located at the University of New South Wales, Australia. Each NICU has an audit officer who collects and checks the data before submission into the central database. The accuracy of the data collection is validated by random data crosschecking by the ANZNN coordinator.
Variables were defined according to the ANZNN data dictionary (https://anznn.net/dataresources/datadictionaries). Gestational age is defined as the best obstetric estimate of completed weeks based on obstetric history, clinical examination and prenatal ultrasound. Infants were considered outborn if they were born outside of a hospital with a tertiary level NICU. Major morbidity was defined as the presence of any of the following: bronchopulmonary dysplasia ,12 grade 3 or grade 4 intraventricular haemorrhage,13 periventricular leucomalacia, medically or surgically treated necrotising enterocolitis or grade 3 or grade 4 retinopathy of prematurity.14 The composite outcome of death and/or major morbidity included infants who died or had at least one of the major morbidities.
Survival and neurodevelopmental outcomes were assessed at 2–3 years corrected age as per the ANZNN study protocol. The presence of neurodevelopmental impairment was defined as blindness, deafness requiring amplification, cerebral palsy with severity graded by Gross Motor Function Classification System (GMFCS) levels 2 to 5, or moderate to severe developmental delay. Moderate to severe developmental delay was determined by a score of >2 SD below the mean on either the cognitive, language or motor scale of the Bayley Scales of Infant Development-Third Edition (BSID-III) or Griffith Mental Developmental Scales; or by individual review by the ANZNN follow-up subcommittee of developmental outcomes from other formal developmental assessments or clinical assessments of development by a clinician.
The NICHD risk prediction for the outcomes investigated for each infant was generated based on the logistic mixed model equation provided by the authors.6 The outcomes predicted by the calculator were compared with the actual outcomes for each patient in our study cohort. The following outcomes were compared: death, death or neurodevelopmental impairment.
Receiver operative characteristics (ROC) curves for death and the composite outcome of death or neurodevelopmental impairment were constructed to assess the accuracy of the NICHD calculator for predicting these outcomes in the ANZNN study population. Calculation of the area under the curve (AUC) and its 95% CI were used to compare the performance of the NICHD calculator. The predicted rates of death and the composite outcome from the NICHD calculator were compared with the actual rates in the ANZNN study population. Separate ROC curves were generated for the different gestational age in weeks to compare the prediction of the calculator with gestational age outcomes, which are typically used for outcome prediction of extremely preterm infants. Statistical analyses were performed by using IBM SPSS Statistics V.22.0. (IBM Corp, Armonk, New York, USA)
Of the 957 ANZNN registrants who were born at 22–25 weeks gestation and at ≥401 grams and <1000 grams, 714 were eligible for inclusion in the study cohort. The overall follow-up rate for neurodevelopmental assessment was 78%. Exclusions were made according to figure 1. The baseline clinical characteristics and outcomes are described in table 1. A total of 100 infants (14.0%) were born at less than 24 weeks. More than a third (36.0%) of the infants were born of mothers who were either ≥35 years or <20 years of age. A total of 590 (82.6%) mothers presented in preterm labour, 240 (33.6%) with antepartum haemorrhage and 52 (7.3%) with hypertension in pregnancy. The distribution of the infants by gestational age: 22 weeks—10 (1.4%), 23 weeks—90 (12.6%), 24 weeks—268 (37.5%), 25 weeks—346 (48.5%). Of the infants, 529 (74.1%) were singleton births, 42 (5.9%) had intrauterine growth restriction and 625 (87.5 %) were born in a tertiary care hospital. Vaginal delivery accounted for 57.1% of all births with the proportions decreasing from 22 weeks to 25 weeks gestation. Antenatal steroids were given to 78.9% of all infants prior to delivery and 84.7% were intubated at delivery.
A total of 458 infants (64.1%) were noted to have at least one of the major neonatal morbidities and 75 (10.5%) were noted to have neurodevelopmental impairment at 2–3 years corrected age. The overall number of deaths in this cohort was 288 (40.3%), of which 278 (38.9%) occurred prior to discharge. The composite outcome of death and/or any major morbidity was noted in 578 (81.0%) infants and death or neurodevelopmental impairment in 363 (50.8%) infants.
The precision of NICHD calculator was low for prediction of death in our cohort, with an overall AUC of 0.65 (0.61–0.69) (figure 2). The accuracy of the calculator for predicting death did not change substantially when stratifying by inborn or outborn status, (AUC 0.65 (0.60–0.70) compared with 0.65 (0.53–0.77), respectively) (supplemental figures 1 and 2). When stratified by gestational age and compared with the outcome of death, the AUC ranged from 0.56 to 0.61 (figure 2). For the composite outcome of death or neurodevelopmental impairment, the accuracy of the NICHD for predicting this outcome was similarly low with an AUC of 0.65 (0.61–0.69) (figure 3). This also did not change substantially when stratified by inborn or outborn status, (AUC 0.66 (0.61–0.70) compared with 0.66 (0.54–0.77), respectively) (supplemental figures 1 and 2). When compared with actual stratified gestational age outcomes, the AUC ranged from 0.57 to 0.62 (figure 3).
Table 2 illustrates the comparison of the predicted death according to the NICHD calculator and the actual deaths in our cohort as stratified by gestational age. For the overall cohort, the actual rate of death of 27.5% exceeded the predicted probability for the group with the lowest estimated probability of death (<20%). The actual rate of death for this group with a predicted probability of <20% ranged between 25% and 100% for infants 23–25 weeks gestation. Comparison of the predicted and actual composite outcomes of death or neurodevelopmental impairment showed similar inaccuracy of the calculator for those with the highest and lowest predicted probabilities of this outcome (table 3). The actual rate of death or neurodevelopmental impairment for the overall cohort was 33.0% for the lowest predicted group of <20% and was 69.4% for those in the highest predicted group of >80%.
In this study, we attempted to determine the validity of the 2009 NICHD calculator for the prediction of outcomes in extremely preterm infants in our contemporary Australia and New Zealand population 2009–2010 cohort. We found the calculator to have underperforming predictive ability in determining the rates of death and the composite outcome of death or moderate to severe neurodevelopmental impairment. This predictive capability did not improve when stratified by birth in a tertiary centre or outside a tertiary centre. Specifically, the calculator was less accurate with predictions at the extreme categories of predicted outcomes. When stratified and compared with actual gestational age outcomes, we found the calculator to have marginally better predictive ability for both outcomes.
The predictive capability of the calculator in our cohort differed somewhat to previous reports of comparisons in the USA and Australia. In an attempt to validate the accuracy of the NICHD calculator in a cohort from California, Lee et al concluded that the model improved the prediction of outcomes compared with gestational age based outcomes in their setting of mostly tertiary level academic and community NICUs.10 In their study the authors determined that the NICHD model resulted in a higher true-positive rate and a lower false-positive rate in this cohort. However, this study was limited to predischarge outcomes and did not assess or compare neurodevelopment. In another cohort study of 114 infants from Victoria, Australia, Boland et al demonstrated that the NICHD calculator was good at predicting the outcomes of death in inborn infants (AUC 0.8 (0.7–0.9)) but was not as precise for outborns (AUC 0.6 (0.3–0.9)).11
Our study expands on the earlier study by Boland et al with the inclusion of more infants (714 infants) from all tertiary level NICUs from Australia and New Zealand. This allows for better generalisability of the results for the population. Our results indicate that the NICHD calculator does not seem to accurately predict outcomes in our contemporary cohort.15 Specifically, overestimation of the probability of death or neurodevelopmental impairment at the highest risk group may affect decisions surrounding initiation and continuation of intensive care. While it is clear from many studies that survival and outcomes cannot be accurately predicted by gestational age alone, there is a need to validate the accuracy of available models and to potentially develop a prediction model that is based on local data for the population. The provision of accurate and current data on outcomes and prognosis is vital to inform decisions made by parents and the medical staff.
The strength of our study is that it includes all the tertiary level NICUs within Australia and New Zealand, making this a good representation of the extremely preterm population. The standardised definitions and rigorous data crosschecking by ANZNN ensures completeness and validity of the data obtained. The NICHD calculator was developed with the inclusion of only mechanically ventilated infants as an indicator of active provision of intensive care. However, with the high usage of early CPAP for many units in our cohort,16 we felt that inclusion of such infants is justified as it would constitute the provision of intensive care for which the calculator is currently being used. There were no babies who died without mechanical ventilation within the first day of life in our study, which likely indicates that there was no active palliation in our study cohort. Even so, it is likely that our inclusion of infants who survive without mechanical ventilation would increase the number of infants with favourable outcome than would be predicted.
While blindness, deafness requiring amplification, GMFCS levels 2 to 5 cerebral palsy were similarly included in the definition of moderate to severe neurodevelopmental impairment for the NICHD calculator, there were some notable differences in the classification of neurodevelopmental impairment in our study and that of the initial NICHD study. For the outcomes calculator, neurodevelopmental impairment was defined as a score of ≤70 (≥2 SD below the mean) on either the Psychomotor Developmental Index or Mental Developmental index of the BSID-Second Edition (BSID-II). The ANZNN uses the definition of >2 SD below the mean on either the cognitive, language or motor scales of BSID-III. Several studies have highlighted the difference between the two tests, where BSID-III tends to underestimate the prevalence of neurodevelopmental impairment by 2%–7%.17–19 There have also been reported difficulties in trying to reconcile the cut-offs between the two tests for the individual components—cognitive and language.20 This could account for the suboptimal accuracy of the NICHD calculator in predicting neurodevelopmental impairment as the ANZNN cut-off may be an underestimate of the incidence when compared with the calculator. Due to the differences in the tests between the study cohorts, the composite outcome of death or profound impairment was also not compared between the groups. The timing of the developmental assessment was at an older age for the ANZNN cohort (≥18 months corrected age with the majority or assessments performed at 2–3 years corrected age) where the NICHD cohort screened infants at 18–22 months. Nonetheless, these early assessments at these time points should similarly provide an important indication of any potential developmental issues that require early intervention.
The NICHD outcomes model does not provide an accurate prediction of outcomes in the extremely preterm infants in the ANZNN cohort. There is a need to validate available models for use in different population cohorts and for the development of neonatal outcomes prediction models that are based on local geographical data, such as that for infants born in Australia and New Zealand.
Supplementary file 1
Supplementary file 2
The authors thank the staff, the mothers and their babies cared for in all ANZ NICUs for contributing to this study.
Contributors KTY, AW, KL were involved in the conception and design of the work; acquisition, analysis and interpretation of the data for the manuscript; and initial drafting of the work. NS, RLM were involved in the acquisition, analysis and interpretation of the data for the work. TS, SB, RH were involved in the interpretation of the data for the manuscript. All authors were involved in the critical revision of the manuscript for intellectual content and approved the paper as submitted. All authors agree to be accountable for all aspects of the work and in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Funding The authors KTY and TS, and the study were supported in part by a grant from the Running for Premature Babies Fund, Royal Hospital for Women Foundation. The Australian and New Zealand Neonatal Network were supported from the ANZNN members’ contribution and in part by a grant from Leslie Steven Grant for Newborn Care, Sydney Children’s Hospital Foundation, Australia, and Royal Hospital for Women Foundation, New South Wales, Australia.
Competing interests None declared.
Ethics approval South Eastern Sydney Local Health District Human Research Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.
Collaborators Advisory Council Members of ANZNN (* denotes ANZNN Executives) Australia: Peter Marshall (Flinders Medical Centre, SA), Peter Schmidt (Gold Coast Hospital, QLD), Paul Craven, Koert De Waal* (John Hunter Hospital, NSW), Karen Simmer, Andy Gill*, Jane Pillow* (King Edward Memorial and Princess Margaret Hospitals, WA), Jacqueline Stack (Liverpool Hospital, NSW), Lucy Cooke (Mater Mother’s Hospital, QLD), Dan Casalaz, Jim Holberton* (Mercy Hospital for Women, VIC), Charles Barfield (Monash Medical Centre, VIC), Lyn Downe, Vjay Singde (Nepean Hospital, NSW), Michael Stewart (Newborn Emergency Transport Service, VIC), Andrew Berry (NSW Newborn & Paediatric Emergency Transport Service), Kathryn Browning Carmo (NSW Newborn & Paediatric Emergency Transport Service), Rod Hunt (Royal Children’s Hospital, VIC), Charles Kilburn (Royal Darwin Hospital, NT), Tony de Paoli (Royal Hobart Hospital, TAS), Kei Lui* (Royal Hospital for Women, NSW), Mary Paradisis (Royal North Shore Hospital, NSW), Ingrid Rieger, Tracey Lutz, Shelley Reid* (Royal Prince Alfred Hospital, NSW), David Cartwright, Pieter Koorts (Royal Women’s Hospital, QLD), Carl Kuschel, Lex Doyle, (Royal Women’s Hospital, VIC), Andrew Numa (Sydney Children’s Hospital, NSW), Hazel Carlisle (The Canberra Hospital, ACT), Nadia Badawi (The Children’s Hospital at Westmead, NSW), Guan Koh* (The Townsville Hospital, QLD), Steven Resnick (Western Australia Neonatal Transport Service), Melissa Luig (Westmead Hospital, NSW), Chad Andersen (Women’s & Children’s Hospital, SA). New Zealand: Adrienne Lyn (Christchurch Women’s Hospital), Brian Darlow (Christchurch School of Medicine), Roland Broadbent* (Dunedin Hospital), Lindsay Mildenhall (Middlemore Hospital), Mariam Buksh (National Women’s Hospital), David Bourchier, Lee Carpenter* (Waikato Hospital), Vaughan Richardson (Wellington Women’s Hospital).ANZNN executives not members of hospitals contributing data: Ross Haslam* Chair of the executives, Georgina Chambers* (National Perinatal Statistics and Epidemiology Unit, University of New South Wales); Adam Buckmaster* (Gosford Hospital, NSW), Victor Samuel Rajadurai*, (KK Women’s and Children’s Hospital, Singapore); Barbara Bajuk* (NSW Pregnancy and Newborn Services Network)