Article Text
PDF
Abstract
Objective There is international variation in recommendations regarding developmental screening and growing recognition of the low sensitivity of commonly used developmental screening tools. Our objective was to examine the predictive validity of the Infant Toddler Checklist (ITC) at 18 months to predict a developmental diagnosis at 3–5 years, in a primary care setting.
Methods We designed a prospective cohort study, recruiting in primary care in Toronto, Canada. Parents completed the ITC at the 18-month visit and reported developmental diagnosis at 3–5 years (developmental delay, autism spectrum disorder (ASD), attention deficit hyperactivity disorder (ADHD), learning problem). We calculated screening test properties with 95% CIs. We used multivariable logistic regression analyses adjusted for important covariates.
Results In the final sample (n=488), mean age at screening was 18.5 (SD 1.1) months, and at follow-up was 46.6 (SD 10.0) months. At screening, 46 (9.4%) had a positive ITC. At follow-up, 26 (5.3%) had a developmental diagnosis, including: developmental delay (n=22), ASD (n=4), ADHD (n=1), learning problem (n=1); parents of two children each reported two diagnoses (total of 28 diagnoses). Of four children with a diagnosis of ASD at follow-up, three had a positive ITC at 18 months. The ITC specificity (92%, 95% CI: 89% to 94%) and negative predictive value (96%, 95% CI: 95% to 97%) were high; false positive rate was low (8%, 95% CI: 6% to 11%); sensitivity was low (31%, 95% CI: 14% to 52%). There was a strong association between a positive ITC at 18 months and later developmental diagnosis (adjusted OR 4.48, 95% CI: 1.72 to 11.64; p=0.002).
Conclusion The ITC had high specificity, high negative predictive value, low false positive rate, and identified children with later developmental delay and ASD. The ITC had low sensitivity, similar to other screening tools underscoring the importance of continuous developmental surveillance at all health supervision visits.
- health services research
Data availability statement
Data are available upon reasonable request. Data available upon reasonable request according to the TARGet Kids! data access policy.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
There is international variation in recommendations regarding developmental screening and growing recognition of the low sensitivity of commonly used developmental screening tools. The Infant Toddler Checklist (ITC) was developed for early identification of children, 6–24 months, who have, or are at risk of developing, a communication impairment. There is little research on the predictive validity of the ITC when used to screen children at the 18-month visit in primary care.
WHAT THIS STUDY ADDS
The ITC at 18 months had high specificity and negative predictive value, and a low false positive rate for a developmental diagnosis at 3–5 years, including developmental delay and autism spectrum disorder. Children with a positive ITC at the 18-month visit had higher odds of a developmental diagnosis at follow-up. The low sensitivity of the ITC suggests that a positive ITC at the 18-month visit cannot accurately identify those who will have a developmental diagnosis at 3–5 years.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE AND/OR POLICY
Practitioners and policymakers may consider the ITC as a developmental screening tool at the 18-month health supervision visit. The low sensitivity of the ITC is similar to other screening tools and underscores the importance of continuous developmental surveillance at all health supervision visits.
Introduction
Early identification of young children with developmental disorders is recommended in many countries. While primary care practitioners often perform developmental surveillance, the addition of standardised developmental screening tools may lead to earlier identification and referral for intervention, as shown in a randomised trial.1 However, there is international variation in recommendations regarding screening, including type of tool (general/broadband or domain/disorder-specific tool), age of screening and one-time versus repeat screening.2–11
In Canada, the Canadian Paediatric Society recommends an enhanced 18-month visit in primary care, including the use of a developmental screening tool to ‘stimulate discussion with parents about their child’s development’.2 However, there is no consensus on which screening tool is best suited for one-time screening at this visit, and there is a growing recognition of the low sensitivity of commonly used developmental screening tools.12–17
Considering the importance of speech, language and social communication at 18 months, we examined the Infant Toddler Checklist (ITC). The ITC was developed by Wetherby and Prizant for early identification of children, 6–24 months, who have, or are at risk of developing, a communication impairment.18 Scoring provides recommendations for monitoring and referral.19 The developers also generated evidence supporting the concurrent and predictive criterion validity of the ITC for detection of a range of developmental concerns, including language delay, global developmental delay and autism spectrum disorder (ASD).20–24 However, an instrument should be validated for the target population and setting in which it will be used; we are interested in the criterion validity of the ITC in routine primary care.
Pierce et al selected the ITC for their screen–evaluate–treat model for early identification of children with ASD and have screened more than 44 000 children at 12, 18 and 24 months in a network of 203 primary care paediatricians in California, USA, demonstrating feasibility and physician satisfaction.25 26 Diagnostic evaluation was completed in those referred (39%), leading to developmental diagnoses including ASD and other delays.26 However, since not all children had a diagnostic evaluation, criterion validity of the ITC could not be examined.25 26
Concurrent criterion validity of many developmental screening tools has been evaluated, with concurrent measurement of screening and criterion measures. However, there is great interest in the predictive validity of developmental tools which ‘offers a critical illustration of whether a screening test measures dimensions of development that are enduring and have a meaningful impact on children’s long-term outcomes’.27 Three recent systematic reviews addressed the predictive criterion validity of developmental screening tools; however, of included studies, few examined screening at 18 months or younger, and none included the ITC.28–30
While the ITC may be a promising developmental screening tool for the 18-month visit in primary care, we have identified gaps in evidence regarding its predictive validity. Our objective was to examine the predictive validity of the ITC at the 18-month visit to predict a developmental diagnosis at 3–5 years of age.
Methods
Design, setting, population
Using a prospective cohort design, we recruited healthy children between 2011 and 2018 during a scheduled 18-month health supervision visit at primary care practices participating in a longitudinal cohort in Toronto, Canada called TARGet Kids! (www.targetkids.ca). The cohort profile has been published and registered at ClinicalTrials.gov (NCT01869530).31 Current Canadian recommendations define screening as use of standardised tools to search for developmental delay in asymptomatic populations.32 To ensure a healthy, asymptomatic cohort, we excluded children with chronic health conditions (except asthma), established diagnosis of developmental delay, gestational age <32 weeks, birth weight <1.25 kg and unscheduled visit. To optimise questionnaire completion, parents unable to communicate in English were not included.
ITC at 18 months
At the 18-month visit, parents completed the ITC, a one-page, 24-item checklist, which takes 5 min to complete and 2 min to score.18 19 The checklist produces three composite scores (social, speech, symbolic) and their sum produces a total score. Age-based cut-offs (at or below the 10th percentile) indicate concern/no concern for each of the four scores (three composite scores and total score). The developers recommend that a child with concern on the speech composite should be monitored with repeat ITC in 3 months and referred for evaluation if there is concern on a second ITC (monitor/refer); and a child with concern on the social composite, symbolic composite or total score should be referred for evaluation (refer).
For this study, the ITC was only measured once at the 18-month visit. We examined three components of the ITC: concern for speech delay (defined as concern on the speech composite); concern for other communication delays (defined as concern on the social composite, symbolic composite or total score); positive ITC (defined as concern for speech delay and/or other communication delays).
Developmental diagnosis at 3–5 years
At 3–5 years, using a standardised questionnaire, parents responded to the question: ‘Has your child been diagnosed with any of the following conditions?’ (response options: developmental delay, ASD, attention deficit hyperactivity disorder (ADHD), learning problem, none). Parent response was dichotomised as a developmental diagnosis (yes/no). Parent report of clinician diagnosis, using similarly worded questions, has been used extensively in national surveys of developmental disorders including ASD.33–36
Statistical analysis
We used all available data from children in the TARGet Kids! cohort meeting the eligibility criteria. Descriptive statistics were used to characterise the study participants. To assess the predictive validity of the ITC, we used two approaches.
First, we calculated the screening test properties (sensitivity, specificity, false positive rate, positive predictive value (PPV), negative predictive value (NPV)), with 95% CIs, for each of the three components of the ITC at 18 months, with a developmental diagnosis at 3–5 years as the criterion measure.
Second, we further evaluated the strength of the relationship between a positive ITC at the 18-month visit and the criterion measure, using multivariable logistic regression analyses. Three models were created corresponding to the three components of the ITC: positive ITC; concern for speech delay; concern for other communication delays. Potential confounding variables selected a priori included child age at follow-up, sex, birth weight, maternal ethnicity and family income. Models were adjusted for all covariates regardless of statistical significance.37 All potential confounders had <13% missing data. Missing covariate data were handled by multiple imputation using the fully conditional specification method.38 To reduce the potential for bias, models were run on 20 imputed data sets.39 Results of the 20 imputed data sets were combined, and the parameter estimates (95% CI) for the adjusted pooled models were reported. Statistical significance was defined as p<0.05; all statistical tests were two sided. Statistical analysis was conducted using SAS V.9.4 statistical software (SAS Institute).
Patient and public involvement
Parents and members of the public were not involved in the design, analysis or interpretation of the research presented here. TARGet Kids! has now developed a Parent and Clinician Team (https://www.targetkids.ca/pact).
Results
Participants
Of 593 children with an ITC at baseline and follow-up at 3–5 years, 488 (82%) had outcomes on developmental diagnosis and were included in the analysis (figure 1). The mean age at screening was 18.5 (SD 1.1) months and at follow-up was 46.6 (SD 10.0) months (table 1). At screening, 46 (9.4%) children had a positive ITC: concern for speech delay (n=28, 5.7%); concern for other communication delays (n=30, 6.2%); concern for both speech delay and other communication delays (n=12, 2.5%). At follow-up, 26 (5.3%) children had a developmental diagnosis, with parents of two children each reporting two diagnoses, for a total of 28 diagnoses: developmental delay (n=22); ASD (n=4); ADHD (n=1); learning problem (n=1). Of the four children with a diagnosis of ASD at follow-up (0.8% of the total sample), three had a positive ITC at 18 months, all with concern for other communication delays.
Study participant flow chart. ITC, Infant Toddler Checklist.
Characteristics of study participants (n=488)
Predictive validity
Sensitivity was 23%–31%, specificity 92%–95%, false positive rates 5%–8%, PPV 17%–21% and NPV 96% (table 2). Children with a positive ITC at 18 months had higher odds of a developmental diagnosis at 3–5 years (adjusted OR (aOR) 4.48, 95% CI: 1.72 to 11.64; p=0.002), as did male sex (aOR 3.05, 95% CI: 1.17 to 7.97; p=0.02) (table 3). There was a strong association between ITC concern for speech delay (aOR 4.78, 95% CI: 1.65 to 13.81; p=0.004) and ITC concern for other communication delays (aOR 4.46, 95% CI 1.71 to 11.64; p=0.002) and a developmental diagnosis at 3–5 years (table 4).
Screening test properties of the Infant Toddler Checklist (ITC) screen at 18 months compared with developmental diagnosis at 3–5 years (n=488)
Logistic regression models for the association between screening with the Infant Toddler Checklist (ITC) at 18 months and developmental diagnosis at 3–5 years (n=488)
Multivariable logistic regression models for the association between screening with the Infant Toddler Checklist (ITC) at 18 months and developmental diagnosis at 3–5 years (n=488)
Discussion
In this study, parents of 488 children completed the ITC at their child’s 18-month health supervision visit. On the basis of a 10th percentile cut-off, 48 children were expected to have a positive ITC screen. In our sample, 46 children (9.4%) had a positive ITC screen, including 5.7% with concern for speech delay, 6.2% with concern for other communication delays and 2.5% with concern for both. At follow-up, at a mean age of 4 years, approximately 5% had a developmental diagnosis, including developmental delay, ASD, ADHD and learning problems. Of four children with a diagnosis of ASD at 3–5 years, three had a positive ITC screen, all with concern for other communication delays, which is notable as this ITC component is thought to capture ASD.24–26 Children with a positive ITC at the 18-month visit had 4.48 higher odds of a developmental diagnosis at follow-up. The high specificity (92%–95%) and NPV (96%) suggest that most children with a negative ITC screen will not have a later developmental diagnosis. The low false positive rate (5%–8%) suggests that use of the ITC will result in few unintended harms related to overdiagnosis and over-referral.
We also identified low sensitivity of the ITC suggesting that a positive ITC at the 18-month visit cannot accurately identify those who will have a developmental diagnosis at 3–5 years. Low sensitivity of developmental screening tools to predict later outcomes has been recognised as a challenge, due to the dynamic nature of children’s development.27 30 Factors associated with lower sensitivity include younger age at screening and longer latency from screening to outcome.30 Therefore, it is not surprising that screening at 18 months resulted in a low sensitivity to predict developmental diagnosis at 3–5 years. This underscores the importance of continuous developmental surveillance at all health supervision visits, as recommended by professional organisations.2 5
There have been three recent systematic reviews of predictive validity of developmental screening in early childhood, highlighting the interest in understanding which tools best identify children who may benefit from early identification and intervention.28–30 Of included studies, few examined screening at ≤18 months, and none examined the ITC.
Sim et al examined the predictive validity of language screening tools at 2–6 years in six studies.28 Studies of children <18 months were not eligible, and only two studies had a time to follow-up of ≥12 months, with a mean sensitivity of 54%. Cairney et al examined the predictive value of developmental assessment at 1–5 years in 13 studies and found a positive association between poor early child development and later educational difficulties, high specificity and NPV, and low sensitivity.30 Three studies included children <18 months, which assessed association but not screening test properties.
Schonhaut et al examined the predictive validity of the Ages & Stages Questionnaire (ASQ), which is of great importance given its common use in many countries.29 Of five included studies, three screened children at 36 or 48 months; and while the remaining two studies screened some children at 18 months, they included children born preterm. Lamsal et al (not included in any systematic review) examined the predictive validity of the ASQ for parent report of a developmental diagnosis at 4–5 years.15 At 24 months (the youngest age examined), using the 1 SD cut-off for the ASQ, sensitivity was 84% and specificity was 69%; using the 2 SD cut-off, sensitivity was 32% and specificity was 91%. In summary, little is known about the predictive validity of the ASQ in healthy term infants screened at 18 months.
Low to moderate sensitivity has been found in concurrent validity studies of the ASQ.12 13 16 Warren et al found a sensitivity of 55% and specificity of 86% in a meta-analysis of four studies.16 Sheldrick et al found a sensitivity of 35% and specificity of 89% in children 9 months–5.5 years (mean 2.6 years).13 Wilson et al found a sensitivity of 55% and specificity of 95% on the ASQ Communication Scale in children 23–30 months (mean 25 months).12
Low sensitivity of ASD-specific tools has also been found. In a predictive validity study of the Modified Checklist for Autism in Toddlers with Follow-Up in children 16–26 months in primary care, Guthrie et al found a sensitivity of 39% and specificity of 95% for ASD; and a sensitivity of 12% and specificity of 97% for any delay.14
The ITC was developed by Wetherby and Prizant in US children.18 For identification of communication disorders, concurrent validity was assessed in children 12–17 and 18–24 months: sensitivity 86%–89% and specificity 74%–77%21; and predictive validity in children 12–24 months, with language assessed at 3 years: sensitivity 83% and specificity 70%.21 For identification of ASD, in children 9–24 months (n=5385), 60 who were ≥4 years received a diagnosis of ASD; of these, the ITC identified 56, for a sensitivity of 93%.24
Pierce et al selected the ITC in their screen–evaluate–treat model embedded in primary care, aiming for early ASD intervention before 2 years.25 26 A network of 203 paediatricians screened more than 44 000 children at 12, 18 and 24 months using the ITC. Approximately 39% of children with a positive ITC were referred for diagnostic evaluation. Of these, almost half received a diagnosis of ASD and about one-third received a diagnosis of developmental delay, language delay or other delays.26 Pierce et al found that a diagnosis of ASD becomes stable starting at 14 months, that the most common diagnostic transition was from language or developmental delay to ASD, and that almost 24% of children with an ASD diagnosis at 3–4 years were late identified.40 In our study, three of four children with an ASD diagnosis at 3–5 years had a positive ITC at 18 months, supporting the potential for early identification of ASD using the ITC.
Strengths of the ITC include its focus on communication, an important developmental domain at 18 months for which interventions are available. The ITC distinguishes concern for speech delay (monitor/refer) and other communication delays such as ASD (refer). Overall, the ITC screen positive rate is about 10% and false positives are low, minimising overdiagnosis and over-referral. Additional advantages include its one-page format, ease of completion and availability free of charge.
Limitations of this study include parent report of a developmental diagnosis rather than a standardised clinician assessment. However, parent report of physician diagnosis of developmental disorders including ASD has been used extensively in national surveys such as the US National Survey of Children’s Health and the National Health Interview Survey.33–36 Kogan et al have summarised the evidence supporting the validity of parent report compared with physician diagnosis.33 In our study, maternal education and family income were high, which may limit the generalisability of our findings. However, family income was included as a covariate in our analysis and the overall prevalence of a positive ITC screen in our sample was close to the expected 10th percentile cut-off score.19
Conclusion
Developmental screening at 18 months may lead to early identification of communication impairments (including speech delay and other communication disorders such as ASD) for which interventions are available. This study provides evidence on the predictive validity of the ITC in primary care at the 18-month visit, suggesting that it performs as well as other currently available developmental screening tools.
Data availability statement
Data are available upon reasonable request. Data available upon reasonable request according to the TARGet Kids! data access policy.
Ethics statements
Patient consent for publication
Ethics approval
Consent was obtained from parents. Ethics approval was granted by the Hospital for Sick Children, Toronto Ethics Board (REB# 1000012436). The ITC was completed for research purposes; practitioners and parents were blind to the results.
Acknowledgments
We thank all participating children and families for their time and involvement in this study. We thank the TARGet Kids! Collaboration for supporting this study (details may be found on our website: www.targetkids.ca). The TARGet Kids! Collaboration is a primary care practice-based research network and includes practice site physicians, research staff, collaborating investigators, trainees, methodologists, biostatisticians, data management personnel, laboratory management personnel and advisory committee members.
References
Footnotes
Contributors CB and PP drafted the manuscript. CB and MA analysed the data. All authors contributed to the design of the study, edited the manuscript and approved the final submission. CB and PP contributed equally, are joint senior authors and act as guarantors for the paper.
Funding Funding to support TARGet Kids! was provided by multiple sources including the Canadian Institutes for Health Research (CIHR), namely the Institute of Human Development, Child and Youth Health and the Institute of Nutrition, Metabolism and Diabetes, as well as the St Michael's Hospital Foundation, Toronto. The Pediatric Outcomes Research Team is supported by a grant from the Hospital for Sick Children Foundation, Toronto.
Disclaimer Funding agencies had no role in the design, collection, analyses or interpretation of the results of this study or in the preparation, review or approval of the manuscript.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.