Objectives To identify and prioritise technical procedures that should be integrated in a curriculum of simulation-based procedural training in paediatrics using the Delphi method.
Study design National general needs assessment using a Delphi process was completed among 93 key opinion leaders in paediatrics in Denmark. Delphi round 1 identified technical procedures. Round 2 explored frequency of procedures, number of paediatricians performing the procedures, risks and/or discomfort for patients and feasibility for simulation-based training. Round 3 included final elimination and reprioritisation.
Results Response rates in the Delphi rounds were 73%, 71% and 72%. We identified 37 procedures in Delphi round 1, preprioritised in round 2, resulting in a final list of 19 procedures in round 3. Strong correlation between the prioritisation from the second and third Delphi rounds was identified, Spearman’s r of 0.94 (p<0.0001). Top five on the final list were acute neonatal airway management, acute non-neonatal airway management, non-neonatal peripheral intravenous and intraosseous access, neonatal vascular access and advanced heart lung resuscitation.
Conclusion We identified and prioritised 19 technical procedures in paediatrics that are suitable for simulation and may be used as a guide for the development of simulation-based curriculum in paediatrics.
- clinical procedures
- general paediatrics
- medical education
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
What is known about the subject?
Paediatricians are expected to possess the skills of managing acutely ill children in a broad specialty in high-stakes time sensitive acute situations.
Rarity of acute events limits exposure and it is impractical and unethical to rely entirely on apprenticeship and clinical interactions to develop the skills required.
Simulation-based training is often driven by opinion of local educators, available simulators or coincidence. Development of a curriculum should follow a structured and systematic process.
What this study adds?
A needs assessment using a Delphi process to identify and prioritise technical procedures that are suitable for simulation in paediatrics.
The procedures represent core competencies in paediatric specialist training that should be practised in a simulation-based environment before applied on patients.
Educators may use this list to develop, plan and implement simulation-based training curricula for paediatric trainees.
Paediatricians are expected to possess the skills of managing acutely ill children. However, the volume of acute events in paediatric emergency departments often limits exposure to many procedures. In addition, paediatrics is a broad specialty considering the broad range of patient age, different subspecialties and rare clinical conditions. Consequently, it is both impractical and unethical to rely entirely on apprenticeship and clinical interactions to develop and master the skills required to manage the treatment of acutely ill children.
Current paediatric training in Denmark includes work-based, supervised and apprenticeship learning blended with elements of simulation-based training. Simulation-based training is often driven by opinion of local educators, available simulators or coincidence,1 which has several limitations such as limited training time due to work-hour restrictions, patient safety considerations and training programmes that are not necessarily aligned with the current needs.1–3 Simulation-based training of procedures in paediatrics has been shown to improve the acquisition of skills,4–8 promote patient safety and reduce errors.2 9 Nonetheless simulation-based training is a complex and resource-demanding educational intervention.10
The development of a simulation-based procedural training curriculum should follow a structured and systematic process, for example, as described by Kern11 in a six‐step approach: (1) Problem identification and general needs assessment, (2) Targeted needs assessment, (3) Goals and objectives, (4) Educational strategies, (5) Implementation and (6) Evaluation and feedback. Unfortunately, the first step is often disregarded or is decided based on availability of simulators or local interests.12 There is a need to bridge the gap between this unstructured way of developing a simulation-based procedural training curriculum and a needs-driven approach. The results may aid the development of simulation-based training programmes as part of residency curricula in paediatrics.
The objective of this study was to perform a national general needs assessment in paediatrics using a Delphi process to identify and prioritise technical procedures that should be integrated in a simulation-based training curriculum in paediatrics.
Materials and methods
The study consisted of a three-round Delphi process to establish consensus and prioritise technical procedures for simulation-based training (figure 1). The Delphi technique is widely accepted as a valid tool to obtain expert opinion, obtain consensus and support decision making.13–15 During each Delphi round, the preceding results are presented to the participants in an anonymous manner to re-evaluate and explore. The Delphi technique allows involvement of participants from across different locations. This means that they do not have to meet in person, that anonymity is maintained, which is key to the Delphi process and avoids strong characters dominating the process. In this modified Delphi, we had planned on the number of rounds a priori but were open to another round if consensus was not achieved. In the most cases, consensus is reached after three rounds,13 and therefore, we decided on three rounds. The results from previous rounds were fed to the next round for review and exploration. This method has been used to develop curricula for simulation-based training in various specialties.16–22 The study was conducted from December 2017 to September 2018.
The participants were key opinion leaders in paediatric education and were identified based on their roles in the specialist training and education of paediatric trainees across Denmark. The study was nationwide and all paediatricians who are involved in residency training and education were invited. These include postgraduate clinical associate professors, heads of clinical education and training of all paediatric departments, heads of paediatric departments, conveners of mandatory courses for paediatric specialisation, professors, members of educational committees, board members of The Danish Society for Paediatrics and board members of The Danish Young Paediatricians Association. Participants were invited individually and introduced to the study via emails. The second and third round were administered to all the participants regardless if they responded or not in first and second round. Participation was voluntary.
Patient and public involvement
No patients participated and there was no public involvement in the study.
The Delphi process was conducted through online survey questionnaires using survey software (Survey Monkey, San Mateo, California, USA) and were distributed by email. Reminder emails were sent up to three times. A multidisciplinary research steering group was formed to facilitate data collection and analysis. It consisted of five members, all investigators and coauthors of the study: two paediatric consultants and postgraduate clinical associate professors (TB and NTH), head of research at a simulation centre (CP), an assistant professor (RDJ) and a paediatric resident, who is also a member of the educational committee in paediatrics and a board member of The Danish Society for Paediatrics (ST). Anonymity of responses was maintained. The primary investigator (ST) was responsible for data organisation, ensuring that the answers were blinded from the rest of the steering group. All suggestions from the participants were considered and were only eliminated when consensus was not reached
Delphi round 1: brainstorming phase
The Delphi process started with a brainstorming phase, where the participants were asked to suggest all technical procedures which a paediatrician who had just completed specialist training in paediatrics should be able to perform. All suggested procedures were reviewed by the research steering group and non-technical skills were removed (eg, team training and communication skills). The remaining technical procedures were consolidated into categories where similar procedures were grouped. This consolidated list of technical procedures was used in Delphi round 2.
Delphi round 2: survey
We used a previously developed needs assessment formula (NAF) from the Copenhagen Academy for Medical Education and Simulation (CAMES) to prioritise the included procedures from the first Delphi round (1, 20). This formula explores the need for simulation-based training of a given technical procedure by investigating four different factors.
Frequency; the number of procedures performed annually.
Operators; number of physicians that should be able to perform the procedure.
Impact; discomfort/risk if the procedure is performed by an inexperienced physician.
Feasibility; suitability for learning the procedure in simulation-based environment.
Each technical procedure/procedure group from Delphi round 1 was evaluated according to these four factors. Frequency, operators and impact were stated as multiple‐choice items on a five-point rating scale ranging from 1 to 5. The fourth element of the formula (feasibility for simulation-based training) was challenging for the participants to estimate and was therefore evaluated by the steering group. Feasibility was calculated as the mean score of three equally weighted factors: suitability for simulation-based training, equipment availability and associated costs. We used the means and not medians for ranking the procedures because using the medians would create many ties in the ranking. The steering group explored feasibility of each technical procedure from Delphi 1 using a five-point Likert scale ranging from 1 to 5.
The CAMES NAF score for the individual technical procedure was the average of the mean scores (1–5) of the four factors (frequency, physicians, impact and feasibility) giving each factor equal weighting of 25%. Alternative weighting was considered but decided against because arguing for and quantitating alternative unequal weights was difficult, that is, is it more important to practice a high-frequency low risk procedure or a very rare high-risk procedure. The resulting total CAMES NAF score from 1 to 5 points determined the procedures’ ranking on the preliminary prioritised list used in the third Delphi round.
Delphi round 3: elimination and prioritisation
The preliminary prioritised list of technical procedures from Delphi round 2 was sent to the participants for re-evaluation. The participants were asked to eliminate technical procedures that they found unsuitable for learning in a simulation-based environment, simple technical procedures that should have been learnt in pregraduate medical education or very advanced technical procedures that are learnt in the later part of specialist training. The participants also had the opportunity to reprioritise the remaining procedures.
Data analysis and statistics
In Delphi round 1, we eliminated non-technical skills and grouped similar procedures into categories. We grouped them together when the procedures could be combined and trained in one training programme. In Delphi round 2, the mean score for each factor was calculated and the CAMES NAF score was used to produce a preliminary ranked list of technical procedures used in Delphi round 3. In Delphi round 3, we calculated the number of occurrences for each response using frequency analysis and applied a degree of agreement of ≥70% to establish consensus. The procedures that had <70% support from the participants were eliminated from the final list. Spearman’s r was calculated to explore the alignment between prioritisation according to the CAMES NAF (Delphi round 2) and the final decisions of the participants (Delphi round 3). All analyses were performed using STATA/IC V.15.1 (StataCorp).
Results of Delphi round 1
A total of 93 participants were identified and invited to the study. Round 1 of the Delphi process had a response rate of 73% (68/93) and 68 clinical procedures were suggested in the brainstorming process. These were reduced to 37 procedures/procedure groups.
Results of Delphi round 2
The response rate in Delphi round 2 was 71% (66/93). A preprioritised list of technical procedures was generated using the CAMES NAF score. In this Delphi round, we identified acute airway management of neonates as highest priority and biopsy of skin, tendon and muscles as least priority (table 1).
Results of Delphi round 3
The response rate in Delphi round 3 was 72% (67/93). Eighteen technical procedures were eliminated. The final list included 19 technical procedures (table 2).
A Spearman’s r of 0.94 (p<0.0001) showed a strong correlation between the prioritisation order of the technical procedures from Delphi round 2 and the final list in Delphi round 3 (figure 2).
We performed a general needs assessment to identify and prioritise technical procedures that should be integrated in a curriculum of simulation-based procedural training in paediatrics. A prioritised list of 19 technical procedure was produced using a three‐round Delphi process among key opinion leaders in education in paediatric. All paediatric departments involved in the specialist training programme were represented, and the response rates were consistently high in all three rounds, indicating strong support from the participants and increasing the credibility of our results.
Airway management of both neonatal and non-neonatal topped the final list with highest priority. Airway management comprises several hands-on procedures and some of the airway management procedures are performed infrequently, making it relevant for simulation-based procedure training. Improvement in airway management skills through simulation-based training among paediatric residents has been shown in several studies.5 23 24 There is a growing body of evidence finding that a simulation-based airway management curriculum is superior to no intervention and to non-simulation intervention for education outcomes.25
Non-neonatal peripheral intravenous and intraosseous access and neonatal vascular access ranked third and fourth, respectively, in the final prioritised list. Neonatal intraosseous access ranked lower in the list, probably because it is rarely performed due to umbilical vein and artery availability. Nevertheless, it has been shown that intraosseous access can be performed more quickly in neonates than umbilical vein catherisation, especially for paediatricians who do not routinely place umbilical vein catheters.26 A low-cost model with real umbilical cord has been described to provide a more realistic training model than currently available commercial simulators.27
Advanced heart lung resuscitation was the fifth procedure on the final prioritised list. Advanced heart lung resuscitation comprises several hands-on procedures in form of cardiopulmonary resuscitation with rescue breathing and chest compressions as well as airway management and defibrillation.28
The top five procedures in the final prioritised list are all part of the Airway, Breathing, Circulation, Disability, Exposure approach of paediatric advanced life support and resuscitation.29 These procedures represent core competencies in paediatric specialist training. The skills are required in high-stakes time sensitive acute settings, and thereby emphasises the need for simulation-based training of these procedures.
The strong correlation between the results from Delphi rounds 2 and 3 indicates minimal changes in the ranking order of the procedures (figure 2). However, a few interesting changes were observed, for example, the change in rank for lumbar puncture and intubation which were both in top five on the preprioritised list from round 2. Lumbar puncture rated second in Delphi round 2 and changed to a sixth placement in Delphi round 3, most likely because lumbar puncture is a less acute procedure compared with airway management, vascular access and advanced heart lung resuscitation which ended up in top five. Intubation ranked fifth after the second Delphi round, but was eliminated in the final round. This is probably because the participants were asked to eliminate the very advanced technical procedures that are not expected for a newly specialised trainee to perform.
Ultrasonography only appeared as no. 17 and 18 in the final list in contrast to several ultrasound related procedures which were found in Delphi round 2. We assume that they were regarded as specialised procedures that only some paediatricians should master. In a recent study on general needs assessment in radiology, the top five procedures included ultrasonography.21 In addition, ultrasonography is listed among the top-10 procedures in cardiology,16 urology,19 pulmonology20 and anaesthesia.22 The relatively low ranking of ultrasonography in our study compared with that seen in other specialties may indicate that experience with ultrasonography among the paediatric participants is relatively low at this point. We may expect a stronger need for training of ultrasonography in paediatrics in years to come as in international proposals.30
The combination of CAMES NAF score and a three-round Delphi not only identified but also prioritised the technical procedures for simulation-based training. CAMES NAF score prioritises procedures if they are frequent, performed by many physicians, potentially harmful to patients if performed by a non‐competent physician, and very feasible to train simulation based.1 The participants were free to remove and reprioritise the procedures from the second round if they did not agree with the prioritisation resulting from the CAMES NAF score. However, we found a strong correlation between the ranking order after the second and third Delphi rounds, indicating that the participants widely agreed with the results based on CAMES NAF score in the second Delphi round. The same strong correlation has been found in previous needs assessment studies in other specialties.18 20
It is a strength of this study that all departments involved in the paediatric specialist training programmes in Denmark were represented, indicating strong support from the departments and increasing the generalisability and credibility of the study results. Furthermore, this study used a well-known method for curriculum development12 and followed international methodological criteria for reporting of Delphi studies.14 Additionally, a strength of the Delphi process is that surveys are sent electronically which reduces the risk of participants influencing each other and allows participants across different geographical locations to participate.
We are aware of the limitations in this study. The disease panorama is different from country to country, influencing the needs and some of the procedures on the list may not be transferable to other countries. However, we believe that the generalisability of this study is quite high, particularly because the top five procedures, on our final prioritised list represents core procedures in acute situations and therefore may be transferable. In perspective, it would be interesting to perform the need assessment in other countries and compare results. Another limitation is the three-round survey design, which takes time for the participants to respond to, and therefore, may have caused some potential participants to decline participation. The response rate in this study varied in the three rounds, with the lowest response rate in Delphi round 2. Overall, we had high response rates as seen in similar studies.16–22 In relation to skills in ultrasound, we have noted above that a change in experience and practice might change ranking of ultrasound skills over time. This may also relate to other skills, both those included and not included in the current report, including views on what is feasible in paediatric skill training may thus change over time. As such, the current results may reflect what is perceived as skills needed today and not skills needed in the future. Educating the future’s clinicians, it is important to bear future needs in mind. Asking the respondents to focus on this aspect may have shifted their responses and priorities somewhat, however, the acute skills rated at the top are unlikely to change markedly over time.
It is important to note that only technical skills for simulation-based procedural training were asked for and identified in this study. Factors that impact training such as communication, leadership and other non-technical skills should be incorporated when developing more team-based simulation-based programmes. Finally, it is important to acknowledge that we only explored which procedures to train in a simulation-based environment. Exploration of other important factors in curriculum development such as volume of training and how to implement training is outside the scope of the current study.
Implementing simulation-based training is associated with economic costs related to doctors spending working hours away from patients and sometimes to training facilities and equipment. The costs involved in the development and implementation of simulation-based training remains incomplete.31 However, the gains related to better training in human and economic terms should also be considered. The cost-effectiveness of simulation-based training in paediatrics according to the procedures identified in this study remains to be studied.
In conclusion, this needs assessment identified and prioritised a list of clinically relevant technical procedures suitable for simulation-based training in paediatrics. The procedures represent core competencies in paediatric specialist training that should be practiced in a simulation-based environment before applied on patients. Educators may use this list to develop, plan and implement simulation-based training curricula for paediatric trainees.
Contributors ST: study design, data collection, data analysis, figures, data interpretation, writing. LJN: study design, data interpretation, writing. CP: study design, data analysis, data interpretation, writing. RDJ: study design, data analysis, data interpretation, writing. LK: study design, data interpretation, writing. NTH: study design, data analysis, data interpretation, writing. TB: study design, data analysis, data interpretation, writing.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as online supplementary information. All data relevant to the study are included in the article.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.