Consensus on patient cases for hospitalised children with a high paediatric track and trigger tool score that raises no mounting concern: a Delphi process study
================================================================================================================================================================

* Claus Sixtus Jensen
* Hanne Vebert Olesen
* Hans Kirkegaard
* Marianne Lisby

## Abstract

**Background** Paediatric track and trigger tools (PTTTs) based on vital parameters have been implemented in hospitals worldwide to help healthcare professionals identify signs of critical illness and incipient deterioration in hospitalised children. It has been documented that nurses do not use PTTT as intended, but deviate from PTTT protocols because, in some situations, PTTT observations make little sense to them. The present study aimed to reach consensus on whether automatically generated PTTT scores that are higher than deemed reasonable by healthcare professionals according to their professional experience and clinical expertise may be downgraded.

**Methods** A two-round modified Delphi technique was used to explore consensus on 14 patient cases for hospitalised children with a high PTTT score that did not raise concerns by systematically collating questionnaire responses. Participants rated their level of agreement on a 9-point Likert scale. IQR and median were calculated for each case.

**Findings** A total of 221 participants completed round 1 and 101 participants completed round 2. Across the two rounds, majority of the participants were from paediatric departments, nurses and women. In round 1, consensus on inclusion was reached on 2 of the 14 cases. In round 2, consensus was reached on one additional patient case. Three of the 11 non-consensus cases remaining after rounds 1 and 2 were included by the research group based on predefined criteria.

**Conclusion** In conclusion, a consensus opinion was achieved on six patient cases where the child had a high PTTT score but where the healthcare professionals were not as concerned as indicated by the PTTT score.

*   Nursing
*   Health services research

### WHAT IS ALREADY KNOWN ON THIS TOPIC

*   Nurses express that routine observations are often unnecessary in stable paediatric patients.

*   Nurses highlight the constant burden of repeated alerts relating to individual patients in situations requiring no clinical intervention.

#### WHAT THIS STUDY ADDS

*   The study identifies the most frequent patient cases in paediatric care for hospitalised children who have a high paediatric track and trigger tool (PTTT) score that, nevertheless, causes no mounting concern.

*   The study provides knowledge that may help reduce alarm fatigue and increase nurses’ perspective of the clinical relevance of PTTT.

#### HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

*   In children who are at risk of deterioration, we may possibly benefit from focusing on more aspects than just vital signs.

*   Giving nurses the opportunity to reduce the PTTT score based on predefined patient cases may potentially help reduce alarm fatigue and increase nurses’ use of PTTT.

## Background

In the past decade, greater focus has been placed on recognising and responding to hospitalised paediatric patients at risk of deterioration.1 It has been documented that a child’s deterioration often goes unrecognised and elicits no timely response, which heighten the risk of negative health outcomes and child deaths.2 3 For most children, acute illnesses develop gradually over many hours, with vital parameters gradually becoming more abnormal as deterioration progresses.4 Paediatric track and trigger tools (PTTTs) may be employed as a screening tool to help healthcare professionals identify signs of critical illness and incipient deterioration and facilitate a prompt and relevant response in hospitalised paediatric patients.5 6 When a child’s clinical condition is deteriorating, deviation from normal vital parameters will yield a high or increased score indicating that intervention may be required. Using PTTT is thus a systematic method for assessing several observations, primarily vital parameters. In other words, PTTTs quantify paediatric patients’ risk of clinical deterioration. Worldwide, a variety of PTTTs are available aiming to provide healthcare professionals with an aggregate PTTT score based on physiological parameters such as respiratory rate, respiratory effort, oxygen saturation, systolic blood pressure, pulse rate and level of consciousness.7 8 Some PTTTs have been validated, whereas others have been modified or developed at a local hospital with no subsequent validation.7 To the best of our knowledge, no consensus exists as to which PTTT is superior, and evidence to determine which PTTT may be superior is limited.7 9 Furthermore, recent literature reviews have questioned the evidence underpinning PTTTs’ effect on children’s outcomes.6 7 9

Studies on adult patients have documented that nurses often also use their intuition to recognise deteriorating patients.10–12 Benner *et al*13 defined intuition as ‘a judgment without a rationale, a direct apprehension and response without recourse to calculative rationality’. PTTTs do not distinguish between diagnoses or the individual child’s characteristics. Therefore, a risk exists of overlooking children who do not have a normal stress response and children with a habitual, chronically impaired physiology due to chronic disease. Children with expected abnormal vital parameters because of, for example, medical treatment are frequently observed unnecessarily, resulting in an increased workload. This is a documented reason for *not* making PTTT observations and not complying with PTTT protocols.14 PTTTs do not leave much room for individual clinical assessment, which has been reported to give nurses a negative perception of PTTT.14 Previously, we have documented how nurses found that PTTTs were an important tool, but they also described that they did not use them as intended because, in some situations, PTTT observations made little sense to them.14 Such situations could be those in which a child had a high PTTT score indicating that the child was at risk of clinical deterioration but where the nurses were not concerned to the point expected given the PTTT score.14 In addition, nurses found that measurement of vital parameters increased their workload without improving the care they provided.8 15 Furthermore, some nurses stated that they spent too much time measuring vital parameters and that routine observations were often unnecessary in stable paediatric patients.16 Thus, attention should be given to further the development of PTTTs. If not, the risk of failing to recognise children’s deteriorating conditions will remain and constitute a barrier to nurses’ use of PTTT.

### Aim

Our aim was to reach consensus on whether automatically generated PTTT scores that are higher than deemed reasonable by healthcare professionals according to their professional experience and clinical expertise may be downgraded.

## Methods

### Study design

We used a modified Delphi study technique to develop a consensus opinion among clinicians, both nurses and medical doctors.17 This method was chosen because it is an extensively used method for transforming individual opinions on a specific issue into a group consensus.18 A two-round modified Delphi approach was chosen. After each round, the patient case statements on which consensus was reached were included and statements on which consensus was not reached were carried through to the next round of questioning. Feedback on individual cases and the panel response were included in the second round to enhance consensus among the panel. This two-round method was used to enhance participation in the study by reducing responder fatigue.

### Development of patient cases

To develop the patient cases used in this study, we enlisted input from multidisciplinary stakeholders in paediatric care, including clinical nursing specialists, registered nurses, paediatric consultants, paediatric residents and nurse managers. We held a group meeting at five different hospitals covering four of Denmark’s five regions and included both university hospitals and regional hospitals. The aim of the study was outlined initially at each meeting, and the stakeholders were asked to brainstorm on any paediatric patient cases they thought would be relevant. The goal of this stage was to incorporate multidisciplinary perspectives into the development of the patient cases. The final version of the patient cases distributed in this study was prepared by CSJ and HVO.

### Delphi panel

The Delphi panel in this study included healthcare professionals who were active in clinical practice and had a minimum of 2 years of experience in the field of paediatrics from hospital settings in Denmark.

The participants were recruited from the following relevant professional national healthcare network and association: the Emergency Paediatric Network representing 73 healthcare professionals and the Danish Association for Nurses working with children and young people representing 400 nurses. Invitations to participate in the study were posted on the network’s and the association’s Facebook sites. Furthermore, we contacted the management of all 19 paediatric departments in Denmark and invited them to distribute the invitation to healthcare professionals who met the above criteria.

### The Delphi process

#### First round

The Delphi survey was distributed electronically using Research Electronic Data Capture (REDCap) hosted by Aarhus University, Denmark. In the first round, the participants were asked independently to rate 14 paediatric patient cases (online supplemental table 1). They were asked to rate their agreement with each of the 14 cases on a scale from 1 (completely disagree) to 9 (completely agree). For each statement, participants were given the opportunity to tick ‘Don’t know’ as an alternative answer. For each case, they were also given the opportunity to contribute free-text comments, including suggestions for rephrasing, elaborating or explaining their response. The participants were also given the opportunity to add other relevant cases. Demographic information such as years of experience in the paediatric field, workplace, gender and year of birth was also collected.

### Supplementary data

[[bmjpo-2022-001564supp001.pdf]](pending:yes)

#### Second round

In the second round, each participant received the Delphi survey with the non-consensus cases from round 1. For each case, the participants were presented with their own response and the panel response outlined as median and IQR, as well as any free-text comments given by the participants (see table 1 for an example). Any new cases being suggested by more than one participant from the first round were also presented and rated in round 2. No option was provided to add new cases in round 2.

View this table:
[Table 1](http://bmjpaedsopen.bmj.com/content/6/1/e001564/T1)

Table 1 
Example of a patient case presented in round 2 including the panel’s score and a reminder of the participant’s score along with any free-text comments

### Analysis

Descriptive statistics were used to analyse participants’ demographic characteristics. IQR and median were calculated for each patient case. A consensus for inclusion was defined by a median score and IQR of 7–9; for exclusion, a consensus was defined by a median score and IQR of 1–3.17 19 20 Non-consensus was defined as all other scores yielding reassessment in round 2, for example, a score and IQR falling within the 4–6 range or an IQR crossing the 4–6 range.

Any non-consensus cases remaining after round 2 were handled in accordance with predefined criteria21 (table 2). In order to ensure that the response being analysed was answered only by clinicians who felt that they possessed the knowledge required to rate the cases, all ‘don’t know’ responses were excluded from the group response.

View this table:
[Table 2](http://bmjpaedsopen.bmj.com/content/6/1/e001564/T2)

Table 2 
Criteria for inclusion or exclusion of non-consensus patient cases

### Patient and public involvement

Patients or the public were not involved in the design, or conduct, or reporting or dissemination plans of our research.

## Results

### Demographics of panel members and response rate

A total of 221 participants completed round 1 and 101 completed round 2, corresponding to 54%. Table 3 presents the participants’ demographic characteristics. Across the two rounds, majority of the participants were from paediatric departments, nurses and women.

View this table:
[Table 3](http://bmjpaedsopen.bmj.com/content/6/1/e001564/T3)

Table 3 
Demographic characteristic of Delphi participants

#### First round

In round 1, consensus was reached on inclusion of 2 of the 14 described cases (Table 4). The proportion of participants who reported ‘don’t know’ to each case ranged from 1.4% to 47%, with the cases regarding oncology patients and those with anorexia and diabetes having the highest number of ‘don’t know’ answers. No cases were excluded in the first round. No new cases were suggested by the participants in round 1. Suggestions were made to implement a minor change in the wording of seven of the described cases (online supplemental table 1).

View this table:
[Table 4](http://bmjpaedsopen.bmj.com/content/6/1/e001564/T4)

Table 4 
Included and excluded patient cases in the Delphi process

#### Second round

Consensus was reached on inclusion of one additional case following round 2 (table 4). The three cases that had a high number of ‘don’t know’ answers in round 1 remained the same in round 2. The proportion of ‘don’t know’ answers in each case ranged from 0% to 50.5%. Based on the response from round 1, some minor changes were made to the wording in some of the cases prior to distribution of the second round of the survey.

In 11 out of the 14 cases, consensus on inclusion was not reached in rounds 1 and 2. An additional three cases were included and eight were excluded according to the predefined criteria outlined in table 2.

## Discussion

In the present study, consensus was achieved on 3 of the 14 patient cases following the two rounds, and an additional three cases were included according to the predefined criteria for inclusion and exclusion.

The qualitative feedback option to each patient case demonstrated that management of different patient cases will differ widely depending on the child’s clinical presentation, the nurse’s or the medical doctor’s experience, individual practice, and variation in PTTT guidelines. Statements like “They [the child] are difficult to assess” and “depending on the situation, knowledge of the patient and the nurses’ competences” were often provided. The differences between the opinions of the non-consensus patient cases were echoed in the wide range of median scores in the responses, which ranged from 1 (strongly disagree) to 8 (agree). This complexity was also highlighted in a study by Lillitos *et al*,22 who aimed to establish a benchmark list defining significant acute paediatric conditions that warrant acute hospital admission from the emergency department, using a Delphi method. They failed to achieve consensus on 37 statements because assessment of sick children’s conditions may vary and depend on many factors, such as the healthcare professional’s experience and the child’s clinical presentation. In the study by Lillitos *et al*,22 the authors identified many neutral answers in round 1 and therefore added the response category ‘I don’t look after children with this condition’. We also identified many patient cases in which the healthcare professionals did not have sufficient experience to answer the question, which underscores the complexity of paediatrics.

As implementation of PTTTs has a considerable impact on both children and resources, it is essential to keep refining and developing PTTTs to establish the best way to identify children who are at risk of clinical deterioration and who would benefit from early intervention. Studies have shown that PTTT implementation has been unsuccessful; healthcare professionals often experience alarm fatigue and PTTTs are therefore often ignored by front-line nursing staff,23 and nurses do not always follow PTTT protocols and medical doctors may be unaware of their role.14 Bedoya *et al*23 documented how nurses highlighted the constant burden of repeated alerts on individual patients in situations requiring no clinical intervention. In a previously published study, we showed that nurses described low PTTT scores in children who did not have a condition that warranted an intervention.14 This discordance led to alert fatigue and a general mistrust in PTTT.14 23 Children at risk of deterioration may therefore benefit from focusing on more aspects than merely vital signs such as respiratory and heart rate. Furthermore, children may present different patterns of illness as the natural history of inpatient paediatric conditions is not uniform in how they progress. We achieved consensus on inclusion of 6 of 14 patient cases. Some of the included patient cases occur frequently in paediatric care, for example, the agitated child; crying when staff are present but observed as calm and ‘happy’ when staff are not in the room; and children with asthmatic bronchitis who responded to treatment with β2-agonists equivalent to adequate saturation but who maintain a high pulse rate. It may possibly have a positive impact on clinical practice if these case types were included in a modified PTTT. Our study provided insight into one aspect of the further development of PTTT. Giving nurses the opportunity to reduce the PTTT score in accordance with the clinical particulars of predefined patient cases may potentially help to reduce alarm fatigue and enhance nurses’ perception of the clinical relevance of PTTT.

### Limitations

This Delphi study was conducted in Denmark. The results may therefore not be applicable to other healthcare settings where clinical practice, cultural attitudes and available resources may be different. To reduce the introduction of persuasive bias that may arise from dominant individuals, the study was deliberately anonymised. If a Delphi process appears too time-consuming or complex, the invited experts may not join or might drop out during the survey. We conducted a modified two-round survey to reduce the risk of dropout; even so, we did experience considerable dropout.

## Conclusion

In conclusion, consensus was achieved on 6 of 14 paediatric patient cases with a high PTTT score in which healthcare professionals were not concerned to the extent indicated by the PTTT score and in which nurses could downgrade the PTTT score with a predefined number. These patient cases may be used for future research into PTTT, and if incorporated into a modified PTTT these cases may also help reduce alarm fatigue and enhance nurses’ perception of the usefulness of PTTT.

## Data availability statement

Data are available upon reasonable request.

## Ethics statements

### Patient consent for publication

Not required.

### Ethics approval

The study was registered with the Danish Data Protection Agency (J-1-16-02-61-20). The Danish National Committee on Health Research Ethics waived ethical approval with reference to Danish law (J. 1-10-72-1-20).

## Acknowledgments

The authors take this opportunity to express their gratitude to the participating healthcare professionals.

## Footnotes

*   Contributors CSJ: conceptualisation, data curation, formal analysis, funding acquisition, investigation, methodology, project administration, original draft, writing - review and editing. HVO: conceptualisation, data curation, formal analysis, review and editing. HK: conceptualisation, supervision, review and editing. ML: conceptualisation, formal analysis, supervision, review and editing. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work. CSJ is responsible for the overall content.

*   Funding This study was supported by the Novo Nordisk Foundation (NNF180C0052020), Lundbeckfonden and the research programme 'The Acute Patient'.

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  Tume L. The deterioration of children in ward areas in a specialist children's Hospital. Nurs Crit Care 2007;12:12–19.[doi:10.1111/j.1478-5153.2006.00195.x](http://dx.doi.org/10.1111/j.1478-5153.2006.00195.x)pmid:http://www.ncbi.nlm.nih.gov/pubmed/17883659
    
    [CrossRef](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=10.1111/j.1478-5153.2006.00195.x&link_type=DOI) 
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=17883659&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

2.  Jensen CS, Kirkegaard H, Aagaard H, et al. Clinical profile of children experiencing in-hospital clinical deterioration requiring transfer to a higher level of care. J Child Health Care 2019;23:522–33.[doi:10.1177/1367493518794400](http://dx.doi.org/10.1177/1367493518794400)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30124066
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

3.  Jensen CS, Olesen HV, Aagaard H, et al. Comparison of two pediatric early warning systems: a randomized trial. J Pediatr Nurs 2019;44:e58–65.[doi:10.1016/j.pedn.2018.11.001](http://dx.doi.org/10.1016/j.pedn.2018.11.001)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30471827
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

4.  McLellan MC, Gauvreau K, Connor JA. Validation of the children's hospital early warning system for critical deterioration recognition. J Pediatr Nurs 2017;32:52–8.[doi:10.1016/j.pedn.2016.10.005](http://dx.doi.org/10.1016/j.pedn.2016.10.005)pmid:http://www.ncbi.nlm.nih.gov/pubmed/27823915
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=27823915&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

5.  Chapman SM, Wray J, Oulton K, et al. Systematic review of paediatric track and trigger systems for hospitalised children. Resuscitation 2016;109:87–109.[doi:10.1016/j.resuscitation.2016.07.230](http://dx.doi.org/10.1016/j.resuscitation.2016.07.230)pmid:http://www.ncbi.nlm.nih.gov/pubmed/27496259
    
    [CrossRef](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=10.1016/j.resuscitation.2016.07.230&link_type=DOI) 
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=27496259&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

6.  Trubey R, Huang C, Lugg-Widger FV, et al. Validity and effectiveness of paediatric early warning systems and track and trigger tools for identifying and reducing clinical deterioration in hospitalised children: a systematic review. BMJ Open 2019;9:e022105.[doi:10.1136/bmjopen-2018-022105](http://dx.doi.org/10.1136/bmjopen-2018-022105)pmid:http://www.ncbi.nlm.nih.gov/pubmed/31061010
    
    [Abstract/FREE Full Text](http://bmjpaedsopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOS81L2UwMjIxMDUiO3M6NDoiYXRvbSI7czoyMzoiL2JtanBvLzYvMS9lMDAxNTY0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

7.  Lambert V, Matthews A, MacDonell R, et al. Paediatric early warning systems for detecting and responding to clinical deterioration in children: a systematic review. BMJ Open 2017;7:e014497.[doi:10.1136/bmjopen-2016-014497](http://dx.doi.org/10.1136/bmjopen-2016-014497)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28289051
    
    [Abstract/FREE Full Text](http://bmjpaedsopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiNy8zL2UwMTQ0OTciO3M6NDoiYXRvbSI7czoyMzoiL2JtanBvLzYvMS9lMDAxNTY0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

8.  Ball S, Parkinson S, Marjanovic S, RAND Corporation. Paediatric early warning systems: a scoping study lessons from a rapid review. Cambridge, 2021.
    
    

9.  Chapman SM, Wray J, Oulton K, et al. 'The score matters': wide variations in predictive performance of 18 paediatric track and trigger systems. Arch Dis Child 2017;102:487–95.[doi:10.1136/archdischild-2016-311088](http://dx.doi.org/10.1136/archdischild-2016-311088)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28292743
    
    [Abstract/FREE Full Text](http://bmjpaedsopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTI6ImFyY2hkaXNjaGlsZCI7czo1OiJyZXNpZCI7czo5OiIxMDIvNi80ODciO3M6NDoiYXRvbSI7czoyMzoiL2JtanBvLzYvMS9lMDAxNTY0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. Douw G, Huisman-de Waal G, van Zanten ARH, et al. Nurses' 'worry' as predictor of deteriorating surgical ward patients: a prospective cohort study of the dutch-early-nurse-worry-indicator-score. Int J Nurs Stud 2016;59:134–40.[doi:10.1016/j.ijnurstu.2016.04.006](http://dx.doi.org/10.1016/j.ijnurstu.2016.04.006)pmid:http://www.ncbi.nlm.nih.gov/pubmed/27222458
    
    [CrossRef](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=10.1016/j.ijnurstu.2016.04.006&link_type=DOI) 
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=27222458&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

11. Douw G, Schoonhoven L, Holwerda T, et al. Nurses' worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review. Crit Care 2015;19:230.[doi:10.1186/s13054-015-0950-5](http://dx.doi.org/10.1186/s13054-015-0950-5)pmid:http://www.ncbi.nlm.nih.gov/pubmed/25990249
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

12. Odell M, Victor C, Oliver D. Nurses' role in detecting deterioration in ward patients: systematic literature review. J Adv Nurs 2009;65:1992–2006.[doi:10.1111/j.1365-2648.2009.05109.x](http://dx.doi.org/10.1111/j.1365-2648.2009.05109.x)pmid:http://www.ncbi.nlm.nih.gov/pubmed/20568317
    
    [CrossRef](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2648.2009.05109.x&link_type=DOI) 
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=20568317&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

13. Benner P, Tanner CA, Chesla AC. Expertise in Nursing practice Caring, Clinical Judgment & Ethics. 2. Springer, 2009: 199–232.
    
    

14. Jensen CS, Nielsen PB, Olesen HV, et al. Pediatric early warning score systems, nurses perspective - a focus group study. J Pediatr Nurs 2018;41:e16–22.[doi:10.1016/j.pedn.2018.02.004](http://dx.doi.org/10.1016/j.pedn.2018.02.004)pmid:http://www.ncbi.nlm.nih.gov/pubmed/29454507
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

15. Dall'Ora C, Griffiths P, Redfern O, et al. Nurses' 12-hour shifts and missed or delayed vital signs observations on hospital wards: retrospective observational study. BMJ Open 2019;9:e024778.[doi:10.1136/bmjopen-2018-024778](http://dx.doi.org/10.1136/bmjopen-2018-024778)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30782743
    
    [Abstract/FREE Full Text](http://bmjpaedsopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOS8xL2UwMjQ3NzgiO3M6NDoiYXRvbSI7czoyMzoiL2JtanBvLzYvMS9lMDAxNTY0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

16. Dall'Ora C, Griffiths P, Hope J, et al. How long do nursing staff take to measure and record patients' vital signs observations in hospital? time-and-motion study. Int J Nurs Stud 2021;118:103921.[doi:10.1016/j.ijnurstu.2021.103921](http://dx.doi.org/10.1016/j.ijnurstu.2021.103921)pmid:http://www.ncbi.nlm.nih.gov/pubmed/33812297
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

17. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376–80.[doi:10.1136/bmj.311.7001.376](http://dx.doi.org/10.1136/bmj.311.7001.376)pmid:http://www.ncbi.nlm.nih.gov/pubmed/7640549
    
    [FREE Full Text](http://bmjpaedsopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTEvNzAwMS8zNzYiO3M6NDoiYXRvbSI7czoyMzoiL2JtanBvLzYvMS9lMDAxNTY0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

18. Powell C. The delphi technique: myths and realities. J Adv Nurs 2003;41:376–82.[doi:10.1046/j.1365-2648.2003.02537.x](http://dx.doi.org/10.1046/j.1365-2648.2003.02537.x)pmid:http://www.ncbi.nlm.nih.gov/pubmed/12581103
    
    [CrossRef](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=10.1046/j.1365-2648.2003.02537.x&link_type=DOI) 
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=12581103&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 
    
    [Web of Science](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=000180928000014&link_type=ISI) 

19. Boier Tygesen G, Kirkegaard H, Raaber N, et al. Consensus on predictors of clinical deterioration in emergency departments: a Delphi process study. Acta Anaesthesiol Scand 2021;65:266–75.[doi:10.1111/aas.13709](http://dx.doi.org/10.1111/aas.13709)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32941660
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

20. Dean B, Barber N, Schachter M. What is a prescribing error? Qual Health Care 2000;9:232–7.[doi:10.1136/qhc.9.4.232](http://dx.doi.org/10.1136/qhc.9.4.232)pmid:http://www.ncbi.nlm.nih.gov/pubmed/11101708
    
    [Abstract/FREE Full Text](http://bmjpaedsopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjkvNC8yMzIiO3M6NDoiYXRvbSI7czoyMzoiL2JtanBvLzYvMS9lMDAxNTY0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

21. Lisby M, Nielsen LP, Brock B, et al. How should medication errors be defined? development and test of a definition. Scand J Public Health 2012;40:203–10.[doi:10.1177/1403494811435489](http://dx.doi.org/10.1177/1403494811435489)pmid:http://www.ncbi.nlm.nih.gov/pubmed/22333835
    
    [CrossRef](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=10.1177/1403494811435489&link_type=DOI) 
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=22333835&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

22. Lillitos PJ, Lyttle MD, Roland D, et al. Defining significant childhood illness and injury in the emergency department: a consensus of UK and ireland expert opinion. Emerg Med J 2018;35:emermed-2018-207802[doi:10.1136/emermed-2018-207802](http://dx.doi.org/10.1136/emermed-2018-207802)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30282629
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom) 

23. Bedoya AD, Clement ME, Phelan M, et al. Minimal impact of implemented early warning score and best practice alert for patient deterioration. Crit Care Med 2019;47:49–55.[doi:10.1097/CCM.0000000000003439](http://dx.doi.org/10.1097/CCM.0000000000003439)pmid:http://www.ncbi.nlm.nih.gov/pubmed/30247239
    
    [CrossRef](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=10.1097/CCM.0000000000003439&link_type=DOI) 
    
    [PubMed](http://bmjpaedsopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjpo%2F6%2F1%2Fe001564.atom)