You are here

  • Home
  • Archive
  • Volume 6, Issue 1
  • Prediction models for SIRS, sepsis and associated organ dysfunctions in paediatric intensive care: study protocol for a diagnostic test accuracy study

Article Text

Article menu
Protocol
Prediction models for SIRS, sepsis and associated organ dysfunctions in paediatric intensive care: study protocol for a diagnostic test accuracy study
  1. Julia Böhnke1,
  2. Nicole Rübsamen1,
  3. Marcel Mast2,
  4. Henning Rathert3,
  5. ELISE Study Group,
  6. André Karch1,
  7. Thomas Jack3,
  8. Antje Wulff2,4
    1. 1Institute of Epidemiology and Social Medicine, University of Münster, Münster, Germany
    2. 2Peter L. Reichertz Institute for Medical Informatics, TU Braunschweig and Hannover Medical School, Hannover, Germany
    3. 3Department of Pediatric Cardiology and Intensive Care Medicine, Hannover Medical School, Hannover, Germany
    4. 4Big Data in Medicine, Department of Health Services Research, Carl von Ossietzky University of Oldenburg, Oldenburg, Germany
    1. Correspondence to Julia Böhnke; boehnkej{at}uni-muenster.de

    Abstract

    Introduction Systemic inflammatory response syndrome (SIRS), sepsis and associated organ dysfunctions are life-threating conditions occurring at paediatric intensive care units (PICUs). Early recognition and treatment within the first hours of onset are critical. However, time pressure, lack of personnel resources, and the need for complex age-dependent diagnoses impede an accurate and timely diagnosis by PICU physicians. Data-driven prediction models integrated in clinical decision support systems (CDSS) could facilitate early recognition of disease onset.

    Objectives To estimate the sensitivity and specificity of previously developed prediction models (index tests) for the detection of SIRS, sepsis and associated organ dysfunctions in critically ill children up to 12 hours before reference standard diagnosis is possible.

    Methods and analysis We conduct a monocentre, prospective diagnostic test accuracy study. Clinicians in the PICU of the tertiary care centre Hannover Medical School, Germany, continuously screen and recruit patients until the adaptive sample size (originally intended sample size of 500 patients) is enrolled. Eligible are children (0–17 years, all sexes) who stay in the PICU for ≥12 hours and for whom an informed consent is given. All eligible patients are independently assessed for SIRS, sepsis and organ dysfunctions using corresponding predictive and knowledge-based CDSS models. The knowledge-based CDSS models serve as imperfect reference standards. The assessments are used to estimate the sensitivities and specificities of each predictive model using a clustered nonparametric approach (main analysis). Subgroup analyses (‘age groups’, ‘sex’ and ‘age groups by sex’) are predefined.

    Ethics and dissemination This study obtained ethics approval from the Hannover Medical School Ethics Committee (No. 10188_BO_SK_2022). Results will be disseminated as peer-reviewed publications, at scientific conferences, and to patients in an appropriate dissemination approach.

    Trial registration number This study was registered with the German Clinical Trial Register (DRKS00029071) on 2022-05-23.

    Protocol version 10188_BO_SK_2022_V.2.0–20220330_4_Studienprotokoll.

    • Adolescent Health
    • Information Technology
    • Technology

    Data availability statement

    The datasets generated and/or analysed during the current study are not publicly available due to data privacy and security matters of patients but are available from the corresponding author on reasonable request16.

    http://creativecommons.org/licenses/by-nc/4.0/

    This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

    https://doi.org/10.1136/bmjpo-2022-001618

    Statistics from Altmetric.com

      Request Permissions

      If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

      WHAT IS ALREADY KNOWN ON THIS TOPIC

      • Currently, healthcare professionals rely on their expertise, experience and the latest scientific findings (evidence-based medicine) to diagnose systemic inflammatory response syndrome (SIRS), sepsis and associated organ dysfunctions.

      • The criteria for SIRS, sepsis and associated organ dysfunctions must be assessed for various age ranges in children; thus, diagnosing can be challenging, in particular for less experienced and overloaded clinicians from paediatric intensive care units (PICUs).

      • Digitalisation, data availability and data accessibility in PICUs are increasing; thus, data-driven prediction models, implemented as clinical decision support systems, can be applied to support clinical decision-making.

      WHAT THIS STUDY ADDS

      • The evaluation of the performance (ie, sensitivity and specificity) of previously developed prediction models for diagnosing SIRS, sepsis and associated organ dysfunctions.

      HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

      • The prediction models could allude the healthcare professionals to possible health deteriorations related to SIRS, sepsis and associated organ dysfunctions in a timely manner.

      Introduction

      Medical staff in paediatric intensive care medicine are faced with the challenge of recognising and treating clinically relevant disease processes safely and early in an environment characterised by high stress levels, time pressure, work interruptions and risk situation. Currently, clinicians rely on a combination of expertise, experience and the best available evidence-based medicine. However, the necessary combined, implicit knowledge is often only represented in individual persons and not permanently accessible to less experienced persons. Risks for medical errors are omnipresent due to high dynamics, uncertainties, immediate decision-making needs and large data volumes.1 2

      Systemic inflammatory response syndrome (SIRS), sepsis and associated organ dysfunctions (OD)3 are common and relevant disease processes in paediatric intensive care. These processes significantly influence the morbidity and mortality of critically ill children.4 The diagnosis and recognition of these diseases are more complex than in adults due to the large number of different vital and laboratory parameters that must be evaluated considering various age-specific reference ranges.4 In this challenging situation, medical informatics tools, such as clinical decision support systems (CDSSs), could assist clinicians5 in everyday problem solving as they summarise, analyse, and present clinically relevant data at the point of care. However, their diagnostic performance must be first proven6 by estimating the sensitivity and specificity with which the index test correctly diagnoses patients. Such an assessment should be part of a diagnostic test accuracy (DTA) study that uses an established reference standard (ie, current gold standard) as the ground truth.7 8

      Within the preceding CADDIE-2 study (ClinicalTrials.gov NCT03661450), the diagnostic accuracy of a knowledge-based model for onset detection of SIRS, its length, and end, was evaluated. This model detected SIRS with a sensitivity and specificity of 91.7% (95% Wald confidence interval [CI]: 85.5% to 95.4%) and 54.1% (95% Wald CI: 45.4% to 62.5%), respectively.9 Additional knowledge-based models for the diagnosis of sepsis and associated OD (ie, hepatic, haematologic, respiratory, renal and cardiovascular OD) were developed and evaluated. The knowledge-based models are able to assess retrospectively the onset, length, and end of the conditions. However, for predicting their onset before knowledge-based models can react, data-driven prediction models are required which we developed using data from the last 7 years of our study centre and intend to assess in a DTA study setting.

      We hypothesise that the early warning of such potential life-threatening diagnoses, supported by prediction models, is beneficial for patients as clinicians can early on implement treatment measures to minimise/avoid further health deterioration.

      Methods and analysis

      Reporting and trial registration

      We report our study in accordance with the ‘Standard Protocol Items: Recommendations for Interventional Trials’ (SPIRIT)10 11 guideline that we altered to fit a DTA study (online supplemental file 1). The trial is registered with the German Clinical Trial Register (DRKS00029071; (table 1)).

      Supplemental material

      View this table:
      Table 1

      Summary of the study’s registration of the German Clinical Trial Registry

      Study objectives and hypotheses

      In this study, the prediction models are externally validated (ie, in the newly sampled prospective dataset) for their DTA in a purely research-oriented analysis (ie, the models are not running as a live application on the ward) addressing the following objectives:

      • Five primary confirmatory objectives are evaluated to determine the performance of the prediction models for SIRS, sepsis, hepatic OD, haematologic OD and respiratory OD. These hypotheses are tested hierarchically (ie, in the listed order) which is based on the underlying pathophysiology as OD can develop from SIRS and sepsis. Each analysis estimates the sensitivity and specificity (co-primary endpoints) of the prediction model for the corresponding target disease and whether the corresponding prediction model has a sensitivity ≥75% and a specificity ≥75%, including their 95% Wald CIs, in detecting the target disease. Only if this is the case, the next hypothesis is tested. The threshold of 75% marks the minimum DTA with which our prediction models should diagnose to still be considered a useful support for clinicians.

      • Two key secondary exploratory objectives

        • To determine the performance of the prediction models for renal OD and cardiovascular OD, and

        • To test if the sensitivities and specificities of the prediction models are superior to the real-time evaluations of clinicians working in routine clinical care.

      Study design and setting

      This study is a monocentric, prospective DTA study, which classifies as non-interventional (according to the German Medical Products Act12) since it does not include any interventions using a medical product yet.5 6 Single study centre is the Paediatric Cardiology and Paediatric Intensive Care Unit (PICU) of the Hannover Medical School (MHH) situated in an urban North-German federal province with a large catchment area and a patient volume of approximately 1000 paediatric patients per year who receive also treatment other than post anaesthesia care. The MHH does not use any diagnostic tools for the target diagnoses yet.

      The study design is outlined in figure 1. Recruitment and eligibility screening started on 2022-07-25 and continues for approximately 9 months until the required sample size is included (see the Intended sample size section).

      Figure 1
      Figure 1

      The study design of the ELISE study. Note that only during the enrolment and clinical assessment patients are recruited in the paediatric intensive care unit (PICU) and all routine data monitoring measures take place for approximately 9 months (ie, prospectively). Only routine data are used; no new or additional data are collected. All subsequent model assessments and the data analysis are performed using the previously recorded routine data (ie, retrospectively).

      Study timeline

      During the enrolment and clinical assessment, clinicians of the study centre continuously screen all PICU patients. Patients who stay for at least 12 hours qualify for enrolment and their physicians start the recruitment process. Simultaneously, the clinicians assess all prospectively collected routine patient data once per shift for potential episodes of SIRS, sepsis and associated OD (ie, standard of care). Each assessment is stored in a secure, closed off file in the patient record.

      Then, during the model assessments, the data integration processes are conducted to extract, transform and integrate the complete dataset of the recruited patients from the primary source system. This is followed by the automatic application of the prediction models (index test) and the knowledge-based models (reference standard) that have been previously trained and tested on the CADDIE-2 dataset. Both assessors evaluate the occurrence of any diagnostic episodes of SIRS, sepsis and associated OD using the patient’s routine data, retrospective to the PICU stay, while being blinded to the other assessor’s (incl. clinicians) evaluations. A comparison between different patient populations (single-arm; except patient strata) or interventions (non-randomised) is not pursued. Adverse events, requiring emergency unblinding, cannot occur due to the retrospective data evaluation.

      During the data analysis, the primary and secondary endpoints are evaluated (see the Research objectives section).

      Eligibility

      Study participants are patients (0 days to 17 years at enrolment) who are eligible if they stay at the study centre - independent of sex, underlying disease or admission time—and if written, informed consent is signed by them (if patients are ≥6 years old) and their legal guardians/representatives. Excluded are patients who stay in the PICU for less than 12 hours (ie, decreased probability of experiencing SIRS, sepsis and/or associated OD) and/or informed consent was not given/withdrawn before data analysis.

      Eligible patients can only be recruited once, but re-admissions to the study centre after being determined ‘eligible’ are included in the final analysis regardless of the length of stay. Non-participation has no negative effects on the patient’s medical care. Patients will not be compensated, neither monetarily nor otherwise.

      Intended sample size

      To investigate all hypotheses, including those with a rather low prevalence, a sample size of 500 patients was considered sufficient using the method by Stark and Zapf13 with an overall power of 80%. However, an adaptive sample size planning method is used and scheduled after including the first 250 patients to account for the actual prevalence among the recruited patients. This new sample size is the target; hence, either enrolment is stopped if the target is met or continued until the target is met, but no longer than 12 months (online supplemental file 2).

      Supplemental material

      Recruitment

      The study centre physicians approach all patients and their legal guardians/representatives if they stay in the PICU for at least 12 hours. They inform them about the research’s scope and ask for their consent. All study-related information is provided orally and written prior to signing the informed consent form (available in German, English, Turkish and Arabic) and is modified for the age groups 6–11 years, 12–14 years and 15–17 years (online supplemental file 3). Consent can be given during the PICU stay at any time, but also later during the enrolment and clinical assessment. All physicians are regularly reminded to engage in the recruitment process.

      Supplemental material

      Diagnostic approaches

      All data of eligible patients are assessed using equal testing methods.

      Index test

      For the prediction of SIRS, sepsis and associated OD, various machine learning techniques such as boosting regression, naive Bayes, support vector machine, decision tree and random forest are tested before determining the optimal approach for usage in a data-driven prediction model (ie, the best performance considering the study objectives).14 The prediction models (ie, specifically tailored for one diagnosis) should trigger an alert for a potential SIRS, sepsis or associated OD episode up to 12 hours prior to the disease onset defined by the reference standard; thus, in future, clinicians can start an early treatment. Each prediction model is trained and tested on an extended CADDIE-2 study dataset15 16 before they are validated using the new prospectively enrolled population of this study.

      Reference standard

      In this study, knowledge-based models (see refs. 17 18 for technical details) are applied as the reference standard for retrospective detection of SIRS, sepsis and associated OD. They are used to label the diagnostic status per patient within the dataset and have been previously validated against the current gold standard of independent retrospective extensive chart reviews by two experienced intensive care clinicians. Using a knowledge-based model as the reference standard is more efficient and reliable given the enormous volume of data to be reviewed per patient. The relevant multimodal parameter values are queried for one patient per day from a previously harmonised and standardised openEHR-based data repository16 and added to a knowledge base, a set of computerised diagnostic expert rules for the research diagnoses. Afterwards, the rule engine activates the appropriate rules. These include, for example, deriving age-specific reference range values, summarising the queried values to medians, checking the exceeding of the limit values, triggering individual parameter alerts, and checking the presence of a relevant combination of alerts, which then lead to a decision. Technically, the knowledge-based procedures were realised via Java and the open-source business rule management system Drools from JBoss (RedHat).19

      Routine assessments

      Inexperienced (<10 years) and experienced (≥10 years) clinicians document the present and suspected conditions of the patients at the end of their shifts using a digital documentation form (online supplemental file 4). These assessments are carried out without an in-depth analysis of the data available in the patient data management system (PDMS), and happen completely under routine care conditions. The aim is to document whether and when a patient was diagnosed clinically with SIRS, sepsis and associated OD and which symptoms led to the assessment. All clinicians are trained on how to use the digital documentation and will be regularly reminded to document their assessments.

      Supplemental material

      Data management and collection

      For eligible patients, intensive care routine data20 are extracted from the PDMS m.life20 and the data warehouse of the MHH, are transferred into a semantic interoperability standard for clinical information representation (openEHR)17 and loaded into an openEHR-based data repository, using internationally agreed on, standardised data models.17 No additional examinations and tests are performed (ie, only available routine data are used). Assessment data from the index tests, the reference tests and the clinicians (ie, start and end time of SIRS, sepsis and associated OD episodes) as well as general documentations of the patient conditions, events or unintended effects will be documented.

      Data monitoring and auditing

      Quality assurance measures are carried out continuously throughout the project. When integrating primary source data into the standardised data repository, plausibility checks—ranging from simple counts to uncover missing data from the primary system to logical checks—are executed. Automatically executed validation checks, like semantic checks or double data entries, are ensured when transferring and loading data into our data repository by using the openEHR standard. We strive to detect missing or wrong values automatically during data integration. The contributing authors and designated clinicians monitor the trial procedures continuously and supervise study protocol compliance and data privacy.

      Data protection: data access and confidentiality

      For compliance with local data protection laws, data are integrated in a pseudonymised way. Patient identifiers and personal data are removed and replaced by pseudonyms. Data from patients who withdrew their consent are deleted immediately from the openEHR-based data repository. The dissolution of a pseudonym requires the involvement of the data repository administrators and the responsible physicians.

      The data repository is located in the MHH Information Technology (MIT) network and is subject to the MIT-specific data protection concept. Unauthorised storage, processing or reproduction of protected data is prevented by standard technical and organisational measures. All system accesses are made via encrypted connections. The usual measures apply for software protection (ie, firewall, virus protection, encryption programmes). Non-pseudonymised data are processed exclusively for the purpose of patient care. All measures were coordinated with the data security officer and recorded in a data protection concept. Patients are informed about these procedures and their rights (ie, the possibility to withdraw consent and to obtain information about collected datasets at any time), and are asked to consent to these (see the Recruitment section).

      Statistical analysis

      The statistical analysis plan (online supplemental file 5) provides details on the evaluation of the primary confirmatory and secondary exploratory analyses. As described above, the primary confirmatory analyses will be performed in a hierarchical approach due to the relations of the diagnoses and their individual diagnostic prevalences. Sensitivities and specificities with their 95% Wald CI for the detection of SIRS, sepsis and associated OD are estimated using a clustered nonparametric approach21–23 that accounts for the longitudinal data format (ie, 24 hours per day and several days per patient). The DTA estimates will be adjusted for potential misclassification caused by using an imperfect reference standard.24 Exploratory subgroup analyses will be performed for age groups, sexes, and age groups by sexes using the same statistical approach as for the main analysis.

      Supplemental material

      Ethics and dissemination

      This study (ie, study protocol, patient information and informed consent forms) received ethical approval from the MHH Ethics Committee (No. 10188_BO_SK_2022; online supplemental file 6). All aspects were reviewed and accepted by the data security and privacy officers of the MHH. Any modifications to the study protocol and/or the patient information and informed consent forms require a formal amendment (ie, ethical approval) to the original study documents.

      Supplemental material

      The patient and/or the legal guardians/representatives sign the informed consent form after a personal prestudy consultation through the attending physician. All information is also provided in written form. The information sheet for children is provided in age-specific forms.

      Results are disseminated via peer-reviewed publications, scientific conference presentations, and in an appropriate way to the participants that is still to be defined. Data will only be stored pseudonymised and processed within the specified concept (see the Data protection section).

      Data availability statement

      The datasets generated and/or analysed during the current study are not publicly available due to data privacy and security matters of patients but are available from the corresponding author on reasonable request16.

      Ethics statements

      Patient consent for publication

      Ethics approval

      This study obtained ethics approval from the Hannover Medical School Ethics Committee (No. 10188_BO_SK_2022).

      Acknowledgments

      We thank our colleagues from the MIT department for Educational and Scientific IT Systems of the Hannover Medical School for supporting in matters of data access and collection, as well as all clinicians for participating in the routine evaluation. This publication is part of the 'Ein Lernendes und Interoperables, Smartes Expertensystem für die pädiatrische Intensivmedizin' (Acronym: ELISE) project, funded by the German Federal Ministry of Health (ZMVI1-2520DAT66C).

      References

      1. ↵
        1. Lighthall GK,
        2. Vazquez-Guillamet C
        . Understanding decision making in critical care. Clin Med Res 2015;13:156–68.doi:10.3121/cmr.2015.1289pmid:http://www.ncbi.nlm.nih.gov/pubmed/26387708
        OpenUrlAbstract/FREE Full Text
      2. ↵
        1. Lincoln P,
        2. Manning M-J,
        3. Hamilton S, et al
        . A pediatric critical care practice group: use of expertise and evidence-based practice in identifying and establishing "best" practice. Crit Care Nurse 2013;33:85–7.doi:10.4037/ccn2013740pmid:http://www.ncbi.nlm.nih.gov/pubmed/23547131
        OpenUrlFREE Full Text
      3. ↵
        1. Chakraborty RK,
        2. Burns B
        . Systemic inflammatory response syndrome. In: StatPearls, 2021. https://www.ncbi.nlm.nih.gov/books/NBK547669/
      4. ↵
        1. Goldstein B,
        2. Giroir B,
        3. Randolph A
        . International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics*. Pediatric Critical Care Medicine 2005;6:2–8.doi:10.1097/01.PCC.0000149131.72248.E6
        OpenUrlCrossRefPubMed
      5. ↵
        1. European Medicines Agency
        . Medical devices. Available: https://www.ema.europa.eu/en/human-regulatory/overview/medical-devices [Accessed 04 Oct 2022].
      6. ↵
        1. European Parliament and European Council
        . Regulation (EU) 2017/746 on in vitro diagnostic medical devices and repealing directive 98/79/EC and Commission decision 2010/227/EU; 2017.
      7. ↵
        1. Leeflang MMG,
        2. Allerberger F
        . How to: evaluate a diagnostic test. Clin Microbiol Infect 2019;25:54–9.doi:10.1016/j.cmi.2018.06.011pmid:http://www.ncbi.nlm.nih.gov/pubmed/29906592
        OpenUrlCrossRefPubMed
      8. ↵
        1. Chassé M,
        2. Fergusson DA
        . Diagnostic accuracy studies. Semin Nucl Med 2019;49:87–93.doi:10.1053/j.semnuclmed.2018.11.005pmid:http://www.ncbi.nlm.nih.gov/pubmed/30819399
        OpenUrlPubMed
      9. ↵
        1. Wulff A,
        2. Montag S,
        3. Rübsamen N, et al
        . Clinical evaluation of an interoperable clinical decision-support system for the detection of systemic inflammatory response syndrome in critically ill children. BMC Med Inform Decis Mak 2021;21:1–9.doi:10.1186/s12911-021-01428-7
        OpenUrlCrossRef
      10. ↵
        1. Chan A-W,
        2. Tetzlaff JM,
        3. Altman DG, et al
        . SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med 2013;158:200–7.doi:10.7326/0003-4819-158-3-201302050-00583pmid:http://www.ncbi.nlm.nih.gov/pubmed/23295957
        OpenUrlCrossRefPubMedWeb of Science
      11. ↵
        1. Chan A-W,
        2. Tetzlaff JM,
        3. Gøtzsche PC, et al
        . SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 2013;346:e7586.doi:10.1136/bmj.e7586pmid:http://www.ncbi.nlm.nih.gov/pubmed/23303884
        OpenUrlAbstract/FREE Full Text
      12. ↵
        1. Federal Law Gazette
        . Medicinal products act (Arzneimittelgesetz – AMG), 2021. Available: https://www.gesetze-im-internet.de/englisch_amg/index.html [Accessed 06 Jan 2022].
      13. ↵
        1. Stark M,
        2. Zapf A
        . Sample size calculation and re-estimation based on the prevalence in a single-arm confirmatory diagnostic accuracy study. Stat Methods Med Res 2020;29:2958–71.doi:10.1177/0962280220913588pmid:http://www.ncbi.nlm.nih.gov/pubmed/32299298
        OpenUrlPubMed
      14. ↵
        1. Mast M,
        2. Marschollek M,
        3. Jack T, et al
        . Developing a data driven approach for early detection of SIRS in pediatric intensive care using automatically labeled training data. Stud Health Technol Inform 2022;289:228–31.doi:10.3233/SHTI210901pmid:http://www.ncbi.nlm.nih.gov/pubmed/35062134
        OpenUrlPubMed
      15. ↵
        1. Wulff A,
        2. Mast M,
        3. Bode L, et al
        . Towards an evolutionary open pediatric intensive care dataset in the ELISE project. Stud Health Technol Inform 2022;295:100–3.doi:10.3233/SHTI220670pmid:http://www.ncbi.nlm.nih.gov/pubmed/35773816
        OpenUrlPubMed
      16. ↵
        1. Wulff A,
        2. Mast M,
        3. Bode L
        . ELISE - An open pediatric intensive care data set, 2022.doi:10.24355/DBBS.084-202203101150-0
      17. ↵
        1. Wulff A,
        2. Haarbrandt B,
        3. Tute E, et al
        . An interoperable clinical decision-support system for early detection of SIRS in pediatric intensive care using openEHR. Artif Intell Med 2018;89:10–23.doi:10.1016/j.artmed.2018.04.012pmid:http://www.ncbi.nlm.nih.gov/pubmed/29753616
        OpenUrlPubMed
      18. ↵
        1. Bode L,
        2. Schamer S,
        3. Böhnke J, et al
        . Tracing the progression of sepsis in critically ill children: clinical decision support for detection of hematologic dysfunction. Appl Clin Inform 2022. doi:doi:10.1055/a-1950-9637. [Epub ahead of print: 26 Sep 2022].pmid:http://www.ncbi.nlm.nih.gov/pubmed/36162433
      19. ↵
        1. The JBoss Drools Team
        . Drools expert user guide: community documentation. Available: https://docs.jboss.org/drools/release/5.4.0.Final/drools-expert-docs/html_single/#d0e128 [Accessed 15 Oct 2020].
      20. ↵
        1. medisite GmbH
        . medisite GmbH. Available: https://medisite.de/mlife/ [Accessed 05 Jul 2022].
      21. ↵
        1. Brunner E,
        2. Zapf A
        . Nonparametric ROC analysis for diagnostic trials. Methods Appl Stat Clin Trials 2014;2:483–95.doi:10.1002/9781118596333.CH28
        OpenUrl
      22. ↵
        1. Lange K
        . Nichtparametrische analyse diagnostischer Gütemaße bei Clusterdaten, 2011. Available: https://ediss.uni-goettingen.de/handle/11858/00-1735-0000-000D-F1D1-B [Accessed 29 Jul 2021].
      23. ↵
        1. Rooney D
        . Covariate adjusted nonparametric estimation of sensitivity and specificity in clustered data; 2017.
      24. ↵
        1. Umemneku Chikere CM,
        2. Wilson KJ,
        3. Allen AJ, et al
        . Comparative diagnostic accuracy studies with an imperfect reference standard – a comparison of correction methods. BMC Med Res Methodol 2021;21:1–12.doi:10.1186/s12874-021-01255-4
        OpenUrlCrossRefPubMed

      Supplementary materials

      Footnotes

      • JB and NR contributed equally.

      • TJ and AW contributed equally.

      • Collaborators ELISE Study Group: Louisa Bode (Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany); Marcel Mast (Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany); Antje Wulff (Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany, and Big Data in Medicine, Department of Health Services Research, School of Medicine and Health Sciences, Carl von Ossietzky University Oldenburg, Oldenburg, Germany); Michael Marschollek (Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany); Sven Schamer (Department of Pediatric Cardiology and Intensive Care Medicine, Hannover Medical School, Hannover, Germany); Henning Rathert (Department of Pediatric Cardiology and Intensive Care Medicine, Hannover Medical School, Hannover, Germany); Thomas Jack (Department of Pediatric Cardiology and Intensive Care Medicine, Hannover Medical School, Hannover, Germany); Philipp Beerbaum (Department of Pediatric Cardiology and Intensive Care Medicine, Hannover Medical School, Hannover, Germany); Nicole Rübsamen (Institute of Epidemiology and Social Medicine, University of Münster, Münster, Germany); Julia Böhnke (Institute of Epidemiology and Social Medicine, University of Münster, Münster, Germany); André Karch (Institute of Epidemiology and Social Medicine, University of Münster, Münster, Germany); Pronaya Prosun Das (Research Group Bioinformatics, Fraunhofer Institute for Toxicology and Experimental Medicine, Hannover, Germany); Lena Wiese (Research Group Bioinformatics, Fraunhofer Institute for Toxicology and Experimental Medicine, Hannover, Germany); Christian Groszewski-Anders (medisite GmbH, Hannover, Germany); Andreas Haller (medisite GmbH, Hannover, Germany); Torsten Frank (medisite GmbH, Hannover, Germany).

      • Contributors JB, NR, TJ and AW were responsible for drafting the manuscript and coordinating the planning and outlining of the study protocol and ethics application. JB, AK and NR were responsible for the design of the statistical analysis, the sample size calculation, and all analysis-related questions for the study. TJ and HR provided clinical expertise for the use case. AW and TJ act as project coordinators of the ELISE project managing the design and implementation of the knowledge-based and predictive models to be applied from both a technical (AW) and a clinical (TJ) perspective. MM is responsible for data management, collection, access and confidentiality and leads the data integration and harmonisation work. HR and TJ are responsible for patient recruitment and monitor the study on the ward. The ELISE study group comprises all project partners contributing to the design and implementation of the knowledge-based as well as the predictive models for detection and prediction of SIRS, sepsis and associated OD and their evaluations within the ELISE project. All authors contributed to the study design and writing of the protocol. All authors read and approved the final manuscrip.

      • Funding This work was supported by the Federal Ministry of Health, Germany, Grant No. 2520DAT66A, and the Open Access Publication Fund of the University of Muenster, Germany.

      • Competing interests None declared.

      • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

      • Provenance and peer review Not commissioned; externally peer reviewed.

      • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.