Scope of the COS

The COS is intended to be used to evaluate the overall success of operative and non-operative treatment among children who are assigned a clinical and/or radiological diagnosis of acute uncomplicated appendicitis. The COS will include outcome measures identified as important within 12 months of treatment initiation and longer term outcomes if applicable. The COS focuses specifically on treatment of acute uncomplicated appendicitis; the treatment of perforated appendicitis (with or without abscess) and appendix mass is outside the scope of this COS. The key objectives of the study are:

1. to determine which outcomes have previously been reported in studies comparing treatments for acute uncomplicated appendicitis in children

2. to prioritise treatment outcomes of children with acute uncomplicated appendicitis from key stakeholder groups’ perspectives (including paediatric surgeons, general surgeons, patients (12–18 years old) and parents of children who have had acute uncomplicated appendicitis)

3. to compare and contrast paediatric acute uncomplicated appendicitis treatment outcomes prioritised by key stakeholder groups (as detailed above)

4. to achieve consensus between key stakeholder groups on a COS to evaluate overall success of treatment for acute uncomplicated appendicitis in children.

Design

COS development will entail four key stages:

1. a systematic review to identify previously reported acute uncomplicated appendicitis treatment outcomes

2. assembly of stakeholder panels

3. a three-phase online Delphi process

4. consensus meeting.

The CONTRACT study, as part of the feasibility study, will involve in-depth qualitative research. Interviews will be conducted with patients, parents and caregivers to explore experiences of treatment for appendicitis. We will explore families’ perceptions of meaningful and important outcomes, which will help to optimise COS participants’ project understanding and engagement throughout the COS development.

Systematic review

The COMET Initiative recommend the use of systematic reviews in informing the first phase of the Delphi process.11 Two recent systematic reviews will be used to inform the initial list of potential outcomes to be considered for the COS. The first review identified outcomes used in studies of paediatric appendicitis,10 and the second review aimed to determine safety and efficacy of non-operative treatment for acute appendicitis.9 Identified outcomes were combined with closely similar outcomes from the operative treatment systematic review10 to develop an initial list of outcomes (online supplementary appendix 1). Online supplementary appendix 1 describes the eligibility criteria for inclusion of papers. All relevant articles will therefore be included that reported any non-operative treatment regimen for acute uncomplicated appendicitis in children with or without a comparative group of children undergoing surgical treatment. All outcomes identified through these reviews and an updated literature search will inform the initial list of outcomes and will be considered by the stakeholder groups as part of the Delphi process.

Finalising and appropriate wording of initial outcomes

To inform and support the CONTRACT study, a Study Specific Advisory Group (SSAG) has been assembled, comprising 15–20 young people and parents. Young people recruited are children who have had appendicitis or children from the existing Clinical Research Network (Children) Young Persons Advisory Groups. Parents are parents of children who have had appendicitis. An SSAG meeting will be held to present the initial list of treatment outcomes, to inform the addition and wording of outcomes and to ensure outcomes are appropriately presented. Three versions of the stakeholder-facing materials will be developed for all rounds of the Delphi process, for each stakeholder panel (clinicians, young people and parents), using appropriate language identified and agreed by the SSAG. The CONTRACT qualitative study will also inform the initial outcomes.

Stakeholder panel assembly: identification and recruitment

For the COS to be meaningful and relevant to those involved in the treatment of acute appendicitis, the COS needs to reflect the views of patients who have been treated for acute appendicitis, their parents and relevant clinicians. As these groups may have different priorities that could obstruct reaching consensus on a COS, the stakeholders will be separated into three panels, which we intend to be equally weighted: (1) patients; (2) parents; and (3) paediatric surgeons and general surgeons. Initially, potential members of each stakeholder panel known to the research team will be invited to participate, and we will develop strategies to identify further experts (see table 1).

Initial contact with potential participants will explain the study and why they have been identified as a potential participants. It will contain a plain language summary of the study aims and procedures, emphasising the importance of commitment to the panel. The wording of this initial contact will be tailored to meet the panel category. It will also contain a link to an online form to enable participants to express their interest in participation in the study and to provide further information on their experience of the treatment of acute appendicitis. We will ask participants to commit to completing three rounds of questionnaires anticipated to take approximately 10 min each to complete.

The process of invitation and enrolment will continue until the optimal number of stakeholders have expressed an interest to participate (with at least 10 in each panel). There is no consensus on the optimal sample size for a Delphi study; recruitment will therefore be based on previous Delphi studies.12 We will aim to achieve 75–100 participants in the first round of the Delphi with at least as many parents/children as clinicians. Efforts will be made to invite a diverse range of participants to each stakeholder group. We will aim to send the first questionnaire to all participants on the same day they each confirm their desire to participate.13 Participants will be sent a link to a customised online database hosted on a secure server, from which they can access and complete phase one questionnaire of the Delphi process. To limit attrition, appropriate procedures will be completed,12 14 including reminder emails.

Participation in this COS development process will be limited to surgeons, patients and parents from the UK. We will not recruit paediatric or general surgeons from outside the UK because the treatment pathway for appendicitis in the UK differs to that in other countries. Furthermore, if we were to recruit surgeons from outside the UK, we would also need to recruit patients and parents from outside the UK to avoid bias; this would become increasingly challenging logistically.

Definition of consensus

During the process, participants will be asked to score each outcome using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) scale, which is recommended by the COMET Initiative.15 The scale will be presented in the format 1–9, with 1–3 labelled ‘not important’, 4–6 labelled ‘important but not critical’ and 7–9 labelled ‘critical’.16 ‘Consensus in’ will be defined as ≥70% of participants rating the outcome 7–9 and <15% rating it as 1–3. Outcomes will defined as ‘consensus out’ if >70% participants rate it 1–3 and <15% rate it 7–9. Outcomes not meeting these definitions will be classified as ‘no consensus’.

Delphi process: phase one data collection

A customised online data system (developed by the National Perinatal Epidemiology Unit (University of Oxford, UK)) will be used to conduct a three-phase Delphi process run in parallel across the stakeholder panels. Participants will be presented with the initial list of outcomes, grouped by domains. As described, the initial list will comprise outcomes identified in our recent systematic review,10 updated review of the literature (online supplementary appendix 1) and any additional outcomes identified during qualitative interviews with key stakeholders. There will also be an option for experts to add further outcomes, but these outcomes will not be scored in phase one.

Surgeon participants will be asked the key question ‘How important do you consider the following outcomes to be when considering which treatment to offer children with uncomplicated acute appendicitis?’ A similar question will be posed to patient and parent stakeholder panels, with the wording altered as necessary based on our SSAG input.

Participants will be asked to complete each round of the Delphi exercise within 3 weeks, and two subsequent reminder emails will be sent. Participants who have not completed the questionnaire within 4 weeks of being requested to complete the questionnaire will be deemed not to have completed that phase.

Delphi process: phase one data analysis

The number of participants who were invited to participate in phase one and the response rate from each stakeholder group will be recorded. Outcomes will be analysed separately for each panel, and descriptive statistics will be calculated. All outcomes will be carried forward to phase two.

Additional outcomes provided by participants will be reviewed by two members of the COS team to ensure they represent new outcomes and will be included in phase two if they were proposed by at least two participants.

Delphi process: phase two data collection

Participants who completed phase one will be invited to participate in phase two. Participants will be individually presented with their own scores, the distribution of scores for each outcome from their stakeholder group in phase one, and will be asked to rescore each outcome. Participants will be asked to score any new outcomes identified in phase one.

Delphi process: phase two data analysis

The data analysis process described for phase one will be repeated. Bias from loss of experts between phases will be assessed. Any outcomes that meet the criteria of ‘consensus out’ will be removed from the outcomes list prior to phase three. All other outcomes from phase two will be carried forward to phase three.

Delphi process: phase three data collection

Participants who completed phases one and two will be invited to participate in phase three. The data collection process described for phase two will be repeated; however, participants will also be shown scores for their own stakeholder panel and separately for each other panel. This will allow participants to consider other stakeholder groups’ views before rescoring the outcomes.17 Participants will be asked to identify the one outcome which they believe is the most important for informing their treatment choice, and if they cannot identify a single outcome, a combination of essential outcomes.

Delphi process: phase three data analysis

The data analysis process described for phase two will be repeated. All outcomes from phase three will be carried forward to the consensus meeting.

Final consensus meeting

The aim of the consensus meeting is to ratify outcomes where consensus (‘in’ or ‘out’) has been achieved, to discuss outcomes where consensus could not be achieved and to finalise the COS. All participants who have completed all three rounds of the Delphi exercise will be invited to attend the consensus meeting. We will aim to have a minimum of 40 stakeholders confirm their attendance with equally weighted panels and disciplines. Representatives from each stakeholder panel will be required in order for the consensus meeting to be quorate.

During the meeting, stakeholders will be provided with an overview of the results of phase three including presentation of each outcome scored, how it was scored by each stakeholder panel and its consensus status. Following moderated discussion, each outcome will be anonymously rescored using the same scoring system as the Delphi process. For outcomes for which ‘no consensus’ was achieved across all stakeholder panels at the end of the Delphi exercise, and for which consensus was achieved in at least one but not all stakeholder groups, further discussion will take place, following which attendees will be asked to score each outcome anonymously. Following rescoring at the consensus meeting, outcomes reaching ‘consensus in’ will be included in the finalised COS. All others will be excluded.

Finalising the COS

The Outcome Measures in Rheumatology initiative provides a COS development framework that is a useful across various healthcare domains.18 In developing the current COS, we will draw on this framework, which comprises three core domains (death, life impact and pathophysiological manifestations) and one strongly recommended domain (resource use). The framework recommends inclusion of at least one applicable measurement instrument for each core domain. It also recommends inclusion of ‘adverse events’. Overall, we aim to achieve a manageable COS with a maximum of approximately 10 outcomes. Beyond the current project, further work may be undertaken to establish optimal methods of measuring each core outcome18 and subsequent work may be undertaken to determine whether the UK-based COS is valid internationally.

Data management

Experts will enter data directly into the customised database when they complete each questionnaire at each phase of the Delphi process. Anonymised data will be stored securely and will be managed as per standard operating procedures. Only selected members of the research team will have access to the data.

Ethics and dissemination

Full ethical approval for CONTRACT was granted by South Central - Hampshire A Research Ethics Committee (REC ref: 16/SC/0596) in November 2016. The finalised COS will be disseminated via publication, events and the COMET Initiative website (http://www.comet-initiative.org/).