Objective The objective of this study is twofold: first, to describe the methods used when involving children and young people (CYP) in developing a paediatric research agenda and, second, to evaluate how the existing literature describes the impact of involving CYP. We distinguish three forms of impact: impact on the research agenda (focused impact), impact on researchers and CYP (diffuse impact) and impact on future research (research impact).
Design A narrative review of MEDLINE, PsycINFO, Web of Science and Google Scholar was conducted from October 2016 to January 2022. The included studies involved at least one CYP in developing a research agenda and were published in English.
Results 22 studies were included; the CYP involved were aged between 6 years and 25 years. Little variation was found in the methods used to involve them. The methods used were James Lind Alliance (JLA) approach (n=16), focus groups (n=2), workshop (n=2), research prioritisation by affected communities (n=1) and combined methods (n=1). Impact was rarely described: focused impact in nine studies, diffuse impact in zero studies and research impact in three studies.
Conclusion This study concludes that the JLA approach is most frequently used to involve CYP and that all methods used to involve them are rarely evaluated. It also concludes that the reported impact of involving CYPs is incomplete. This study implies that to convince sceptical researchers of the benefits of involving CYPs and to justify the costs, more attention should be paid to reporting these impacts.
- Data Collection
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
This study provides an overview of different methods used when involving children and young people (CYP) in research priority setting.
The James Lind Alliance method is most frequently used to involve CYP in priority setting even though the method is rarely evaluated.
This study shows that little is reported about the impact of research priority setting with children or young people.
Implementation plans of research agendas are rarely described, while it is considered a waste of resources should the project end with publishing the research agenda.
The idea that children should be treated as passive subjects in research is changing. They are more and more involved as active agents.1 The involvement of children is now recognised as a best practice and is an essential requirement for paediatric research funding allocation by funders in the UK, Australia, the USA and the Netherlands.1 2
Children should be involved in every phase of the research, starting with what research should be about, in so-called research agendas. Paediatric research agendas used to be predominantly developed by professionals and researchers.3 Increasing evidence illustrates that research questions prioritised by professionals may not be aligned to those experiencing the disease.4 At worst, this results in limited research money being spent on research that is not important to patients, and money is wasted.3 This raised a call for collaboration with children and young people (CYP) as equal partners to develop research agendas.
Thus far, the involvement of CYP in developing research agendas appears to be limited. Few studies purely include CYP in developing those agendas. More often, adults act as a proxy for CYP’s views.5 A systematic review by Odgers and colleagues published in 2017 showed that 25% of studies reported some parental or caregiver involvement. Only in 5% of the studies were children involved directly.6 This is partly explained because there is no agreement on what might constitute best practice for involving CYP in developing a research agenda.7 Moreover, the involvement of CYP may bring age-specific barriers and challenges such as increased workload, unknown impact on the research agenda and power imbalances.7
Efforts to develop engaging and developmentally appropriate strategies that involve CYP in developing a research agenda are lacking. The most well-known example is the James Lind Alliance (JLA) method. The JLA unites patients, carers and clinicians to identify and prioritise the top 10 unanswered research questions in so-called priority setting partnerships (PSPs). Odgers and colleagues question the extent to which the JLA method may be well suited to involve CYP, although they do not clarify this claim.6 Previous studies have not dealt with identifying what methods are well suited to involve CYP in PSPs.8
One of the most significant discussions about involving CYP is that the impact of their involvement is not clear.9 Reasons for assessing this are numerous: to improve the involvement of CYP, to convince sceptical researchers of its benefits, to reduce tokenistic involvement, to justify the cost of the involvement of CYP, and to increase funding for their involvement.10 Therefore, it is strongly recommended to conduct more research that critically examines this impact.11 12 We distinguish three forms of impact, of which the first two were described before.13 (1) The effect of the involvement of CYP on the research agenda (focused impact), (2) The effect of the involvement of CYP on researchers and CYP themselves (diffuse impact) and (3) What is reported on action plans for assessing the effect of the research agenda on future research (research impact). Assessing these forms of impact may be challenging but documenting the contributions and incorporations of these contributions into the research priority setting may be feasible and would be welcomed by many contributors.10 This paper has two key aims. First, we will identify the methods used to involve CYP in formulating a research agenda and perform a first exploration on the evaluation of these methods. Second, the study aims to assess what is reported about the impact of involving CYP in research priority setting.
We conducted a narrative review to gain a qualitative perspective on the methods used to involve CYP in developing a research agenda and the reported impact of this involvement.
The research team cocreated the literature search strategy in collaboration with an information librarian. We used the Medical Subject Headings and text words for ‘children’, ‘priority setting partnerships’ and ‘research agenda’. Online supplemental file 1 provides more details about the search strategy. Each search term within the three categories were combined with the Boolean operator “OR” and the three different categories were combined with the Boolean operator “AND”. Databases searched were MEDLINE, EBSCOhost, Web of Science, Google Scholar and the JLA website. The included articles were uploaded in the programme Rayyan QCRI (Qatar Computing Research Institute (Data Analytics), Doha, Qatar) and duplicates were removed.
The research team specified the inclusion criteria after a thorough consultation. Articles were included in this review if developing a paediatric research agenda with the involvement of at least one CYP aged below 18 years was reported and if the articles were written in English and were published between October 2016 and March 2022 (to add more research agendas that have been developed with CYP to the four already identified by Odgers and colleagues).6 For the inclusion, we have chosen a three-step approach: (1) the first author screened the title and abstracts of 557 articles; (2) all articles for which it was unclear whether they should be included were intensively discussed with the last author; moreover, the articles that were already included were discussed in detail; (3) in the final step, the inclusion was discussed with the research team. The same three-step approach was chosen for the inclusion of the 89 full-text articles.
A narrative synthesis was performed. To systematically describe data from the included studies, two data extraction forms in Microsoft Excel were developed. Descriptive information of the studies (eg, title, authors and method used to involve CYP) were reported on the first data extraction form. The second form was developed to chart data on the age and the number of the CYP involved, the phase of the involvement and the impact of the involvement. To assess the impact of the research priority setting, we divided impact into three forms: focused impact, diffuse impact and research impact. The data were extracted by LP and discussed with the research team.
We used the 32-item checklist developed by Odgers and colleagues to assess the transparency of reporting of research priority setting. They extracted items from good practice principles to develop the checklist. Another frequently used checklist, the Guidance for Reporting Involvement of Patients and Public checklist (GRIPP2),14 was developed to help improve the quality, consistency and transparency of reporting patient and public involvement in research. The checklist of Odgers differed from the GRIPP2 checklist in that it was developed to assess the reporting of research priority setting specific. Therefore, we decided to use the checklist of Odgers instead of the GRIPP2 checklist.
The original checklist of Odgers was not developed to specifically assess the reporting on developing a research agenda together with CYP. Therefore, we added three items to make sure the checklist covers important aspects of involving CYP. Next, the items will be further explained. The first item, ‘describes the method used to involve CYP in developing a research agenda’, was added to the list because we agree with Flynn and colleagues that appropriate strategies that involve CYP are lacking.15 The second and third items were added to the list to assess different forms of impact: ‘describe the impact of the involvement of CYP on the research agenda’ (focused impact) and ‘describe the impact of the research priority setting on the participants (diffuse impact). We rephrased the original item 29: ‘describe how impact will be measured’ as ‘describe how the impact of the research agenda on future research will be measured’ (online supplemental file 5).
Twenty-two studies were included in this review (figure 1). Most of the studies were conducted in the UK (n=13) (online supplemental file 2 and figure 1). The CYP involved were aged between 6 and 25 years. Seventeen studies involved children below the age of 18, and 2 studies did not report the age of the CYP involved. The number of the CYP involved in the included studies ranged from 1 to 108. Four studies did not report the number of CYP involved (see tables 1 and 2).
The transparency of reporting score was averaged across the studies. The scores of those included ranged from 11 to 27 items out of 36 items (online supplemental file 3, figure 2). Strikingly, few studies reported the impact of the CYP on the agenda (n=9), the action plans for implementing priorities (n=8), the evaluation of the PSP (n=6), methods used to involve CYP (n=5) and how impact of the research agenda will be measured (n=3). No studies reported how the feedback was integrated and whether the research priority setting impacted the participants (online supplemental file 3, figure 3). The completed checklist can be found in table 3.
Methods used in paediatric priority setting
Little variation was found in the methods used to involve CYP in paediatric research priority setting. The JLA approach was the most frequently used method (n=16).16–31 This was followed by focus groups (n=2),9 32 a workshop approach (n=2),33 34 the research prioritisation by affected communities (RPAC) method (n=1). The RPAC method directly involves individuals from under-represented groups in identifying, ranking and prioritising their unanswered questions about their health conditions.35 In one study, different methods were combined36 (online supplemental file 4, figure 4).
The JLA method divided the involvement of children into four phases. A total of 358 children were involved in the identification of research questions16 17 19 20 22–25 27; 287 children were involved in the prioritisation of research questions16 17 19 20 22 23 25–27; 38 children were involved in the prioritisation workshop17 22–24 26 33 34 36 and seven children were involved in the steering group16 18 23 (online supplemental file 4, figure 5). To ensure the involvement of paediatric patients of all age categories, Verwoerd and colleagues added focus groups with children in all phases of the JLA method.17 Similarly, Grant and colleagues organised additional focus groups for younger participants but only at the identification phase.18 Nonetheless, Lim and colleagues found that focus groups were problematic for the younger participants; therefore, they were contacted individually.26 The advantages of the JLA were as follows: it is a rigorous method for the establishment of priorities16; CYP reported their involvement as positive and powerful16 23; and it fulfils many of the criteria for good practice in priority setting.27 Examples of the criteria that have been used were using a comprehensive approach and inclusiveness of stakeholders.37 Disadvantages of the JLA were as follows: prioritisation in this manner is highly subjective16 18; CYP are less represented in almost all phases of the priority setting process20 23 26 27; and researchers themselves need to refine the research questions.25
Two studies used focus groups to involve CYP.9 32 Manikam and colleagues organised two focus groups involving 7–10 CYP.32 They were asked to prioritise research topics that were submitted by healthcare professionals. Parsons and colleagues organised thirteen focus groups, in which a total of sixty CYP were involved.9 In these focus groups, CYP were asked to identify the research questions themselves. No advantages or disadvantages were reported using focus groups to involve CYP.
A workshop was used to involve CYP by two research teams.33 34 Both teams used the JLA method as a basis for their workshop. Lopez-Vargas and colleagues organised a workshop in which CYP first had to present their prepared research questions and then had to vote for their top three priority questions.33 Schilstra and colleagues used the workshop to clarify why each priority mattered to the CYP and how they would address the priorities. This approach extended the impact of survey-based approaches by enabling CYP to compare their experiences and actionable research questions were developed.34 In contrast, survey-based approaches may require less of the CYP’s time than workshops. Furthermore, Schilstra and colleagues found that recruitment to an in-person workshop can be challenging and time-consuming.34
Another method used to involve CYP was the RPAC.35 Following the RPAC method, two focus groups were organised. In the first focus group, individuals shared their experiences and generated a list of research questions. In the second focus group, individuals prioritised the topics they want researchers to focus on. In both focus groups, 11 CYPs were involved. An advantage of the RPAC is that it was developed to directly involve patients using their personal experiences rather than beginning with survey data.35 No disadvantages were reported.
Reported impact of paediatric priority setting
This study focused on three forms of impact: focused impact, diffuse impact and research impact. Diffuse impact was not described at all.
In nine studies, the focused impact was described.17–19 22 24 27–29 33 Focused impact of the included studies can be divided into two categories: different research questions and different research priorities. In the first category, CYP have different research questions than researchers have. In the second category, CYP have the same research questions, but they prioritised the questions differently than the researchers did (table 4).
Action plans for assessing the research impact were described in three studies26 28 29 (table 5). Noteworthy is that assessing the research impact of research priority setting is as challenging as assessing focused impact. Assessing the research impact takes a long time, and this requires the research team to be involved for a longer time span.
In this study, we identified that the JLA method is most frequently used to involve CYP in developing a research agenda and that the impact is insufficiently described at best. The results add to the rapidly expanding field of involvement of CYP. Our study showed that the involvement of CYP in developing research agendas has grown since 2016. Previously, only four research agendas were formulated together with CYP.6 Five years later, this involvement has increased fivefold, resulting in 22 research agendas. This growth indicates the change in the position of CYP in research.
JLA method most frequently used method
The JLA method was most frequently used to involve CYP in developing a research agenda. van Seventer and colleagues argue that although the outcomes of involving CYP in developing a research agenda have been described, reflecting on the method used to involve CYP is hardly performed.8 Yet, Verwoerd and colleagues did evaluate the JLA method, and they were one of the first who integrated additional focus groups to involve the younger children in developing a research agenda.17 They found it to be of added value because otherwise, the views of adolescents and young adults would have been over-represented.38 Our results indicate that only six studies evaluated the method used. Therefore, more information is needed to justify the statement about that JLA method not being well suited to CYP.6
Impact is insufficiently described at best
There is widespread acknowledgement that analysing the focused impact is challenging because it is difficult to know which contribution of the CYP made the difference in developing the research agenda. Yet, this study shows that nine of the included studies attempted to describe the contribution of CYP. It is noteworthy that no studies reported the diffuse impact. The main goal of developing a research agenda together with CYP is to provide the most important research questions. Yet, we should keep in mind that researchers with a positive experience in partnering with CYP in research are most likely to implement a similar collaboration in the future.39 CYP with a positive involvement experience gain knowledge and confidence which can affect their own lives and work and can provide motivation to be involved in later studies.39 Therefore, diffuse impact could also be an important argument for involving CYP.
The JLA recognises that the partnerships between patients, clinicians and professionals may have an impact on the people who participate in them and on the research agenda itself. Interestingly, the JLA guidebook does not elaborate on how to evaluate this impact. The guidebook does provide valuable recommendations on how to maximise the research impact of the agreed priorities.40 The guidebook might have been more all-encompassing if it encouraged researchers to evaluate the focused and diffuse impact as well.
Publishing a research agenda should be a tool, not a stand-alone goal
Only eight of 22 studies reported the action plans to implement the research agenda, and only 3 of these reported keeping track of the research impact. This marginal reporting on the postprioritisation phase is seen in JLA PSPs in general.39 As a result, little information is available about whether the research agenda is implemented. Jongsma and colleagues interviewed the participants involved in their PSP. Participants considered the PSP a waste of money and time, should the project end with the publication of the top 10 priorities.8 This is a striking outcome because our study showed that only a few studies described continuing the project after publishing the research agenda. Staley and colleagues suggested extending the partnership to cover impact-oriented activity beyond publishing the agenda.39 Taking the results of our study into account, we agree with this proposal so plans can be implemented, and the impact of the research agenda can be measured. Awareness about the fact that publishing the research agenda is not a stand-alone goal is important. Influencing research practice and thereby changing paediatric care should be the goal striving for. Publishing a research agenda is an important tool for achieving that.
A limitation of this study is the inability to retrieve how many CYP of a specific age group were included. In the included studies, the age of the CYP was divided into broad categories. Although the agendas developed together with children have increased from 4 to 22 in 5 years, we did not compare the number of the research agendas that have been developed together with children to the total of research agendas. Therefore, we cannot state anything about the relative growth compared with the total.
Future research and conclusion
This study aimed to identify the methods used to involve CYP in developing a research agenda and to assess what is reported about the impact of involving CYP in research priority setting. We found that the JLA method is most frequently used even though it is rarely evaluated as to whether it is appropriate for involving CYP. This study suggests that an evaluation on the methods should be performed to understand if these are appropriate for the involvement of CYP. Furthermore, this study concludes that reporting the impact remains rare. To be able to measure the impact, researchers should perform a qualitative study focusing on what researchers and CYP believe are important characteristics when measuring the impact of developing a research agenda together. This could lead to an operationalised definition of impact. In our follow-up study, we will start with this. Furthermore, we recommend expanding the guidelines on involving children in developing a research agenda with information on how to evaluate the impact.
Patient consent for publication
We thank T van Wulfften Palthe, PhD, for correcting the English manuscript.
Contributors LP coordinated this project, collected and analysed the data, wrote the first version of the manuscript, and drafted and wrote the final version of the manuscript. EM was involved in data collection, analysis and drafting the manuscript. EV and ML were involved in drafting the manuscript and writing the final version of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.