Methods
Trial design
We conducted a randomised within-subject trial to estimate the impact of the CDSS PEDeDose on the number of dose calculation errors and the time needed for the derivation. As interventions, we defined either the Swiss Summary of Product Characteristics (SmPC)15 used together with a pocket calculator (control) or the CDSS PEDeDose14 with its built-in calculator (full). Furthermore, we exploratively assessed the impact of the PEDeDose web application without using the built-in calculator but using a pocket calculator instead (basic). A pool of 18 items, each representing one drug prescription for a hypothetical paediatric patient, was created (online supplemental file 1). The items were developed by the main author (LH) and reviewed by two clinical pharmacists (KK and PV) with extensive experience in the field of paediatrics and neonatology. Only drugs with a paediatric label were selected so that a reference dosage was available in the Swiss SmPC. For each participant the trial consisted of three consecutive blocks. To each block one of the three interventions and six items drawn from the pool were randomly assigned without replacement. The trial design is visualised in figure 1.
Figure 1Visualisation of the trial design. The name of the interventions correspond to the CDSS PEDeDose with built-in calculator (full), PEDeDose used together with a pocket calculator (basic), and SmPC used together with a pocket calculator (control). CDSS, clinical decision support system; SmPC, Summary of Product Characteristics.
We report this study in concordance with the ‘Reporting Guidelines for Healthcare Simulation Research: Extensions to the CONSORT and STROBE Statements’.16 17
Participants
Our target population consisted of physicians and pharmacists in Switzerland. We focused the recruitment on physicians and pharmacists working in children’s hospitals, general hospitals with paediatric clinics and HCPs working in the ambulatory setting that is, public pharmacists and general practitioners. To ensure a high quality of the collected data, the trial was conducted under the supervision of the main author. Participants gave informed consent to the data collection and received a small monetary compensation for their participation.
The participants were mainly recruited via convenience sampling by directly contacting the responsible head of department in Swiss children’s hospitals, general hospitals with paediatric clinics or by the company’s newsletter. Furthermore, snowball sampling was used as many of the participating HCPs were also helping recruiting their colleagues.
Interventions
The CDSS PEDeDose encompasses a database with general paediatric dosing information and a built-in calculator for individualised dosing. The built-in calculator makes PEDeDose a CDSS. However, the general dosing information can also be consulted without using the built-in calculator. Thus, we defined three interventions: The Swiss SmPC used together with a pocket calculator (control), the CDSS PEDeDose (full) and the PEDeDose dosing information used together with a pocket calculator (basic). The study was powered to compare the CDSS PEDeDose (full) to the SmPC used together with a pocket calculator (control). The SmPC is a full-text electronic resource, while the data of the PEDeDose database is highly structured. Thus, to isolate the effect of structuring drug dosing information, we exploratorily assessed the impact of using PEDeDose without the built-in calculator. An example of the structured dosing information from PEDeDose is shown in online supplemental file 2.
Simulation setup
The trial was developed using the Gorilla Experiment Builder (www.gorilla.sc), a web-based trial platform.18 We conducted the trial at the participants workplace. Depending on the availability participants were using their own computers or were provided with a notebook for the trial. The Gorilla website was opened in a browser while the interventions (ie, SmPC or PEDeDose websites) were opened either in a different browser tab or window, depending on the participants preferences. Before the trial started, every participant was briefed about the aim and the design of the trial. Subsequently, the participants were required to solve a dedicated test example with the PEDeDose built-in calculator (full). This ensured that the participants fully understood the capabilities of PEDeDose, such as the possibility to convert the calculated dosage to the correct dosing unit (eg, mg to mL).
Participants were instructed to round the calculated dosage to a maximum of two decimal places. If a dose range was provided by the respective dosing information, participants were asked to submit a range as a result, too.
Outcomes
The primary outcome was the correctness of the derived dosage, a binary variable with 1=error and 0=correct. The secondary outcome was the time needed to solve an item.
Since the dosing information that the participants were required to use was specified in advance and no additional clinical evaluation was required, there was an objectively correct dosage for every item within the corresponding dosing information (SmPC or PEDeDose). Errors were defined as submitted responses that exceeded clinically non-relevant deviations of 5% or 10% for drugs with narrow or wide therapeutic windows, respectively (online supplemental file 1). Even though the participants were required to submit dose ranges as a range, we did not consider it an error if the submitted dosage was a single dosage that was within the correct window. We reviewed all erroneous responses and tried to determine the possible cause of error. For the errors that were found in the full block (ie, PEDeDose with built-in calculator), the logging data of the PEDeDose built-in calculator were additionally analysed. This allowed us to assess whether the participant had specified the calculator inputs incorrectly (ie, drug, indication, route of administration, birthdate, weight, height and gestational age).
The secondary outcome response time was defined as the time difference (in seconds) for each item between the time stamp on the mouse click that initialised item loading and the click that submitted the result. We defined outliers in the time outcome as values greater than three SD for each intervention. We removed outliers and missing values and analysed only complete items.
Covariates
The following categorical participant covariates were assessed prior to the trial start: The type of institution where the participant was working as an unordered factor (children’s hospital, general hospital with children’s clinic, public pharmacy or doctor’s office), their profession (physician, pharmacist), their working experience as an ordered factor (<5 years, 5–10 years, >10 years), and whether they had been already using PEDeDose in their daily work (yes, partly, no).
Sample size
The sample size estimation was done in collaboration with the Clinical Trial Unit of the University of Basel, Switzerland. An a priori error rate of 20% for the control study arm was assumed based on the results of previous research estimating a 26.5% error rate for dose calculation using a pocket calculator.19 A 50% overall error reduction at a significance level of 5% with >80% power resulted in a total of 600 items that need to be rated. We aimed to test the two arms for the confirmatory analysis with six items per arm, which resulted in an estimated sample size of 50 participants (600 items/12 items per participant=50 participants) (online supplemental file 3). Adding an equal number of items for the exploratory arm, the resulting total number of items that need to be rated was 900, which corresponds to 18 items per participant.
Randomisation
Randomisation was done on the level of the interventions and the items (figure 1). Thus, for each participant the order of the three interventions was randomised, while for each intervention 6 out of the pool of 18 items were randomly drawn without replacement. The Gorilla Experimental Builder enabled to design the randomisation procedure directly into the trial, thus taking care of the participant allocation.18
Statistical methods
Statistical analyses were performed in R V.4.1.1.20 The relevant functions and additional packages used are denoted as function {package}. The only continuous variable was the secondary outcome response time per item, which was transformed using the natural logarithm to achieve normality of the residuals. Orthogonal sum-to-zero contrasts for the unordered factors ’institution’ and ‘profession’ applying contr.sum {stats} were used. The lower-level effects were thus estimated at the level of the grand mean and interpreted accordingly. We applied difference coding for the ordered factors ‘experience’, ‘PEDeDose user’ and the exploratory version of the variable ‘intervention’ using contr.sdif {MASS}.21 Thus, each level of the ordered factors was compared with their previous level. The contrast coding scheme is provided in online supplemental file 4.
For the primary outcome ‘error’, we fitted a generalised linear mixed-effects model (GLMM) with a logit-link function using glmer {lme4}.22 The secondary outcome ‘time’ was assessed by fitting a linear mixed-effects model (LMM) using lmer {lmerTest}.23 All models were derived by starting with maximal model specification based on the trial design, and then sequentially reducing model complexity until a non-singular fit was achieved.24 We started by defining by-subject and by-item random intercepts and slopes (ie, crossed-random effects) on each type of intervention. The main variable ‘intervention’ and the additional covariates were treated as fixed variables.
In exploratory analyses, we assessed the impact of structuring the dosing information by adding the intervention basic (ie, PEDeDose without the built-in calculator). Thus, the binary variable for the intervention became an ordered three-level factor (control, basic, full). As a sensitivity analysis, we created a model that is only adjusted for the order of the interventions.
For all models also an unadjusted model was built, containing only the variable ‘intervention’ as well as only random intercepts for both subject and item, respectively. We derived Wald confidence intervals. The p values for the linear models were derived via Satterthwaite’s df method.23 The estimated marginal means for the ‘intervention’ variable for all the models were calculated using emmeans {emmeans}.25 The summary outputs of the models are reported in online supplemental file 2.