Objective To validate a three-step protocol that assesses the clinical risk associated with using blood glucose monitoring systems (BGMS) in neonates for the management of dysglycaemia.
Method The three-step validation approach included confirmation of the accuracy of the reference method using National Institute of Standards and Technology (NIST) glucose standards, assessment of analytical risk performed on whole blood collected from paediatric patients routinely tested for glucose and a clinical risk assessment performed using heel stick capillary samples collected from 147 new-born babies and neonates admitted to intensive care. BGMS glucose measurements were compared with the NIST aligned laboratory reference method.
Results The accuracy of the laboratory reference method was confirmed with the NIST standards. Specificity studies demonstrated that the accuracy of one of the BGMS was affected, particularly, in the hypoglycaemic range, by known interference factors including haematocrit, ascorbic acid, lactose, galactose, N-acetylcysteine and glutathione. The accuracy of the other BGMS was unaffected. The clinical performance of this BGMS in neonates met the system accuracy criteria of Clinical and Laboratory Standards Institute (CLSI) POCT 12-A3 standard for evaluating hospital BGMS with 95.1% of glucose measurements within±0.67 mmol/L for samples ≤5.55 mmol/L and 95.6% within±12.5% for samples>5.55 mmol/L.
Conclusions This three-step validation protocol provides a challenging approach for determining the accuracy and reliability of BGMS for managing dysglycaemia in neonates. StatStrip BGMS achieved analytical and clinical performance criteria confirming its suitability for use in neonates. We advocate that this validation approach should be considered for performance evaluations of both BGMS and continuous glucose monitoring systems going forward.
- blood glucose monitoring
- isotope dilution Liquid Chromatography Mass Spectrometry
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- blood glucose monitoring
- isotope dilution Liquid Chromatography Mass Spectrometry
What is already known on this topic?
There is increasing debate about the accuracy and reliability of current bedside blood glucose monitoring systems (BGMSs).
Published case studies and real-time as well as simulation modelling studies have highlighted that the use of inaccurate BGMS can lead to mismanagement of dysglycaemia resulting in adverse clinical outcomes.
Evaluations to assess clinical performance and risk often neglect to confirm the accuracy of the reference method, questioning the validity of data analysis.
What this study hopes to add?
A three-step protocol to assess the clinical accuracy risk of a diagnostic method that includes: (1) isotope dilution mass spectrometry alignment of the reference method, (2) an analytical risk assessment focusing on specificity studies and (3) a clinical risk assessment performed on the target patient populaton.
The three-step validation protocol used provides a platform for more effective validation of the performance of BGMS in neonates.
First demonstration of BGMS meeting Clinical and Laboratory Standards Institute Point-of-Care Testing (POCT) system accuracy criteria in a neonatal population.
Accurately identifying and managing dysglycaemia is important in high-risk neonates. Neonates with prolonged or recurrent episodes of hypoglycaemia can have a poor prognosis and are at risk of developing long-term neurocognitive impairments. This is most marked in symptomatic hypoglycaemia, but long-term neurological sequelae have also been associated with asymptomatic hypoglycaemia in small-for-gestational age babies, babies of diabetic mothers or preterm babies.1 2 Hyperglycaemia is common in preterm babies particularly extremely low birthweight babies during the first days of life. Unmanaged hyperglycaemia is associated with short-term negative influence on neurological and behavioural outcomes and is a risk factor for early mortality in very preterm infants.3 Close monitoring of blood glucose levels in high-risk babies is important in avoiding the consequences of hypoglycaemia and hyperglycaemia.4 Bedside handheld blood glucose monitoring systems (BGMCs) are widely used for measuring glucose in neonates.5 However, there is increasing debate about the accuracy and reliability of current bedside BGMS.4–7 Analytical accuracy evaluations, published case studies and real-time as well as simulation modelling studies have highlighted that the use of inaccurate BGMS can lead to mismanagement of dysglycaemia resulting in adverse clinical outcomes.8–14 Falsely elevated BGMS glucose values in adult and neonatal hospitalised patients particularly those in intensive care can be associated with pathophysiological factors or medication present in the patient’s whole blood.8 9 Abnormal haematocrit values, for example, have been shown to adversely affect the accuracy of BGMS glucose measurements in neonates with low haematocrit causing falsely high glucose values.15 The consequence of this can result in a missed neonatal hypoglycaemia, and it has been reported that currently used BGMS affected by haematocrit can miss up to 50% of hypoglycaemias in neonates.16 The failure to recognise hypoglycaemia could have clinical risk consequences for preterm neonates. The increased scrutiny of BGMS accuracy has led to the introduction of new guidelines and standards for assessing glucose metre performance. Prior to 2012, the only international standard available was the International Organization for Standardization (ISO) 15197 applied only to blood glucose systems used for self-monitoring.17 In 2013, this was updated to include tighter system accuracy criteria.18 In 2012, the first standard for use in hospital patients was published from the Clinical and Laboratory Standards Institute (CLSI POCT 12-A3) with system accuracy criteria tighter than included in ISO 15197:2013.19 To date, there are limited studies assessing the clinical risk accuracy of BGMS in neonates in comparison with the CLSI performance criteria. In China, there has also been little validation of the wide range of blood glucose metres used in hospitals. Many of the glucose metres in use are those that are intended for use only for self-monitoring in individuals with diabetes but have migrated into hospital setting. A recent international multisite regulatory compliant clinical risk assessment study advocates that an isotope dilution mass spectrometry (IDMS) aligned laboratory reference method should be used as the comparator method in BGMS performance evaluations.20 The aim of this study was to establish and validate a three-step protocol to evaluate the clinical risk associated with using BGMS in neonatal and paediatric patients for management of dysglycaemia, using an IDMS traceable reference method.
Material and methods
A three-step validation approach was used to assess BGMS clinical accuracy risk as according to Chinese Consensus Glucose Performance Guidelines for hospital BGMS21 and CLSI POCT 12-A3 guideline.19 The three-step approach comprised: (1) validation of the accuracy and IDMS traceability of the laboratory reference method with international reference, (2) an analytical risk assessment including method correlation, precision and specificity studies performed on two BGMS, and (3) a clinical accuracy risk assessment was performed with the BGMS demonstrating better specificity and the lowest analytical risk.
IDMS calibration alignment of the reference method
The validation of the IDMS alignment of the central laboratory reference method was performed with National Institute of Standards and Technology (NIST) standard reference materials 917c (NIST certified pure glucose, purity 99.7%±0.3%) and 965b (glucose in frozen serum with NIST-certified concentration values), both of which were assessed by isotope dilution gas chromatography mass spectrometry. The reference laboratory method used glucose oxidase run on a Hitachi 7180 laboratory analyser (Roche Diagnostics, Indianapolis, Indiana, USA).
Analytical risk assessment
For the analytical risk assessment, venous whole blood specimens were collected in lithium heparin blood collection tubes from paediatric patients routinely tested for glucose as part of our institution’s glycaemic control programme. Each whole blood specimen was tested directly after collection using the two BGMS systems: StatStrip Glucose (Nova Biomedical, Waltham, Massachusetts, USA), AccuChek Inform II (Roche Diagnostics). The two BGMSs were randomly used throughout the analytical study. Following BGMS testing, the samples were immediately centrifuged, and the plasma was tested on the hospital central laboratory reference analyser glucose method. For the specificity studies, the influence of interfering factors such as ascorbic acid, lactose, galactose, β-hydroxybutyrate, N-acetylcysteine, glutathione and haematocrit on the accuracy of glucose measurements was assessed. Adult donated blood specimens were used for preparation of specificity samples. The donated blood was left overnight for glycolysis to occur resulting in the removal of native glucose present in the blood sample. Blood samples were prepared with five different glucose concentrations (1.1–3.3, 6.9–9.7, 12.5–15.3, 18.1–20.8 and 25.4–29.2 mmol/L), and at each glucose level, two different concentrations of each interference were tested along with a zero concentration interference control. The concentrations of the interference factors tested reflected the levels that can be seen in patients at the upper end of the clinical and therapeutic ranges and were based on published guidelines.19 22 Five different blood glucose levels were also tested in combination with five different levels of haematocrit representing the abnormal ranges seen in neonatal intensive care unit (NICU) patients. The interference blood samples were tested using the two BGMS and by the NIST aligned central laboratory reference method.
Clinical risk assessment
Heel stick capillary samples were collected from hospitalised new-born babies including neonates admitted to intensive care. To ensure sufficient glucose levels within the hyperglycaemic or hypoglycaemic range, 20 neonatal specimens were spiked with a glucose solution or left to undergo glycolysis in order to ensure sufficient glucose levels within the hyperglycaemic or hypoglycaemic range. Each whole blood specimen was tested directly after collection using the StatStrip Glucose BGMS, and the haematocrit level was determined in each whole blood specimen using microhaematocrit centrifugation (StatSpin VT Centrifuge, Statspin, Norwood, Massachusetts, USA). Following this, each sample was immediately centrifuged, and the plasma was tested on the hospital central laboratory analyser glucose method.
All data analysis and calculations were performed using Analyse-It software V.4.0 (Analyse-IT Software, Leeds, UK) and Windows Excel. Passing-Bablok regression analysis was used for the determination of correlation coefficient, intercept and slope. For assessment of method correlation the mean, bias, SD and % bias difference between the BGMS and the laboratory reference glucose oxidase method were determined. Bias plot analysis was used to assess BGMS system accuracy in comparison with the criteria defined in Chinese Consensus Glucose Performance Guidelines21 and CLSI POCT 12-A3 guideline19 for hospital blood glucose monitoring. The % bias difference between the glucose value for the control interference samples (zero), and the glucose value for each interference dosed samples was calculated for each interfering factor and for each BGMS
IDMS calibration alignment of the reference method
The glucose oxidase laboratory reference method showed good agreement and alignment with the NIST SRM917c and 965b tests confirming IDMS alignment and the accuracy of the reference method (online supplementary table S1) for use in the analytical and clinical risk assessment studies.
Supplementary file 1
Analytical risk assessment
The analytical interference study showed that ascorbic acid, lactose, galactose, β-hydroxybutyrate, N-acetylcysteine and glutathione did not interfere with the accuracy of StatStrip BGMS glucose measurements. For the AccuChek Inform II BGMS, only β-hydroxybutyrate did not interfere with the accuracy of glucose measurements. The interference factors particularly influenced the accuracy of the low level glucose measurements for the AccuChek BGMS indicating a risk of misidentifying hypoglycaemia (figure 1). Varying haematocrit (HCT) levels did not interfere with the accuracy of StatStrip BGMS glucose measurements, but for the AccuChek Inform II BGMS, the medium and higher glucose levels glucose readings were inversely proportional to the HCT levels (figure 2). Both StatStrip and AccuChek BGMS showed a good initial correlation to the NIST aligned reference method (online supplementary figure 1). The overall coefficient of correlation (r) for StatStrip compared with the NIST aligned reference method was 0.998 with a slope of 0.943 and intercept of 0.041 and an overall mean % bias difference of −4.46%. The overall coefficient of correlation (r) for AccuChek Inform II compared with the NIST aligned reference method was 0.998 with a slope of 0.907 and intercept of 0.254 and an overall mean % bias difference of −3.44%. The agreement between the two BGMS and the NIST aligned reference method was also analysed using the Bland-Altman plot (online supplementary figure 2).
Supplementary file 2
Supplementary file 3
Although both BGMS showed a good correlation to the laboratory reference method with the residual specimens, we only selected the StatStrip BGMS for the clinical risk assessment validation step because the accuracy of AccuChek Inform 2 was affected by substances that may be present in the whole blood matrix of hospitalised neonatal and paediatric patients.
Clinical risk assessment
Heel stick capillary blood specimens were collected and tested from 147 neonatal patients including 86 males and 61 females. The mean age of the neonatal population was 9.6 days and the ages ranged from 1 to 28 days old. Demographic breakdown by gender and additional information on birth weight and clinical disorder categories can be seen in online supplementary table S2. Each heel stick capillary blood specimen was analysed using StatStrip BGMS and the NIST aligned reference laboratory glucose method. The StatStrip BGMS achieved concordance to both the Chinese consensus performance criteria and the CLSI POCT12-A3 guideline system accuracy criteria (table 1, figure 3). Haematocrit present in the neonatal samples, values ranged from 27% to 70% (mean 49%), also had no effect on the accuracy of glucose measurements (figure 4).
Supplementary file 4
Accurate glucose measurements are important for identifying and managing neonatal dysglycaemia. Handheld BGMSs are widely used for this purpose despite concern about their accuracy for identifying and managing neonatal hypoglycaemia.1 4
However, it is also recognised that continuous glucose monitoring systems, although useful for monitoring glycaemic variability and with improving overall performance, are still less accurate than BGMS for use in hospitalised patients.23 Blood gas analysers are regularly used for glucose measurements in patients in NICU, but a recent report indicates that these may not show good agreement with central laboratory results and may not meet the accuracy criteria of new hospital use performance guidelines.24 The advent of CLSI POCT 12-A3 guideline as the first guideline published providing performance criteria for use in hospital patients provides a protocol for evaluating clinical risk prior to selection of a BGMS. In this study, this protocol was adopted for the first time in China in a three-step validation approach to determine the safest BGMS option for use in a neonatal patient population. A key step in the process is to ensure the accuracy and traceability of the reference method glucose measurements, which was established in our study using international NIST standards. Confirmation of NIST IDMS alignment is rarely presented in published accuracy studies undertaken in neonatal patients. In our three-step protocol, the analytical risk assessment step raised concerns about the neonatal application of AccuChek BGMS because of the influence of some of the interference substances (which can be present in the whole blood matrix of hospitalised neonates) on the accuracy of glucose measurements. This has been previously reported for AccuChek Inform II in accuracy assessments performed on adult patient populations with the BGMS not achieving the CLSI performance criteria.25 26 Erroneous blood glucose readings have been reported due to galactose interference in neonates with galactosaemia-inherited disease due to transferase deficiency leading to mismanagement and misdiagnosis of galactosaemia.11 27 However, transient non-persistent galactosemia can also occur in neonates due to liver disease, delay in liver maturation and portasystemic shunting,28 29 and if unrecognised could lead to inaccurate glucose readings. Lactose is a key component of breast milk and non-glucose sugars including lactose, and galactose can be present in neonatal formula feeds. It is not known if at the concentrations present in blood this could impact on neonatal blood glucose readings, but it has been reported that the AccuChek metre is susceptible to interference by low concentrations of galactose that could potentially influence the ability to detect hypoglycaemia.30 The accuracy of StatStrip BGMS was unaffected by these interference factors, and a third validation step was undertaken to assess clinical accuracy risk. In previously published studies, StatStrip was reported to be acceptable for use in neonates, but in these studies, SMBG and not hospital glucose performance criteria was used to assess accuracy.31 32 In our study, StatStrip BGMS met these tighter system accuracy criteria of CLSI POCT 12-A3. This outcome is similar to that recently reported in a large multisite study performed on critically ill adult patients.20 In this larger study, a trend analysis of glucose measurements in the hypoglycaemic range was performed to confirm the accuracy in this range. A limitation of our study was the small number of glucose measurements in the hypoglycaemic range limiting the scope for a trend analysis. Our findings indicate that StatStrip BGMS is accurate across the glycaemic range found in neonates, and two recent publications have reported improved clinical outcomes in paediatric and neonatal patients following a switch to using StatStrip BGMS.33 34
This three-step validation approach is a useful tool for identifying clinical risk associated with using a BGMS in a neonatal setting. The laboratory glucose oxidase method used in our hospital is IDMS traceable and as such is validated for undertaking an evaluation of the accuracy of BGMS. StatStrip shows strong measurement accuracy in neonates either in normal inpatient ward or in NICU. Consideration should be given to adopting the three-step validation approach for performance evaluations of both BGMS. At present, we do not use continuous glucose monitoring systems (CGMS) in our hospital neonatal setting but recognise that CGMS is being evaluated for neonatal use and as such we believe that our three-step protocol should also be considered for determining clinical accuracy of CGMS.
YB and JX contributed equally.
Contributors CC was the lead investigator of the study devising the design of the study protocol and undertaking data collection and analysis, data interpretation, literature searches, preparation editing and submission of the manuscript. All authors contributed to the overall conception and final design of the study and to the interpretation of results and drafting of the manuscript. All authors read and approved the final manuscript. CC affirms that the manuscript is an honest, accurate and transparent account of the study being reported.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Parental/guardian consent obtained.
Ethics approval Approved by the Ethics Committee of Children’s Hospital of Fudan University.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Supplemental data available.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.