The cooking component

The resource use associated with the cooking component of the intervention was measured using school staff-completed logbooks. School staff recorded how much time was spent on both the preparation and the delivery of the workshops and classroom sessions. On most occasions it was the teacher who was involved, and sometimes TAs were used. When ‘other staff’ members were listed but their roles were not described, a TA-level role was assumed. The materials used in the workshops were purchased and delivered by the research team in person. Parents were also invited, and their time and travel costs were acquired from a resource use questionnaire. Receipts for the materials used in the cooking workshops were logged and costed to estimate the average material cost per class. The costs associated with delivering the materials to the schools were calculated from the mileage and time it took for the research assistant to travel to each school.

The physical activity component

Each school within the intervention arm was asked to pick two of the offered PA packages. The costs associated with these packages were recorded as implementation costs and are detailed in Table 29. The PA component was delivered either during class time or at lunchtime. All resource use associated with the preparation and delivery of the packages was recorded using school staff-completed logbooks.

TABLE 29

Unit costs

Villa Vitality programme

This part of the intervention was split into the following subcomponents:

• two Villa Vitality days
• one Villa Vitality school-based session (run by football club staff)
• classroom challenges and class project.

The Villa Vitality days were purchased at a fixed cost, which was then averaged across all of the classes to derive a cost per class. When school staff had supervised the children during the Villa Vitality days, the costs that were associated with their time were included. Parents were also welcome to attend the Villa Vitality days and, when this happened, their time and resource use were recorded in travel cost questionnaires and included within the sensitivity analysis, for which a wider perspective was adopted.

Teaching staff completed logbooks to record the resource use that was associated with the delivery of the classroom sessions.

Signposting

The resource use associated with the signposting included printing and the delivery of materials to the schools. All of the purchase orders and receipts for printing were recorded and the cost of delivery was estimated based on the number of sheets posted. This item of expenditure was treated as an ongoing cost, as it required either updating (generic signposting sheet) or complete revision (school-specific signposting sheet) at the start of each intervention year and was therefore included as part of the intervention cost within the base-case analysis.

Unit costs and assumptions

This section outlines the justification of, and source for, the unit costs applied to each component of the intervention, as outlined in Table 29.

School unit costs

Unit costs for an hour of a teacher’s time were calculated using the mid-scale point on the standardised Department for Education salary scales.⁹² Resource use was measured on a per-hour basis. The annual salary scales were based on a contracted 1265 hours per year. Thus, to estimate the cost per hour, the annual salary was divided by 1265 hours. Annual salary scales were not available for TAs, LTAs or dinner ladies, and so to calculate the hourly unit costs for these roles the estimates were based on unit costs published by the Office for National Statistics in its 2014 New Earnings Survey.⁹⁰

University unit costs

Included within the sensitivity analyses were the costs associated with the set-up and implementation of the intervention. Many of these costs were incurred by the research institute at the University of Birmingham. These costs included administration and research staff time. To calculate the unit costs for each of these roles, the 2014 University of Birmingham academic/support staff salary scales were used. Appropriate mid-points of the salary scale were selected for each staff position. These salaries were then converted to an hourly rate assuming that staff worked 7.5 hours per day excluding weekends, university ‘closed days’ and public holidays.

Other unit costs

For the cooking workshops, the unit costs for the cooking materials used were based on the purchase price of each item. For the PA elements of the intervention, the fixed costs associated with the activity packages chosen were recorded by the trial team and the receipts were retained.

All of the costs were adjusted to 2014 prices. Finally, for various aspects of the intervention, there were associated travel costs. When travel by car was recorded, the Her Majesty’s Revenue & Customs guidance was used and a £0.45-per-mile cost was applied in line with standard practice.

Measuring quality-adjusted life-years

After initial pilot research on the acceptability, reliability and validity of competing preference-based measures for children,⁹³ the CHU9D instrument was used to collect quality-of-life information for the children. The CHU9D⁴⁶ features nine dimensions of child HRQL: worried, sad, pain, tired, annoyed, schoolwork/homework, sleep, daily routine and ability to join in activities. Each dimension contains five severity levels, resulting in 1,953,125 unique health states that are associated with the measure. Responses from the CHU9D questionnaires were transformed into quality-of-life (utility) weights derived from a UK general population sample using an algorithm developed by Stevens et al.⁴⁶ This gives a possible utility value set of between 0.33 (worst health state) and 1 (best health state). The QALYs were then calculated for each individual child using the area under the curve method,⁹⁴ which uses the trapezium rule.

Assessing quality-adjusted life-year differences

To control for differences in baseline utility between the intervention and control arms,⁹⁴ prespecified covariates were adjusted for based on a statistical analysis plan. These were cluster-level variables, which were used in the randomisation (size of school, proportion of pupils eligible for free school meals, ethnic mix of pupils), and pupil-level factors (sex, baseline CHU9D score, ethnicity, deprivation, baseline total energy consumption and baseline PA energy expenditure).⁹⁵

Thus, three models are reported within the analysis:

1. a linear regression model
2. a multilevel regression model controlling for baseline utility
3. a multilevel regression model controlling for baseline utility and prespecified covariates.

The first model is an unadjusted model, that is, a linear regression of costs (or QALYs) on the independent intervention dummy variable. The data, however, were clustered, and, to account for this, the second model adopts a hierarchical approach to account for clustering while also controlling for baseline utility to address baseline differences. The third and final model, and the one used for the primary analysis, adds the prespecified covariates to model 2. This model, therefore, adjusts for clustering and baseline utility, as well as the covariates specified within the analysis plan. All of the multilevel models were implemented using Stata’s ‘mi estimate: mixed’ command, using maximum likelihood estimation to fit a multilevel mixed-effect linear regression including a random effect for the level 2 school variable using multiply imputed data.

Multiple imputation

Resource use data were collected at the cluster-level, whereas health outcome data were collected at the individual level; consequently, any reason for missing data (the missingness mechanism) varied for these two different types of data. The cost and QALY data were therefore imputed separately to include the relevant covariates within the imputation model and then combined to form a complete data set. During the imputation process, to account for the hierarchical nature of the data, all of the individual-level data were imputed using multilevel multiple imputation. This was implemented through REALCOM-IMPUTE software in conjunction with Stata 13. Thirty imputations were conducted, resulting in 30 complete data sets. Rubin’s rule,⁸⁴ which incorporates uncertainty around the predicted values, was used to calculate pooled estimates of the mean costs and QALYs, as well as CIs. Given the number of missing data, the base-case analysis uses the imputed data.

Sensitivity analysis methods

A series of sensitivity analyses were conducted.

Secondary analysis

Examining the impact on quality-adjusted life-years

Table 32 describes the unadjusted mean QALYs for each arm of the trial. At FU2, the intervention group accrued 2.17 QALYs, compared with 2.14 QALYs for the control group. This difference was not significant at the 0.05 significance level, given the way that QALYs are calculated using the ‘area under the curve’ method, and as highlighted in Figure 9, the extra QALYs within the intervention arm are probably due to the large imbalance at baseline.

TABLE 32

Unadjusted QALYs accrued (pre-imputation)

After conducting multiple imputation, the unadjusted QALY estimates remain similar to those pre imputation (Table 33).

TABLE 33

Unadjusted QALYS accrued (post imputation)

Incremental analysis: effectiveness

Table 34 describes the incremental difference in mean QALYs between the intervention and control group for the data with no adjustment, with adjustment for clustering and baseline differences, and with adjustments for clustering, baseline differences and the prespecified covariates using multilevel multivariate regression.

TABLE 34

Incremental difference in QALYs

When controlling for baseline utility, clustering and the covariates, the mean QALY difference for FU1 and FU2 was negligible and insignificant.

Of interest, we explored the differences in mean QALYs between the intervention and control groups by trial intervention year. Table 35 outlines the results. The coefficient for incremental QALYs in G1 schools at all follow-up points is positive, indicating that more QALYs were attained in the intervention group than in the control group. In contrast, however, the corresponding coefficient for G2 schools is negative, indicating the opposite effect, namely that fewer QALYs were attained in the intervention group than in the control group.

TABLE 35

Mean QALY difference by school group

Resource use and costs

Details of all resource use and costs for the intervention group are displayed in Tables 36–38. Compared with the running costs, the set-up and development costs of the intervention were relatively low. In terms of the set-up costs, the largest component of cost was the time attributed to research staff developing materials for the intervention, in particular the development of the cooking workshop and classroom materials. The largest cost driver for implementing the intervention related to the time of the teaching staff and the cost of staff cover for attending the training sessions. Other significant costs related to the creation of the signposting materials and the purchasing of the PA packages. The ongoing running/delivery costs had the biggest impact on the overall costs of the intervention. Of the main components of the intervention, in terms of delivery, the cheapest by far was the signposting. The most expensive component of the trial was the Villa Vitality sessions, which accounted for over half of the running costs.

Set-up/development costs

Implementation costs

Ongoing/intervention delivery costs

Missing cost data

For all of the components of the intervention for which the resource use data were collected by the research team, there were no missing data. For much of the intervention, however, the cost data were collected from logbooks and there were extensive missing data. Some teachers failed to complete the logbooks, citing time constraints; other logbooks were returned, but without completion of the requested cost data; some logbooks were completed but lost by the school; and other logbooks were reported to have been returned but were never received. Given the multifaceted nature of the intervention, there were large numbers of missing data (Table 39).

TABLE 39

Missing data by cost component

The high levels of missing data for both costs and QALYs provide a strong case for using multiple imputation.

Incremental analysis of cost

Multiple imputations (30 imputations) were run for each subcomponent of cost, and these were then combined to calculate the total costs of the intervention. Therefore, the analysis of cost was conducted on 30 data sets.

Table 40 outlines the incremental costs of the intervention arm compared with the control arm after adjusting for clustering, baseline utility and the covariates. Unsurprisingly, given the assumed ‘no costs’ associated with the control arm, the intervention arm was statistically significantly more expensive than the control arm.

TABLE 40

Incremental cost analysis

Cost–utility analysis

The CUA combines the incremental costs with the incremental QALYs to produce an ICER. The ICER associated with the base case is £46,083 per QALY and, thus, the intervention is not cost-effective. Through the net benefit framework it is possible to assess the uncertainty around the ICER, while also considering clustering and the correlation between costs and outcomes. Given the very small effect size, and the uncertainty around the effect size, it is unsurprising to find extremely large levels of uncertainty around the ICER. At the NICE-recommended willingness-to-pay (WTP) threshold of £20,000–30,000 per QALY, the control arm is more likely to be cost-effective than the intervention arm. At the lower threshold of £20,000 per QALY, there is just a 30% chance that the intervention is more cost-effective than usual practice. Even at a WTP threshold of £100,000 per QALY, the intervention arm has only a 59% chance of being the more cost-effective option. The reason underlying this uncertainty can be attributed to the lack of effect of the intervention. As the effect size approaches zero, the CIs around the net benefit widen.

Figure 10 shows the net benefits associated with the intervention at different levels of WTP. As the WTP threshold increases, the CIs widen, showing the increasing levels of uncertainty.

FIGURE 10

Net benefit of intervention at differing WTP levels.

This is reflected in the CEAC (Figure 11), which shows the probability of the intervention being cost-effective at different levels of WTP for a QALY. Owing to the negligible treatment effect, even at high levels of WTP per QALY there is only a slightly better than 50 : 50 chance that the intervention is the more cost-effective option.

FIGURE 11

Cost-effectiveness acceptability curve.

Sensitivity analysis 1: different multiple imputation method

When an alternative imputation strategy that utilised a fixed effect for clusters was applied, there was little impact on the results (Table 41). The overall ICER reduced to just under £42,000 per QALY; however, the estimates remained extremely uncertain, given the small and inconsistent effect size. This is reflected in the CEAC shown in Figure 12.

TABLE 41

Incremental QALYs, adjusted for clustering, baseline utility, covariates, with alternative imputation strategy

FIGURE 12

Cost-effectiveness acceptability curve with alternative fixed-effect imputation strategy.

Sensitivity analysis 2: including set-up and implementation costs

As expected, the inclusion of set-up and implementation costs led to an increase in the ICER associated with the intervention and an even less favourable CEAC. The addition of the set-up and implementation costs increased the mean costs by £43.66, increasing costs to £311.07 per child. This increase in cost had a notable impact on the ICER, increasing the ICER to £53,610 per QALY.

The CEAC (Figure 13) shows the impact of the higher levels of cost moving the CEAC downwards, especially at the lower levels of WTP than with the base-case analysis. Again, because of the lack of effect of the intervention, there is a great deal of uncertainty surrounding the results.

FIGURE 13

Cost-effectiveness acceptability curve: including set-up and implementation costs.

Sensitivity analysis 3: including wider costs

As with sensitivity analysis 2, the inclusion of wider costs results in the ICER rising to just under £52,000 per QALY. As shown by the CEAC in Figure 14, there was little chance that the intervention is cost-effective. At a WTP of £20,000 per QALY, there is just a 26% chance that the intervention is more cost-effective than usual care. When wider costs are considered, the intervention remains not cost-effective.

FIGURE 14

Cost-effectiveness acceptability curve: including wider costs.

Sensitivity analysis 4: best-case scenario

In this scenario it is assumed that there are 30 children in every class; therefore, the cost has reduced to £155.53 per child. This produced a more favourable cost-effectiveness result, with the ICER falling to approximately £26,804 per QALY. Although this would be borderline cost-effective, it is again important to note the uncertainty around this best-case estimate, as demonstrated in the CEAC in Figure 15. Again, because of the lack of treatment effect within the trial, there is a great deal of uncertainty surrounding all cost-effectiveness results. Even at a WTP of £100,000 per QALY, there would be only a 62% probability that the intervention is more cost-effective than usual care.

FIGURE 15

Cost-effectiveness acceptability curve: best-case scenario.

Cost per obesity case prevented

The purpose of the intervention is to prevent obesity and it is therefore important to consider the intervention in terms of its success at preventing children transitioning to an obese state. In the control arm, 7% of the children transitioned to an obese health state at FU2, having been in a non-obese state at baseline. In contrast, 10% of those in the intervention arm transitioned into an obese state in the intervention arm. That is, more children in the intervention arm transitioned into an obese state than those in the control arm. This is reflected by the results of the multilevel logit model. To demonstrate the lack of effect of the intervention, the odds ratio associated with children in the intervention arm transitioning into an obese state is 1.17 (95% CI 0.66 to 2.09) when controlling for covariates. This indicates that those in the intervention arm are more likely to transition into an obese state than those in the control arm. It should be noted, however, that this is not a significant difference. This negative finding, however, makes it impossible to assess the cost per obesity case prevented as a result of there being zero cases of obesity prevented by the intervention. These results reflect the primary trial analysis of the health outcomes data, which indicates that there was no notable impact of the intervention on clinical outcomes.