Characteristics and quality risk of bias of outcome evaluations

Chris Bonell; Kelly Dickson; Kate Hinds; GJ Melendez-Torres; Claire Stansfield; Adam Fletcher; James Thomas; Katrina Lester; Elizabeth Oliver; Simon Murphy; Rona Campbell

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Bonell C, Dickson K, Hinds K, et al. The effects of Positive Youth Development interventions on substance use, violence and inequalities: systematic review of theories of change, processes and outcomes. Southampton (UK): NIHR Journals Library; 2016 May. (Public Health Research, No. 4.5.)

Appendix 16Characteristics and quality risk of bias of outcome evaluations

TABLE 19

Characteristics of outcome evaluations (studies included in the review to answer RQ3)

Study details	Characteristics of outcome evaluations
Berg et al.⁸⁶
Methods	Non-randomised trial: matched comparison group
Unit of allocation	Individual
Participants’ details	Country: USA Sample number: 114 (IG); 202 (CG) Sex: 51% female, 49% male in intervention group Ethnicity: 47% African American/black; 41% Latino; 12% mixed/white/newly arrived SES: not stated
Intervention details	Description: YARP Targeted/universal: targeted Target population: youth aged 14 to 16 years Provider(s): prevention research educators Theory: prevention, cognitive and critical theory including: (1) ecological theory; (2) identity theory; (3) learning and instructional theories (including social learning, social construction and multiple intelligence); and (4) critical, transformative theories including voice, empowerment and action research Training: educators are trained in action research methods and empowerment-oriented, social construction facilitation skills Setting: community-based after-school and summer programme Content: participatory action research involving formative community ethnography (e.g. training youth to identify adolescent risk behaviours, develop a collective action plan and carry out activities) as a group, including using research to understand their community better Length/intensity: 4 hours/day for 8 weeks, 8-month implementation period for action plan Control: measurement
Outcomes	At post intervention (less than unity favours intervention): Frequency of marijuana use in past 30 days: r = –0.12; p-value = 0.053 Frequency of alcohol use in past 30 days: decrease but not significant
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Not applicable	This was a non-randomised trial
Allocation concealment: was the allocation adequately concealed?	Not applicable	This was a non-randomised trial
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	This was a non-randomised trial
Incomplete outcome data: were complete data for each outcome reported, and, if not were adequate reasons for incomplete outcome data provided?	No	By outcome: reasons for dropout not reported, and all drug-use outcomes not reported 17.4% attrition overall Reasons for dropout not reported; however, authors report no differences between groups on dropout characteristics
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Yes	The evaluation discussed collecting outcome data on a variety of substance use measures, but only one is presented in sufficient detail to calculate an effect size
Key confounders: Were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Yes	Intervention and control samples matched, and differences between groups used as covariates in analyses
Clustering: was clustering of participants accounted for in the analysis?	No	No evidence of accounting for clustering
Other source of bias: were there any other sources of bias that might affect the results of the study?	No
Cross et al.⁹⁰ (linked studies: Gottfredson et al.⁹¹^,⁹²)
Methods	RCT
Unit of allocation	Individual
Participants’ details	Country: USA Sample number: 224 (IG); 223 (CG); analysed: 211 (IG); 205 (CG) Sex: 48% female, 52% male Ethnicity: 71% African American/black; 17% white; 8% multiracial SES: 58% received subsidised school meals
Intervention details	Description: the All Stars prevention curriculum: an enhanced ASP Targeted/universal: universal Target population: pupils aged 11–14 years from five under-performing middle schools Theory: not stated Provider(s): a county-level government agency that specialising in providing recreation and leisure activities for youths Training: 6 hours in tutoring model, 19 hours in running ASP, 3 days in All Stars Setting: middle schools Content: leisure activities (e.g. fitness activities, board games, arts and crafts, field trips, computer projects or computer free time, service learning, workforce skills and holiday or other special event celebrations) Length/intensity: 3 days per week, for 3 hours Control: monthly ‘fun activity’ at programme
Outcomes	At post intervention (less than unity favours treatment): Any use of alcohol, cigarettes or marijuana in the past month: OR 1.03; p-value = 0.91 Smoking initiation: OR 0.95; p-value = 0.90 Drinking initiation: OR 1.10; p-value = 0.75 Marijuana initiation: OR 0.72; p-value = 0.47 Inhalant initiation: OR 0.73; p-value = 0.17 Other drug initiation: OR 1.73; p-value = 0.40
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Yes	Random number generator used
Allocation concealment: was the allocation adequately concealed?	Unclear	No information provided
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	Unclear	No information provided
Incomplete outcome data: were complete data for each outcome reported, and, if not were adequate reasons for incomplete outcome data provided?	No	Sample restricted to post-test questionnaire completers; dropout characteristics analysed but missing data on post-test questions as well 7% attrition overall
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Unclear	We could not assess this with the given information
Key confounders: Were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Not applicable	RCT
Clustering: was clustering of participants accounted for in the analysis?	Yes	Analysis controlled for site
Other source of bias: were there any other sources of bias that might affect the results of the study?	Yes	Unbalanced missing and differential follow-up patterns are present
Gottfredson et al.⁹³
Methods	RCT/non-randomised trial
Unit of allocation	Individual and group
Participants’ details	Country: USA Sample number: 239 in IG?, 201 in CG? (only older children) Sex: 44% female, 56% male in IG; 50% female, 50% male in CG Ethnicity: 76% non-white in IG, 53% in CG SES: no information
Intervention details	Description: Maryland After School Community Grant Program Targeted/universal: universal Target population: elementary- and middle-school children Theory: not stated Provider(s): ASP providers Training: 6 hours in tutoring model, 19 hours in running ASP, 3 days in All Stars Setting: public schools, community centres Content: three activity areas: (1) academic assistance; (2) social skill; or (3) character development, and recreational/leisure activities aimed at retaining young people in the programme. Recreational activities mainly included: sports, arts and crafts, in additional to specialty activities, such as entrepreneurial activities, karate, sailing, or soccer Length/intensity: 3 days/week for 90 sessions (middle school), 4 days/week for 120 sessions (elementary school); average of 3 hours/session Control: waiting list or community-recruited comparison
Outcomes	At post intervention: Past year variety of drug use: IG: pre-test 0.036 (SD 0.119, n = 234) post-test 0.038 (SD 0.127, n = 222) CG: pre-test 0.053 (SD 0.146, n = 199) post-test 0.086 (SD 0.198, n = 173)
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Not applicable	Non-randomised trial
Allocation concealment: was the allocation adequately concealed?	Not applicable	Non-randomised trial
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	Non-randomised trial
Incomplete outcome data: were complete data for each outcome reported, and, if not were adequate reasons for incomplete outcome data provided?	No	Dropout characteristics analysed, uneven attrition and inadequate treatment of missingness (e.g. imputed means were not reported)
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Unclear	We could not assess this using the information provided
Key confounders: Were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Yes	Differences between conditions tested across randomisation and non-random assignment
Clustering: was clustering of participants accounted for in the analysis?	No	No evidence of accounting for clustering
Other source of bias: were there any other sources of bias that might affect the results of the study?	No
Kuperminc et al.⁹⁵
Methods	Non-randomised trial
Unit of allocation	Individual
Participants’ details	Country: USA Sample number: 86 (IG), 89 (CG) (additionally, 29 girls assigned to mentors) Sex: 100% female Ethnicity: 76% non-white in IG, 53% in CG SES: 80–96% of students attending programme schools lived in economic disadvantage
Intervention details	Description: Cool Girls, Inc. Targeted/universal: universal Target population: young girls aged 9 to 15 years from disadvantaged and low-income communities Theory: not stated Provider(s): community-based organisation Training: not stated Setting: public schools Content: three main components: (1) Girls Club; a comprehensive life skills curriculum covering a range of topics (e.g. positive sexual health, hygiene, conflict resolution, self-esteem and cultural awareness); (2) Cool Scholars: providing homework assistance, individual tutoring and supporting girls to complete projects (e.g. making presentations, journaling and participating in academic tournaments). Participants are also eligible for (3) ‘Cool Sisters’, a one-to-one mentoring programme, after actively taking part in the programme for 1 year. Additional components include: weekend workshops (e.g. computer skills, financial literacy and career development); field trips (e.g. tours of the Cable News Network, the Weather Channel and visits to local museums); special events; and summer programmes Length/intensity: weekly meetings during the school year Control: comparators nominated by participants and schools
Outcomes	At post intervention (greater than unity favours intervention): Avoidance of drug use (cigarettes, alcohol, marijuana, cocaine, or inhalants) in past 6 months: OR 0.62 (ns) for full sample; OR 1.53 (ns) for mentored
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Not applicable	Non-randomised trial
Allocation concealment: was the allocation adequately concealed?	Not applicable	Non-randomised trial
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	Non-randomised trial
Incomplete outcome data: were complete data for each outcome reported, and, if not were adequate reasons for incomplete outcome data provided?	Yes	Dropout characteristics analysed, uneven attrition and inadequate treatment of missingness but multiple imputation carried out
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Unclear	Unable to assess based on provided information
Key confounders: were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Yes	Differences between conditions tested using covariates in regression
Clustering: was clustering of participants accounted for in the analysis?	Yes	Cluster effects were small and not statistically significant
Other source of bias: were there any other sources of bias that might affect the results of the study?	No
Rhodes et al.⁹⁶ (linked studies: Grossman and Tierney;⁹⁷ Tierney⁹⁸)
Methods	Randomised trial
Unit of allocation	Individual
Participants’ details	Country: USA Sample number: 487 (IG), 472 (CG) Sex: about 25% female, 75% male Ethnicity: 45% white; 42% black; 14% Hispanic SES: 37% welfare recipients, 36% incomes below poverty level
Intervention details	Description: BBBS Targeted/universal: targeted Target population: young people between 5 and 18 years of age, with minimal social skills, who live in an agency catchment area, with priority given to young people who only have one parent engaged in their life Theory: matched youth mentoring; based on sex and various other factors such as shared interest, reasonable geographic proximity and same ethnicity Provider(s): trained community volunteers Training: initial training on abuse and programme rules, additional training often provided on youth development, monthly supervision Setting: community Content: 1 : 1 weekly mentoring aimed at developing the ‘whole person’, with the mentoring relationship is seen is the mechanism that enables the mentor to support the mentee transition from childhood and/or adolescence into adulthood Length/intensity: weekly meetings with mentor Control: unmatched children
Outcomes	At post intervention: Initiation of drug abuse: CG mean 11.47%, IG less by 45.8% (p < 0.05) Initiating alcohol use: CG mean 26.72%, IG less by 27.4% (p < 0.10) Cigarette smoking: CG mean 17.2%, IG less by 19.7% (p > 0.10) Number of times hit someone: CG mean 2.68, IG less by 0.85 (p < 0.05) Number of times involved in a fight: CG mean 1.54, IG less by 0.02 (p > 0.10)
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Unclear	Not enough information was provided
Allocation concealment: was the allocation adequately concealed?	Yes	Allocation done by an external survey subcontractor
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	Not possible in this intervention
Incomplete outcome data: were complete data for each outcome reported and, if not, were adequate reasons for incomplete outcome data provided?	No	Only people with all measurement occasion included
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Unclear	Could not assess based on provided information
Key confounders: Were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Not applicable	Randomised trial
Clustering: was clustering of participants accounted for in the analysis?	No	No evidence of accounting for clustering
Other source of bias: were there any other sources of bias that might affect the results of the study?	No
Rodriguez-Planas⁷⁸ (linked studies: Maxfield;⁹⁹^,¹⁰⁰ Rodriguez-Planas;¹⁰¹ Schirm et al.;¹⁰³ Schirm and Rodriguez-Planas;¹⁰⁴ Schirm and McKie¹⁰⁵)
Methods	RCT
Unit of allocation	Individual
Participants’ details	Country: USA Sample number: 580 (IG), 489 (CG) Sex: IG: 52.2%% male, 47.8% female; CG: 55.8% male, 44.2% female Ethnicity: IG: 26.2% Hispanic, 68.3% Black, CG: 25.7% Hispanic, 67.9% Black SES: not reported
Intervention details	Description: QOP Targeted/universal: targeted Target population: youth who met the following criteria: (1) began the ninth grade with a dropout rate of 40% or more; (2) grade point average below the 67th percentile of entering ninth graders; (3) not repeating the ninth grade; (4) not so physically disabled or learning disabled that participation in programme is inappropriate Theory: youth development model Provider(s): case managers and mentors, community-based organisations Training: unclear Setting: schools and community-based organisations Content: intensive case management, mentoring and educational, developmental, cultural and recreational and community-based activities. Education services: academic assessment, an individual education plan, one-on-one tutoring and computer-assisted instruction in specific coursework and basic reading and mathematics. Visiting nearby college campuses and other activities designed to promote awareness of and planning for college. Developmental activities: life-skills training, employment-readiness training, cultural awareness and recreation. Community service activities (e.g. visiting the residents of a local nursing home or volunteering at a neighbourhood food bank). Cultural and recreational activities: movies, ice skating, bowling, swimming, sailing, golfing, mountain biking, amusement/water parks, haunted houses, board/computer games, local fairs, picnics, attending sporting events, pizza lunches, dinners in restaurants Length/intensity: 750 hours per year, or over 14 hours per week on average throughout the year for up to 5 years Control: no treatment
Outcomes	In the fourth year of the demonstration programme (i.e. post intervention): Near the end of the fourth academic year: Drinking in past 30 days: 40% (IG) vs. 33% (CG) Frequent drinking in past 30 days: 11% (IG) vs. 11% (CG) Binge drinking in past 30 days: 24% (IG) vs. 20% (CG) Frequent binge drinking in past 30 days: 7% (IG) vs. 5% (CG) Drunk or high at school in past 12 months: 20% (IG) vs. 20% (CG) Used an illegal drug in past 30 days: 34% (IG) vs. 28% (CG) Involved in gang fight in past 12 months: 16% (IG) vs. 14% (CG) Telephone survey 7 months later: Binge drinking in past 30 days: 19% (IG) vs. 23% (CG) Frequent binge drinking in past 30 days: 5% (IG) vs. 4% (CG) Used an illegal drug in past 30 days: 16% (IG) vs. 19% (CG) 3 to 4 years post intervention: Binge drinking in past 30 days: 25% (IG) vs. 31% (CG) Binge drinking on eight or more days in past 30 days: 7% (IG) vs. 5% (CG) Used an illegal drug in past 30 days: 12% (IG) vs. 18% (CG) Approximately 6 years post intervention: Smoked cigarettes or used tobacco in past 30 days: 34% (IG) vs. 34% (CG) Smoked cigarettes or used tobacco daily in past 30 days: 22% (IG) vs. 24% (CG) Binge drinking in past 30 days: 31% (IG) vs. 31% (CG) Binge drinking on eight or more days in past 30 days: 8% (IG) vs. 6% (CG) Used an illegal drug in past 30 days: 12% (IG) vs. 13% (CG)
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Unclear	No information provided
Allocation concealment: was the allocation adequately concealed?	Unclear	No information provided
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	Impossible given the intervention
Incomplete outcome data: were complete data for each outcome reported and, if not, were adequate reasons for incomplete outcome data provided?	Yes	Weighted analysis partially accounts for missingness
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Unclear	Could not assess given information
Key confounders: Were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Not applicable	Randomised trial
Clustering: was clustering of participants accounted for in the analysis?	Yes	Analysis controlled for site
Other source of bias: were there any other sources of bias that might affect the results of the study?	No
Schwartz et al.⁷⁷ (linked studies: Millenky et al.;¹⁰⁷^–¹⁰⁹ Perez-Arce et al.¹¹⁰)
Methods	RCT
Unit of allocation	Individual
Participants’ details	Country: USA Sample number: 722 (IG), 451 (CG) Sex: 88.0% male, 12.0% female Ethnicity: 18.1% Hispanic; 42.3% white; 33.8% black; 5.7% other SES: 26.4% of families receive public assistance
Intervention details	Description: NGYCP YIM programme Targeted/universal: targeted Target population: youth 16 to 18 years old who have dropped out or been expelled from school, who are drug free at the time of entry into the programme and not currently on probation or parole for anything beyond juvenile status offences, not serving time or awaiting sentencing, not under indictment or charged and not convicted of a felony or capital offence Theory: military training model that believes incorporating caring relationships with non-parental adults can contribute to a range of PYD outcomes Provider(s): National Guard Training: unclear Setting: community military-style boot camp Content; The Pre-ChalleNGe Phase is a 2-week period of orientation and assessment in which young people adjust to an intensive, structured lifestyle required at the programme site. The residential phase is a 20-week period during which youth are working towards their high school diploma or GED and take classes on life skills, health and job skills, while participating in other activities such as physical training, sports, leadership and citizenship activities and community service. The post-residential phase is characterised by a post-residential action plan in which youth identify specific post-residential activities (e.g. GED programme, community college, vocational training, a job, or military service) Length/intensity: 5 months’ full-time residential, 1 year’s post-residential with job placement and structured mentoring Control: waitlist
Outcomes	At post intervention: Charged with a violent crime: 3.4% (IG) vs. 3.6% (CG) Convicted of a violent crime: 1.4% (IG), 1.2% (CG) Any violent incidents: 54.0% (IG) vs. 57.3% (CG) Number of violent incidents: 0.9 (IG) vs. 1.3 (CG) Binge drinking in past 14 days: 2.8% (IG) vs. 4.7% (CG) Frequent marijuana use in past 12 months: 22.5% (IG) vs. 25.2% (CG) Ever used other illegal drugs (LSD, cocaine, methamphetamine, amphetamine sulphate, barbiturates, heroin, PEDs): 24.1% (IG) vs. 23.1% (CG) Frequent illegal drug use in past 12 months: 5.6% (IG) vs. 4.4% (CG) Charged with a drug crime: 2.9% (IG) vs. 5.3% (CG) Convicted of a drug crime: 1.4% (IG) vs. 1.9% (CG) At 18 months post intervention: Convicted of a violent crime: 2.1% (IG) vs. 2.3% (CG) Any violent incidents: 48.7% (IG) vs. 44.5% (CG) Number of violent incidents: 0.9% (IG) vs. 0.8% (CG) Binge drinking in past 14 days: 26.1% (IG), 30.2% (CG) Frequent marijuana use in past year: 26.0% (IG), 24.4% (CG) Ever used other illegal drugs (LSD, cocaine, methamphetamine, amphetamine sulphate, barbiturates, heroin, PEDs): 28.2% (IG) vs. 23.2% (CG) Frequent illegal drug use in past 12 months: 4.7% (IG) vs. 4.2% (CG) Convicted of a drug crime: 8.1% (IG) vs. 5.9% (CG)
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Unclear	No information provided
Allocation concealment: was the allocation adequately concealed?	Unclear	No information provided
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	Unclear	No information provided
Incomplete outcome data: were complete data for each outcome reported and, if not, were adequate reasons for incomplete outcome data provided?	Yes	Weighted analysis partially accounts for missingness, response bias tested
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Unclear	We could not assess this question given the information provided
Key confounders: were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Not applicable	Randomised trial
Clustering: was clustering of participants accounted for in the analysis?	Yes	Analysis accounted for clustering by site
Other source of bias: were there any other sources of bias that might affect the results of the study?	No
St Pierre and Kaltreider¹¹¹
Methods	Non-randomised trial
Unit of allocation	Group
Participants’ details	Country: USA Sample number: 52 (IG), 54 (IG with booster), 55 (CG) Sex: about 25% female, 75% male Ethnicity: 45% white; 42% black; 14% Hispanic SES: 37% welfare recipients, 36% incomes below poverty level
Intervention details	Description: Stay SMART is a component of the SMART moves National prevention programme of Boys and Girls Clubs of America Targeted/universal: universal Target population: 13-year-old members of Boys and Girls Clubs of America Theory: personal and social competence approach Provider(s): Boys and Girls Clubs employees Training: compulsory training on Stay SMART and yearly training of group leaders on SMART Leaders component Setting: local ‘clubhouses’ for children Content: life skills training programme includes structured small group sessions on: gateway drugs; decision-making; advertising; self-image and self-improvement; coping with change; communication skills; social skills (meeting and greeting people; and boy meets girl); assertiveness; relationships; life planning skills. The booster programme SMART Leaders added leadership sessions on: (1) orientation; improving self-image; coping with stress; resisting media pressures; being assertive in pressure situations and (2): resisting alcohol; other drugs; and early sexual activity. Sessions included culturally relevant experiential activities and videos. The video format introducing the session objectives; gave background information; and discussion questions on the video. Participants were also encouraged after sessions to become involved in other prevention activities (e.g. wearing drug-free t-shirt; being drug-free role models; helping with general club activities; helping with specific drug-prevention activities) Length/intensity: 12 sessions in 3 months; 8 sessions over 2 years added in booster sessions Control: Boys and Girls Clubs without a prevention programme
Outcomes	At post intervention for binary substance-use outcomes: Alcohol behaviour: not reported Marijuana behaviour: not reported Cigarette behaviour: not reported At 12 months post intervention for binary substance-use outcomes: Alcohol behaviour: not reported Marijuana behaviour: IG2 vs. CG: b = 0.80; p-value = 0.19 Cigarette behaviour: IG2 vs. CG: p-value = 0.91; IG2 vs. CG: p-value = 0.54 At 24 months post intervention for binary substance-use outcomes: Alcohol behaviour: not reported Marijuana behaviour: IG2 vs. IG: b = 1.09; p-value = 0.12; IG2 vs. CG: b = 1.23; p-value = 0.09 Cigarette behaviour: IG2 vs. IG: b = 0.86; p-value < 0.12; IG2 vs. CG: b = 0.96; p-value < 0.08 Chewing tobacco at all time points: unclear and unreported
Notes	Further clarification on outcomes was unavailable
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Not applicable	Non-randomised trial
Allocation concealment: was the allocation adequately concealed?	Not applicable	Non-randomised trial
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	Non-randomised trial
Incomplete outcome data: were complete data for each outcome reported and, if not, were adequate reasons for incomplete outcome data provided?	No	Only people with all measurement occasions were included
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	No	All outcomes reported, though incompletely
Key confounders: were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Unclear	Covariate results inadequately presented
Clustering: was clustering of participants accounted for in the analysis?	No	No evidence of accounting for clustering
Other source of bias: were there any other sources of bias that might affect the results of the study?	Yes	Outcomes unclear and poorly presented
Tebes et al.⁷⁹
Methods	Non-randomised trial
Unit of allocation	Group: comparator chosen by intervention programmes in neighbouring cities
Participants’ details	Country: USA Sample number: 149 (IG), 155 (CG) Sex: 47% female, 53% male Ethnicity: 75.7% African American, 19.7% Hispanic, 3.9% Caucasian SES: about 13% of parents less than high school educated
Intervention details	Description: PYDC Targeted/universal: universal Target population: 12- to 16-year-olds Theory: not stated Provider(s): community group leaders Training: 12 hours in curriculum and facilitation, bi-weekly supervision Setting: public schools Content: two core components: (1) The Adolescent Decision-Making for the PYDC (ADM-PYDC) a substance-use prevention curriculum which included: (a) programme introduction and overview; (b) understanding and coping with stress and learning stress-reduction strategies; (c) learning the steps of effective decision-making; (d) learning essential information about tobacco, alcohol and other drug use (two sessions); (e) applying the decision-making process to one’s life through identifying positive personal attributes, dealing with job and school stressors, setting positive goals for healthy living and enhancing one’s social networks and resources (four sessions); and (f) programme close and review; and (2) participation in health education and cultural heritage activities e.g. regular field trips to community agencies, civic organisations, businesses and schools to promote learning about community service and understanding one’s cultural heritage. The field trips promoted after-school experiences and access to academic and vocational support, counselling services and participating in intergenerational programmes and community theatre Length/intensity: school year Control: ASPs without PYD/adolescent decision-making components
Outcomes	At post intervention (less than unity favours intervention): Use of alcohol in past 30 days: OR 1.179, 95% CI 0.49 to 2.87 Use of marijuana in past 30 days: OR 1.759, 95% CI 0.66 to 4.68 Use of other drugs (amphetamines, cocaine, heroin, non-prescription methadone, hallucinogens, tranquillisers, inhalants) in past 30 days: OR 1.266, 95% CI 0.52 to 3.10 Use of any drug in past 30 days: OR 1.694, 95% CI 0.74 to 3.90 At 3 months post treatment: Use of alcohol in past 30 days: OR 0.365, 95% CI 0.15 to 0.90 Use of marijuana in past 30 days: OR 0.178, 95% CI 0.08 to 0.42 Use of other drugs in past 30 days: OR 0.188, 95% CI 0.08 to 0.44 Use of any drug in past 30 days: OR 0.289, 95% CI 0.013 to 0.67
Notes	ORs as expressed in study are change indices rather than controlled for baseline values
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Not applicable	Non-randomised trial
Allocation concealment: was the allocation adequately concealed?	Not applicable	Non-randomised trial
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	Non-randomised trial; research team responsible for recruiting comparison group
Incomplete outcome data: were complete data for each outcome reported and, if not, were adequate reasons for incomplete outcome data provided?	No	Dropout characteristics analysed, but missing data unimputed though missingness roughly balanced across arms
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Yes	Some outcomes (e.g. tobacco) not reported
Key confounders: Were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Yes	Propensity score matching entered into level 2 of multilevel model
Clustering: was clustering of participants accounted for in the analysis?	No	No evidence of accounting for clustering
Other source of bias: were there any other sources of bias that might affect the results of the study?	No
Wiggins et al.¹¹² (linked studies: Wiggins et al.⁴⁶)
Methods	Non-randomised trial: group matched
Unit of allocation	Group
Participants’ details	Country: UK Sample number: 1637 (IG), 1087 (CG) Sex: IG: 62% male, 38% female; CG: 56% male, 44% female Ethnicity: IG: 23% BME, CG: 20% BME SES: 73% IG, 61% CG in non-private housing; 39% and 35% in workless households
Intervention details	Description: YPDP Targeted/universal: targeted Target population: universal targeting of young people Provider(s): youth service providers Theory: youth development model Training: training of volunteers on programme requirements, recognising child abuse, working with youth; monthly supervision Setting: community youth services Content: activities focusing on young people’s health and education as well as their broader social development with specific programme content determined by the individual projects delivering services. These could include: education (literacy, numeracy, IT, vocational skills). training/employment opportunities life skills (e.g. communication, decision-making, goal-setting, relationships, negotiation, anger-management), mentoring (weekly one-to-one sessions with staff), volunteering (both career-oriented and community-based), health education (particularly sexual health, substance misuse) arts and sports, advice on accessing services (health, contraceptive, drug and alcohol services, welfare, benefits advice, counselling and advice, housing) Length/intensity: 6–10 hours’ weekly provision for 1 year Control: matched comparison
Outcomes	Post intervention (9-month follow-up, less than unity favours intervention): Cannabis use weekly or more in previous 6 months: OR 1.41, 95% CI 0.75 to 1.71 Cannabis use monthly or more in previous 6 months: OR 1.56, 95% CI 0.93 to 2.63 Drunkenness monthly or more in previous 6 months: OR 0.98, 95% CI 0.65 to 1.47 9 months post intervention (18-month follow-up, less than unity favours intervention): Cannabis use weekly or more often in previous 6 months: OR 1.97 95% CI 0.93 to 4.17 Drunkenness monthly or more often in previous 6 months: OR 1.20 95% CI 0.78 to 1.84
Risk of bias
Item	Author judgement	Description
Sequence generation: was the allocation sequence adequately generated?	Not applicable	Non-randomised trial
Allocation concealment: was the allocation adequately concealed?	Not applicable	Non-randomised trial
Blinding: was knowledge of the allocation intervention adequately prevented during the study?	No	Non-randomised trial
Incomplete outcome data: were complete data for each outcome reported and, if not, were adequate reasons for incomplete outcome data provided?	Yes	Weighted analysis partially accounts for missingness
Selective outcome reporting: were the findings of the study subject to selective outcome reporting?	Yes	Full outcomes presented at follow-up 1 not presented at follow-up 2
Key confounders: were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for?	Yes	Covariates included in model
Clustering: was clustering of participants accounted for in the analysis?	Yes	Analysis accounted for clustering
Other source of bias: were there any other sources of bias that might affect the results of the study?	No

: CG, control group; IG, intervention group; LSD, lysergic acid diethylamide; PED, performance-enhancing drug; SD, standard deviation.

TABLE 20

Quality assessment of outcome evaluations

Study	Sequence generation		Allocation concealment		Blinding		Incomplete outcome data		Selective outcome reporting		Confounder		Other sources of bias
Study	CB/SM	KD	CB/SM	KD	CB/SM	KD	CB/SM	KD	CB/SM	KD	CB/SM	KD	CB/SM	KD
Berg et al. ⁸⁶	0	0	0	0	3	3	1	1	2	2	2	2	3	3
Cross et al.⁹⁰	1	1	1	1	1	1	1	1	1	1	0	0	2	2
Gottfredson et al.⁹³	0	0	0	0	1	1	2	2	1	2	2	1	2	1
Kuperminc et al.⁹⁵	0	0	0	0	3	1	2	1	1	2	2	2	3	3
Rodriguez-Planas⁷⁸	2	1	1	1	1	1	1	1	2	2	0	0	3	3
Rhodes et al.⁹⁶	1	1	1	1	1	1	2	2	1	3	2	2	3	2
Schwartz et al.⁷⁷	1	1	1	1	1	1	1	1	1	1	0	0	3	1
St Pierre et al.¹¹¹	0	0	0	0	3	3	2	2	1	3	3	3	2	2
Tebes et al.⁷⁹	0	0	0	0	0	3	1	1	3	1	2	2	3	3
Wiggins et al.¹¹²	0	0	0	0	0	0	1	1	2	2	2	2	3	3

: 0, not applicable; 1, not stated/unclear; 2, yes; 3, no.
: All studies were coded between CB and KD except for Wiggins et al.¹¹²

Copyright © Queen’s Printer and Controller of HMSO 2016. This work was produced by Bonell et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Included under terms of UK Non-commercial Government License.

Bookshelf ID: NBK362304

Contents

< Prev Next >