In the UK, most patients with AAA, PAD, CAD, VLU and VVs are treated by the NHS’s secondary care services. Decisions on how and when to treat patients with these vascular conditions are subject to variation across the UK. Understanding the impact that the disease has on the quality of life (HRQoL) of patients can help both clinicians and patients to make treatment decisions. HRQoL is central to the individual patient’s experience of health and disease; measures of HRQoL must do more than describe a patient’s health in terms of what health professionals and society believe constitutes health.174
One method of collecting data about HRQoL is using PROMs. PROMs are made of structured questions that ask patients about their health and HRQoL from their point of view.175 PROMs provide information about the impact of a disease, or its associated treatment, from a patient’s perspective.
A key objective of the research programme was to identify or develop PROMs that could be collected from users of vascular services in an electronic form, to improve and monitor their treatment and to evaluate the services that they receive. This section describes an overview of the development of the ePAQ-VAS that consisted of four key stages and followed the process recommended in the US Food and Drug Administration (FDA)’s framework for PROM development.176 A substudy was also undertaken to generate utility values of vascular patients from the EQ-5D.
Methods
Stage 1: themes generation
The various themes for the ePAQ-VAS were generated from systematic literature reviews of existing outcome measures and qualitative evidence (see Reviews of published literature).
Qualitative interviews with vascular patients
In addition, semistructured interviews were conducted with five vascular patient groups: patients with PAD, AAA, CAD, VLU and VVs. Users of vascular services attending the vascular department, Sheffield Vascular Institute (SVI) at the Sheffield Teaching Hospitals were recruited using purposive sampling techniques to ensure a range of participants at different ages, sex and stages of treatment. A clinician approached each patient and, following informed consent, a date and time were agreed for one of three researchers to visit the participant at home to carry out an interview [ethics approval: Yorkshire and Humber – Bradford Leeds Research Ethics Committee (REC) number 14/YH/1117 on 25 September 2014; an amendment to the interview schedule was approved on 18 June 2015]. The interview protocol, informed by the prior literature reviews, explored the signs, symptoms and impact that the condition had on function and the lifestyle (see Appendix 2).
Field notes were taken to aid interpretation and each interview was recorded and transcribed verbatim. Anonymised transcripts were entered into NVivo 10 (QSR International, Warrington, UK) for management and analysis using standard framework analysis techniques.177 Between 10% and 20% of the interviews were double coded by a second researcher in NVivo 10. Inter-rater comparisons were calculated using Cohen’s kappa. Regular meetings were set up with an experienced qualitative researcher (AT) to review the frameworks and guide the analysis.
Consensus exercise with clinicians
Clinicians involved in the care of vascular patients were invited to list the key issues, the symptoms and the impact that AAA, PAD, CAD, VLU and VV had on patients suffering with these conditions. The evidence from this initial round was used to inform the qualitative evidence synthesis.
Stage 2: item generation and scale development
Items under the themes above were generated by a new PROMs Steering Committee (SCR, GJ, PP, EL and AA) to establish the initial version of the ePAQ-VAS. Face validity was assessed by separate exercises with clinicians and patients.
A different group of clinicians involved in the care of vascular patients were invited to score the relevance of items (questions) in the provisional version of the ePAQ-VAS. Participants were invited to rate the appropriateness of each question on a five-point Likert scale of ‘strongly disagree’ (0) to ‘strongly agree’ (4). This process was repeated, with members of the clinicians’ panel being presented with the aggregate findings of the previous round and again asked to score each question. This process examined the relevance of each item from the clinicians’ perspective and identified new items suggested by the clinicians.178
Based on the work reported above, the Steering Committee employed an iterative process to incorporate evidence from the systematic reviews of qualitative studies, the qualitative study and the clinicians’ consensus study, as recommended by the FDA,176 to develop the initial item pool for the ePAQ-VAS.179 A draft instrument was then adjusted based on further service users’ input and review by the steering committee.
Face validity of the electronic Personal Assessment Questionnaire – Vascular
Semistructured interviews were conducted with 19 patients, purposefully sampled from the five key clinical areas identified in workstream 2. A paper and/or electronic ePAQ-VAS (version 1) was presented to participants and a focused interview was conducted to investigate vascular patients’ perceptions of the whole questionnaire and individual items. Questions were asked under the headings of overall impressions, clarity, relevance and emotional response. Interviews were audio-taped, transcribed and analysed. A pragmatic approach was used for the analysis, with comments collated and presented back to the Steering Committee, which made consensus decisions on revisions to the ePAQ-VAS.
Stage 3: reducing the items, confirming the domain structure and scoring algorithms of the electronic Personal Assessment Questionnaire – Vascular
Participant recruitment
To reduce the items and confirm the conceptual framework of the ePAQ-VAS generated in stage 1, a quantitative survey was undertaken. Consecutive patients attending outpatient clinics run by SVI between June 2017 and June 2018 were invited to complete the questionnaire online before their clinic appointment using an established encrypted voucher system. Patients who had not completed the questionnaire online, and inpatients, had the option to complete the questionnaire using electronic tablet computers or computer terminals on site. The five key vascular conditions that had been identified (AAA, CAD, PAD, VLU and VV) were all represented in the sample.
Statistical analyses
The sample size calculation was based on previous studies suggesting that a ratio of at least 4–10 respondents per item would enable factor analysis and internal reliability calculations.180,181 Because 55 items in the ePAQ-VAS contributed to eight scales, up to 550 patient completions were considered necessary.
A one-factor confirmatory factor analysis model for ordinal data182 was fitted to each of the eight scales to identify their structure and enable item reduction. Ordinal items were regressed on the domain factor by regressions estimated by a robust weighted least squares estimator with mean and variance adjustment.183 Appropriateness of the confirmatory factor analysis model for each domain was assessed by examining the comparative fit index (CFI) and the root-mean-square error of approximation (RMSEA), where CFI of > 0.95 and RMSEA of < 0.08 were regarded as an appropriate fit.184 Furthermore, items factor loadings (> 0.4), model residual correlations and modification indices were considered to examine local dependence within domains.184 The magnitude of these three indices was evaluated in comparison with other items in the scale. When the modification indices (MI) were > 100 and residual correlations (RCs) were > |0.10| this was taken as the indicator of lack of fit and items were removed from the scale.184 Following confirmatory factor analysis, redundant items were removed. Mplus version 8.2 (Muthén & Muthén, Los Angeles, CA, USA) was used for the statistical analyses.
Internal consistency reliability measurement was used to investigate the reliability of the remaining items in each domain. The Cronbach’s alpha coefficient was calculated for each domain to measure internal reliability. A Cronbach’s alpha score of ≥ 0.70 was considered acceptable; however, scores exceeding 0.92 were taken to indicate that items in the scale may be redundant.82,185,186
Stage 3: further psychometric testing of the electronic Personal Assessment Questionnaire – Vascular
To further establish the psychometric properties of the ePAQ-VAS, the results of relevant items from a previous survey and an additional survey were used to assess test–retest reliability, construct validity and the responsiveness of the measure186 (see Report Supplementary Material 12).
Participants’ recruitment
All consecutive patients invited to outpatient clinics run by SVI from June 2018 to January 2019 were asked to participate in this study using the same voucher arrangements as for the conceptual framework analysis (CFA) study. For test–retest reliability, patients were asked to complete a second questionnaire 3–7 days later, provided that there was no change to their health status. Only patients with AAA, PAD, VLU and VV were included in this survey because CAD patients were available around the time of admission for treatment only. For responsiveness testing, patients completed the ePAQ-VAS before and 6 weeks after PAD and VV procedures. The second survey for test–retest reliability and responsiveness was administered by telephone by one of the researchers.
Statistical analyses
Scoring
Summated rating was used to score the results of the eight scales in the ePAQ-VAS and was standardised to a 0–100 scale, where 0 indicates the best HRQoL and 100 the worst HRQoL. Skipped items were allocated a score of 0; the electronic format of the instrument allows patients to skip items that are not relevant to them, so it was assumed that patients skipping any items have no problems relating to these items.
Test–retest reliability
Intraclass correlation coefficients (ICCs) were used to assess test–retest reliability. ICCs exceeding 0.7 are generally regarded to indicate reliability for population-based research and ICCs exceeding 0.9 are considered to indicate reliability for use clinically with individuals.15,84,187
Known group validity
Known group validity was examined using hypothesis testing to examine whether or not the scales correlate well with expected clinical group differences. Correlations are considered low if r is < 0.3, moderate if r lies between 0.30 and 0.49 and high if r is > 0.5.188 Hypotheses were stated a priori, including the postulated direction.78,188
Responsiveness
Responsiveness was measured using standardised effect size, calculated as the change in score between post intervention and baseline divided by the standard deviation (SD) at baseline. An effect size of 0.30–0.49 is regarded as ‘small’, an effect size of 0.50–0.79 as ‘moderate’ and an effect size of ≥ 0.80 as ‘large’.188 The standardised response mean was also calculated by measuring the mean difference between baseline and post intervention divided by the SD of the change. Similar cut-off points as for the standardised effect size above were used. Statistical analyses were performed using SPSS version 24 (IBM Corporation, Armonk, NY, USA).
Substudy EuroQol-5 Dimensions, five-level version, utility values
The purpose of this substudy was to generate utility values for use in economic evaluation models.189 The EuroQol-5 Dimensions, five-level version (EQ-5D-5L), was incorporated into the ePAQ-VAS and participants were asked to complete an electronic version of the EQ-5D and the EQ-5D visual analogue scale. A series of clinical health states were identified by the steering group, in each of the five vascular categories, as being representative of specific vascular conditions and identifiable from the ePAQ-VAS responses. Utility values based on the EQ-5D-5L responses were calculated for each of these clinical states using the van Hout et al.190 mapping function.
Results
Stage 1: themes generation
Overlapping themes were identified from the literature reviews for the various vascular conditions. For PAD, there were six main themes: symptoms, physical functioning, impact on social functioning, psychological impact, financial impact and process of care. For AAA, the four overarching themes identified were symptoms, functional outcomes, psychological outcomes and social outcomes. For CAD, the five main themes were anxiety, impact on personal roles and activities, effect on independence, psychological impact and symptoms. For VV, the five main themes were physical impact of VV, psychological impact, social impact, adapting to VV and reasons for seeking treatment. For VLU, the main themes were physical impact, psychological impact, social impact and treatment.16,19,21–24
Qualitative primary study with vascular service users
Out of 111 patients invited to participate in the primary qualitative study, 56, aged 35–77 years, were interviewed to explore the impact of vascular disease on daily living: 13 patients with AAA, nine with CAD, 14 with PAD, 10 with VLU and 10 with VV. The framework analysis of the primary data identified six overarching themes relating to the impact of these five vascular conditions. These were symptoms, impact on physical function, social impact, psychological impact, financial impact and lifestyle. Identified signs, symptoms and impact of the conditions were then mapped and tabulated to see which themes were relevant to which condition and where the similarities and differences lay.179
Pain and mobility were the most commonly reported domains by participants with PAD. The extent to which they had an impact on HRQoL was associated with the severity, age expectations and social support. Fear of the symptoms worsening and amputation were also reported. Most participants with AAA reported no physical symptoms, with a small number reporting abdominal pain or pain in their legs. Uncertainty, anxiety and fear of rupture and death appeared to have the greatest impact on HRQoL.
Patients with CAD had the widest range of signs and symptoms, reporting nine different issues. The condition had the least impact on physical and social function, although some participants reported a sense of worry and anxiety, which was most often related to fear of having a major stroke.
Most of the participants reported that VV had little impact on their overall HRQoL. Pain was reported by 8 of the 10 participants; one reported significantly diminished HRQoL owing to reduced mobility. The perceived unpleasant appearance of the VV had the greatest psychological impact and was described in detail by several of the group. Many of the participants had had their VV for very long periods of time, often just ‘putting up with it’ for numerous years before seeking help.
The impact of VLU on HRQoL differed in the group. For some there were no major issues and having a VLU was accepted as part of their current life, with the hope that it would heal eventually. For others, the impact was far more significant. Pain was reported by six of the group and in some participants was quite severe, was quite severe, leading to a significantly reduced HRQoL. Pain also had a bearing on people’s mobility and their ability, or desire, to go out and socialise. Sleep was also often disturbed because of pain. The persistence of VLU had resulted in participants suffering for long periods of time. In addition, the non-healing or recurring nature of the condition had a significant impact for many. VLU had a significant psychological impact, causing some a high degree of distress.
There were many overlapping domains between the conditions. Pain was experienced in varying degrees within and across the conditions. Many of the participants had more than one vascular condition that contributed to some of the overlap in reported symptoms. For instance, many participants with AAA reported symptoms of claudication. Comorbidities were common, particularly among patients with PAD or VLU, who described the cumulative impact that the other conditions had on their HRQoL and functioning. Overall, those patients with VVs were the youngest and reported the least impact on HRQoL. Age expectations were evident in all groups; some of the older participants reported that they expected ill health because of their age, whereas younger people often reported being much more distressed by the condition.
In total, 13 vascular clinicians reviewed the items and they scored on the relevance of the items in a two-round consensus exercise. Clinicians’ rating generally aligned with the issues identified by patients, but some additional items were added, particularly relating to specific symptoms of swallowing difficulty for CAD and abdominal pain or throbbing for AAA.
Stage 2: item generation and scale development
Based on the results above, the items of the ePAQ-VAS were arranged into four sections: generic, AAA, CAD and lower limb vascular conditions. The generic section included the EQ-5D-5L instrument that had been identified in the reviews (see Reviews of published literature) and through Steering Committee discussions as being the most appropriate generic measure. A single lower limb section was developed as common themes were identified for PAD, VLU and VVs affecting the lower limb, regardless of whether the underlying pathology was venous or arterial. An inclusive approach to development was used and a comprehensive questionnaire was produced with 168 questions.
Face validity of electronic Personal Assessment Questionnaire – Vascular
Overall, the response to the semistructured patient interviews was favourable, with participants reporting that they felt that it was fit for purpose and potentially useful. There was little consistency in items that participants found difficult. No single item was identified for which many participants had a difficulty. Issues raised included the use of abbreviations; font size and contrast between text and background; response options and scales; electronic format versus paper format; relevance to patients and clinicians; use of free-text boxes and the language and wording used; when and how to use the skip button; repetition of items and subject matter; and the possibility of emotional distress associated with questions about the possibility of deterioration or death.
In total, 59 items were eliminated owing to overlap and five items were added, as requested by clinicians, because they were relevant to clinical management. Generic items and the EQ-5D-5L questionnaire were included in the first section and all respondents answered questions about pain, altered sensation, weakness, weight/height, smoking habit, medical history and regular medication.
The subsequent three sections are condition specific and relate to CAD, AAA and lower limb vascular disease, each of which is further divided into scales. There were 55 items in eight scales in addition to the generic EQ-5D-5L instrument. The remainder of the questions did not contribute to scales but were kept because of their clinical relevance. Individual items, scales and sections and the source of the evidence for their inclusion have been detailed, along with illustration of the structure of the questionnaire following further revisions.179
Stage 3: reducing the items, confirming the domain structure and scoring algorithms of the electronic Personal Assessment Questionnaire – Vascular
The first version of the ePAQ-VAS that was presented to patients had 114 items, of which 55 items contributed to eight factors. The other questions were either clinically relevant questions (i.e. asking about smoking, weight, etc.) or screening questions to ensure that only relevant questions are presented to the patients based on their specific vascular complaint. The ePAQ-VAS was completed by 638 vascular patients; 159 patients (24.9% of the total) completed the questionnaire online before their clinic appointment and the remainder were recruited face to face in the outpatient clinic and the vascular inpatient ward. The majority (65.2%) of patients were male. All of the patients were asked to complete relevant sections of the generic dimension. Patients were presented only with relevant disease-specific scales. In total, 49 patients completed CAD questions, 112 respondents answered the AAA dimension, 323 completed PAD-related questions and 117 and 172 patients completed the VLU and VV questions, respectively.
The CFA models were developed to test the a priori model for the eight-scale structure.191 In the CAD section, two scales were modelled and all items in the ‘CAD-related anxiety’ section were relevant to measure the latent factor. However, two items were dropped from the ‘Impact of CAD on activities of daily living (ADL)’ domain. The first was a generic item about the impact of CAD diagnosis on enjoyment of life, which had high MI and RCs with two other items. The other asked about the impact on mood and had low factor loading. In the AAA section, only one item was deleted because of high MI and RCs with two items on the same scale, as was a similar item asking about the impact that lower limb symptoms have on the enjoyment of life. Items with a low factor loading relating to ‘cold feet’ and VLU symptoms domain were also dropped. The opinion of clinicians was sought before deleting these items.
Following item reduction, the internal consistency of each scale was examined; all scales had a Cronbach’s α coefficient of ≥ 0.70 but none exceeded 0.92. The results suggest that the items in each scale measure the same latent construct. For CFA model fit statistics see and Aber et al.186
Confirmatory factor analyses of the ePAQ-VAS
Stage 4: further psychometric testing of the electronic Personal Assessment Questionnaire – Vascular
In total, 721 patients completed the ePAQ-VAS. The mean age of the participants was 63.45 years (SD 15.7 years) and 64.9% were men (n = 468). A total of 76% of patients (n = 553) completed the questionnaire in a clinical environment (clinic or ward) and the remaining patients completed the questionnaire online before their clinic. Scores were calculated for each scale ().
Number of respondents and mean score for each scale of the ePAQ-VAS
Test–retest reliability
For the test–retest survey, 150 patients (60 with PAD, 39 with VLU and 51 with VVs) completed the relevant sections of a second questionnaire after 3–7 days. Test–retest results were calculated for the symptom scale and the impact that lower limb vascular disease had on ADL for patients with PAD, VLU and VVs separately. The ICC ranged from 0.59 for anxiety related to AAA to 0.94 for the impact that VLU had on ADL.179
Known group validity
Correlations between the proposed clinical hypotheses and the CAD impact on ADL and CAD-related anxiety scores were low, ranging from –0.089 to 0.094. The correlation between size of AAA and AAA-related anxiety score was significant, and this scale captured this latent variable. There was no significant correlation between size of AAA and AAA impact on ADL. There was low correlation between AAA-related anxiety and AAA impact on ADL in preoperative patients. There was a significant correlation between rest pain and PAD symptoms and impact of PAD on ADL. The presence of an ulcer had a statistically significant correlation with the score of PAD impact on ADL.
Ulcer recurrence had a significant correlation with VLU symptom scale score. The correlation of ulcer recurrence with other scales was weak. The presence of VV with VLUs had no significant correlation with any of the scales’ correlations. The presence of VV in both legs had a significant correlation with VV symptoms only, and the presence of VV in both legs did not have strong correlations with scores of VV impact on ADL. Some of the results of known group validity are in line with proposed clinical hypotheses, for instance, ‘the larger the size of AAA, the greater the anxiety caused by the condition’ and ‘the presence of rest pain or an ulcer has a significant impact on PAD scales scores’. However, some clinical hypotheses, particularly in relation to CAD scale scores, were not in line with what was proposed. This could be because of the small size of this disease group.
Responsiveness
In total, 55 patients with VV undergoing VV surgery and 37 patients undergoing lower limb revascularisation procedures for PAD completed the ePAQ-VAS preoperatively and once more at least 6 weeks after their procedure. The effect size and standardised response mean were measured for all of the relevant scales of the ePAQ-VAS ().179
Effect size for the ePAQ-VAS measuring responsiveness
Substudy: EuroQol-5 Dimensions, five-level version, utility values
The average EQ-5D-5L derived utility values ranged from 0.396 for PAD patients with a major amputation to 0.886 for VV patients following treatment ().
EQ-5D-5L utility values for vascular health states
Discussion
This section documents the steps undertaken to develop the ePAQ-VAS, a new tool to collect PROMs and other clinical information from users of vascular services. This can be used as a holistic clinical assessment tool to be completed by patients before seeing a clinician and during follow-up. The information generated can be used to help shared decision-making and to monitor the impact of intervention on the patients’ quality of life. As an electronic online measure it can potentially be completed remotely, which facilitates virtual clinics,192 and may contribute to electronic patient records or aggregated data for service evaluation. A demonstration version of the final version of the web-based form is available online at www.epaq.co.uk/Demo/VascularDemo (accessed 1 December 2020).
A major strength of the work is that the instrument has been developed through a rigorous process in line with FDA guidelines for developing PROMs.176 Evidence from this study suggests that scales in the lower limb section of the ePAQ-VAS have good test–retest reliability, particularly the PAD and VLU scales. The results of the known group validity show that the AAA-related anxiety scale correlates with the size of AAA. Patients’ anxiety is increased and the impact that the disease has on patients’ daily living is greater for preoperative patients than for those under surveillance. The instrument shows good correlation between rest pain in PAD and increased PAD impact on ADL scores. The results of the survey also confirmed that the presence of an ulcer with PAD diagnosis increased PAD impact on ADL scores, and recurrent ulcer is associated with an increase in ulcer symptom scores. The results of the responsiveness analyses show that the following scales can detect changes following intervention: AAA-related anxiety, PAD symptoms, impact of PAD on ADL, VV symptoms scale and impact of VVs on ADL.
Another strength of the work is the inclusion of the EQ-5D-5L alongside disease-specific scales. This allows the generation of utility values for the different health states that can be used in cost-effectiveness modelling, in line with NICE’s methodology.193 It may also facilitate further research to consider the relationship between such generic measures and the more detailed symptomatic and disease-specific description of vascular conditions provided by the new instrument.
These studies had a number of limitations. For pragmatic reasons, the development and validation were largely based on evaluation at a single site. Sample size was small for some groups of patients, because of resources and patient availability, particularly for patients with CAD and AAA. Access to some patient groups before and after treatment was difficult and the fact that some forms were completed online, some in outpatient clinics and some by telephone may have been a confounding factor. Furthermore, the sample size for responsiveness and test–retest was small for all groups, and future analysis with a larger sample size is desirable. The follow-up data for test–retest and responsiveness were collected by telephone, which may have introduced some bias. Further studies are also required to explore acceptability and completion rates, to consider the practicalities of wider implementation and to refine aspects of the process, such as reporting formats and the analysis of longitudinal and aggregate data.
Overall, the ePAQ-VAS provides a holistic data collection process that is relevant to most vascular service users and has the potential to contribute to patient-focused care and the collection of aggregate data for service evaluation.
