Within the present study the three first phases (0–2) of the mHealth Agile Development & Evaluation Lifecycle25 were completed. In the present study phase zero will be described briefly, while phase one and two are described in more detail. Figure 6 illustrates the lifecycle, adapted to the ACTsmart development project. In phase zero, the project identification phase, the agile and user centered development method Lean UX32 and scrum methodology33 were used as project management approaches. Phase one and two were divided into five sprints. Each sprint had specific objectives and continued during ~30 days. The project leader compiled all proposed changes for the solution and prioritized among possible functionality enhancements. Early in phase one, the decision was made to build an independent, cloud-based and flexible technical solution, rather than further develop an existing platform, to maximize flexibility in the development. Also, it was decided to focus on the patient interface during phase one, and to prepare the patient interface for beta testing in phase two. Consequently, the therapist interface was still rudimentary when the beta testing/clinical feasibility trial was conducted.
The mHealth Agile Development & Evaluation Lifecycle (Wilson et al.25) adapted with permission to the ACTsmart development project.
Alpha testing is an acceptance testing preferably carried out with potential end-users. It involves simulating a real user environment by carrying out tasks that actual users might perform. The alpha test is made to identify as many potential problems as possible before releasing a product for beta testing, and can be performed on early versions of the product such as paper sketches, versions that lacks all features and on versions that is yet too unstable for reliable use. The alpha testing also gives preliminary end-user feedback, to get feedback when adjustments are still easy to make.
In phase one, nine individuals (age 19–65 years, 78% women) with complex chronic pain (potential end-users) were recruited for alpha testing from a tertiary care pain clinic after completing a standard face-to-face ACT-treatment. Alpha testing began with end-user interviews to inform personas and work flow ideas. Based on these nine interviews three personas were generated. The interface was then built based on the personas and continuously and repeatedly tested with the alpha users. See Table 4 for a detailed description of the test flow.
Six therapists were continuously involved in the alpha testing of the therapist interface which was carried out in the same way as with the patient alpha testers. See Table 4 for a detailed description of the test flow. Figure 7 shows an example of a development of one of the features in the therapist interface throughout the different phases of alpha testing.
Paper sketches showing the evolution of the list of patients in active treatment in the therapist interface during alpha testing.
The alpha testing phase is followed by beta testing (phase 2), in which the intervention is tested by real users in a real environment. In the present study, the beta testing sample consisted of 31 individuals (87% women, aged 25–57) with chronic pain (mean pain duration 19.74 years, range 0.5–40 years) with no prior exposure to behavioral treatment for their chronic pain condition, actively requesting participation in a research study on internet-delivered ACT. In the beta sample, seven out of 31 (23%) underwent a UX-interview and 16 out of 31 (52%) answered a UX-survey. Four therapists trained in ACT and behavioral treatment of chronic pain delivered treatment during the beta testing.
After both alpha (phase 1) and beta (phase 2) testing were completed, preparation of the next phase included minor adjustments identified during the sprints and debugging. See Table 4 for detailed information on the development and evaluation of phase 0–2.
ACTsmart is based on the clinical treatment program developed at Karolinska Institutet and Karolinska University Hospital during the past 18 years. To date, the ACT based treatment program for chronic pain patients has been evaluated in nine clinical trials, including five RCT’s31,34,35,36,37 with results illustrating the efficacy of the protocol. Improvements are primarily seen in functioning/disability and psychological flexibility, with mostly large effect sizes. Results are consistent across all studies, supporting the external validity of the findings.
The overarching goal of the ACTsmart treatment is to improve functioning and quality of life by increasing the participants psychological flexibility, defined as the ability to act in accordance with life values in the presence of pain and distress.38 Psychological flexibility is established through promoting greater acceptance of negative inner experiences22,31 as well as increasing ability to observe, rather than being entangled with, thoughts and engage in valued action.38,39
In treatment, participants were encouraged through content, solution design and by their therapist to redirect behaviors and shift focus from avoiding or reducing pain and distress to act in alignment with values in the presence of interfering pain and distress. Thus, patients were encouraged to engage in value-based exposure. Treatment content was divided into different themes that roughly corresponds to the core processes of ACT;38 acceptance, creating distance to thoughts, creating distance to emotional and bodily experiences, noticing and changing behaviors, self-observation, and values. Content was also categorized depending on type; educational, exercise or value-work. See Figs. 8 and 9 for screenshots of the patient interface.
Screenshot of patient interface, home screen, introduction to values, and values work section.
Screenshot of patient interface, educational content, and exercise.
The pre-defined criterion of treatment completion was the combination of (a) completing at least eight exercises, (b) having defined at least one formulated value, and (c) reporting behavioral actions towards that value. This criterion was chosen as exercises and values-work are based on experiential learning and therefore expected to be the active ingredients in treatment,40 in contrast to educational content that have the purpose of preparing for experiential learning.
The intervention was arranged in a micro-learning format, which is the combination of micro-content delivery and micro interactions that seeks to enable the user to gain knowledge and skills without risking information overload.41 Micro-content learning through a mobile device can give the user personal control and ownership of the learning process.42,43 All content in the treatment could be either read or listened to, in order to accommodate different preferences and needs.
The present study was part of an open-label pilot trial with one intake. The pilot trial was approved by the Regional Ethics Committee in Stockholm, Sweden 3 November 2015 (2015/1638-31/2) and followed the Helsinki declaration. The trial was registered at clinicaltrials.gov at 17 November 2017 with registration number NCT03344926. Participants provided written informed consent prior to enrollment in the study.
Treatment was delivered on a secure platform and log in required double authentication from both patients and therapists. Data storage differed depending on the type of data collected. Personal data that could be traced back to a specific individual was stored on a secure server and anonymized data was stored in a cloud solution using a cloud storage provider that was certified according to security and auditing standards ISO 27001 and SAS 70/SSAE 16 as well as connected to the Privacy Shield principles of data processing and thus complied with the European legislation (GDPR) requirements for the processing of personal data. Security was also managed through various levels of access, for example therapists only had access to data regarding their specific patients while research admin had access to all patient data.
Data collected during the alpha testing consisted of qualitative data on user behaviors and experiences. All tests were documented by the test leader and then brought back to the development team to guide further development.
In the beta testing, system generated quantitative user data was extracted during the course of the treatment/testing. Qualitative UX-data was derived from interviews with seven patients (23%) and all four therapists post treatment/testing as well as through a UX-survey that was completed by 16 (52%) of the patient beta testers. Data was compiled, organized into themes, and sorted for immediate re-iteration, future research questions to address or future development.
The main purpose of the beta testing was to examine feasibility. The variables of interest were acceptability, usage and practicality. Acceptability concerns to what extent the intervention program and format of delivery is suitable, satisfying, and attractive to the users, both recipients (patients) and deliverers (therapists).44 Usage data provides information on to what extent the participants accept the treatment, and at what level participants use the solution throughout the course of the treatment. Practicality refers to the possibilities of carrying out the intervention based on existing means, resources, and circumstances and without outside intervention44 as well as technical feasibility.
Acceptability data was collected pre-treatment (during alpha testing) as well as post treatment (as part of the beta testing) in qualitative UX-interviews. The alpha testing investigated user friendliness, comprehensiveness, and usability and led to iterations and continuous retests until they met a satisfactory level for all alpha testers before the intervention was ready for beta testing. Beta testing investigated satisfaction with treatment content, satisfaction with treatment format, satisfaction with values section as well as the treatment’s ability to motivate the user. Qualitative acceptability data was also collected from therapists in a post-treatment focus interview on satisfaction with treatment format, intent to continue use, use of therapist time, user friendliness as well as if the intervention supports patient communication.
Usage data was generated by the solution through completion rate, number of chat messages to therapist as well as post treatment logins.
Practicality data was collected during treatment and included therapist minutes per patient, text message reminders and phone calls from therapist to patient. Technical feasibility, as part of the practicality aspect, concerned how well the system worked in real-life situations through the need for technical support and reason for the need of technical support. Post-treatment quantitative data was collected in a UX-survey investigating device used, and in what working context patients engaged in treatment.
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
This content was originally published here.