Identifying accurate and cost-effective fidelity measurement methods for therapists using CBT

Optimal usage of evidence-based psychotherapy treatments requires tools to ensure that clinicians are successfully implementing new techniques. Research to improve client outcomes in community mental health has been hindered by an inability to accurately and inexpensively measure fidelity, or the type and quality of treatment provided. Measuring fidelity accurately is critically important, because it mediates the effect of evidence-based practices on client outcomes. Fidelity is also an indicator of quality of care. Fidelity is composed of adherence (how closely the components of a protocol are followed) and competence (how skillfully the components are implemented and how responsive the therapist is to the client and situation). Measuring fidelity is challenging because there are few reliable, valid, and efficient fidelity measurement methods. Direct observation of therapist behavior, the gold standard for measuring fidelity to psychosocial interventions, requires extensive resources. When fidelity is measured in community settings, which occurs rarely, the most commonly used and least expensive method is therapists’ self-report. Unfortunately, concordance between observation and self-report is low. There is a critical need to identify and evaluate methods of fidelity measurement that are accurate (i.e., measure what they intend) and cost-effective. Our rationale for this study stems from ongoing collaboration with community partners, who have raised fidelity measurement as a significant barrier to ensuring quality implementation of evidence-based practices. The timing of the study is ideal, as the Affordable Care Act will mandate agencies to monitor the fidelity of the therapies they use. Our long-term research goal is to strengthen the public health impact of psychosocial interventions by identifying fidelity measurement methods that can be used for both research and practice. Our objective in this measurement proposal is to compare the accuracy and cost-effectiveness of therapists’ self-report (fidelity measurement-as-usual) and 2 innovative methods (chart-stimulated recall and behavioral rehearsal) in assessing fidelity to cognitive-behavioral therapy for children and families, an established evidence-based practice. Chart-stimulated recall is a brief interview with a clinician about the care provided to a client, during which the clinician reviews the client’s file to aid recall.Behavioral rehearsal, also known as standardized patient methodology, is a role-play between a clinician and a trained actor. Our central hypothesis is that chart-stimulated recall and behavioral rehearsal are a compromise between direct observation and self-report. We will randomize 135 therapists to 1 of 3 conditions: self-report (n = 45), chart-stimulated recall (n = 45), and behavioral rehearsal (n = 45). We will compare the fidelity measurement method in each arm to direct observation using the Therapy Process Observational Coding System13 as the gold-standard comparison. Specifically, we propose to: Specific Aim #1: Identify the most accurate fidelity measurement method. We hypothesize, based on preliminary data, that chart-stimulated recall and behavioral rehearsal will be more accurate than self-report when each is compared to direct observation. Specific Aim #2: Estimate the economic costs and cost-effectiveness of the proposed fidelity measurement techniques. We hypothesize that chart-stimulated recall and behavioral rehearsal will be more cost-effective than self-report. Specific Aim #3: Compare stakeholder willingness to use each method, and identify their perceived barriers and facilitators to using each method. We hypothesize that stakeholders will be most willing to use chart-stimulated recall in future endeavors. For each method, we will identify barriers and facilitators to use. Successful completion of this project will identify accurate and cost-effective innovative fidelity measurement methods and identify factors that can increase their use. This study will have a significant positive impact by determining which fidelity measurement methods can be used by implementation scientists for research and by community mental health clinics to monitor fidelity of evidence-based practices. The significance of this study is its potential to reduce the research-to-practice gap by identifying accurate and cost-effective fidelity measurement methods that both researchers and stakeholders in community mental health clinics can use to monitor fidelity of evidence-based practices, thereby improving the public health impact of evidence-based psychosocial interventions.