Kraus, D. R., Seligman, D., & Jordan, J. R. (2005). Validation of a behavioral health treatment outcome and assessment tool designed for naturalistic settings: The Treatment Outcome Package. Journal of Clinical Psychology, 61(3), 285–314. https://doi.org/10.1002/jclp.20084
Sample:
Participants — 9,801 adult patients treated in 383 different behavioral health services across the United States who completed all questions of the TOP at intake as part of standard treatment
Race/Ethnicity — Not specified
Summary:
This article discusses a number of studies that evaluate the initial psychometrics of the items that comprise the mental health symptom and functional modules of The Treatment Outcome Package (TOP). Results indicate that the adult version of the TOP has an excellent factor structure, good test-retest reliability, promising initial convergent and discriminant validity, measures the full range of pathology on each scale, and has some ability to distinguish between behavioral health clients and members of the general population. These results document good convergent and excellent discriminant ability of many of the TOP scales. In most cases these correlation coefficients were large (in the 0.60 to 0.90 range), demonstrating good convergent validity. All but one (TOP LIFEQ and BSI-OCD, 0.31) expected discriminant relationships were below 0.30, demonstrating excellent discriminant validity.
Kraus, D. R., Castonguay, L., Boswell, J. F., Nordberg, S. S., & Hayes, J. A. (2011). Therapist effectiveness: Implications for accountability and patient care. Psychotherapy Research, 21(3), 267–276. https://doi.org/10.1080/10503307.2011.563249
Sample:
Participants — 6,960 patients seen by 696 therapists in the context of naturalistic treatment
Race/Ethnicity — Patients: 48% White/Caucasian, 24% Hispanic, 17% Asian, 8% Other, and 3% African American. Therapists: 71% White/Caucasian, 10% Hispanic, 6% African American, 6% Asian, and 6% Other
Summary:
Significant therapist variability has been demonstrated in both psychotherapy outcomes and process (e.g., the working alliance). In an attempt to provide prevalence estimates of "effective" and "harmful" therapists, the outcomes of 6,960 patients seen by 696 therapists in the context of naturalistic treatment were analyzed across multiple symptom and functioning domains. Therapists were defined based on whether their average client reliably improved, worsened, or neither improved nor worsened. Results varied by domain with the widespread pervasiveness of unclassifiable/ineffective and harmful therapists ranging from 33 to 65%. Harmful therapists demonstrated large, negative treatment effect sizes (d = -0.91 to -1.49). while effective therapists demonstrated large, positive treatment effect sizes (d = 1.00 to 1.52). Therapist domain-specific effectiveness correlated poorly across domains, suggesting that therapist competencies may be domain or disorder specific, rather than reflecting a core attribute or underlying therapeutic skill construct.
Nordberg, S. S., Castonguay, L. G., Fisher, A. J., Boswell, J. F., & Kraus, D. (2014). Validating the rapid responder construct within a practice research network. Journal of Clinical Psychology, 70(9), 886–903. https://doi.org/10.1002/jclp.22077
Sample:
Participants — 147 clients attending weekly individual psychotherapy sessions at a large university-based psychotherapy training clinic in the Northeast United States between July 2002 and September 2008
Race/Ethnicity — 86.4% Caucasian
Summary:
This study identifies multiple groups of clients in treatment with high-symptom severity and markedly different recovery trajectories (rapid/early response vs. little or no response). Using data collected through repeated administrations of the Depression subscale of The Treatment Outcome Package, growth mixture modeling was employed to determine whether clients fell into discrete groups of response trajectories during 15 sessions of psychotherapy. Additionally, logistic regressions were conducted to assess possible predictors of group membership. Three separate groups of treatment responders were identified: 2 high-symptom groups - rapid responders and nonresponders – and 1 low-symptom group of nonresponders. Elevated social conflict and suicidality predicted increased likelihood of membership in the high-symptom nonresponder group. Increased feelings of interpersonal hostility and better sexual functioning predicted increased likelihood of membership in the rapid responder group. This study provides further evidence for the usefulness of modeling change during psychotherapy using multiple trajectories. Predictors of group membership indicate the influence of functional impairment on recovery and support the importance of multidimensional measurement of client problems.
Kraus, D. R., Bentley, J. H., Alexander, P. C., Boswell, J. F., Constantino, M. J., Baxter, E. E., & Castonguay, L. G. (2016). Predicting therapist effectiveness from their own practice-based evidence. Journal of Consulting and Clinical Psychology, 84(6), 473-483. https://doi.org/10.1037/ccp0000083
Sample:
Participants — Archival data set included 3,540 clients and 59 therapists
Race/Ethnicity — Clients: 80% European American, 7% Hispanic, 5% African American, 1% Asian American, and 1% Other Race/Ethnicity; 6% did not answer item. Therapists: 68% European American, 14% Hispanic, 8% Asian American, 5% African American, and 5% Other Race/Ethnicity
Summary:
This study examined the stability and predictive validity of therapist effectiveness across multiple outcome domains using risk-adjusted outcomes. Initial and follow-up outcome data on the Adult version of The Treatment Outcome Package were collected on clients who were treated in naturalistic settings by a sample of therapists. After risk-adjusting outcomes based on case-mix variables using random forest models, outcome data from the first 30 clients of each therapist were used to classify each therapist's effectiveness on 12 outcome domains. These results were then compared with outcome data from the therapist's next 30 clients. Results demonstrated that therapist effectiveness was relatively stable, although somewhat domain specific. Therapists classified as "exceptional" were significantly more likely to remain above average with future cases, suggesting that a therapist's past performance is an important predictor of their future performance.
