Ashish Srikanth - Measuring Change in Coping Skills: A Pre/Post Walkthrough for MH&A Training

Data Byte

Context

Most evaluation conversations in mental health and addictions (MH&A) start from the same place: “We ran a new group, we distributed our pre- and post-training surveys. Did it help?” I recall reading a Quick Tips guide from the University of Toronto’s Continuing Professional Development (CPD) team a short while ago, which nudged me to slow down and decide what part of the story we are actually trying to measure, who needs the answer, and how they will use it before I even think about getting started with data analysis.

An important note to be made here is that a pre/post scale is not a verdict on the entire program. It is one lens, mainly about learning and early competence, that can sit alongside participation, satisfaction, performance, and patient outcomes in a broader program evaluation plan.

Suppose we have a six-session “Coping Skills for Recovery 101” group at your trusted MH&A hospital. The group serves outpatients from two streams: one focused on mood and anxiety, and the other on concurrent substance use. At session one and session six, participants complete a brief 10-item coping confidence scale scored from 0 to 10.

Objectives

We want to know whether participants who complete the group report higher coping confidence, we want the result to inform whether to keep or adapt the group, the main audience is the clinical lead and manager, and the output will be a short summary that feeds into next year’s planning cycle.

Data Sources

Here, let’s build a small fictitious dataset in R for 270 clients from a hospital coping skills group, one row per person. Each row has a client ID (client_id), a stream label for either mood and anxiety or concurrent use (stream), and two coping knowledge scores: one before the group (pre_knowledge) and one after (post_knowledge). Pre scores are simulated from a normal distribution centred around the middle of a 0 to 10 scale, then I add an average improvement of about 1.2 points plus noise, trimming everything to stay within 0 and 10. That produces a realistic mix of partial gains, a few declines, and some ceiling effects. Once that synthetic run is working, one can swap the dataset for an export from REDCap or your clinical system with the same basic structure and reuse the rest of the analysis.

Code

set.seed(2025)

library(dplyr)
library(tibble)

n <- 270

dat <- tibble(
  client_id     = sprintf("P%03d", 1:n),
  stream        = sample(
    c("MOOD/ANXIETY", "CONCURRENT USE"),
    size    = n,
    replace = TRUE,
    prob    = c(0.6, 0.4)
  ),
  pre_knowledge = pmin(10, pmax(0, rnorm(n, mean = 5.5, sd = 1.5)))
)

dat %>%
  select(client_id, stream, pre_knowledge) %>%
  slice(1:9)

# A tibble: 9 × 3
  client_id stream         pre_knowledge
  <chr>     <chr>                  <dbl>
1 P001      CONCURRENT USE          7.84
2 P002      MOOD/ANXIETY            3.98
3 P003      MOOD/ANXIETY            5.35
4 P004      MOOD/ANXIETY            3.94
5 P005      CONCURRENT USE          6.91
6 P006      MOOD/ANXIETY            4.47
7 P007      CONCURRENT USE          2.63
8 P008      MOOD/ANXIETY            6.90
9 P009      CONCURRENT USE          6.45

Analysis

Instead of comparing post-group averages only, we calculate a change score for each person and treat them as their own control.

Code

set.seed(2025)

library(dplyr)
library(tibble)

n <- 270

dat <- tibble(
  client_id     = sprintf("P%03d", 1:n),
  stream        = sample(
    c("MOOD/ANXIETY", "CONCURRENT USE"),
    size    = n,
    replace = TRUE,
    prob    = c(0.6, 0.4)
  ),
  pre_knowledge = pmin(10, pmax(0, rnorm(n, mean = 5.5, sd = 1.5)))
) %>%
  mutate(
    change         = rnorm(n, mean = 1.2, sd = 1.2),
    post_knowledge = pmin(10, pmax(0, pre_knowledge + change))
  )

dat %>%
  select(client_id, stream, pre_knowledge, post_knowledge, change) %>%
  slice(1:9)

# A tibble: 9 × 5
  client_id stream         pre_knowledge post_knowledge change
  <chr>     <chr>                  <dbl>          <dbl>  <dbl>
1 P001      CONCURRENT USE          7.84          10     2.24 
2 P002      MOOD/ANXIETY            3.98           5.30  1.31 
3 P003      MOOD/ANXIETY            5.35           6.97  1.62 
4 P004      MOOD/ANXIETY            3.94           6.75  2.81 
5 P005      CONCURRENT USE          6.91           7.39  0.478
6 P006      MOOD/ANXIETY            4.47           6.09  1.61 
7 P007      CONCURRENT USE          2.63           3.56  0.931
8 P008      MOOD/ANXIETY            6.90           8.11  1.21 
9 P009      CONCURRENT USE          6.45           5.88 -0.567

In terms of the analysis in R, that means taking post_score - pre_score within each row, summarising the mean and confidence interval of that change, and running a paired t-test with t.test(post_score, pre_score, paired = TRUE).

Code

summary_tbl <- dat %>%
  summarise(
    n           = n(),
    mean_pre    = mean(pre_knowledge),
    mean_post   = mean(post_knowledge),
    mean_change = mean(post_knowledge - pre_knowledge)
  )

tt <- t.test(dat$post_knowledge, dat$pre_knowledge, paired = TRUE)

list(
  summary = summary_tbl,
  t_test  = tt
)

$summary
# A tibble: 1 × 4
      n mean_pre mean_post mean_change
  <int>    <dbl>     <dbl>       <dbl>
1   270     5.55      6.77        1.23

$t_test

    Paired t-test

data:  dat$post_knowledge and dat$pre_knowledge
t = 16.807, df = 269, p-value < 2.2e-16
alternative hypothesis: true mean difference is not equal to 0
95 percent confidence interval:
 1.082761 1.370097
sample estimates:
mean difference 
       1.226429

This answers a very specific question: among participants who finished the group and completed both surveys, what is the average gain in coping confidence and how precisely is it estimated? It aligns with the “learning” and early “competence” levels from above, which focus on whether people know what to do and can demonstrate that knowledge in the structured setting of the program.

The Preprocessing Step(s) Once Removed?

On a real dataset, the workflow would involve another step: reshaping the data so that each participant has one row with clearly named pre and post columns and filtering to rows where both are non-missing (super important!).

We can also check the distribution of change scores; a quick histogram will tell you whether most people are clustered around zero or shifting in a meaningful way.

Code

suppressPackageStartupMessages({
  library(dplyr)
  library(tibble)
  library(plotly)
  library(htmltools)
})

set.seed(2025)

n_clients <- 270L

mh <- tibble(
  client_id = sprintf("P%03d", 1:n_clients),
  stream    = sample(
    c("MOOD/ANXIETY", "CONCURRENT USE"),
    size = n_clients,
    replace = TRUE,
    prob = c(0.6, 0.4)
  ),
  pre_coping = pmin(10, pmax(0, rnorm(n_clients, mean = 5.5, sd = 1.5)))
) %>%
  mutate(
    change_raw  = rnorm(n_clients, mean = 1.2, sd = 1.2),
    post_coping = pmin(10, pmax(0, pre_coping + change_raw)),
    change_coping = post_coping - pre_coping
  )

bg_col            <- "#FAF8F1"
point_color       <- "#751F2C"
font_family       <- "Ramabhadra"
.fixed_fig_width  <- 1100L
.fixed_fig_height <- 390L
.wrapper_max_w    <- 790L

fig_change <- plotly::plot_ly(
  mh,
  x    = ~change_coping,
  type = "histogram",
  marker = list(color = point_color),
  hovertemplate = "CHANGE: %{x:.1f}<br>COUNT: %{y}<extra></extra>"
) %>%
  layout(
    width  = .fixed_fig_width,
    height = .fixed_fig_height,
    font   = list(family = font_family, size = 12),
    paper_bgcolor = bg_col,
    plot_bgcolor  = bg_col,
    margin = list(l = 18, r = 18, t = 36, b = 36),
    showlegend = FALSE,
    xaxis = list(
      title = list(text = "CHANGE IN COPING SCORE (POST − PRE)", standoff = 20),
      showspikes = TRUE,
      tickfont   = list(size = 16),
      gridcolor  = "#E8E8E8",
      zeroline   = FALSE,
      fixedrange = TRUE
    ),
    yaxis = list(
      title = list(text = "NUMBER OF PARTICIPANTS", standoff = 20),
      showspikes = TRUE,
      tickfont   = list(size = 16),
      automargin = TRUE,
      fixedrange = TRUE
    ),
    uniformtext = list(minsize = 16),
    hovermode   = "x unified",
    hoverlabel  = list(
      font = list(family = font_family, size = 14, color = "#313131"),
      bgcolor = "#FFF",
      namelength = -1
    ),
    dragmode = FALSE
  ) %>%
  config(
    responsive = FALSE,
    scrollZoom = FALSE,
    doubleClick = FALSE,
    modeBarButtonsToRemove = list(
      "zoom2d","pan2d","select2d","lasso2d",
      "zoomIn2d","zoomOut2d","autoScale2d","resetScale2d",
      "toggleSpikelines","toImage"
    ),
    displaylogo   = FALSE,
    displayModeBar = TRUE,
    showTips      = FALSE
  )

htmltools::browsable(
  htmltools::tagList(
    htmltools::tags$style(htmltools::HTML(sprintf("
      #coping-change-hist-wrap {
        border-radius: 6px;
        overflow: hidden;
        overflow-x: auto;
        -webkit-overflow-scrolling: touch;
        background: %s;
        padding: 0.75rem;
        width: 100%%;
        max-width: %dpx;
        margin: 0 auto 1rem;
      }
      #coping-change-hist-wrap > div {
        min-width: %dpx;
      }
      @media (max-width: %dpx) {
        #coping-change-hist-wrap { padding: 0.5rem; }
      }
    ", bg_col, .wrapper_max_w, .fixed_fig_width, .wrapper_max_w))),
    htmltools::tags$div(id = "coping-change-hist-wrap", fig_change)
  )
)

Methodology

As a way of ensuring our analysis steps are sound but also delivering a more rich analysis to leadership, we can repeat the summary by stream or other meaningful subgroups; by doing so, the clinical team would be able to see whether certain populations are benefiting more.

Results & Next Steps

The value of this approach is not the p-value; it is the clarity about what we are and are not claiming. A pre/post scale will not tell you whether clinicians changed their practice, whether relapse rates dropped, or whether community outcomes improved. Those sit higher in the evaluation hierarchy and rely on other methods such as chart reviews, administrative data, or follow-up surveys—the CPD calls these complementary options. What a paired analysis does give you is a compact, defensible view of learning for the subset of clients who completed the group, expressed in units that are easy to explain in supervision, at a program meeting, or in a one-page briefing to leadership.

If you are an evaluator or analyst in a similar setting, the takeaway from all of this is simple. Treat your pre/post scales as one building block in a larger evaluation design, set up a clean paired analysis with a small, well-documented R script, and connect the numerical change back to the specific decisions your team needs to make about keeping, scaling, or reshaping the program. Once that basic pattern is in place, it is easy to “rinse and repeat” the same code across groups, gradually layer in other evaluation levels, and turn scattered surveys into a reusable feedback loop for your services.