My client, a small online training provider specializing in certification courses, collaborates with local firms and universities to offer structured learning programs. Their goal was to expand beyond institutional clients and attract new individual learners through a targeted social media campaign. While working on this project, we identified an additional opportunity: leveraging customer feedback to refine course structure and better understand learner behavior.
This blog post focuses on that initiative, how small adjustments to training sessions led to measurable improvements in learning outcomes and student satisfaction.
Disclaimer: This blog post was reviewed by the client before publication, and some sections, specifically those detailing the training activities and aspects of the heterogeneity analysis, have been removed.
Course organization
The course in focus runs for six weeks, with two-hour sessions held online. Three groups follow the same curriculum in parallel:
- Tuesday evening
- Thursday evening
- Saturday morning
Each group consists of 25 to 50 students and professionals. At the end of the course, a mock certification exam is conducted before students enroll for the official certification. In theory, there is no incentive to cheat on this assessment.
One recurring complaint, particularly among the Thursday evening group—was that the sessions felt too long and were difficult to follow. To address this, the client introduced:
- A 10-minute break after one hour for the Thursday group
- A 5-minute break for the Tuesday group
These breaks allowed students to ask questions freely or engage with the instructor. The key questions were:
- Was the break effective in improving test performance?
- Is a 5-minute break sufficient, or does a longer break provide additional benefits?
Identification strategy
To evaluate the impact of these changes, we used a Difference-in-Differences (DiD) approach. This method helps identify causal effects by comparing the changes in test scores before and after the intervention across different groups. The key assumption is that, in the absence of an intervention, all groups would have followed similar learning trends.
Since this was the third time the course was offered, we had access to past test score data for comparison. Additionally, my client collects data from registration, allowing us to control for individual characteristics.
By analyzing test score trends for comparable students across cohorts, we could isolate the impact of the breaks introduced in the third cohort:
Results
Depending on the comparison group, we observed test score increases ranging from 3% to 10%. To account for small sample size concerns, we applied bootstrapping, a resampling technique that allow for statistical inference. Results were significant at the 10% level. Similar results were observed for satifaction scores.
Since students self-select into groups, the estimated effect represents a Local Average Treatment Effect (LATE), applicable to individuals whochose each session. To explore general implications, we used Inverse Probability Weighting (IPW) to adjust for selection bias and conducted additional group-level analysis.
Interestingly, after accounting for differences in group composition, we found that the difference in impact between a 5-minute and 10-minute break was statistically insignificant. Instead, the primary driver of variation was the type of students, certain individuals benefited more from longer breaks, rather than it being a universally superior option.
Business Actions
These findings provided my client with the confidence to implement 5-minute breaks as a standard practice in most courses. However, some courses will continue without breaks or with extended 10-minute breaks where student benefit most.
Beyond improving student experience, even modest increases in test scores can strengthen the client’s reputation with institutional clients. Future analysis will further refine these strategies as more cohorts complete the program under the updated structure.