Not Too Late: Guryan, Ludwig et al. (2023)

What this is. A distilled skeleton of Guryan, Ludwig et al. (2023). Read the original article to replicate or extend; this page records the headline results with PDF locators, the model equations, and the datasets used, as extracted by an LLM and not yet human-verified.

TL;DR

Two separate randomized controlled trials (RCTs) of high-dosage tutoring for disadvantaged high school students in Chicago test whether paraprofessional tutors working at a 2:1 student-to-tutor ratio for 50 minutes per school day can raise math achievement. Study 1 (n = 2,633, 2013-2014) and Study 2 (n = 2,710, 2014-2015) both find large positive treatment effects on math test scores (0.18 SD and 0.40 SD, respectively) and math course grades, with no detectable effect on arrests or disciplinary outcomes. Pooling the two studies, the treatment-on-the-treated (TOT) effect on math test scores is 0.28 SD. Effects persist: one to two years after tutoring, math test scores remain 0.23 SD higher in eleventh grade. The benefit-cost ratio (2.4-8.0 depending on study) is in the range of well-known early childhood programs such as the Abecedarian Project and the Perry Preschool Program. The results are consistent with Lazear (2001)‘s model of classroom production. Evidence is suggestive of personalization as a mechanism, while measured outcomes provide little support for a generic mentoring explanation.

Core results

#	Result	Locator	Magnitude as reported
R1	TOT effect on math test score, Study 1 Year 1	Table 3, p. 749	TOT = 0.179 SD (0.066), ITT = 0.091 (0.035); CCM = -0.111
R2	TOT effect on math GPA, Study 1 Year 1	Table 3, p. 749	TOT = 0.571 GPA pts (0.079); CCM = 1.617 (~C- to ~C+)
R3	TOT decline in math course failure rate, Study 1 Year 1	Table 3, p. 749	TOT = -0.086 (0.026); 48% decline from CCM 0.178
R4	TOT effect on math test score, Study 2 Year 1	Table 4, p. 750	TOT = 0.398 SD (0.105), ITT = 0.135 (0.036); CCM = -0.172
R5	TOT effect on math test score, pooled Year 1	Table 5, p. 751	TOT = 0.282 SD (0.059); CCM = -0.143
R6	TOT effect on math GPA, pooled Year 1	Table 5, p. 751	TOT = 0.516 GPA pts (0.069); CCM = 1.675
R7	TOT decline in math course failure rate, pooled Year 1	Table 5, p. 751	TOT = -0.086 (0.022); 47% decline from CCM 0.184
R8	Persistent TOT on 11th grade math test score	Table 7, p. 753	TOT = 0.232 SD (0.065)***; CCM = -0.147
R9	TOT effect on on-time high school graduation	Table 7, p. 753	TOT = 1.3pp (3.2pp); CCM = 78.3%; FDR q = 0.677 (null)
R10	Supplemental research-team math test, Study 1	Table 3, p. 749	TOT = 0.199 SD (0.090); FDR q = 0.029
R11	Nonmath GPA and nonmath course failures, Study 1	Table 3, p. 749	GPA TOT = 0.173 (0.068), CCM = 1.611, q = 0.018; failure TOT = -0.057 (0.022), CCM = 0.221, q = 0.018
R12	Reading-test null, Study 1	Table 3, p. 749	TOT = 0.033 (0.074); FDR q = 0.657
R13	Disciplinary outcomes, Study 1	Table 3, p. 749	TOT: incidents = 0.189 (0.235), absences = 0.441 (1.960), suspensions = 0.424 (0.345); FDR q = 0.631-0.823
R14	Arrest outcomes, Study 1	Table 3, p. 749	TOT estimates: violent -0.038 (0.035), property -0.025 (0.025), drug 0.047 (0.032), other -0.005 (0.053), ever -0.018 (0.031), any -0.021 (0.087); FDR q = 0.624-0.929
R15	Math GPA, Study 2	Table 4, p. 750	TOT = 0.412 (0.122); CCM = 1.795; FDR q = 0.002
R16	Math course failures, Study 2	Table 4, p. 750	TOT = -0.080 (0.037); 43% decline from CCM = 0.185; FDR q = 0.029
R17	Nonmath, behavioral, and arrest spillovers, Study 2	Table 4, p. 750	No statistically significant spillovers after FDR correction; nonmath GPA TOT = 0.181 (0.100), q = 0.208; total arrests TOT = -0.286 (0.151), q = 0.219
R18	Nonmath academic spillovers, pooled studies	Table 5, p. 751	Nonmath GPA TOT = 0.184 (0.058), q = 0.005; course failures TOT = -0.048 (0.018), q = 0.012
R19	Reading, behavior, and arrest null results, pooled studies	Table 5, p. 751	Reading TOT = 0.019 (0.065), q = 0.774; discipline, suspension, and arrest results not detectable (arrest q = 0.287)
R20	Year 2 test-score effect and participation bounds, Study 1	Table 6, p. 753	Per year of tutoring TOT = 0.228 SD (0.050); upper bound = 0.305 for at least one year and 0.835 for two years
R21	Year 2 math GPA and course-failure effects, Study 1	Table 6, p. 753	Per-year TOT: GPA = 0.180 (0.065); failures = -0.080 (0.035)
R22	Persistent 11th-grade math GPA	Table 7, p. 753	TOT = 0.251 (0.086); CCM = 1.837; FDR q = 0.004
R23	Ever graduated from high school	Table 7, p. 753	TOT = 0.000 (0.029); CCM = 0.865; FDR q = 0.996
R24	Grade-retention balance check	text p. 752	Authors rule out treatment-control differences in grade retention larger than +/- 3 percentage points in ITT terms
R25	Missing-outcome and Lee-bound robustness	text p. 754	Lee lower bounds are beneficial except Study 1 math scores = -0.008 and Study 2 nonmath GPA = -0.009
R26	Interaction with the Becoming a Man intervention	text p. 754	Tutoring interactions: math test score = 0.08, math GPA = -0.04, math course failures = 0.004; imprecise and insignificant
R27	Similar results across ninth- and tenth-grade cohorts	text p. 754	Authors report similar results despite different counterfactual courses
R28	No detectable treatment-effect difference by race/ethnicity	text p. 754	Treatment interactions do not show detectably different effects for Black and Hispanic students
R29	Spillover attenuation check	text p. 754	Estimated block treatment effects rise with treatment assignment share, opposite the attenuation prediction
R30	Heterogeneity by baseline achievement	Figure 1, p. 756	Math GPA effects positive in all four quartiles; math-test effects positive in the top three, consistent with a bottom-quartile floor effect
R31	Classroom discipline as a mechanism	Figure 2, p. 758	Interaction estimates are modest and imprecise with one exception: math-test effects are significantly smaller where baseline misconduct prevalence is higher
R32	Classroom math-achievement heterogeneity as a mechanism	Figure 3, p. 759	Tutoring interaction estimates are positive in 11 of 12 plotted estimates, generally imprecise
R33	Null evidence for a generic mentoring channel	text p. 758	No detectable change in caring-adult count; 95% CIs rule out effects above 0.10 SD for grit, 0.16 SD for conscientiousness, and 0.10 SD for locus of control
R34	Earnings-based benefit-cost analysis	text pp. 759-760	PDV earnings gains: $11,500 (Study 1), $25,700 (Study 2); benefit-cost ratios: 2.4-3.6 and 5.4-8.0, respectively
R35	Randomization balance diagnostic	Table 2, p. 746	Joint baseline-characteristic balance test p = 0.798 (Study 1), p = 0.273 (Study 2)
R36	First-stage take-up of at least one tutoring session	text p. 749	Treatment take-up = 40.2% (Study 1), 36.9% (Study 2); control participation = 1.1%, 7.8%
R37	Year 2 participation first stage, Study 1	Table 6, p. 753	Assignment effect: years of tutoring = 0.563 (0.020), at least one year = 0.423 (0.014), two years = 0.139 (0.010)

Overall (paper’s conclusion). High-dosage tutoring with paraprofessional tutors raises math test scores by 0.18-0.40 SD within one academic year for disadvantaged high school students - effect sizes comparable to what Fryer (2014) finds for tutoring as a component of no-excuses charter schools. Effects persist at 0.23 SD in eleventh grade math test scores and 0.25 GPA points (Table 7). No statistically significant effects emerge on disciplinary outcomes, arrests, or graduation (the graduation point estimate is positive but imprecise at 1.3pp). Benefit-cost ratios of 2.4-8.0 are comparable to the Abecedarian Project (1.9-2.2) and Perry Preschool (3.9-6.8). The evidence is suggestive of personalization of instruction as a mechanism, supported by Banerjee et al. (2007) and by heterogeneity analysis showing larger gains in classrooms with more heterogeneous math achievement levels (Figure 3, p. 759), though those interaction estimates are generally imprecise. No detectable treatment effects on grit, conscientiousness, or locus of control provide little support for a generic mentoring explanation (p. 758).

Theory / model

The paper adapts Lazear (2001)‘s classroom-production framework. A school has S students and budget M for teachers; teacher quality is V(w), and average class size is n = wS/M. Student achievement heterogeneity is indexed by variance (\sigma^2), and the probability of learning without disruption falls as heterogeneity rises. The paper’s model equations are unnumbered in the article (pp. 756-757):

$$n = \frac{wS}{M}$$ $$p(\sigma^2) = \frac{e^{-\sigma^2}}{1 + e^{-\sigma^2}}$$ $$\max_w SV(w) p(\sigma^2)^{wS/M}$$ $$\frac{\partial w^*}{\partial \sigma^2} = \frac{\left\{\frac{S}{M}[1-p(\sigma^2)]V(w^*)^2\right\}}{V(w^*)V''(w^*)-[V'(w^*)]^2}$$

The comparative static is negative when teacher quality is not too convex in wages: more classroom achievement dispersion makes smaller classes or tutoring more valuable. The paper tests two channels: reducing behavior-related classroom disruption, and personalizing instruction to achievement levels. The author’s interpretation is that achievement heterogeneity, rather than baseline classroom misconduct, predicts larger gains (Figures 2-3, pp. 758-759).

Method

Assignment to tutoring was randomized. The intention-to-treat equation (1), printed on p. 745, is:

$$\text{Y}_i = \pi_0 + \pi_1\text{Z}_i + \text{X}_i\pi_2 + \text{B}_i + \varepsilon_i \tag{1}$$

Here Z is the offer, X contains baseline covariates, and B denotes randomization block fixed effects. The treatment-on-the-treated estimate uses random assignment as an instrument for participation. The first stage and outcome equation (2)-(3), printed on pp. 746-747, are:

$$\text{D}_i = \gamma_0 + \gamma_1\text{Z}_i + \text{X}_i\gamma_2 + \text{B}_i + \mu_i \tag{2}$$ $$\text{Y}_i = \beta_0 + \beta_1\hat{\text{D}}_i + \text{X}_i\beta_2 + \text{B}_i + \vartheta_i \tag{3}$$

Equation (3) uses the fitted participation indicator from the first stage; Z is the instrument and beta1 is the complier effect under the IV assumptions. The paper also reports FDR-adjusted q-values by outcome family and permutation-test inference using 100,000 randomizations (analysis plan, pp. 745-748).

Empirical specifications

Equations (1)-(3) include randomization-block fixed effects and baseline controls for sociodemographics, pre-randomization test scores, prior GPA, absences, suspensions, disciplinary incidents, and arrest history. Missing covariates are set to zero with missingness indicators; only observations with observed outcomes are used. Standard errors are heteroskedasticity robust. Study 2 and pooled estimates cluster standard errors by individual because 65 students were randomized twice. Table notes for Tables 3-7 (pp. 749-753) report these details.

Study 1 includes 2,633 male ninth- and tenth-graders; Study 2 includes 2,710 students of both sexes; pooled Year 1 analyses include 5,343 students. The main outcome tables estimate ITT and 2SLS TOT separately by study and pooled cohort. The Year 2 specification uses years of participation as the endogenous treatment and random assignment as its instrument (Table 6, p. 753). As described on pp. 751-752, the TOT is a weighted average of persistent Year 1 effects and the incremental Year 2 effect. Its reported upper bounds assume one component is zero while the component effects are nonnegative.

Study 1 included Becoming a Man (BAM) in a 2 × 2 factorial design, building on the Chicago intervention reported in Heller et al. (2017).

The paper’s subgroup specification replaces the treatment indicator with interactions for baseline achievement quartiles (Figure 1, p. 756; written out from the figure note):

$$\text{Y}_i = \alpha + \sum_{q=1}^{4} \beta_q \text{Z}_i\mathbf{1}(\text{Q}_i=q) + \sum_{q=1}^{4} \lambda_q\mathbf{1}(\text{Q}_i=q) + \text{X}_i\gamma + \text{B}_i + \varepsilon_i$$

The classroom interaction specification (Figures 2-3, pp. 758-759; written out from the figure notes) is the TOT model:

$$\text{Y}_i = \alpha + \beta \text{D}_i + \delta \text{H}_i + \theta(\text{D}_i \times \text{H}_i) + \text{X}_i\gamma + \text{B}_i + \varepsilon_i$$

Here H is a classroom-discipline or achievement-dispersion measure. The endogenous terms D and D x H are instrumented by assignment Z and Z x H. Both specifications retain block fixed effects and the standard baseline controls; the figures report confidence intervals for the interaction estimates. These specifications are not numbered in the article.

Datasets used

Dataset	Role in paper	Wiki page
Chicago Public Schools (CPS) Student Administrative Records	Primary outcomes (test scores, GPA, course failures, attendance, disciplinary actions); enrollment and school records; baseline covariates for Study 1 and Study 2	no page yet
CPS standardized tests (EXPLORE, PLAN by ACT Inc.)	Primary math outcome, expressed as CPS-wide z-scores	no page yet
Chicago Police Department (CPD) Arrest Records	Secondary outcome family (violent, property, drug, other arrests)	no page yet
ISR Survey Data (Institute for Social Research, Univ. of Michigan)	Math achievement test administered by research team; survey measures of noncognitive skills, adult relationships, risky behavior	no page yet
Saga Education internal records	Tutoring attendance and dosage; tutor characteristics; Saga internal math assessments	no page yet

Sample scope. Study 1: 2,633 ninth and tenth grade male students in 12 CPS high schools, 2013-2014 academic year. Study 2: 2,710 ninth and tenth grade students (male and female) in 15 CPS high schools, 2014-2015. Pooled N = 5,343. Average baseline math score in both samples was 8-15 percentile points below the CPS-wide average (Table 1, p. 746).

When to read the full paper

Read the original if you are: (a) designing or scaling a tutoring program for secondary students and need the benefit-cost analysis (Section IV, pp. 759-761); (b) studying the mechanisms of educational production (Lazear (2001) model, Section III, pp. 755-759); (c) assessing the credibility of ITT/TOT estimates in large-scale school RCTs (Section II.D, pp. 745-748 for the analysis plan and multiple-testing corrections); or (d) building on the evidence for high-dosage tutoring reviewed in Nickow, Oreopoulos, and Quan (2020). Table 3 and Table 4 are the primary results tables by study; Table 5 (pooled) and Table 7 (persistent effects) are the synthesis tables of most interest for policy.

Attribution and rights

Guryan, Jonathan, Jens Ludwig, Monica P. Bhatt, Philip J. Cook, Jonathan M. V. Davis, Kenneth Dodge, George Farkas, Roland G. Fryer Jr., Susan Mayer, Harold Pollack, Laurence Steinberg, and Greg Stoddard. 2023. “Not Too Late: Improving Academic Outcomes among Adolescents.” American Economic Review 113(3): 738-765. https://doi.org/10.1257/aer.20210434

Replication data: https://doi.org/10.3886/E182903V1

This page contains LLM-distilled extracts only (extract-only; paywalled article). Not human-verified. Not reproduced. Read the original for any replication or downstream use.