The continuous version of the ASPES method allows researchers to estimate the relationship between a continuously-defined endogenous variable and impact magnitude. As described below, the method involves the following two stages: (1) predict values of the continuously-defined endogenous variable symmetrically for treatment and control group members; and (2) estimate the relationship between the predicted endogenous variable and impact magnitude.
Stage 1: Construct Predicted Endogenous Variable
In Stage 1, a cross-validation approach is used to predict the continuously-defined endogenous variable of interest. The cross-validation approach involves the following steps:
- Step 1: Randomly partition the experimental sample (both treatment and control) into 10 or more groups of equal size.
- Step 2: To obtain predictions for group 1, estimate the prediction model on the subsample of treatment individuals in groups 2-10+. Using the parameters obtained from estimating this prediction model, predict endogenous subgroup membership for both treatment and control individuals in group 1.
- Step 3: To obtain predictions for group 2, estimate the prediction model on the subsample of treatment individuals in groups 1, 3-10+. Using the parameters obtained from estimating this prediction model, predict endogenous subgroup membership for both treatment and control individuals in group 2.
- Step 4: Repeat for groups 3-10+.
This process provides each individual in the sample (both treatment and control) with a predicted value of the endogenous variable of interest.
Stage 2: Estimate Relationship Between Predicted Endogenous Variable and Impact Magnitude
In Stage 2, estimate the relationship between the predicted continuously-defined endogenous variable (constructed in Stage 1) and intervention impacts. To do so, analysts should estimate the relationship between the outcome of interest and the following combination of measures: (1) a treatment group indicator; (2) the predicted continuously-defined endogenous variable; and (3) the interaction of the treatment group indicator and the predicted continuously-defined endogenous variable. The coefficient on the interaction term provides an estimate of the effect of the predicted continuously-defined endogenous variable on the intervention’s impact.
Practical Example
Harvill et al. (2017) apply the ASPES method to restricted-use data from the evaluation of Comprehensive Teacher Induction (CTI) Study (Glazerman et al., 2010) to test whether increased mentorship for novice teachers is associated with increased impacts on student achievement outcomes. They use the continuous ASPES method to produce estimates of the effect of mentorship on CTI impacts. They find that a one-unit increase in the average number of times per semester the novice teacher was observed teaching by their mentor is associated with a 27 percent of a standard deviation increase in the impact of the CTI intervention on student math achievement. The authors note that a one-unit increase in this measure of mentorship represents a relatively large increase, akin to jumping from about the 25th percentile to the 75th percentile in terms of the amount of mentorship received.
The following assumption is required to attribute this increase in impacts to the influence of the actual endogenous variable (as opposed to the predicted endogenous variable): the baseline characteristics that predict the endogenous variable do not have a direct or indirect effect on impacts apart from their indirect effect on impacts through the endogenous variable. In this application, this assumption would be violated if the teacher personal and professional characteristics used to predict mentorship receipt affect CTI impacts through channels other than mentorship (e.g., through professional development workshops).
| Student Math Achievement Score | |
|---|---|
| The effect of mentorship on the impact of CTI | 0.27** (0.11) |
| Number of Students | 1,190 |
| Number of Teachers | 70 |
| Number of Schools | 50 |
| Number of Districts | 10 |
Notes: Reported results include coefficient and standard error, clustered at the school level, in parentheses.
*** p<0.01, ** p<0.05, * p<0.1. Reported sample sizes are rounded to the nearest 10 to minimize disclosure risk.
Discussion
The continuous version ASPES method provides a promising avenue for researchers interested in estimating effect of a continuously-defined endogenous variable. The method provides an experimental estimate of the relationship between predicted values of a continuously-defined endogenous variable and the magnitude of an intervention’s impact. This estimate allows researchers to address mediational questions like the following: What is the relationship between the predicted endogenous variable and intervention impacts?
Researchers may be more interested in understanding the relationship between actual values of the endogenous variable and impact magnitude. The following key assumption is required to interpret the continuous ASPES estimates as the influence of the actual endogenous variable on impacts: the baseline characteristics that predict the endogenous variable do not have a direct or indirect effect on impacts apart from their indirect effect on impacts through the endogenous variable.