The discrete version of the ASPES method estimates impacts on subgroups defined by post-randomization experiences. The method proceeds in three stages: (1) use baseline characteristics to sort the experimental sample into discretely-defined endogenous subgroups; (2) estimate the impacts on these predicted endogenous subgroups; and (3) convert estimated impacts on predicted endogenous subgroups to represent impacts on actual endogenous subgroups. In what follows, we provide more detail on each of these three stages of the method. For additional practical guidance on applying the discrete version of the ASPES method we refer readers to the SPI-Path|Individual User Guide (Moulton, Peck, & Bell, 2014).
Stage 1: Construct Predicted Endogenous Subgroup Indicator
In Stage 1, the probability (or propensity) that a study participant is in an endogenous subgroup is modeled as a function of baseline characteristics using a logit, probit, or linear probability model. The goal of this prediction process is to identify which treatment and control group members have the baseline characteristics associated with membership in a given subgroup (Moulton, Peck, & Bell, 2014). In order to ensure the comparability of treatment and control group members within a subgroup without loss of sample, Harvill, Peck, & Bell (2013) recommend using a cross-validation approach to predicting subgroup membership. The cross-validation approach involves the following steps:
- Step 1: Randomly partition the experimental sample (both treatment and control) into 10 or more groups of equal size.
- Step 2: To obtain predictions for group 1, estimate the prediction model on the subsample of treatment individuals in groups 2-10+. Using the parameters obtained from estimating this prediction model, predict endogenous subgroup membership for both treatment and control individuals in group 1.
- Step 3: To obtain predictions for group 2, estimate the prediction model on the subsample of treatment individuals in groups 1, 3-10+. Using the parameters obtained from estimating this prediction model, predict endogenous subgroup membership for both treatment and control individuals in group 2.
- Step 4: Repeat for groups 3-10+.
This process provides each individual in the sample (both treatment and control) with a continuous score that represents their probability of being in the endogenous subgroup based on their baseline characteristics. This continuous score is then converted into a binary indicator that divides the sample into two predicted endogenous subgroups (for example, a predicted high dosage subgroup and a predicted low dosage subgroup).
We refer readers to the SPI-Path|Individual User Guide for additional practical guidance on the construction of predicted endogenous subgroups, steps involved in the cross-validation approach to subgroup prediction, selection of the baseline characteristics included in the prediction model, assessing performance of the prediction model, and more.
Stage 2: Estimate Impacts on Predicted Endogenous Subgroups
In Stage 2, the impact of the intervention is estimated for each predicted subgroup constructed in Stage 1. Since the predicted subgroups were constructed using only baseline characteristics, the integrity of the original randomized evaluation remains intact and the impacts for each of the predicted subgroups are experimental impacts, unbiased by selection or other influences (e.g., Moulton, Peck, & Greeney, 2017). However, not everyone who is predicted to be in the subgroup based on their baseline traits will actually be in that subgroup. This is because our prediction model is unlikely to perfectly predict subgroup membership. As such, the estimated impacts on the predicted subgroup represents the impact of the intervention on those with a profile of characteristics that makes them most likely to be in that subgroup.
While the estimated impacts on predicted subgroups are experimental, researchers may be more interested in estimating the impact of the intervention on study participants who actually belong to a given subgroup. Stage 3 of the discrete version of the ASPES method provides a method for converting these estimated impacts on predicted endogenous subgroups to represent impacts on subgroups that actually had the experience of interest. However, estimates of impacts on actual endogenous subgroups require additional assumptions, as discussed in more detail below.
Stage 3: Convert Impacts on Predicted Endogenous Subgroups to Impacts on Actual Endogenous Subgroups
In this final stage, the estimated impacts on predicted endogenous subgroups estimated in Stage 2 are converted to estimated impacts on actual endogenous subgroups. For example, converting the impact on the subgroup of study participants predicted to receive a high dosage of the intervention to represent the impact on the subgroup of study participants that actually received a high dosage of the intervention. The SPI-Path|Individual User Guide describes how to estimate impacts on actual endogenous subgroups and corresponding assumptions.
Bell & Peck (2013) note that the Stage 3 conversion assumptions hinge on whether the impacts on the actual endogenous subgroup vary systematically with the baseline characteristics used to predict subgroup membership. For example, if baseline educational attainment were included in the prediction model, then the assumptions would be violated if the impact on the actual high dosage subgroup differed for those with high and low levels of baseline education. Although we cannot directly test this assumption, Bell & Peck (2013) show that we are seeking instrumental variables as predictors of subgroup membership that affect impact magnitude through the actual endogenous factor but not by other means. This implies that the best baseline characteristics to include in the Stage 1 prediction model are those that (1) strongly predict subgroup membership and (2) bear little relationship to actual subgroup impacts (Bell & Peck, 2013).
Practical Example
Harvill et al. (2017) apply the ASPES method to restricted-use data from the evaluation of Comprehensive Teacher Induction (CTI) Study (Glazerman et al., 2010) to test whether increased mentorship for novice teachers is associated with increased impacts on student achievement outcomes. They use the discrete version of the ASPES method to estimate the impact of CTI on students taught by teachers predicted to receive a high dosage of mentorship (more than the median dosage received by treatment teachers) and students taught by teachers predicted to receive a low dosage of mentorship (less than the median dosage received by treatment teachers). For the predicted high dosage subgroup, treatment group students’ math achievement scores were 36 percent of a standard deviation higher than their control group counterparts. In comparison, the estimated CTI impact on math achievement for the predicted low dosage subgroup is near zero.
Following Stage 3 of the ASPES method, the analysts convert impacts on the subgroup predicted to receive a high (or low) dosage of mentorship to represent impacts on the subgroup who actually received a high (or low) dosage of mentorship. For the actual high dosage subgroup, they find that the CTI impact on math achievement is 69 percent of a standard deviation. However, due to the large standard error associated with this impact the authors are unable to reject more modest impacts. In this example, the Stage 3 conversion assumptions would be violated if the teacher personal and professional characteristics used to predict subgroup membership affect CTI impacts through channels other than mentorship (e.g., through professional development workshops).
| Impact on Student Math Achievement Score | ||
|---|---|---|
| High Dosage Subgroup | Low Dosage Subgroup | |
| CTI Impact on Predicted Subgroup | 0.36*** (0.10) |
-0.02 (0.09) |
| CTI Impact on Actual Subgroup | 0.69*** (0.20) |
-0.53*** (0.25) |
| Number of Students | 640 | 550 |
| Number of Teachers | 40 | 30 |
| Number of Schools | 30 | 30 |
| Number of Districts | 10 | 10 |
Notes: Reported results include coefficient and standard error, clustered at the school level, in parentheses.
*** p<0.01, ** p<0.05, * p<0.1. Reported sample sizes are rounded to the nearest 10 to minimize disclosure risk.
Discussion
The discrete version of the ASPES method produces both (1) impacts on the subgroup of study participants predicted to be in a given endogenous subgroup and (2) impacts on the subgroup of study participants who are actually in a given endogenous subgroup. The Stage 2 impacts estimated on predicted subgroups are considered experimental because subgroups are symmetrically constructed for the treatment and control groups using only baseline characteristics (Harvill, Peck, & Bell, 2013). Stage 3 converts estimated impacts on predicted subgroups computed in Stage 2 to represent impacts on actual dosage-defined subgroups, but requires additional assumptions.
Both the impacts on predicted endogenous subgroups (estimated in Stage 2) and impacts on actual endogenous subgroups (estimated in Stage 3) may be of policy interest. Impacts on predicted subgroups may be of interest for analyses of predicted “risk groups”, defined by their likelihood to experience a given event (such as students at risk of dropping out of school). There may be more policy interest in the impact on the subgroup of study participants who are actually in the endogenous subgroup (as opposed to those who are likely to be in the group), but converting to impacts on actual subgroups requires additional strong assumptions. As such, impacts on predicted subgroups may be preferred to impacts on actual subgroups if researchers are uncomfortable with the assumptions necessary to convert from predicted to actual impacts (Moulton, Peck, & Bell, 2014).