Inter-American Development Bank
facebook
twitter
youtube
linkedin
instagram
Negocios SosteniblesCaribbean Development Trends¿Y si hablamos de igualdad?Puntos sobre la iIdeaçãoSeguridad CiudadanaSostenibilidadFactor TrabajoImpactoEnfoque EducaciónGobernarteKreatopolisPrimeros PasosCiudades SosteniblesEnergía para el FuturoGente SaludableMás Allá de las FronterasBeyond BordersIdeas MatterIdeas que CuentanAbierto al públicoMoviliblogVolvamos a la fuente Gestión fiscalHome
Citizen Security and Justice Creative Industries Development Effectiveness Early Childhood Development Education Energy Envirnment. Climate Change and Safeguards Fiscal policy and management Gender and Diversity Health Labor and pensions Open Knowledge Public management Science, Technology and Innovation  Trade and Regional Integration Urban Development and Housing Water and Sanitation
  • Skip to content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer

Impacto

  • HOME
  • CATEGORIES
    • Beyond development effectiveness
    • Evaluation methods and techniques
    • Measuring our performance
    • What does and doesn’t work in development
  • Authors
  • English
    • Español

The synth_runner Package: Utilities to Automate Synthetic Control Estimation Using synth

March 22, 2016 by Autor invitado | Leave a Comment


By Brian Quistorff.

 The new module for Synthetic Control Methods “synth_runner” helps conduct multiple estimations, inference, diagnostics, and generate visualizations of results.”

synth_runner
Image: iStock

The Synthetic Control Methodology (SCM) (Abadie and Gardeazabal, 2003, Abadie et al., 2010, hereafter ADH) is a data-drive approach to small-sample comparative case-studies for estimating treatment effects.

Similar to a difference-in-differences design, SCM exploits the differences in treated and untreated units across the event of interest. However, in contrast to a difference-in-differences design, SCM does not give all untreated units the same weight in the comparison.

Instead, it generates a weighted average of the untreated units that closely matches the treated unit over the pre-treatment period and uses that as the counterfactual. Along with their paper, ADH released the -synth- Stata command for single estimations. This blog post details the new module -synth_runner- that builds on top of the previous command to help conduct multiple estimations, inference, diagnostics, and generate visualizations of results.

Let’s take a look at the example used in ADH in which they estimate the effect of Proposition 99 in California on cigarette sales. The proposition passed in 1988 and increased the tax on cigarettes and instituted several other restrictions on tobacco. First, have a look at the state trends in per-capita cigarette sales.  synth 1

California is graphed along with states that did not enact large tobacco regulations or change their tobacco tax rates. These 38 other states are termed “donors” as they can potential be used to form the counterfactual. As one can see, California looks quite different than most of these other states during the pre-treatment period.

A simple difference-in-differences strategy would therefore not be appropriate as the parallel trends assumption is not satisfied. Instead of using the untreated states equally, SCM finds an optimal weight over the untreated states to construct a counterfactual.

The weights are found so that the counterfactual’s pre-treatment outcomes (and any other important pre-treatment variables) match that of the treated unit. Below is the estimated counterfactual for California.

synth 2

One sees that the counterfactual matches California well during the pre-treatment period. In the post-treatment period, however, California is much lower than its counterfactual. The estimated effect is then the difference between the treated unit and its synthetic control for the post-treatment period. Below is the difference between California and its counterfactual.synth 3

ADH show that if weights can be found so that the counterfactual matches well the treated unit in the pre-treatment period then the estimated effect will be unbiased even in the presence of unobserved confounders that take a factor structure.

The allowed factor structure is more general than the standard panel estimation framework where unobserved confounders are limited to time-invariant characteristics. In SCM, the factor structure can accommodate units on different time trends.

For inference, SCM conducts a series of in-place placebo tests. For each of the untreated units, temporarily assume that it received treatment at the same time and construct a synthetic control for each using the rest of the untreated units. Collect the placebo effects (differences between units and their synthetic controls) to get a distribution against which one can gauge the relative size of the main effect.synth 4

To calculate a p-value for each post-treatment effect, find the share of placebo effects that are as large as the main effect.

If many placebo effects are as large as the main effect (i.e. the p-value is high) than it is likely that the main effect was observed by chance. As one can see in the distribution of differences above, post-treatment differences will be larger if the pre-treatment match was bad. A common alteration then is to scale each post-treatment effect by a measure of pre-treatment match quality (the pre-treatment root mean-squared prediction error, RMSPE). Comparing those “pseudo t-statistics”, the following are the p-values for Proposition 99’s effects.

synth 5

Cavallo et al. (2013) extent SCM by allowing more than one unit to experience treatment and at possibly different times.

An overall treatment effect is constructed as an average over the treated units, with effects all relative to their treatment date. This averaging removes noise from the estimate so the same should be done for the comparison distribution when conducting inference.

For each treatment, consider the group of placebo estimates where the never-treated donors are thought of as experiencing treatment at the treatment period. Select one placebo effect from each group and then take the average to construct a member of the comparison distribution. There will be many such averages with the size of the comparison group growing exponentially in the number of treatments.

The -synth_runner- package also performs several diagnostics used in the aforementioned papers. The first is to check if a weighted average of donors is able to approximate the treated unit in the pre-treatment period. This should be satisfied if the treated unit lies within the convex hull of the control units. -synth_runner- calculates from the distribution of pre-treatment RMSPEs, what proportion of control units match worse than the treated unit.

The second diagnostic, is that if one constructs the counterfactual by only matching on the initial part of the pre-treatment period, then the counterfactual should do well to match the rest of the pre-treatment period. The initial section of the pre-treatment period is often designated the “training” period with the later part being the “validation” period. As an example, Cavallo et al. (2013) set aside the first half of the pre-treatment period as the training period. When a training period is used, -synth_runner- will provide the proportion of control units that match worse than the treated unit during the validation period.

Hopefully, this package will be useful for those using this new method. The synth_runner module file may be downloaded from here.

 

Brian Quistorff has a degree in Computer Science from Stanford University, a Master’s in Economics from University of British Columbia and he will finished is Ph.D. in Economics from University of Maryland at College Park this May 2016.


Filed Under: Evaluation methods and techniques Tagged With: Brian Quistorff Data data-drive Economics IDB Inter-American Development Bank SCM STATA synth synth_runner Synthetic Control Methodology University of Maryland

Autor invitado

Autor invitado

Reader Interactions

Comments

  1. click says

    May 9, 2016 at 8:46 am

    fantastic points altogether, you just gained a brand new reader. What would you suggest about your post that you made some days ago? Any positive?

    Reply
    • BID-Blog says

      May 16, 2016 at 9:37 am

      We are happy you liked this post. About your question… what post are you referring to?

      Reply
  2. click says

    May 9, 2016 at 8:46 am

    fantastic points altogether, you just gained a brand new reader. What would you suggest about your post that you made some days ago? Any positive?

    Reply
    • BID-Blog says

      May 16, 2016 at 9:37 am

      We are happy you liked this post. About your question… what post are you referring to?

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Search

About this blog

This blog highlights effective ideas in the fight against poverty and exclusion, and analyzes the impact of development projects in Latin America and the Caribbean.

Suscription

Categories

Footer

Banco Interamericano de Desarrollo
facebook
twitter
youtube
youtube
youtube

Blog posts written by Bank employees:

Copyright © Inter-American Development Bank ("IDB"). This work is licensed under a Creative Commons IGO 3.0 Attribution-NonCommercial-NoDerivatives. (CC-IGO 3.0 BY-NC-ND) license and may be reproduced with attribution to the IDB and for any non-commercial purpose. No derivative work is allowed. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB's name for any purpose other than for attribution, and the use of IDB's logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC- IGO license. Note that link provided above includes additional terms and conditions of the license.


For blogs written by external parties:

For questions concerning copyright for authors that are not IADB employees please complete the contact form for this blog.

The opinions expressed in this blog are those of the authors and do not necessarily reflect the views of the IDB, its Board of Directors, or the countries they represent.

Attribution: in addition to giving attribution to the respective author and copyright owner, as appropriate, we would appreciate if you could include a link that remits back the IDB Blogs website.



Privacy Policy

Banco Interamericano de Desarrollo

Aviso Legal

Las opiniones expresadas en estos blogs son las de los autores y no necesariamente reflejan las opiniones del Banco Interamericano de Desarrollo, sus directivas, la Asamblea de Gobernadores o sus países miembros.

facebook
twitter
youtube