By Xiomara Alemán, Suzanne Duryea, Nancy Guerra, Patrick McEwan, Rodrigo Muñoz, Marco Stampini, and Ariel Williamson
Who could have imagined four decades ago that a few Venezuelan musicians rehearsing in a garage would lay the foundation for a project that to date has enrolled millions of children and adolescents?
But that’s just what the Venezuelan National System of Youth and Children’s Choirs and Orchestras (El Sistema) has done.
Founded in 1975 and administered by the Fundación Musical Simón Bolivar (Fundamusical), the program has received national and international awards honoring its musical and social initiatives. More than 25 countries have either completely or partially replicated the program. Some of those efforts have been supported by the IDB through technical cooperation grants.
Despite its international reputation, however, El Sistema had never been subject to an impact evaluation to rigorously measure its benefits. To address that need, the IDB and Fundamusical agreed in 2011 to assemble an interdisciplinary team comprised of economists, engineers, sociologists, and psychologists to design an impact evaluation to identify the causal effects of the program.
To get started, they conceptualized a “change theory” or model that described how the music program could contribute to a chain of intermediate effects, and then ultimately to the results expected in the long term.
At the time of the evaluation, approximately 400,000 children and youth were participating in the program at 370 training centers throughout Venezuela devoted to teaching music theory and applied music.
The evaluation makes two key contributions: It is the first experimental evaluation in any country of a scaled-up, government-implemented music intervention. It also presents experimental evidence on the effects of a group music program in a developing country with high rates of violence.
Following an ongoing dialogue between the IDB and Fundamusical and a thorough review of the literature on psychology, four constructs were identified to explain how the music system could potentially benefit children and adolescents. The four constructs were self-control skills, prosocial skills and connections, behavior, and cognitive skills.
Because of the excess demand for the program in some training centers, it was possible to conduct an experimental impact evaluation to measure the short-term causal effects in 16 training centers in the Capital District and in the departments of Aragua, Bolívar, Lara, and Miranda.
All families with children aged 6 to 14 applying for admission were divided randomly into two groups: the treatment group was offered admission into the program in September 2012, and the control group was offered a guaranteed admission into the program in September 2013.
To obtain information directly related to the four constructs, data collection occurred in two phases. The first took place between October 2012 and February 2013, and the second occurred between September and October 2013.
The analysis focused on the changes in children’s outcomes related to the four constructs over one academic year.
Did the Evaluation Find an Impact of the Music Program on Children and Youth?
Through a comprehensive bibliographic review, a set of instruments was selected to estimate the effects in the four constructs. The set of instruments included computerized questionnaires and exercises completed by program applicants and their parents or guardians. This in turn allowed 26 indicators to be defined that captured the dimensions of the four constructs.
The evaluation results showed that, on average, children in the treatment group increased their scores on the self-reported self-control scale by 1.13 percent compared to the control group.
This increase means that receiving an offer of early admission to the music program helped the children improve their capacity to control their attention, behavior, and emotions.
In addition, children in the treatment group reduced their scores on the scale of self-reported total difficulties by 2.05 percent. This means that children who received an offer of early admission into the program suffered fewer conduct problems and improved relations with their peers.
Finally, the evaluation found that the effects observed were, on average, more positive for children from the most vulnerable situations.
For the subgroup of boys exposed to violence, the offer of admission improved their scores on the scale of self-control on average by 2.42 percent and reduced their scores on the scale of difficulties on average by 6.03 percent. Similarly, the offer of admission reduced aggressive behavior on average by 6.9 percent in this subgroup.
The design of the evaluation provides important lessons for the group of evaluators. It points to the importance of working with an interdisciplinary group to design the change theory that guides impact evaluation. In addition, it shows the value of incorporating psychometric analysis to confirm the precision and reliability of the measures of the results that are being sought.
Finally, it underlines the value of unifying, standardizing, and automatizing the information processes that collect the registrations and achievements of the participating training centers, especially with regard to pre-registration processes for students.
The participant registration forms used in the impact evaluation will be used at all the training centers in the music program, which will help in monitoring the program in the years ahead.
Although this evaluation estimated only the short-term effects of the program, El Sistema seems to be particularly effective for boys exposed to violent situations. Multiple studies of other interventions working with groups of vulnerable boys or young males have not shown such positive impact.
This story is one of the impact evaluations included in the Development Effectiveness Overview, an annual publication that highlights the lessons learned from IDB projects and evaluations.
About the authors:
Xiomara Alemán is a social protection specialist in the IDB headquarters in Washington, DC.
Suzanne Duryea is a principal research economist in the Social Sector in the IDB headquarters in Washington, DC.
Marco Stampini is a social protection lead specialist in the IDB headquarters in Washington, DC.
Nancy Guerra is a professor of the Department of Psychological and Brain Sciences at the University of Delaware.
Patrick McEwan is a professor of Economics at Wellesley College.
Rodrigo Muñoz is an engineer and researcher specialized in social metrics and evaluation.
Ariel Williamson is a graduate student of the Department of Psychological and Brain Sciences at the University of Delaware.