In 2008 a couple of economists from the IDB visited the Peruvian Ministry of Education to meet with the Education Technology Director. Our purpose was to evaluate an old IDB-project that provided computer labs to middle schools.
We were welcomed and were provided with all the support to carry out that evaluation. We were told that the Government’s real interest was in conducting the most rigorous evaluation possible of another project, the Peruvian One Laptop per Child (Una Laptop por Niño).
Although that program was not financed by the Bank, we formed a group with experts from various units of the Bank, and in early 2009 worked with the Government’s team to design the first impact rigorous impact evaluation of the program. The details of the evaluation are available in a recent working paper, here I just want to mention some salient features.
This is the first evaluation of OLPC that focuses on educational outcomes, namely student learning. There are many stories about implementation of OLPC, and many opinions for and against the program (see for example OLPC News and the Technology Salon), but no evaluations that focus on learning, and none using a rigorous evaluation approach.
We used a randomized control trial: out of 320 schools, 210 were randomly selected to receive XO´s (the laptops), while the rest did not. The schools were identical before the program, and except from the computers nothing differed between them, so we are pretty sure that any difference after the program can be attributable to OLPC.
Our main focus was on academic achievement in Math and Language, that were the declared objective of the program. We also looked at cognitive skills as measured by Raven’s Progressive Matrices, a verbal fluency test and a Coding test.
The results are striking.
Our results indicate that the program dramatically increased access to computers. There were 1.18 computers per student in the treatment group, compared with 0.12 in control schools at follow-up. This massive rise in access explains substantial differences in use.
Eighty-two percent of treatment students reported using a computer at school in the previous week compared with 26 percent in the control group. Effects on home computer use are also large: 42 percent of treatment students report using a computer at home in the previous week versus 4 percent in the control group. Internet use was limited because hardly any schools in the study sample had access.
However, we find no evidence that the program increased learning in Math or Language. This is not surprising, as the program did not include specific interventions to integrate the laptop to the curricula, nor the computers include specific math or language software.
The program did not affect attendance or time allocated to doing homework, nor did it increase motivation or reading habits and the program did not seem to have affected the quality of instruction in class.
On the positive side, the results indicate some benefits on cognitive skills. In the three measured dimensions, students in the treatment group surpass those in the control group. A back-of-the-envelope calculation suggests that the estimated impact on the verbal fluency measure represents the progression expected in six months for a child.
Similarly, the estimated impact for the Coding and Raven tests accounts for roughly the expected progression during five and four months, respectively.
The results are quite striking: as implemented, the program did not increase learning. Is this surprising? What is next for OLPC, in Peru and elsewhere? Let us know your opinion…
Please see the month-long discussion of OLPC’s impact in Peru and analysis of this report on http://EduTechDebate.org
Please see the month-long discussion of OLPC’s impact in Peru and analysis of this report on http://EduTechDebate.org
I don’t see the lack of difference in math or reading to be a major concern. The deciding measure should have been computer literacy. By making computers and computer skills part of the daily life of students at a young age, those children will develop a measure of comfort with the technology – putting them much closer to higher income peers in their own country and abroad, and able to keep up with other opportunities that may come down the line.
Look at developed countries – even here we have a computer literacy gap. Our is generational – how often have you seen the 20-something student intern deftly fix something for a much more senior staff member? Our gap will diminish as the time passes, but without projects like OLPC, the gap will persist in impovrished countries, further alienating the have-nots from the rest of society.
Closing computer literacy gaps is a massive accomplishment. When that child goes on to a more advanced school, they won’t be held back by not having figured how to navigate through software, etc.
Computer literacy is not enough for anyone to be educated. Over 61% of all mainland Chinese students use computers for games and to hack–definitely not the definition of education. It is far more important that a student masters subject matter: mathematics, science, reading, and so forth. Computers are tempting, but several studies that I have conducted show that computers are at best expensive laptops and toys. Students studied, read, thought, debated, and obtained far better grades before computers were introduced into the classroom. Computers only encourage plagiarism, theft of intellectual property, and laziness.
I don’t see the lack of difference in math or reading to be a major concern. The deciding measure should have been computer literacy. By making computers and computer skills part of the daily life of students at a young age, those children will develop a measure of comfort with the technology – putting them much closer to higher income peers in their own country and abroad, and able to keep up with other opportunities that may come down the line.
Look at developed countries – even here we have a computer literacy gap. Our is generational – how often have you seen the 20-something student intern deftly fix something for a much more senior staff member? Our gap will diminish as the time passes, but without projects like OLPC, the gap will persist in impovrished countries, further alienating the have-nots from the rest of society.
Closing computer literacy gaps is a massive accomplishment. When that child goes on to a more advanced school, they won’t be held back by not having figured how to navigate through software, etc.
Computer literacy is not enough for anyone to be educated. Over 61% of all mainland Chinese students use computers for games and to hack–definitely not the definition of education. It is far more important that a student masters subject matter: mathematics, science, reading, and so forth. Computers are tempting, but several studies that I have conducted show that computers are at best expensive laptops and toys. Students studied, read, thought, debated, and obtained far better grades before computers were introduced into the classroom. Computers only encourage plagiarism, theft of intellectual property, and laziness.
I saw this paper presented at Columbia University in NYC. There are some problems here that aren’t discussed in the report or presentation.
This project has yet to identify or model the actual treatment they are offering. Student may get a laptop, but there was no incorporation of the amount of time the students actually spent on the computer for each session they used it. The time on the laptop is what we think is important, not being in the near vicinity of a laptop.
The one piece of info we got out of the presenter was that most sessions were no longer than 15 mins and it was mainly used to copy notes from the chalkboard. This is the real treatment. But in the study, if the student turned the computer on at all in a day, it was coded as one, for being treated. They should really weight the treatment indicator by the proportion of the class day that the computer was used, since most students only used it in school and never took it home. There might actually be an effect when the specification error is removed.
As it stands though, given how the laptops were used, is there any wonder that we don’t find any positive impact when the treatment is 15 mins a day to do transcription?
I think the main finding is that it is not enough to hand technology to students that came with absolutely no programs for guided learning in it and to teachers who have no idea how to use technology in a productive way. But we already knew this.
So I really don’t know why the OLOC project didn’t install guided learning programs on these laptops. If you are going to spend this kind of money to put a laptop in the hands of each child, give them something to actually do with it!
Steve–the program in Peru, with its lack of integration in the curriculum, lack of teacher training, and lack of use is broadly representative of OLPC projects in other geographies. It is of no use to suggest that the lack of impact was due to the fact that schools weren’t using the OLPCs properly or intensively. This is what has happened in most places where technology is deployed into classrooms in this manner.
The conclusion, based on current evidence, has to be that OLPC and it’s like are a waste of funds. Similar impact can be achieved far more cheaply with non-technological interventions (for reference see J-PALs education portfolio).
I saw this paper presented at Columbia University in NYC. There are some problems here that aren’t discussed in the report or presentation.
This project has yet to identify or model the actual treatment they are offering. Student may get a laptop, but there was no incorporation of the amount of time the students actually spent on the computer for each session they used it. The time on the laptop is what we think is important, not being in the near vicinity of a laptop.
The one piece of info we got out of the presenter was that most sessions were no longer than 15 mins and it was mainly used to copy notes from the chalkboard. This is the real treatment. But in the study, if the student turned the computer on at all in a day, it was coded as one, for being treated. They should really weight the treatment indicator by the proportion of the class day that the computer was used, since most students only used it in school and never took it home. There might actually be an effect when the specification error is removed.
As it stands though, given how the laptops were used, is there any wonder that we don’t find any positive impact when the treatment is 15 mins a day to do transcription?
I think the main finding is that it is not enough to hand technology to students that came with absolutely no programs for guided learning in it and to teachers who have no idea how to use technology in a productive way. But we already knew this.
So I really don’t know why the OLOC project didn’t install guided learning programs on these laptops. If you are going to spend this kind of money to put a laptop in the hands of each child, give them something to actually do with it!
Steve–the program in Peru, with its lack of integration in the curriculum, lack of teacher training, and lack of use is broadly representative of OLPC projects in other geographies. It is of no use to suggest that the lack of impact was due to the fact that schools weren’t using the OLPCs properly or intensively. This is what has happened in most places where technology is deployed into classrooms in this manner.
The conclusion, based on current evidence, has to be that OLPC and it’s like are a waste of funds. Similar impact can be achieved far more cheaply with non-technological interventions (for reference see J-PALs education portfolio).
How can you measure a 5-month advancement in cognitive skills (over 2 years) and say “the program did not increase learning”? Something is clearly wrong with your analysis, if not with the method.
How can you measure a 5-month advancement in cognitive skills (over 2 years) and say “the program did not increase learning”? Something is clearly wrong with your analysis, if not with the method.
Striking? perhaps the most glaring ISSUE in this and many of the other OLPC reports is the pervasive lack of internet access and library resources at these schools. Why is everyone harping on about maths and literacy results with or without ICTs in the classroom, when, as Wayan’s news of the destructive Peruvian warehouse fire yesterday, the cost of 61,000 OLPC “US$200” accounted for less than 15% of total cost of destruction – which included 500,000 extremely expensive books destined for these schools! I think we should be rethinking one laptop per child to one library/media centre with a well-paid suitably motivated ‘informationist’ and FREE high speed Internet access per school for folks to access increasingly more OERS!
Striking? perhaps the most glaring ISSUE in this and many of the other OLPC reports is the pervasive lack of internet access and library resources at these schools. Why is everyone harping on about maths and literacy results with or without ICTs in the classroom, when, as Wayan’s news of the destructive Peruvian warehouse fire yesterday, the cost of 61,000 OLPC “US$200” accounted for less than 15% of total cost of destruction – which included 500,000 extremely expensive books destined for these schools! I think we should be rethinking one laptop per child to one library/media centre with a well-paid suitably motivated ‘informationist’ and FREE high speed Internet access per school for folks to access increasingly more OERS!
I agree with Tim. We need to acknowledge that not every project is going to be a success, and we need to accept and learn from results that show some interventions don’t accomplish much. It’s unfortunate that the resources dedicated to buying the laptops weren’t used better, but at least the resources were allocated to evaluating the intervention so we could learn something from the project. What did we learn? That there are many better ways to help disadvantaged children than by providing them access to video games (which, I understand, is mostly what the laptops functioned as to these children). Bravo to the investigators for being willing to call attention to a negative result: my negative results have been repeatedly quashed by the funding agency. Bring on the objective analysis- let’s learn from our mistakes!
I agree with Tim. We need to acknowledge that not every project is going to be a success, and we need to accept and learn from results that show some interventions don’t accomplish much. It’s unfortunate that the resources dedicated to buying the laptops weren’t used better, but at least the resources were allocated to evaluating the intervention so we could learn something from the project. What did we learn? That there are many better ways to help disadvantaged children than by providing them access to video games (which, I understand, is mostly what the laptops functioned as to these children). Bravo to the investigators for being willing to call attention to a negative result: my negative results have been repeatedly quashed by the funding agency. Bring on the objective analysis- let’s learn from our mistakes!
Perhaps the fault here lies not with the technology, nor with the teachers who may or may not know what they were to do with such numbers of devices but with the comparison of testing against use of technology to improve it.
You say your results ‘were striking’ and so they should be when you have not actually detailed anything for comparison apart from relating that a child with a laptop does not equate to improved test results; it can also be said a child with a large paper weight does not equate to improved test results, the focus on test results is at fault here, not the technology. Technology can and is used effectively to improve learning, develop understanding, solve problems, collaborate on world wide projects, provoke and debate, create and develop, and for the narrow minded, improve test results.
Review the project, don’t let your narrow mindedness let it fail. Go back and see where the use of technology can be improved, where the hindrance of testing can be removed and where every learner can improve.
Perhaps the fault here lies not with the technology, nor with the teachers who may or may not know what they were to do with such numbers of devices but with the comparison of testing against use of technology to improve it.
You say your results ‘were striking’ and so they should be when you have not actually detailed anything for comparison apart from relating that a child with a laptop does not equate to improved test results; it can also be said a child with a large paper weight does not equate to improved test results, the focus on test results is at fault here, not the technology. Technology can and is used effectively to improve learning, develop understanding, solve problems, collaborate on world wide projects, provoke and debate, create and develop, and for the narrow minded, improve test results.
Review the project, don’t let your narrow mindedness let it fail. Go back and see where the use of technology can be improved, where the hindrance of testing can be removed and where every learner can improve.