Development that Works
  • About

    This blog highlights effective ideas in the fight against poverty and exclusion, and analyzes the impact of development projects in Latin America and the Caribbean.
  • Randomized Control Trials vs. Structural models: sometimes a question of life or death (and millions of dollars….)



    I ran into this Wired article that tells the interesting story of the Pfizer failure in launching torcetrapib, a potentially revolutionary compound for cardiovascular treatment.

    Although the conclusion is too skeptical and the article gets a little bit lost on the “correlation vs. causation” argument, I believe it’s a good example of what can happen when people become overconfident about their “theories of change.”

    Cognitive biases like those of the Michotte balls are pervasive, as shown by Daniel Kahneman in his book Thinking, Fast and slow.

    Pfizer’s scientists were so sure they had precisely understood the cholesterol pathway, that the company not only had invested more than US$1 billion in the development of this drug, but also US$90 million to expand the factory that would manufacture the compound, even before having the results of a full scale RCT.

    Luckily, in the pharmaceutical industry RCTs are a legal obligation: instead of saving lives, the drug would have killed people.

    2 Responses to “Randomized Control Trials vs. Structural models: sometimes a question of life or death (and millions of dollars….)”

    • RCTs aren’t perfect and shouldn’t be above criticism, but yes a thousand times. Every time there’s a major trial failure the reporting will have quotes from the people behind it saying they’re baffled by what went wrong; they were that certain it would work. See this recent NYT article for instance: that ends with “Quite frankly, the results were a surprise”. They’re always a surprise to those who think they know the outcome based on theory or priors, which is why we test them (or should).

    Comment on the post