The power of field experiments
Mar 2020
Small is beautiful
Last February, José Vila-Belda gave a nanoTalk at the University of Zurich explaining the key contributions for which professors Abhijit Banerjee, Esther Duflo and Michael Kremer were awarded the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2019. By breaking down big open questions into smaller but rigorous and policy-relevant scientific tests and leveraging the power of field experiments, Duflo and her colleagues have revolutionized the study of the causes of poverty and the policies that can effectively reduce it.
by Tania Manríquez
Esther Duflo is a member of the advisory board at the UBS Center. She is the Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics in the Department of Economics at the Massachusetts Institute of Technology (MIT) and a co-founder and co-director of the Abdul Latif Jameel Poverty Action Lab (J-PAL).
José Vila-Belda is a PhD candidate in Economics at the University of Fribourg and a Research Manager at the Department of Economics, University of Zurich. Prior to joining the University of Zurich, he was based at the Latin American office of the Poverty Action Lab, where he worked with policymakers in Chile, Uruguay, Peru and Mexico using scientific evidence and designing experiments to evaluate the impact of their policies. At UZH, he manages research projects led by UBS Foundation Professor Dina Pomeranz, a J-PAL affiliate.
Last February, José Vila-Belda gave a nanoTalk at the University of Zurich explaining the key contributions for which professors Abhijit Banerjee, Esther Duflo and Michael Kremer were awarded the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2019. By breaking down big open questions into smaller but rigorous and policy-relevant scientific tests and leveraging the power of field experiments, Duflo and her colleagues have revolutionized the study of the causes of poverty and the policies that can effectively reduce it.
by Tania Manríquez
Esther Duflo is a member of the advisory board at the UBS Center. She is the Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics in the Department of Economics at the Massachusetts Institute of Technology (MIT) and a co-founder and co-director of the Abdul Latif Jameel Poverty Action Lab (J-PAL).
A new approach
How can we eradicate poverty? One way in which governments in high income countries are trying to do so is through foreign aid. But how do we know whether development aid is working? As Vila-Belda said, this a question that development economists have been trying to answer for years.
One simple approach to try to examine this is by looking at whether developing countries that receive more foreign aid tend to experience faster economic growth. A Forbes article published online in 2012 looked at this and found a negative correlation: countries receiving more aid tend to experience less growth, not more. The article therefore concluded that economic aid is an “economic somnolent”. Vila-Belda invited the audience to question this interpretation of the data: correlation does not imply causation. A more likely interpretation of this negative correlation, he said, is that developing countries that are expected to experience less growth are the ones that receive more aid, which sounds like a sensible policy decision!
While development economists have used more sophisticated methods than simple correlations to try to understand the causal relationship between foreign aid and economic growth or poverty, Vila-Belda used this example to illustrate two important limitations in past research on poverty and development that Duflo and her colleagues have helped to overcome.
First, questions such as whether aid works in general are simply too big to be answered scientifically and too general to be of any practical significance for policy making. Whether aid works for economic growth and poverty, Vila-Belda argued, depends on the details of exactly how aid funds are used in each specific case. So rather than a single big question on whether aid works, we need to ask many small questions about how to maximize the impact of every dollar that is available to reduce poverty. The Nobel Prize winners propose to take the big puzzle of poverty and break it down into smaller questions that matter for practical policy decisions. In other words, these professors may argue that ‘small is beautiful’.
Secondly, Vila-Belda pointed out, the old-school approach of using existing data to answer research questions is too limited. Even when the relevant data exists or can be collected, our ability to establish cause and effect relationships has been severely limited by the tools on which development economists have relied for decades. Duflo, Banerjee and Kremer do not only invite us to ask radically different questions, but also propose a powerful tool to answer them: randomized experiments.
How can we eradicate poverty? One way in which governments in high income countries are trying to do so is through foreign aid. But how do we know whether development aid is working? As Vila-Belda said, this a question that development economists have been trying to answer for years.
One simple approach to try to examine this is by looking at whether developing countries that receive more foreign aid tend to experience faster economic growth. A Forbes article published online in 2012 looked at this and found a negative correlation: countries receiving more aid tend to experience less growth, not more. The article therefore concluded that economic aid is an “economic somnolent”. Vila-Belda invited the audience to question this interpretation of the data: correlation does not imply causation. A more likely interpretation of this negative correlation, he said, is that developing countries that are expected to experience less growth are the ones that receive more aid, which sounds like a sensible policy decision!
Esther Duflo © Nobel Media AB 2019 by Alexander Mahmoud.
Research meets policy
To illustrate what randomized experiments are and how they are used to study poverty, Vila-Belda used the example of research on malaria prevention. In 2018, there were 228 million cases of malaria and over 400'000 deaths caused by it, 90% of which happened in Sub-Saharan Africa. Against this background, Vila-Belda said, policy makers tried to figure out ways to expand the use of bed nets and to best allocate the limited resources available for that purpose.
Imagine you have 1,000 dollars to spend in subsidies for bed nets and you are asked to decide how to use these funds. You have two options. First, you can fully subsidize the cost of bed nets. If the market price of each bed net is 10 dollars, you can buy 100 bed nets and give them for free to 100 people. Alternatively, you could charge half of the market price to each person, that is, 5 dollars. If you find enough people willing to buy bed nets for this price, you could benefit up to 200 people with your subsidies, twice as many as in the first approach.
Which option is best for the ultimate goal of preventing malaria transmission? With the second approach, you may be able to reach more people, but if those willing and able to pay for bed nets are too spread out, malaria transmission may not be contained. Vila-Belda illustrated how the methodology of randomized experiments, coupled with precise and carefully targeted research questions, allowed a group of researchers to produce crucial scientific evidence that helped policy makers figure out which approach to take.
The researchers Jessica Cohen and Pascaline Dupas, broke down this problem into three key smaller aspects in their study ‘Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment’ in 2010. The first aspect was access: how can access to malaria bed nets be maximized? Would people be willing to pay small fees to buy bed nets, so that more people can be reached with partial subsidies? The second aspect was use: if given for free, would people not value and not use the bed nets? If so, would it therefore be better to charge small fees, based on the idea that when you pay for something you are more likely to use it? Finally, the third key aspect to take into consideration was cost per user: if users contribute a bit to the costs, more funds will be available to partially subsidize more bed nets. The combination of these three elements would determine which option is the most cost-effective, that is, which approach prevents more malaria cases for every dollar spent on bed net subsidies.
To illustrate what randomized experiments are and how they are used to study poverty, Vila-Belda used the example of research on malaria prevention. In 2018, there were 228 million cases of malaria and over 400'000 deaths caused by it, 90% of which happened in Sub-Saharan Africa. Against this background, Vila-Belda said, policy makers tried to figure out ways to expand the use of bed nets and to best allocate the limited resources available for that purpose.
Imagine you have 1,000 dollars to spend in subsidies for bed nets and you are asked to decide how to use these funds. You have two options. First, you can fully subsidize the cost of bed nets. If the market price of each bed net is 10 dollars, you can buy 100 bed nets and give them for free to 100 people. Alternatively, you could charge half of the market price to each person, that is, 5 dollars. If you find enough people willing to buy bed nets for this price, you could benefit up to 200 people with your subsidies, twice as many as in the first approach.
Abhijit Banerjee © Nobel Media AB 2019 by Alexander Mahmoud.
Rolling the dices
The experiment carried out by Cohen and Dupas took place in rural Kenya. They selected a representative sample of sixteen health clinics in rural areas and randomly split the sixteen clinics into four experimental groups. This process of organizing a pool of subjects into randomly selected groups is the key defining characteristic of the experimental approach championed by the Nobel Prize winning trio. When we split in two or more groups a pool of subjects in a randomized way, we obtain groups that are equal on average in every characteristic. One can then introduce a policy manipulation and accurately measure its causal effects. After the policy intervention, any observed difference between these groups will be the causal effect of the policy.
Vila-Belda explained that in the Kenyan experiment, the policy manipulation involved was the level of fees that users had to pay to get a bed net: from no fees at all, to a fee of 70 cents of a dollar, equivalent to 10% of the market price. Because these groups only differed in the fee they had to pay to get a bed net, this experiment allowed the researchers to accurately measure the effect of charging fees on the access and use of bed nets, and ultimately on malaria incidence.
This research and other randomised controlled experiments on the use of malaria bed nets produced two relevant findings. First, charging fees did not substantially promote use: contrary to what some expected, people who obtained the bed nets for free were as likely to use them as those who paid for them. Second, and most importantly, charging fees caused large reductions in access. The researchers working in Kenya found that practically all of the users offered a bed net for free accepted it. However, charging a fee of 70 cents reduced the percentage of people getting a bed net to less than 40%. These findings were replicated in 15 other experiments involving bed nets and other preventive health products, such as hand soap, vitamins, or water filters, and not only in African countries but also in India and Guatemala.
These studies demonstrated that in terms of access, free distribution of bed nets is much better than charging fees, and even if free distribution implies a higher cost per user, overall free distribution is much more cost-effective to prevent malaria.
The experiment carried out by Cohen and Dupas took place in rural Kenya. They selected a representative sample of sixteen health clinics in rural areas and randomly split the sixteen clinics into four experimental groups. This process of organizing a pool of subjects into randomly selected groups is the key defining characteristic of the experimental approach championed by the Nobel Prize winning trio. When we split in two or more groups a pool of subjects in a randomized way, we obtain groups that are equal on average in every characteristic. One can then introduce a policy manipulation and accurately measure its causal effects. After the policy intervention, any observed difference between these groups will be the causal effect of the policy.
Vila-Belda explained that in the Kenyan experiment, the policy manipulation involved was the level of fees that users had to pay to get a bed net: from no fees at all, to a fee of 70 cents of a dollar, equivalent to 10% of the market price. Because these groups only differed in the fee they had to pay to get a bed net, this experiment allowed the researchers to accurately measure the effect of charging fees on the access and use of bed nets, and ultimately on malaria incidence.
Michael Kremer © Nobel Media AB 2019 by Alexander Mahmoud.
Policy influence
Partly thanks to these studies, Vila-Belda pointed out, organizations like the World Health Organization, the United Nations Millennium Project, and the Department for International Development of the UK, now recognize that the elimination of user fees is the most effective way of quickly increasing take-up of key preventive health products.
The promising research findings based on randomized controlled experiments also contributed to increase funding to eradicate malaria: development assistance funds more than doubled in recent years. A rise in funding to distribute free bed nets has led to a rapid increase in the percentage of people sleeping under bed nets in Sub-Saharan Africa and this has been key to achieve a sustained reduction in the incidence of malaria across the region, saving millions of lives.
To wrap up, Vila-Belda invited the audience to learn about the work of J-PAL, the organization co-founded by Esther Duflo that conducts experiments to produce evidence about what work the areas such as agriculture, conflict, education, environment, finance, gender, governance, health, and labour markets.
Feel free to download the commented presentation slides Economics Nobel Prize 2019 nanoTalk
Partly thanks to these studies, Vila-Belda pointed out, organizations like the World Health Organization, the United Nations Millennium Project, and the Department for International Development of the UK, now recognize that the elimination of user fees is the most effective way of quickly increasing take-up of key preventive health products.
The promising research findings based on randomized controlled experiments also contributed to increase funding to eradicate malaria: development assistance funds more than doubled in recent years. A rise in funding to distribute free bed nets has led to a rapid increase in the percentage of people sleeping under bed nets in Sub-Saharan Africa and this has been key to achieve a sustained reduction in the incidence of malaria across the region, saving millions of lives.
Quotes
We had two closely related ambitions. The first one was to contribute to improving the lives of the poor, here and now. The second was to build a better understanding of how they live their lives, from the ground up, by building a fuller picture, one question at a time.
Esther Duflo
Today, I am also proud to represent women, and in particular women in economics. Tellingly, Elinor Ostrom, the only other woman before me also relied on fieldwork and studied what we can learn from poorer societies, from Nepal to Indonesia. I don’t think this is an accident. Development is one field of economics that has its fair share of women.
Esther Duflo
Videos
Contact
José Vila-Belda is a PhD candidate in Economics at the University of Fribourg and a Research Manager at the Department of Economics, University of Zurich. Prior to joining the University of Zurich, he was based at the Latin American office of the Poverty Action Lab, where he worked with policymakers in Chile, Uruguay, Peru and Mexico using scientific evidence and designing experiments to evaluate the impact of their policies. At UZH, he manages research projects led by UBS Foundation Professor Dina Pomeranz, a J-PAL affiliate.
José Vila-Belda is a PhD candidate in Economics at the University of Fribourg and a Research Manager at the Department of Economics, University of Zurich. Prior to joining the University of Zurich, he was based at the Latin American office of the Poverty Action Lab, where he worked with policymakers in Chile, Uruguay, Peru and Mexico using scientific evidence and designing experiments to evaluate the impact of their policies. At UZH, he manages research projects led by UBS Foundation Professor Dina Pomeranz, a J-PAL affiliate.