A natural experiment is a type of causal analysis that has been widely adopted by many organisations and research fields. It was such a game-changer that the pioneers of the idea were awarded the 2021 Nobel Prize in economics. This article will explore what makes natural experiments unique compared to regular experiments and how you can apply this innovative approach within your organization.

Causal analysis

Let’s start with an example.

There are many reasons why someone might want to go to university. One of them is the expectation of higher income. The assumption that a few more years of education will lead to higher income becomes even more critical if you need to pay for it. However, the figure above shows correlation, not causation. In other words, it shows us that education and income are related, but it is not a causal analysis.

So, how can you find out whether getting that master’s degree will result in higher income?

When we think about exploring causal relationships, we think about experimental design. Traditionally, experiments include a controlled environment and random assignment of participants. 

Can we apply this to our education and income question?

No, because once we move from pharmacy and medicine to social issues, it’s often impossible to get all the factors under control. We still have questions that need some type of causal analysis.

Natural experiments

Natural experiments, also called quasi-experiments, are similar to regular experiments, except that everything is under control.Going back to the education and income question, Joshua Angrist (one of the Nobelists) and Alan Krueger answered it. They compared people born in the first and in the last 3 months of the year, and they noticed that those born earlier in the year spent less time in education on average. They also had lower incomes compared to those born in the last quarter. 

This is a natural experiment as it uses the fact that nobody can decide when they are born – it’s random. Additionally, their analysis relies on the US educational policy that allows children to leave school when they turn 16 or 17 (depending on the state). The same study design wouldn’t work in countries with years-of-education limits rather than age-limit for leaving school.

Causal analysis is not reserved only for scientists.

Causal analysis in business

If you are familiar with user research, you know that it relies on causal analysis. Imagine you are a media outlet, and you want to determine which title will lead to more clicks. Come up with two titles and assign them to users randomly. Then you calculate the click-through rate for both groups, and you will find out which title is more clickable. This is an example of a regular experiment because we can perform the random assignment of the interventions (i.e. titles). 

Now, imagine you are in the leadership of a company in 2019. You are considering the option for the employees to work from home. You know that it works in some companies, but you are still hesitant. This is not something you want to experiment with on a massive scale. Suppose you tell your workforce to work from home for 2 months. What if your company loses money due to decreased productivity? You might think allowing the most reliable employees to work remotely would reduce the risk and help you answer the question. Yet, this would only lead to biased results. 

Do you remember the COVID-19 pandemic? 

😑😑😑

The pandemic is terrible, of course, but it occurred naturally and created an environment for natural experiments. When the pandemic happened, people whose jobs allowed them to work from home started working from home. This allowed us to understand the benefits and drawbacks of remote work much better.

But how exactly can you leverage the pandemic for causal analysis?

Regarding remote work, your natural experiment could compare the productivity level before the pandemic and when people worked from home. You still want to make the time frames as similar as possible. E.g. comparing the last quarter of the year before the pandemic with a second-quarter during the pandemic is not the best idea.

Generalisability

We can even apply causal analysis to our personal life, but we need to be very careful about our conclusions. Let’s say you take vitamins every day for 3 months and you notice you didn’t become ill. One of your conclusions could be that these vitamins strengthen immunity which protects you from diseases. Even though you may be right, just because something works for you doesn’t mean it will work for everyone in the same way. We don’t know for sure what would have happened if you didn’t take the vitamins. Maybe you wouldn’t get ill anyway. Every type of causal analysis has a specific level of uncertainty. Natural experiments are great, but randomised experiments still offer the strongest proof of a causal relationship.

Other resources:

Nobel Prize 2021 Press Release 

Note: One half of the 2021 Nobel prize was awarded to David Card “for his empirical contributions to labour economics” and the second half went to Joshua D. Angrist and Guido W. Imbens “for their methodological contributions to the analysis of causal relationships”.

Originally published on Hackernoon.com