When using data to answer a question, your results might not be perfect. Lets think about two scenarios:
- Lets say you are trying to figure out if heads and tails are equally likely when you flip a coin. You flip a coin 6 times, and get THTTTH. That's odd... they should be the same. But numbers don't lie right? So you decide that your original hypothesis that they were equally likely is wrong.
- Having lost all confidence in your understanding of probability, you set out to test something you know is different, your expected winnings vs. expected losses when gambling. So you buy $100 worth of lottery tickets, and surprisingly receive $100 in winnings. You conclude that you are expected to break even in the long run.
In statistics, the first case is called "Type 1 error," deciding that the hypothesis that there is no difference (called the null hypothesis) is false when it is true. The second case is called "Type 2 error," deciding that there is no difference, when there is one.
Understanding that statistics can be misleading is important when making choices. But even more important is asking the right questions. The focus of this blog is avoiding this "Type 3 error," where you might get the right solution to a problem, but it is not really the problem you wanted to solve. The really unique thing about Type 3 error is that it can be eliminated completely by making sure you know which question you need to answer.
