Top Pages
biases
cognitive effects
fallacies
deception with statistics
effective business writing
body language
Small Sample Pool
The Risk of a Small Sample Pool in Statistics
If you needed to know what a large group of people thinks or how a product affects them, it would be impractical to ask them all. So, such information is usually gathered by testing a smaller "sample" pool. There's an issue though. If the sample group is too small, it increases the risk of chance affecting the outcome. If the sample size is too big, this risk is reduced, but those impracticalities start to mount up. So, there's a balance to be struck. More specifically, the size of the sample group should be sufficient to meet the required confidence levels and margin of error for the test.Of course, using a sample pool of sufficient size is what a scrupulous researcher would do. There are, however, lots of unscrupulous researchers out there who are only too happy to use small, even hand-picked, sample pools to ensure the outcome they need.


That statement looks pretty conclusive, but what if the sample was only 10 cats and it was just a statistical anomaly? Small samples can create errors - it's the opposite of the Law of Large Numbers.
That aside, there are other questions. What did those cats prefer the cat food to? How many other options were offered? What if they were hungry or greedy cats who would eat the first thing plonked down in front of them? How did they even measure preference? However it was done, it looks like the figure was manipulated.
Beware Small Sample Pools When Reviewing Claims
Using a small sample pool is a common way to achieve the results you need. Let's imagine I was trying to sell you a coin that guaranteed heads every time you tossed it:"Guaranteed head every time you toss.
Watch.
See, a head.
Right, who wants to buy it?"
Watch.
See, a head.
Right, who wants to buy it?"

Look for the P-Value in Statistics
If you read a scientific paper describing intellectually rigorous and repeatable (a key point) tests, you might come across something called a P-value. A P-value gives a percentage chance of the answer being arrived at by chance alone (at least that is a good enough definition for our purposes). Obviously, the lower the percentage, the better. Quite simply, a P-value of 1% means that there is a 1 in a 100 chance that the result was "luck". Because of the way the Law of Large Numbers works, the larger the sample size, the lower the P-value.Critical Thinking Test
Are you good at spotting the biases, fallacies, and other cognitive effects? Can you spot when statistics have been manipulated? Can you read body language? Well, let's see!
Gold

Silver

Bronze
0
- This test has questions.
- A correct answer is worth 5 points.
- You can get up to 5 bonus points for a speedy answer.
- Some questions demand more than one answer. You must get every part right.
- Beware! Wrong answers score 0 points.
- 🏆 If you beat one of the top 3 scores, you will be invited to apply for the Hall of Fame.
Scoring System
Guru (+)
Hero (+)
Captain (+)
Sergeant (+)
Recruit (+)
Guru (+)
Hero (+)
Captain (+)
Sergeant (+)
Recruit (+)
- Do you disagree with something on this page?
- Did you spot a typo?
- Do you know a bias or fallacy that we've missed?