Probability and statistics abounds with many fallacies, "paradoxes", and otherwise unintuitive statements, often caused by the presence of various sampling biases that might not be immediately obvious without some mathematical (or similarly careful) analysis. Some of my favorite examples of these phenomena are Simpson's paradox, Berkson's fallacy, and the friendship paradox (all of which can be found for instance on Wikipedia). 1/n
One such unintuitive fact (basically a special case of Simpson's paradox) is the following: the number of failures that a person experiences in a field can be positively correlated with how competent that person is in that field: the more failures a person generates, the more likely that person is to be competent! 2/
Let's see how. For sake of discussion, let's say that the field is book writing, and that there are only two types of book one can produce: good books (successes) and bad books (failures). Let's also say for simplicity that authors fall into two classes: competent writers, who produce good books 3/4 of the time and bad books 1/4 of the time, and incompetent writers, who produce good books only 1/4 of the time and bad books 3/4 of the time. 3/n
Intuitively, one would then expect that an author who has written a lot of bad books is more likely to be an incompetent author than an author who has written very few bad books. But this is not necessarily the case! The problem is that there is a confounding variable, namely the number of books each author writes. 4/n
To make a concrete numerical illustration of this, let us consider four types of authors, who are either competent or incompetent, or persistent or not persistent.
1. A competent, not persistent author might write just 4 books (say) during his or her career, of which 3 are good and 1 is bad.
2. An incompetent, not persistent author might also write just 4 books in his or her career, of which 1 is good and 3 are bad.
3. A competent, persistent author might write as many as 40 books over his or her career, of which 30 are good and 10 are bad.
4. An incompetent, persistent author might aspire to write 40 books in his or her career; however, faced with frequent negative reviews and poor sales, he or she may eventually quit after writing only 8 books, of which 2 are good and 6 are bad.
With this simple example, one sees that it is the competent author who ends up writing the most bad books; also, out of all the bad books published by the above four authors, the majority were written by competent authors rather than incompetent ones. Because of the biasing effect of discouragement of persistent, incompetent authors, the fact that an author has a large number of bad books to his or her name becomes evidence in favor of competent writing, rather than incompetent writing! 8/n
Searching Google suggests I paraphrased https://twitter.com/silvervvulpes/status/643474990290092032
@Geoff Oh, cool, thanks for tracking that down. I'll have a look and see what seems to be the right balance.
Lots of these clever, pithy sayings have many variants over time as the evolve into a "best version". Attributions in these cases can be hard, and sometimes not really justified.
Everyone is welcome as long as you follow our code of conduct! Thank you. Mastodon.cloud is maintained by Sujitech, LLC.