Great article, but I wish it would have made a more explicit mention of the* central limit theorem (CLT), which I think is what makes the normal distribution "normal." For those not familiar, here is the jist: suppose you have `n` independent, finite-variance random variables with support in the real numbers (so things like count R.V.s work). Asymptotically, as n->infinity, the distribution of the mean will approach a normal distribution. Usually, n doesn't have to be big for this to be a reasonable approximation. n~30 is often fine. The CLT extends in a
To me, this is one of the most astonishing things about probability theory, as well as one of the most useful.
The normal distribution is just one of a class of "stable distributions," all sharing the properties of sums of their R.V.s being in the same family.
The same idea can be generalized much further. The underlying idea is the distribution of "things" as they get asymptotically "bigger." The density of eigenvalues of random matrices with I.I.D entries approach the Wigner Semicircle Distribution, which is exactly what it sounds like. It plays the role of the normal distribution in the very practically-promising theory of free (noncommutative) probability.
Good for you for stating the assumptions properly that go into the CLT and for mentioning other stable distributions.
I disagree about the Gaussian being the "normal" case or the "one that usually matters". Finite variance is a big assumption and one that's routinely violated in practice.
For those that are interested, Levy-stable distributions are the general class of convergent sums of random variables [0], synonymously called "fat-tailed" or "heavy-tailed" distributions and include Pareto [1] and the Cauchy distributions [2].
Is there an intuitive explanation for why the Wigner semicircular law is basically the "logarithm" the Gaussian in some respect?
> The best way to do that, I think, is to do away entirely with the symbolic and mathematical foundations, and to derive what Gaussians are, and all their fundamental properties from purely geometric and visual principles. That’s what we’ll do in this article.
Perhaps I have a different understanding of "symbolic". The article proceeds to use various symbolic expressions and equations. Why say this above if you're not going to follow through? Visuals are there but peppered in.
Agree. This text relies heavily on traditional mathematics to define and work through things. It's quite good at that! But it does become weird when it starts out by declaring that it won't do what it then does.
Gaussian, Gaussian, Gaussian. Important to understand Gaussians, but also to recognize how profoundly non-Gaussian, in particular multimodal, the world is. And to build systems that navigate and optimize over such distributions.
(Not complaining about this article, which is illuminating).
A particularly interesting case is Maxwell-Boltzmann distributions of the speeds of molecules in a gas in a 3D space. Even though the individual velocities of gas molecules along the x, y and z directions do follow Gaussian distributions, the distributions of scalar speeds do not (since the speed is obtained from the velocities by a non-linear transformation), resulting in a long tail of high velocities, and a median value less than the mean value.
Incidentally human expertise and ability seems to follow the Maxwell-Boltzmann model far more than the Gaussian 'bell curve' model - there's a long tail of exceptional capabilities.
> You can see that the data is clustered around the mean value. Another way of saying this is that the distribution has a definite scale. [..] it might theoretically be possible to be 2 meters taller than the mean, but that’s it. People will never be 3 or 4 meters taller than the mean, no matter how many people you see.
The way the author defines definite scale is that there is a max and a minimum, but that is not true for a gaussian distribution. It is also not true that if we keep sampling wealth (an example of a distribution without definite scale used in the article), there is no limit to the maximum.
Jinlian (1964–1982) of China was 8 feet, 1 inch (2.46 centimeters) when she died, making her the tallest woman ever. According to Guinness World Records, Zeng is the only woman to have passed 8 feet (about 2.44 meters)
Mean from article 163.
So the facts check out.
Author is correct.
Also very interesting the suggestion that human height is not Gaussian.
Snip :
“
Why female soldiers only? If we were to mix male and female soldiers, we would get a distribution with two peaks, which would not be Gaussian.
“
Which begs the question what other human statistics are non Gaussian if sexes are mixed and does this apply to other strong differentiators like historical time, nutrition, neural tribes ?
It's an oversimplification but at some point there is really no difference between impossible and 'incredibly small probability'.
I mean sure it is possible for all air molecules to randomly all go to the same corner of the room at the same time (heck it is inevitable in some sense), you can play it back in reverse to check no laws of physics were broken, but practically that simply does not happen.
Great article, but I wish it would have made a more explicit mention of the* central limit theorem (CLT), which I think is what makes the normal distribution "normal." For those not familiar, here is the jist: suppose you have `n` independent, finite-variance random variables with support in the real numbers (so things like count R.V.s work). Asymptotically, as n->infinity, the distribution of the mean will approach a normal distribution. Usually, n doesn't have to be big for this to be a reasonable approximation. n~30 is often fine. The CLT extends in a
To me, this is one of the most astonishing things about probability theory, as well as one of the most useful.
The normal distribution is just one of a class of "stable distributions," all sharing the properties of sums of their R.V.s being in the same family.
The same idea can be generalized much further. The underlying idea is the distribution of "things" as they get asymptotically "bigger." The density of eigenvalues of random matrices with I.I.D entries approach the Wigner Semicircle Distribution, which is exactly what it sounds like. It plays the role of the normal distribution in the very practically-promising theory of free (noncommutative) probability.
https://en.wikipedia.org/wiki/Wigner_semicircle_distribution
Further reading:
https://terrytao.wordpress.com/2010/01/05/254a-notes-2-the-c...
*there's a few normal distribution CLTs, but this is the intuitive one that usually matters in practice
Good for you for stating the assumptions properly that go into the CLT and for mentioning other stable distributions.
I disagree about the Gaussian being the "normal" case or the "one that usually matters". Finite variance is a big assumption and one that's routinely violated in practice.
For those that are interested, Levy-stable distributions are the general class of convergent sums of random variables [0], synonymously called "fat-tailed" or "heavy-tailed" distributions and include Pareto [1] and the Cauchy distributions [2].
Is there an intuitive explanation for why the Wigner semicircular law is basically the "logarithm" the Gaussian in some respect?
[0] https://en.wikipedia.org/wiki/L%C3%A9vy_distribution
[1] https://en.wikipedia.org/wiki/Pareto_distribution
[2] https://en.wikipedia.org/wiki/Cauchy_distribution
> The best way to do that, I think, is to do away entirely with the symbolic and mathematical foundations, and to derive what Gaussians are, and all their fundamental properties from purely geometric and visual principles. That’s what we’ll do in this article.
Perhaps I have a different understanding of "symbolic". The article proceeds to use various symbolic expressions and equations. Why say this above if you're not going to follow through? Visuals are there but peppered in.
Agree. This text relies heavily on traditional mathematics to define and work through things. It's quite good at that! But it does become weird when it starts out by declaring that it won't do what it then does.
It also felt like this could be a good topic for a 3b1b video... and... here's the 3b1b video on gaussians: https://www.youtube.com/watch?v=d_qvLDhkg00
Gaussian, Gaussian, Gaussian. Important to understand Gaussians, but also to recognize how profoundly non-Gaussian, in particular multimodal, the world is. And to build systems that navigate and optimize over such distributions.
(Not complaining about this article, which is illuminating).
There was an opportunity when heights of soldiers were discussed. Gaussians have infinite extent, but soldier heights must be positive.
Good example
A particularly interesting case is Maxwell-Boltzmann distributions of the speeds of molecules in a gas in a 3D space. Even though the individual velocities of gas molecules along the x, y and z directions do follow Gaussian distributions, the distributions of scalar speeds do not (since the speed is obtained from the velocities by a non-linear transformation), resulting in a long tail of high velocities, and a median value less than the mean value.
Incidentally human expertise and ability seems to follow the Maxwell-Boltzmann model far more than the Gaussian 'bell curve' model - there's a long tail of exceptional capabilities.
> You can see that the data is clustered around the mean value. Another way of saying this is that the distribution has a definite scale. [..] it might theoretically be possible to be 2 meters taller than the mean, but that’s it. People will never be 3 or 4 meters taller than the mean, no matter how many people you see.
The way the author defines definite scale is that there is a max and a minimum, but that is not true for a gaussian distribution. It is also not true that if we keep sampling wealth (an example of a distribution without definite scale used in the article), there is no limit to the maximum.
I think he’s saying that the distribution of human heights has definite scale, not the Gaussian?
Nothing is Gaussian then. What probability distribution allows for Graham's Number to be a possibility?
Jinlian (1964–1982) of China was 8 feet, 1 inch (2.46 centimeters) when she died, making her the tallest woman ever. According to Guinness World Records, Zeng is the only woman to have passed 8 feet (about 2.44 meters)
Mean from article 163.
So the facts check out.
Author is correct.
Also very interesting the suggestion that human height is not Gaussian.
Snip :
“ Why female soldiers only? If we were to mix male and female soldiers, we would get a distribution with two peaks, which would not be Gaussian.
“
Which begs the question what other human statistics are non Gaussian if sexes are mixed and does this apply to other strong differentiators like historical time, nutrition, neural tribes ?
Statistics is highly non-trivial. “
It's an oversimplification but at some point there is really no difference between impossible and 'incredibly small probability'.
I mean sure it is possible for all air molecules to randomly all go to the same corner of the room at the same time (heck it is inevitable in some sense), you can play it back in reverse to check no laws of physics were broken, but practically that simply does not happen.
https://gregorygundersen.com/blog/2020/04/11/moments/
> If we want to stretch a function f(x) vertically by a factor of y, we should multiply its input by 1/y: f(1/y x)
I didn't quite follow this part.
Possibly the author meant "horizontally".
Thanks for sharing! The Gaussian distribution never gets old. And nice plot of this:
> 100 000 points drawn from a Gaussian distribution and passed through a randomly initialized neural network.
It gives you a sense of how complex the folding of the space by NNs can be. And also the complexity of the patterns that they can pick up.
Now explain Gaussian splatting