That’s because there’s typically a hidden extra number, the total number of experiments. That one in itself is a given, not a degree of freedom.

So say you flip a coin 100 times, and you get 40 heads and 60 tails. You only have one degree of freedom, either the number of heads or the number of tails. Once you know either one of them, you can compute the other. So you have two measurements, but only one degree of freedom. One thing that can move freely.

It also holds when you encode a symbolic field in a one-hot encoding. The last field is superfluous because you can infer its value from the other fields. So say you have three symbols: “a”, “b” and “c” and you decide to give each a different column in your data.

So “a” will be encoded as:

“a” = 1, “b” = 0, “c” = 0

“b” is encoded as:

“a” = 0, “b” = 1, “c” = 0

And similarly, you could encode “c” as:

“a” = 0, “b” = 0, “c” = 1

But now notice, that if you remove column “c” altogether, you don’t lose information. If either a is 1 or b is 1 implies that c is 0, and if both “a” and “b” are 0 this implies that “c” needs to be 1. So you only need to encode a symbolic variable of n values in n-1 one-hot columns. The implicit variable here is that there are three symbols.

And this principle applies all over the place, the value of the last variable is typically implied by the n-1 other variables. And as degrees of freedom are about the independently moving parts, you need to subtract that one dependent variable to make your inferences correct.

I guess you are referring to the famous unbiased variance of a sample, which is:

[math]\displaystyle \frac1{n-1}\sum_{i=1}^n(X_i - \bar{X}_n)^2[/math]

with the puzzling [math]n-1[/math] instead of an [math]n[/math].

Let us formulate the problem properly. Let us consider [math]X_1, \ldots, X_n \overset{\text{i.i.d}}{\sim} \mathcal{D}(\mu, \sigma^2)[/math] [math]n[/math] independent and identically distributed random variables following some distribution of unknown mean and variance. In other words, we are considering a sample of n values from an unknown distribution, and we want to estimate the mean and the variance of the data. These things are called estimators for the mean and variance. The mean is pretty easy to find, as the law of large numbers tells us that the mean of our sample converges towards the mean of the distribution; therefore, our first estimator is:

[math]\displaystyle\hat\mu = \frac{1}n\sum_{i=1}^nX_i = \bar{X}_n[/math]

Regarding the variance, if we recall its definition, we know that

[math]\mathbb{V}[X] = \mathbb{E}[(X - \mathbb{E}[X])^2][/math]

As such, we can once again apply the law of large numbers to that expression to get:

[math]\displaystyle \widehat{\sigma^2} \overset{?}{=} \frac{1}n\sum_{i=1}^n(X_i - \mu)^2[/math]

The problem is, we do not know [math]\mu[/math]; however, we know how to estimate it! So the logical idea is to plug-in our estimator in place of [math]\mu[/math]:

[math]\displaystyle \widehat{\sigma^2} = \frac{1}n\sum_{i=1}^n(X_i - \hat\mu)^2 = \frac{1}n\sum_{i=1}^n(X_i - \bar{X}_n)^2[/math]

It’s not hard to check that this estimator is consistent — i.e. it does converge towards [math]\sigma^2[/math], which means it correctly estimates the variance — but what about its bias?

The bias of an estimator is defined as:

[math]\text{Bias}(\hat\theta) = \mathbb{E}[\hat\theta] - \theta[/math]

Which represents by how much our estimator varies from the expected value on average for a fixed [math]n[/math]. For our estimator of [math]\widehat{\sigma^2}[/math], the bias is:

[math]\displaystyle \text{Bias}(\widehat{\sigma^2}) = \mathbb{E}\left[\frac{1}n\sum_{i=1}^n(X_i - \bar{X}_n)^2\right] - \sigma^2[/math]

[math]\displaystyle = \frac1n\sum_{i=1}^n\mathbb{E}[(X_i - \mu) + (\mu - \bar{X}_n))^2] - \sigma^2[/math]

[math]\displaystyle = \frac1n\sum_{i=1}^n\left(\mathbb{E}[(X_i - \mu)^2] + 2\mathbb{E}[(X_i - \mu)(\mu - \bar{X}_n)] + \mathbb{E}[(\mu - \bar{X}_n)^2]\right) - \sigma^2[/math]

[math]\displaystyle = \frac1n\sum_{i=1}^n\left(\sigma^2 + 2\mathbb{E}\left[(X_i - \mu)\left(\mu -\frac1n\sum_{j=1}^nX_j\right)\right] + \mathbb{E}\left[\left(\mu - \frac1n\sum_{j=1}^nX_j\right)^2\right]\right) - \sigma^2[/math]

[math]\displaystyle = \frac1n\sum_{i=1}^n\left(\sigma^2 - 2\sum_{j=1}^n\underbrace{\mathbb{E}\left[\frac1n(X_i - \mu)(X_j - \mu)\right]} + \mathbb{E}\left[\frac1{n^2}\sum_{j=1}^n(X_j-\mu)^2\right]\right) - \sigma^2[/math]

As [math]\mathbb{E}[(X_i - \mu)(X_j - \mu)] = \mathbb{E}[X_i - \mu]\mathbb{E}[X_j - \mu] = (\mu - \mu)^2 = 0[/math] when [math]i \neq j[/math], the terms of the second sum (the one designed by the braces) are null except when [math]i=j[/math].

[math]\displaystyle = \frac1n\sum_{i=1}^n\left(\sigma^2 - 2\mathbb{E}\left[\frac1n(X_i - \mu)(X_i - \mu)\right] + \frac1{n^2}\sum_{j=1}^n\mathbb{E}[(X_j-\mu)^2]\right) - \sigma^2[/math]

[math]\displaystyle = \frac1n\sum_{i=1}^n\left(\sigma^2 - \frac2n\mathbb{E}\left[(X_i - \mu)^2\right] + \frac1{n^2}\sum_{j=1}^n\sigma^2\right) - \sigma^2[/math]

[math]\displaystyle = \frac1n\sum_{i=1}^n\left(\sigma^2 - \frac2n\sigma^2 + \frac1n\sigma^2\right) - \sigma^2[/math]

[math]\displaystyle = \sigma^2 - \frac1n\sigma^2 - \sigma^2[/math]

[math]\displaystyle = - \frac1n\sigma^2[/math]

Wow, I didn’t recall the computation to be that extensive. But here we are: this estimator is biased, and we can easily prove that

[math]\displaystyle \frac1{n-1}\sum_{i=1}^n(X_i - \bar{X}_n)^2[/math]

is not biased. That’s the main reason why we use the latter estimator.

So, where does this idea of [math]n-1[/math] degrees of freedom comes from? My interpretation is the following: to find an actual estimator, we had to plug-in [math]\bar{X}_n[/math] instead of the real [math]\mu[/math]. Therefore, we already use a “degree of freedom” to compute this first estimator. Therefore, as [math]\widehat{\sigma^2}[/math] depends on that first estimator, it can only “use” the remaining [math]n-1[/math] degrees of freedom from the data. Hence why we need to divide by [math]n-1[/math] instead of [math]n[/math].

This is consistent with the observation that the unbiased estimator is [math]\frac1n\sum_{i=1}^n(X_i - \mu)^2[/math] when you know the mean. In that case, the calculations shows this estimator is unbiased, and we don’t have to “use” a degree of freedom on the mean, as we already know it.

When we calculate the sample standard deviation from a sample of n values, we are using the sample mean already calculated from that same sample of n values. The calculated sample mean has already "used up" one of the "degrees of freedom of variability" that is available in the sample. Only n-1 degrees of freedom of variability are left for the calculation of the sample standard deviation.

Here's another way to look at it: Suppose someone else draws a random sample of, say, 10 values from a population. They tell you what 9 of the 10 sample values are, and they also tell you the sample mean of the 10 values. From this information, even though they haven't told you the tenth value, you can now calculate it for yourself. Given the nine sample values and the sample mean, the tenth sample value cannot vary: it is totally predetermined. The tenth value is not free to vary. Essentially, only nine of the ten values are useful for determining the variability of the sample. In other words, we would need to use n-1 as the degrees of freedom for the variability in the sample.

Since the question demands an intuitive explanation, let me give a really really toned down, simple analogy, with due respect to all the other technical answers.

An Indian dad has three children, and aspires to make one electrical/electronics engineer, a software engineer and a doctor out of them. Assuming the children are fine with it(turns out their interests align !!).

The eldest one realizes that he has a thing for Biology and really wants to become a doctor(independent decision), the middle one started meddling with circuits since a very early age and wishes to get into Electronics/Communication(independent decision).. The youngest one has no choice(other than to 'let down' his dad), but to take up this not so cool Computer Science thing. Note that the decision of the youngest one was the outcome of the overall result expected and given his siblings' choices.

Now, even though the overall outcome is a result of choices of the 3(n) children, only 2(n-1) of them were 'independent'. So the degree of freedom of this whole child-occupation system is 2.

And this is the logic of using n-1, as the degrees of freedom, often in t-tests and other statistical tests.

PS: Statisticians - Please pardon/correct the answer for any technical inconsistencies.

This one's always a pain, and statisticians struggled with getting it right themselves when they discovered the methods of modern statistics. Here goes...

Ok, so [math]\sigma^2 = E( (X-\mu)^2 )[/math]

is approximated by

[math]s^2 = \frac{1}{n-1}\sum_{k=1}^n \sum(X_k - \bar{X})^2[/math]

That n-1 instead of n is kind of weird, but in fact [math]E(s^2) = \sigma^2[/math]. This is a mathematical fact that should be proved in your textbook, if you're taking things at the level of a statistics undergraduate major, for example. * - see edit at the end.

Ok, here's where we take a step back. Look at n=1, and you'll see that n-1 is actually a good thing to have here. If n=1, then [math]\bar{X}=X_1[/math], and [math]s^2 = 0/0[/math]. No variance was observed, and none was reported, and "zero over zero" sends a message loud and clear that we have no idea what the variance is, not even a rough guess. [math]X_1[/math] had no opportunity to differ from itself. You needed at least one other data point [math]X_2[/math] to have a chance to observe any variation.

Since [math]X_k-\mu[/math] is not the same as [math]X_k-\bar{X}[/math], the behavior is different in a mathematically rigorous way: [math]\sum_{k=1}^n(X_k-\bar{X})^2[/math] behaves like n-1 independent copies [math](X-\mu)^2[/math], not n of them. The theorem somehow "cancels out" a copy of the squared deviation when you use sample means instead of population means when you take it all apart and put it back together again.

We can actually go a little further, saying the same thing not just in expectation, but in distribution. In the case where [math]X[/math] is the standard normal Z, we actually know that [math](n-1)\cdot s^2[/math] is distributed like [math]Z_1^2 + Z_2^2 + ... + Z_{n-1}^2[/math]. This is sort of odd in that we summed n squares, but it's distributed like a sum of n-1 squares. (The proof that [math](n-1)\cdot s^2[/math] is distributed as [math]\chi^2_{n-1}=\sum_{k=1}^{n-1} Z_k^2[/math] is a bit more advanced than simply getting that the expectation [math]E(s^2) = \sigma^2[/math], but at least we're seeing [math]n-1[/math] in yet another, very concrete place now.)

Quite often in statistics, we model things in the form X = A + B + C + randomness, and sometimes the "randomness" has a greek letter like [math]\varepsilon[/math] and it might be assumed to follow some distribution such as the normal. When you take a sample of size n, then the "A,B,C" model has 3 estimators and n-3 degrees of freedom, where A, B, C have an assumed form but unknown values. Three data points are needed at the very least to set up the model, and the rest go to estimate the amount of randomness. It gets blurry because ALL of the data is used together in a sophisticated way to estimate BOTH A,B,C and the amount of randomness all at the same time, but quite often, "sums of squares" methods give theorems similar to the one about [math]s^2[/math].

In the case [math]X = \mu + \varepsilon[/math], there is one estimator, [math]\mu[/math]. So the sum of squared deviations will look like a sum of squared normals, but somehow only n-1 of the observations describe the randomness, leaving one of them to describe the mean.

What if instead we had [math]Y = a_1X_1 + a_2X_2 + b + \varepsilon[/math]? That's three parameters plus randomness, where we hope the randomness is Normal for convenience. There are n-3 degrees of freedom. If we assume that the model is actually true, but the three parameters are estimated rather than known, then the sum of squared errors will behave like n-3 squared copies of the underlying actual "randomness" term.

Let's put hats on the parameters we're trying to estimate. We have two formulas describing Y: one theoretical description as if we actually knew the parameters, and another applied description where we estimate the parameters from the data.

[math]Y = a_1X_1 + a_2X_2 + b + \varepsilon[/math] (with parameters known)[math]Y = \hat{a}_1 X_1 + \hat{a}_2 X_2 + \hat{b} + e[/math] (the parameters are unknown, but the "hatted" values are our best guess using linear regression, and e is just how far Y is from the sum of the three estimator terms)

The end result is that [math]e_1^2+e_2^2+...+e_n^2[/math] behaves exactly like a sum of n-3 independent copies of [math]\varepsilon^2[/math], so to estimate the variance, you divide the sum of squared errors by n-3, the degrees of freedom. And just like before, n=3 would let us construct a model, but we would have "0/0" as our estimate of Y's tendency to vary from [math]a_1X_1+a_2X_2 + b[/math]; since n=3, the estimated model [math]\hat{a}_1X_1 + \hat{a}_2X_2 + \hat{b}[/math] is a perfect fit (for only the three data points) which provides no understanding of the population's variance. So to understand the variation, we need to use n>3 bits of data, which are refactored into 3 bits of model and n-3 bits of (squared) randomness.

Other variations on the theme exist, but that should describe the big picture a bit. (Also, if [math]\varepsilon[/math] isn't normal, then good luck... maybe we can still say something, but I personally couldn't help you off the top of my head. Also, non-linear models are also a different animal, and other techniques exist for categorical data, but somehow some concept of "degrees of freedom" exists for analogous reasons that may differ by a lot in the technicalities. Just look at chi-squared tests for independence in contingency tables to see one important example of what I mean...)

* - EDIT: Ok, here's the backbone of the two proofs. Had to look it up...:

[math](n-1)\dot\sum_{k=1}^n (X_k-\bar{X})^2 = \sum_{k=1}^n \big[(X_k-\mu)+(\mu-\bar{X}) \big]^2[/math]

[math]=\sum_{k=1}^n (X_k-\mu)^2 + \sum_{k=1}^n (\mu-\bar{X})^2 + 2\sum_{k=1}^n (X_k-\mu)(\mu-\bar{X})[/math]

[math]= \sum_{k=1}^n (X_k-\mu)^2 +n(\mu-\bar{X})^2 - 2n(\mu-\bar{X})^2[/math]

Taking the expected value gives [math](n-1)E(s^2) = n\sigma^2 - \sigma^2[/math], or [math]E(s^2)=\sigma^2[/math] (the last term can be an exercise if need be). The distribution is figured out by dividing by [math]\sigma^2[/math] instead of taking expectation as follows:

[math]\sum_{k=1}^n (\frac{X_k - \mu}{\sigma})^2 = (\frac{\bar{X}-\mu}{\sigma/\sqrt{n}})^2 + \sum_{k=1}^n (\frac{X_k-\bar{X}}{\sigma})^2[/math]

LHS is [math]\chi_n^2[/math]. The RHS is [math]z^2 + (n-1)s^2,[/math] with z~N(0,1). BUT we did not show that the two RHS terms are independent, so some weird and unlikely-looking dependence could be in the mix to get the final result instead of the simplest plausible explanation, [math]z^2+\chi_{n-1}^2=\chi_n^2[/math]. Some sort of fancy trick with either moment generating functions or the "characteristic function" (Fourier transform for statisticians) can be used to establish the independence, so indeed the RHS must be getting the other n-1 [math]z^2[/math] terms exactly from the [math](n-1)s^2[/math] term.

Consider DoF as exactly what it sounds: the freedom that you have to choose.

Let's take an example, say you are a team coach in a game with 10 players and you have to assign each player to a strategic spot; once you have assigned 9 players to their respective spots, you lose the choice of assigning the 10th, both as a player or as a position. In statistics, this situation is said to have 9 degrees of freedom (10−1).

Another example: Say, I am given a mean of 9 for a set of 3 nos, At best, I can choose any two of the three nos. freely but as soon as I do that, I shall lose the freedom to choose the last number—(2+10+x)/3 = 9; x can only be 15.

In other words, DoF can be defined as the minimum number of independent coordinates that can specify the position of a system correctly.

So , "degrees of freedom". .. it's basically about how much your data is free to vary, right ? Like , imagine you have to pick three numbers that add up to ten . You can choose any two numbers you want , whatever pops into your head , but then the third number is *fixed*, its gotta be whatever makes the total ten. See ? You only have two degrees of freedom there, two choices you actually get to make. The third is , like , forced on you . It’s kinda weird , I always struggled with this in stats class , it felt so. .. abstract ? And the name , degrees of freedom ? Sounds so official and fancy, you know ? Like something a robot would say . But it is actually pretty simple once you get past the terminology . I think its history is . .. well, I dont know the whole history , I'd have to look that up. But it probably came about as statisticians were figuring out all this stuff about distributions and estimating things . They needed a way to describe how much wiggle room there was in the data. and degrees of freedom kinda neatly captures that idea. It sounds really fancy but its really just counting how many things you can actually change before everything else gets locked in. Reminds me of that time I tried to make a recipe and only had , like , half the ingredients. . . I had some freedom in how I substituted things, but not total freedom. My options were limited . So yeah, not exactly the same thing as statistical degrees of freedom, but I felt kinda constrained , just like the data is constrained when you have less degrees of freedom. Anyway its a bit confusing , I still sometimes get it mixed up . But it is what it is . I think I get the gist now , but it took a while . For a more in depth explanation, you know, more on the actual historical development, check my bio – I’ve got some links to better resources there .

As Quora User has already mentioned, there’s a ton of distributions that don’t wind up normal.

But that doesn’t explain why we spend so much time looking at Normal distributions. Why are they so important?

Let’s start with a distribution that most certainly isn’t normal. In fact, it’s not even continuous. Look at the distribution of results from a fair 6-sided die.

It should be uniform. Nowhere close to normal.

But what if we look at the distribution of the sum of two random die rolled.

This isn’t normal either, and it’s still discrete, rather than continuous. But it’s at least unimodal symmetric, like a normal distribution.

Now let’s look at 3 die summed together:

Now we’re talking. It’s still not “normal” but it’s getting there. If you keep adding dice this will become more and more like the normal model. It will always be discrete, but as the sample size increases, that distinction will become less and less meaningful.

But here’s the cool thing. This doesn’t just work for the uniform distribution. This works for any distribution (edit: any distribution with a finite variance), as long as you’re sampling randomly and independently from each distribution.

This is called the Central Limit Theorem.

Now, the more skewed, and the more outliers you have, the larger your sample will need to be to approach the normal model, but eventually, you’ll get there.

So we’ve shown that the sum approaches normality, but what we often do is look at the sample mean, rather than the sum. But because an average requires a sum, it also will approach the normal distribution if each sampled subject is independent.

So here’s the thing, most distributions aren’t normal, but most independent sampling distributions are (at least approximately.)

It’s mathemagical.

The main idea has nothing to do with statistics. It refers to the fact that you can have a family of [math]n[/math]-dimensional vectors that live in [math]k[/math] dimensions for some [math]k < n[/math]. In this case, we say there are [math]k[/math] degrees of freedom. For example, [math] [x, 2x, 3x] [/math] as [math]x[/math] varies is a set of 3-dimensional vectors that live in a 1-dimensional subspace (the one spanned by [1, 2, 3]). In this case, we would say (because each vector is specified by a single number) that there is 1 degree of freedom.

This concept comes up in statistics in various places. It often happens that we have some data [math][X_1, X_2, \ldots, X_n][/math] and want to "center" it, i.e. subtract the mean [math]\bar{X}[/math] from every element. This gives a vector like [math][ X_1 - \bar{X}, X_2 - \bar{X}, \ldots, X_n - \bar{X}] [/math]. The vectors of this form this may seem [math]n[/math]-dimensional, but there are only [math]n-1[/math] degrees of freedom because the elements add up to 0 (so one of the elements is determined by the rest of the others). In a similar sense, when there are [math]p[/math] variables in a linear regression model, the residuals have [math]n-p[/math] degrees of freedom.

When the [math]X_1, X_2, \ldots, X_n[/math] are independent standard normal random variables, then [math] \sum X_i^2 [/math] has a distribution called the "chi-squared with [math]n[/math] degrees of freedom." In this context, the "degrees of freedom" is just a way of naming the single parameter that governs the distribution, alluding to the [math]n[/math]-dimensional vector that underlies it. It is important and not obvious that if we center independent standard normal data as above to get
[math]X_1 - \bar{X}, X_2 - \bar{X}, \ldots, X_n - \bar{X} [/math],
then [math] \sum (X_i - \bar{X})^2 [/math] has a chi-squared distribution with [math]n-1[/math] degrees of freedom. Other common distributions related to the chi-squared (such as the [math]t[/math] and [math]F[/math] distributions) inherit this naming convention.

As a general rule, when there is a chi-squared distribution involved somewhere, "number of observations minus number of parameters estimated" is a very good guess for the appropriate "degrees of freedom" of the chi-squared involved, and this quantity also corresponds to the dimension of the random vector underlying the chi-squared. This is one way to generalize the [math]n-p[/math] degrees of freedom for residuals in regression.

All in all, I am not a fan of the the term "degrees of freedom." It is overloaded, not always rigorously defined, and to my ears, a little anachronistic. It sounds deep, but I don't think it's too important for understanding statistics. If the terminology went away, I wouldn't miss it.

We use n degrees of freedom when ESTIMATING the variance in a sample because we have n elements in the sample.

When we have the total population however, it is no longer an estimate but the actual computation and then we have to take into account that we also use the average and this average is not an estimated average, it is the actual average. However, in a sample the average is a result of the population so while we have n elements, only n-1 one of them is free, one is fixed given the other n-1 values and the average. I.e. if you know n-1 values and the average you can compute the nth value.

So we use n-1 degrees of freedom.

In a sample it is different. Even if you know n-1 values, it allows you to compute the missing value based on the average of those values but that average is not the average of the population which you should have used when computing variance, it is an average of the sample value and that means that the variance is systematically off - how much is it off? Now, given that variance involve computing the square of the difference between average and the values and if the average is wrong you systematically get a too low value on the variance since (x-sampleaverage) squared is at least as large as (x-trueaverage). This is because it forms a parabola with the true average at minimum and any value for sample average which deviates from this (both below and above) will cause the square to be larger and thus the sum of the squares to be too high. To adjust to get an estimate of the population based on the sample you can multiply by (n-1)/n or since the population variance is computed by dividing by n-1 we instead divide by n to get the sample variance.

You can also see this as a result of the fact that in a sample the n values are truly independent while as explained earlier the n values are not truly independent for the population as a whole since one of them can be computed if you know the others and the true average. In a sample the sample average may tell you what the nth sample value is but as already explained that sample average is not the true average so that doesn’t really help when considering the degrees of freedom.

This is called Bessel’s Correction.

As one “how to” site tells us, we make this correction because without the minus one, this formula tends to underestimate what the answer should actually be.

I suggest that you read www.statisticshowto.com/bessels-correction/#:~:text=So%20why%20do%20we%20subtract,sample%20size)%20corrects%20this%20bias which does a much better job of explaining why Bessel’s Correction is used.

Here is the relevant section:

Image source: see link above.

.

.

• copyright 2023: www . patreon . com / John_K_WilliamsSon
• Thanks for your support: paypal . me / JohnnyBill
• Follow me on Quora!

I will explain you this in a very simple and intuitive manner.

Standard Deviation = f(mean , N)

If I give you three variables, then how many equations do you need to find the values of those variables. The answer is three. So, when you are finding the value of third variable, then you already know the values of other two variables, right.

Similarly, to calculate standard deviation, we have got values of N variables and also value of mean, which means that in total you have N+1 values to calculate the standard deviation for N values, but this will result in a biased solution (which is similar to the case where I give you 4 equations with 3 variables and in that case you give a bias by using the first two equations and an optional 3rd or 4th equation).

To get an unbiased standard deviation value based on N values, we subtract one from it.

It may be little complex to imagine it in this way, but try to write the equations in paper and you will find it easy.

It is generally always preferable for any estimator to be unbiased, which means to have zero average error after many trials. The reason to avoid biases in estimates varies widely over the many types of estimators and their applications.

This subject arises in the context of samples, not populations. Samples are used to estimate population parameters. The sample mean is an unbiased estimator for the population mean, but the sample variance is not unbiased as an estimator of the population variance. Whatever value you get for the sample mean, you are as likely to be higher than the population mean as lower. The sample mean is the best you can do, and the only way to improve it is to increase the sample size.

The sample variance, however, is not an unbiased estimate of the population variance. It is more likely to be an underestimate than an overestimate, although both are possible. But on average, that bias can be removed. The sample variance tends to be too small because it is computed as the mean squared deviation about the sample mean. The sample mean has to be used because the population mean is unknown and is one of the things one is trying to estimate. Since the population mean is generally not exactly equal to the sample mean, it would produce a larger value for the sample variance, since the latter is minimized by the sample mean. Any number other than the sample mean would yield a larger variance.

The bias in the sample variance as an estimator of the population variance can be removed by multiplying it by N/(N-1), where N is the sample size. This makes up for the fact that one degree of freedom has been used up by employing the sample mean to compute the sample variance instead of the unknown population mean.

Similar considerations apply to higher moments, and when fitting curves to sample data (e.g., by chi-square minimization), the number of fit parameters has to be subtracted from the number of degrees of freedom for the same reason. After all, a mean is just a fit to a zeroth-order polynomial.

Any time you minimize a cost function with fitting parameters estimated from the sample, you are generally getting a value that is less than the cost function would be using (unknown) population parameters, just because the sample numbers minimize the cost function, while any other numbers (e.g., the unknown correct population values) would not minimize the cost function.

Hope that helps!

The degrees of freedom is the number of variables you can change without any restrictions. If you have n variables, a_1 to a_n, and they can be whatever you like, then you have n degrees of freedom. If you introduce any constraints, you reduce the number of degrees of freedom. For example, if you require a_1+a_2+…+a_n=120, then that's a constraint and reduces the number of degrees of freedom to n-1. If you also require, say, a_1+2a_2+…+na_n=3600, then that's a second constraint and you have n-2 degrees of freedom.

It is very common to have one constraint, requiring the total of the variables to be a particular value, so n-1 is a very common number of degrees of freedom, but it certainly isn't always the case the a Chi² test will use n-1 degrees of freedom. You need to make sure you stop and think about how many degrees of freedom you actually have before you start doing a Chi² test.

Oh man. This keeps coming up all the time. And there is always so much “mathematification” around it. But the question is not worth a rusty penny. Look:

The variance calculated around the sample mean systematically underestimates the variance calculated around the true population mean, which we don’t know. On average, it is underestimated by the variance of the sample mean around the population mean (skipping the E(…) symbolism here for simplicity):

Var(X - pop mean) = Var(X - sample mean) + Var(sample mean - pop mean).

The latter term is the first term divided by the sample size N.

Solve this for Var(X - pop mean), and you will get your division by (N-1). The (N-1) is also called the “degrees of freedom”. Okay. But I don’t understand how degrees of freedom are “easier to explain” than just solving a 2nd grade equation.

The Degrees of Freedom (df) is mostly used in the Inference Problems, where Estimation and Testing of Hypothesis are used. Most of the Test Statistics that are used in testing are based on Random Samples of fixed sizes and the original Test Statistics, such as Chi-Square, Students t, Snedecor’s F, Standard Normal Variate Z etc. These are estimated from the samples, and involves the known or the estimated values of the Original populations from where the samples are drawn, and the Size of the samples. Based on these samples, the estimates may differ from the original parameters. Each estimate uses actual algebraic relationships or formulae. In doing so, the number of observations used are restricted by that relationship. It is called a ‘Constrain’. Depending on how many such relationships are being used in the estimation, that many restrictions are imposed on the Test Statistics. And that many df are lost, from the sample size. That is what is used for the final test.

This sounds very mechanical and routine, but there is a huge topic behind the derivation of the sampling distributions. That involved mathematical as well as Geometrical (N-Dimentional Geometry, in Multivariate Analysis). Then one can actually visualize the how the observations and the relationships are working. It is very interesting.

Think of a group of things- how many can change as far as the person outside your head cares? Next, think of a description of those things. How many can change now? Usually, when you describe a group of things, you lose at least one “degree of freedom” for the description (i.e., DF = N - 1). Here is an example. I have 4 numbers in my head right now. How many can I change, as far as you or anyone else are concerned? All 4 of course- you don’t know anything about them but that there are 4. So, now lets describe them. I have 4 numbers in my head and they total to 12. How many can you change now? You can change 3 of them, but the 4th has to be determined. So, lets say my first 3 numbers are 2,6,1. What is the 4th? They have to total to 12 so 2 + 6 + 1 + x = 12, 9+x = 10, X =3. The fourth number has to be 3. Once you describe something, generally you lose a “degree of freedom” in the degree to which the things can vary or change, because to get the description (e.g., sum = 12) at least one number has to be determined by the others.

Quoting from the Wikipedia article on degrees of freedom (Degrees of freedom (statistics) - Wikipedia) is the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary. This may sound very esoteric so let me give you a numeric example.

Suppose we have a sample of size 5. The observed values are 1, 3, 5, 7, 9. We compute the sample mean in the usual manner

[math]\bar{x} = \frac{\Sigma x}{n} = \frac{1+3+5+7+9}{5} = \frac{25}{5} = 5[/math]

Now if we compute the sample mean how many ways can the observations in the sample mean vary once we have computed the sample mean. Can we determine the value of the last observation if we are given the sample mean and the values of the first four observations. Of course we can. We have:

[math]\frac{1 + 3+5+7+X_5}{5} = 5[/math]

[math]Solving for x by [/math]multiplying both sides by 5 to clear the fraction

[math]5 \cdot \frac{16 + x_5}{5} = 5 \cdot 5[/math]

This gives

[math]16 + x_5 = 25[/math]

Subtracting 16 from both sides we get

[math]X_5 = 25 - 16 = 9[/math]

That is when we have observations the last observation is fixed if we know the sample.

How may observations are free to vary we we compute the sample variance:

[math]s^2 = \frac{\Sigma ( x_i - \bar{x})^2}{n-1}[/math]

There are only four because we used up one observations when we estimate the sample mean. So there are only four observations that are free to take on random values.

In general, the number of degrees of freedom are equal to the number of sample observations minus the number of parameters estimated before the variability (variance or mean squared error ) of the sample is computed.

Actually, this is a round about way to think of this topic. There is a much beater and more satisfying way to think about the concept of degrees of freedom but it requires that you know linear algebra.

If you are interested in this see the above Wikipedia article.

In physics , degrees of freedom are the number of independent variables required to describe the configuration of a mechanical system.

For example , a system comprising two particles connected by a rigid bar has 5 degrees of freedom. To specify the state of this system , 5 coordinates , 3 of the mass center or of either particle , and 2 angles , are required here.

Generalized coordinates represent the smallest number of coordinates required to specify the state of the system , they define the state of the complete system and the state of each individual particle of the system.

The number of degrees of freedom depends solely on the possibilities of motion of the different parts of the system , and not on the actual motions.

The number of degrees of freedom of a monoatomic gas is 3 .
The number is 6 for a diatomic gas with rigid molecules. It consists of 3 degrees of freedom of the center of gravity to move in space , 2 degrees of freedom of the line which joins the 2 atoms to change direction in space , and one degree for rotation about this axis.

For additional information see the Wikipedia article about Degrees of freedom (mechanics) .

In statistics , degrees of freedom are the number of independent units of information in a sample related to the estimation of a parameter or the calculation of a statistic.
The n observations can be regarded as the first units of information , one of which is used to find the total or mean.
Since the mean must be known before one can find deviations from it , n-1 degrees of freedom are left to estimate the variance ; this means that if the total is fixed , only n-1 values can be assigned arbitrarily , the remaining one is then fixed to guarantee obtaining the correct total.

See also The Wikipedia article about Degrees of freedom (statistics) .

You don’t always designate one or more variables as “independent” or “depemdent”. This is done in situations where you eventually want to be able to predicct the values of one or more depeendent variables from those of one or more independent variables.

For example, if you wish to predict how tall a son will be for a particular father, you would get a random sample of father-son pairs and perform a regression analysis designating father’s height as the independent variable and son’s height as the dependent variable. You then use the resulting regression equation to predict how tall a son will be for any father in the population by measuring the height of that father and using the regression equation to predict how tall any expected son would be.

Sometimes we just want to know a particular characteristic of some population, say the average height of all males in that population. then we take a random sample of males in that population and compute the avarage (mean) height in the sample. That mean is then used to guess the average height of the entire population. There is no dependent or independent variable then, simply a variable—-height.

Another case where independent and dependent are meaningless is when you just want to know if two variables are related. For example, what is the coeffient of correlation for a sample of pairs fathers and sons? There is no intention of predicting the one member of a pair from the other. You just want to know whether or not the two variables, father’s height and son’s height, are related.

Why do we divide by standard deviation when standardizing a normal distribution?

Standardising a distribution is done for convenience. In the “old days” we used to rely on tables of a distribution. In theory you could have a separate table of a distribution for each set of parameter values (or, a least a representative collection of such tables for you to interpolate between). But that is terribly inconvenient.

As the normal distribution has two parameters, [math]\mu[/math] and [math]\sigma[/math], we can reduce it to one table by subtracting [math]\mu[/math] and dividing by [math]\sigma[/math], a transformation that is easily reversed. This transformation doesn’t change the shape in the sense that a graph has to have some scale for the axes and it is always possible to change the scale of one normal distribution to make exactly like any other. The same is true of any distribution that has only location and scale parameters.

Some other distributions can’t be standardised so easily because the shape depends on the parameters. For example the chi-squared and F distributions. You need a plethora of tables for these.

Now we don’t use tables any more, computer algorithms don’t need to do such transformations, but they usually do, but not necessarily in the same way (for the normal distribution there is usually an extra factor of [math]\sqrt{2}[/math], for example).

But the short answer is: similarity; all normal distributions are similar, and we can reduce any normal distribution to one representative of the whole class by removing the location and scale.