This is an expansion on some of the material in the textbook on pages 375 - 381. You should read that material first - especially “the null hypothesis” on p378 and “the alpha level (level of significance)” on p. 379-380.
This is a difficult concept for some students. Although it is a concept that you should have learned in your statistics class, we go over it again in Psyc 220 because some students have trouble with it. In the past, some students have found it helpful to think of it the following way.
When you carry out an experiment and get data, you evaluate your data using the following logical steps:
1. Identify your hypothesis. This could be something like
this:
There will be a difference between scores in Group A and Group B
There is a correlation between variable K and variable L
2. Assume the null hypothesis.
That is, assume that there is NO difference, or NO effect,
or NO correlation, and the results you observed were just due to chance.
3. Ask yourself:
Given the data I obtained, how likely is it that the null hypothesis
is correct?
or, in other words,
What is the probability that the results I obtained are due to chance?
4. Do a statistical test to determine that probability. (For now, we are not worrying about WHAT statistical test, or HOW that statistical test gives you an estimate of the probability that results like yours could occur due to chance. We just believe that, because we learned all about this in statistics class, there IS some test somewhere that will tell us the probability that results like yours would occur due to chance - that is, the probability that results like yours would occur if the null hypothesis were correct and there were no real effect or no real difference between the groups.)
5. Find out whether the statistical test gives a very low probability that the null hypothesis is correct. If that is the case, we can reject the null hypothesis. For example:
IF our statistical test gives the result of p<.05
THEN this means
with our data, the probability is less than 5% that the null hypothesis is correct.
NOTE THAT THE FOLLOWING PHRASES ALL MEAN THE SAME THING:
- With our data, the probability is less than 5% that the null hypothesis is
correct.
- With our data, the probability is less than 5% that the results were due to
chance.
- It is very unlikely that our results were due to chance.
- The probability is very low that our results were due to chance.
- The probability is very low that the null hypothesis is correct.
6. And if this is true, we can say:
- The results were statistically significant.
- We can reject the null hypothesis.
- We have a high level of confidence that the results were not
just due to chance.
- We have a high level of confidence that there was a real effect.
Once again, these are all different ways of saying the same thing.
ADDITIONAL POINTS:
7. Why did we choose a level of 5%? We wanted to know whether “the probability
is very low” that the null hypothesis is correct and the results were
due to chance. Exactly what is meant by “the probability is very low”
is debatable. We chose 5%, or less than 5 in 100, to be an acceptable level.
We could have demanded a more rigorous level - for example, .02 (that is, 2%,
or only 2 in 100), or .01 (that is, 1%, or only 1 in 100). The standards usually
used in psychology are 5% and 1%.
We might find: the probability is only 5% (or less than 5%). This is p<.05
We might find: the probability is only 1% (or less than 1%). This is p<.01
8. The lower the probability that the results could have been
due to chance, the more confidence we have that the results
were NOT due to chance.
or, if we are examining a difference between two groups,
The lower the probability that the results could have been
due to chance, the more confidence we have that the results
show a real difference between groups.
or, if we are examining a correlation,
The lower the probability that the results could have been
due to chance, the more confidence we have that a correlation
we observe shows a real relationship between two variables.
TEST QUESTION: which experimenter has MORE confidence that the results she
obtained represented a real effect and were NOT just due to chance?
Experimenter A, whose results were statistically significant at p<.05
OR
Experimenter B, whose results were statistically significant at p<.01?