This week p-value made it to the Science news feed. The whole story
started already in the middle of last year, when Bejamine and coworkers
published in Nature Human Behaviuor a paper titled „Redefine Statistical Significance“. The group of some 80 scientists propose, that the major part of
research studies should not be conducted at the significance level of 0.05 but
at 0.005. They claim that this could be an efficient measure to increase the
reproducibility of the scientific results. Now, Lakens together with ca 100
coworkers published a response in defence of assigning p-value = 0.05 as statistically significant. They argue, that this would increase tremendously the size of the studies. The
studies would need to analyse far more data points or samples, would take much
longer and of course would be much more expensive. The latter may result in a
decrease of confirmative studies.
So why really all this fuss? Obviously, p-value is somewhat complicated
to understand, students hate it and the scientist just like to state that the
effects observed were significant. I personally was pretty much surprised by
this topic popping up in the first place as it somewhat goes against my
fundamental understanding about statistics.
So I try explaining the concept and the problems very shortly and what
is the problem. First, usually when we do science we are looking into some
differences or similarities. Unfortunately statistics allow us only testing if
two or more things are the same and it does not allow us to directly test if
our observations are different. Therefore, the whole concept relies on two
hypotheses: the so called Null-hypotheses, that our observation is indifferent
from something else. Let’s look in on the examples. So let’s say that we are
investigating if the temperature within the last ten years is higher than is
was between 1950 and 1960 in our hometown. So our Null-hypotheses is that the
temperature from 2007 to 2017 is indistinguishable from the average temperature
between 1950 and 1960. The other possibility, logically called alternative
hypotheses, is that, the average temperature is not the same.
Now comes the p-value into play. Because of these random variations in
our measurements caused by very-very different reasons starting from
fluctuations in instruments up to sudden geological processes such as volcanic
eruption the averages can not be exactly the same even if there is no real
increase in the temperature. Therefore, we take into account the variation in
temperature, called standard deviation, and conduct a suitable statistical test
to reach a conclusion. However, due to these variations we are also not 100%
sure in our average values and can only with some certainty say if the
temperatures really are the same. p-value represents this certainty, or better
yet uncertainty. For example, if we conclude that the temperatures are significantly
different with p-value of 0.05 we actually mean, that even though we state,
that T1 does not equal T2 it has a 5% change that they actually are the same.
So from this perspective it would be better to use a lower p-value as an
indication of significance. If we state something is significant at p-value of
0.005 (or lower) it means that we would assign a significant difference only to 0.5% of
the cases where there actually is no difference.
However, the picture is not so simple. It also has a completely other
pole. The p-value we talked above is associated with a so called false positive
results. This means that in reality there is no effect but based on our test we
would be claiming an effect. Still, it could also be the other way round. It
can be that in reality the temperature is different between now and then but we
do not reach this conclusion with our tests. This is called a false negative
result. Unfortunately there is basically no easy way to assign a probability to
such a false negative result. But one thing we know for sure is that if we
decrease the probability of one type of error the probability of the other type
of error will increase if the sample size is constant. Therefore if we really-really
do not want to allow us being mistaken if we claim that the temperature now and
then is different even if it is not, we have to realize that it becomes more
likely, that we say that we conclude that the temperature is the same even if
it actually is not. The only way to reduce both type of error is to increase
the sample size, this is the main point of Lakens and the co-workers in the
paper published this week.
But let’s look at another example. Say that we are trying to find a new
drug for a disease. So basically the null hypotheses is here that the drug is
as efficient as a placebo and the alternative hypotheses is that the drug has
an effect compared to placebo. In reality there are two possibilities. That the
drug actually has no effect and that it has an effect. The result of our
analyses could also be either, that the drug has an effect or it does not have
an effect. There is no problem if the drug really has no effect and we also say
that that there is no effect. Also it is very much desired if our analyses conclude
that the drug has an effect if it really does. However there are also two undesired
possibilities. First, that we say that the drug has an effect if it actually
does not have an effect. This is exactly the case we can describe with a
p-value. So if we do not wish to label something as a drug if it is not a drug
we should lower the p-value we assign as statistically significant and gain more confidence in this way. However, it
is also possible that we state that the drug has no effect even though it
actually does. This means, that we have lost a possible drug candidate due to
our rigidness. As stated before if one error reduces the other increases at the
same number of studies. So in one case we assign something as a possible drug
that is not and in the second case we lose a drug that could be of use.
As we can not reduce both, the question now is, which is a worst
possibility? It is obvious that there is no easy answer and in my personal
opinion it highly depends on the circumstances. For example if you are in the
beginning of the process trying to find a suitable drug, you most probably
start with some computational chemistry approach to investigate which compounds
at all could be reasonable. It is already well known that down in the next
steps you will lose most of the possible candidates due to various problems. It
is very likely that you do not want to lose a best possible drug already in the
first step. So in this case you do not want to make a false negative decision.
As we can not adjust the probability of false negative result itself we need to
adjust false positive. So to lose as little as possible useful drug candidates
but still not to test every single possible compound in the next stages we should adjust the
p-value we call statistically significant reasonably high, for example to 0.10.
Now, let’s look a situation if one of our drugs that we selected in this
first step has made it through a punch of other trials and we are about to
undertake some of the last trials. Last trials usually need human experiments,
take extremely long time and are followed by enormous costs. This means that we
do not want to put all this time, effort and money into testing some compound,
that is not very prominent. This means, that now we need to be very sure that
the drug really is about to work. Meaning that we want to reduce the false
positive risk and this time should observe as small as possible p-value to be
certain, that the yet-to-be-drug actually has an effect.
These are the main reasons why there can and should not be a fixed
p-value that we call statistically significant and it should be adjusted each time depending on the aim of the study.
Even more so, it is much better to report the original data and the probability
of false positive calculated from the data in addition to stating the author’s
conclusion. This would allow readers comparison with their data and independent
evaluation of the data.
