You have spent months coming up with your experiment, gathering information and cleaning your spreadsheets. After running your analysis, heart throbbing, you see it: p = 0.12.
The non-significant outcome that nobody wanted can be a dead end. In academia, where competition is the order of things, there is a tendency to consider p < 0.05 to be the golden ticket to publication. A non-significant p-value however may not imply that your hypothesis is false. Maybe it was just that your study design or data handling could not identify the effect with the accuracy that it required.
These five critical checks are to be run through before you archive your project.
Reconsider Your Sample Size (Statistical Power)
The most widespread explanation of the non-significant result is the low power of the study. With a sample size that is very small, your test might be too weak to pick up a true effect, in the event that it actually exists.
Consider the statistical power, as the focus of a microscope. When the resolution is too low, you can not see the bacteria, however, it does not imply that they are not there.
What to do: Find a post-hoc power analysis. When your power is less than 0.80, then you probably should have additional subjects or observations.
Pro Tip: Mars Publications offers Scientific and Technical Editing that includes a critique of your methodology section so that peer reviewers can easily understand and check your power calculations.
Data Entry Checks and Outliers
Noise in statistics is simply astonishing. One mistake in a spreadsheet, typing in 100 in place of 10, can swell your variance, and hide your p-value. Extreme outliers can draw the mean off the real middle of your data, which raises the standard error.
What to do: Plot your data in box plots or in scatter plots. Detect more than three standard deviations of the points around the mean.
The Fix: Find out whether or not the outlier is a measurement error or a biological (but rare) variation. In case of an error, delete it and explain in your manuscript why it is an error.
Reconsider Your Statistical Test
Not everything is data of the same kind. Most researchers fail at using t-test or ANOVA without ensuring their data satisfies the assumptions, including that it is normal or has the same variance.
When your data are skewed, then a parametric test may lose its efficiency. Under such instances, the p-value will be high since the test is confused with the data.
Common Mismatches:
Type of data Misplaced Test
Non-normal distribution , Independent t-test , Mann-Whitney U Test , Categorical results, Linear Regression, Logistic Regression, Repeated measures ANOVA, Simple ANOVA, Mixed-Effects Model.
Examine the Size of the Effect, and Not the P-Value
The p-value will inform you whether an effect is probable owing to probability, however, the resultant size (such as Cohen’s d or Pearson’s r) will inform you how huge that effect is.
When you get a large effect size but non-significant p-value, then you have virtually a problem with sample size. On the other hand, when your effect size is close to zero, there is a possibility that your hypothesis is indeed not supported.
Why it is important: Journals are abandoning “p-hacking” and adopting instead so-called estimation statistics. It is common to find it more interesting, scientifically, to report a large effect size with a confidence interval that barely crosses zero than it is to report a tiny effect that is only significant because the sample size was huge.
Control Confounding Variables
You may also consider the possibility that a relationship exists but is confounded by some other variable that you have not captured in your model. For illustration, a new medication may perform well, but your study does not observe the effect of this type of medication due to your failure to consider the age and pre-existing conditions of the participants.
What to do: multivariate analysis or ANCOVA (Analysis of Covariance) should be used to control these variables. With the background noise removed, you can discover that the major relationship comes into focus and is statistically significant.
Summary: It is better not to give up when the results are negative
Even when the p-value is not significant, it is a result. It narrates the limits of your hypothesis. You might have discovered a true null result, in case you have checked your power, cleaned your data, verified your tests and have taken care of confounders. These play an essential role in avoiding publication bias within scientific society.
We are Mars Publications, and our business is to get these findings into perspective. Our Scientific and Technical Editing services assist you in putting across whatever finer points you may have on your data, ensuring that even the negative results receive the statistical credit and clarity they require.