- Ten Ways to Cheat on Statistical Tests When Writing Up Results (continued)
- The authors used the r-value to claim causation when it only suggested association.
- The authors used outliers to their advantage: assumed that the ones which helped them were real and the ones which hurt them were mistakes.
- The authors left out the confidence intervals if they overlapped "0".
- The authors found that the difference between 2 groups became significant 4 months into a 6-month trial. They stopped the trial arbitrarily at that point. On the other hand, if they couldn’t make statistical significance at 6 months, they extended the trial a few more weeks.
- If their results proved uninteresting, they went back and looked for more subgroups. Chinese females between 15 and 37 might prove interesting.
- If their first tests didn't work, they found some weird test no one knew anything about and couldn't challenge.
Other important questions
- Did the authors state what their original protocol was?
- Did they define subgroups in advance and calculate how large their study needed to be to account for all the subgroups?
- Or, did they sift through the results afterwards to see if they could find something interesting?
- Did they stop the study when they said they were going to or did they stop it when things became interesting and "significant"?
