An Introduction to Reviewing the Statistical Methods in a Study

Peer Reviewer Resources

Editor’s Note: The following post is part of a series of Peer Reviewer Resources written by some of Academic Medicine‘s top peer reviewers. Check back each Thursday for the next post in the series. Read more about our Peer Reviewer Resources

By: Colin P. West, MD, PhD, associate professor of medicine and biostatistics, Division of General Internal Medicine, Department of Medicine, Mayo Clinic

A complete discussion of the review of the statistical methods reported in a manuscript would span multiple chapters. However, in this blog post, I would like to highlight selected key elements I focus particular attention on when I examine the statistical methods and reporting in a paper.

First, a conceptual point that is critical: the statistical analysis plan should be considered a fundamental element of the study design, not an afterthought. Sir Ronald Fisher is quoted as saying: “To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of.” The study design should be appropriate for the hypothesis being tested, the analysis plan should be consistent with the research design, and the study protocol should fully specify the planned analyses. Analysis of biased data is simple, as statistical software cannot evaluate the validity of the data’s source. Therefore, the onus is on the investigative team to design the study carefully and apply statistical methods only when the data arise from an appropriate study design.

In examining a study’s analytic methods, I try to determine the primary outcome and how it is measured. It should be clear in the reporting of the paper. The measurement of the primary outcome should provide an accurate reflection of the underlying process it is intended to measure. Similarly, I examine the explanatory or associated variables and their measurement. It should be clear that all important and relevant variables have been measured and accounted for in the analyses, so that the reported associations are not simply a result of confounding by a key unmeasured factor.

The study sample and measurement plans are also important. Are there groups of participants, and are these groups independent? Is there more than one observation per participant, and if so has this been accounted for in the analytic approach? For example, a paired t-test may be appropriate when the outcome is measured in a pre-post fashion on the same participant, while a two-sample t-test would generally not be appropriate for this design. If a pre-post design like this is not well understood in advance, and participants do not have identifiers allowing linkage of their pre and post data, the correct analysis cannot be performed and Fisher’s warning may come to pass.

The nature of the outcome variable is also an important consideration. Dichotomous variables are analyzed using different methods from continuous variables. The analysis should be consistent with the type of data being collected. In addition, many statistical tests require that a variety of assumptions be satisfied for the tests to be valid, so authors should confirm that these assumptions have indeed been met by the study design and data.

Regarding reporting of statistical results, one guiding comment is to use the statistics to tell the study’s story—the statistics are not the story. The discussion of a study’s results should be consistent with the descriptive data and with any more complex statistical analyses. Reporting of a study’s results should add clarity, not mislead readers into a false sense of the importance or magnitude of the results. Figures should be clear and fairly represent the data, and statistical reporting should be precise and informative: I strongly prefer actual p values rather than ranges, and confidence intervals whenever possible.

As noted in the opening, a comprehensive treatment of the appropriate application of statistics in study reporting would be an endeavor beyond the scope of this posting. However, resources to guide authors do exist. Here are a few useful references:

Lang TA, Secic M. How to Report Statistics in Medicine, 2nd E. American College of Physicians.

McGahie WC, Crandall S. Review criteria for research manuscripts: Data analysis and statistics. Acad Med. 2001;76:936-938.

Regehr G. Review criteria for research manuscripts: Results: Reporting of statistical analyses. Acad Med. 2001:76: 938-939.

Regehr G. Review criteria for research manuscripts: Presentation of results. Acad Med. 2001:76: 940-942.