Practical 10 Critical thinking about methods and analyses

Often it is tempting accept the results of papers at face value. They were published so they must be correct, right? Sadly no. Even the best papers have flaws. There may be problems with the data, method or interpretation of the results. Some of these are unavoidable, some reflect misunderstandings of the methods used, and others are just mistakes. Learning to critically read the scientific literature (or indeed any literature in this age of fake news!) is therefore a key skill to develop.

As practice we will split into groups and critically evaluate recent papers using some of the methods we’ve worked with in this module. I’ll assign papers to everyone on the first day. Read the paper before class, and make notes of things you don’t understand or disagree with. I have provided some guidance of things to look for below. The question you should keep asking yourself throughout is given the data and methods, do I trust the conclusions of the paper?

10.1 How to critically evaluate a paper

10.1.1 Logic/interpretation

• What questions does the paper address?

• Do the analyses/data actually answer the questions the paper is meant to be asking, or do they answer a different question?

• What are the conclusions? Do the analyses/data support the conclusions?

• Have the authors exaggerated the importance of their conclusions (e.g. evidence of one species shifting range in the ice age and the conclusions are that climate change is going to be fine?)

• Is the logic of the paper clear and justifiable, given the assumptions?

• Are there any flaws in the authors reasoning?

• Do you agree with how the authors interpret their results?

10.1.2 Data

• What’s the sample size? Is it large enough to support the conclusions of the paper?

• How many species are missing from the analysis? Does this worry you?
  • Is two species missing from a clade of 50 species a problem?
  • Can 50 species be used to make conclusions about a clade of 1000s of species?
• Are species missing in a way which might influence the results?
  • Would you be concerned if all species from one clade were missing?
  • Are the species present well distributed across the phylogeny?

• Are fossil/extinct species considered? Would this influence the results/conclusions?

• How were the data collected? Could this bias the results at all?

• Are there biases in the age, sex, geographic locality etc. of species included?

• Do you think the data quality is high enough?

• Would other data have been better to answer this question?

10.1.3 Methods

• Check the text carefully for caveats. These may appear in the introduction, methods, results or discussion. Did the authors deal with them or just mention them?

• What are the assumptions/limitations of the method being used? These may be mentioned in the text, or you may need to dig into the literature to find them (don’t worry about this for the class though do check the handouts from practicals for some pointers).

• Are the assumptions the authors make reasonable? For example, a big assumption underlying all phylogenetic methods is that the phylogeny is correct. Do you agree?

• Be aware that some older methods may have been superseded by better methods.

• Be aware that sometimes there is debate in a community about the best method to use (e.g. the BAMM debate).

10.1.4 Moving forwards

• What are the good things in this paper? Make sure that you don’t ignore the positive in your hunt for the negative!

• Do these ideas have other applications or extensions that the authors might not have thought of?

• How would you fix the flaws in this paper?

• How might the paper be useful to you? For example, as a paper to cite in your thesis, a method to use, or a cautionary tale of what not to do?!

10.2 References

These papers involve critiques/reviews of some of the methods we’ve been learning about in this module. They may be helpful for some of the papers.

• Cooper et al. 2016a. Shedding light on the “Dark Side” of phylogenetic comparative methods. Methods Ecology and Evolution.
• Freckleton 2009. The seven deadly sins of comparative analysis. Journal of Evolutionary Biology.
• Losos 2011. Seeing the forest for the trees: the limitations of phylogenies in comparative biology. American Naturalist.
• Cooper et al. 2016. A cautionary note on the use of Ornstein Uhlenbeck models in macroevolutionary studies. Biological Journal of the Linnaean Society.
• Kamilar and Cooper 2013. Phylogenetic signal in primate behaviour, ecology, and life history. Phil Trans Roy Soc B.
• Moore et al. 2016. Critically evaluating the theory and performance of Bayesian analysis of macroevolutionary mixtures. PNAS.