Step 3: appraise

As discussed in Step 2: acquire, we want to base our answers to the clinical question posed on the "highest level of evidence" available. Where syntheses are unavailable, identification of the study design of individual papers is important in determining the overall quality of the paper. Our advice to patients should always include information on the degree of reliability or otherwise of the evidence.

According to the Centre for Evidence-based Medicine (CEBM), the evidence from well-designed systematic reviews and randomised control trials are likely to have greater quality and more reliable evidence. This is because they are based on larger participant samples and have design strengths to reduce bias. Conversely, if the best available evidence is very low level/low quality (e.g., a case reports based on before and after treatment effects), our advice to patients should be even more conservative. Case reports are less reliable as they are observations of individual cases and are based on the characteristics of that case.

There are some quick steps that can help you determine the study design of the research paper you are reading:

1. Read at the "Method" section to find the name of the design used according to the authors. (This may also be written in the title and/or abstract).
2. Then use this link from the Centre for Evidence-Based Medicine to determine what that study design means, where it fits in the types of studies that could be used and the advantages and disadvantages of that study design.

Tip: If you cannot find the name of the study design reported, it may be because another synonym or term is used. A quick internet search might help you find an alternative term for the same design so you can use the CEBM resource.

Critical appraisal is the process of assessing and interpreting evidence by systematically considering its internal and external validity and its relevance to the question. Internal validity is the extent to which the research is reliable; flaws and confounding factors in the methodology or the interpretation of results would reduce internal validity (see the UNSW Medicine tutorial on critical appraisal). External validity is an indication of the generalisability of the findings. A study with high external validity would have findings that are likely to apply to populations other than those involved in the study.

To be able to critically appraise clinically relevant research, it is vital to understand validity. Many resources are available to help you understand the various factors contributing to internal and external validity. A good place to start is the "How to read a paper" series by Trish Greenhalgh, particularly "How to read a paper: Assessing the methodological quality of published papers". There are guides for specific study designs.

By following a process it is possible to evaluate evidence quite quickly. Many critical appraisal checklists or tools have been developed as guides to appraisal of research evidence. The example critical appraisal tool provided here is modified from the University of New South Wales Medicine.

Tools of this kind are essentially checklists that prompt us to look for various potentially confounding factors in a research study. Critical appraisal tools are generally designed for the critique of intervention studies, in which a treatment or other form of intervention is trialed, but the tools and the overall critical appraisal process are quite flexible and can be adapted to different types of studies (e.g. randomised controlled or case-control trials; see Study Design to understand the differences) and even for evaluation of reviews and meta-analyses. A set of excellent critical appraisal tools applicable to each type of study design can be found at the Centre for Evidence-based Medicine (CEBM) website.

You won't be surprised to know that critical appraisal of research is a set of skills, and tools such as critical appraisal tools do not give you those skills, they help you to use the skills efficiently and effectively. The skills are learned by practicing critical appraisal, by reading about it (see the links above and further reading below) and by taking continuing education and other learning opportunities that aim to develop appraisal skills. One such opportunity, available free of charge online, is the Cochrane Eyes and Vision Group's course on Translating Critical Appraisal of a Manuscript into Meaningful Peer Review, particularly Module 2: How to Critically Appraise the Literature.