Reviewing Literature Reviews for Quality

Literature reviews offer a convenient way to understand a field of research quickly. Packed full of peer-reviewed information, they provide an entry point to broad topics, saving us time and effort sifting through countless isolated papers. However, we need a systematic method for reviewing literature reviews, ensuring we only use those that avoid bias and contain high-quality references.

Literature reviews are up-to-date summaries of a given topic. Even though they involve no original research, they can be susceptible to bias and poor quality. There are four practical methods for reviewing a literature review paper (applicable to all academic fields):

1. Look for external motivations behind the review
2. Sample cited research papers for citations and relevance
3. High-quality reviews also address weaknesses
4. Check for the quality of the journal it is published in

Pinpoint External (Often Financial) Motivations

Unlike standalone studies, literature reviews don’t involve original research; it is up to the author to collate and publish existing information in the field. While government grants often fund research, external parties often fund literature reviews.

In general, literature reviews that receive funding from charities, unions and advocacy groups should warn readers that the writing is likely to be biased toward their cause. Government and independent non-profit groups tend to commission more objective and neutral studies. By law, authors must disclose their source of funding in the review.

Ultimately, review papers will frequently lean toward a financially or socially-backed outcome. Still, keeping these motivations in mind as we move into the next step of checking for bias is essential.

Inconsistencies in the Reference List

Most literature reviews will cite the most authoritative papers in the field; you can check the number of citations in each article from Google Scholar. Sometimes, authors cite their own studies or studies published by people they have obvious academic connections with. It’s not necessarily bad practice, but it can lower the overall quality of the review.

Since it would be too time-consuming to review each study for quality, you can take a sample of papers from the reference list and look for consistency and relevance between them. Papers with a low citation count or with little relevance to the review topic can signify:

1. The author is referencing it because it fits their narrative/outcome
2. There are arrangements in place that are not purely academic
3. The literature review has not been thoroughly researched

One way to ensure the literature review isn’t cherry-picking data is to identify if any meta-analysis was done. Meta-analyses are typical in literature reviews, where the author combines the data from multiple studies and represents it as a more robust result. If the data set omits specific sources to favor one outcome over others, it is a biased and low-quality review.

Researcher Bias in the Discussion

Next, we can look for biases simply by reading the content! Most authors who write literature reviews are also from the field and will want to promote the potential of research within it, but robust reviews will also talk about the field’s current limitations. Some will also include the strengths and weaknesses of specific studies and datasets.

Review papers that don’t discuss the pros and cons of a topic should be considered low-quality because they don’t paint the complete picture of the field’s current state. Always be wary of literature reviews that avoid addressing the weaknesses of the cited research.

Beware of Predatory Publishing

The peer-review process is integral to academic publishing, ensuring the data is reliable and trustworthy. However, some predatory journals publish papers—including literature reviews—only for financial gain.

These journals usually provide incomplete contact information with an obscure editorial board or use fake profiles (no links to ResearchGate, LinkedIn or Google Scholar). At the time of writing, this website maintains a list of potentially predatory and poor-quality journals and publishers.

With these tools for reviewing literature reviews, you can avoid low-quality ones like the plague while benefiting from those that are trustworthy and informative!