What is a Systematic
Review? Ultimate Guide to Systematic Reviews

Why are systematic reviews valuable?

Simply speaking, better research benefits the world. As a top method for gathering and reporting on data, systematic reviews are essential to the global population in terms of quality scientific research and innovation. Systematic reviews aim to provide credible, reproducible studies, and reduce bias.

Policymakers and guideline developers must have the full picture on any given topic before making recommendations or informing new policies and guidelines. They rely on systematic reviews as one of the main tools to inform their decisions which impact the general population, such as when setting regulatory standards.

Systematic Review or Literature Review?

In certain industries or academic streams, “literature review” is synonymous with “systematic review” (for example, in preparation of clinical evaluation reports), while in others, the defining aspect of a systematic review is that it is systematic and follows a set of specific guidelines, whereas a general literature review may not adhere to the same kinds of standards and expectations.

However, the name and definition depends entirely on specific terminology and context. Depending on the geographic location, industry, or even the organization, you could call it a “systematic review,” “literature review,” or even “systematic literature review.” Read about the types of Systematic Reviews.

Systematic Review Best Practices

Over the years, systematic literature review methodology has changed, improved, and innovated with technological advances, and the need for fast, detailed, and evidence-based research has reached new levels. 

As early as 20 years ago, systematic reviews were done with pen and paper. Researchers and staff were physically printing off references, screening, highlighting, and keeping references in separate stacks or filing cabinets. As you can imagine, this manual process was extraordinarily time-consuming and resource-intensive.

Fifteen years ago, systematic reviews moved into the digital sphere and were completed using spreadsheets. Researchers would copy and paste data into cells. This saved a lot of paper, but as the demand for evidence grew, the spreadsheet method began to severely lacked the efficiency and accuracy needed by modern researchers. With the help of technology, systematic review tools have become more advanced and new methods have been implemented.

Today, systematic review software is the status quo for completing fast and accurate systematic reviews–more on that later. 

Regardless of the purpose of your review, there are several universally accepted systematic review best practices. These best practices improve efficiency, transparency, and reproducibility in systematic reviews, which are goals that all reviews strive to achieve. 

• Dual independent screening is the gold standard best practice for systematic reviews. Humans make errors, so it’s critical to have methods in place that will catch the errors before they become catastrophic to your research. 
• Establish an experienced team to carry out the systematic review tasks. If there are certain elements of the review that your organization is not experienced with, it would be advisable to hire an information specialist or other subject matter experts to help including people with experience in systematic literature reviews. 
• Search all relevant databases and alternative information sources for grey literature. An incomplete search is one of the foremost reasons that a systematic literature review could be invalidated.
• Determine inclusion and exclusion criteria and develop a conflict resolution protocol beforehand. Stay away from ad hoc processes.
• Capture the reason for exclusion. Researchers can easily reference the article exclusion list during the peer review or audit stage. The list of excluded articles could also help expedite the search and screening process in different but similar reviews that look at some of the same references. 
• Systematic reviews hinge on the idea that evidence should be transparent. A brief look at the PRISMA checklist shows that transparency in every step of the review from developing the search strategy to reporting the results ought to be detailed and as transparent as possible.

Literature Review for EU MDR

EU MDR is a new medical device regulation directive in Europe. It impacts all new and legacy (existing) medical devices sold in the European Union. The new directive was released in 2017, and since then, medical device manufacturers have been working towards ensuring their products are MDR-compliant. 

Under EU MDR, medical devices are reviewed by notified bodies who are designated by the European Commission. These notified bodies have a duty to perform the conformity assessments and audits to confirm if medical devices meet the current standards. They are for-profit organizations hired by medical device manufacturers to perform “unannounced audits,” so it’s critical for manufacturers always to be prepared. 

So what constitutes compliance? For medical device manufacturers, the literature review is just one (albeit important) part of their conformity assessment. Notified bodies look for a few things when assessing a CER literature review. Some of the biggest challenges medical device manufacturers face with their literature reviews include:

• Incomplete audit trail: NB’s want to see transparency in the literature review. They want to know the path researchers take to reach their conclusions. An audit trail is essential for showing work. 
• Ad hoc processes: A proper literature review must follow a systematic and reproducible process. 
• Incomplete search coverage: Missing studies is a critical error in a CER literature review, and missing relevant information could invalidate your work and lead to time to market delays. 
• Data integrity: CER literature reviews are used to inform important healthcare decisions, so accuracy is essential. 
• Efficiency: When dealing with massive amounts of data to make informed decisions for evidence-based healthcare, it’s important to work as efficiently as possible.

Although these are common challenges for organizations seeking EU MDR-compliance, there are several ways to mitigate the challenges and make it easier to comply with the new medical device standards, so your device doesn’t fail its audit. Medical device manufacturers must establish “State of the Art” in their CER literature review, which is essentially a section of the report that >“describes what is currently and generally considered standard of care, or best practice, for the medical condition or treatment for which the device is used.” However, the State of the Art section impacts the entirety of the review, and if you fail to establish it, your review could be considered incomplete.

How systematic review software can save costs and time

If you consider the “old way” of doing a manual systematic review, with pen and paper or with clunky spreadsheet programs, it makes sense to use specialized systematic review software instead. After all, the research is of critical importance; doesn’t it make sense to do everything you can to make sure it’s done correctly? 

Systematic review software (SRS) benefits research teams in many different ways, but the two most important things to know are that it saves time and money. 

With SRS, researchers can work more efficiently, reduce errors, and, therefore, spend less time fixing mistakes when completing reviews. This efficiency results in lower costs and faster access to better evidence-based research. 

Other significant benefits of Systematic Review Software include:

• Real-time collaboration: Specialized systematic review software like DistillerSR enables users to work on their review from any browser, which means you can work on projects regardless of your location. 
• Audit-trail: The software also has a clear audit trail with timestamping and user stamping, which promotes transparency, ensures the team follows proper protocol and helps pinpoint errors and changes to the project. 
• Simple reporting: Access to reports including Kappa, PRISMA, User Metrics, and Reference Progress with the simple click of a button. 
• Automation: Other features such as deduplication and reference prioritization are automations that save even more time when completing the review.
• Simplified data extraction: Custom data extraction forms enable users to manage complex datasets including capturing related, repeating blocks of information (also known as hierarchical data.)

So how does it work?

AI in systematic reviews is powered by a subfield of AI known as “Natural Language Processing” (NLP). This technology uses algorithms to break down sentences and words to their absolute base. By doing this and assigning mathematical properties to sentences, NLP can take large amounts of unformatted text (e.g., articles, blog posts, books, etc.) and turn it into data that can be understood by a computer. 

Natural language processing is the subfield of AI that powers many tools we use every day, such as spam filters, spelling and grammar checkers, and search engines. 

Natural language processing is the AI that powers the real heavy hitter when it comes to automating systematic review tasks. Classifiers are a statistical model that uses NLP to classify information and process it accordingly. 

In a systematic review, a classifier can answer a specific question about the text you are reviewing. “Is this an RCT?” “Is the study about humans?” are just a couple of questions that can be answered by natural language processing.

As you can imagine, this type of technology holds limitless possibilities for systematic reviewers who spend most of their time deciding whether or not to include or exclude references based on the presence or absence of specific data. 

However, before getting too excited, it’s important to remember to take a pragmatic approach to AI in systematic reviews. While the technology is fascinating and holds many opportunities to save time and automate many time-consuming SR tasks, we need to remember that the AI of today is not perfect. 

The current gold standard for accuracy in a systematic review is using dual independent screening, so we should expect AI to at least meet the accuracy rating of humans instead of expecting it to be perfect. 

It’s important to use AI in a way that it doesn’t need to perfect — use safeguards that would prevent any catastrophic problems if the AI makes a mistake, and make sure to double-check and test the AI frequently to make sure it’s working the way it should. Consider it “training” your robot to do the tasks you need it to. 

In research communities where accuracy is the difference between vital resources getting allocated to specific populations, and potential harm done to humans, we must take a pragmatic, cautious approach to automating tasks. By taking smaller steps towards integrating AI into systematic review workflows, we can build confidence in the tools, so they are used in a way that both saves time and reduces errors.