From Fragmented Evidence to Actionable Insight: How RTI Used AI-Enabled Workflows to Build a Cross-Sector Economic Mobility Framework

Traditional approaches couldn’t synthesize a complex, fragmented evidence base on economic mobility. With DistillerSR, RTI’s AI-enabled, human-in-the-loop workflow efficiently processed hundreds of references to produce a rigorous, policy-ready framework.

RTI International is an independent scientific research institute dedicated to improving the human condition. Founded in 1958 in Research Triangle Park, North Carolina, RTI combines scientific rigor, multidisciplinary expertise, and real-world implementation experience to help clients solve complex challenges. From discovery to deployment, RTI closes the distance between evidence and action, transforming research and innovation into practical solutions that deliver measurable impact. Learn more at RTI.org.

About the Researchers

Tatiana Elisa Bustos, PhD, is a Research Scientist and Project Director at RTI International, where she works on policy and systems innovation. Drawing on nearly four years at RTI, Tatiana focuses on research, evaluation, and technical assistance projects that center the voices of people from different backgrounds and health conditions—embedding their perspectives into clinical trial design, decision-making processes, and systems change. A W.K. Kellogg Foundation Community Leadership Network Fellow and published researcher, she brings expertise in economic mobility, participatory evaluation, and community-engaged research. On this project, Tatiana served as Project Director, overseeing the scoping review and economic mobility framework development awarded through RTI.

Government decision makers and foundation leaders need clear, timely evidence to decide which strategies to invest in, adapt, and scale. Difficulty arises when relevant information is spread across many fields, uses different terms, and measures success in different ways. This is certainly the case in the field of economic mobility.

Economic mobility refers to how a person’s economic well-being changes over a lifetime. It is shaped by multiple pathways that span education, employment, health, and infrastructure. The literary evidence on economic mobility is fragmented, leaving those trying to work in the space with several barriers to navigate:

• Cross-sector evidence: Literature and data is spread across sectors such as housing, education, workforce development, health, and human services
• Terminology mismatch: Programs use different terms and concepts to describe similar goals and strategies
• Outcome variation: Studies measure different outcomes, making it difficult to compare results

These barriers make it harder to identify what works in a particular context, compare strategies, and translate findings into action. To address this challenge, RTI’s research team needed an efficient and rigorous way to organize and synthesize a large, diverse evidence base, identify common concepts across sectors, compare strategies and outcomes, and transform the findings into practical guidance for decision-makers.

RTI applies responsible artificial intelligence to turn scientific expertise into faster, greater real-world impact. Guided by its independent, data-driven approach, the institute uses AI where it strengthens insight, accelerates outcomes, and upholds the highest standards of rigor, objectivity, and privacy. As part of this approach, RTI recently combined AI with scientific and subject- matter expertise to process and organize large volumes of literature, identify related concepts expressed through different terminology, and surface patterns across sectors and outcomes. Researchers then reviewed, interpreted, and validated the results to ensure that the final synthesis remained grounded in evidence and met RTI’s standards for rigor, objectivity, transparency, and privacy. This combination of AI-enabled efficiency and expert oversight made it possible to move more quickly from fragmented evidence to actionable insight.

The Solution: Expert-Led, AI-Enabled Evidence Synthesis with DistillerSR

To address these challenges, we conducted a scoping review on economic mobility and we implemented an expert-led, AI-enabled workflow using DistillerSR which provided a centralized platform to:

• Manage large volumes of literature
• Standardize screening and data extraction
• Maintain a transparent, auditable workflow
• Support AI-assisted prioritization and coding
• Streamline efficiencies for a rapid review process
• Automate processes for faster results

The platform’s AI capabilities, such as continuous reprioritization and automated screening support, enabled us to accelerate
labor-intensive steps while maintaining methodological rigor and reproducibility. Critically, AI was not used to replace expert
judgment. Instead, it was embedded within a human-in-the-loop process, ensuring that all decisions remained interpretable, validated, and defensible.

The Approach: Combining Structured Methods with Scalable Workflows

The project followed a multi-stage, systematic workflow:

1. Structured Search and Data Integration

We conducted a comprehensive search across academic and gray literature sources, importing all records into DistillerSR for centralized management and deduplication. This resulted in 397 records being included in the Title and Abstract review phase. A more comprehensive search strategy is published in Open Science Framework 2.

2. AI-Assisted Screening with Reviewer Oversight

Across 397 unique references, a total of 104 references (26.2%) were reviewed exclusively by two human reviewers, including 51 references that were dual-reviewed for calibration and consistency. An additional 83 references (20.9%) were reviewed by both the DistillerSR automated reviewer and a human reviewer, supporting targeted quality assurance, while 210 references (52.9%) were screened solely by the automated reviewer. We utilized continuous reprioritization via DistillerSR Reranking scores to continually surface the most promising papers to the top of the list for human review. Once all highly scored papers had been processed, the AI reviewer was used to complete the remaining screening task.

Applying the observed human screening rate from the project, the use of AI represents an estimated savings of approximately 7–9 hours of manual title/abstract screening effort.

A total of 85 references were excluded during the title/abstract review process, with 312 moving onto full text review. To ensure consistency and rigor, we implemented multiple quality assurance checks throughout the screening process. DistillerSR’s error- checking tools were used to identify references that may have been excluded incorrectly, enabling targeted reviewer verification. In parallel, the AI model was iteratively trained on subsets of reviewer- labeled records, allowing it to continuously refine its predictions and improve performance across diverse study types. Inter-rater reliability analyses demonstrated high agreement across reviewers and AI-assisted decisions; agreement between human reviewers was perfect on overlapping records, and alignment between reviewers and the AI remained strong. The weighted overall kappa statistic was 0.93, indicating strong agreement and supporting the reliability and reproducibility of the screening process.

3. Standardized Data Extraction and Coding

A two-stage full-text review and data extraction process was used to identify the studies to be included in the final scoping review. Initially, full-text screening assessed the following criteria:

• Based on the full-text review, does this literature present a case example of economic mobility or related lever(s)?
• Does the article include a sufficient description of the economic mobility strategy or lever?
• Is the asset considered a peer-reviewed article or technical report from a trustworthy institution?
• Is the article written in English AND published between 2019-2026 (past 7 years)?
• Is the article focused on U.S contexts? Note: Assets may include U.S. and other contexts.

If the answer to any of these questions was ‘No’ the reference was excluded. This resulted in the inclusion of 251 references (64.7% of the original sample). Once the reference passed the final full text screening phase it continued to full data-extraction.

We leveraged DistillerSR’s Smart Evidence Extraction (SEE) capabilities to streamline and standardize the collection of study information. SEE accurately finds, suggests, explains, extracts, and links the supporting evidence within the reference in a human-in- the-loop and fully automated workflow.

SEE enabled automated extraction of structured data elements directly from full-text articles, populating predefined extraction forms while maintaining traceability to source text. In some cases, fully automated extraction forms were configured to complete fields without manual entry, further reducing reviewer burden for routine data elements. Reviewer oversight remained central to the process, with team members validating extracted content and refining extraction logic as needed. This combination of automated extraction and human verification allowed us to efficiently process large volumes of full-text articles while ensuring accuracy, consistency, and transparency in the resulting dataset. We developed a structured extraction framework capturing:

• Study characteristics
• Economic mobility pathways and levers
• Contextual factors and populations
• Outcome measures

As an additional quality assurance step, we conducted targeted validation of AI-assisted screening and extraction during full-text review. A random sample of 26 records (approximately 10% of the final set of 251 extracted sources) was drawn from references with below-median DAISY rank scores created in the initial title/abstract review stage, focusing on cases where AI confidence was lower. This sample represented approximately 20% of all lower-ranked records. Each sampled record was manually reviewed to verify both inclusion decisions and the accuracy of extracted data fields. Only one record was identified as a false inclusion, indicating strong AI performance even among lower-confidence predictions and high reliability in the automated extraction process. Using observed human extraction times from the project (approximately 15 minutes per full-text record), manually extracting data for the 281 records processed by DistillerSR AI would have required an estimated 70+ hours of reviewer effort. DistillerSR AI completed these submissions in approximately one hour of recorded review time, representing a substantial reduction in manual extraction workload.

4. Synthesis and Framework Development

With screening and coding streamlined, we shifted effort toward higher-value activities:

• Thematic synthesis
• Cross-sector comparison
• Development of a logic model linking levers – mechanisms – outcomes

This culminated in a theory-driven economic mobility framework designed for policy and program application.

The Results: Efficiency Gains Enabled Deeper Insight

The use of AI-enabled workflows within DistillerSR allowed us to efficiently process a high-volume, cross-sector evidence base while maintaining strong methodological rigor. From an initial set of 397 records, 312 advanced to full-text screening and 251 high-quality sources were ultimately included for synthesis, demonstrating an efficient and focused screening process.

The hybrid screening approach balanced efficiency with oversight. More than half of all references (52.9%) were screened solely by the automated reviewer, while 26.2% were reviewed exclusively by human reviewers to support calibration and consistency. An additional 20.9% of records were reviewed by both the automated reviewer and a human reviewer, enabling targeted quality assurance without requiring full duplication of effort across the entire dataset.

Quality assurance metrics further reinforced the reliability of the workflow. Inter-rater agreement across reviewers and AI-assisted decisions was high, with a weighted kappa of 0.93, indicating strong consistency in screening decisions. Targeted validation of lower-confidence cases showed similarly strong performance: in a structured sample of records drawn from below-median AI confidence scores, only one false inclusion was identified. This validation also confirmed the accuracy of automated data extraction, with reviewer checks demonstrating high alignment between extracted fields and source documents.

Together, these results indicate that the AI-assisted workflow not only accelerated screening and extraction, but did so while preserving transparency, reproducibility, and data quality. By reducing the time spent on routine review tasks, we were able to dedicate more effort to synthesis, cross-sector comparison, and framework development.

The Outcome: A Reusable Evidence Framework for Economic Mobility

The project resulted in a rigorously developed, cross-sector evidence base and a theory-driven economic mobility framework designed for policy and program application. Using AI-enabled workflows within DistillerSR, we efficiently processed 397 records through screening and full-text review. The structured workflow supported both efficiency and analytical depth. By leveraging AI-assisted screening and automated extraction, we reduced time spent on repetitive tasks and redirected effort toward higher- value activities, including cross-sector comparison and framework development.

At the same time, the workflow maintained a high standard of methodological rigor. Screening consistency was supported
by strong inter-rater agreement (K = 0.93), and targeted quality assurance confirmed the accuracy of both inclusion decisions and extracted data. In a focused validation sample of lower-confidence records, only one false inclusion was identified, demonstrating reliable AI performance even in more uncertain cases. Together, these results demonstrate that AI-enabled workflows can not only accelerate evidence synthesis, but also support the development of high-confidence, decision-ready evidence products.

The Takeaway: Scaling Complexity Without Sacrificing Rigor

Managing complex, cross-sector reviews requires more than just high-volume processing; it demands sustained consistency, transparency, and analytical coherence. This project highlights that AI-enabled workflows achieve the best results when integrated with structured methodology and expert oversight. By utilizing DistillerSR, we successfully scaled our screening and data extraction while ensuring an auditable, quality-assured process. Our hybrid approach—merging human expertise with AI-driven prioritization and extraction—allowed us to:

• Efficiently process a high-volume, heterogeneous evidence base
• Maintain strong reviewer alignment and reproducibility
• Validate both screening decisions and extracted data through targeted QA
• Focus researcher effort on synthesis and interpretation rather than manual processing

Ultimately, AI delivers the greatest value by amplifying, not replacing, expert judgment. For organizations navigating complex policy landscapes, this model offers a proven path to faster, scalable, and actionable evidence that maintains both rigor and trust.