Despite its ubiquity, RAG-enhanced AI still poses accuracy and safety risks

Retrieval-Augmented Generation (RAG) — a method used by genAI tools like Open AI’s ChatGP) to provide more accurate and informed answers — is becoming a cornerstone for generative AI (genAI) tools, “providing implementation flexibility, enhanced explainability and composability with LLMs,” according to a recent study by Gartner Research.

And by 2028, 80% of genAI business apps will be developed on existing data management platforms, with RAG a key part of future deployments.

There’s only one problem: RAG isn’t always effective. In fact, RAG, which assists genAI technologies by looking up information instead of relying only on memory, could actually be making genAI models less safe and reliable, according to recent research.

Alan Nichol, CTO at conversational AI vendor Rasa, called RAG “just a buzzword” that just means “adding a loop around large language models” and data retrieval. The hype is overblown, he said, adding that the use of “while” or “if” statements by RAG is treated like a breakthrough.

(RAG systems typically include logic that might resemble “if” or “while” conditions, such as “if” a query requires external knowledge, retrieve documents from a knowledge base, and “while” an answer might be inaccurate re-query the database or refine the result.) 

“…Top web [RAG] agents still only succeed 25% of the time, which is unacceptable in real software,” Nichol said in an earlier interview with Computerworld. “Instead, developers should focus on writing clear business logic and use LLMs to structure user input and polish search results. It’s not going to solve your problem, but it is going to feel like it is.”

Two studies, one by Bloomberg and another by The Association for Computational Linguistics (ACL) found that using RAG with large language models (LLMs) can reduce their safety, even when both the LLMs and the documents it accesses are sound. The study highlighted the need for safety research and red-teaming designed for RAG settings.

Both studies found that “unsafe” outputs such as misinformation or privacy risks increased under RAG, prompting a closer look at whether retrieved documents were to blame. The key takeaway: RAG needs strong guardrails and researchers who are actively trying to find flaws, vulnerabilities, or weaknesses in a system — often by thinking like an adversary.

How RAG works — and causes security risks

One way to think about RAG and how it works is to compare a typical genAI model to a student answering questions just from memory. The student might sometimes answer the questions from memory — but the information could also be outdated or incomplete.

A RAG system is like a student who says, “Wait, let me check my textbook or notes first,” then gives you an answer based on what they found, plus their own understanding.

Iris Zarecki, CEO of data integration services provider K2view, said most organizations now using RAG augment their genAI models with internal unstructured data such as manuals, knowledge bases, and websites. But enterprises also need to include fragmented structured data, such as customer information, to fully unlock RAG’s potential.

“For example, when customer data like customer statements, payments, and past email and call interactions with the company are retrieved by the RAG framework and fed to the LLM, it can generate a much more personalized and accurate response,” Zarecki said.

Because RAG can increase security risks involving unverified info and prompt injection, Zarecki said, enterprises should vet sources, sanitize documents, enforce retrieval limits, and validate outputs.

RAG can also create a gateway through firewalls, allowing for data leakage, according to Ram Palaniappan, CTO at TEKsystems Global Services, a tech consulting firm. “This opens a huge number of challenges in enabling secure access and ensuring the data doesn’t end up in the public domain,” Palaniappan said. “RAG poses data leakage challenges, model manipulation and poisoning challenges, securing vector DB, etc. Hence, security and data governance become very critical with RAG architecture.”

(Vector databases are commonly used in applications involving RAG, semantic search, AI agents, and recommendation systems.)

Palaniappan expects the RAG space to rapidly evolve, with improvements in security and governance through tools like the Model Context Protocol and Agent-to-Agent Protocol (A2A). “As with any emerging tech, we’ll see ongoing changes in usage, regulation, and standards,” he said. “Key areas advancing include real-time AI monitoring, threat detection, and evolving approaches to ethics and bias.”

Large Reasoning Models are also highly flawed

Apple recently published a research paper evaluating Large Reasoning Models (LRMs) such as Gemini flash thinking, Claude 3.7 Sonnet thinking and OpenAI’s o3-mini using logical puzzles of varying difficulty. Like RAG, LRMs are designed to provide better responses by incorporating a level of step-by-step reasoning in its task.

Apple’s “Illusion of Thinking” study found that as the complexity of tasks increased, both standard LLMs and LRMs saw a significant decline in accuracy — eventually reaching near-zero performance. Notably, LRMs often reduced their reasoning efforts as tasks got more difficult, indicating a tendency to “quit” rather than persist through challenges.

Even when given explicit algorithms, LRMs didn’t improve, indicating they rely on pattern recognition rather than true understanding, challenging assumptions about AI’s path to “true intelligence.”

While LRMs perform well on benchmarks, their actual reasoning abilities and limitations are not well understood. Study results show LRMs break down on complex tasks, sometimes performing worse than standard models. Their reasoning effort increases with complexity only up to a point, then unexpectedly drops.

LRMs also struggle with consistent logical reasoning and exact computation, raising questions about their true reasoning capabilities, the study found. “The fundamental benefits and limitations of LRMs remain insufficiently understood,” Apple said. “Critical questions still persist: Are these models capable of generalizable reasoning or are they leveraging different forms of pattern matching.”

Reverse RAG can improve accuracy

A newer approach, Reverse RAG (RRAG), aims to improve accuracy by adding verification and better document handling, Gartner Senior Director Analyst Prasad Pore said. Unlike typical RAG, which uses a workflow that retrieves data and then generates a response, Reverse RAG flips it to generate an answer, retrieve data to verify that answer and then regenerate that answer to be passed along to the user.

First, the model drafts potential facts or queries, then fetches supporting documents and rigorously checks each claim against those sources. Reverse RAG emphasizes fact-level verification and traceability, making outputs more reliable and auditable.

RRAG represents a significant evolution in how LLMs access, verify and generate information, Pore said. “Although traditional RAG has transformed AI reliability by connecting models to external knowledge sources and making completions contextual, RRAG offers novel approaches of verification and document handling that address challenges in genAI applications related to fact checking and truthfulness of completions.”

The bottom line is that RAG and LRM alone aren’t silver bullets, according to Zarecki. Additional methods to improve genAI output quality must include:

Structured grounding: Fragmented structured data, such as customer info, in RAG.

Fine-tuned guardrails: Zero-shot or few-shot prompts with constraints, using control tokens or instruction tuning.

Human-in-the-loop oversight: Especially important for high-risk domains such as healthcare, finance, or legal.

Multi-stage reasoning: Breaking tasks into retrieval → reasoning → generation improves factuality and reduces errors, especially when combined with tool use or function calling.

Organizations must also organize enterprise data for GenAI and RAG by ensuring privacy, real-time access, quality, scalability, and instant availability to meet chatbot latency needs.

“This means that data must address requirements like data guardrails for privacy and security, real-time integration and retrieval, data quality, and scalability at controlled costs,” Zarecki said. “Another critical requirement is the freshness of the data, and the ability of the data to be available to the LLM in split seconds, because of the conversational latency required for a chatbot.”Despite its ubiquity, RAG-enhanced AI still poses accuracy and safety risks – ComputerworldRead More