Feature
Countless times throughout the day, employees look for a piece of information to help them do their job — from how to perform a task in a workplace app to locating the latest version of a report. The type and number of queries are vast.
But every minute your employees spend looking for an answer is time they aren't focusing on their core tasks. Microsoft's 2023 Work Trend Index found that 62% of employees surveyed struggle with spending too much time searching for information during their workday.
For decades, businesses have turned to contextual knowledge retrieval, which refers to finding information stored digitally, to find data throughout the workday. For example, a salesperson can use a CRM to quickly locate customer information stored in a single location. HR management systems help HR professionals find the latest performance review about an employee.
In recent years, the speed and accuracy of contextual knowledge retrieval has improved through advancements in natural language processing (NLP) and artificial intelligence (AI). However, the recent evolution of retrieval augmented generation (RAG) has made the process significantly more effective.
What Is RAG?
The RAG process starts when an employee requests a specific piece of information, such as through a chatbot or search engine. The technology retrieves the data based on the request and generates a response that puts the information discovered into a format and context that answers the query’s intent.
While some tools focus on retrieving and others focus on generating, RAG represents a significant improvement in contextual knowledge retrieval because it combines both functions into a single response.
“Businesses struggle with vast, unstructured data across documents, databases and APIs,” said Mithilesh Ramaswamy, senior engineer at Microsoft. “RAG helps surface the most relevant information instantly, digesting data in various formats [and] ensures that AI responses stay up to date.”
While static models rely on fixed training data, RAG dynamically retrieves real-time information, ensuring up-to-date responses, Ramaswamy said.
Benefits and Limitations of RAG
While purely generative models can generate incorrect or misleading information, Ramaswamy explained that RAG improves accuracy and trust through integrating retrieval-based grounding. Because many businesses cannot expose proprietary or sensitive data to external models, RAG ensures data remains internal and controlled while still enhancing AI interactions, he said.
For example, many companies are turning to RAG for internal customer support. Instead of creating a ticket or waiting on hold for assistance, employees can use AI-powered virtual assistants that can retrieve the latest FAQs, product manuals and troubleshooting steps. So, instead of paging through documents to find the exact piece of information they need, the employee has the exact answer to their question and can successfully complete their task more quickly.
Meanwhile, one of the challenges organizations face with RAG starts with not understanding its purpose, which results in inefficiencies and inaccuracies.
“We are now facing this moment where organizations are making available this tool to everyone, and it's a retrieval tool for large language models,” said Dr. Rosina Weber, professor of Information Science at Drexel University. “RAG tools are not problem-solving tools, but this is how people use them. Instead of making predictions and connections like AI tools, RAG tools simply regurgitate information that already exists.”
RAG vs. LLM
While AI technology becomes more integrated with processes, businesses often consider creating a new LLM for contextual information retrieved. However, Ramaswamy said that instead of the expensive retraining cycles that are required with a new LLM, businesses can plug in new information dynamically via RAG.
“While LLMs provide powerful, retrieval-based grounding is necessary to make them truly useful for individual customers with different needs. RAG achieves hyper-personalization at scale,” said Ramaswamy.
This helps small businesses with scalability without massive AI investments. “Companies don’t need billion-dollar GPUs to keep AI relevant; RAG allows small- to mid-sized enterprises to integrate AI cost effectively,” he said.
Emerging RAG Technology
Until recently, RAG primarily focused on retrieving text from databases using keywords. This has provided some benefits but also offered limited use cases.
“Once RAG began using vector databases, which are representatives of the tokens that the large language models use, efficiency improved,” explained Weber.
Users now have more opportunities to retrieve documents that are relevant to the prompts due to the embedding that offers context, making it superior to keywords. Recent technology also enables query expansions, making it possible for employees to make more sophisticated and detailed requests.
Query expansions are especially powerful when combined with domain-specific RAG because the tool accesses specialized databases and improves the quality of the retrieval, Weber explained. But data scientists take RAG to the next level by using a technique known as “attention” to improve the power of the generative aspect of RAG and provide the ability for supervised training on a passage of text where the model learns how important the tokens in the context are to the request based on the specific of the domain. According to Weber, attention provides a higher level of sophistication that improves the overall accuracy.
Many industries, especially biomedical, are now integrating RAG with knowledge graphs that connect the nodes with links. With this technology, data scientists addressed the challenges for natural language in the domain and converted the natural language into a knowledge graph.
“If you retrieve from the documents, you have to deal with the ambiguities of natural image. But if you’re going to retrieve directly from a knowledge graph, there’s no ambiguity, and you have the relationships between the concepts right there. It’s much more powerful,” said Weber.
What’s Next for RAG
When asked what the future holds for RAG, Ramaswamy said that with on-device and edge-based retrieval getting cheaper, companies will soon have the ability to deploy RAG on employees’ smartphones. Because RAG systems will move from cloud-heavy architectures to on-premises or edge deployments, the mobile version will have reduced latency and improved performance.
Webinar
Sep
25
CMS Briefing: A Live Look at What’s Next in AI-Driven Platforms
Learn how leading organizations are using AI‑driven tools to publish faster, personalize smarter and stay secure.
Register
Webinar
On demand
Ready or Not: How Data-First Organizations Are Unlocking Agentforce Potential
Learn how to cut through the noise, activate Agentforce and build a Salesforce AI strategy that actually delivers.
Watch Now
Webinar
On demand
AI in Customer Service: Faster Resolutions, Happier Customers
Don’t let rising demand burn out your team. See how to build a smarter, more resilient support org.
Watch Now
Webinar
On demand
From Hype to High-Impact CX Strategies That Actually Scale
Turn buzzworthy AI and outsourcing trends into measurable CX wins with fresh 2025 data.
Watch Now
Webinar
On demand
Insights to Action Rethinking the Contact Center for Real Business Impact
Join our exclusive webinar to hear CX executives share their innovative strategies for transforming service delivery.
Watch Now
Webinar
On demand
Accelerating Healthcare Ops with AI
Sign up to see how your peers are putting AI to work — and making it stick.
Watch Now
Through memory augmented retrieval, RAG will also retain conversational memory, learning from past interactions while continuously retrieving new information.
Editor's Note: Read up on other trends in the search and knowledge management space:
- GenAI Can Improve Enterprise Search, But Remains a Work In Progress — Generative AI could overcome traditional problems associated with enterprise search. However, it still has a way to go before its fully functional.
- What's Next for Enterprise Search? AI Knows — People don't want to search, they want to find information. A new breed of solutions are changing the boundaries of enterprise search.
- AI-Driven Knowledge Management Turns Repositories Into Intelligent Ecosystems — Knowledge management is undergoing a profound transformation, driven by the rapid integration of artificial intelligence.