Editorial
We at SWOOP Analytics have recently released our third global SharePoint Online intranet benchmarking report. In this study, we examined how employees from 28 organizations are using their organization’s intranet, on an hour-to-hour basis, over a three-month period.
A key theme from the report is the status quo on the use of artificial intelligence (AI) with intranets. Responses from benchmarking partners ranged from “AI is currently blocked in our organization” through “we use it extensively and are working on an AI chatbot interface to our intranet.” The main theme, though, was “not if, but when.”
The most pervasive use, and the one that is relatively risk-free, was the early adoption of AI tools to help improve intranet content. However, our benchmarking partners found an over-reliance on AI-generated content can be problematic. For news in particular, human editors were better at targeting content for employee consumption, with nuance, empathy, cultural alignment and relationships. In other words, co-pilot, not auto-pilot!
Many of our respondents identified the potential for AI chatbots to provide a new level of engagement for their intranets.
What Is So Special About AI Chatbots with Intranets?
For decades, intranets have had a reputation for poor search function. Our benchmarking showed that, on average, “search” made up less than 1% of all intranet sessions. What’s worse is that even when search was used, more than 60% were abandoned as unsuccessful. Yet if you could search and find what you are looking for, you wouldn’t need to use less-efficient navigation schemes for this task.
The reasons for poor intranet search are both large and varied, but consistently relate to poor content management, such as poor structure, outdated information, lack of metadata, simple keyword search, spread across multiple platforms or little or no personalization.
AI chatbots are finally providing an intranet search that works. If you have used a chatbot like ChatGPT or Microsoft Copilot, you will notice the first stage is “searching ....” Formally called retrieval-augmented generation (RAG), it uses cognitive search techniques to identify chunks of information that relate to the user query. These chunks can be selected paragraphs, headings, bullet lists and so on. This identified content then augments the user query to create the prompt to be fed to the foundation AI model, whether it’s GPT-4, Claude, Gemini or another.
Unlike a traditional search, an AI chatbot:
- Understands natural language, meaning content will be identified even if keywords are absent; "How do I claim expenses after a work trip?" will identify content labelled "travel policy 2025" even without matching keywords
- Provides direct answers, not just links to potentially relevant content; instead of: "Here are 10 documents about leave policies,” you get: "You are entitled to 20 days of annual leave per year. Here's the form link”
- Clarifies queries through conversation; just like a human conversation, the questioner invokes a conversation thread to further clarify the request, such as “is 20 days available even if I am resident in the U.S.A.?”
- As AI chatbots can store all previous queries and can ‘remember’ previous queries and contexts, searches are not started with a ‘clean sheet’. Over time the chatbot can learn to tailor its responses using your own acronyms and contexts.
Sourcing information from your intranet through a natural language user interface is better for users. We found, however, it cannot paper over poorly structured and poorly governed intranet content.
Is Your Intranet AI-Ready?
We have found the world needs some form of AI Readiness Index. As part of our benchmarking study, we looked to create such an index. The common dimensions identified are:
- Content: Content that is up-to-date, for better AI-powered responses. If content hasn’t been updated, or the content creator isn’t working for the organization anymore, you risk not being able to rely on the content. Given how reliant an intranet-based AI bot is on the underlying content, we assigned this a weighting of 60%.
- Engagement: Content that people engage with indicates it is being read, and we can assume it is more likely to be useful. The more unused content you have, the less likely it is to be useful. While AI will also surface content from sources that people aren’t using, we suggest usage does reflect value. Therefore, we assigned this a weighting of 30%.
- Search effectiveness: Low search effectiveness is reflected in a high proportion of failed searches, implying a content gap and/or a lack of good metadata. Either of these mean your AI-bot is less likely to be able to generate high-quality responses. We have weighted search effectiveness at 10% because there is such a blend of how well search is perceived to work.
Using these dimensions against our benchmarking dataset, we calculated a prospective AI Readiness Index for each organization:
The average weighted AI Readiness Index across the 28 organizations was 51.12 out of 100. Our top performer scored 64.8 and the lowest scored 41.48. These findings aligned with our interviews: Organizations with strong governance and clean content felt AI-ready; those with messy or outdated content did not.
Preparing Your Intranet for AI Success
A key finding from our benchmarking study is that, while more employees are using the intranet (on average, more than 93%), they are spending less time there (on average, less than 6 minutes/day). With employees increasingly relying on the intranet to do their work, the need to provide them with the information they require when they need it has never been greater.
Our AI readiness index shows that all organizations will have some scope for improvement by:
- Intranet content being up-to-date for AI to be helpful. Minimizing outdated or rarely visited content reduces the chance for the AI engine to provide inaccurate responses.
- Filling information gaps, especially for more common factual queries. AI chatbots can hallucinate “facts” to fill an information gap.
- Good information governance, such as naming conventions, archiving documents and setting permissions and sensitivity on documents.
It's tempting to say “Keep doing what you're doing,” but AI raises new questions about what qualifies as “good enough.” While one can’t fault the sentiment, there is still much gray in how much is “good enough for AI.” Perfection isn’t realistic, so where’s the tipping point where AI still works reliably? How much can AI fix our inadequacies?
“AI success isn't about hoarding data or making it perfect,” said future of work commentator Charlene Li in her article Stop Waiting for Perfect Data. Here’s How to Win with AI Now. It's about using what you have to create real business impact.” This is consistent with my earlier article, Looking to Replace Your Intranet with AI . Manage your chatbot scope to the scope of your current intranet governance priorities: If you have invested in strong governance of company policies, limit your initial chatbot scope to these sites.
How to Build an Intranet Chatbot
Intranet content pages are largely factual. Prefacing your AI prompts with a system prompt such as "Be brief, factual, and cite sources if available" will help influence the tone and reasoning style. Chatbot systems also offer temperature control, with low-temperature settings providing deterministic and precise results (preferred for intranets) as opposed to high-temperature settings creating more diverse, creative (but possibly hallucinated) results.
Ideally, we want chatbot responses to be 100% drawn from intranet content, with the AI foundation model largely providing conversational facilitation only. The more incomplete the information is for end-user questions, the more the AI foundation model will be asked to fill the gaps. The result will be over-generalized responses or at worst, erroneous or hallucinated results.
The “search effectiveness” component of the AI readiness index is intended to give some insight into information gaps.
Webinar
Sep
25
CMS Briefing: A Live Look at What’s Next in AI-Driven Platforms
Learn how leading organizations are using AI‑driven tools to publish faster, personalize smarter and stay secure.
Register
Webinar
On demand
Ready or Not: How Data-First Organizations Are Unlocking Agentforce Potential
Learn how to cut through the noise, activate Agentforce and build a Salesforce AI strategy that actually delivers.
Watch Now
Webinar
On demand
AI in Customer Service: Faster Resolutions, Happier Customers
Don’t let rising demand burn out your team. See how to build a smarter, more resilient support org.
Watch Now
Webinar
On demand
From Hype to High-Impact CX Strategies That Actually Scale
Turn buzzworthy AI and outsourcing trends into measurable CX wins with fresh 2025 data.
Watch Now
Webinar
On demand
Insights to Action Rethinking the Contact Center for Real Business Impact
Join our exclusive webinar to hear CX executives share their innovative strategies for transforming service delivery.
Watch Now
Webinar
On demand
Accelerating Healthcare Ops with AI
Sign up to see how your peers are putting AI to work — and making it stick.
Watch Now
Why Chatbot AI Analytics Are Essential
While not yet commonly available, chatbot performance analytics will quickly become essential. There will be a continual need to:
- Assess what information employees are actually seeking through their queries. Clustering query content can expose emerging themes of employee concern.
- Create an intranet content/AI foundation ratio that provides a more accurate measure of information completeness. This measure relies on a similarity score for a query measured during the RAG search of intranet content. A similarity score threshold can trigger a "I'm not sure. I couldn’t find anything” response, rather than risk a hallucinated response from the AI foundation model.
- Assess how well user queries are being responded to, perhaps by allowing end users to rate responses. Otherwise, it could be inferred by how often queries receive a “I'm not sure. I couldn’t find anything” response or by overly long discussion threads.
- Assess how well top-down strategic content is engaging employees. For example, when a flexible working’ policy or a compliance policy are announced, do chatbot conversations identify areas of confusion or incompleteness in a policy?
- Assess what percentage of employees are engaging with this information to help do their work.
The bottom line is to be mindful of the importance of well-governed content, but don’t let that stop you from experimenting with AI chatbots. Preface your AI chatbot responses with appropriate warnings, then monitor the quality of the responses. As a minimum, you will find out what employees are seeking, and how aligned they are with your most important strategic communication goals.
Learn how you can join our contributor community.
