Identify hallucinations in Slackbot responses

The large language models (LLMs) that power Slackbot can sometimes generate responses that are factually inaccurate or that don’t make sense in the context of a conversation. These responses, known as hallucinations, are often plausible-sounding and delivered confidently.

Read on to learn more about recognising hallucinations, ways to avoid them and how to report suspected hallucinations to Slack. 

Note: It’s important to review AI-generated responses critically and verify information yourself before using it to make decisions or sharing responses with others.

How to recognise hallucinations

AI responses in Slack only contain information you have access to (like public channels, private channels you belong to and your direct messages), so if you see unfamiliar information it’s likely a hallucination. 

Hallucinations that Slack can detect

Here are the types of hallucinations that we can identify and flag in Slackbot responses: 

• Files and messages
  Links to files and messages that lead to a dead end because they don’t exist in your Slack workspace or Enterprise organisation.
• People and channels
  References to people or channels that don’t exist.

When we detect one of these hallucinations, links, channel names or people will be replaced with a Link not found, Channel not found or User not found tag. If you see flagged hallucinations in a response, you should carefully review the full response for accuracy. 

What to do if Slackbot hallucinates

Slackbot has been trained to be as helpful as possible, but in some cases it won’t have enough information or context to provide a quality response. If Slackbot says ‘I don’t know’ or ‘I don’t have enough information to answer that question’ and you ask it to try again without providing new instructions, it’s likely to start hallucinating in order to give you an answer that you’re satisfied with. 

Use these tips to verify the information that Slackbot sends to you or which helps you get back on track when it’s having trouble providing an answer: 

• Ask Slackbot again
  Ask Slackbot the same question in a new chat and see if the new response is consistent with the original reply. 
• Keep conversations short
  If you’ve sent several messages in the same conversation, start a new discussion to reduce the risk of Slackbot getting confused by earlier replies. 
• Provide specific sources
  Rather than saying “Search for other places for more info’ try ‘Those aren’t the right channels, please use #channel-name and #channel-name instead. You can also look for messages from @person.’

Report hallucinations

If AI response feedback submissions are enabled for your workspace or organisation, you can report responses with hallucinated information. These reports help us identify patterns and improve Slackbot. Here’s how: 

1. From a Slackbot response, click the   thumbs down button. 
2. Tick the boxes next to the issues that you want to report, and include the messages in your conversation if you’d like. We’ll always receive the last message that you sent to Slackbot as part of a report. 
3. Click on Send.