For people like us operating in a dynamic and competitive landscape, providing exceptional customer service and support is our key differentiator. So, our support team works hard to respond to client questions, troubleshoot issues, and ensure our point-of-sale (POS) software enables their success. So, when we heard about AI conversation bots that could potentially assist our support team, we were eager to test this emerging technology.
After developing our own AI chatbot prototype and trialling it on actual customer queries, we found there are indeed limitations with AI support today.
We feed it thousands of our customers' queries.
Depending on who accessed its answers, we got these disputed scores. I say disputed as some judges were much more negative than others. This is a consensus score, not an individual view.
- Approximately 23% of the chatbot responses were largely to wholly accurate. These tended to be the stuff that our support staff felt were things they knew well and would have little trouble doing without AI.
- 50% were partially correct but lacked vital details.
- 15% were unclear due to poorly phrased customer questions. This is a common problem that those who ask questions in the real world to get accurate information need to know much. Rubbish in = Rubbish out.
- 10% were utterly wrong.
Our AI chatbot clearly has room for improvement before it goes live.
However, AI can still provide value when implemented correctly by augmenting our support team’s capabilities.
Why AI struggles with complex support queries
Through our testing process, we gained insight into factors that limit AI chatbots’ capabilities for software support:
1. Insufficient training data
Like humans, AI systems require extensive ‘practice’ through training data to become highly skilled at specific tasks. Our prototype only had access to a thousand pre-recorded customer support calls from which to learn - this is far too little data to fully grasp the complexity and nuances of troubleshooting what is our system probably the most advanced POS software in Australia.
2. Incomplete understanding of context and ambiguity
Humans draw upon years of world knowledge and experience when interpreting language. AI chatbots currently lack this contextual awareness, so they struggle to clarify vague questions and handle ambiguities. Our clients have a range of English knowledge; some know plenty, we have clients with doctorates and some that barely speak English. So 15% of customer queries stumped our chatbot due to unclear wording that we could easily interpret.
3. Inability to grasp nuanced domain knowledge
POS software is feature-rich and built to handle intricate retailer needs. Most AI systems today would need explicit training to comprehend the industry-specific terminology, workflows, exceptions, integrations, and technical considerations involved. Without directly relevant domain training, the chatbot’s knowledge gaps led it to miss critical details in 50% of test cases.
Best practices for implementation
While the technology still has maturing to do, useful AI assistance is within reach by following guiding principles for implementation:
Ensure sufficient high-quality training data
For accurate learning, training datasets must cover each type of support scenario the AI may encounter. Data should be cleanly recorded without errors that could propagate misleading information. The problem here is that our information is real, and in the real world information with errors is common.
Provide ongoing human feedback.
Humans must continually verify and correct the AI’s responses, allowing its knowledge to evolve through feedback over months and years. This human-in-the-loop oversight is critical. This will take a lot of work.
The future role of AI in customer support
Though AI chatbots cannot wholly substitute for human support staff at this stage, we see bright possibilities for AI assistance going forward:
-
Handling high volumes of repetitive queries - Freeing support staff to focus on higher-level, more complex problem-solving for customers.
-
Providing 24/7 self-service - Enabling customers to access basic troubleshooting after hours when support staff are offline.
-
Supporting human agents - Assisting our support staff with knowledge lookup, documentation, and tracing issues.
While AI support today is imperfect, we are excited by its prospects through careful implementation. Over the next several years, we aim to refine our chatbot to be an invaluable teammate for our human support staff - combining the best human and machine capabilities to deliver exceptional customer assistance. With a spirit of experimentation and patience, as this emerging technology matures, we know that AI promises to improve customer service.
Let me know if you are interested in learning more because this is really interesting stuff.
