One of AI's great promises is to take data and produce advanced information. The problem is that today, we have too much information that is hard to process; it's just too much. So, let's see what happens when we take real point-of-sale software data and put this promise to the test.
So, we decided to test the AI's understanding of a typical SMB retail shop from its customer's sales report. Knowing your customers is essential for retail success, and we will check whether an AI customer data analysis can help us make informed decisions about them.
The added question is which free AI tools can perform this task most effectively. Only a few SMB retailers have paid accounts for AI ChatBots, so we decided to restrict our tests to the free version. As you will see, we immediately ran into problems with this during this test.
Okay, let's test that and see.
Our Test
We have already established a comprehensive evaluation process to determine which AI systems deliver the most actionable insights here.
So, we challenged six leading free AI systems to analyse real customer sales performance data from 2023 and 2024.
Each AI system received identical instructions: "Using data from the attached CSV files, analyse and compare sales performance and visit frequency for top customers in an Australian Newsagency between 2023 and 2024. Identify key changes in behaviour for the highest-value customers."
For a fair assessment, we evaluated each system's customer analysis report using nine key performance indicators and scored out of 10:
Evaluation Framework
Quality of Information
How accurate was the data presented about customer behaviour and trends? If it's not correct, it's worse than useless.
####Usefulness Do these insights improve a retail business? We do not need useless information.
Clarity
Was the analysis presented straightforwardly and clearly?
Actionability
Did the report provide clear, implementable recommendations?
Accuracy
Were the calculations and conclusions drawn from the data correct? AI is known for giving wrong results. This is a common problem of AIs.
Adaptability
Did the analysis take into account specific retail contexts and customer demographics?
Depth of Analysis
Did the report go beyond surface-level observations to uncover meaningful patterns?
Creativity
Were there innovative suggestions for addressing the identified changes in customer behaviour?
Customer-Centric Approach
Did the analysis effectively segment customers and provide tailored approaches?
Testing Limitations
Unlike our pricing study, we encountered several practical limitations with free AI access immediately:
Claude and Chatgpt required paid accounts to process our instructions, so they failed the free test. However, as they are very important AI chatbots, we decided to test them with our paid accounts, but they failed immediately.
QWEN would only process our prompt by editing it in the window's notebook, so we had to redo the tests with consolidated data. Doing this with QWEN requires some computer skills, which others do not need. We marked it down for this.
All the other AI systems could process our request without issue using their free tier offerings.
Immediately, we saw that when we analysed the solutions, not all of them offered the same ability.
Results Summary
The performance rankings revealed interesting shifts from our previous pricing analysis:
| AI Tool | Score (out of 90) |
|---|---|
| ChatGPT | 78 |
| Grok | 68 |
| DEEPSEEK | 67 |
| QWEN | 66 (marked down) |
| Claude | 59 (failed) |
| Google AI | 55 |
Interestingly, while Google AI topped our previous pricing study, it ranked last for customer analysis.
Detailed Analysis of Each AI Tool
ChatGPT (Score: 78/90)
On the paid version, and I stress the paid-only
Strengths
- We thought it showed exceptional clarity. The report was well-organised, with sections in a logical progression We liked its innovative customer segmentation framework, which includes categories like "Growth Champions" and "Frequent Shoppers."
- We all liked its analysis and felt it was comprehensive
- What we liked was its tailored recommendations for different customer segments
Problems
- There was nothing we thought was innovative, but it worked by the rules. It ignored the location, that we were in Australia, and the same problem we had in the previous test.
Overall
ChatGPT transformed customer data into strategically actionable segments that you can use in your marketing and sales approaches. It saved time but gave us little originality.
Grok (Score: 68/90)
Since it's now free, this is a brilliant AI. You should be delighted with it.
Strengths
- We notice that it is outstanding in its accuracy and attention to detail
- The report was an in-depth and comprehensive analysis
- We liked its comparative analysis of individual customers
- It came up with a category, and we liked the emerging high-value customers
Problems
- We thought that most of the insights were limited in actionability; it's nice to know something, but what is the point of telling us if you cannot use it?
- Also limited in creative solutions
- The report sections did not follow a logical order.
Overall
Grok provided an excellent analysis and identified some subtle patterns in customer behaviour. It would be good to investigate why, though you'll need to develop action plans based on its findings.
DEEPSEEK (Score: 67/90)
Strengths
- Excellent formatting and visual emphasis that make your information accessible
- We liked that it had specific, implementable recommendations tied to analysis findings.
- It was accurate in its data presentation, with correctly identified trends
- Effective customer segmentation approaches
Problems
- Did not give a detailed customer-by-customer examination
- Not that good in a specific Australian retail context
- Recommendations were sound but somewhat conventional
Overall
DEEPSEEK struck a good balance between analytical depth and practical recommendations.
QWEN (Score: 66/90)
Do not underestimate this AI. The only reason it did not win was that it required some advanced computer skills to use. It was not just a simple matter of using it.
Strengths
- We liked its brevity, which offered highly actionable recommendations
- We liked its concise format, clear headings and bullet points
- It provided good creative thinking in recommendations for improving customer visits
- We liked its suggestion for improving a loyalty program and bundling.
Problems
Small report: Now, this may not be a problem for you. Some of us liked that it produced a good condensed report. They felt they did not want a book; the quicker it could be said, the better. I disagreed; I wanted to know as much as possible. Let me know your thoughts on this below. We thought it didn't consider that we were talking about retail.
Overall
Many people like QWEN's concise, action-oriented approach, which provides quick and practical guidance without requiring extensive analysis.
Claude (Score: 59/90)
It was a paid version, yet its score was bad. You expect it to be better if you pay, but you don't get what you pay for with AI.
Strengths
- It provided accurate data analysis throughout
- We did like its breakdowns of top customers and their performance changes
- It had clear section headings and a well-organised structure
- It identified key account growth opportunities
- It also identified at-risk relationships
Problems
- Overall, it lacked specific implementation strategies
- The segmentation of customer types was bad.
- Again, as with some others, not much is known about Australian retail
- Report sections varied in detail
Overall
Claude was solid but struggled to translate those insights into tailored, implementable recommendations.
Google AI (Score: 55/90)
Now, Google AI is a good AI. Last time, it came out the best, but here, it's the worst. Google changed the free version because it wants you to switch to the paid version.
Strengths
- Methodical approach to analysis
- Accurate calculations
- We thought that it produced logically sound conclusions
- It considered many factors
Problems
- It presented an analysis without any specific recommendations
- Little creative
- Minimal segmentation of customer types
- Limited adaptation to retail
Overall
The free Google AI fell short of providing the strategic interpretation and actionable guidance that retailers need.
Key Findings and Implications for Your Retail Business
Our evaluation revealed several important considerations for using AI:
Different tools excel at different retail tasks
It's all good to analyse, but an ideal AI solution should connect the dots between "what's happening," "why it's happening," and, most importantly, "what to do about it." Not all AI ChatBots are equal.
The significant shift in rankings between our pricing study, where Google AI dominated, and this customer analysis evaluation highlights the critical point that AI is not a one-size-fits-all approach. It also showed that suppliers are moving to a user-paid model.
Usability
Busy retailers need an AI that produces a clear organisation and logical flow.
Practical Recommendations for Australian Retailers
Your specific need
The reality is that different AI systems excel at other tasks. There's no way around it; you need to test which one works better in your circumstances.
Prioritise actionability
Choose AI tools that connect analysis to specific actions you can take. Ask yourself: "What can I do tomorrow based on this information?" If the answer isn't clear, the tool may not be right for your needs.
Provide detailed context
We tested with more specific information about a business, location, and customer base. We did get better results, but that's outside of this test. I would include such information if I were doing it in a shop. Don't assume the AI knows your business environment; even if it does, it can use it.
Verify findings
We found that AI made errors and provided advice that didn't align with business reality. Always check.
Continue with a focused question.
These reports should be considered the start of your analysis. To get more insights, I would ask specific questions about the customer, the segments, loyalty patterns, or purchase frequency. For example, "Which customers decreased their visit frequency in the last quarter?" will yield more actionable results than "Analyse my customers."
Integrating AI Insights with Your POS System
For retailers using our POS systems, connecting AI analysis has tremendous potential. Exporting customer sales data in formats your AI can analyse is easy; you can do this in the report section. I will discuss this in a future post.
The Future of AI
This evaluation demonstrates that AI tools have already reached impressive capabilities. Using them, you can get free access to dedicated data analysts. They offer you access to sophisticated insights previously available only to large organisations.
My thoughts
While it's free, I recommend looking at Grok, our winner.
Written by:
Bernard Zimmermann is the founding director at POS Solutions, a leading point-of-sale system company with 45 years of industry experience. He consults to various organisations, from small businesses to large retailers and government institutions. Bernard is passionate about helping companies optimise their operations through innovative POS technology and enabling seamless customer experiences through effective software solutions.
