During our 2025 survey, we found that many of our POS System users see real AI risks, including hallucinations that invent facts and privacy breaches under the Privacy Act. Our 2025 survey revealed a clear split: about a third were really positive, about a third were neutral, and the rest were deeply sceptical, with comments like "Hell No," "Don't trust AI," and fears of privacy invasion or AI "making stuff up."
Let us discuss these concerns with real examples, to help people to protect their operations while exploring AI's potential.
Hallucinations: AI's Plausible Errors
Everyone makes mistakes, your accountants, your electricians, etc. AI stands out because it makes so many errors (hallucinates). The problem is that it delivers its advice so confidently with so much information that it's hard to disagree with the wrong answers that sound right. No AI system eliminates this fully; claims to the contrary are misleading.
Real Retail Impacts:
- One of our clients faced copyright issues over a gift design card. Their AI advised sending a letter and including that they would take this issue to the Small Claims Tribunal if the people continued. When they checked with a lawyer, they were told it was pure nonsense, as copyright falls under the Federal Court. That letter would have made it worse. Much worse.
- In 2025, a legal case in the US involved an Agentic AI that deleted a company database, created 4,000 fake user records, and covered up the mess with false reports.
- I heard of an AI that was set to tidy up stock; it wiped many slow-selling greeting cards from the POS System files. The physical items were in the shop, but the POS System had no record of them because they had been deleted.
- I set an AI to examine one of our e-commerce client sites, which reported false stock sold on its invented promotions that never happened.
Always treat AI as a draft and verify manually to reduce problems if you are using an Agentic AI. Never let the AI act until you check the answer.
Privacy Invasion: Data Slips Out of Control
Survey users flagged "invades privacy," and they're spot on. Sending customer details, like say a book buyer's habits, to AI often means exporting data beyond your reach. This risks breaching Australia's Privacy Act.
Now, once that data hits US or Chinese servers, which is where most Australian AI is going, your control ends, and if you get into a privacy legal problem, I doubt the large AI providers will try to shield you. On the contrary, I am sure they will do everything they can to pass the blame onto you.
If you are sending user information, I strongly recommend avoiding your clients' names; instead, use their account numbers. You know their account numbers; no one else does.
Legal situation
Your data legally remains yours, but AI firms host it abroad, and those records are considered their property. US companies may hand it over to Australian courts without telling you if asked. If you use a Chinese system, this further complicates legal matters for the Australian Courts.
I doubt that relying on "privacy settings" will protect you from legal orders related to employee disputes, supplier contracts, or customer lists. Secret AI accounts and VPNs might help if you want privacy.
Human-in-the-Loop: The Essential Safeguard
AI is still new. We are all experimenting with it; currently, we view AI as a time-saver, not a replacement. You need human oversight to supervise.
Simple Steps for Your POS:
- Start with aggregated data only, no names, limited addresses and no emails.
- Always review AI suggestions.
- Measure how AIs work for you.
AI does work; our clients who use our free AI stock inventory system report massive benefits.
Info: If you have any concerns, I am happy to discuss.
Written by:
Bernard Zimmermann is the founding director of POS Solutions, a leading point-of-sale system company with 45 years of industry experience, now retired and seeking new opportunities. He consults with various organisations, from small businesses to large retailers and government institutions. Bernard is passionate about helping companies optimise their operations through innovative POS technology and enabling seamless customer experiences through effective software solutions.
