Confidence in workplace artificial intelligence (AI) remains low among Australian employees, with just 14% fully trusting the AI-generated outputs used by their organisation, according to new research from Appian (Nasdaq: APPN).
The findings suggest concerns about AI reliability are well founded, with 60% of Australian workers reporting they have personally identified an AI error at work. More than one in five also said employees in their organisation are discouraged from challenging AI recommendations or decisions, raising concerns about whether mistakes are being identified and addressed before they affect business operations.
Charlie Hutchinson, SVP Asia-Pacific and Japan at Appian, said organisations cannot expect workers to embrace AI if they do not trust its outputs or feel able to question them.
“It is hardly surprising that complete trust in AI is so low when workers are catching it making mistakes and are discouraged from questioning its outputs,” said Hutchinson.
“For businesses, mistrust creates a lose-lose situation. Employees may avoid AI or spend time checking everything it produces, wiping out the productivity gains it was meant to deliver. Alternatively, they may continue using an output they suspect is wrong because they do not feel able to challenge it. AI needs to operate within the same governance processes and controls as the rest of the business, with people able to step in before a significant error occurs.”
AI mistakes are affecting business operations
The research found that errors generated by AI are already creating problems for businesses and their customers.
One in five respondents said the most significant AI mistake they had encountered led to a customer service issue, while 13% reported operational disruption. Almost half said their most serious experience with an AI error resulted in consequences greater than a minor inconvenience, including financial, compliance or reputational impacts.
The possibility of AI making incorrect decisions was the leading concern for 25% of workers, followed by data privacy and security risks (22%) and uncertainty about how AI reaches its conclusions (20%).
“Every visible error gives workers another reason to question whether AI should be trusted with more consequential work,” said Hutchinson. “Employees are often the last line of defence before an incorrect output reaches a customer or becomes part of a business decision. If they are discouraged from speaking up, organisations remove one of their most important safeguards.”
These concerns are reflected in workers’ attitudes towards AI decision-making, with 94% opposing the use of AI to make decisions affecting employees or customers without limits or human oversight.
Senior management also retained a considerable trust advantage over AI, with 52% of respondents placing greater confidence in senior leaders, compared with 29% who trusted AI more.
Businesses face growing pressure to establish AI safeguards
As organisations introduce AI into more business processes, the need for clear governance is becoming increasingly important.
Separate global research from Harvard Business Review Analytic Services, sponsored by Appian, found that while 92% of respondents agreed AI agents need rules-based guardrails to operate safely and effectively, only 48% said their organisation had defined them.
The findings point to a disconnect between awareness of AI risks and the measures businesses are taking to manage them, particularly as AI moves into processes involving customers, employees and operational decisions.
“Businesses know AI needs rules, but many are deploying it before those rules are in place. Doing so without the right controls can quickly lead to unexpected token costs, operational disruption and accountability gaps,” said Hutchinson. “Governance cannot be bolted on after AI is already influencing decisions. AI needs to be embedded within governed business processes that control what it can access, define the rules it must follow, and determine when a decision must be handed to a person. That gives employees the visibility and confidence to use AI while allowing businesses to capture its productivity benefits without giving up accountability.”
Human oversight remains a priority for workers
Australian employees expressed a clear preference for retaining human involvement in decisions that affect people.
Two-thirds (66%) believed decisions concerning employees or customers should either require human review and approval or be made entirely by people. A further 28% supported restricting AI to low-risk decisions.
The results suggest that while AI is becoming part of everyday workplace operations, employees want greater assurance that its recommendations can be questioned and its decisions reviewed.
“Australian workers are rejecting unchecked AI. Businesses will only realise the full value of AI if employees build trust, are willing to use it within critical processes, and confident enough to act on its outputs. That requires governed processes where AI recommendations and autonomous agent outputs can be systematically reviewed, validated, or overridden and human judgement remains part of important decisions,” concluded Hutchinson.
For more information about AI automation for critical business processes, visit Appian.
