Employees overtrust humanlike AI, ignoring its flaws

Trust in generative AI (genAI) has surged globally despite gaps in AI safeguards because of its humanlike ability to respond to queries or prompts, according to a new study by IDC.

Only 40% of organizations invest in “trustworthy AI,” or AI with guardrails. Yet, those investing the least view genAI as 200% more trustworthy than traditional, proven machine learning — despite the latter being more established and having greater reliability and explainability.

“Our research shows a contradiction: that forms of AI with humanlike interactivity and social familiarity seem to encourage the greatest trust, regardless of actual reliability or accuracy,” said Kathy Lange, research director of the AI and Automation Practice at IDC.

IDC’s study, sponsored by SAS, found that organizations that build governance, ethics, and transparency guardrails are 60% more likely to double AI project ROI — highlighting the cost of ignoring responsible AI practices. The global survey of 2,375 IT professionals and line-of-business leaders found that strategic AI use, not just cost-cutting, drives market share and customer gains.

GenAI has rapidly outpaced traditional AI, and as organizations move toward agentic AI, its influence on decision-making — often hidden — will only grow. (AI agents are autonomous programs, often containerized, that make decisions in dynamic environments.)

“What stands out in this research is how rapidly the center of gravity has shifted from traditional machine learning toward generative and agentic AI,” Chris Marshall, IDC’s vice president of Data, Analytics, and AI research wrote in the report.

Yet, without trust, progress stalls, IDC found. AI trust isn’t just ethical — it’s financial. Nearly half of companies face a trust gap, leading to lower ROI. Mature AI adopters invest more in responsible AI and see better results.

While only a quarter of survey respondents have dedicated AI governance teams, most plan to boost investment, especially in ethics training, bias detection, and responsible AI platforms. “Trust is key to unlocking AI’s full value,” IDC said in its report.

According to Gartner research, 40% of agentic AI projects will be canceled by 2027 due to rising costs, unclear value, or poor risk controls. An even more recent study by MIT found that up to 95% of AI pilot projects fail.

MIT’s study found that companies are failing to implement genAI effectively, not because of bad models, but due to a “learning gap” in how tools and organizations adapt. While execs blame regulations or technology limits, MIT said the real issue is poor enterprise integration.

Most budgets go to sales and marketing tools, yet the highest ROI comes from back-office automation, where genAI can cut outsourcing and streamline operations. How companies adopt AI matters: partnerships with vendors succeed twice as often as building AI in-house, according to MIT.

Another study by Carnegie Mellon University (CMU) and Salesforce assessed the performance of AI agents and found the technology failed at tasks 70% of the time — and many of those tasks were pedestrian at best.

CMU researchers created a simulated small software firm called “TheAgentCompany,” where they tested leading AI agents (including Claude 3.5 Sonnet, Gemini 2.0 Flash, GPT‑4o, and others) on multistep office tasks — a mix of engineering, sales, HR, and finance jobs.

The agents struggled with simple actions, such as closing pop-up dialogs, interpreting common file formats, or identifying contacts correctly. Some even “cheated” by renaming users to simulate progress. The study also found that AI agents had limited human‑like performance. Even top agents only reliably completed a quarter of workplace tasks in a controlled setting.

Graham Neubig, an associate professor in CMU’s Language Technologies Institute (LTI) who directed the development of TheAgentCompany, said the low achievements by the AI agents “met or slightly exceeded” expectations based on benchmarking tools he’d used before.

For example, website navigation can be tough for AI tools, as shown by one agent’s inability to close a pop-up window. “It’s a silly little thing that wouldn’t bother a human at all,” Neubig said in a CMU article on the study.

The lack of social skills was evident when one AI agent never bothered to connect with the company’s HR manager when instructed to do so. And the failure to recognize the relevance of a “.docx” extension in another case is proof that the tools often lack common sense.

The pressure to prove generative AI’s value is growing. While some executives aren’t demanding ROI yet, that’s changing as more organizations move from pilots to real deployment with measurable results.

“Organizations are no longer experimenting on the edges; they are embedding these technologies into workflows that span customer service, coding and decision support,” Marshall wrote in the IDC report. “Yet, the real differentiator is not just adoption but integration: The ability to unify structured and unstructured information, apply governance and embed explainability into automated processes.”

Companies using “trustworthy AI” in automation see the biggest efficiency gains, while others risk scaling inefficiencies.

In the meantime, leaders should focus on what they can control, such as data readiness, governance, compliance, and talent in order to set the foundation for long-term genAI success, IDC said.

Likely to no one’s surprise, IDC’s study found agentic AI needs strong data infrastructure and talent to succeed; in other words, AI agents cannot be fully left to their own devices but must have human oversight.

The study also found that quantum AI — the combination of quantum computing and artificial intelligence (AI) — is emerging, with real potential across industries.

Quantum tech is still experimental but it is “generating real excitement” across industries like finance, logistics, and climate science, IDC said.

Agentic AI has been adopted by 52% of those surveyed by IDC, and it is poised to further expand the boundaries of automation and intelligence. Likewise, quantum AI promises to solve problems previously considered intractable due to computational limitations.

“While these technologies are still emerging, their potential has captured the imagination of decision makers eager to experiment and innovate (61% seek greater process efficiency, only 32% seek cost savings),” IDC said.Microsoft Planner cheat sheet: How to get started – ComputerworldRead More