KPMG Pulls AI Report After Fabricating Client Case Studies

Hey everyone and welcome back to the show. This is your weekly roundup of news in the profession. I'm Blake Oliver

and I'm David Liry

and we've got our lead story today about KPMG's AI report that is full of vibe citations. That's a new phrase for me. We'll dig into that and more in this episode. But David, first let's thank our sponsors.

Yeah, our sponsors this week we have Canopy, Valuebuilder Systems, OnPay, and STR Search.

Let me ask you something. How much of your day is actually spent doing accounting? If you're like most firm owners, 30 to 40% of your time is eaten alive by the work around the work. We're talking chasing client documents, drafting the exact same emails over and over again, manual filing, and trying to remember what a client said on a call last Tuesday. It's an administrative tax, and it's killing your profitability. That's where Canopy comes in. Canopy actually delivers on the all-in-one practice management promise. It handles everything from proposal to payment and all the steps in between. Smart client intake, tax workflows, month-end close automation, and billing are all in one unified platform. No more duct taping 10 different apps together. Plus, they have Canopy Co-worker. It's a secure AI assistant that lives right inside the platform and actually does real work. It drafts contextaware emails, summarizes client histories, takes meeting notes, and turns them into tasks automatically. Early access firms are already seeing what's possible when AI works inside your workflow instead of alongside it. To see what a truly modern automated practice looks like, head over to accountingpodcast.promo/canopy. That is accountingpodcast.promocanopy. And that's a great leadin to our lead story about this KPMG report. David, we both spotted this and uh it's like another one another another big four report that is is full of citations that just are not real or are mis uh misidentified or all sorts of stuff. So, what's going on here?

And this has happened now so many times that I asked you if we already talked about this story, but no, it's another new story. It's another new incident. So KPMG just had to pull a major AI report after getting caught with completely fabricated case studies. All the victim of AI hallucinations. So bigname organizations like UBS, NHS, Greater Manchester, Transport of London all claimed that all the claims about their AI usage were completely false or misleading.

This is a 2025 report titled total experience redefining excellence in the age of agentic AI. And this is, you know, a month after we had this happen with EY and then I think Deote had some stuff maybe a month ago. It's just we have to create a database and just track these because the big four just keeps doing it over and over and over again. Um, and the ironic part of this whole thing is, you know, KPMG just like Deote has an AI marketing page with an article on it that says essential elements of responsible AI how solid god rails can help you scale AI faster. like they're ma they're marketing themselves as the AI experts. The only way I could think this could work is now big for four can go to Fortune 500 and be like look we know all the mistakes that can be made now. Listen to us, hire us because we keep screwing up so we know what to do or not to do. This was detected by a platform called GPT0 which is designed to detect whether text was written by a human or generated by an AI model such as Chad GPT or Claude or Gemini. And it was originally developed to help educators maintain academic integrity, but it's grown into a broader toolkit used by journalists and enterprise teams and investigators to audit content for authenticity and factual accuracy. and they put out a report on their website about this uh KPMG report and they found 45 citations that uh only five of which were accurate. 28 of the citations pointed loosely to real sources but included altered or fabricated components. 12 citations were too vague or flawed to verify confidently. And at least 16 of those citations qualify as hallucinations under GPT0's methodology. Its hallucination check tool rated the report 89% flawed. The company also said that many of the reports underlying claims appear false, exaggerated or misattributed, especially where the report describes companies using agentic AI in customer experience settings. So these are examples of agentic AI that were hallucinated and some of the examples are uh interesting. KPMG described an Austrian utility Verbund as using AI agents for real-time household energy optimization, including smart appliances and EV charging. But the cited source is a press release about Verbund's venture arm investing in a startup working on energy optimization. And so the report conflates the investor and the startup while adding unsupported details about Agentic AI and household level use cases. And there's just many, many more of these examples. The report incorrectly claimed that the airline Emirates had a mobile chatbot named Sarah that could converse with passengers and change flights, but Sarah was actually introduced as a robot assistant in 2023 and did not have the flight change capability described. Another example is the report says that East Japan Railway was using AI agents for travel recommendations and disruption prediction back in 2019. But that's impossible because this press release that was cited uh predates the commercial emergence of Agentic AI and doesn't mention COVID. I mean that's before COVID even. Yeah.

Yeah. So, uh, another great case study, another example of why you need a human in the loop and you need to verify, uh, what the AI is generating for you and not just push it out to clients or to the world.

Well, especially if you're going to claim to be the expert on AI and train people, um, one of the things kind of related to this, I noticed, you know, how we we we shifted our spend on AI to claude and we kind of shut down our subscription to uh, Open AI. Well, I'm still been using Open IIA, but now I'm getting ads.

Guess who my ads are from?

PWC offering me like AI expertise to implement AI. Like the these big firms are just like you can't play off you're the expert and then make mistakes like this. It's not going to work long term.

So, we're talking about AI. Let's do a follow-up to a previous story we've discussed that now includes AI. And this is the uh Pentagon's inability to pass an audit. Well, the Pentagon is turning to AI to help it do so ahead of the 2028 deadline that the government has set. They've awarded a nearly 49 million dollar contract to Groundswwell Corp. for an artificial intelligence-based platform called Agentic Auditor. It runs through 2023, no 2031. and it's going to help the Army, Navy, Air Force, and other Defense Department agencies with data collection and audit preparation. The Defense Department is the only federal agency that has never received an unmodified financial audit according to the Government Accountability Office.

Did it say how much they're paying for this?

$49 million.

$49 million. So, basically, they went and signed another contract for $50 million to track their other millions and millions and millions of dollars of contracts. No, billions and billions of contracts.

What is it? Uh, the defense department budget is now like a trillion. I think

it's somewhere in there. Yeah,

something like that.

So, let's hope that it works. More AI news. Copilot Co-work is now generally available from Microsoft. This is well, I haven't seen it myself, but it appears to be Microsoft's version of Claude Co-work. So more than a chatbot, it can run complex multi-step tasks from start to finish and give you the finished result, not just a draft. More than half of the Fortune 500 have now used it since it was in a three-month preview. And uh so if you have not yet used Claude Co-work because you're limited in what you can use in your firm or in your company, check out Copilot Co-work. They're saying

probably most accounting firms are in the Office 365 stack and the Microsoft stack. You probably have people in your firm using it right now if you or you probably have it installed in your computer right now.

They also say it's cheaper than um Claude Co. 30 to 40% cheaper on average per prompt. And that's based on Microsoft's own internal analysis of 125 test runs across 12 prompts. It's available in Microsoft 365 co-pilot user subscription license. So if you've got that, you should now have access to it. It's built based on usage and um the the price is going to depend for each task on the model, the context amount, the tool calls, the runtime, that sort of thing. So, this is great news because Cloud Co-work has blown away the finance community as we've talked about on uh previous episodes. And I've actually got an example that I'd love to share with you about uh some writeup work that I did, some bookkeeping, accounting work that I did,

please. And then I'm going to share with you a gripe about AI and accounting. So, continue. You go first.

Okay. Well, you know, here's what happened is I needed to do two years of writeup work, 2024 and 2025 for a uh for a simple, you know, service-based business. And this is something that in the past would have taken a lot of time um trying to go get CSV statements uh or CSVs from, you know, bank accounts online, right? credit cards, you know, AMX, uh, bunch of different bank accounts, some relay accounts, some not relay accounts. Sometimes you can get PDFs, sometimes you can get CSVs. You can't always get everything. So, I got everything I could. I went in and I spent an hour or two downloading every single statement that I could find in whatever format I could get it. So, it's a mix of PDFs and CSVs and I dumped them all into folders. And I pointed cloud co-work at the folders and I wanted to do this right up in zero. And so I I was working with cloud co-work to