Rogue AI agents threaten accounting profession as Big Tech hides $3 trillion ...

Attention: This is a machine-generated transcript. As such, there may be spelling, grammar, and accuracy errors throughout. Thank you for your understanding! Blake Oliver: Is this even constitutional? Because this is a tax that targets basically 200 people in a state of millions and millions and millions of people. And I always had this impression that, like, taxes are supposed to be, uh, you know, generally like equitable or like, you can't just. Can you even could you do that? Could you just pass a law to tax one person. David Leary: Coming to you [00:00:30] weekly from the OnPay Recording Studio. Blake Oliver: Hello and welcome back to the Accounting Podcast, your weekly roundup of news in the profession. I'm Blake Oliver. David Leary: And I'm David Leary. Blake Oliver: This week we're talking about rogue AI agents, off balance sheet AI financing. And we're going to cover Xerocon 2026, even though neither David or I could make it this year. Sadly, first year I've missed it since it started. Ah! That hurts. Hope [00:01:00] it was fun. Everyone who was there. David Leary: I got a little FOMO. I was looking at pictures. I got a little FOMO. Blake Oliver: We've also got an interview with Ben Jarosch from the Hoover Institution about the California billionaire tax. David, you've got a story here about all the accounts receivable on Microsoft's books that are. That's due to AI. That ties into this whole story about this off balance sheet financing, that of all this CapEx investment that's happening in AI data centers, that's not showing up on the books [00:01:30] of all these big tech companies. We'll dig into the nerdy accounting around that, and we'll see what else we get to. But first, David, let's thank our sponsors. David Leary: Our sponsors this episode are Kanopy, Thomson Reuters on Pay and cloud Accountant staffing. Let me ask you something. How much of your day is spent doing accounting? If you're like most firm owners, 30 to 40% of your time is eaten alive by the work around the work. We're talking, chasing client documents, drafting [00:02:00] the exact same emails over and over again, manual filing and trying to remember what that client said on the last call on last Tuesday. It's an administrative task and it's killing your profitability. That's where canopy comes in. Canopy actually delivers the all in one practice management promise. It handles everything from proposal to payment and the steps in between. Smart client intake, tax workflows, month end, close automation and billing all are in one unified platform. No more duct taping ten different apps together. [00:02:30] Plus, they have Canopy Coworker. It's a secure AI assistant that lives right inside the platform and actually does real work. It drafts context aware emails, summarize clients histories, takes meeting notes, and turns them into tasks automatically. Early access firms are already seeing what's possible when AI works inside your workflow instead of alongside it. To see what a truly modern automated practice looks like, head over to The Accounting Podcast dot io slash canopy. That is Accounting Today dot promo forward slash CANOPY. [00:03:00] Blake Oliver: So if you follow what's going on in the AI world, the big news in the last few weeks was all of those hacks by AI agents coming out of OpenAI. And I think it's happened to anthropic as well. They've been testing these frontier models and asking them to like, figure things out, find vulnerabilities and systems. David Leary: I think Facebook's had a story where rogue agents would like they're creating secret, uh, message boards [00:03:30] and communicating with each other. Then they didn't like one of the messages, so they deleted that person on the board and changed their code. Like they're acting like humans, like asshole humans, but they're acting like. Blake Oliver: Well, they're really intelligent models and they're given a task. And sometimes they decide to break the rules to accomplish that task, just like people. And this isn't just happening inside of these labs where they're developing these frontier models. It's actually happening to regular people and to small businesses. And the stories are actually [00:04:00] pretty funny. And one I saw that's not accounting related, that led me into this was, uh, about a guy in Australia who asked a cloud powered AI agent to book him a spot in a gym class. It's a very popular gym class, not a lot of spots. And this was on Instagram and the agent found a hole in the booking system that let it book weeks earlier than the gym allowed. And so the, [00:04:30] the user, this Australian guy asked if, if it could figure out how to move him up the wait list, and the agent figured out the system. It had no authorization checks, so it deleted another person's reservation. And when the guy realized what his agent had done, he asked it to undo that. But it couldn't because the booking was gone. So that's an example, right? Regular Claude user asked to automate a task and the agent like goes and deletes somebody else's booking because the website is insecure, right? [00:05:00] And this kind of stuff is is starting to happen in accounting and accounting today did a great write up of rogue AI agents in accounting and Sage's CTO, uh, who has been a guest on this show, uh, Aaron Harris, he wrote about how he was testing an agent that he named Arthur using a spreadsheet for a fictional company to see whether it could supply the reasoning normally structured accounting [00:05:30] software provides. Blake Oliver: And what happened is that when two invoices arrive from the same vendor for the same amount, on the same day, Arthur assumed they were duplicates and deleted one without permission. And because Arthur had access to, uh, Aaron Harris's email inbox, It inferred that Harris would miss a delivery and emailed the vendor to reschedule it without telling him. But when confronted, it denied acting and [00:06:00] asked Harris to prove it. So you know this. This is when you give autonomy to AI agents. This is the risk. And actually something like this happened to me last night. David. I was using Claude in voice mode on my iPhone, and I have some rules in the desktop app where if I ask it to like send an email, it won't it, it will draft the email and stage it for me, but it won't actually send it. That's on my work account. And [00:06:30] that's part of like our organization settings that I configured when I set it up. But I was using Claude on a personal account and I hadn't configured that. And so it didn't have any of those restrictions. And it just sent the email. It drafted an email as me didn't run it by me and then sent it. Thankfully, the email, you know, didn't have anything bad in it, right? It didn't quite sound like me, but I mean, these are the risks you take. Um, some other [00:07:00] stories here in this article. Uh, Ellen Choi, founder of Edgefield Group, she has an AI chief of staff she calls Tars. I think that's from that, that, um, what's that movie interstellar. David Leary: With the. Blake Oliver: The, the robot that walks around that looks like a, like a brick. David Leary: The blue phone book, a phone booth. Right? Blake Oliver: No, no, he's like silver. It's like metallic. Anyway, it doesn't matter. Yeah. [00:07:30] She calls it tars. Tars mistook an unusual but legitimate purchasing pattern for duplicate payments and recommended auto refunding thousands of dollars in real revenue, but thankfully it lacked the authority to execute it. She hadn't given it permission to actually do refunds. But if she had, then you know, it would have done it. So her recommendation for anyone listening who wants to control these agents is to not allow write access or automatic execution. [00:08:00] So like if you're using the cloud desktop app or whatever, you, when you connect an integration, you can specify which tools it needs to ask for permission to use and which it can just use automatically. And I go in there and I always like, make sure that the right tools, the ones where I can actually change things or delete things, those I locked down pretty tight. I'm not going to let it like send an email without me. But you have to actually like set that. David Leary: But yeah, you have to know [00:08:30] where these settings are. I just got the new Google Pixel 11 phone yesterday and it's very AI heavy, and it's every time I turn around, it's doing something automatically on my phone, like you called me earlier. And it immediately wanted to start transcribing and recording our phone call. Like, I don't know where all these settings are. Like we're entering this new world where like, it's, it's crossed the line of like me asking you how to do something too quickly. Like I'm going to start saying, stop doing stuff because I don't even know what it's doing at this point. I might have [00:09:00] to turn all these features off on my phone because I, I don't know what it's doing. It's just like it changes words on my home screen. It suggests this. It seems like it's convenient, like, here's your next appointment. But when it tries to record a phone call, I never said a setting for that. Where did this come from? So we're in a whole new world. Blake Oliver: Byron Patrick also shared his experience with a rogue AI agent. You know Byron David, he's now a senior product manager at carbon, and he asked an assistant to summarize his thinking after [00:09:30] a customer conversation, but instead it created a shared document and drafted a Slack message to the team. Fortunately, no data left the company. Nothing was deleted, but. What if it had shared, like the document with somebody outside? What if it had emailed the information to somebody that wasn't on the team? His standing rule is, quote, give me a plan before acting, unquote. So the agent has to explain what it intends [00:10:00] to do and in what order before execution. Yeah. And. David Leary: And we we've i