Building an AI accounting document agent from scratch

Okay, welcome to uh today's live stream. Um so this is the third of five that I'm doing um this week. So one every day, one hour uh for each. And today's um live stream is a accounting document agent um that's being requested um by ticker. Hopefully I've pronounced your name correctly there. Um uh the two other agents that I've built was a a community manager that um passed content into um or monitored internal systems and um drafted social media content that then could be reviewed in Slack and approved and was automated. And then the one yesterday was a um a mental health coach that um provided mental health um questionnaires and screening and ongoing support and reviews and and things like that. And today's very different again. So today's account uh accounting document agent um which I'm going to build from scratch and um if you got any questions as I go along please um throw them in the comments and I'll be trying to keep an eye on those uh as we go. So I've got a bit of a plan. Um got uh a bit of a scoping document here which I'm going to talk through I guess now. So, um the the request was basically how do we um you know, we've got an admin team that processes invoices, receipts, statements, things like that. Um a human opens them up, works out who it should go to, what it belongs to, files in the correct folder, um chases the client if there's stuff that's still missing, and then repeats that obviously over and over again. So, it's like, can we automate that in some way? So from a workflow perspective, the idea is that the client sends um uh a document, attachment, invoice, whatever it happens to be, [snorts] um the agent needs to get that in some way. It needs to process the ideally there's a bit of an audit log. It gets filed. Um and then obviously it can be processed. Now, I'm doing this as a bit of a demo. Um uh so I'm going to have to take a few shortcuts because I've only got an hour here, but um we'll work through this uh as as best we can. So, I've got a rough plan here. Um, for the purposes of the demo, um, I'm going to focus on I've got a test email account here. So, I'm going to from a different email account, I'm going to email in, um, some receipts and pretend that this is like the agent's email address. Um, and the agent will be able to fetch um, these emails, process them. Um, in terms of where the data goes, um, in the agent, um, it's possible to connect the agent to say zero. So, I could connect a zero account. Um, I could connect QuickBooks. Um, I could connect to a a database. Um, there's a lot of flexibility in terms of where the um files get stored or where the processing happens. So, that piece doesn't really probably matter too much. Obviously, that's you need to configure that based on what you're trying to achieve. But for today's purpose, what I'm going to do to keep this as simple as I can for the demo is I've got a folder, a Google Drive folder here, um, which has all the different clients. Um, and then I have two notion databases. So, I have a database of clients. Now, obviously, you know, if you were, um, if you had your own accounting software, you could talk to the API and and pull that list. Maybe you've got some other system. Um, whatever that looks like, the agent would be able to connect to it. And then one of the requirements was sort of a bit of an audit log of what the agent does. So we've created a filing audit log here as a database um uh as well. So the agent should populate the aim is the agent will populate this of everything that it does um as it goes along. So hang on a sec. I'm not sure if I was sharing my screen. Apologies for that. Let me um let me quickly just run through that again. Uh rookie mistake there. Uh so as I said, I've got a scoping document here. I have um a Google uh account here, which is going to um be my test account, pretending this is like the email address for the agent. I have this folder here of Google Drive um uh clients where we can file the actual documents as they come through. And then we have two notion databases here. This is the list of clients and this is going to be our audit log that's going to populate um as the agent processes the different information. Um what's kind of interesting is this is the the agent requirement here um is focused on I guess an accounting firm receiving these documents and processing them and that's kind of what the process is uh is a typical process right um with AI it's actually kind of possible to do things a little bit differently sometimes and this is an interesting example so a completely different way to achieve achieve the same goal could be that instead of um the client emailing stuff, the client could actually connect to an agent, connect their email address, connect their existing systems and there could be an agent that automatically just monitors all of their activity of that client and when it picks up something that's relevant for accounting purposes, it automatically pushes that across to um the accounting firm's agent. So you could have an agent to agent communication happening um or the agent could automatically talk to the accounting firm software um and populate things that way. So um that would be a completely automated way of doing this where the client doesn't have to email anything at all. Um so if I get time I might try and demonstrate how that would potentially work but um for the brief that I've got today um I'll focus on sort of the traditional um approach. Okay. So, um let's jump in and start the process of building um this agent. So, we're going to click here and click add new agent and start the um there's a little interview process here that we're going to to run through. Um for this one, this is going to be uh an agent for me for now. And we're going to call this accounting document agent. All right. So, we're going to give a higher level description of what we're trying to get the agent to do. Um, so we're going to say, uh, I've got copy and paste here from the original spec. Oops. So, we've got, um, the agent is supporting a small Australian tax and accounting firm. Client send tax documents by email. Um, I'll remove text. Obviously, we could support text. We have um uh we can support things like Twilio to pull text messages and things like that as an integration. Um but we'll focus on email for this demo. A human opens it mind, works out who is files against the right client job and chases what's missing. We want to have the agent automate this process. Um the agent should be technical and precise. I think that makes sense. Um now normally in this process I would um uh kind of continue to be interviewed by the agent here and create this initial kind of scope. Um I'm going to keep this one pretty short though. So I'm just going to answer one more question and then jump straight to um to the agent build because I want to kind of sort of do this a bit more manually. So current workflow for email that comes in with tax. So okay. So the email is received. Um it's uh the client it belongs to is identified. Um the document is reviewed to work out what type it is and where it should be filed. Um um there is an audit log recorded for each file received. Um uh that's it for now. Oh um uh yeah. Okay, we'll leave it like that. Keep it simple. Okay. Okay, for each of those four stages, walk me through the exact data points. So, I'm not going to do that. I'm basically just going to go, I'm done. I'm going to build it and I'm going to um sort of do this in a two-phase process. So, I will manually get this to work and then I'm going to get the agent to build the skills and make this a repeatable process. So, what's actually happening at the moment is um it's drafting a a plan for the agent. So it's proposed that there's a a set of skills here. So identify the client in the job, classify the document, file the document, record the document audit log, chase any missing documents, um any low confidence documents present for review, so that makes sense. Um connections, so connecting the email inbox, obviously that makes sense, which I've already got a plan here for that. The firm, so the document and filing system. Now that could be as I said anything like um you know you can configure you might have you probably got your own internal document storage system perhaps um there could be uh clients that are using zero so maybe automate just push stuff into the client zero account or QuickBooks um there's obviously different ways that that could be configured um and any other practice management system or something like that you you potentially connect as well so the agent can can work um some tools so agents can have like a custom tool So, um, these is almost like custom code, I guess. So, um, it's possible to kind of have this custom code to do things. I'm going to just remove those because I don't think we would need those the way that I'm planning on doing this. Um, knowledge base. So, let's say we wanted to have a knowledge base of the most up-to-date tax information um, or documentation or internal processes. Um, it's possible to to give that agent that insight, that knowledge. Maybe there's internal processes that's or IP in terms of how the firm can firm works. You could add that um to the agent, but for now we'll remove that. Uh and then we can do automation and loops and and scheduling. So here we've got recurring work of um uh identifying clients and jobs and chasing missing documents. Um I'm going to remove those because I want to just set them up myself manually. So um we'll go from there. So here it is. This is the agent. It's given it a name tally. Sure. Um, and we have the ability here to uh review. Those connections are set up and there's an onboarding conversation. So, the idea here is obviously this is how a um uh we're building an agent that an accounting firm could use. So this onboarding would be an onboarding for the accounting firm to set up the email addresses,