How to Save 50% on AI Tokens Using Cloudflare Queues

Cloudflare Q's recently hit the workers free plan and I want to show you my absolute favorite way to use them by cutting your AI token bills. Now look, look, I know that sounds kind of clickbaity, but both OpenAI and Enthropic give you a 50% discount if you use their badge APIs. If a user isn't actively waiting on an AI response, don't call the model in line. You're paying full price if you do. Drop the messages on the queue, let the queue ship it off to Entropic and OpenAI and get a 50% discount. It's cheaper. It's resilient. And let's dive right in. [snorts] Here we have a little demo app. It's a feedback classifier. We can choose between bug, feature request, praise, and billing. Let's start with a bug. We have a web app, and the user says, "The homepage didn't load on Windows. Send it to the web app." Beautiful. Okay. A sales call. The guy was really nice. Great. Another one. And see how fast this is. We're not waiting on a response. And finally, a support email. You need a haircut. Okay, let's go to D1. And here are our entries. The homepage is loaded, guys. Really nice. And they're in progress. And on the left, you see a chron job. And you'll see it's processing all three. It's in in progress. Because batch APIs are asynchronous, we use a Cloudflare cron trigger to pull for completion. Once the batch is ready, the chrona pulls the data and puts it right back into our D1 database. So, while this is turning out, let me show you some of the code. It all starts, of course, at Wrangler. And here we have our chron trigger. This means it runs every minute, which is good for a demo. In real life, you'd probably have it run a little slower, maybe every hour or so. Let's go to the implementation of the chron job. It's on scheduled. This gets invoked every minute. We do some logging. We get all the batch IDs that are in progress. And then we pull entropic. And of course, this code will be available on GitHub. So that's the crunch job. Now how do we actually get things on the queue? It's with the producer and the producer is our web app feedback q. Here you can see we have this API call. This is our API where we have handle feedback submission. We look at the feedback create an ID and then we insert it into D1. Then we put it on the queue and then we just return the JSON. That's how it was so fast when I added the feedback. We talked to our producer. We talked about the chron job. Now we need the consumer. This is the Q function on our worker and it just takes a batch and a batch is a list of those messages. I've set in our consumer here, I've set the size to three. That's why we needed three messages for it to run and a timer to 60 seconds. Whichever one hits first. If batch size is first, then the uses batch size. If 60 seconds have passed and there's only two items in the queue, then let's send those two items. And we have a maximum of three tries. So we go to this process batch mode and all it does is it gets the jobs, we turn them into requests and then we just send those requests to the entropic API. If there's an error then we retry. If there's an error more than three times, it goes to the dead cube. If it's successful, we mark the job as submitted. And then very important, we acknowledge the message here. Let's see what we got here. It is Yay. It worked. The batch is completed. We pulled. After a couple minutes, it was completed. So the web app, the homepage didn't load on Windows. That's a bug. High priority. The sales call, the guy was really nice, praise, but low priority. And the support email that I need a haircut, I mean, I know, I know, is low. So, this is cool because we're only paying half price for this. Because batch APIs are asynchronous, we use the Clifair Chron batch is ready, the chronop pulls the data and drops it in our D1 database. And if I go to our D1 database, you'll see it has upset and we have the categories, the priorities, and then a summary what the bug actually is. But this isn't just about money. It's also about reliability. LLM APIs are flaky. We get rate limited. We get overloads, transient errors. This is exactly what Q's are used for. That's why we use max retries. Let me show you again here. Max retries three. If we hit the max retries, then it goes to a dead letter Q, which is probably one of the coolest names for a paradigm. That means even if we retry it three times, if Entropic errors out, our consumer will just back off and retry the whole batch. If Entropic is down for a longer time, it will go to the dead letter Q and you actually never lose a message. Now, that's pretty crucial. You just never lose a message cuz it will either be processed or on the dead letter Q. So to summarize, if you're building with AI and the work doesn't need to happen in real time, use cues. Save some money with the batch API. Cloudflare cues are available on the free plan now. You get 10,000 operations per day. And if you're on the workers paid plan, [snorts] you get a million operations per month. After that, it's 40 cents per million operations. I hope this was useful for you and that you can save some money on tokens. I use Q's all the time. They're one of my favorite and underrated products and I hope with this video you're excited to try them too. The repos in the description. I hope you build something awesome. I'll see you later.