Building a Scalable Click Tracker with Cloudflare Pipelines

Hey there, I'm Craig. So, I am a developer educator and I give a lot of talks. So, you maybe have seen one of mine. If you have, can I ask you a personal question? You can be honest with me. Did you actually click those links that I shared at the end of the talk? What an awkward question. I can't ask that. But I want to know that, right? As an educator, I I want to know that I got you. I delivered to you what you wanted to keep learning. So, I built a thing for myself. It's called linked out cuz you link you send links out. I guess I actually I built this uh using Cloudflare pipelines. So, you or someone you know that might find this useful, you can use it too to host your own content pages. So, here it is. Here's what one of the pages looks like. So this is an Audi page linked out Audi that I I built for the Cloudflare data platform. Now after giving the talk, I always want people to learn more. See, it has this really nice clean URL. It's got a nice outcloudflare data platform. Nice slug. And uh often the talk is live. I can't just drop into a chat. So and share that link. So what I've done is I've made it so that if I press Q, there's a QR code. It looks pretty great on mobile. I've made it so the page kind of looks good on mobile. So, I can get a bunch of links out to you. Now, when I open this page or I did that QR code scan, uh an event was sent and uh event was sent uh onto a stream, right? So, uh, if I click into that and if I if I come here and I look at the the worker binding, I click into these links, you'll see they're going they're going pretty quick to the to jump into these docs pages that I was was thinking about. Um, and what I've done is I've made a pipeline that uh sends to a sync which is an R which goes to an R2 data catalog. Right? So there's this R2 data catalog which is a managed Apache iceberg uh data catalog uh and it's in the Apache parquet format which is pretty cool. Now if you missed this Cloudflare data platform announcement there's a really great blog post here. So come into here read all about this. You can kind of get up to speed with everything that's happening here. How we're thinking about data here at Cloudflare. Uh and again if you need that link it's in the notes and also here's a QR code and click it. I would love for you to click that action. And I'm curious how many clicks it will get. So, let me show you how I take care of that curiosity. I'll show you what the backend looks like for the creator of the Audi. So, if you just go to linkedout.dev, uh, everybody's welcome here. Anybody with a GitHub account can log in. So, if you just click this and you log in with your GitHub. I built this using Cloudflare Access. Check check the notes for that as well. So, if you need to share links and track things, please, please grab this, feel free to post and take things for a spin. So, that map, pretty rad, right? Everybody's rough geol locations of where they're clicked and I would love to see that map fill up. So, uh after I'm logged in to my Audi page, I want to show off that it has these uh analytics. So, I can click here and I can see uh popular terms that people clicked, right? Which is important to me because I'm looking about I'm looking at about these destination URLs. Good. I drove a lot of traffic over here to Apache uh to to park and iceberg which I want to see one was not a lot of traffic but also I can see what words here were here that I kind of like that there's this grouping u and I just kind of I put this out here this wouldn't be in real time but I just put this out here to kind of get a vibe of the things that are coming in of what I kind of how I want to feel like people coming through and clicking this. I'm not exactly sure what I want here and that's kind of nice right. Oh there's also an aggregation across all of them. So, I could go to go do this thing and I can go view all links and I can see how many came across of all of my Audi pages, right? Because I have access to it. I've I've gone and I've created this and I can do all of my links and I've seen what the different links of how how many people clicked the most popular pages that I've had out there. And like I said, this is just the start of my analysis. I'm I'm sure I'm going to want more reports and different slices. And the nice thing is that I'm out of the way of my transactional database, you know, like where the users are are working, right? The the real users of this app. We want to get the information to them. I don't want to get in their way reporting. In fact, the only transactional bits here are where the creator updates or edits the page. I'll I'll show you that here in a second because we want the production transactional database to be super snappy. We want it to be able to render the page as fast as possible. I don't want my tracking attempts at reporting to slow anybody down. So, let's let's look at that code really quick and see how I accomplished that. Let's see what that felt like. So, uh, in my Hono app here, right, it's a Hono app. I am in this, uh, app.get out and then slug. And I pull the slug. I get the Audi out. I come in here. I start building the things that I'm I'm going to want to do. I start building this click event. So, I have a a structure to it. Uh, and I get the properties off of the Cloudflare object, right? That's how I'm getting the location. That's where I know people are at. And then I take that and I have this wait until, which now you can import from Cloudflare workers. You just import this wait until it's off the context here. And I'm going to do this event stream which is a binding to a stream. And I'm going to send that page view event. Uh and that page view event then I use a pipeline that will write to a sync. And that sync in this case happens to be our R2 data catalog. We'll take a look at that here in a second. And I wanted to make sure before the page rendered, I'm just going to go define all that stuff and it's going to go render the page out. So, we get a page view, but we also get a track as that happens. Oh, yeah. The the editing is pretty straightforward, too. Let me show you that really quick, just so you get a a feel for that. So, if I'm back in the dashboard here and I do this manage, I can come in here and I can edit the content. This what it looks like when you create a new one, too. Uh, kind of give a theme. There's different themes that you can do, but it's just straight markdown. I just want you to be able to write straight markdown. And you'll notice that you just put the links in here, and I automatically take care of that for you, right? So I I transparently will do all of the the link tracking that you need to happen have happen there. Uh there's a live live preview here as well. Uh and let's take a look at what that link tracking looks like. Right? So I actually used AI for this transparently. I want to be uh transparent about that. I wanted to find the the latest least intrusive way. And what it showed me was there's a thing called send beacon. So if I am in my my front end here, right? So my front-end JavaScript has a file called uh track and I just do this uh send beacon and I I send it to that that endpoint. And if I look at my API, pass that into my in this routes tracking and we go to API track, you'll see that I do a very similar pattern. I build the events and here I'm not doing any logging. I just do this wait until and then I return right away because the the way that the send beacon works is you want right away you want to return and so I return it and then this thing runs and this is a really nice clear. I think this is very super clear uh out of the path writing to the stream writing that event. Let's take what it looks like in the dashboard. So, uh, I've got an event stream and that's where I'm sending the information to and I've got this events pipeline set up and that is what is connected to the sync. And so the sync is going to say, "Hey, I want to to get some stuff and it's going to pull the pipeline's going to pull off the buffer and push it into the sync." And I'll show you. Let's click into each one of these. That's probably the quickest way to get there. So, there's the event stream. And the event stream has a schema. I defined this at the beginning, but I don't need to when I when I created it. It could also be unstructured. And so, uh, that's what the stream looks like. And if we go back and we take a look here, we have the the the pipeline here. And, uh, the pipeline, we're on the pipeline page. Actually, there's a pipeline SQL. And I'm just passing it straight through into the sync at this this time. But insert into event sync, select star from event stream. I could do filtering, right? I could do validation, whatever I want in there. I could change this pipeline is in control of that. And if we go and we look at the sync, we can uh take a look at this event sync. And it has uh it it's going to a linked out uh uh data catalog, which we'll take a look at here in a second. It's parquet. It's Zstandard. Um and uh let's take a look at this this data catalog. So the data catalog itself has uh some settings. And if we look at the R2 data catalog, it has a catalog URI. So I can use that, you know, in my my whatever I want to, my Spark, my Duck DB, and uh yeah. So, and I'm not using that. I'm actually using inside my app because there's this R2SQL. Check this out. This is awesome. So, uh in my Let's do R2 SQL query. So, I I wrote a little wrapper here. It's just uh it's it's doing an HTTP call. The binding is not here yet. This is an open beta just so you know that currently at the time of this and uh I'm just writing straight SQL. It's fun. It's gorgeous. For the more orange pill of you, you probably already know that we have a tool called workers analytics engine which we offer for quick OLAP or online analytical processing. Very similar to this. You stream your event through when you need those tasks. And this is great. The workers analyze engine is great especially for times when you know what you're planning out or querying out of the gate and you need things really fast. Perfect for that. Now for me I'm not sure right now that I know all that I want to report on. But I do know that I can pass way more information than I need initially and then I can shape these reports and I can query that SQL to get the more advanced SQL statements and I can navigate this through through bigger tools. Right. Check the notes for more information. I'll document the differences and the limitations there. And in fact, I'll I'll add that to my Audi page, if you will. The link is in the description. And again, if you want to uh uh come to this page, I'll I'll add those notes here. Uh there's the QR code for you. Remember, if you want to make your own Audi pages and get your own analytics or share this with the deval people in your life that you know and love, go for it. It's at linkedout.dev. What would you like to see next? the Cloudflare data platform. I feel like I'm just getting to the tip of the Apache iceberg. The tip of the iceberg. Uh yeah, because I I plan to spend a ton more time exploring and analyzing what's possible here. Thanks so much for hanging out and we'll see you real