Replacing Kubernetes with stateful Edge functions, Daniel Troger, Depict AI | Immerse Stockholm 2026

Hi guys. Um, while you all settle in, do I have a vote? Uh, it's a menty presentation, so maybe you can all uh scan this QR code and uh answer the question uh what's the most expensive cloud bill mistake you ever made? Can you guys hear me actually or should we turn it up? Okay, excellent. So while you're answering this, I'll be telling you a bit about myself and depict where I work. So my name is uh Daniel Croger. Um I work as a tech lead at depict and um what is the picked? Depict is a company making like personalization and merchandising for e-commerce. Um I uh started at depict in 2020 as an intern back when it was just an Usyka, Oliver Edmol, and Looney Sale. And uh you might know Antonica because he went on to found lovable. Uh I've worn a lot of hats uh at the pic since I was intern. And uh yeah a lot of or some some of my former colleagues are here too today. Um so what what does the pic do? We do like if you go on an e-commerce page um and you open the search box the search results could be powered by the picked and also the recommendations when it says like you might also like that's probably also the picked. And then lastly, merchandising is this kind of art of sorting collection pages. So if you go to trousers on like Zara or something, um then it turns out that they care a lot about the sorting of these trousers because most people will look at the top 10 and not at all the 200. Um yeah, damn. We have uh a lot of people who who've spent some money here on the mistakes, but uh it's all good. That's where you live, right? Um, and we have four people above 100K USD. That That is kind of crazy. [laughter] Who are you guys? Damn. All right. Negotiations. Yeah. Okay. Yeah. Yeah. We'll get there. So, in the next uh 20 minutes, I'm going to tell you about how we move the real production workloads off from Kubernetes onto the Cloudflare edge. uh what nearly killed the project and then we're going to do two deep dives into specific problems that I hope is interesting for you guys and answer your question if it was all worth it. So we essentially blew uh $123,000 uh US dollars on uh um cloud mistake. Sorry. And this was from our uh 250k startup credits and yeah we'd actually do it again. Um so why did we want to move our stuff onto the edge on Cloudflare? Um essentially we had been building 8020 for a lot of years and that meant just uh pushing the solutions out of there like shipping as quickly as possible because we didn't know if it would actually provide value and it was totally the right call because most most things that we built uh got deleted. Um but then ultimately luckily for us we found the product market fit. So people wanted to buy our product and then our infrastructure um started crumbling which led us to this architecture sorry rearchitecture. Uh we have three products that they picked. Um we basically started with the first one but we're going to focus on the middle one. I'll I'll think is anyone from here about e-commerce? Oh okay. Okay. [laughter] Anyways, [snorts] I think uh it's not a lot of you, so I I won't go into the products more, but uh what we were doing here in uh the like merchandising product that I'm going to focus on today uh is that we're syncing the products from Shopify into our database. Um so essentially Shopify is like a database where people have their products and we need a mirror of this that has at most like 2 minutes latency and um the best solution architecturally would be to just avoid doing it. But since we already had built our product around it, we were kind of sunk cost fellows in. So our old solution uh running in Kubernetes was like a homemade uh queue and it was doing non-incremental syncs which meant that every 3 hours we were downloading all the products of like 1,500 stores and uh that's like a lot of data because uh the biggest store has 165,000 products. So we're like constantly just just churning through this data and because this was non incremental it was super slow. So sometimes it took six hours after someone like created a new product for it to show up in the picked. Um our biggest customers threatened to churn uh because the syncs were so slow. We made like a last ditch hack to give them priority above everyone else. Uh but it still took 14 minutes to get any change into the picked and they turned which is super sad because now we have this awesome system on Cloudflare. But yeah, apart from the customer impact, one major problem for us uh was or for us developers was the database. So we have this database in Google cloud and usually the storage is like hovering here at 72 gigabytes and uh one Saturday uh super annoying it was a Saturday it was just growing and growing you can see it on the graph here and uh a colleague called me like around here somewhere uh we were trying to figure out why is the database disk space just growing uncontrollably and it turns out it's because of the um MVCC so the multi-version concurrency control uh which is like a performance part of postgress So in Postgress when you um delete data it doesn't actually get deleted. It's still in the table. It's just kind of marked as deleted. And uh the reason for that is that you should be able to write and read data at the same time. So a reader kind of always has a snapshot of how the database look like at the time they did the read. But what we were doing is um two things actually. The the Q system uh had like locks for a really long time. And then also because we're deleting and writing all the data for a really long time, we got to kind of drown in dead pupils which is like this deleted data. Um and uh yeah the database couldn't keep up uh with the vacuum. So that's uh where we had this problem and also we had the web hook handling uh with like Kubernetes pods and they couldn't scale fast enough. So it was basically every week an engineer would get an alert that we can't handle the web hooks and it was kind of just waiting uh and then would solve itself. So it was a dire situation um like uh for for us with the infrastructure and uh also some issues with Kubernetes because the talk is also about that. Um personally I think the the worst part about Kubernetes is that if you don't set up like affinity or a pod disruption budget um node upgrades will give you like brief outages of your service and uh for me who doesn't really care about infra I just want my code to not go down. Uh it's super confusing why why you have to like put in effort for the service to stay up. So um it was time for something new and basically I I want to clarify I don't think like Kubernetes is bad. It's just for most use cases it's over complicated. um it's inefficient. Our GCP bill is still way too high even though we moved most of the stuff out of here and uh once your infrastructure person lives and you're just like developers um then it's very expensive to actually fix the mistakes. So we wanted to keep it simple and uh cloud for uh looks promising but I want to ask you guys uh who here has been woken up by an alert from their infrastructure. All right, we have a few people here then it's great that you're uh at the cloud conference. Um yeah so we didn't actually like bet code moving all this to Cloudflare. Uh we progressively tried it out. We tried with a growth hack. Then we moved like a serverside rendering of the front end of one of our products to Cloudflare like the lovable guys. But then we actually went uh further and uh moved the like data ingestion. So the the syncing and processing part of the data to Cloudflare workflows and I think that's kind of uncharted territory. Uh most people do like routing or or server side rendering on Cloudflare. uh like on the edge, but we actually went with the whole data layer. Um so what did we build? A quick overview. Um this is the merchandising app where we moved large part large parts to Cloudflare. You can see we still have a lot of stuff running in Google Cloud. Here um it's essentially the front end where our customers log in and drag around the products or set sorting rules. Um, we left that in GCP because it's just a simple fast API server and there's not a lot of users. There's the 1,500 merchants. Um, and that has worked fine. But the part that we have moved is all the like data syncing because uh a store can have up to 165,000 products. So, uh this is where we have to like handle a lot of load and a lot of data. Um yeah, and this is just the cloud part. Uh you see it's a lot of durable objects and stuff. Uh I provide durable objects. Who here has a durable object in production? Uh four people not a lot. Okay. So for those of you who don't know, maybe it's uh like a tiny stateful uh function with its own database. Um yeah. Okay. So now let's do the deep dive. Um and uh this is a very hard problem that cost us like a month of iterating before we were able to solve it on Cloudflare. And uh it's a story of betrayal because every obvious answer is broken in like a non-obvious way. Um what is the problem that we're actually trying to solve? So as I said, we need uh the sync to from Shopify within 2 minutes. But more specifically, we need to know what products are in which collections. So like trousers are in trousers and t-shirts and t-shirt and so on, right? You think it's dead simple. Shopify could just send a web hook when like the t-shirts collection changes and then you know uh this product has been added. But uh the catch is that for automated collections which are like rule based. So for example, if you have a collection that has like on sales products, there would be a rule that if the sale price and the original price are different uh then it's on sale. Um so for this sale uh sorry uh rule-based collections, Shopify doesn't give you a collection update web hook. The only thing that you actually uh get to know is that like product Y has had its price changed and then you have to figure the rest out yourself. Um and that means on every like product update web hook. So when any product changes we need to get more data from Shopify and this is the graphical query that we're getting essentially uh for this product we have to fetch all the collections that it's in. And then we only want to know the position in this collection but we actually have to fetch fetch all the products in the collection because there's no way to just get the position. And so if you multiply this app up we actually batch this. I'll talk more about the batching later, but we might fetch up to 2k products. Um, and the product, every product might be in up to 1k collection and every collection might have up to 165k products. And that's like 330 billion [laughter] like connections to download. And so you might be thinking now like um it doesn't workers have 128 megabyte memory limit. Um, and you would be right. It does. So we had to solve that somehow. And uh the naive approach is you just stream it. So the result from Shopify is like JSONL. You would think like uh so you see every color is a collection and every block is a product. So you would think okay uh we just um stream one whole collection and then at this point we like flush to the database. We get the next one we flush to the database and that way the most we have ever in memory is 165,000 products. Uh we built it. It like worked great in the test but it turns out that in uh like 5% of cases Shopify randomly return products out of order. So you would get like a product from collection A and then suddenly a product from collection B and then it would go back to product from collection A. And that means that you can never flush to the database. So you need to know everything to process like anything. Um and we figured it would be super um nice if we could increase the memory limit. But before we did that, we found there's a flag in Shopify which is called the group objects. Um and it's supposed to like sort this. And uh we had to go through three levels of support because it didn't work. and they just said it's like intended behavior. [laughter] Yeah, very annoying. So then we did the limit increase form for Cloudflare and uh we begged for more RAM. Uh but uh they said no and uh I thought like uh we're cooked. Um I thought Cloudflare is for to-do apps like you seen on Twitter probably like oh I built a to-do app on Cloudflare but you can't actually build products. Um but then Andrea from Cloudfare who was actually working on the workflows and also was in a call uh where we asked about the RAM uh he came off mute and he had this uh super nice idea. So we would essentially stream every JSON line into one row in a table in a durable object and we could have one durable object per workflow. Um and it was uh super nice because the durable objects scale indefinitely. So uh we could just like have one per workflow, right? And um also then you can just query the durable object. So you can say give me uh the lines where collection equals a and then you get everything back sorted without using any RAM. Um so we shipped it and also I I was asking about the cost and he was saying like Amina knows um and uh I was kind of getting the impression it won't be an issue. Well, you can see it coming already. Um we were checking the billing dashboard uh religiously like every day and uh at the end of the month we were at $529 and uh yeah given what we pay for GCP this was very very cheap. Um, yeah, and we had our our target 2 minutes, so we thought this was solved. But then actually I got an email and it's really good that I opened the email because it was the actual bill and the actual bill was $121, uh,399. So this was the the cloud mistake and uh, we didn't see it coming. Um, so this is this is actually more than my yearly salary. So, it was quite quite a shock to get this and it was uh really great to uh how how Cloud handle it. Uh you guys promised you would top us up up if we ran out of credits and we got on a call with uh the workflows team and asked like uh you saved us once, can you please save us again? Um and uh they gave us advice to do less work. Oh did my Okay, no, my claw just broke here. But uh essentially we did less work. So we um checked the updated that time stamp and that brought the uh cost down to 6,000 per month. So you can get from Shopify like when when the data actually changed, right? Um it was still not cheap enough. Um so we tried to get desperate. Yeah. And the idea was essentially to um kind of as you diff these indexes only if an index actually starts to deviate then you like extract that change and push that to the database. Um it didn't work. it uh or I mean it kind of worked um but the code became unreadable and it surfaced one crazy limitation with durable objects which is that just by making the SQL query in a durable object more complex you might be able to overload the durable object. So this worked like in development but in production uh the durable object suddenly had to do like more CPU work and they got overloaded and you can see this PR that I've reverted like five times because you just merge it and then you see oh uh durable object overloaded in production then you reverted and you like optimize your SQL query and then you try again um yeah but essentially we had to ditch durable objects it didn't work so the next idea was to um do temporary tables in Postgress um if you pin a connection then you can create a temporary table when you drop it or actually when you disconnect it gets automatically dropped and there's no dead pupils. Uh we shipped it. It worked perfectly in dev again. Um but then in production customers started reporting that all the products disappeared from their collections and I did went on like a frantical debugging journey and uh the crazy thing u that turns out to be happening was that SQL pin doesn't work on cloud for workers. So uh you guys have this pooler called hyperdrive and uh when you imagine it's a it's a sort of race. So if you imagine workflow A creates a temporary table and it's empty because it hasn't gotten to do any work yet and workflow B has just created it temporary table and written all the products. Now when workflow B wants to read back its temporary table, it might swap the table with workflow A which is like totally insane. And then it sees like oh it's an empty table. And then it goes like okay all the products were deleted. I'm going to delete them from the database. and uh the customer doesn't get any products. So um yeah, we raised it with the hyperdrive team and they were super empathetic which was great and they said we're going to make it fail loudly and update the docs. Uh the docs have still not been updated but [snorts] I I hope it fails loudly at least. Well then we did the next iteration and they was basically doing unlock tables in Postgress. They're like faster and if you drop them you get no dead pupils. Um Postgress took it like a champ until we got like a lot of web hooks and what happened then was that we got the dead pupils again and the funny thing is like uh we thought okay maybe it's the this dropping these tables does create dead pupils but it doesn't the problem is every time you create a table postgress writes one row in an internal table that this table exists and if you delete the table it deletes this row and that's one dead pupil and we were creating and deleting tables so often that like PG class uh started dying uh which was crazy and Then um we did the bucketed instead. So we have one table per 10 minutes. And the highest I've seen so far is 41 million rows in a 10-minute bucket. Uh so it's like 4 million rows per per minute. Um pretty insane for just temporary data that you sort through. Um and this worked great except that we were still like writing and then reading and then writing again. It was too slow. So finally we have like one query which uh basically does everything inside the database and it's fast enough. Uh so that's super great and also around this time cloud for fixed the billing dashboard. Um so last month we were at $3,000 US which is very manageable and you see there's still some like cost savings here that were kicking in. Um so this month we're extrapolated at $1,800 which is a lot cheaper than GCP. Um unfortunately GCP is building stuff is like super unreadable. Um so I can't tell you how much cheaper but like I would say from two to 10x somewhere. Um yeah sir. Yeah so in summary it was like uh six six layers that betrayed us and now it finally works well. Um it's probably the only solution that you can get if you want to have Shopify and Cloudflare and this uh requirements. Um also some brief mentions here about the like Cloudflare reliability. Um in the start of this we had a bug with hyperdriver 50% of workflows just couldn't connect to database. Uh but you guys at Cloudflare fixed it in like one two weeks. um which was great but the the weeks were painful. Um and then also we had the one workflows outage uh where you let you know on like Friday evening and it was fixed Sunday morning. So I think uh yeah it's it's fair uh for a weekend fix. Um and but the funny thing was when the workflows product was down uh I think engineers at cloud like deleted the workflows manually and our code was expecting them to always like settle into a state. Uh so now our code is handling like in case someone clever deletes the workflows and lastly unfortunately the workflows are not fully reliable. Uh so there's one thing which is retries in the workflow. Um but then there's one thing where workflows just randomly fail and we have like a 0.018% failure rate due to like Cloudflare issues. Um we can actually recover from that now most cases but it will still be great if you fix it. So I'm using this for a bit visibility. [gasps] Um yeah, but this is what like uh moving your data processing uh production to to cloud looks like. Uh this was the first deep dive. Let's let's speed through the second one. I hope you guys are still with me. Good. So uh this is um related problem. Uh you know the web hooks, they don't have enough data, right? We need to get the collection memberships and and other things. Um on Shopify there's an admin API has rate limits. If you would just every time you get the web hook, you would get more data. You would just get instantly rate limited. Now the thing is there's um another API called the bulk operations API and it has no limits except that you can only do five concurrent queries and when it's done you get the web hook and this is also one thing uh with the Kubernetes based ingestion it was basically impossible to use this because it's really hard to route the incoming web hook to the right pod where you're waiting for the work and also your pod could get like evicted while it's waiting or something. Um but in Cloudflare it's super easy because a workflow can just pause and then um you can have a durable object uh which takes all the web hooks and resumes the right workflow. So this was uh something that was like hard to solve previously was now like 200 lines of code. Uh that was really good but uh now we had this uh way of super easily starting the bulk operations and then we did the nave approach of just one web hook equals one bulk operation and uh to make it scale we needed the cues. Um we actually went to 300 cloud for cues. Each Q has 250 max concurrent invocations. So that gave us um 75,000 concurrent invocations and 9 terabytes of available RAM. Uh which I think no one else would give you that cheaply. But uh it was still like queuing for too long. So we need kind of architectural solution. And the solution is to think about it like a train station. So if you uh think about the five slots in Shopify like a five train tracks um and every bulk operation needs to use one of these tracks right there can only be one train on each track and then um the idea is that the one durable object is essentially like a train station. Um and then any incoming web hook while the tracks are full just jumps on the train which means it goes into the table of a durable object. And then once a track uh is free then you get all of the ids that are batched in the durable object or in the in the train and uh they depart simultaneously into one big bulk operation and uh that gave us an insane amount of throughput uh of data that we get from Shopify and that's how we can get the 41 million rows from the Shopify API essentially um per 10 minutes. Um yeah and one super cool engineering important thing here is that this is like a positive feedback loop. it kind of self-stabilizes um because if you get a web hook storm like you get a lot of updates um then the bulk operations become slower because we need to get more data but as the bulk operation gets slower um the kind of train is waiting longer in the station and because the this traffic is mostly people buying the same products it's like many people buying few products uh because of that then you can duplicate while the train is waiting so if two people buy the same product before the train has departed we only one update and uh that way we can like dduplicate away a lot of the work. Um and that means the harder you can uh spam us with web hooks, the better we get at managing them, which is super cool. Um yeah, and and that way basically the constraint that we had with B operations became a feature. Um okay, what is it like to live here? Um I think yeah, there are a few pain points. the 128 megabyte limit um is annoying because uh if you could just add more memory then we could have solved the JSL streaming thing at iteration one just give us 5 GB we have everything in RAM boom um I think generally like building on cloud is unapologetic you can't do quick fixes so when your durable object is overloaded or uh you run out of RAM you can't just like go in and add more and then you can solve it the next week but the good part of that is that you don't end up with infrastructure where you have moved everything to next week. Um, and then also one downside is that you can't really observe these durable objects. Imagine you had a web server and you couldn't see the memory or CPU usage. Like that's durable objects. Um, you never know if like your SQL query is going to take them down, which is not good. Um, yeah. So, I would say in summary like Cloudfare makes it easy to build complex systems, but it's still hard to build them way well and get like the the last edge cases out of the way. Um but the wins are also pretty good. Yeah, you can see that. Um so basically you don't need any infrastructure people anymore, any Kubernetes nerds. Um like as an app [laughter] like uh if you're an app engineer and you can write code, you just need to understand the cloud for primitives which I think are a lot easier than Helm charts. Uh I've gotten those explained multiple times but I still don't get it. Um and also a super cool advantage is like the customer isolation. Um if we have one big customer where everything like needs a lot of resources because on Cloudflare everything's running like in another data center around the globe the the resources taken by one customer like don't make it slower for everyone else that's using depict uh which is amazing also because you can parallelize massively you can run I think like 50k concurrent workflows and with workers it's unlimited. Um because of that our ingestion time of doing a full sync which we still do once a week has gone down from three hours to uh 20 minutes like less than 20 minutes for most people say one minute. So that's like 10 times faster. Um and we're only paying when we actually use this capacity which is different between other providers. Yeah. So basically we if we just have to scale the database if we get more requests now um and and the rest is handled which is great. Um summing up I would say that the closest analogy is cloud is like an EV [laughter] there's a a higher upfront cost um but when you have everything built up and it's working in the abstractions um especially also because you're using streaming you can just handle like uh any volume of uh products or or customers for that part because the your blog you also have per customer right um and yeah now everything has been running smoothly for the the last time which is great so I would say you're betting on a trae Tory and it could be worth it. Um, would we do it again? I would say yes. If I like started the company tomorrow, I would build on workers. Um, I would not miss Kubernetes, but also I'd be probably reaching out to Cloud occasionally about like bugs and and limits and stuff. Um, yeah. So, that's the talk. Happy to take your questions. [applause] Thank you so much. Any questions? Nope. Yeah. I just want to say brilliant presentation. Thank you. Yes. Yes. Yes. [applause] Observability is number one request. engineering definitely I mean the lack of observability in dos is so painful yeah for sure and then you end up using so much uh resource in the do on logging or pushing to log stash or something so yeah and there's many optimizations that of course you would expect that you would get in your graph query out of um out of Shopify you know on that initial uh query but yeah as you say intended behavior [laughter] Um but how much how much uh sharding of the dos like how were you using interdo communications using sockets between them? No, we generally have uh like one dribble object per merchant kind of per task I would say. Um and then there's just uh the workflow which is running everything which is communicating with the dribble objects. And did you call D1s from dos? Uh no, we do not use D1. We use hyperdrive and then Postgress on uh Google cloud still. Yeah. So I was thinking about using D1 as a cache from your DO. Like when you got out of order, you could have used D1 there. Oh yes, but I think the right pricing per right to D1 is similar to durable objects. So we would still have the 121,000 per month bill. Right. Right. Okay. Ending. Brilliant talk. Thanks. Yeah, I'm not sure if this is like an inappropriate question because we're at the Cloudflare events, but I'm just wondering like with moving so much of the infrastructure onto Cloudflare, do you have any like concerns about I don't know like vendor lock in and that sort of thing or any like ways to mitigate it? Yeah, I mean it's a good point. I think if it came to build a cloud with like 10 extra prices overnight, [laughter] what we could still do is either run the wrangler development environment in Kubernetes. It would probably be a bit rough. Uh but technically would work or we could just write like a workflow wrapper that runs the same code essentially. Um so I think there would be a way out. It would be painful, but given the benefits you get, I think it's still like worth the bet for today.