Casey Destroys Optimization Myths | TheStandup

If you guys do not know, chat, chat, everybody in here, Casey, trash, the startup is going through a bit of a change, all right? There's been some investment interest. And we are now raising a series A like any proper multi-billion dollar unicorn startup. Cool. And so therefore Decacorn. Decacorn, really? We're a decacorn? And so Yeah. I don't know what that means yet, but someone told me it's like that's the best kind of unicorn. I think it means it has multiple horns. I don't know what it means. 10 10 horns. 10 horns. Okay, nice. So it's kind of like the the beast in the Bible, the devil, right? Very cool. have 10, does it? Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah. Uh anyways, I'm sorry. Fact checkers? Fact checkers? Fact checkers? And so with that teacher who will fact check that one. I actually thought TJ was going to fact check me, but he fact checked me incorrect. It didn't seem 10, but that it could be, could be. Bunch of you know who's read Revelation. What the hell's going on here, guys? Hold on, hold on. Well, that is kind of what they what it answers, you know. [laughter] Anyway. All right, so he has 10 horns. Called it. Okay, primary reference, I got it. Okay, I was I was correct on that one. Okay, good. Referenced. All right, so with that in mind, we for our theology. All right. Just let it happen, everybody, okay? My gosh. All right, this is not starting off the way I wanted it to. So anyways, nor does any standup. All right, so with all of that said, that means as part of our amazing content, I have now organized us an amazing linear board, which will help us walk through a more organized standup, because if there's one thing I know about in raising a series A is they want to see agile, okay? I'm pretty sure that's correct still in today's day and age, so we're getting really agilized. No, it's actually going to be it's going to make us hopefully have a more organized and amazing standup, so that the world's most attended standup, which is our current standup right now, right now, just in case you're wondering. You know, you may not know this, Casey. There's There is There's 2,500 people on the stand-up right now. This is the world's most attended stand-up. Oh. It's like you couldn't even like it we previously no one knew if that many people could even stand at the same time. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. We shattered records. Okay. Oh, yeah, definitely. What was the record before? Like 100 people, maybe? Yeah, like 100 to 200, I don't know. Maximum. Now we're at 2,500 is with world It's It's world shattering. Yes, totally. It's a game changer. Yeah. I want to add, though, we didn't get an invite or the topics until this morning, like 2 hours No, yeah, so this This stand-up's going to be amazing cuz cuz Prime's like, uh we're going to talk about this new Well, you know what? I won't spoil it, but let's put it put it this way. It probably was a topic that would have been nice to research beforehand if we're going to talk about it. That's That's all I'm going to say. I don't want to spoil anything. I had to listen to Jim listen to YouTube videos about it. Would have BEEN NICE. WELL, YOU KNOW, THAT That makes this authentic, uh Casey. You get a react to Trash's workout gym knowledge. Okay. Right. Rather than that, Trash is the expert. You get Trash back. I am not the expert. Trash is the expert. Trash is the expert. I was like, "Oh, no." Dude, if there was a trash fact every episode, that would be That would be fire. It's It's literally going It's going on the linear board. I literally have endless lore since then. Um another thing that I would like to to uh ask at this point is if we are serious about this series A, which I think we should be, uh is it time for us to start putting together a pitch deck? Oh, wow. don't know what one of those are, but I think we need to do that. I think you're I think you're going to do it. We need some slides. My startup, dailybjj.ai, is going nuclear. Congratulations on product market fit. Sorry I called you the Diffeler. I knew you weren't the Diffeler. I always thought it was the intern with the weird glasses. You should have seen his code. But anyways, if you want to get in the ground floor of this thing, we're taking angel investments. So, you finally cracked the case, Merge Cop. You put the white space in the diff. You made the synthetic traffic. You made me approve that PR. You're the diffler. I ALWAYS KNEW IT. [screaming] YOU'VE MERGED YOUR LAST PR, DIFFLER. I already clicked merge on my PR 35 seconds ago. Huh? My merge is blocked. CURSE YOU, CODE RABBIT. YOU SAVED THE CITY from a code review-related crime. Thanks, Commish. Well, technically, it was Code Rabbit that saved the day. Not only do they have advanced AI features that can detect security vulnerabilities, like what the diffler was trying to merge, they also have ways to enforce styling, [music] linting, and a variety of other tools. And so, you can stop wasting time reviewing code that humans don't need to review. You can try it yourself at coderabbit.ai. So, Merge Cop, I would say it's a little misleading to say you did it all by yourself. Huh. Would you like some cake? Oh, sure. Thanks. All right. So, Casey, I saw this tweet and I figured you could maybe give me like a little I think I understand the purpose here and what the problem is of it. But, I thought maybe you could help me a little bit on this one. This is my blocker, by the way, for the day. Which is I saw this where someone said, "Hey, don't use division. Use reciprocal math instead. Can you can you can you can you maybe give us like us dumb [clears throat] people why this is either good or bad idea? Oh boy. I have no idea what that means. Well, right here he's doing array index I divided by pi versus array index I multiplied by the reciprocal. So, 1 divided by pi. So, he does the division thing once and then hits it with a bunch of multiplications. Yeah. Um So, there's a lot of things that you would want to talk about here and that's I actually saw that tweet that you're talking about. Yes. And unfortunately your account? No. I mean, see here's the here's the problem, right? So, delete your account. I love delete your account and I love what Ryan's doing there. But what mostly what he's doing there is he's just like and he even described this on one of his own streams. He's just saying like look, basically like you guys are all the people who like come up and and and beg for change on the street or something. Like you're just totally making the world difficult to live in by like trying to get all of like just just hounding everyone who goes into a particular area to try and get them to give you something, right? Which is your attention or whatever. They're they're like these AI shill accounts or whatever. People trying to get engagement. They're trying to get follows, whatever it is, right? And it just really degrades the experience of being in that place. That's basically what he said on like one of his streams. I'm probably phrasing it poorly. Um I I think he he said I think he said the phrase "If you go to a third-world country like France." Was the WAS [laughter] LITERALLY I THINK the phrase he used. I love Ryan. in that stream. So, I don't know don't don't blame me. Don't come after me for this. I'm just repeating what I believe it was. Um But anyway, the point is just that uh that was sort of the point of delete your account. And I And I'm He's right. Like, there's just all of this kind of, you know, really bad behavior that's going on there. And the uh this tweet I don't really like being harsh on people who are trying to talk about performance because I'm glad that we somehow went, you know, uh over the past, you know, 10 years or so to people actually starting to think that it matters that you're thinking about how the CPU is going to do things. So, I don't want to like be mean to anyone who's at least trying to say like, "Hey, let's think about what this is doing." The problem is that it requires some actual knowledge to like actually get these things right. And a lot of times, if you just ask the AI or if you just read something on a blog and you then repeat that, it's not accurate, right? Because performance is a very specific thing. It's It's kind of like repairing a car or something. You can't just know like one thing about like, "Oh, yeah, you know, the carburetor or something." Right? It's like, you have to actually know how the whole thing works. Right? [laughter] Carburetor. car is short for. No, me exactly. Like, I'd use it I'm using that as as an example. Like Like, I'd just be like, "Uh the spark plug." Or Right? You know, it's like, I don't even really know much beyond like there's a thing that ignites the gas, right? Then okay, that's it. I couldn't tell you like is it broken? Is it working properly? Does it need to be tuned? What's going on? Right? Spark plugs are for electric cars, right? Cuz you get it from the battery. I mean, that's There you go. That's Must be. I mean, clearly. So, yeah, this particular tweet is not is not good though because it misleads people, I think, on a lot of points. I'm not sure if the person actually tested this. They've got like seconds listed there, which is a little weird cuz I don't The Those numbers um I I mean, there's there's a lot of things that could have happened there. I don't know. So, I don't know the circumstances of it. But, I just don't want people to be misled. When you're looking at a loop like this both of those loops that are in that tweet, cuz I remember this. Of course, I can't read what's on prime screen cuz Riverside won't let me, but I remember it fairly well. You're basically going You're going through arrays and it's basically a read-modify-write, right? It's It's loading a value out of array. The value is a floating-point number. Um loading a floating-point value out of an array. It's modifying that floating-point number and writing it back to the array, right? And uh the operation that's being done on the array, the only difference between the loops is one of them is doing a divide, so it's loading, dividing, and rewriting. And the other one is loading, multiplying, and rewriting. Correct. Now, obviously, in math if you do a uh divide of a number, if you say, "I'm going to divide by X." There's no difference between that and saying I'm going to multiply by one over X, True. That's true in math, meaning in like in grade school, you write it down. If someone said, "Oh, you know, A divided by What's sorry? Okay. All right. All right. [laughter] TJ was a math major, so okay, he's flexing that he knows these things, okay. Math major, get out of here. Yeah, TJ. This is This is like in in like junior high, right? You Or earlier probably, Yeah, yeah, totally junior high, yep. Yeah, totally. Come on, guys. LIKE SERIOUSLY, LIKE I AM NOT GOOD AT MATH. I AM NOT GOOD AT MATH. IF I know this, you know this. Come on. Yes, Casey, this is true. This is true and and straightforward, yes. Um this is just something people did in grade school, right? And so uh the idea, this was certainly true in the old days um for computing, right? Is Look, if I have floating point, right? If I have the ability to use floating-point numbers. Integers, obviously, if we're using integers, then you have to do a lot of much more crazy stuff if you wanted to divide instead I mean multiply instead of divide. Go into that cuz that's a whole 'nother ball game. But if assuming you're in floating point, then that means that you could literally just store the number one over X, whatever you're going to divide by, and then multiply by it instead. So if you're going to do a bunch of multiplies, this is potentially an attractive option if dividing is slow, right? So if we look at divide and we go the dividing's too slow, we could just do it once and then multiply if the if the multiply is faster. Now, it's worth noting that these are not necessarily the same operation because in floating point, right? An actual divide, this divided by that, right? May produce a different answer because you don't know like when you do it an inversion like that, you're only capturing part of the answer. Like the answer might have gone on for quite some time and you're only getting a smaller part of it. So you're not doing infinite precision math and so you also have to be aware of the fact that this will change the answer. So you have to you have to know, right? If that mattered to you or not when you're doing this. So if it's just some random thing you're doing for some UI element or something like that, then it probably will never matter. If this is scientific computing, it could be a very big difference. And so this is the first thing that I want to point out is that this is not a thing you can just tell somebody always invert your numbers and then multiply because that may be very bad advice in certain circumstances. That's thing one that wasn't mentioned in the tweet that you have to understand. The butterfly effect. Uh yeah, well, it's yeah, basically like, you know, Jeff Goldblum with the water going down his hand effect. It might even just be the Good movie. the the the 1,000 mile butterfly. Like it's this huge butterfly because it could be literally it could just be just literally the answer you get right here could be wrong enough that you care, right? Depending on the circumstances and depending on what those input numbers were and what their scales were and all this stuff, right? Casey, can you just I there is a decent chance a few people in the audience don't understand why that happens on a computer. Well, so to be fair, I'm the wrong one to explain it because I don't work on scientific computing, right? And people who work on scientific computing are much better at analyzing things like, you know, what's called ULP or basically like the last what happens to the final bits in floating point numbers and things like that. But in general, when you're working with floating point on a computer, what's actually happening there is that you're taking the size of the storage that you, you know, made for those numbers. So let's say that you're doing 32-bit floating point numbers. In scientific computing it might be 64-bit numbers, whatever it is. If you just have 32 bits, you only have a certain amount of space to store what you would prefer was an infinite series. Like let's say we want to compute with pi. Right? Everyone knows from grade school pi is this infinitely long series. Like there is no storage that can store pi. It just keeps going. So we have to approximate that, right? By putting it into some format. And all the numbers that we work with in floating point computing, they have this kind of format. Just something simple. you for a quick second? Trash, when he says pi, he's not talking about a snack. Just in case you're wondering. I don't even like pi, actually. I actually hate [laughter] pi. Me neither. Trash, yes. There's another reason Trash is the star of the stand-up. No pi zone. Yeah, that's true. Yeah, what's your favorite pi adjacent kind of thing? Like is it cake or brownies or like what is it then? Love brownies. Pi is not adjacent to cake and you'll take that back. We can We know we just got to move on cuz that was very offensive. Okay, that's it. Put that next next episode. I'm putting it in the backlog. How adjacent How adjacent [laughter] is cake to pi? Okay, so anyway, if you if you look at what happens to these things during all your computations, every computation that you do is going to have some infinitely precise answer that's the answer you actually wanted and that you will want you would have wanted to carry through all of your subsequent operations, but at every step of the way, the computer has to like round it basically to whatever will fit back into that 32-bit encoding. And that 32-bit encoding is basically, you know, it's a sign bit that says whether the thing's positive or negative. It's some number of bits for the exponent, right? Which says I think is the sign. is the next thing. Oh, the next thing. That's right. That's right. Uh the exponent basically says what the scale is, like how big roughly is this And then the mantissa are the bits that actually say like what the number is. So, the reason it's called floating point is because it's broken up into these pieces. Floating point, the point is the decimal point, if you will. The floating part is that we store an exponent, so we say where the decimal point is, and then the mantissa is just the actual digits. So, we're basically floating the point to where it needs to go using the exponent, right? And the reason we do this is because we're trying to compute with extremely large and extremely small and everything in between, and so we can't just take 32 bits, which has a very small range if we just use an integer, right? If we fixed point, if we just said the decimal point is here, it would kind of suck. That's what floating point's for. It's why we use it for everything that isn't just nice, you know, typical whole number kinds of stuff that we're doing, right? You know, usually when we when we use uh just answer things like that, just integer numbers. The common The common one that people laugh at, right, is it's what 0.2 + 0.1 in JavaScript, and you get the 0.299999, right? So, it's like this is This is why that happens, though. So, just I feel like there's probably a decent amount of people who are like just haha JavaScript stupid. I mean, you're right, but like that's not that's like a right for the wrong reason JavaScript problem. Exactly. That works in C, it works in Java, it works in all of them. yeah, JavaScript is weird in that it kind of only has double precision arithmetic period. Like it's just like that's just what we do everywhere, I think. Yeah, unless you use bitwise operations, then it converts to a signed 32-bit No, it will just magically at that time one moment you can actually do a you can actually do a a overflow at 32 bits. Only if you use uh bitwise stuff. Again, this is always why JavaScript compilers are so hard, right? Like cuz they have to go like, "Oh, I don't you don't want to do double precision arithmetic for like your loops and like counter and things like that." So, it has to analyze and go, "Okay, I can turn this into an integer for this code." Right? It's got to do that work. Normal languages, it's already knows, right? It's just been told like this integer doesn't have to think about that, right? But anyway, so that's that's the floating-point math thing. And that's sort of what this this uh this snippet was trying to say is just invert the floating-point number so you only pay for the divide once, then multiply by it cuz that'll give you the same answer. Again, the tweet didn't say no, it's not quite the same answer, and it might matter depending on what you're doing and depending on what the inputs are. But anyway, uh the other problem I have with this is it misrepresents the performance as well for a number of reasons. So, the first reason it mis- misrepresents the performance is because it has some weird statements about how long a division is. So, it said something like 20 to 40 cycles for a divide or something like that, I think. Yes, 20 to 40. And I just have no idea where that's coming from, right? Like I've no idea where they got that number from. They like I really wish it had said where it got that number from, right? came from his heart, all right? It came from his He could feel it. He could feel those cycles. The CPU whisperer. So, a floating-point divide um you know, if you look So, if you look at a modern chip, like let's say a Zen a Zen 4 or Zen 5 chip, right? Something that you might be running today that might be in a computer you'd buy today. Typically, their floating point divide units are extraordinary. Very fast. I want to say that they can complete these things in I don't know, five, six cycles, something like five cycles, maybe. I'm not sure what it is. We should probably look it up. But, 40 is no no where to be found. I mean, that's it's not quite off by 10, but it's something like 10. Like, it's very very wrong. I think it's from uh ChatGPT cuz if I'm not mistaken, I actually asked this question not too long ago and I got the answer of like eight to 30 or something like that cycles for a divide depending on the CPU and architecture. I mean, the part the part that the ChatGPT got right is the depending on the the architecture part cuz it [laughter] does depend. Okay, okay. Uh but like let's sit here, I'll just look it up on my machine right now. Like, we'll just see what roughly it is. So, if I go in and I say I want to do a, you know, 32-bit um uh like we're going to do a div PS here or a div uh SS, let's say. And we look at what's going to happen on a Zen 5 chip, um the the total latency, right, is less than 10 cycles for for that operation, latency, right? So, that means that if you about multiply then? Sorry? What what about for multiply? Multiply will be four. I want to say, right? Um so, three three on the Zen 5. though, Casey? We just need to know how many [laughter] FPS's 10 cycles. Right. If it were a bouncing ball Casey, how fast is that ball? So, three on that and I think four four is on my CPU, which is Zen 4. I think it should be four, right? Uh is that correct? Zen 4. Also three. So, all right, this is better than I would have expected. I'm used to four, these get three, Now, so in general, that is the number that you would use if you were trying to analyze how long it takes you to get the answer back, right? These loops don't care how long it takes to get the answer back because nothing is waiting for that, right? On the iteration of the loop, as you go through, right? You're going to do a load, the floating point op, either divide or multiply, and then the store, When you do that, the next iteration of the loop doesn't have to wait for that to complete. So, those are just in the scheduling queue going, they can be dispatched. The next iteration of the loop is already decoded and probably also in the scheduler, given this is a hot loop, presumably we're just running about, you know, It's good looking at that. It's good looking, for sure. It's [laughter] just cooking. So, these are all going to pile up in the scheduler, scheduler's going to dispatch them every cycle as it can. And so, what we care about is not how long it takes to get the answer cuz no one cares. What we care about is how long will it take us to issue the next one. So, if we issue them on this cycle, how long does it take us to issue the next one, right? And so, that's what we typically call that's a throughput number. So, we typically call that, right? And if you look at the throughput numbers for multiply, right? Typically, you can issue two floating point multiplies per cycle, which means it takes half a cycle effectively to multiply. Half a cycle, okay? That is what you That is the number you should be listing or thinking about when you're looking at this loop. For divides, the throughput again is remarkably good on modern CPUs, so it's like in the two to three range. So, is multiply faster than divide? Sure, right? Half a cycle versus three cycles when we're analyzing this loop, let's say, or something along those lines. But then you have to ask the question, do you care? This is a load a op and a store. Unless you're entirely out of the L1 cache, which is going to service, you know, at about, you know, like let's say, uh one, you know, it's probably like two per cycle, three per cycle, it could get you answers back from the L1 cache, right? If you're talking about the L2 cache, then, I mean, I don't know, the number of clocks it's probably going to take 14 cycle latency, something like this. If we're looking at how long it takes you to actually get the memory back, it's not clear to me that unless the unless the CPU was predicting extremely well what you were going to do and always had things, like you'd have to go look at the actual bandwidth to see if you were actually going to get enough back such that you could operate them on them at that speed. Maybe you can. It's it's not in my brain right now as to whether you would actually be able to do that at a speed that would matter at the like, let's say, three cycles, uh throughput number that you're going to get on the divide. So, it really just it was very misleading, the whole thing, and as I you know, you heard me say all that stuff, I would want to go look at this loop and and check it first, and I'd want to think about it a little bit before I would post anything like that, and it was just that clearly wasn't done. Because the none of those numbers make sense that are placed there. Is it true that multiply is faster than divide? Yes. Does it even matter in this case? No. I don't think it probably does, but maybe. And furthermore, usually, in most cases, When you're looking at actual loops, more is going on. And so, it's also going to mislead people. So, point number three, it's also going to mislead people in this final way, which is to think that they always have to do this. It's the This is the problem with these kind of like out of context performance things. People read it, they think, "Oh, I just need to invert all my numbers, and then my code goes fast." And then they go around inverting all of their numbers and turning them into multiplies. And the problem with that is in a lot unless you were unless you had serial dependency chains that really could have been unblocked by this change, you're just wasting your time. The divide would have been just as fast. It would have overlapped with other things you're doing in the loop. You were The thing was waiting on a on an uncashed read or something, so the whole loop was was waiting 80 cycles anyway, so none of this mattered, right? So, the problem again is that like performance is a thing, performance is a process. It's an understanding. You have to understand all those things that I just said. They're not actually that hard to understand. You just You can't You just need to learn them. You need to spend a few weeks learning this stuff. And then you can think about do I need to invert this number? Do I not need to invert it at that point. What you don't want to do is try to package it into a tweet like that when you haven't really done the work of understanding why it matters or anything like that because other people will take away the wrong lessons. So, that was a very long-winded way of trying to explain everything I didn't like about that tweet. Again, no offense. Sorry to the person who posted it. It's just I don't think that kind of thing is helpful. It's sort of like saying always call memset or never call memset or these other weird performance things that people pass around, which out of context don't help anybody because you're just going to make incorrect decisions if you think that the key to performance is always do X or never do X. That's not how it works. Spark plugs. Great job, Casey. Thank you. I did my best. I figured you'd be able to help me on that one because I had the rough idea that it would introduce error, but I had no idea to the extent that divide and multiply may not make any sort of actual real difference in your program. Plus all the crappy things I do anyways probably destroy my program's performance long [laughter] before we get to this loop. Yeah, Prime, we've got other issues before we get to whether we're multiplying or dividing, I think. Yeah. I just really want to take my Lua program seriously, TJ, okay? It's It's usually I'll The one thing I can put in someone's head that's generally not wrong is Don't confuse integer multiply with floating-point multiply. Floating-point multiplies are much faster than integer multiplies typically. And so some of the time when you see people saying, "Oh my god, in like divide is so bad." Sometimes they're thinking about integer divide. And there's a lot of reasons There are a lot of reasons why integer divide is bad. A lot of reasons. to put that on the block list for next week. You can We can We can talk about that cuz that's actually a That seems very interesting. Hey guys, if you like this episode, you can watch the rest of it on Spotify. And don't forget to like AND SUBSCRIBE. WOO! SEE YOU LATER. [singing] Five code errors on my screen. [music] Terminal coffee Living the dream.