AI Coding Still Sucks (without validation)

What's up? CJ here. You might have seen the video I released a few months back called AI coding sucks where I talked about all the problems I was having with AI when trying to use it to code. And here we are a few months later and um I I' I've gone through phases, right? So, I took an entire month break after that using no AI at all. After that, I tried to figure out how to incorporate it into my workflow in a way that didn't suck or didn't make me hate my job. Um, and so I I've learned a lot since then. And so in this video, I want to talk about some of the things that I've learned, talk about areas where AI coding still sucks and then maybe give you some guidance as to how you should go forward go forward in this world where AI is everywhere and you're being forced to use it. Um, and really the the the biggest thing and the main thing that I want to get across and the main thing that I found is that tests are more important now in your codebase than they ever were. And and here's why. So let's talk about it. So first of all, AI writes code at 100x speed, right? It can just poop out code all day long and that's fine. But one of the main issues is with that all of this hype comes around focused on the output, right? So sure, it can it can just produce so much code and it can produce code all day long. And so people have opinions that oh because it can produce so much code, we don't have to care about code quality, you don't have to look at the code ever again. Just prompt your way to to glory. Um but really when you you take something like AI that's producing code at 100x you can compare it to the old days I guess say let's like a few years ago where Joe Schmo copy paste Stack Overflow developer could come along and could copy paste and glue libraries together to get things working and they were employable right they could do that but they're not moving at 100x speed, right? They're they're capped at the amount of Stack Overflow that they could copy paste to turn into somewhat working software. And so that's the old world. In the new world, we basically have this AI that can do that, but it can do it at 100x speed. And if you're not validating its output, and if you don't have systems in place to validate its output, you're basically 100xing yourself to the point of extreme technical debt or to the point of not being able to maintain your codebase anymore. And so that's the key insight is we're too focused on output and we basically have replicated the worst part and the worst kinds of software developers in this in this world of AI coding. Um, and and it's really interesting to think about because before, like I said, you could have someone who was somewhat decent at assembling the stuff together, copy pasting, not caring about code quality or not having static analysis tools or not having really good test suites and basically just just fumbling their way into success. And we're trying to replicate that now with AI. And that's basically the a lot of the code that AI was trained on. And so that's why we're ending up with outages and technical debt and and basically systems that are are no longer trustworthy because we've basically handed over the keys to AI without actually validating its output or without actually having systems in place that can validate the stuff that it puts out. So with that in mind, this is where the idea of verification systems or tests or static analysis come into play and why they're more important than ever because um I've been testing out a lot of different models. So I use cloud opus 4.6 probably mainly and then I use uh GBT 5.4 for planning. Sometimes I'll use GBT codeex 5.3 for coding. And I have this little uh local AI machine back there. And I've been testing out some local coding models like uh Quinn 3 coder and Quinn 3.5. And what I found is that these models have been trained in a way and basically they've had feedback loops as in their training that causes them to want to validate things. Right? So even on this little machine back here using Quinn 3 coder I could give it the most basic prompt and it searches my codebase. Do I have a llinter installed? Do I have tests that I can run? And I say I I'm speaking from from the perspective of the AI but essentially the the way that it has been trained is it is it wants to validate itself. It wants to make sure that the code that it wrote compiles or the code that it wrote type checks or the code that it wrote passes all of the tests. And so that's a very interesting behavior of these new models that we should try to exploit because if you're using AI in a codebase that doesn't have those checks and balances, right? That doesn't have a really good test suite, that doesn't have static analysis tools or type checking or anything like that, AI has no way to attempt to verify itself. So it's just going to write some code and say, "Hey, I'm done." And so my main point here is because the models are trying to do this. If you set them up for success, if you have a codebase that has a really good existing test suite that has static analysis tools like llinters and formatterers and also potentially like security review static analysis tools, those kinds of things, AI can run those in its loops to essentially validate itself. and you're going to end up with code quality that's much better than AI that's just going off and pooping out 100x code. And so with a really good test suite, AI will write the code, it will run the tests, fix the failures, and then keep on iterating. And what I found is if you have this in place, it'll try to do it automatically. Now you can uh supplement that with decent prompts and skills and potentially like sub agents that are who they're whose sole job is to write tests or to to run the validations because that's really one of the issues I was running into months ago. I would prompt and and have basically like an agents.mmd or a claude.mmd that would describe hey you need to run the tests for everything or you need to make sure the llinter passes for everything but eventually the context would get bloated and would forget a lot of those things. So what I'm finding is with these latest models, they're much more reliable at actually running the checks and you essentially need to have gated checks. So you need to have a prompt that says, "Okay, implement the feature. Make sure all of the existing tests pass and make sure you write tests for the new code that you have implemented, right?" And so with this iteration, your your test suite starts to grow and grow. And if you're at least validating the test because the test like there it's possible to have bad tests, right? It's possible to have tests that test for the wrong thing. And so that's that's a whole other video. That's a whole other can of worms. But if you can make sure that your tests are validating the right things and are essentially proving the software that you're writing is what you're trying to build, AI AI is going to have a much time working within those guardrails. Um, and so one of the main issues I had and and what I talked about a lot in in the last rant I did on AI coding was it it felt like my job had changed from uh being able to like implement these systems myself to just being an AI validator like like what what was my role other than to just make sure that the code output was decent. And I'm finding with these new validation loops and kind of like shifting my thinking, I'm no longer the code validator. I'm more the design validator. So, I make sure that the thing that AI is building is the thing that I actually want it to build. And I make sure that the things that we're testing for is the things that we care about. Um, and so with that shift in mindset, um, at least for me, I'm I'm having a better time. So, I have at least six different projects I'm working on right now that have had heavy design phases where I didn't write a single code for a week. But this is another area where AI has been useful for me is kind of as an ideation tool and and as someone to bounce ideas off of. Um because AI is really good at taking the status quo, right? All the code that it's ever been trained on and just spitting that out. But if you combine this ability to generate code with um basically just like being able to think outside the box or combining other ideas from other places in the world that may not be programming, it is capable of coming up with interesting new ideas. So I'm actually building a framework, a full stack framework from scratch. And like I said, I spent a week prompting with AI just talking about, okay, here's what exists and here's what I don't like about it and how would we change that? and it would come back with with several ideas and then we could iterate on those ideas and before actually implementing code, we were designing interfaces. We were designing how these things connect together without any code to make sure that it made sense from a design perspective. And so that type of iterating with AI for me has been a huge unlock. There's been so many ideas I've had over the years that just over the past two months, I've been able to iterate with AI to really flesh out those ideas. things that were just like a seed of an idea in my head that I'm actually putting onto paper now. Um, and so that entire design phase, like I said, no code. And then we we go into the coding phase, but we make sure all of the systems are in place that I talked about earlier, like we have tests, we have linting, we have static analysis, we have feedback loops. There also skills you can use. So, Centry has a uh skills repo, and there's a bunch of other people have skills repos. basically just markdown files with instructions for the AI, but I specifically use the code simplifier skill and the code review skill and the security review skill. So, anytime AI is implementing a feature, it makes sure all the existing tests pass. It writes new tests and tests of all kinds, too. I I'm not going to get into it in this video, but I'm going to do a separate video on how to write good tests and different types of tests, but you've got unit test, you've got integration test, and end to-end test. and you need to make sure that you have a a good suite of all of those for AI to validate itself from. Um, but it'll essentially write its new tests, make sure the test suite pass, and then before committing any changes, it goes back and does the code simplifier skill. And so this removes duplication. And as I found recently, there are static analysis tools that can find code duplication um in a consistent way. Like you don't need to prompt AI for it. They're essentially CLI tools that will index your codebase and find common patterns. And I found that that is really good input for AI as well. So simplify the code, reduce duplications whether or not the AI finds the duplications or the static analysis tools find the duplications. So that iterates that keeps the code clean and then um we do a code review which makes sure that the code aligns with the the uh practices of the codebase and then we do a security review and the uh security review skill that is in the Sentry repo has all types of things to check for whether it's front-end code or backend code. Um, and essentially with these checks and balances, like all of that happens before a commit actually happens. And so, everyone is hyping up and touting like how much faster you can move. I'm finding that yes, I'm moving faster, but I'm also being more methodical. I'm AI is basically forcing me to be more disciplined. And that's a good thing, right? Because even without AI, a codebase with a really good test suite and really good static analysis tools is going to be much better to work in than one that has just been slopped together, whether it was by a human or by an AI. So this this is basically um my my current mindset, my current state of things. And in some upcoming videos, I'm going to talk about my workflows. I'll show you some of the things I've built. I'll show you some of my massive test suites. That's I don't know if that's something to to brag about, but yeah, I have test suites with literally 6,000 tests. Um, and I have uh test rubrics and test validation because again, like I talked about earlier, you have to have good tests. And so I have actually worked with AI to go back and forth and come up with a rubric that says like what makes a good test. And every time we write a test, we run it against this rubric to say like are we catching all of the edge cases? And then a lot of times that rubric will cause AI to discover like flaws in the design, things we didn't think about. And instead of just guessing, it will come back to me with questions and be like, "Hey, this is what I discovered. What should we do about it?" Right? Instead of assuming how to implement it or or skirting around certain edge cases I've prompted my AI in a way that we're basically a team now. And instead of just going off and doing whatever it wants, it'll come back to me and be like, "Hey, I found this. What do you think?" And and I have found that iterative flow works a lot better. So all of that to say like AI does allow me to move faster but in different ways and it's not the 100x code output that everyone is touting. It is just being more methodical. Um it's uh spending more time in the design phase and having tons and tons of checks and balances in place. Um now you might argue okay but do we still need AI for all of that? Probably not. But like I said I I'm able to iterate in the sense of design and and making choices faster. not iterating faster in the sense of code because AI running in a loop to write the code and write the test and do force phases of validation and come back with questions that takes a long time, right? So, I'm still it still takes a while. It's not it's not cra um but it is forcing me to care and write better software. So, that's where I'm at with all of this. Let me know down in the comments if you have a similar experience. Let me know down in the comments if you think I am now an AI shill because because my mindset has shifted so much. Um, but yeah, like I said, videos are coming on my testing strategy, on how I've been using sub aents, how I've been able to code remotely, how I've been able to design bigger systems, how I've been able to code in languages that I'm not super familiar with. Like I'm writing a desktop app with Rust, I'm writing some performance evaluations in Python. Um, all this stuff I've written before but I'm not super familiar with, but AI allows me to move a little bit quicker. But if you add on all of the those other validation layers, I'm attempting to do my best at writing good code in these languages that I'm not super familiar with. So that's all I got for you. Let me know what you think and I'll catch you in the next one.