How to Train a Neural Network like a Boss

DesignCourse| 00:06:15|May 18, 2026

Chapters9

Describes a live stream of a sorting robot that uses camera input and sensors to identify a label orientation and push items accordingly.

Synthetic data plus cloud GPUs can train a robust pool-ball vision model in hours, not days, unlocking real-world robotics and apps.

Summary

In DesignCourse's latest short, the host walks through training a neural network to recognize pool balls across varied environments, using Blender renders and synthetic data. He highlights the challenge of real-world labeling and explains why synthetic data can dramatically scale training without manual annotation. The process starts with a realistic pool-table model from Turbos Squid, refined with Claude Code to produce top-down views and diverse camera distortions via ISP sensor simulations. The goal is to generate 10,000 renders that cover different ball positions, lighting, and table conditions so the model generalizes beyond a single scene. The host contrasts real-data labeling (which requires precise crops and ball IDs) with synthetic labeling, where the computer already knows the labels, speeding up data generation. To handle the heavy compute load, he sponsored cloud solutions from modal.com, spinning up nine GPUs to create and label data and to train the model, cutting a multi-day job to roughly five to six hours for about $80. After training, the model is tested in his and his brother’s environments, showing successful generalization even when images come from unfamiliar setups. He emphasizes the broader value: applying neural networks to real-world, hands-on projects makes coding more exciting and practical, beyond CRUD apps. The video ends with a nudge to creators to explore neural networks in apps that actively interface with data and perception, all while having fun with the process.

Key Takeaways

Synthetic data can dramatically accelerate neural-network training by providing automatically labeled, labeled-by-default training samples (e.g., 10,000 Blender renders with labeled ball positions).
Cloud-based GPU farms (e.g., modal.com with nine GPUs) can reduce a multi-day data-generation and training pipeline to roughly five–six hours and a small cost (about $80).
ISP sensor simulations (camera distortions, exposure, barrel distortion, focus) are crucial to make synthetic data robust to real-world variations.
Real-world validation across multiple environments (host and brother’s setup) demonstrates reasonable generalization even without retraining on those specific images.
Using synthetic data removes the bottleneck of manual labeling, enabling scalable experimentation and iteration in computer-vision projects.
Tools mentioned include TurboSquid for assets, Claude Code for scene setup, Blender for renders, and modal.com for cloud computation.

Who Is This For?

This is essential viewing for developers and designers who want to bring perception and robotics ideas into real apps, especially those exploring computer vision with synthetic data. It shows practical, repeatable steps to go from concept to a trained model on a budget.

Notable Quotes

"So many of you know I've been building this pool projection system and it's for helping you get better at pool."

—Sets up the personal project driving the neural-network training example.

"The goal is to create 10,000 renders of just different layouts where the balls are in different positions."

—Describes the scale of synthetic data needed for robust training.

"This is not a sponsored video. And it worked really well."

—Clarifies monetization and shared results, reinforcing credibility.

"Synthetic data, you don't have to do the labeling because the computer is the one who already positioned each ball."

—Key advantage of synthetic data over real data labeling.

"It would have taken days on my end, end up taking like five or six hours total, and it was $80."

—Shows the time and cost savings from cloud-based training.

Questions This Video Answers

how to train a neural network with synthetic data for object detection
why use synthetic data instead of real data in computer vision
how to simulate camera distortions and ISP sensors in training data
what is Modal.com and how can it speed up ML training
how to validate a neural network model in different real-world environments

DesignCourseNeural NetworksComputer Vision Synthetic Data BlenderBlender rendersISP sensorCamera distortionCloud GPUsModal.com (cloud compute)`,`Claude Code`,`TurboSquid`

Full Transcript

So, I'm sure most of you have seen the figure robot. There's a live stream going on. It's day five right now where it's just continually sorting these packages. And the objective for the robot is to use its cameras, its sensors, to analyze each package to ensure that the white label on it is down. It's facing down and then it just pushes it off. Now, how does that work? Well, that is basically a neural network. It uses a camera to identify a label, a white label on differently colored packages and if it's facing up, flip it over and push it off. That is a neural network and it's exactly the thing that I had to create and figure out how to create myself. So many of you know I've been building this pool projection system and it's for helping you get better at pool. You can create really cool drills and really cool games. And one of the problems that I recently solved with a neural network is being able to accurately identify each ball. Not just in my environment, that's easy. But in all environments, like look at all these images right here. These are not real images. These are actually Blender renders of different layouts of pool balls in different colors like a different lighting environments, different felt and cloth colors and also different camera distortions as well like a barrel roll, focus, exposure, etc. And so you might be wondering, well, what's the purpose of that? Well, in order for this system, this software to work in a variety of different lighting environments and pool tables and pool balls and all that good stuff, it has to be trained. A neural network has to be trained on literally thousands of images. So, I'm going to show you exactly how the process works. So, the first step involved finding a realistic pool table. If you ask Claude Code or any other AI agent to use Blender to create its own 3D, you know, pool table, for instance, it's going to look like crap. So, I went on Turbos Squid, which is a 3D asset website. I found a pool table that I liked. I purchased it for like $30, and it gives you access to all the files that you need to make it custom and unique. I fed that into Claude Code, and I told it, listen, we need to get a top-down view of the actual pull table, and we need to make it look as realistic as possible. Now, I had to do a bunch of back and forth in order to get it really dialed in. But then the next step involved also running it through what's called an ISP sensor. And an ISP is just a camera sensor essentially. And we need to simulate different types of cameras. So maybe this camera is overexposed a little bit. Maybe this one's slightly out of focus. You know, maybe there's barrel distortion, barrel roll at the edges. These are all things that we need to as accurately as possible represent in the training data so that when it comes across it in real life, it'll say, "Oh, I've seen that before." So, the goal is to create 10,000 renders of just different layouts where the balls are in different positions. Some are in the pockets, some are, you know, against the rails, all that good stuff. It has to be able to identify each one. Now, there are two different ways to approach training a neural network. You can train it on real data from like my actual camera, which I already did. I did about 50 different pictures of ball layouts in different areas. But the problem with that is you have to make sure that they're all labeled. Each ball needs to have its own crop around it along along with the accurate label like is this the one ball, is this the nine ball, the two ball, etc. That can take a long time even with autoleabeling because the auto labeling doesn't always get it right. Now, when you use synthetic data, you don't have to do the labeling because the computer is the one who already positioned each ball. It knows by default. So, it creates the labels for you. That's why you're able to generate way more training data because it doesn't take that human in a loop um in approving each label. So, generating 10,000 Blender renders on a single 4090 GPU like the like the workstation that I use here in the PC would take about 3 days. So, instead of waiting 3 days, you can just use a service. you can rent cloud compute. So, I used one service called modal.com. This is not a sponsored video. And it worked really well. Essentially, instead of having just one computer, I was able to have nine computers, nine different GPUs, all generating over 10,000 images of the Blender render. And once all the images are done, okay, you have an image and then you also have the meta data associated with that image, which says, you know, all the different labels of each ball and which ball is which. At that point, you then have to actually train the model. And this takes a long time, too. It's a it's a timeintensive process. So, I used modal.com as well to take care of that. So, the whole process, which would have taken days on my end, end up taking like five or six hours total, and it was $80. That's about it for me. That's definitely worth it. It could have been free if I just ran it on my own computer, but it was $80. Now, once the model's complete, that's where the real test comes into play. So I tested out on my environment and I also tested it out at my brother's environment. He's the first user of the software and it was not trained on any images in his environment and guess what it works. So so to me this provides such a massive unlock to understanding computer vision and neural networks. Not obviously at the very low level but a high level. It unlocks a lot of possibilities in terms of what you can actually build. There's a lot of people right now, vibe coders, they're focused on CRUD apps. Very boring in my opinion. I love infusing real world stuff into these apps because fewer people are doing it. And it's also so freaking fun. So, if you're looking to build something that's truly unique and you're not sure what to build, consider building a neural network in an app that interfaces with it somehow in some way, shape, or form because it's so freaking incredibly fun. Just a real quick video here. I wanted to share that because it was a problem that I was facing. I could get it working in my environment with my 50, you know, organic images, but to really make it work in a robust amount of environments, then synthetic data is the way to go. All right, everybody. I will see you soon. Goodbye.