Google’s New AI Just Broke My Brain

Google made a huge announcement about their new  method that lets us run AI techniques cheaper.  The news took the world by storm.  This came at the best possible time,   because we have a worldwide memory  shortage. So the prices for capable   laptops and GPUs and anything that can run  these AI systems is up by…insane amounts. And this work would make them much  cheaper to run. They call it TurboQuant. Roughly speaking, they claim 4 to 6  times less memory, that is insane.   And 8 times faster computation for a part  of the neural network called attention.   No meaningful loss in output quality. And  it works on top of existing models as is. If true, that is a total game changer.  The news was so huge it even moved the   stock price of huge semiconductor companies. Because of that, I did not want to publish an   early video on the huge sensation.  No. I really wanted to wait a bit,   and find out whether it actually  works in practice. I’ll tell you   about that. And I’ll also tell you  that not everyone is happy about it. So, three questions. What does it do? Does  it work? What is the controversy about? Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér.   It feels so good to do it like this. Well,  this compresses the KV cache of AI systems,   like large language models. This is the  short-term memory of an AI assistant. If   you would look into that, you would see  tons and tons of numbers. These numbers   relate to what you are currently talking about.  Movies, a bunch of documents or a huge codebase. Now, personally, I am a research scientist, what  caught my eye was not the media hype, but this.   Oh! A formal mathematical proof that it works. Now  we’re talking. Okay, one, so what does this do? And these numbers have lots of digits. Scientists  propose that we chop off the end of the numbers   to save memory. Is that a new idea? No. Is that a good idea? No, unless you are very   careful. Because you can lose a lot of information  and your neural network might output nonsense. So, how do you do that? Well, imagine a vector,  this is like an arrow pointing somewhere.   Sometimes that arrow points mostly along one  axis. So most of its "energy" is in one direction,   and a little in other directions.  When you chop that information off,   it snaps on to the grid, you basically  lose everything except that one direction. That is not useful. Now here’s a brilliant idea:  before chopping it off, rotate the arrow in   a random direction. Now the energy spreads  more evenly across all directions. So when   you round off parts of it, you lose a little from  everywhere instead of everything from most places. The result? Much less information lost. Is  this idea new? No. This is a very old idea. Now they do one more thing. They use a   Johnson–Lindenstrauss Transform to  compress the data. What is that? Remember, we have a bunch of numbers,  representing arrow directions. And we   want fewer numbers to describe these  directions. But very carefully. You   do this in a way that guarantees that the  distances between these arrows is roughly   the same after squishing. If you want to sound  really cool, just call it the JL transform. Is that new? Not really. 40-year old  technique. And I think that is the   key. Everyone loves to invent shiny new  stuff. But here, quantization is not new,   rotating things around is not new. This  transform is not new. These are three   age old ideas combined together to great  effect. Sometimes you don’t need to invent   grand new theories. Sometimes you need  a smart combination of existing methods. Okay, second big question:  so does it work in practice?  To conclude that it works in practice,   I wanted to see other scientists reproduce  the technique and benchmark it for themselves.  This is why this video appears later than most  others, but I think it makes it more truthful.  So were other scientists able to  reproduce this technique? Yes.  Did they also benchmark it? Yes. Does this technique help? Yes. But, not so fast! The first tests reveal that  it decreased the memory cost of the KV cache,   short term memory by 30-40%. That is fantastic.  I would have been very happy with this. But it   doesn’t end there. Typically you have a tradeoff  where you decrease memory usage at the cost of   something. So something needs to slow down.  Now hold on to your papers Fellow Scholars,   because it also sped up processing the  prompts by about 40% as well. What? That is…my brain crashed. We get faster  AI assistants that need less memory at   almost zero cost. That is insane. In a world  where it’s harder and harder to own things,   this is a blessing. Thank you so much! It is also remarkable that the paper has  barely been out for a week and some of   you Fellow Scholars already coded it up.  Nice work. Link is in the description. So, it’s not quite like the media says.  Based on the results, we cannot conclude   that every AI machine suddenly needs 6 times  less ram. No. That is a bit idealistic and only   true for some corner cases. You know when  you see an official benchmark of a phone   battery or electric car mileage with somewhat  idealized conditions? It is a bit like that.   So careful with the media hype. Experienced  Fellow Scholars like you know that in your mind,   you have to tone these numbers down a  little. This is why we wait for more   data and analyze experiments here, to get  the highest quality information for you. But it’s still good. Really good!  It helps most people who run AI   systems with very long contexts. When you  chuck in a huge pdf document, or a movie,   or a huge codebase for an AI to analyze. Yes,  you will be able to do that cheaper, with   meaningfully less memory. Often a few gigabytes  less. And I think that is absolutely amazing news. Third, I will note that other researchers  point out that the paper overlaps with previous   techniques. They felt that it has similarities  that should be discussed more thoroughly. There   was more. Eventually, the paper was accepted  for publication, though not all researchers   agree the concerns were fully addressed. I put the  links to all of these in the video description. But this proves that even in modern AI, there  are still basic things we haven’t invented   yet. And that makes this a very exciting  area to be in. What a time to be alive! And if you agree that this is  the way of talking about papers,   please consider subscribing  and hitting the bell icon.