You must log in or register to comment.
Know what uses less? No LLMs
Yay, I’m doing my part!
Try using a 1-bit LLM to test the article’s claim.
The perplexity loss is staggering. It’s like 75% accuracy lost or more. It turns a 30 billion parameter model into a 7 billion parameter model.
Highly recommended that you try to replicate their results.
But since it takes 10% of the space (vram, etc.) sounds like they could just start with a larger model and still come out ahead
There’s actually a perplexity improvement parameter-to-paramater for BitNet-1.58 which increases as it scales up.
So yes, post-training quantization perplexity issues are apparent, but if you train quantization in from the start it is better than FP.
Which makes sense through the lens of the superposition hypothesis where the weights are actually representing a hyperdimensional virtual vector space. If the weights have too much precision competing features might compromise on fuzzier representations instead of restructuring the virtual network to better matching nodes.
Constrained weight precision is probably going to be the future of pretraining within a generation or two looking at the data so far.
There is some research being done with fine tuning 1-bit quants, and they seem pretty responsive to it. Of course you’ll never get a full generalist model out of it, but there’s some hope for tiny specialized models that can run on CPU for a fraction of the energy bill.
The big models are great marketing because their verbal output is believable, but they’re grossly overkill for most tasks.
Making ai more efficient will just mean more ai
Generative AI is great if used as a tool instead of a solution.
Since I find AIs to be useful that sounds fine to me.
So?
We invented multi bit models so we could get more accuracy since neural networks are based off human brains which are 1 bit models themselves. A 2 bit neuron is 4 times as capable as a 1 bit neuron but only double the size and power requirements. This whole thing sounds like bs to me. But then again maybe complexity is more efficient than per unit capability since thats the tradeoff.
Human brains aren’t 1 bit models. Far from it actually, I am not an expert though but I know that neurons in the brain encode different signal strengths in their firing frequency.
Firing of on and off.
Human brains aren’t digital. They’re very analog.
Neuronal firing is often understood as a fundamentally binary process, because a neuron either fires an action potential or it does not. This is often referred to as the “all-or-none” principle. Once the membrane potential of a neuron reaches a certain threshold, an action potential will fire. If this threshold is not reached, it won’t fire. There’s no such thing as a “partial” action potential; it’s a binary, all-or-none process.
Frequency Modulation: Even though an individual neuron’s action potential can be considered binary, neurons encode the intensity of the stimulation in the frequency of action potentials. A stronger stimulus causes the neuron to fire action potentials more rapidly. Again binary in nature not analog.
Neuronal firing is often understood as a fundamentally binary process, because a neuron either fires an action potential or it does not. This is often referred to as the “all-or-none” principle.
Isn’t this true of standard multi-bit neural networks too? This seems to be what a nonlinear activation function achieves: translating the input values into an all-or-nothing activation.
The characteristic of a 1-bit model is not that its activations are recorded in a single but but that its weights are. There are no gradations of connection weights: they are just on or off. As far as I know, that’s different from both standard neural nets and from how the brain works.
Human brains aren’t binary. They send signals in lot of various strength. So “on” has a lot of possible values. The part of the brain that controls emotions considers low but non zero level of activation to be happy and high level of activation to be angry.
It’s not simple at all.
Multi bits models exist because thats how computers work, but there’s been a lot of work to use e.g. fixed point over floating for things like FPGAs, or with shorter integer types, and often results are more than good enough.
So, first, that’s just a reduction. But set that aside, and let’s talk big picture here.
My GPU can use something like 400 watts.
A human is about 100 watts constant power consumption.
So even setting aside all other costs of a human and only paying attention to direct energy costs, if an LLM running on my GPU can do something in under a quarter the time I can, then it’s more energy-efficient.
I won’t say that that’s true for all things, but there are definitely things that Stable Diffusion or the like can do today in a whole lot less than a quarter the time it would take me.
That said, the LLM isn’t running an array of bonus functions like breathing and wondering why you said that stupid thing to your Aunt’s cousin 15 years ago and keeping tabs on your ambient noise for possible phone calls from that nice boy who promised to call you back.
Chat GPT can output an article in a much shorter time than it’d take me to write one but people would probably like mine more
The problem is that using those tools no matter how energy efficient will add to the total amount of energy humans use, because even if an AI generates an image faster than a human could, the human still needs 100W constantly.
This doesn’t mean, that we shouldn’t make it more efficient but let’s be honest, more energy efficient AI just means that we would use even more AI everywhere.
But speaking of efficiency, a human can do more useful tasks while AI is crunching numbers. But that is very subjective.
It depends what you mean by useful. Most humans are (at least at the moment) more versatile than even the most advanced AI we have. But you have to keep in mind that there are jobs with pretty mundane tasks where you don’t really need the intelligence and versatility of a human.
Smaller and speedier means larger token windows and greater variety of models.
Not less energy.
deleted by creator
