How does INT8 quantization differ from INT4?

INT8 and INT4 quantization are methods of reducing the size and computational load of machine learning models by lowering the precision of weights and activations.

• INT8 quantization uses 8-bit integer values to represent parameters. This allows maintaining a good model quality while significantly reducing size (about 4 times compared to 32-bit float) and speeding up computations on supported hardware.

• INT4 quantization is even more aggressive — parameters are encoded with 4-bit values, reducing the model size by approximately 8 times. However, this often leads to greater loss of accuracy and requires more complex error compensation techniques (e.g., calibration, mixed precision).

Example: if a weight in float32 takes 4 bytes, then in INT8 — 1 byte, and in INT4 — 0.5 bytes.

The choice between INT8 and INT4 depends on the trade-off requirements between speed, model size, and accuracy. INT8 is more versatile and widely supported, while INT4 is more experimental and requires careful tuning.