Breaking the 1.58-bit Barrier for Ternary LLMs

6 comments

• om8 21 minutes ago
  Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
  [-]
  • om8 19 minutes ago
    If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS
• plqbfbv 23 minutes ago
  Very interesting, I was just exploring this to hopefully fit one of the latest quantized models in 16GB of VRAM.
• NooneAtAll3 23 minutes ago
  This is the only time "1.58 bit" phrase makes more sense than "1 trit"

  Who knew that if you actually look at information entropy you can pack stuff better!
• Kevcmk 46 minutes ago
  Woah. Good science.
• kadushka 2 minutes ago
  [dead]
• kadushka 40 minutes ago
  [flagged]