Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation.
However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Secondly, the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time.
> None of this would matter if it changed the model's answers
If they want to assert that the answers don’t change, then perhaps they should calculate the statistical distance between the token probability outputs or something to that effect. I doubt the results would indicate that the answers don’t change by any reasonable interpretation.
vLLM's study also concluded that "FP8 can deliver meaningful latency and capacity gains with small or negligible accuracy loss". Their benchmarks include LiveCodeBench 6.
I was interested in reading this until my slop detector went off at the paragraph starting with “It's worth being precise about where the benefit comes from, because it isn't raw speed.”
I have had to stop commenting this because it would end up on 50% of the posts here. I really wish we could flag prose as ai-generated on here and just filter it out.
Next up, LinkedIn starts using this feedback to train a classifier. They then announce an officially approved "not slop" classification only for LinkedIn Gold member posts. The classified posts have a wider reach due to everyone filtering out AI slop. Non-members automatically get bucketed in with the slop bc they don't pay to have the verified classifier run on them.
Cloudflare blogs are not meant to be human-read, AFAIK. They're raw material meant to be fed into an agent to be filtered down. I rarely read the contents because they are usually word-expanded to a greater degree than an article from The Atlantic.
I don't disagree, but at some point in the last year they ended up severely word-expanded. So in a revealed sense, they are no longer meant for human consumption except for those who don't significantly value their own time. There is very little information in the post that an agent can't pull for you:
* they use quantized models
* they quantize KV cache
* they have a cache tagging mechanism to prevent cache misuse (neat)
The agent can extract numbers without filler prose as well.
However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Secondly, the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time.
> None of this would matter if it changed the model's answers
If they want to assert that the answers don’t change, then perhaps they should calculate the statistical distance between the token probability outputs or something to that effect. I doubt the results would indicate that the answers don’t change by any reasonable interpretation.
Maybe the results are still good enough.
https://vllm-project.github.io/2026/04/22/fp8-kvcache.html
Why… I wanted to see if it’s worth it to use cloudflare’s endpoint but I can’t even see the pricing
I love AI, but I really hate reading it.
How well it would work on this site, I'm not sure.
* they use quantized models
* they quantize KV cache
* they have a cache tagging mechanism to prevent cache misuse (neat)
The agent can extract numbers without filler prose as well.