KVCache Meetup Korea 2026: Event Recap
The experiments used two H100 GPUs, Llama 3.1 8B, and vLLM. The benefit depends on the amount of data transferred and the available bandwidth.
The experiments used two H100 GPUs, Llama 3.1 8B, and vLLM. The slides used a 12,370-token prompt, with one request at a time and three measurements for each condition. Retrieving the host cache and generating the first token took about 92 ms. The benefit depends on the amount of data transferred and the available bandwidth.