GPU¶
聊天模型生成文本的瓶颈是**内存带宽**而不是**计算能力**，因为它必须为模型生成的每个 token 从内存中读取每一个active parameter。这意味着您每秒可以从聊天模型生成的 token 数量通常与它这个表达式成正比：
内存总带宽除以模型的大小。一个8B的模型，以 bfloat16 精度加载时，模型大小为 ~16GB。这意味着必须为模型生成的每个令牌从内存中读取 16GB。总内存带宽从消费类 CPU 的 20-100GB/秒到消费类 GPU、Intel Xeon、AMD Threadripper/Epyc 或高端 Apple Silicon 等专用 CPU 的 200-900GB/秒不等，最后高达 2-3TB/秒的数据中心 GPU，如 Nvidia A100 或 H100。这应该可以让您很好地了解这些不同硬件类型的生成速度。
In our quickstart example above, our model was ~16GB in size when loaded in bfloat16 precision. This means that 16GB must be read from memory for every token generated by the model. Total memory bandwidth can vary from 20-100GB/sec for consumer CPUs to 200-900GB/sec for consumer GPUs, specialized CPUs like Intel Xeon, AMD Threadripper/Epyc or high-end Apple silicon, and finally up to 2-3TB/sec for data center GPUs like the Nvidia A100 or H100. This should give you a good idea of the generation speed you can expect from these different hardware types.
参考¶
Chatting with Transformers: https://huggingface.co/docs/transformers/main/en/conversations