NVIDIA on Jiang Yi(姜祎)'s Homepage

megatron-lm

Wed, 21 Jan 2026 18:04:12 +0800

Introduction

随着模型参数变大，现有的 GPU 已经很难使用单一 GPU 来训练模型。对于多 GPU 训练场景，目前主要采用了 pipeline parallelism, 比如 GPipe 等，但是，这些策略需要我们对代码进行比较大的改动，这提高了开发成本。

为了解决多 GPU 训练大规模 LLM 的效率，降低开发成本，目前主要使用了 model parallelism 策略，即对模型进行切分部署在多个 GPU 上。model parallelism 有两种范式：

pipeline parallelism (PP): 将模型按照 layer 进行切分，如 GPipe 等，这种方法的问题是需要额外的逻辑来处理通信以及存在 pipeline bubbles
tensor parallelism (TP): 将模型的按照权重进行切分，部署在不同的 GPU 上。

作者在本文中基于 TP 策略来对 attention, FFN layer 进行简单改动来实现训练效率的提升。

作者通过实现验证了 tensor parallelism 的有效性和高效率，结果发现在 512 张 GPU 的场景下，TP 可以达到 $76\%$ 的 scaling efficiency (相比于 1 张 GPU 带来的性能提升)

Method

作者使用的 transformer 架构如下图所示

本文中，作者探究了 BERT 和 GPT-2 两种架构。

首先，我们假设 transformer layer 输入为 $X\in\mathbb{R}^{bs\times d}$, 这里 $b, s$ 分别为 batch size, sequence length, 接下来我们介绍如何针对 FFN, attention 以及 embedding 构建 TP 策略

FFN

论文中使用的 FFN 为 Linear-GeLU-Linear 的结构，对应第一层权重为 $W_1\in\mathbb{R}^{d\times d_{ff}}$, 第二层权重为 $W_2\in\mathbb{R}^{d_{ff}\times d}$, 对应数学表达式为

$$ Y = \mathrm{GeLU}(XW_1)W_2\in\mathbb{R}^{bs\times d} $$

我们首先对 $W_1$ 按照 column 进行切分，得到

$$ W_1 = [W_{11}, W_{12}]\in\mathbb{R}^{d\times d_{ff}}, \text{ where } W_{11}\in\mathbb{R}^{d\times d_1}, W_{12}\in\mathbb{R}^{d\times d_2}, d_1+d_2=d_{ff} $$

这里 $d_1, d_2$ 与我们并行的 GPU 数 (x-way TP) 相关，这样，我们就有

$$ \mathrm{GeLU}(XW_1) = \mathrm{GeLU}(X[W_{11}, W_{12}]) = \mathrm{GeLU}([XW_{11}, XW_{22}]) = [\mathrm{GeLU}(XW_{11}), \mathrm{GeLU}(XW_{12})] $$

从而我们可以分别将 $W_{11}$ 和 $W_{12}$ 部署在两个 GPU 上，然后并行计算。

论文中还介绍如果我们对 $W_1$ 按照 row 进行切分，则最终由于 $\mathrm{GeLU}(A+B)\neq \mathrm{GeLU}(A)+\mathrm{GeLU}(B)$ 计算时会产生一次额外的同步。

接下来，对于 $W_2$, 我们按照 row 进行切分得到

$$ W_2 = \begin{bmatrix} W_{21}\\ W_{22} \end{bmatrix}\in\mathbb{R}^{d_{ff}\times d}, \text{ where }W_{21}\in\mathbb{R}^{d_1\times d}, W_{22}\in\mathbb{R}^{d_2\times d}, d_1+d_2=d_{ff} $$

计算时，我们有

$$ \mathrm{GeLU}(XW_1)W_2 = [\mathrm{GeLU}(XW_{11}), \mathrm{GeLU}(XW_{12})]\begin{bmatrix} W_{21}\\ W_{22} \end{bmatrix} = \mathrm{GeLU}(XW_{11})W_{21} + \mathrm{GeLU}(XW_{12})W_{22} $$

可以看到，通过按照 row 进行切分，我们可以将 $W_{11}, W_{21}$ 部署在一个 GPU 上，将 $W_{12}, W_{22}$ 部署在另一个 GPU 上，分别计算出 $\mathrm{GeLU}(XW_{11})W_{21}$ 和 $\mathrm{GeLU}(XW_{12})W_{22}$ 之后，再通过一此 all-reduce 操作得到最终的输出结果。计算图如下所示

这里 $f$ 和 $g$ 是两个对偶算子，代表了 TP 产生的额外通信开销

operator	forward	backward
$f$	identity	`all-reduce`
$g$	`all-reduce`	identity

如果说我们使用的是 SwiGLU FFN, 即

$$ Y = (XW_3\odot \mathrm{Swish}(XW_1))W_2 $$

我们按照 column 对 $W_1, W_3$ 进行切分，按照 row 对 $W_2$ 进行切分（假设我们有 2 个 GPU），得到

$$ \begin{aligned} W_1 &= [W_{11}, W_{12}]\in\mathbb{R}^{d\times d_{ff}}, \text{ where } W_{11}\in\mathbb{R}^{d\times d_1}, W_{12}\in\mathbb{R}^{d\times d_2}, d_1+d_2=d_{ff}\\ W_3 &= [W_{31}, W_{32}]\in\mathbb{R}^{d\times d_{ff}}, \text{ where } W_{31}\in\mathbb{R}^{d\times d_1}, W_{32}\in\mathbb{R}^{d\times d_2}, d_1+d_2=d_{ff}\\ W_2 &= \begin{bmatrix} W_{21}\\ W_{22} \end{bmatrix}\in\mathbb{R}^{d_{ff}\times d}, \text{ where }W_{21}\in\mathbb{R}^{d_1\times d}, W_{22}\in\mathbb{R}^{d_2\times d}, d_1+d_2=d_{ff} \end{aligned} $$

然后我们将 $W_{11}, W_{31}, W_{21}$ 放在第一个 GPU 上，将 $W_{12}, W_{32}, W_{22}$ 放在第二个 GPU 上，此时，

$$ \begin{aligned} \mathrm{Swish}(XW_1) &= \mathrm{Swish}(X[W_{11}, W_{12}]) = \mathrm{Swish}([XW_{11}, XW_{12}])=[\mathrm{Swish}(XW_{11}, \mathrm{Swish}(XW_{12}]\\ XW_3\odot \mathrm{Swish}(XW_1) &= [XW_{31}, XW_{32}]\mathrm{Swish}(XW_1) = [XW_{31}\mathrm{Swish}(XW_{11}), XW_{32}\mathrm{Swish}(XW_{12})]\\ Y = (XW_3\odot \mathrm{Swish}(XW_1))W_2&=(XW_3\odot \mathrm{Swish}(XW_1))\begin{bmatrix} W_{21}\\ W_{22} \end{bmatrix} = XW_{31}\mathrm{Swish}(XW_{11})W_{21}+ XW_{32}\mathrm{Swish}(XW_{12})W_{22} \end{aligned} $$

这样我们通过一次 all-reduce 也可以完成 SwiGLU FFN 的 tensor parallelism, 示意图如下所示

Attention

Attention 的处理与 MLP 非常相似，论文中的做法就是将不同 head 部署到不同 gpu 上分别进行计算，最后在计算 output projection 时再通过一次 all-reduce 来合并输出，这里我们假设有 $h$ 个 heads, 每个 head 的 dimension 为 $d_h$, 我们先对 query, key, value layer 的 weight $W_Q, W_K, W_V\in\mathbb{R}^{d\times hd_h}$ 进行切分

$$ W_Q = [W_{Q1}, \dots, W_{Qh}], W_K = [W_{K1}, \dots, W_{kh}], W_V = [W_{V1}, \dots, W_{Vh}] $$

其中 $W_{Qi}, W_{Ki}, W_{Vi}\in\mathbb{R}^{d\times d_h}$ 为每个 head 对应的 query, key, value weight. 我们将切分后的 $W_{Qi}, W_{Ki}, W_{Vi}$ 部署在一个 GPU 上（也可以将若干个 head 部署在一个 GPU 上），然后分别计算出每个 GPU 的 attention 结果，最后再进行汇总，如下所示

$$ \begin{aligned} o_i &= \mathrm{softmax}\left(\frac{(XW_{Qi})(XW_{Ki})^T}{d_h}\right) XW_{Vi}, i=1,\dots,h\\ O &= [o_1,\dots,o_h]W_O \end{aligned} $$

下面是 multi-head attention 对应的 TP 示意图

Embedding

对于 Input embedding, 作者将 embedding matrix $E\in\mathbb{E}^{V\times d}$ 按照 row 进行切分（论文中使用了转置，因此是按照 column 进行切分），得到 $E=[E_1,E_2]^T$, 这里 $E_i\in\mathbb{R}^{d\times V_i}$, $V_1+V_2=V$, 接下来我们把切分后的 embedding matrix 部署在不同的 GPU 上，由于每个 GPU 只有部分结果，因此我们还需要进行 all-reduce 来进行汇总。

而对于 output embedding, 我们也可以使用类似的做法进行切分，每个 GPU 上计算完结果之后我们还需要一个 all-gather 来汇总结果。

作者在这里还额外介绍了针对 output embedding 的优化方法，由于 embedding 的输出大小为 $[bs, V]$, 而 $V$ 通常比较大，因此，为了降低通信开销，作者将 cross-entropy-loss 与 output embedding kernel 进行融合，这样我们传输的数据量就减少到了 $bs$.

Experiments

作者首先对 GPT-2 模型进行了修正，首先将 vocab_size 从 50257 提升到 128 的倍数，即 51200. 对于 model+data parallelism, 作者固定 global batch size 为 512. (64-way DP)

配置如下表所示（head size 为 96）

Hidden size	attention heads	layers	parameters (B)	TP	TP+DP
1536	16	40	1.2	1	64
1920	20	54	2.5	2	128
2304	24	64	4.2	4	256
3072	32	72	8.3	8	512

对应的 scaling （使用多卡训练后，每个 GPU 相对于单卡训练的利用率）如下表所示

parallelism	TP-1	TP-2	TP-4	TP-8	TP-1+DP-64	TP-2+DP-64	TP-4+DP-64	TP-8+DP-64
scaling	100%	95%	82%	77%	96%	83%	79%	74%

Implementation

首先是 linear layer 的 TP 版本，如下所示

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


import torch
import torch.nn as nn
import torch.distributed as dist


class ColumnParallelLinear(nn.Module):
 def __init__(self, in_features, out_features, bias=True):
 super().__init__()
 self.rank = dist.get_rank()
 self.world_size = dist.get_world_size()

 self.local_out = out_features // self.world_size
 self.weight = nn.Parameter(torch.empty(in_features, self.local_out))


 def forward(self, x):
 out = x @ self.weight

 gather_list = [torch.empty_like(out) for _ in range(self.world_size)]
 dist.all_gather(gather_list, out)
 return torch.cat(gather_list, dim=-1)


class RowParallelLinear(nn.Module):
 def __init__(self, in_features, out_features, bias=True):
 super().__init__()
 self.rank = dist.get_rank()
 self.world_size = dist.get_world_size()

 self.local_in = in_features // self.world_size
 self.weight = nn.Parameter(torch.empty(self.local_in, out_features))

 def forward(self, x):
 x_local = torch.chunk(x, self.world_size, dim=-1)[self.rank]

 out = x_local @ self.weight

 dist.all_reduce(out, op=dist.ReduceOp.SUM)
 return out

接下来是针对 LLM 中使用的 SwiGLU FFN 进行的优化，基于前面的介绍，我们不需要对基于 column linear 进行 all-reduce, 代码如下所示

SwiGLU

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


import torch
from torch import nn
import torch.nn.functional as F
import torch.distributed as dist


world_size = 1
rank = 0

class ColumnParallelLinear(nn.Module):
 def __init__(self, in_features: int, out_features: int, dtype = None):
 assert out_features % world_size == 0, f"Output features must be divisible by world size (world_size={world_size})"
 self.part_out_features = out_features // world_size
 self.weight = nn.Parameter(torch.empty(part_out_features, part_in_features, dtype=dtype))

 def forward(self, x: torch.Tensor) -> torch.Tensor:
 y = x @ self.weight
 return y


class RowParallelLinear(nn.Module):
 def __init__(self, in_features: int, out_features: int, dtype = None):
 assert in_features % world_size == 0, f"Input features must be divisible by world size (world_size={world_size})"
 self.part_in_features = in_features // world_size
 self.weight = nn.Parameter(torch.empty(out_features, part_in_features, dtype=dtype))

 def forward(self, x: torch.Tensor) -> torch.Tensor:
 y = x @ self.weight
 if world_size > 1:
 dist.all_reduce(y)
 return y


class MLP(nn.Module):
 def __init__(self, dim: int, inter_dim: int):
 super().__init__()
 self.w1 = ColumnParallelLinear(dim, inter_dim)
 self.w2 = RowParallelLinear(inter_dim, dim)
 self.w3 = ColumnParallelLinear(dim, inter_dim)

 def forward(self, x: torch.Tensor) -> torch.Tensor:
 return self.w2(F.silu(self.w1(x)) * self.w3(x))

attention

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


class TPMultiHeadAttention(nn.Module):
 def __init__(self, d_model: int, num_heads: int, head_dim: int = None):
 self.d_model = d_model
 self.num_heads = num_heads
 self.head_dim = head_dim if head_dim is not None else d_model // num_heads

 assert self.num_heads % world_size == 0, "num_heads must be divisible by world size"
 assert self.d_model == self.num_heads * self.head_dim, "d_model must equal num_heads * head_dim"

 # heads of different GPU
 self.local_num_heads = self.num_heads // world_size
 self.local_qkv_dim = self.local_num_heads * self.head_dim

 self.qkv_proj = ColumnParallelLinear(in_features=d_model, out_features=3 * self.local_qkv_dim)
 self.out_proj = RowParallelLinear(in_features=d_model, out_features=d_model)

 self.scale = 1.0 / torch.sqrt(torch.tensor(self.head_dim, dtype=torch.float32))

 def _split_heads(self):
 batch_size, seq_len, _ = x.shape
 # [batch, seq_len, local_num_heads, head_dim]
 x = x.reshape(batch_size, seq_len, self.local_num_heads, self.head_dim)
 # [batch, local_num_heads, seq_len, head_dim]
 return x.transpose(1, 2)

 def forward(self, x: torch.Tensor, attn_mask: torch.Tensor = None):
 batch_size, seq_len, _ = x.shape
 # [batch, seq_len, local_qkv_dim * 3]
 qkv = self.qkv_proj(x)
 # [batch, seq_len, local_qkv_dim]
 q, k, v = torch.split(qkv, self.local_qkv_dim, dim=-1)
 # [batch, local_num_heads, seq_len, head_dim]
 q = self._split_heads(q)
 k = self._split_heads(k)
 v = self._split_heads(v)

 attn_scores = (q @ k.transpose(-2, -1)) * self.scale

 if attn_mask is not None:
 attn_scores = attn_scores + attn_mask

 attn_weights = F.softmax(attn_scores, dim=-1)
 # [batch, local_num_heads, seq_len, head_dim]
 attn_output = attn_weights @ v
 # [batch, seq_len, local_num_heads, head_dim]
 attn_output = attn_output.transpose(1, 2)
 # [batch, seq_len, local_qkv_dim]
 attn_output = attn_output.reshape(batch_size, seq_len, self.local_qkv_dim)

 gather_list = [torch.empty_like(attn_output) for _ in range(world_size)]
 dist.all_gather(gather_list, attn_output)
 # [batch, seq_len, d_model]
 full_attn_output = torch.cat(gather_list, dim=-1)

 out = self.out_proj(full_attn_output)

 return out

Embedding

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


class ParallelEmbedding(nn.Module):
 def __init__(self, vocab_size: int, dim: int):
 super().__init__()
 self.vocab_size = vocab_size
 self.dim = dim
 assert vocab_size % world_size == 0, f"Vocabulary size must be divisible by world size (world_size={world_size})"
 self.part_vocab_size = (vocab_size // world_size)
 self.vocab_start_idx = rank * self.part_vocab_size
 self.vocab_end_idx = self.vocab_start_idx + self.part_vocab_size
 self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))

 def forward(self, x: torch.Tensor) -> torch.Tensor:
 if world_size > 1:
 mask = (x < self.vocab_start_idx) | (x >= self.vocab_end_idx)
 x = x - self.vocab_start_idx
 x[mask] = 0
 y = F.embedding(x, self.weight)
 if world_size > 1:
 y[mask] = 0
 dist.all_reduce(y)
 return y

Conclusion

作者提出了针对 transformer 架构的 tensor parallelism 策略来提高整体的训练效率，通过在训练过程加入四次 all-reduce 通信我们就可以训练更大规模的模型。

References

Nvidia-GPU specs

Wed, 14 Jan 2026 11:09:19 +0800

V100

V100 关键改进

Volta architecture
SM architecture: 支持深度学习
2nd NVIDIA NVLink
HBM2 memory
Volta Multi-process Service

V100 技术规格

Tesla Product	Tesla K40	Tesla M40	Tesla P100	Tesla V100
GPU	GK180 (Kepler)	GM200 (Maxwell)	GP100 (Pascal)	GV100 (Volta)
SMs	15	24	56	80
TPCs	15	24	28	40
FP32 Cores / GPU	2880	3072	3584	5120
FP64 Cores / GPU	960	96	1792	2560
Tensor Cores / GPU	NA	NA	NA	640
GPU Boost Clock	810/875 MHz	1114 MHz	1480 MHz	1530 MHz
Peak FP32 TFLOPS²	5	6.8	10.6	15.7
Peak FP64 TFLOPS²	1.7	.21	5.3	7.8
Peak Tensor TFLOPS²	NA	NA	NA	125
Memory Size	Up to 12 GB	Up to 24 GB	16 GB	16 GB
Memory Interface	384-bit GDDR5	384-bit GDDR5	4096-bit HBM2	4096-bit HBM2
TDP	235 Watts	250 Watts	300 Watts	300 Watts
Manufacturing Process	28 nm	28 nm	16 nm FinFET+	12 nm FFN

内存规格

GPU	Kepler GK180	Maxwell GM200	Pascal GP100	Volta GV100
Compute Capability	3.5	5.2	6.0	7.0
Threads / Warp	32	32	32	32
Max Warps / SM	64	64	64	64
Max Threads / SM	2048	2048	2048	2048
Max Thread Blocks / SM	32	32	32	32
Max 32-bit Registers / SM	65536	65536	65536	65536
Max Registers / Block	65536	65536	65536	65536
Max Registers / Thread	255	255	255	255
Max Thread Block Size	1024	1024	1024	1024
FP32 Cores / SM	192	128	64	64
Ratio of SM Registers to FP32 Cores	341	512	1024	1024
Shared Memory Size / SM	16 KB/32 KB/ 48 KB	96 KB	64 KB	Configurable up to 96 KB

系统规格

Specification	DGX-1 (Tesla P100)	DGX-1 (Tesla V100)
GPU	8x Tesla P100 GPUs	8x Tesla V100 GPUs
TFLOPS	170 (GPU FP16) + 3 (CPU FP32)	1 (GPU Tensor PFLOP)
GPU Memory	16 GB per GPU / 128 GB per DGX-1 Node	16 GB or 32 GB per GPU / 128-256 GB per DGX-1 Node
CPU	Dual 20-core Intel® Xeon® E5-2698 v4	Dual 20-core Intel® Xeon® E5-2698 v4
FP32 CUDA Cores	28,672 Cores	40,960 Cores
System Memory	Up to 512 GB 2133 MHz DDR4 LRDIMM	Up to 512 GB 2133 MHz DDR4 LRDIMM
Storage	4x 1.92 TB SSD RAID 0	4x 1.92 TB SSD RAID 0
Network Interconnect	Dual 10 GbE, 4 IB EDR	Dual 10 GbE, 4 IB EDR
System Dimensions	866 D x 444 W x 131 H (mm)	866 D x 444 W x 131 H (mm)
System Weight	80 lbs	80 lbs
Max Power TDP	3200 W	3200 W
Operating Temp	10 - 35°C	10 - 35°C

A100

A100 关键改进

Ampere 架构：使用 MIG 来将 A100 切分为更小的实例或者链接更多 GPU
Tensor Cores: 312 TFLOPs/s
NVLink: 更高的 throughput
MIG (multi-instance GPU): 一个 A100 可以切分为至多 7 个硬件层面隔离的实例
HBM2e: 更大的 HBM, 更快的 bandwidth, 更高的 DRAM 使用效率
structure sparsity: 稀疏运算可以带来 2 倍的算力提升

A100 技术规格

	A100 80GB PCIe	A100 80GB SXM
FP64	9.7 TFLOPS	9.7 TFLOPS
FP64 Tensor Core	19.5 TFLOPS	19.5 TFLOPS
FP32	19.5 TFLOPS	19.5 TFLOPS
Tensor Float 32 (TF32)	156 TFLOPS \| 312 TFLOPS	156 TFLOPS \| 312 TFLOPS*
BFLOAT16 Tensor Core	312 TFLOPS \| 624 TFLOPS*	312 TFLOPS \| 624 TFLOPS*
FP16 Tensor Core	312 TFLOPS \| 624 TFLOPS*	312 TFLOPS \| 624 TFLOPS*
INT8 Tensor Core	624 TOPS \| 1248 TOPS*	624 TOPS \| 1248 TOPS*
GPU Memory	80GB HBM2e	80GB HBM2e
GPU Memory Bandwidth	1,935 GB/s	2,039 GB/s
Max Thermal Design Power (TDP)	300W	400W ***
Multi-Instance GPU	Up to 7 MIGs @ 10GB	Up to 7 MIGs @ 10GB
Form Factor	PCIe Dual-slot air-cooled or single-slot liquid-cooled	SXM
Interconnect	NVIDIA® NVLink® Bridge for 2 GPUs: 600 GB/s ** PCIe Gen4: 64 GB/s	NVLink: 600 GB/s PCIe Gen4: 64 GB/s
Server Options	Partner and NVIDIA-Certified Systems™ with 1-8 GPUs	NVIDIA HGX™ A100-Partner and NVIDIA-Certified Systems with 4,8, or 16 GPUs NVIDIA DGX™ A100 with 8 GPUs

H100

H100 关键改进

Hopper 架构
Tensor Core: 更强的 tensor core
transformer engine: 加速基于 transformer 架构模型的训练
NVLink: 900GB/s 的 bandwidth
2nd MIG: 支持 multi-tenant, multi-user 使用
DPX: 基于 DPX 指令集加速动态规划算法

H100 技术规格

	H100 SXM	H100 NVL
FP64	34 teraFLOPS	30 teraFLOPs
FP64 Tensor Core	67 teraFLOPS	60 teraFLOPs
FP32	67 teraFLOPS	60 teraFLOPs
TF32 Tensor Core*	989 teraFLOPS	835 teraFLOPs
BFLOAT16 Tensor Core*	1,979 teraFLOPS	1,671 teraFLOPS
FP16 Tensor Core*	1,979 teraFLOPS	1,671 teraFLOPS
FP8 Tensor Core*	3,958 teraFLOPS	3,341 teraFLOPS
INT8 Tensor Core*	3,958 teraFLOPS	3,341 teraFLOPS
GPU Memory	80GB	94GB
GPU Memory Bandwidth	3.35TB/s	3.9TB/s
Decoders	7 NVDEC 7 JPEG	7 NVDEC 7 JPEG
Max Thermal Design Power (TDP)	Up to 700W (configurable)	350-400W (configurable)
Multi-Instance GPUs	Up to 7 MIGS @ 10GB each	Up to 7 MIGS @ 12GB each
Form Factor	SXM	PCIe dual-slot air-cooled
Interconnect	NVIDIA NVLink™: 900GB/s PCIe Gen5: 128GB/s	NVIDIA NVLink: 600GB/s PCIe Gen5: 128GB/s
Server Options	NVIDIA HGX H100 Partner and NVIDIA- Certified Systems™ with 4 or 8 GPUs NVIDIA DGX H100 with 8 GPUs	Partner and NVIDIA-Certified Systems with 1–8 GPUs
NVIDIA AI Enterprise	Add-on	Included

H200

H200 关键改进

更高的 HBM 内存和带宽
更高的 LLM inference 速度

H200 技术规格

	H200 SXM	H200 NVL
FP64	34 teraFLOPS	30 teraFLOPs
FP64 Tensor Core	67 teraFLOPS	60 teraFLOPs
FP32	67 teraFLOPS	60 teraFLOPs
TF32 Tensor Core*	989 teraFLOPS	835 teraFLOPs
BFLOAT16 Tensor Core*	1,979 teraFLOPS	1,671 teraFLOPS
FP16 Tensor Core*	1,979 teraFLOPS	1,671 teraFLOPS
FP8 Tensor Core*	3,958 teraFLOPS	3,341 teraFLOPS
INT8 Tensor Core*	3,958 teraFLOPS	3,341 teraFLOPS
GPU Memory	141GB	141GB
GPU Memory Bandwidth	4.8TB/s	4.8TB/s
Decoders	7 NVDEC 7 JPEG	7 NVDEC 7 JPEG
Confidential Computing	Supported	Supported
Max Thermal Design Power (TDP)	Up to 700W (configurable)	Up to 600W (configurable)
Multi-Instance GPUs	Up to 7 MIGS @ 18GB each	Up to 7 MIGS @ 18GB each
Form Factor	SXM	PCIe dual-slot air-cooled
Interconnect	NVIDIA NVLink™: 900GB/s PCIe Gen5: 128GB/s	2- or 4-way NVIDIA NVLink bridge: 900GB/s per GPU PCIe Gen5: 128GB/s
Server Options	NVIDIA HGX H200 Partner and NVIDIA- Certified Systems™ with 4 or 8 GPUs	NVIDIA MGX™ H200 NVL partner and NVIDIA-Certified Systems with up to 8 GPUs
NVIDIA AI Enterprise	Add-on	Included

相比于 H100, H200 升级了 HBM 和 bandwidth

B200

B200 关键改进

blackwell 架构： GPU 之间的通信效率大幅度提升
Grace CPU: GPU 可以与 Grace CPu 之间达到 900GB/s 的 bidirectional bandwidth
5th NVIDIA NVLink: 可以链接 576 块 GPU 来支持计算，NVlink 的带宽可以达到 130TB/s
RAS engine: 自动识别故障来提高效率
NVIDIA networking

B2100 技术规格

system specification 如下

Specification	GB200 NVL72	GB200 NVL4	HGX B200
NVIDIA Blackwell GPUs \| Grace CPUs	72 \| 36	4 \| 2	8 \| 0
CPU Cores	2,592 Arm® Neoverse V2 Cores	144 Arm Neoverse V2 Cores	-
Total NVFP4 Tensor Core²	1,440 \| 720 PFLOPS	80 \| 40 PFLOPS	144 \| 72 PFLOPS
Total FP8/FP6 Tensor Core²	720 PFLOPS	40 PFLOPS	72 PFLOPS
Total Fast Memory	31 TB	1.8 TB	1.4 TB
Total Memory Bandwidth	576 TB/s	32 TB/s	62 TB/s
Total NVLink Bandwidth	130 TB/s	7.2 TB/s	14.4 TB/s

individual specification 如下

Specification	GB200 NVL72	GB200 NVL4	HGX B200
FP4 Tensor Core	20 PFLOPS	20 PFLOPS	18 PFLOPS
FP8/FP6 Tensor Core²	10 PFLOPS	10 PFLOPS	9 PFLOPS
INT8 Tensor Core²	10 POPS	10 POPS	9 POPS
FP16/BF16 Tensor Core²	5 PFLOPS	5 PFLOPS	4.5 PFLOPS
TF32 Tensor Core²	2.5 PFLOPS	2.5 PFLOPS	2.2 PFLOPS
FP32	80 TFLOPS	80 TFLOPS	75 TFLOPS
FP64 / FP64 Tensor Core	40 TFLOPS	40 TFLOPS	37 TFLOPS
GPU Memory Bandwidth	186 GB HBM3E 8 TB/s	186 GB HBM3E 8 TB/s	180 GB HBM3E 7.7 TB/s
Multi-Instance GPU (MIG)	-	7	-
Decompression Engine	-	Yes	-
Decoders	-	7 NVDEC³ 7 nvJPEG	-
Max Thermal Design Power (TDP)	Configurable up to 1,200 W	Configurable up to 1,200 W	Configurable up to 1,000 W
Interconnect	-	Fifth-generation NVLink: 1.8 TB/s PCIe Gen5: 128 GB/s	-
Server Options	NVIDIA GB200 NVL72 partner and NVIDIA-Certified Systems™ with 72 GPUs	NVIDIA MGX partner and NVIDIA-Certified Systems	NVIDIA HGX B200 partner and NVIDIA-Certified Systems with 8 GPUs

B300

B300 关键改进

Blackwell 架构
AI reasoning inference: 支持 test-time scaling, 对 attention layer 和 FLOPs 都有加速
HBM3e: 支持更大的 batch size 和 throughput
ConnectX-8 SuperNIC, 一个 host2 个 ConnectX-8 设备，支持 800Gb/s 的 GPU 之间通信
Grace-CPU: 更强的表现和带宽
5th NVIDIA NVLink: 更高的通信效率

B3100 技术规格

system specification 如下

	GB300 NVL72	HGX B300
Blackwell Ultra GPUs\| Grace CPUs	72 \| 36	8 \| 0
CPU Cores	2,592 Arm Neoverse V2 Cores	-
Total FP4 Tensor Core	1 1,440 PFLOPS \| 1,080 PFLOPS	144 PFLOPS \| 108 PFLOPS
Total FP8/FP6 Tensor Core	2 720 PFLOPS	72 PFLOPS
Total Fast Memory	37 TB	2.1 TB
Total Memory Bandwidth	576 TB/s	62 TB/s
Total NVLink Switch Bandwidth	130 TB/s	14.4 TB/s

individual specification 如下

	GB300 NVL72	HGX B300
FP4 Tensor Core	20 PFLOPS \| 15 PFLOPS	18 PFLOPS \| 14 PFLOPS
FP8/FP6 Tensor Core2	10 PFLOPS	9 PFLOPS
INT8 Tensor Core2	330 TOPS	307 TOPS
FP16/BF16 Tensor Core	5 PFLOPS	4.5 PLFOPS
TF32 Tensor Core2	2.5 PFLOPS	2.2 PFLOPS
FP32	80 TFLOPS	75 TFLOPS
FP64/FP64 Tensor Core	1.3 TFLOPS	1.2 TFLOPS
GPU Memory \| Bandwidth	279 GB HBM3E \| 8 TB/s	270 GB HBM3E \| 7.7 TB/s
Multi-Instance GPU (MIG)	7	7
Decompression Engine	Yes	Yes
Decoders	7 NVDEC3 7 nvJPEG	7 NVDEC3 7 nvJPEG
Max Thermal Design Power (TDP)	Configurable up to 1,400 W	Configurable up to 1,100 W
Interconnect	Fifth-Generation NVLink: 1.8 TB/s PCIe Gen6: 256 GB/s	Fifth-Generation NVLink: 1.8 TB/s PCIe Gen6: 256 GB/s
Server Options	NVIDIA GB300 NVL72 partner and NVIDIA-Certified Systems™	NVIDIA HGX B300 partner and NVIDIA-Certified Systems