当前位置：首页 > news >正文

DeepSpeed Monitoring Comm. Logging

news 来源：原创 2024/9/29 15:31:49

Monitoring

支持多种后端：Tensorboard、WandB、Comet、CSV文件；

TensorBoard例子：

自动监控：DeepSpeed自动把重要metric记录下来。只需在配置文件里enable相应的看板后端即可：
{"tensorboard": {"enabled": true,"output_path": "output/ds_logs/","job_name": "train_bert"}"wandb": {"enabled": true,"team": "my_team","group": "my_group","project": "my_project"}"comet": {"enabled": true,"project": "my_project","experiment_name": "my_experiment"}"csv_monitor": {"enabled": true,"output_path": "output/ds_logs/","job_name": "train_bert"}
}
自定义监控：

# Step 1: Import monitor (and DeepSpeed config, if needed)
from deepspeed.monitor.monitor import MonitorMaster
from deepspeed.runtime.config import DeepSpeedConfig

# Step 2: Initialized monitor with DeepSpeed config (get DeepSpeed config object, if needed)
ds_config = DeepSpeedConfig("ds_config.json")
monitor = MonitorMaster(ds_config.monitor_config)

for epoch in range(2):

running_loss = 0.0
for i, data in enumerate(trainloader):
pre = time.time()
inputs, labels = data[0].to(model_engine.local_rank), data[1].to(
model_engine.local_rank)
if fp16:
inputs = inputs.half()
outputs = model_engine(inputs)
loss = criterion(outputs, labels)

model_engine.backward(loss)
model_engine.step()
post = time.time()
# Step 3: Create list of 3-tuple records (single entry in this case)
events = [("Time per step", post-pre, model_engine.global_samples)]
# Step 4: Call monitor.write_events on the list from step 3
monitor.write_events(events)

[("Time per step", post-pre, model_engine.global_samples)]，<表名，纵轴值，横轴值>

通信Logging

注意：加了logging, 所有通信将改为同步，对性能会有伤害。

所有deepspeed.comm下的通信，都将被统计上。

在配置文件里打开：
"comms_logger": {"enabled": true,"verbose": false,"prof_all": true,"debug": false
}
verbose: 边跑，边把发生的通信，一条条写下来。例：
[2022-06-26 01:39:55,722] [INFO] [logging.py:69:log_dist] [Rank 0] rank=0 | comm op: reduce_scatter_tensor | time (ms): 9.46 | msg size: 678.86 MB | algbw (Gbps): 1204.52  | busbw (Gbps): 1129.23
[2022-06-26 01:39:56,470] [INFO] [logging.py:69:log_dist] [Rank 0] rank=0 | comm op: all_gather_into_tensor | time (ms): 0.11 | msg size: 6.0 MB | algbw (Gbps): 954.41  | busbw (Gbps): 894.76
[2022-06-26 01:39:56,471] [INFO] [logging.py:69:log_dist] [Rank 0] rank=0 | comm op: all_gather_into_tensor | time (ms): 0.08 | msg size: 6.0 MB | algbw (Gbps): 1293.47  | busbw (Gbps): 1212.63
algbw: algorithm bandwidth, 发生的通信size/实际通信时间；

busbw: 硬件理论带宽；是个固定值；

algbw如果比busbw小太多，说明糟糕，有待进一步优化；

总结式：deepspeed.comm.log_summary()
Comm. Op            Message Size        Count               Total Latency(ms)   Avg Latency(ms)     tput_avg (Gbps)     busbw_avg (Gbps)
broadcast2.0 KB              146                 11.12               0.08                0.43                0.4198.25 MB            1                   8317.12             8317.12             0.20                0.19
reduce_scatter_tensor678.86 MB           40                  602.29              9.69                1468.06             1376.31
展示通信等待时长：
dist.log_summary(show_straggler=True)
这么计算的：（一次组播通信里，每个rank的完成时间，减去，所有rank里完成最快的，这些"等待"时间，加和到一起）
straggler = sum(t_collectives - allreduce(t_collectives, MIN))