docker搭建普罗米修斯监控gpu
ip8的服务器监控ip110和ip111的服务器
被监控的服务器110和111只需要安装node-export和nvidia-container-toolkit
下载镜像包
docker pull prom/node-exporter
docker pull prom/prometheus
docker pull grafana/grafana
新建目录
mkdir /opt/prometheus
cd /opt/prometheus/
vim prometheus.yml
global:scrape_interval: 60sevaluation_interval: 60sscrape_configs:- job_name: prometheusstatic_configs:- targets: ['localhost:9090']labels:instance: prometheus- job_name: linuxstatic_configs:- targets: ['10.20.13.8:9100']labels:instance: master- job_name: nodestatic_configs:- targets: ['10.20.13.111:9100','10.20.13.110:9100']
启动普罗米修斯
docker run -d \-p 9090:9090 \-v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \prom/prometheus启动node-export
docker run -d -p 9100:9100 \-v "/proc:/host/proc:ro" \-v "/sys:/host/sys:ro" \-v "/:/rootfs:ro" \prom/node-exporter
新建目录
mkdir /opt/grafana-storage
chmod 777 -R /opt/grafana-storage
启动grafana
docker run -d \-p 3000:3000 \--name=grafana \-v /opt/grafana-storage:/var/lib/grafana \grafana/grafana
访问grafana url
10.20.13.8:3000
默认会先跳转到登录页面，默认的用户名和密码都是admin
添加data source时，ip地址要填写本机Ip地址 http://ip:9090
安装显卡监控
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.listapt update
apt upgrade
apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker运行容器
docker run -d --restart always --gpus all -p 9400:9400 --name gpu-exporter nvcr.io/nvidia/k8s/dcgm-exporter:3.2.5-3.1.8-ubuntu22.04在配置文件中加入端口
vim /opt/prometheus/prometheus.yml加入一段
- job_name: gpu_metricsstatic_configs:- targets: ['10.20.13.111:9400','10.20.13.110:9400']在grafanan导入监控gpu模板 id12239
