centos7 k8s安装部署nvidia-docker2 排坑记录
1.安装docker-ce
因为nvidia-docker2依赖docker-ce，所以得先安装docker-ce。
1.1 如果主机上已经安装了docker，则先运行如下命令卸载
service docker stop
yum remove docker1.2 安装yum管理依赖包
sudo yum install -y yum-utils device-mapper-persistent-data lvm21.3添加docker-ce仓库命令
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo如果报以下错误
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
File "/bin/yum-config-manager", line 135
except yum.Errors.RepoError, e:
^
SyntaxError: invalid syntax则运行如下命令修改/bin/yum-config-manager中的配置
vi /bin/yum-config-manager
按下 i 字母
把文件开头 #!/usr/bin/python -tt 改为 #!/usr/bin/python2 -tt
再按下esc键，键入:wq!保存。最后再次执行添加docker-ce仓库命令
1.4 运行如下安装docker-ce命令
sudo yum install docker-ce docker-ce-cli containerd.io如果报以下错误
Transaction check error:
file /usr/bin/docker from install of docker-ce-cli-1:19.03.8-3.el7.x86_64 conflicts with file from package docker-common-2:1.13.1-109.gitcccb291.el7.centos.x86_64
file /usr/bin/dockerd from install of docker-ce-3:19.03.8-3.el7.x86_64 conflicts with file from package docker-common-2:1.13.1-109.gitcccb291.el7.centos.x86_64
Error Summary
则先运行如下命令删除docker残余的组件
sudo yum erase docker-common-2:1.13.1-109.gitcccb291.el7.centos.x86_64
再次运行安装docker-ce命令
1.5 开机启动/启动/查看状态
systemctl enable docker // 设置开机启动
systemctl start docker // 启动docker
systemctl status docker //查看状态
2.安装nvidia-docker2
2.1设置仓库
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \
sudo tee /etc/yum.repos.d/nvidia-docker.repo2.2更新仓库中的key
DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
sudo yum makecache2.3安装nvidia-docker2
sudo yum install nvidia-docker2报错container-selinux not found。原因是docker仓库没有该依赖包，自己在阿里云仓库手动下载离线安装，
http://mirrors.aliyun.com/centos/7/extras/x86_64/Packages/
安装依赖包：
rpm -ivh fuse3-libs-3.6.1-4.el7.x86_64.rpm
rpm -ivh fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm
rpm -ivh slirp4netns-0.4.3-4.el7_8.x86_64.rpm
yum install policycoreutils-python
rpm -ivh container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm
2.4 重新载入docker daemon的设定
sudo pkill -SIGHUP dockerd2.5 测试是否安装成功
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
第一次运行会花几分钟下载组件，最后显示如下结果则表示安装成功
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
Wed Mar 25 04:58:46 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:1A:00.0 Off | N/A |
| 16% 26C P8 1W / 250W | 0MiB / 10989MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
2.6 k8s中调用GPU
k8s在v1.8版本后推荐使用NVIDIA/k8s-device-plugin来调用GPU。其实就是集群版的nvidia-docker...都是暴露gpu分配调度接口，并进行监听。原理参照k8s插件原理。
预准备
- Kubernetes 的节点必须预先安装了 NVIDIA 驱动
- Kubernetes 的节点必须预先安装 nvidia-docker 2.0
- Docker 的默认运行时必须设置为 nvidia-container-runtime，而不是 runc
- NVIDIA 驱动版本 ~= 384.81
- 接下来设置默认runtime.
vim /etc/docker/daemon.json
重启docker使配置生效
systemctl restart docker
- 安装k8s-device-plugin
在master节点上运行
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta4/nvidia-device-plugin.yml返回提示
daemonset.apps/nvidia-device-plugin-daemonset created
查看安装后是否有可用gpu资源
kubectl describe nodes
重启docker使配置生效
可以看到在有GPU的node上，守护Pods中多了nvidia-device-plugin-daemonset用于监听和分配gpu，在Allocated resources（可分配资源）中多了gpu。