kubevirt基于CDI创建虚拟机
CDI介绍
KubeVirt 的 Containerized Data Importer (CDI) 是一个 Kubernetes 原生的数据管理组件,专门为虚拟机 (VM) 提供存储支持,尤其在虚拟机的镜像管理和数据导入方面非常有用。CDI 的主要用途是帮助用户轻松地将外部数据源导入到 Kubernetes 集群中,并将这些数据转换为虚拟机可以使用的持久化存储卷 (Persistent Volume, PV)。
containerized-data-importer(CDI) 项目提供了一些设施,使持久卷声明(PVC) 能够通过DataVolumes用作 KubeVirt VM 的磁盘。 CDI 的三个主要用例是:
- 将磁盘映像从 Web 服务器或容器注册表导入到 DataVolume
- 将现有 PVC 克隆到 DataVolume
- 将本地磁盘镜像上传到DataVolume
本文档涉及第三个用例。因此,您应该在集群中安装 CDI、要上传的 VM 磁盘,并在路径中安装 virtctl。
前置要求
已准备可用的kubernetes集群
root@node40:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node40 Ready control-plane 161d v1.29.3 192.168.72.40 <none> Ubuntu 22.04.2 LTS 5.15.0-122-generic containerd://1.7.15
node41 Ready <none> 161d v1.29.3 192.168.72.41 <none> Ubuntu 22.04.2 LTS 5.15.0-76-generic containerd://1.7.15
node42 Ready <none> 161d v1.29.3 192.168.72.42 <none> Ubuntu 22.04.2 LTS 5.15.0-122-generic containerd://1.7.15
root@node40:~#
已安装kubevirt以及virtctl命令行工具。
root@node40:~# virtctl version
Client Version: version.Info{GitVersion:"v1.4.0-alpha.0", GitCommit:"946f894f472b3a355ebb8eefaf89b871a06415ab", GitTreeState:"clean", BuildDate:"2024-09-12T11:50:12Z", GoVersion:"go1.22.6 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{GitVersion:"v1.4.0-alpha.0", GitCommit:"946f894f472b3a355ebb8eefaf89b871a06415ab", GitTreeState:"clean", BuildDate:"2024-09-12T13:21:26Z", GoVersion:"go1.22.6 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}
已准备可用的默认storageclass
root@node40:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 10d
longhorn-static driver.longhorn.io Delete Immediate true 60s
CDI安装
安装最新的 CDI 版本
export TAG=$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)
export VERSION=$(echo ${TAG##*/})
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
确认安装成功
root@node40:~# kubectl -n cdi get pods
NAME READY STATUS RESTARTS AGE
cdi-apiserver-55dd9447cb-ctsww 1/1 Running 0 5d22h
cdi-deployment-55c6d9fc49-88vf8 1/1 Running 14 (2d5h ago) 10d
cdi-operator-7f5bc68fc5-zp54v 1/1 Running 5 (2d5h ago) 5d22h
cdi-uploadproxy-5b76f7c876-9gkqs 1/1 Running 0 10d
公开 cdi-uploadproxy 服务
参考:https://github.com/kubevirt/containerized-data-importer/blob/main/doc/upload.md
$ cat cdi-uploadproxy-nodeport.yaml
apiVersion: v1
kind: Service
metadata:name: cdi-uploadproxy-nodeportnamespace: cdilabels:cdi.kubevirt.io: "cdi-uploadproxy"
spec:type: NodePortports:- port: 443targetPort: 8443nodePort: 31001protocol: TCPselector:cdi.kubevirt.io: cdi-uploadproxy
应用yaml文件
kubectl apply -f cdi-uploadproxy-nodeport.yaml
确认存在NodePort类型的service:cdi-uploadproxy-nodeport
root@node40:~# kubectl -n cdi get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cdi-api ClusterIP 10.96.1.45 <none> 443/TCP 11d
cdi-prometheus-metrics ClusterIP 10.96.3.131 <none> 8080/TCP 11d
cdi-uploadproxy ClusterIP 10.96.0.118 <none> 443/TCP 11d
cdi-uploadproxy-nodeport NodePort 10.96.2.193 <none> 443:31001/TCP 3h4m
上传镜像
CDI 支持 qemu 支持的raw
和qcow2
镜像格式。还可以使用可启动 ISO 映像,并将其视为raw
映像。镜像可以使用gz
或xz
格式进行压缩。
下载镜像,以rocky linux cloud image为例:
wget https://dl.rockylinux.org/pub/rocky/9.4/images/x86_64/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2
使用virtctl上传镜像,确认已安装virtctl工具。
virtctl image-upload dv rocky-vm-disk \
--size=10Gi \
--image-path=/root/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2 \
--uploadproxy-url=https://192.168.72.40:31001/v1beta1/upload \
--insecure
参数说明:
virtctl image-upload
:子命令用于上传一个磁盘镜像到 Kubernetes 集群内,通常会存储在一个DataVolume
中。dv rocky-vm-disk
:dv
是DataVolume
的缩写,表示要上传的镜像会创建或关联到一个名为rocky-vm-disk
的DataVolume
资源。DataVolume
是 CDI 提供的一个资源类型,用于管理虚拟机磁盘镜像的数据导入和上传。--size=10Gi
: 指定了要创建的虚拟磁盘的大小,这里是 10GiB。DataVolume
会申请一个大小为 10GiB 的持久化卷(PVC)来存储上传的镜像文件。--image-path
: 指定了本地磁盘上待上传的镜像文件路径。这里的文件是Rocky Linux 9.4
的云基础镜像,格式为qcow2
,位于/root
目录下。--uploadproxy-url
: 指定了 CDI 的上传代理服务 (uploadproxy
) 的 URL,该 URL 用于上传镜像文件至 Kubernetes 集群。这个 URL 由 Kubernetes nodeport类型的服务暴露,使用节点NODE IP 地址以及31001
端口访问。通过这个代理服务,virtctl
将本地镜像文件上传至集群中的DataVolume
。--insecure
: 表示在与uploadproxy
服务通信时,不使用 TLS 证书验证。这在测试或非生产环境中很常见,目的是避免因为证书问题而导致上传失败。生产环境中建议使用带有证书验证的安全连接。
查看创建的cdi-upload pod,上传完成后该pod被自动清理
root@node40:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
cdi-upload-prime-d9d2776b-bda2-4852-a0f7-6e69c6582031 1/1 Running 0 3m22s
查看上传日志信息
root@node40:~# kubectl logs -f cdi-upload-prime-e2055ca6-92a0-42fa-89da-ba6211c9aa2a
I0929 07:40:34.705880 1 uploadserver.go:81] Running server on 0.0.0.0:8443
I0929 07:40:37.181186 1 uploadserver.go:438] Content type header is ""
I0929 07:40:37.181236 1 data-processor.go:348] Calculating available size
I0929 07:40:37.188220 1 data-processor.go:356] Checking out block volume size.
I0929 07:40:37.188243 1 data-processor.go:368] Request image size not empty.
I0929 07:40:37.188275 1 data-processor.go:373] Target size 10737418240.
I0929 07:40:37.188641 1 data-processor.go:247] New phase: TransferScratch
I0929 07:40:37.189035 1 util.go:96] Writing data...
I0929 07:41:08.076678 1 data-processor.go:247] New phase: Convert
I0929 07:41:08.076712 1 data-processor.go:253] Validating image
E0929 07:41:08.191970 1 prlimit.go:156] failed to kill the process; os: process already finished
I0929 07:41:08.192141 1 qemu.go:115] Running qemu-img with args: [convert -t writeback -p -O raw /scratch/tmpimage /dev/cdi-block-volume]
E0929 07:43:58.201878 1 prlimit.go:156] failed to kill the process; os: process already finished
I0929 07:43:58.202649 1 data-processor.go:247] New phase: Resize
I0929 07:43:58.205172 1 data-processor.go:247] New phase: Complete
I0929 07:43:58.206054 1 uploadserver.go:465] Wrote data to /dev/cdi-block-volume
I0929 07:43:58.206845 1 uploadserver.go:230] Shutting down http server after successful upload
I0929 07:43:58.208180 1 uploadserver.go:115] UploadServer successfully exited
确认上传成功
root@node40:~# virtctl image-upload dv rocky-vm-disk \
--size=10Gi \
--image-path=/root/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2 \
--uploadproxy-url=https://192.168.72.40:31001/v1beta1/upload \
--insecure
PVC default/rocky-vm-disk not found
DataVolume default/rocky-vm-disk created
Waiting for PVC rocky-vm-disk upload pod to be ready...
Pod now ready
Uploading data to https://192.168.72.40:31001/v1beta1/upload578.31 MiB / 578.31 MiB [----------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2.88 MiB p/s 3m21sUploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
Processing completed successfully
Uploading /root/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2 completed successfully
root@node40:~#
查看创建的pvc
root@node40:~# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
rocky-vm-disk Bound pvc-0679a050-51d4-4c62-8d52-1b20dc2f9cfb 10Gi RWO longhorn <unset> 3m25s
查看创建的pv
root@node40:~# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-0679a050-51d4-4c62-8d52-1b20dc2f9cfb 10Gi RWO Delete Bound default/rocky-vm-disk longhorn <unset> 3m53s
查看创建的datavolume
root@node40:~# kubectl get dv
NAME PHASE PROGRESS RESTARTS AGE
rocky-vm-disk Succeeded N/A 4m17s
查看datavolume详细信息
root@node40:~# kubectl get dv rocky-vm-disk -o yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:annotations:cdi.kubevirt.io/storage.usePopulator: "true"creationTimestamp: "2024-09-29T07:40:17Z"generation: 1name: rocky-vm-disknamespace: defaultresourceVersion: "5568675"uid: 54d1741c-169e-4919-933e-37abe416cb62
spec:contentType: kubevirtsource:upload: {}storage:resources:requests:storage: 10Gi
status:claimName: rocky-vm-diskconditions:- lastHeartbeatTime: "2024-09-29T07:43:59Z"lastTransitionTime: "2024-09-29T07:43:59Z"message: PVC rocky-vm-disk Boundreason: Boundstatus: "True"type: Bound- lastHeartbeatTime: "2024-09-29T07:43:59Z"lastTransitionTime: "2024-09-29T07:43:59Z"status: "True"type: Ready- lastHeartbeatTime: "2024-09-29T07:43:58Z"lastTransitionTime: "2024-09-29T07:43:58Z"message: Upload Completereason: Completedstatus: "False"type: Runningphase: Succeededprogress: N/A
创建虚拟机
示例yaml文件
root@node40:~/cdi# cat rocky-vm.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:name: rocky-vm
spec:runStrategy: Alwaystemplate:metadata:labels:kubevirt.io/size: smallkubevirt.io/domain: rocky-vmspec:domain:devices:disks:- name: datavolumevolumedisk:bus: virtio- name: cloudinitdiskdisk:bus: virtiointerfaces:- name: defaultmasquerade: {}resources:requests:memory: 1Gilimits:memory: 2Ginetworks:- name: defaultpod: {}volumes:- name: datavolumevolumedataVolume:name: rocky-vm-disk- name: cloudinitdiskcloudInitNoCloud:userData: |-#cloud-configpassword: rockychpasswd: { expire: False }
应用yaml文件
kubectl apply -f rocky-vm.yaml
查看创建的虚拟机
root@node40:~/cdi# kubectl get vm
NAME AGE STATUS READY
rocky-vm 7m49s Running True
root@node40:~/cdi#
root@node40:~/cdi# kubectl get vmi
NAME AGE PHASE IP NODENAME READY
rocky-vm 7m52s Running 100.64.1.111 node42 True
root@node40:~/cdi#
通过virtctl 访问虚拟机,默认账号密码为rocky/rocky
root@node40:~# virtctl console rocky-vm
Successfully connected to rocky-vm console. The escape sequence is ^]rocky-vm login: rocky
Password:
Last login: Sun Sep 29 07:52:19 on ttyS0
[rocky@rocky-vm ~]$
[rocky@rocky-vm ~]$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
[rocky@rocky-vm ~]$
虚拟机写入数据
[rocky@rocky-vm ~]$ echo $(date) > data.txt
[rocky@rocky-vm ~]$
[rocky@rocky-vm ~]$ cat data.txt
Sun Sep 29 08:01:03 AM UTC 2024
[rocky@rocky-vm ~]$
删除虚拟机
root@node40:~# kubectl delete vmi rocky-vm
virtualmachineinstance.kubevirt.io "rocky-vm" deleted
等待虚拟机重建,验证持久化数据是否存在
root@node40:~# virtctl console rocky-vm
[rocky@rocky-vm ~]$ cat data.txt
Sun Sep 29 08:01:03 AM UTC 2024
[rocky@rocky-vm ~]$