kubeflow Pods stuck in Pending state
问题如图
查看详细
kubectl -n kubeflow describe pod minio-d56488484-5drv8发现pvc没有绑定
解决：
查看pvc
kubectl -n kubeflow get pvc
查看pvc描述
kubectl get pvc minio-pv-claim -n kubeflow -o yaml...
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: storageclass-default
volumeMode: Filesystem
volumeName: pv-volume1
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
phase: Bound
capacity:
storage: 20Gi 即为所需的磁盘空间大小。
kubectl describe pv local-pv-0
Name: local-pv-0
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"local-pv-0"},"spec":{"accessModes":["ReadWriteOnce"],"ca...
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: storageclass-default
Status: Bound
Claim: kube-operator/f2c-prometheus-server
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Gi
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [worker1.prophecis.software.dc]
Message:
Source:
Type: LocalVolume (a persistent volume backed by local storage on a node)
Path: /opt/prophecis/local/f2c0
Events: <none>
Path: /opt/prophecis/local/f2c0 为pv绑定的主机实际空间，需到各worker节点查看有没有这个目录，如果没有就创建。
mkdir -p /opt/prophecis/local/f2c0
并用df -h 查看剩余空间是否足够。
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 84M 7.7G 2% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/centos-root 44G 18G 26G 41% /
There are three pods that have Persistent Volume Claims (PVCs) that will get stuck in pending state if they are unable to bind their PVC. The three pods are minio, mysql, and katib-mysql. Check the status of the PVC requests:
kubectl -n ${NAMESPACE} get pvc
- Look for the status of “Bound”
- PVC requests in “Pending” state indicate that the scheduler was unable to bind the required PVC.
If you have not configured dynamic provisioning for your cluster, including a default storage class, then you must create a persistent volume for each of the PVCs.
You can use the example below to create local persistent volumes:
sudo mkdir /mnt/pv{1..3}
kubectl create -f - <<EOF
kind: PersistentVolume
apiVersion: v1
metadata:
name: pv-volume1
spec:
storageClassName:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/pv1"
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: pv-volume2
spec:
storageClassName:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/pv2"
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: pv-volume3
spec:
storageClassName:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/pv3"
EOF
Once created the scheduler will successfully start the remaining three pods. The PVs may also be created prior to running any of the kfctl commands.