部署K8S
防火强的初始化：
[root@k8s-node-12 ~]# systemctl stop firewalld NetworkManager
[root@k8s-node-12 ~]# systemctl disable firewalld NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@k8s-node-12 ~]# sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
[root@k8s-node-12 ~]# setenforce 0
[root@k8s-node-12 ~]# iptables -F
[root@k8s-node-12 ~]# iptables -X
[root@k8s-node-12 ~]# iptables -Z
[root@k8s-node-12 ~]# iptables -P FORWARD ACCEPT关闭swap：
k8s默认禁用swap功能
[root@k8s-node-11 ~]# swapoff -a
防止开机自动挂载swap分区
[root@k8s-node-11 ~]# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstabyum源的配置：
[root@k8s-node-11 yum.repos.d]# cat CentOS-Base.repo
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client. You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7[root@k8s-node-11 yum.repos.d]#
源配置具体步骤一：
[root@k8s-node-11 network-scripts]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 2523 100 2523 0 0 287 0 0:00:08 0:00:08 --:--:-- 777
[root@k8s-node-11 network-scripts]# curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 664 100 664 0 0 77 0 0:00:08 0:00:08 --:--:-- 162
[root@k8s-node-11 network-scripts]# sed -i '/aliyuncs/d' /etc/yum.repos.d/*.repo
[root@k8s-node-11 network-scripts]# yum clean all && yum makecache fast
已加载插件：fastestmirror, langpacks
正在清理软件源： base epel extras updates
Cleaning up list of fastest mirrors
已加载插件：fastestmirror, langpacks
Determining fastest mirrors
base | 3.6 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/7): epel/x86_64/group_gz | 99 kB 00:00:08
(2/7): epel/x86_64/updateinfo | 1.0 MB 00:00:08
(3/7): base/7/x86_64/group_gz | 153 kB 00:00:08
(4/7): epel/x86_64/primary_db | 7.0 MB 00:00:01
(5/7): base/7/x86_64/primary_db | 6.1 MB 00:00:09
(6/7): updates/7/x86_64/primary_db | 24 MB 00:00:02
(7/7): extras/7/x86_64/primary_db | 250 kB 00:00:12
元数据缓存已建立
[root@k8s-node-11 network-scripts]#源配置具体步骤二：
[root@k8s-master-10 ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 2523 100 2523 0 0 288 0 0:00:08 0:00:08 --:--:-- 783
[root@k8s-master-10 ~]# curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 664 100 664 0 0 76 0 0:00:08 0:00:08 --:--:-- 160
[root@k8s-master-10 ~]# sed -i '/aliyuncs/d' /etc/yum.repos.d/*.repo
[root@k8s-master-10 ~]# yum clean all && yum makecache fast
已加载插件：fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Repository contrib is listed more than once in the configuration
正在清理软件源： base docker-ce-stable epel extras updates
Cleaning up list of fastest mirrors
已加载插件：fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Repository contrib is listed more than once in the configuration
Determining fastest mirrors
base | 3.6 kB 00:00:00
docker-ce-stable | 3.5 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/9): epel/x86_64/group_gz | 99 kB 00:00:08
(2/9): base/7/x86_64/group_gz | 153 kB 00:00:08
(3/9): docker-ce-stable/7/x86_64/updateinfo | 55 B 00:00:08
(4/9): docker-ce-stable/7/x86_64/primary_db | 117 kB 00:00:08
(5/9): epel/x86_64/updateinfo | 1.0 MB 00:00:00
(6/9): base/7/x86_64/primary_db | 6.1 MB 00:00:09
(7/9): epel/x86_64/primary_db | 7.0 MB 00:00:00
(8/9): extras/7/x86_64/primary_db | 250 kB 00:00:08
(9/9): updates/7/x86_64/primary_db | 24 MB 00:00:10
元数据缓存已建立
[root@k8s-master-10 ~]#NTP配置：
[root@k8s-node-12 network-scripts]# yum install chrony -y
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
软件包 chrony-3.4-1.el7.x86_64 已安装并且是最新版本
无须任何处理
[root@k8s-node-12 network-scripts]# systemctl start chronyd
[root@k8s-node-12 network-scripts]# systemctl enable chronyd
[root@k8s-node-12 network-scripts]# date
2023年 10月 24日 星期二 21:01:39 CST
[root@k8s-master-10 ~]# ntpdate -u ntp.aliyun.com
24 Oct 21:21:01 ntpdate[2672]: adjust time server 203.107.6.88 offset 0.005337 sec
[root@k8s-master-10 ~]# hwclock -w
修改Linux内核参数，开启数据包转发功能：
#容器跨主机通信，底层走的iptables，内核级别的数据包转发:
[root@k8s-master-10 ~]# cat <<EOF> /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> vm.max_map_count=262144
> EOF加载读取内核参数配置文件：
[root@k8s-node-12 network-scripts]# sysctl -p /etc/sysctl.d/k8s.conf
报错如下：
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 没有那个文件或目录
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: 没有那个文件或目录
net.ipv4.ip_forward = 1
vm.max_map_count = 262144
解决方法：
root@k8s-node-12 network-scripts]# modprobe br_netfilter
[root@k8s-node-12 network-scripts]# sysctl -p
完美解决：
[root@k8s-node-12 network-scripts]# sysctl -p /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.max_map_count = 262144安装docker基础环境：
yum remove docker docker-common docker-selinux docker-engine -y
curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum list docker-ce --showduplicates
yum install docker-ce-19.03.15 docker-ce-cli-19.03.15 -y
mkdir -p /etc/docker/
vi daemon.json
{"registry-mirrors":["https://ms9glx6x.mirror.aliyuncs.com"],"exec-opts":["native.cgroupdriver=systemd"]
}systemctl start docker && systemctl enable docker安装K8S的初始化工具kubeadm命令（主节点执行）
# 安装k8s集群环境初始化的工具
# kubelet-1.19.3 #组件，增删改查pod在具体机器上，pod可以运行主节点上，node节点上；
# kubeadm-1.19.3 #k8s版本 1.19.3，自动拉取k8s基础组件镜像的一个工具；
# kubectl-1.19.3 #管理，维护k8s客户端和服务端交付的一个命令行工具；
所有机器执行
设置阿里云源
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repocat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOFyum clean all && yum makecacheyum list kubeadm --showduplicatesyum install kubelet-1.19.3 kubeadm-1.19.3 kubectl-1.19.3 ipvsadm
k8s安装完毕后，设置所有节点的kubelet开机运行
#查看kubeadm版本信息，初始化k8s版本信息
[root@k8s-node-12 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}设置开机自启
systemctl enable docker
systemctl enable kubelet初始化k8s-master主节点（只在主节点执行）
kubeadm init \
--apiserver-advertise-address=192.168.208.128 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.19.3 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.2.0.0/16 \
--service-dns-domain=cluster.local \
--ignore-preflight-errors=Swap \
--ignore-preflight-errors=NumCPU
k8s-master成功装好
Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:===========================================================================================
#创建k8s集群配置文件
#制定了，默认的ssl证书在那？api-server的地址等mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config===========================================================================================
#pod分布再多机器上，pod互相之间链接，得部署，集群网络，选用flannel网络插件
#安装使用即可。
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/#使用如下命令，将k8s-node加入集群即可
Then you can join any number of worker nodes by running the following on each as root:kubeadm join 192.168.208.128:6443 --token n8zyxz.7w7kbbd1rjiygo21 \--discovery-token-ca-cert-hash sha256:e0f3fa4a1f0b2edd106828f586f7d9d78cb6ca51334f05456cc70a7ead7bceac 查看工作节点信息
[root@k8s-master-10 docker]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-10 NotReady master 26m v1.19.3
[root@k8s-master-10 docker]# kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master-10 NotReady master 27m v1.19.3 192.168.208.128 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://19.3.15
NotReady：未就绪，因为网络环境没配置；其他节点加入集群
kubeadm join 192.168.208.128:6443 --token n8zyxz.7w7kbbd1rjiygo21 \--discovery-token-ca-cert-hash sha256:e0f3fa4a1f0b2edd106828f586f7d9d78cb6ca51334f05456cc70a7ead7bceac 加入集群后，查看工作节点信息。查看k8s集群用到了那些节点。
[root@k8s-master-10 docker]# kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master-10 NotReady master 41m v1.19.3 192.168.208.128 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://19.3.15
k8s-node-11 NotReady <none> 8m21s v1.19.3 192.168.208.130 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://19.3.15
k8s-node-12 NotReady <none> 8m14s v1.19.3 192.168.208.129 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://19.3.15
如何让集群就绪？部署网络插件
#1、下载网络插件，配置文件，yaml以及配置文件
git clone --depth 1 https://github.com/coreos/flannel.git
#2、在k8s主节点上，应用这个yaml，基于yaml，创建具体的pod过程
#3、如果需要修改pod运行网络话，要修改配置文件
/root/flannel-master/Documentation/kube-flannel.yml#创建k8s资源，都是写这种yml文件了；
[root@k8s-master-10 Documentation]# grep 'Network' -A 5 kube-flannel.yml "Network": "10.2.0.0/16","Backend": {"Type": "vxlan"}}
---
--hostNetwork: truepriorityClassName: system-node-criticaltolerations:- operator: Existseffect: NoScheduleserviceAccountName: flannel
[root@k8s-master-10 Documentation]#修改第二处，夸主机的容器通信，最终不得走宿主机的物理网卡。
#告诉flannel物理网卡是谁
containers:- name: kube-flannelimage: docker.io/flannel/flannel:v0.22.3command:- /opt/bin/flanneldargs:- --ip-masq- --kube-subnet-mgr- --iface=ens33
基于kubectl命令，应用这个yml文件，读取、以及创建pod资源
配置k8s命令补缺（重要）
k8s命令太多，务必要配置补全
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrckubectl create -f ./kube-flannel.yml
查看当前机器的容器，关于flannel网络插件的进程
[root@k8s-master-10 Documentation]# docker ps | grep flannel
53844bcdd167 e23f7ca36333 "/opt/bin/flanneld -…" 8 minutes ago Up 8 minutes k8s_kube-flannel_kube-flannel-ds-mpgf6_kube-flannel_9cf25cba-14b6-4768-8ff0-6ec821f22769_0
4a86528f1db0 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 9 minutes ago Up 9 minutes k8s_POD_kube-flannel-ds-mpgf6_kube-flannel_9cf25cba-14b6-4768-8ff0-6ec821f22769_0
status状态都是Ready，至此，所有机器，都走flannel进行集群通信了。
[root@k8s-master-10 Documentation]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-10 Ready master 4h53m v1.19.3
k8s-node-11 Ready <none> 4h20m v1.19.3
k8s-node-12 Ready <none> 4h20m v1.19.3
[root@k8s-master-10 Documentation]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master-10 Ready master 4h53m v1.19.3 192.168.208.128 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://19.3.15
k8s-node-11 Ready <none> 4h21m v1.19.3 192.168.208.130 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://19.3.15
k8s-node-12 Ready <none> 4h21m v1.19.3 192.168.208.129 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://19.3.15
创建一个Pod
[root@k8s-master-10 Documentation]# kubectl run linux0224-pod-1-nginx --image=nginx:1.14.1
pod/linux0224-pod-1-nginx created查看pod信息
[root@k8s-master-10 Documentation]# kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
linux0224-pod-1-nginx 1/1 Running 0 107s 10.2.1.2 k8s-node-11 <none> <none>
pod的ip是k8s集群，才能访问的一个ip，无法在外部访问，外部访问得设置更多访问规则
[root@k8s-master-10 Documentation]# curl 10.2.1.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>body {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8s-master-10 Documentation]#
[root@k8s-master-10 Documentation]# curl 10.2.1.2 -I
HTTP/1.1 200 OK
Server: nginx/1.14.1
Date: Sun, 29 Oct 2023 12:44:34 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 06 Nov 2018 13:28:31 GMT
Connection: keep-alive
ETag: "5be196ff-264"
Accept-Ranges: bytes
修改页面显示信息
直接在容器上修改
[root@k8s-node-11 docker]# docker exec 9395505ba67d sh -c "echo '<meta charset-utf-8> 书季辛苦了~' > /usr/share/nginx/html/index.html "[root@k8s-master-10 Documentation]# curl 10.2.1.2
<meta charset-utf-8> 书季辛苦了~容器集群上修改，基于k8s命令
[root@k8s-master-10 Documentation]# kubectl exec linux0224-pod-1-nginx -- sh -c "echo '辛苦了书季，散会，下午好好消化下' >/usr/share/nginx/html/index.html"
[root@k8s-master-10 Documentation]# curl 10.2.1.2
辛苦了书季，散会，下午好好消化下