单节点k8s—自签名证书—四层负载均衡—helm安装rancher
目录
一：环境准备
1、服务器
2、环境配置（两台都要）
二：RKE部署k8s集群
三：下载helm并配置自签名证书安装rancher
四：nginx四层负载均衡实现高可用rancher
一：环境准备
1、服务器
两台服务器：一台部署rancher服务（用的阿里云4C8G），另一台起nginx容器做负载均衡（2C4G）
| 主机名 | ip |
| rancher | 39.108.216.50 |
| nginx | 39.108.222.169 |
2、环境配置（两台都要）
修改主机名
hostnamectl set-hostname rancher/nginx下载常用模块（也可以不用下）
yum install -y lrzsz
yum install -y net-tools
yum install -y wget
yum install -y vim-enhanced
yum install -y bash-completion
yum install -y git
yum install -y telnet
yum install -y telnet-server关闭防火墙、selinux、交换分区
####全部建议永久关闭，避免重启后服务报错
# 关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
systemctl stop NetworkManager && systemctl disable NetworkManager
# 关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 临时
# 关闭swap
swapoff -a # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久启动ipv4转发
echo "
# 需要加载br_netfilter内核模块，用于遍历bridge的数据包，由iptables进行处理以进行过滤和端口转发，值为0则iptables不对bridge的数据进行处理（必要）
# 非ipv6场景不需要设置net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
# 允许接口间转发报文（必要）
net.ipv4.ip_forward=1
# 转发源路由帧，NAT建议开启
net.ipv4.conf.all.forwarding=1
" >> /etc/sysctl.conf
使之生效sudo sysctl -p下载docker
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum makecache fast
yum -y install docker-ce-18.06.3.ce-3.el7配置daemon.json
tee /etc/docker/daemon.json << EOF
{
"registry-mirrors" : [
"https://registry.docker-cn.com",
"https://docker.mirrors.ustc.edu.cn",
"http://hub-mirror.c.163.com",
"https://cr.console.aliyun.com/"
]
}
EOF然后使之生效：systemctl daemon-reload
设置开机自启并启动：systemctl start docker && systemctl enable docker
测试：docker run hello-world
显示 Hello from Docker! 即表示安装成功
二：RKE部署k8s集群
上传RKE包 包下载地址：Releases · rancher/rke · GitHub
安装命令
cp rke /usr/local/bin/rke
chmod +x /usr/local/bin/rke
rke version 测试下载kubectl
# 配置阿里云 Kubernetes yum 软件源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubectl-1.20.15创建用户
useradd -G docker rancher
echo -n rancher | passwd --stdin rancher
su - rancher建立免密通道
ssh-keygen -t rsa
ssh-copy-id localhost配置cluster.yml文件
[rancher@rancher ~]$ vim cluster.yml
[rancher@rancher ~]$ cat cluster.yml
nodes:
- address: 172.19.103.166 #服务器私网ip
user: rancher
role: [controlplane,etcd,worker]
kubernetes_version: v1.20.15-rancher2-2RKE安装k8s
rke up cluster.yml将 kube_config_cluster.yml 文件添加到系统变量中
echo export KUBECONFIG=/home/rancher/kube_config_cluster.yml >> ~/.bash_profile
source ~/.bash_profile测试kubectl get nodes
[rancher@rancher ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
39.108.216.50 Ready controlplane,etcd,worker 4m17s v1.20.15三：下载helm并配置自签名证书安装rancher
下载helm
# 下载 helm 二进制包
wget https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz
# 解压
tar -zxvf helm-v3.5.0-linux-amd64.tar.gz
# 这一步需要 root 用户操作，否则可能会有权限不足的问题
mv /home/rancher/linux-amd64/helm /usr/bin自签名证书脚本vim ssl.sh
#!/bin/bash -e
help ()
{
echo ' ================================================================ '
echo ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；'
echo ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；'
echo ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；'
echo ' --ssl-size: ssl加密位数，默认2048；'
echo ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
echo ' 使用示例:'
echo ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
echo ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
echo ' ================================================================'
}
case "$1" in
-h|--help) help; exit;;
esac
if [[ $1 == '' ]];then
help;
exit;
fi
CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
case "$key" in
--ssl-domain) SSL_DOMAIN=$value ;;
--ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
--ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
--ssl-size) SSL_SIZE=$value ;;
--ssl-date) SSL_DATE=$value ;;
--ca-date) CA_DATE=$value ;;
--ssl-cn) CN=$value ;;
esac
done
# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca
# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}
## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}
SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt
echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m | 生成 SSL Cert | \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"
if [[ -e ./${CA_KEY} ]]; then
echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m"
mv ${CA_KEY} "${CA_KEY}"-bak
openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi
if [[ -e ./${CA_CERT} ]]; then
echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m"
mv ${CA_CERT} "${CA_CERT}"-bak
openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi
echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOM
if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} || -n ${SSL_DOMAIN} ]]; then
cat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOM
IFS=","
dns=(${SSL_TRUSTED_DOMAIN})
dns+=(${SSL_DOMAIN})
for i in "${!dns[@]}"; do
echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
done
if [[ -n ${SSL_TRUSTED_IP} ]]; then
ip=(${SSL_TRUSTED_IP})
for i in "${!ip[@]}"; do
echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
done
fi
fi
echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}
echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}
echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
-CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
-days ${SSL_DATE} -extensions v3_req \
-extfile ${SSL_CONFIG}
echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/ /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/ /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/ /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/ /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/ /'
echo
echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/ /'
echo
echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt创建证书
./ssl.sh --ssl-domain=www.test.com --ssl-trusted-ip=39.108.216.50 --ssl-size=2048 --ssl-date=3650创建namespace，添加TLS secret
kubectl create ns cattle-system
kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=tls.crt \
--key=tls.key
kubectl -n cattle-system create secret generic tls-ca \
--from-file=cacerts.pem=./cacerts.pemhelm部署rancher
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm repo update
#修改对应域名
helm install rancher rancher-stable/rancher \
--namespace cattle-system --version 2.5.15\
--set hostname=rancher.my.org \
--set ingress.tls.source=secret \
--set privateCA=true本机设置域名解析
C:\Windows\System32\drivers\etc\hosts
39.108.216.50 rancher.my.org访问如图
四：nginx四层负载均衡实现高可用rancher
配置nginx.conf文件
vim /etc/nginx.conf
worker_processes 4;
worker_rlimit_nofile 40000;
events {
worker_connections 8192;
}
http {
server {
listen 80;
return 301 rancher.my.org;
}
}
stream {
upstream rancher_servers {
least_conn;
server 39.108.216.50:443 max_fails=3 fail_timeout=5s;
}
server {
listen 443;
proxy_pass rancher_servers;
}
}
起nginx容器
docker run -d --privileged --restart=unless-stopped \
-p 80:80 -p 443:443 \
-v /etc/nginx.conf:/etc/nginx/nginx.conf \
nginx:1.14本机配置nginx域名，并注释rancher域名
C:\Windows\System32\drivers\etc\hosts
#39.108.216.50 rancher.my.org
39.108.222.169 rancher.my.org访问nginx
