当前位置：首页 > news >正文

K8s安装doris踩坑记录

news 来源：原创 2024/5/10 8:34:52

K8s安装Doris

官方文档： https://doris.apache.org/zh-CN/docs/install/k8s-deploy/

整体流程：配置yml文件，k8s执行apply命令

以官方文档为基础，进行修改。官方文档镜像拉取存在问题，直接拉取test会出现拉取异常。

以下是一个fe节点和一个be节点的配置文件参考：

doris_follower.yml:

apiVersion: v1
kind: Service
metadata:name: doris-follower-cluster1labels:app: doris-follower-cluster1
spec:ports:- port: 8030name: http-port- port: 9020name: rpc-port- port: 9030name: query-port- port: 9010name: edit-log-port #This name should be fixed. Doris will get the port information through this nameclusterIP: Noneselector:app: doris-follower-cluster1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: doris-follower-cluster1labels:app: doris-follower-cluster1
spec:selector:matchLabels:app: doris-follower-cluster1serviceName: doris-follower-cluster1# 修改创建副本数量为1replicas: 1template:metadata:name: doris-follower-cluster1labels:app: doris-follower-cluster1spec:## 给当前pod加入host#hostAliases:#- ip: "x"#  hostnames:#  - "apiserver.cluster.local"containers:- name: doris-follower-cluster1#修改镜像 依据https://hub.docker.com/r/apache/doris/tags获取image: apache/doris:2.0.0_alpha-fe-x86_64imagePullPolicy: IfNotPresentenv:# 增加APP_NAMESPACE与FE_IPADDRESS环境变量 - name: APP_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: FE_IPADDRESSvalueFrom:fieldRef:fieldPath: status.podIP#Specify the startup type as k8s to bypass some restrictions of the official image initialization script- name: BUILD_TYPEvalue: "k8s"#Initialize the fe of three nodes- name: FE_INIT_NUMBER# fe创建数量改为1value: "1"#ServiceName of bakend_cn node,（if do not have bakend_cn node,do not configure this environment variable）- name: CN_SERVICEvalue: "doris-cn-cluster1"#StatefulSetName of bakend_cn node,（if do not have bakend_cn node,do not configure this environment variable)- name: CN_STATEFULSETvalue: "doris-cn-cluster1"#ServiceName of bakend node,（if do not have bakend node,do not configure this environment variable）- name: BE_SERVICEvalue: "doris-be-cluster1"#StatefulSetName of bakend node,（if do not have bakend node,do not configure this environment variable)- name: BE_STATEFULSETvalue: "doris-be-cluster1"#ServiceName of follower node,（if do not have follower node,do not configure this environment variable）- name: FE_SERVICEvalue: "doris-follower-cluster1"##StatefulSetName of follower node,（if do not have follower node,do not configure this environment variable)- name: FE_STATEFULSETvalue: "doris-follower-cluster1"ports:- containerPort: 8030name: http-port- containerPort: 9020name: rpc-port- containerPort: 9030name: query-port- containerPort: 9010name: edit-log-portvolumeMounts:#Mount the configuration file in the way of configmap- name: confmountPath: /opt/apache-doris/fe/conf#In order to call the api of k8s- name: kubemountPath: /root/.kube/configreadOnly: truevolumes:- name: confconfigMap:name: follower-conf- name: kubehostPath:path: /root/.kube/config
---
apiVersion: v1
kind: ConfigMap
metadata:name: follower-conf
data:fe.conf: |# 更改为对应的k8s网络地址priority_networks = 172.16.0.0/24#It can automatically maintain node information by getting the number of replicas of StatefulSet, similar to alter system add/drop backenable_deploy_manager = k8s#Automatically adjust the IP of the node according to the domain name (for example, after the pod is restarted, the domain name is still doris-be-cluster1-0-doris-be-cluster1.default.svc.cluster.local, but the IP may change from 172.16.0.9 to 172.16.0.10)enable_fqdn_mode = trueLOG_DIR = ${DORIS_HOME}/logsys_log_level = INFOhttp_port = 8030rpc_port = 9020query_port = 9030edit_log_port = 9010#Doris needs to generate the log4j configuration file according to the fe.yml configuration information, which is written in the same directory as fe.yml by default, but the config we mount is readonly, so specify this configuration to write the log4j file to another locationcustom_config_dir = /opt/apache-doris/#when set to false, the backend will not be dropped and remaining in DECOMMISSION statedrop_backend_after_decommission = false

k8s对应的网络地址请修改为使用命令获取pod的CIDR参数–cluster-cidr

ps -ef | grep kube-controller-manager

如果您没有直接访问控制平面节点的权限，可以使用 kubectl 命令：

kubectl -n kube-system get pods -l component=kube-controller-manager -o yaml

使用命令启动fe

kubectl create -f doris_follower.yml

可能出现异常和问题处理：

Failed to pull image “apache-doris-fe:test”: rpc error: code = Unknown desc = failed to pull and unpack image
检查镜像是否填写正常
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [StatefulSet] with name: [doris-follower-cluster1] in namespace: [default] failed.
- 首先检查命名空间是否正确，如果设定了命名空间但是sts还是去default获取，检查APP_NAMESPACE是否增加
- 如果已经获取到对应命名空间，但是sts还是报错，请为当前namespace添加role权限
- 如果出现 Caused by: java.net.UnknownHostException: apiserver.cluster.local: Name or service not known 请检查/root/.kube/config 是否写了 server: https://apiserver.cluster.local:6443 地址，并且检查/etc/hosts 对应地址，将它加入hostAliases中
current node is not added to the group. please add it first
检查网络地址priority_networks是否正确

doris_be.yml:

apiVersion: v1
kind: Service
metadata:name: doris-be-cluster1labels:app: doris-be-cluster1
spec:ports:- port: 9060name: be-port- port: 8040name: webserver-port- port: 9050name: heartbeat-port #This name should be fixed. Doris will get the port information through this name- port: 8060name: brpc-portclusterIP: Noneselector:app: doris-be-cluster1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: doris-be-cluster1labels:app: doris-be-cluster1
spec:selector:matchLabels:app: doris-be-cluster1serviceName: doris-be-cluster1# 修改创建副本数量replicas: 1template:metadata:name: doris-be-cluster1labels:app: doris-be-cluster1spec:containers:- name: doris-be-cluster1# 修改镜像地址image: apache/doris:2.0.0_alpha-be-x86_64imagePullPolicy: IfNotPresentenv:#Specify the startup type as k8s to bypass some restrictions of the official image initialization script- name: BUILD_TYPEvalue: "k8s"# 写明FE的IP与端口 (FQDN)- name: FE_MASTER_IPvalue: "doris-follower-cluster1-0.doris-follower-cluster1.default.svc.cluster.local"- name: FE_MASTER_PORTvalue: "9030"ports:- containerPort: 9060name: be-port- containerPort: 8040name: webserver-port- containerPort: 9050name: heartbeat-port- containerPort: 8060name: brpc-portvolumeMounts:#Mount the configuration file in the way of configmap- name: confmountPath: /opt/apache-doris/be/conf#Ifnot mounted, when enable_profile, error will be reported when querying the data from jdbc catalog#Error message: error setting certificate verify locations: CAfile:/etc/pki/tls/certs/ca-bundle.crt CApath: none- name: sysmountPath: /etc/pkireadOnly: truevolumes:- name: confconfigMap:name: be-conf- name: syshostPath:path: /etc/pki
---
apiVersion: v1
kind: ConfigMap
metadata:name: be-conf
data:be.conf: |PPROF_TMPDIR="$DORIS_HOME/log/"sys_log_level = INFObe_port = 9060webserver_port = 8040heartbeat_service_port = 9050brpc_port = 8060# 修改为k8s网络priority_networks = 172.16.0.0/24

使用命令启动be

kubectl create -f doris_be.yml

可能出现异常和问题处理：

be找不到fe
检查服务的完全限定域名 (FQDN) 是否配置错误

doris-svc.yaml
对外暴露端口提供访问

apiVersion: v1
kind: Service
metadata:name: doris-svcnamespace: udm
spec:type: NodePortports:- port: 8030nodePort: 8030name: "p8030"- port: 9030nodePort: 9030name: "p9030"- port: 8040nodePort: 8040name: "p8040"selector:app: doris-follower-cluster1