Centos搭建Docker和Kubernetes

搭建Docker+Kubernetes环境，然后部署微服务，最后开放对外访问。

由于网络问题，可能下载安装过程中会失败，耐心多安装几次就OK了。

安装一定要有耐心！！！安装一定要有耐心！！！安装一定要有耐心！！！

环境

Centos：7.5

Docker：19.03

Kubernetes：1.1.18

准备两台CentOS机器，这里为了测试，只准备两台，1台做Master，1台做Worker。

安装Docker

使用脚本安装Docker

使用root登录Centos

确保yum包更新到最新

yum update

执行Docker安装脚本

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

执行这个脚本会添加 docker.repo 源并安装 Docker。

启动Docker并且设置为开机启动

systemctl start docker
systemctl enable docker

至此Docker在Centos系统的安装已完成。

配置镜像加速器

鉴于国内网络问题，拉取Docker镜像十分缓慢，我们需要配置加速器来提速。

增加配置：

vim /etc/docker/daemon.json

{
  "registry-mirrors": ["https://847pb1vj.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

生效配置、重启Docker：

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

关闭防火墙

systemctl stop firewalld	# 关闭服务
systemctl disable firewalld	# 禁用服务

禁用SELinux

修改/etc/selinux/config，设置SELINUX=disabled，重启机器。

禁用交换分区

swapoff -a
vim /etc/sysconfig/kubelet

修改为：KUBELET_EXTRA_ARGS=”–fail-swap-on=false”

安装Kubernetes

安装kubelet、kubeadm、kubectl、dashboard、添加woker。

添加源

由于国内网络原因，官方文档中的地址不可用，本文替换为阿里云镜像地址：

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF

安装

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet && systemctl start kubelet

修改网络配置

cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF
sysctl --system

注意：至此，以上的全部操作，在Worker机器上也需要执行，包括安装Docker。

初始化Master

生成初始化文件

kubeadm config print init-defaults > kubeadm-init.yaml

该文件有3处需要修改：

将advertiseAddress: 1.2.3.4修改为本机地址（***注意是内网地址***）
将imageRepository: k8s.gcr.io修改为imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
nodeRegistration:name，改为k8s-master（woker节点的话，则修改为别的名称）

修改完毕之后：

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.237
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.15.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

下载镜像

kubeadm config images pull --config kubeadm-init.yaml

输入如下信息：

执行初始化

kubeadm init --config kubeadm-init.yaml

等待执行完毕后, 会输出如下内容：

W0328 21:31:14.954124   17338 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.0
[preflight] Running pre-flight checks
	[WARNING Hostname]: hostname "k8s-master" could not be reached
	[WARNING Hostname]: hostname "k8s-master": lookup k8s-master on 100.125.1.250:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.237]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.237 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.237 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0328 21:31:18.364428   17338 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0328 21:31:18.365138   17338 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.501912 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.0.237:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:964c7b0c007ce17c979e631da17ad047dfa3bad76e407b6ee76d729ecf3cd9c7

最后两行需要保存下来， kubeadm join ...是Worker节点加入所需要执行的命令。

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

注意此处：可能安装时一直等待，最后超时，错误如下。

可能是因为kubeadm-init.yaml中advertiseAddress没配置成内网，导致网络问题。

接下来配置环境，让当前用户可以执行kubectl命令:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

测试一下：此处的NotReady是因为网络还没配置。

[root@ecs-babc-0004 kubernetes]$ kubectl get node
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   2m25s   v1.17.3

配置网络

安装calico网络插件：

[root@ecs-babc-0004 ~]$ wget https://docs.projectcalico.org/v3.9/manifests/calico.yaml
[root@ecs-babc-0004 ~]$ cat kubeadm-init.yaml | grep serviceSubnet: serviceSubnet: 10.96.0.0/12

vim calico.yaml，将192.168.0.0/16修改为10.96.0.0/12

需要注意的是, calico.yaml中的IP和kubeadm-init.yaml需要保持一致, 要么初始化前修改kubeadm-init.yaml, 要么初始化后修改calico.yaml.

执行kubectl apply -f calico.yaml初始化网络。

[root@ecs-babc-0004 kubernetes]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

过几分钟再查看node信息，master的状态已经是Ready了。

[root@ecs-babc-0004 kubernetes]# kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   11m   v1.18.0

此处有坑，半支烟被坑了好久，后来才明白，是因为拉取不到calico/cni:v3.8.8镜像，或者拉取很慢，找到坑之后，果断换版本calico/cni:v3.9.5，分分钟就OK了。

以下是排查过程：

首先执行kubectl get pods -n kube-system发现pods一直pending，发现镜像一直在pulling，有时1个小时都拉取不下来。

异常情况：

而正常情况应该是：

[root@ecs-babc-0004 kubernetes]# kubectl get pods  -n kube-system
NAME                                       READY   STATUS     RESTARTS   AGE
calico-kube-controllers-5fc5dbfc47-4gvcb   0/1     Pending    0          43s
calico-node-gvnnc                          0/1     Init:2/3   0          43s
coredns-546565776c-sx5rj                   0/1     Pending    0          6m52s
coredns-546565776c-xtb92                   0/1     Pending    0          6m52s
etcd-k8s-master                            1/1     Running    0          6m51s
kube-apiserver-k8s-master                  1/1     Running    0          6m51s
kube-controller-manager-k8s-master         1/1     Running    0          6m51s
kube-proxy-7jk4h                           1/1     Running    0          6m52s
kube-scheduler-k8s-master                  1/1     Running    0          6m50s

再执行kubectl describe po calico-node-gvnnc -n kube-system查看：

Events:
  Type    Reason     Age    From                 Message
  ----    ------     ----   ----                 -------
  Normal  Scheduled  2m14s  default-scheduler    Successfully assigned kube-system/calico-node-gvnnc to k8s-master
  Normal  Pulling    2m14s  kubelet, k8s-master  Pulling image "calico/cni:v3.9.5"
  Normal  Pulled     99s    kubelet, k8s-master  Successfully pulled image "calico/cni:v3.9.5"
  Normal  Created    98s    kubelet, k8s-master  Created container upgrade-ipam
  Normal  Started    98s    kubelet, k8s-master  Started container upgrade-ipam
  Normal  Pulled     97s    kubelet, k8s-master  Container image "calico/cni:v3.9.5" already present on machine
  Normal  Created    97s    kubelet, k8s-master  Created container install-cni
  Normal  Started    97s    kubelet, k8s-master  Started container install-cni
  Normal  Pulling    96s    kubelet, k8s-master  Pulling image "calico/pod2daemon-flexvol:v3.9.5"
  Normal  Pulled     74s    kubelet, k8s-master  Successfully pulled image "calico/pod2daemon-flexvol:v3.9.5"
  Normal  Created    74s    kubelet, k8s-master  Created container flexvol-driver
  Normal  Started    74s    kubelet, k8s-master  Started container flexvol-driver
  Normal  Pulling    74s    kubelet, k8s-master  Pulling image "calico/node:v3.9.5"
  Normal  Pulled     47s    kubelet, k8s-master  Successfully pulled image "calico/node:v3.9.5"
  Normal  Created    47s    kubelet, k8s-master  Created container calico-node
  Normal  Started    47s    kubelet, k8s-master  Started container calico-node

安装Dashboard

部署Dashboard

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml

kubectl apply -f recommended.yaml

部署完毕后, 执行kubectl get pods --all-namespaces查看pods状态，过几分钟全部变成Running

[root@ecs-babc-0004 kubernetes]# kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
kube-system            calico-kube-controllers-5fc5dbfc47-4gvcb     1/1     Running             0          5m15s
kube-system            calico-node-gvnnc                            1/1     Running             0          5m15s
kube-system            coredns-546565776c-sx5rj                     1/1     Running             0          11m
kube-system            coredns-546565776c-xtb92                     1/1     Running             0          11m
kube-system            etcd-k8s-master                              1/1     Running             0          11m
kube-system            kube-apiserver-k8s-master                    1/1     Running             0          11m
kube-system            kube-controller-manager-k8s-master           1/1     Running             0          11m
kube-system            kube-proxy-7jk4h                             1/1     Running             0          11m
kube-system            kube-scheduler-k8s-master                    1/1     Running             0          11m
kubernetes-dashboard   dashboard-metrics-scraper-66b49655d4-bb6px   0/1     ContainerCreating   0          8s
kubernetes-dashboard   kubernetes-dashboard-74b4487bfc-5dw84        0/1     ContainerCreating   0          8s

创建用户

创建一个用于登录Dashboard的用户，创建文件dashboard-adminuser.yaml内容如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

执行命令kubectl apply -f dashboard-adminuser.yaml

生成证书

官方文档中提供了登录1.7.X以上版本的登录方式，但并不清晰，半支烟没有完全按照该文档的方式进行操作。进入到.kube目录中执行3个命令：

[root@ecs-babc-0004 .kube]# grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
[root@ecs-babc-0004 .kube]# grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
[root@ecs-babc-0004 .kube]# openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"

第三条命令生成证书时会提示输入密码，可以直接两次回车跳过，然后再查看文件列表，看到多了3个证书相关文件：

kubecfg.p12即需要导入客户端机器的证书，将证书拷贝到客户端机器上，导入即可，半支烟的电脑是MAC，直接在~目录下执行以下命令即可，（xxx.xxx.xxx.xxx，是服务器的外网ip）

scp root@xxx.xxx.xxx.xxx:/root/.kube/kubecfg.p12 ./

MAC电脑，下载完p12文件，需要双击安装证书。安装证书时输入证书密码即可。

此时我们可以登录面板了，访问地址： https://{k8s-master-ip}:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login，登录时会提示选择证书，确认后会提示输入当前用户名密码(注意是电脑的用户名密码)。此处的{k8s-master-ip}是指服务器的外网ip。

然后会弹出dashboard的登录界面：

登录Dashboard

在服务器上的.kube目录下执行kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')，获取Token。

[root@ecs-babc-0004 .kube]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Name:         admin-user-token-9gjgz
Namespace:    kube-system
Labels:       <none>
Annotations: kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: a70bac13-dc07-49d2-9f4d-4296654ad66f

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWRoaGtiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiMjBkMTE0My1jZTk0LTQzNzktOWUxNC04ZjgwZjA2ZDg0NzkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.f6IbPGwIdFZWStzBj8_vmF01oWW5ccaCpPuVQNLSK1pgEqn0kNVK_x0RYSuKEnujObzpQQdFiRYcI6ITHja2PIVc5Nv83VCn5IaLvZdYuGZWUYRw0efJUBMA4J4N8-pRkiw6fYAuWLeGYghLNXL_nDdC_JkG75ASqrr3U1MVaikOcfrEPaI-T_AJ3TMYhI8aFoKiERpumu5W1K6Jl80Am9pWDX0Ywis5SSUP1VYfu-coI48EXSptcaxEyv58PrHUd6t_oMVV9rpqSxrNtMZvMeXqe8Hnl21vR7ls5yTZegYtHXSc3PKvCaIalKhYXAuhogNcIXHaMzvLSbf-DSQkVw

复制该Token到登录页，点击登录即可，主界面如下：

添加Worker

重复执行 安装Docker ~ 安装Kubernetes-修改网络配置 的全部操作，初始化一个Worker机器。

执行如下命令将Worker加入集群：

kubeadm join 192.168.0.237:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:964c7b0c007ce17c979e631da17ad047dfa3bad76e407b6ee76d729ecf3cd9c7

注意: 此处的秘钥是初始化Master后生成的, 参考前文.

添加完毕后，在Master上查看Worker节点状态：

[root@ecs-babc-0004 ~]# kubectl get nodes
NAME            STATUS   ROLES    AGE    VERSION
ecs-babc-0006   Ready    <none>   93s    v1.18.0
k8s-master      Ready    master   174m   v1.18.0