[root@harbor ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@test02 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'test02 (192.168.0.17)' can't be established. ECDSA key fingerprint is SHA256:Omg7RAlyXPvDcwNyJdufEmAMHnwcS3eh/gsaHPZVP6I. ECDSA key fingerprint is MD5:5d:55:0f:b2:75:4d:39:ee:47:c1:a8:3f:0f:12:96:30. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@test02's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@test02'" and check to make sure that only the key(s) you wanted were added.
[root@test01 ~]# kubeadm init --apiserver-advertise-address=192.168.0.16 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.16.4 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16 [init] Using Kubernetes version: v1.16.4 [preflight] Running pre-flight checks [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [WARNING Hostname]: hostname "test01" could not be reached [WARNING Hostname]: hostname "test01": lookup test01 on 10.16.140.4:53: no such host [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [test01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.0.16] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [test01 localhost] and IPs [192.168.0.16 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [test01 localhost] and IPs [192.168.0.16 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 33.002262 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs1 [mark-control-plane] Marking the node test01 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node test01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: 7e169l.60ykxxr8md7sb8ak [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
[root@test01 ~]# kubectl apply -f kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created
[root@test02 yum.repos.d]# kubeadm join 192.168.0.16:6443 --token 7e169l.60ykxxr8md7sb8ak \ > --discovery-token-ca-cert-hash sha256:c0b04a34ac7bb55fdf5b04a70233c14c15a11341a2bde0ea33b30579d84c0ce4 [preflight] Running pre-flight checks [WARNING Hostname]: hostname "test02" could not be reached [WARNING Hostname]: hostname "test02": lookup test02 on 10.16.140.4:53: no such host [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@test01 ~]# kubectl apply -f dashboard-recommended.yaml namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created
[root@test01 ~]# kubectl get po -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-7b8b58dc8b-x5p9p 1/1 Running 0 18m kubernetes-dashboard-557f4b4587-zjglq 1/1 Running 0 18m [root@test01 ~]# [root@test01 ~]# kubectl get svc -n kubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.1.80.221 <none> 8000/TCP 21m kubernetes-dashboard NodePort 10.1.241.233 <none> 443:30001/TCP 21m
[WARNING Hostname]: hostname "test02" could not be reached [WARNING Hostname]: hostname "test02": lookup test02 on 10.16.140.4:53: no such host
问题2、kube-proxy没有开启ipvs，默认用了iptables Proxier
1 2 3 4 5 6 7 8 9
[root@test01 ~]# kubectl logs kube-proxy-fr7wk -n kube-system W0213 14:48:50.804540 1 proxier.go:592] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules W0213 14:48:50.805641 1 proxier.go:592] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules W0213 14:48:50.806607 1 proxier.go:592] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules W0213 14:48:50.807514 1 proxier.go:592] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules W0213 14:48:50.811786 1 server_others.go:330] Flag proxy-mode="" unknown, assuming iptables proxy I0213 14:48:50.819927 1 node.go:135] Successfully retrieved node IP: 192.168.0.18 I0213 14:48:50.819966 1 server_others.go:150] Using iptables Proxier. I0213 14:48:50.820484 1 server.go:529] Version: v1.16.4
问题3、etcd有异常日志出现
1 2 3 4 5 6 7 8
2020-02-13 14:48:04.732974 W | wal: sync duration of 1.126809415s, expected less than 1s 2020-02-13 14:48:04.733537 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:429" took too long (760.032104ms) to execute 2020-02-13 14:48:04.733800 W | etcdserver: read-only range request "key:\"/registry/minions/test02\" " with result "range_response_count:0 size:5" took too long (975.286916ms) to execute 2020-02-13 14:48:04.733918 W | etcdserver: read-only range request "key:\"/registry/jobs/\" range_end:\"/registry/jobs0\" limit:500 " with result "range_response_count:0 size:5" took too long (332.81178ms) to execute 2020-02-13 14:48:04.734030 W | etcdserver: read-only range request "key:\"/registry/pods\" range_end:\"/registry/podt\" count_only:true " with result "range_response_count:0 size:7" took too long (377.827782ms) to execute 2020-02-13 14:48:04.734145 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:447" took too long (431.157766ms) to execute 2020-02-13 14:48:04.734246 W | etcdserver: read-only range request "key:\"/registry/minions/test03\" " with result "range_response_count:0 size:5" took too long (1.05179271s) to execute 2020-02-13 14:48:04.734349 W | etcdserver: read-only range request "key:\"/registry/pods/\" range_end:\"/registry/pods0\" limit:500 " with result "range_response_count:8 size:18170" took too long (292.020624ms) to execute
问题4、api-server报错
1
I0213 16:31:39.167022 1 log.go:172] http: TLS handshake error from 192.168.0.10:59610: remote error: tls: bad certificate
问题5、kube-scheduler报错
1
User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
问题6、kubectl get cs问题（上游问题，影响版本从1.16之后。）
1 2 3 4 5 6 7 8 9 10 11
[root@test01 ~]# kubectl get cs NAME AGE scheduler <unknown> controller-manager <unknown> etcd-0 <unknown> #workround方案： [root@test01 ~]# kubectl get cs -o=go-template='{{printf "|NAME|STATUS|MESSAGE|\n"}}{{range .items}}{{$name := .metadata.name}}{{range .conditions}}{{printf "|%s|%s|%s|\n" $name .status .message}}{{end}}{{end}}' |NAME|STATUS|MESSAGE| |controller-manager|True|ok| |scheduler|True|ok| |etcd-0|True|{"health":"true"}|