Deploy KubeSphere on QingCloud Instances
Introduction
For a production environment, you need to consider the high availability of the cluster. If key components (for example, kube-apiserver, kube-scheduler, and kube-controller-manager) are all running on the same control plane node, Kubernetes and KubeSphere will be unavailable once the control plane node goes down. Therefore, you need to set up a high-availability cluster by provisioning load balancers with multiple control plane nodes. You can use any cloud load balancer, or any hardware load balancer (for example, F5). In addition, Keepalived and HAproxy, or Nginx is also an alternative for creating high-availability clusters.
This tutorial walks you through an example of how to create two QingCloud load balancers, serving as the internal load balancer and external load balancer respectively, and of how to implement high availability of control plane and etcd nodes using the load balancers.
Prerequisites
- Make sure you already know how to install KubeSphere on a multi-node cluster by following the guide. For detailed information about the configuration file that is used for installation, see Edit the configuration file. This tutorial focuses more on how to configure load balancers.
- You need a QingCloud account to create load balancers, or follow the guide of any other cloud provider to create load balancers.
- For a production environment, it is recommended that you prepare persistent storage and create a StorageClass in advance. For development and testing, you can use the integrated OpenEBS to provision LocalPV as the storage service directly.
Architecture
This example prepares six machines of Ubuntu 16.04.6. You will create two load balancers, and deploy three control plane nodes and etcd nodes on three of the machines. You can configure these control plane and etcd nodes in config-sample.yaml
created by KubeKey (Please note that this is the default name, which can be changed by yourself).
Note
Install an HA Cluster
Step 1: Create load balancers
This step demonstrates how to create load balancers on the QingCloud platform.
Create an internal load balancer
Log in to the QingCloud console. In the menu on the left, under Network & CDN, select Load Balancers. Click Create to create a load balancer.
In the pop-up window, set a name for the load balancer. Choose the VxNet where your machines are created from the Network drop-down list. Here is
pn
. Other fields can be default values as shown below. Click Submit to finish.Click the load balancer. On the detail page, create a listener that listens on port
6443
with the Listener Protocol set toTCP
.- Name: Define a name for this Listener
- Listener Protocol: Select
TCP
protocol - Port:
6443
- Balance mode:
Poll
Click Submit to continue.
Note
After you create the listener, check the firewall rules of the load balancer. Make sure that port6443
has been added to the firewall rules and that external traffic is allowed to port6443
. Otherwise, the installation will fail. You can find the information in Security Groups under Security on the QingCloud platform.Click Add Backend, and choose the VxNet you just selected (in this example, it is
pn
). Click Advanced Search, choose the three control plane nodes, and set the port to6443
which is the default secure port of api-server.Click Submit when you finish.
Click Apply Changes to use the configurations. At this point, you can find the three control plane nodes have been added as the backend servers of the listener that is behind the internal load balancer.
Note
The status of all control plane nodes might show Not Available after you added them as backends. This is normal since port6443
of api-server is not active on control plane nodes yet. The status will change to Active and the port of api-server will be exposed after the installation finishes, which means the internal load balancer you configured works as expected.Record the Intranet VIP shown under Networks. The IP address will be added later to the configuration file.
Create an external load balancer
You need to create an EIP in advance. To create an EIP, go to Elastic IPs under Networks & CDN.
Note
Similarly, create an external load balancer while don't select VxNet for the Network field. Bind the EIP that you created to this load balancer by clicking Add IPv4.
On the load balancer's detail page, create a listener that listens on port
30880
(NodePort of KubeSphere console) with Listener Protocol set toHTTP
.Note
After you create the listener, check the firewall rules of the load balancer. Make sure that port30880
has been added to the firewall rules and that external traffic is allowed to port30880
. Otherwise, the installation will fail. You can find the information in Security Groups under Security on the QingCloud platform.Click Add Backend. In Advanced Search, choose the
six
machines on which you are going to install KubeSphere within the VxNetpn
, and set the port to30880
.Click Submit when you finish.
Click Apply Changes to use the configurations. At this point, you can find the six machines have been added as the backend servers of the listener that is behind the external load balancer.
Step 2: Download KubeKey
Kubekey is the next-gen installer which provides an easy, fast and flexible way to install Kubernetes and KubeSphere.
Follow the step below to download KubeKey.
Download KubeKey from its GitHub Release Page or use the following command directly.
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.7 sh -
Run the following command first to make sure you download KubeKey from the correct zone.
export KKZONE=cn
Run the following command to download KubeKey:
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.7 sh -
Note
export KKZONE=cn
again before you proceed with the steps below.Note
Make kk
executable:
chmod +x kk
Create an example configuration file with default configurations. Here Kubernetes v1.22.12 is used as an example.
./kk create config --with-kubesphere v3.3.2 --with-kubernetes v1.22.12
Note
Recommended Kubernetes versions for KubeSphere 3.3: v1.20.x, v1.21.x, * v1.22.x, * v1.23.x, and * v1.24.x. For Kubernetes versions with an asterisk, some features of edge nodes may be unavailable due to incompatability. Therefore, if you want to use edge nodes, you are advised to install Kubernetes v1.21.x. If you do not specify a Kubernetes version, KubeKey will install Kubernetes v1.23.10 by default. For more information about supported Kubernetes versions, see Support Matrix.
If you do not add the flag
--with-kubesphere
in the command in this step, KubeSphere will not be deployed unless you install it using theaddons
field in the configuration file or add this flag again when you use./kk create cluster
later.If you add the flag
--with-kubesphere
without specifying a KubeSphere version, the latest version of KubeSphere will be installed.
Step 3: Set cluster nodes
As you adopt the HA topology with stacked control plane nodes, the control plane nodes and etcd nodes are on the same three machines.
Property | Description |
---|---|
hosts | Detailed information of all nodes |
etcd | etcd node names |
control-plane | Control plane node names |
worker | Worker node names |
Put the control plane nodes (master1
, master2
and master3
) under etcd
and master
respectively as below, which means these three machines will serve as both the control plane and etcd nodes. Note that the number of etcd needs to be odd. Meanwhile, it is not recommended that you install etcd on worker nodes since the memory consumption of etcd is very high.
config-sample.yaml Example
spec:
hosts:
- {name: master1, address: 192.168.0.2, internalAddress: 192.168.0.2, user: ubuntu, password: Testing123}
- {name: master2, address: 192.168.0.3, internalAddress: 192.168.0.3, user: ubuntu, password: Testing123}
- {name: master3, address: 192.168.0.4, internalAddress: 192.168.0.4, user: ubuntu, password: Testing123}
- {name: node1, address: 192.168.0.5, internalAddress: 192.168.0.5, user: ubuntu, password: Testing123}
- {name: node2, address: 192.168.0.6, internalAddress: 192.168.0.6, user: ubuntu, password: Testing123}
- {name: node3, address: 192.168.0.7, internalAddress: 192.168.0.7, user: ubuntu, password: Testing123}
roleGroups:
etcd:
- master1
- master2
- master3
control-plane:
- master1
- master2
- master3
worker:
- node1
- node2
- node3
For a complete configuration sample explanation, see this file.
Step 4: Configure the load balancer
In addition to the node information, you need to provide the load balancer information in the same YAML file. For the Intranet VIP address, you can find it in the last part when you create an internal load balancer. Assume the VIP address and listening port of the internal load balancer are 192.168.0.253
and 6443
respectively, and you can refer to the following example.
The configuration example in config-sample.yaml
## Internal LB config example
## apiserver_loadbalancer_domain_name: "lb.kubesphere.local"
controlPlaneEndpoint:
domain: lb.kubesphere.local
address: "192.168.0.253"
port: 6443
Note
- The address and port should be indented by two spaces in
config-sample.yaml
, and the address should be VIP. - The domain name of the load balancer is
lb.kubesphere.local
by default for internal access. If you need to change the domain name, uncomment and modify it.
Step 5: Kubernetes cluster configurations (Optional)
Kubekey provides some fields and parameters to allow the cluster administrator to customize Kubernetes installation, including Kubernetes version, network plugins and image registry. There are some default values provided in config-sample.yaml
. You can modify Kubernetes-related configurations in the file based on your needs. For more information, see Kubernetes Cluster Configurations.
Step 6: Persistent storage plugin configurations
Considering data persistence in a production environment, you need to prepare persistent storage and configure the storage plugin (for example, CSI) in config-sample.yaml
to define which storage service you want.
Note
Available storage plugins and clients
- Ceph RBD & CephFS
- GlusterFS
- QingCloud CSI
- QingStor CSI
- More plugins will be supported in future releases
Make sure you have configured the storage plugin before you get started. KubeKey will create a StorageClass and persistent volumes for related workloads during the installation. For more information, see Persistent Storage Configurations.
Step 7: Enable pluggable components (Optional)
KubeSphere has decoupled some core feature components since v2.1.0. These components are designed to be pluggable which means you can enable them either before or after installation. By default, KubeSphere will be installed with the minimal package if you do not enable them.
You can enable any of them according to your demands. It is highly recommended that you install these pluggable components to discover the full-stack features and capabilities provided by KubeSphere. Make sure your machines have sufficient CPU and memory before you enable them. See Enable Pluggable Components for details.
Step 8: Start to bootstrap a cluster
After you complete the configuration, you can execute the following command to start the installation:
./kk create cluster -f config-sample.yaml
Step 9: Verify the installation
Inspect the logs of installation. When you see output logs as follows, it means KubeSphere has been successfully deployed.
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
#####################################################
### Welcome to KubeSphere! ###
#####################################################
Console: http://192.168.0.3:30880
Account: admin
Password: P@88w0rd
NOTES:
1. After you log into the console, please check the
monitoring status of service components in
the "Cluster Management". If any service is not
ready, please wait patiently until all components
are up and running.
2. Please change the default password after login.
#####################################################
https://kubesphere.io 2020-08-13 10:50:24
#####################################################
Step 10: Verify the HA cluster
Now that you have finished the installation, go back to the detail page of both the internal and external load balancers to see the status.
Both listeners show that the status is Active, meaning nodes are up and running.
In the web console of KubeSphere, you can also see that all the nodes are functioning well.
To verify if the cluster is highly available, you can turn off an instance on purpose. For example, the above console is accessed through the address IP: 30880
(the EIP address here is the one bound to the external load balancer). If the cluster is highly available, the console will still work well even if you shut down a control plane node.
See Also
Kubernetes Cluster Configurations
Persistent Storage Configurations
Feedback
Was this page Helpful?
Receive the latest news, articles and updates from KubeSphere
Thanks for the feedback. If you have a specific question about how to use KubeSphere, ask it on Slack. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement.