Air-gapped Installation
KubeKey is an open-source, lightweight tool for deploying Kubernetes clusters. It allows you to install Kubernetes/K3s only, both Kubernetes/K3s and KubeSphere, and other cloud-native plugins in a flexible, fast, and convenient way. Additionally, it is an effective tool for scaling and upgrading clusters.
In KubeKey v2.1.0, we bring in concepts of manifest and artifact, which provides a solution for air-gapped installation of Kubernetes clusters. A manifest file describes information of the current Kubernetes cluster and defines content in an artifact. Previously, users had to prepare deployment tools, image (.tar) file, and other binaries as the Kubernetes version and image to deploy are different. Now, with KubeKey, air-gapped installation can never be so easy. You simply use a manifest file to define what you need for your cluster in air-gapped environments, and then export the artifact file to quickly and easily deploy image registries and Kubernetes cluster.
Prerequisites
Host IP | Host Name | Usage |
---|---|---|
192.168.0.2 | node1 | Online host for packaging the source cluster |
192.168.0.3 | node2 | Control plane node of the air-gapped environment |
192.168.0.4 | node3 | Image registry node of the air-gapped environment |
Preparations
Run the following commands to download KubeKey.
Download KubeKey from its GitHub Release Page or use the following command directly.
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.13 sh -
Run the following command first to make sure you download KubeKey from the correct zone.
export KKZONE=cn
Run the following command to download KubeKey:
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.13 sh -
On the online host, run the following command and copy content in the manifest-example.
vim manifest.yaml
--- apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Manifest metadata: name: sample spec: arches: - amd64 operatingSystems: - arch: amd64 type: linux id: centos version: "7" repository: iso: localPath: url: https://github.com/kubesphere/kubekey/releases/download/v3.0.10/centos7-rpms-amd64.iso - arch: amd64 type: linux id: ubuntu version: "20.04" repository: iso: localPath: url: https://github.com/kubesphere/kubekey/releases/download/v3.0.10/ubuntu-20.04-debs-amd64.iso kubernetesDistributions: - type: kubernetes version: v1.23.15 components: helm: version: v3.9.0 cni: version: v1.2.0 etcd: version: v3.4.13 calicoctl: version: v3.23.2 ## For now, if your cluster container runtime is containerd, KubeKey will add a docker 20.10.8 container runtime in the below list. ## The reason is KubeKey creates a cluster with containerd by installing a docker first and making kubelet connect the socket file of containerd which docker contained. containerRuntimes: - type: docker version: 20.10.8 - type: containerd version: 1.6.4 crictl: version: v1.24.0 docker-registry: version: "2" harbor: version: v2.5.3 docker-compose: version: v2.2.2 images: - docker.io/kubesphere/kube-apiserver:v1.23.15 - docker.io/kubesphere/kube-controller-manager:v1.23.15 - docker.io/kubesphere/kube-proxy:v1.23.15 - docker.io/kubesphere/kube-scheduler:v1.23.15 - docker.io/kubesphere/pause:3.6 - docker.io/coredns/coredns:1.8.6 - docker.io/calico/cni:v3.23.2 - docker.io/calico/kube-controllers:v3.23.2 - docker.io/calico/node:v3.23.2 - docker.io/calico/pod2daemon-flexvol:v3.23.2 - docker.io/calico/typha:v3.23.2 - docker.io/kubesphere/flannel:v0.12.0 - docker.io/openebs/provisioner-localpv:3.3.0 - docker.io/openebs/linux-utils:3.3.0 - docker.io/library/haproxy:2.3 - docker.io/kubesphere/nfs-subdir-external-provisioner:v4.0.2 - docker.io/kubesphere/k8s-dns-node-cache:1.15.12 - docker.io/kubesphere/ks-installer:v3.4.1 - docker.io/kubesphere/ks-apiserver:v3.4.1 - docker.io/kubesphere/ks-console:v3.4.1 - docker.io/kubesphere/ks-controller-manager:v3.4.1 - docker.io/kubesphere/kubectl:v1.22.0 - docker.io/kubesphere/kubectl:v1.21.0 - docker.io/kubesphere/kubectl:v1.20.0 - docker.io/kubesphere/kubefed:v0.8.1 - docker.io/kubesphere/tower:v0.2.1 - docker.io/minio/minio:RELEASE.2019-08-07T01-59-21Z - docker.io/minio/mc:RELEASE.2019-08-07T23-14-43Z - docker.io/csiplugin/snapshot-controller:v4.0.0 - docker.io/kubesphere/nginx-ingress-controller:v1.1.0 - docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.4 - docker.io/kubesphere/metrics-server:v0.4.2 - docker.io/library/redis:5.0.14-alpine - docker.io/library/haproxy:2.0.25-alpine - docker.io/library/alpine:3.14 - docker.io/osixia/openldap:1.3.0 - docker.io/kubesphere/netshoot:v1.0 - docker.io/kubeedge/cloudcore:v1.13.0 - docker.io/kubeedge/iptables-manager:v1.13.0 - docker.io/kubesphere/edgeservice:v0.3.0 - docker.io/openpolicyagent/gatekeeper:v3.5.2 - docker.io/kubesphere/openpitrix-jobs:v3.3.2 - docker.io/kubesphere/devops-apiserver:ks-v3.4.1 - docker.io/kubesphere/devops-controller:ks-v3.4.1 - docker.io/kubesphere/devops-tools:ks-v3.4.1 - docker.io/kubesphere/ks-jenkins:v3.4.0-2.319.3-1 - docker.io/jenkins/inbound-agent:4.10-2 - docker.io/kubesphere/builder-base:v3.2.2 - docker.io/kubesphere/builder-nodejs:v3.2.0 - docker.io/kubesphere/builder-maven:v3.2.0 - docker.io/kubesphere/builder-maven:v3.2.1-jdk11 - docker.io/kubesphere/builder-python:v3.2.0 - docker.io/kubesphere/builder-go:v3.2.0 - docker.io/kubesphere/builder-go:v3.2.2-1.16 - docker.io/kubesphere/builder-go:v3.2.2-1.17 - docker.io/kubesphere/builder-go:v3.2.2-1.18 - docker.io/kubesphere/builder-base:v3.2.2-podman - docker.io/kubesphere/builder-nodejs:v3.2.0-podman - docker.io/kubesphere/builder-maven:v3.2.0-podman - docker.io/kubesphere/builder-maven:v3.2.1-jdk11-podman - docker.io/kubesphere/builder-python:v3.2.0-podman - docker.io/kubesphere/builder-go:v3.2.0-podman - docker.io/kubesphere/builder-go:v3.2.2-1.16-podman - docker.io/kubesphere/builder-go:v3.2.2-1.17-podman - docker.io/kubesphere/builder-go:v3.2.2-1.18-podman - docker.io/kubesphere/s2ioperator:v3.2.1 - docker.io/kubesphere/s2irun:v3.2.0 - docker.io/kubesphere/s2i-binary:v3.2.0 - docker.io/kubesphere/tomcat85-java11-centos7:v3.2.0 - docker.io/kubesphere/tomcat85-java11-runtime:v3.2.0 - docker.io/kubesphere/tomcat85-java8-centos7:v3.2.0 - docker.io/kubesphere/tomcat85-java8-runtime:v3.2.0 - docker.io/kubesphere/java-11-centos7:v3.2.0 - docker.io/kubesphere/java-8-centos7:v3.2.0 - docker.io/kubesphere/java-8-runtime:v3.2.0 - docker.io/kubesphere/java-11-runtime:v3.2.0 - docker.io/kubesphere/nodejs-8-centos7:v3.2.0 - docker.io/kubesphere/nodejs-6-centos7:v3.2.0 - docker.io/kubesphere/nodejs-4-centos7:v3.2.0 - docker.io/kubesphere/python-36-centos7:v3.2.0 - docker.io/kubesphere/python-35-centos7:v3.2.0 - docker.io/kubesphere/python-34-centos7:v3.2.0 - docker.io/kubesphere/python-27-centos7:v3.2.0 - quay.io/argoproj/argocd:v2.3.3 - quay.io/argoproj/argocd-applicationset:v0.4.1 - ghcr.io/dexidp/dex:v2.30.2 - docker.io/library/redis:6.2.6-alpine - docker.io/jimmidyson/configmap-reload:v0.7.1 - docker.io/prom/prometheus:v2.39.1 - docker.io/kubesphere/prometheus-config-reloader:v0.55.1 - docker.io/kubesphere/prometheus-operator:v0.55.1 - docker.io/kubesphere/kube-rbac-proxy:v0.11.0 - docker.io/kubesphere/kube-state-metrics:v2.6.0 - docker.io/prom/node-exporter:v1.3.1 - docker.io/prom/alertmanager:v0.23.0 - docker.io/thanosio/thanos:v0.31.0 - docker.io/grafana/grafana:8.3.3 - docker.io/kubesphere/kube-rbac-proxy:v0.11.0 - docker.io/kubesphere/notification-manager-operator:v2.3.0 - docker.io/kubesphere/notification-manager:v2.3.0 - docker.io/kubesphere/notification-tenant-sidecar:v3.2.0 - docker.io/kubesphere/elasticsearch-curator:v5.7.6 - docker.io/kubesphere/elasticsearch-oss:6.8.22 - docker.io/opensearchproject/opensearch:2.6.0 - docker.io/opensearchproject/opensearch-dashboards:2.6.0 - docker.io/kubesphere/opensearch-curator:v0.0.5 - docker.io/kubesphere/fluentbit-operator:v0.14.0 - docker.io/library/docker:19.03 - docker.io/kubesphere/fluent-bit:v1.9.4 - docker.io/kubesphere/log-sidecar-injector:v1.2.0 - docker.io/elastic/filebeat:6.7.0 - docker.io/kubesphere/kube-events-operator:v0.6.0 - docker.io/kubesphere/kube-events-exporter:v0.6.0 - docker.io/kubesphere/kube-events-ruler:v0.6.0 - docker.io/kubesphere/kube-auditing-operator:v0.2.0 - docker.io/kubesphere/kube-auditing-webhook:v0.2.0 - docker.io/istio/pilot:1.14.6 - docker.io/istio/proxyv2:1.14.6 - docker.io/jaegertracing/jaeger-operator:1.29 - docker.io/jaegertracing/jaeger-agent:1.29 - docker.io/jaegertracing/jaeger-collector:1.29 - docker.io/jaegertracing/jaeger-query:1.29 - docker.io/jaegertracing/jaeger-es-index-cleaner:1.29 - docker.io/kubesphere/kiali-operator:v1.50.1 - docker.io/kubesphere/kiali:v1.50 - docker.io/library/busybox:1.31.1 - docker.io/library/nginx:1.14-alpine - docker.io/joosthofman/wget:1.0 - docker.io/nginxdemos/hello:plain-text - docker.io/library/wordpress:4.8-apache - docker.io/mirrorgooglecontainers/hpa-example:latest - docker.io/fluent/fluentd:v1.4.2-2.0 - docker.io/library/perl:latest - docker.io/kubesphere/examples-bookinfo-productpage-v1:1.16.2 - docker.io/kubesphere/examples-bookinfo-reviews-v1:1.16.2 - docker.io/kubesphere/examples-bookinfo-reviews-v2:1.16.2 - docker.io/kubesphere/examples-bookinfo-details-v1:1.16.2 - docker.io/kubesphere/examples-bookinfo-ratings-v1:1.16.3 - docker.io/weaveworks/scope:1.13.0
Note
If the artifact file to export contains ISO dependencies, such as conntarck and chrony, set the IP address for downloading the ISO dependencies in .repostiory.iso.url of operationSystem. Alternatively, you can download the ISO package in advance and fill in the local path in localPath and delete the
url
configuration item.You need to enable harbor and docker-compose configuration items, which will be used when you use KubeKey to build a Harbor registry for pushing images.
By default, the list of images in the created manifest is obtained from docker.io.
You can customize the manifest-sample.yaml file to export the desired artifact file.
You can download the ISO files at https://github.com/kubesphere/kubekey/releases/tag/v3.0.7.
If you already deployed a cluster, you can run the following command in the cluster to create a manifest file and configure the file according to the sample in Step 2.
./kk create manifest
Export the artifact.
Run the following command directly:
./kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz
Run the following commands:
export KKZONE=cn ./kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz
Note
An artifact is a .tgz package containing the image package (.tar) and associated binaries exported from the specified manifest file. You can specify an artifact in the KubeKey commands for initializing the image registry, creating clusters, adding nodes, and upgrading clusters, and then KubeKey will automatically unpack the artifact and use the unpacked file when running the command.
Make sure the network connection is working.
KubeKey will resolve image names in the image list. If the image registry requires authentication, you can configure it in .registry.auths in the manifest file.
Install Clusters in the Air-gapped Environment
Copy the downloaded KubeKey and artifact to nodes in the air-gapped environment using a USB device.
Run the following command to create a configuration file for the air-gapped cluster:
./kk create config --with-kubesphere v3.4.1 --with-kubernetes v1.23.15 -f config-sample.yaml
Run the following command to modify the configuration file:
vim config-sample.yaml
Note
- Modify the node information according to the actual configuration of the air-gapped environment.
- You must specify the node where the
registry
to deploy (for KubeKey deployment of self-built Harbor registries). - In
registry
, the value oftype
must be specified as that ofharbor
. Otherwise, the docker registry is installed by default.
apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Cluster metadata: name: sample spec: hosts: - {name: master, address: 192.168.0.3, internalAddress: 192.168.0.3, user: root, password: "<REPLACE_WITH_YOUR_ACTUAL_PASSWORD>"} - {name: node1, address: 192.168.0.4, internalAddress: 192.168.0.4, user: root, password: "<REPLACE_WITH_YOUR_ACTUAL_PASSWORD>"} roleGroups: etcd: - master control-plane: - master worker: - node1 # If you want to use KubeKey to automatically deploy the image registry, set this value. You are advised to separately deploy the registry and the cluster. registry: - node1 controlPlaneEndpoint: ## Internal loadbalancer for apiservers # internalLoadbalancer: haproxy domain: lb.kubesphere.local address: "" port: 6443 kubernetes: version: v1.23.15 clusterName: cluster.local network: plugin: calico kubePodsCIDR: 10.233.64.0/18 kubeServiceCIDR: 10.233.0.0/18 ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni multusCNI: enabled: false registry: # To use KubeKey to deploy Harbor, set the value of this parameter to harbor. If you do not set this parameter and still use KubeKey to create an container image registry, the docker registry is used by default. type: harbor # If Harbor or other registries deployed by using KubeKey requires login, you can set the auths parameter of the registry. However, if you create a docker registry using KubeKey, you do not need to set the auths parameter. # Note: If you use KubeKey to deploy Harbor, do not set this parameter until Harbor is started. #auths: # "dockerhub.kubekey.local": # username: admin # password: Harbor12345 # Set the private registry to use during cluster deployment. privateRegistry: "" namespaceOverride: "" registryMirrors: [] insecureRegistries: [] addons: []
Run the following command to install an image registry:
./kk init registry -f config-sample.yaml -a kubesphere.tar.gz
Note
The parameters in the command are explained as follows:
config-sample.yaml: Specifies the configuration file of the cluster in the air-gapped environment.
kubesphere.tar.gz: Specifies the image package of the source cluster.
Create a Harbor project.
Note
As Harbor adopts the Role-based Access Control (RBAC) mechanism, which means that only specified users can perform certain operations. Therefore, you must create a project before pushing images to Harbor. Harbor supports two types of projects:
Public: All users can pull images from the project.
Private: Only project members can pull images from the project.
The username and password for logging in to Harbor is admin and Harbor12345 by default. The installation file of Harbor is located in /opt/harbor, where you can perform O&M of Harbor.
Method 1: Run the following commands to create a Harbor project.
a. Run the following command to download the specified script to initialize the Harbor registry:
curl -O https://raw.githubusercontent.com/kubesphere/ks-installer/master/scripts/create_project_harbor.sh
b. Run the following command to modify the script configuration file:
vim create_project_harbor.sh
#!/usr/bin/env bash # Copyright 2018 The KubeSphere Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. url="https://dockerhub.kubekey.local" #Change the value of url to https://dockerhub.kubekey.local. user="admin" passwd="Harbor12345" harbor_projects=(library kubesphereio kubesphere argoproj calico coredns openebs csiplugin minio mirrorgooglecontainers osixia prom thanosio jimmidyson grafana elastic istio jaegertracing jenkins weaveworks openpitrix joosthofman nginxdemos fluent kubeedge openpolicyagent ) for project in "${harbor_projects[@]}"; do echo "creating $project" curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" -k #Add -k at the end of the curl command. done
Note
Change the value of url to https://dockerhub.kubekey.local.
The project name of the registry must be the same as that of the image list.
Add -k at the end of the curl command.
c. Run the following commands to create a Harbor project:
chmod +x create_project_harbor.sh
./create_project_harbor.sh
Method 2: Log in to Harbor and create a project. Set the project to Public, so that any user can pull images from this project. For more information, please refer to Create Projects.
Run the following command again to modify the cluster configuration file:
vim config-sample.yaml
Note
- In auths, add dockerhub.kubekey.local and the username and password.
- In privateRegistry, add dockerhub.kubekey.local.
... registry: type: harbor auths: "dockerhub.kubekey.local": username: admin password: Harbor12345 privateRegistry: "dockerhub.kubekey.local" namespaceOverride: "kubesphereio" registryMirrors: [] insecureRegistries: [] addons: []
Note
- In auths, enter dockerhub.kubekey.local, username (admin) and password (Harbor12345).
- In privateRegistry, enter dockerhub.kubekey.local.
- In namespaceOverride, enter kubesphereio.
Run the following command to install a KubeSphere cluster:
./kk create cluster -f config-sample1.yaml -a kubesphere.tar.gz --with-packages
The parameters are explained as follows:
- config-sample.yaml: Specifies the configuration file for the cluster in the air-gapped environment.
- kubesphere.tar.gz: Specifies the tarball image from which the source cluster is packaged.
- --with-packages: This parameter is required if you want to install the ISO dependencies.
Run the following command to view the cluster status:
$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
After the installation is completed, the following information is displayed:
************************************************** ##################################################### ### Welcome to KubeSphere! ### ##################################################### Console: http://192.168.0.3:30880 Account: admin Password: P@88w0rd NOTES: 1. After you log into the console, please check the monitoring status of service components in the "Cluster Management". If any service is not ready, please wait patiently until all components are up and running. 1. Please change the default password after login. ##################################################### https://kubesphere.io 2022-02-28 23:30:06 #####################################################
Access KubeSphere's web console at
http://{IP}:30880
using the default account and passwordadmin/P@88w0rd
.
Note
Feedback
Was this page Helpful?
Receive the latest news, articles and updates from KubeSphere
Thanks for the feedback. If you have a specific question about how to use KubeSphere, ask it on Slack. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement.