CentOS7-based Kubernetes Cluster Setup using Ansible (en)
1. Introduction
This guide will take you through the process for a multi-node kubernetes cluster on CentOS 7 using Ansible automation tool. The setup will have one master and two minions.
2. Cluster Setup
Here are the system spec and network information.
- System Spec.
| Role | Hostname | Model | CPU | RAM | HDD |
|---|---|---|---|---|---|
| master | dns-man1 | HP ProLia nt DL380p Gen8 | 24C HT Xeon E5-2 667@ 2.90 GHz | 16GB | 300GB |
| minion | ares68 | HP ProLia nt DL380p Gen8 | 24C HT Xeon E5-2 667@ 2.90 GHz | 16GB | 300GB |
| minion | ares70 | N/A | 8C HT Xeon E560 6@2. 13GH z | 8GB | 299GB |
| minion | ares-k26 | N/A | 32C HT Xeon E5-2 667v 3@3. 20GH z | 32GB | 600GB |
- Network Info.
|Internet|
|
|ext. switch|
|
|-------------|----------------|-----------------|
.63 .68(10.5.4.68) .70(10.5.4.70) .26(10.5.6.26)
dns-man1 ares68 ares70 ares-k26
.254.189 .200.67 .200.70 .200.16
|------------|----------------|-----------------|
|
|int. switch|
- External IP prefix: x.x.x (intentionally hidden by the author :))
- LB IP range: 10.5.4.0/24, 10.5.6.0/24
- Internal IP range: 192.168.0.0/16
3. Component Installation
Kubernetes installation is very cumbersome and tedious work. I decided to use Ansible to automate Kubernetes Cluster installation.
3.1. Set up Ansible master
Ansible by default manages machines over the SSH protocol.
- no database, no daemons
- 4 month release cycles
- python-based (so possible to install using pip)
- Python 2.6/2.7 based (python3 is not compatible yet.)
I’ll install ansible master on dns-man1. Before installing ansible master, the pre-requisite packages should be installed.
git: need it to get ansible source from git repo.
python-virtualenv: need it to wrap ansible env inside my home.
# yum install git python-virtualenv gcc python-devel libffi-devel openssl-devel
Now install ansible and set up virtualenv.
# git clone git://github.com/ansible/ansible.git --recursive
# virtualenv ansible
# source ansible/bin/activate
(ansible)# cd ./ansible
(ansible)# source ./hacking/env-setup
(ansible)# pip install paramiko PyYAML Jinja2 httplib2 six
3.2. Set up Ansible minions
Ansible uses ssh protocol to manage minions. So it is easier to manage when each minion has a master ssh public key as an authorized key. Let’s set that up.
First, generate ssh keys on master.
# ssh-keygen
Second, copy the public key to each minion.
# ssh-copy-id root@area68
# ssh-copy-id root@ares70
Now let’s get into the ansible playground!!!
3.3. Create Ansible playbook
I created ansible playbook to install and set up the kubernetes components.
First, get the playbook files.
# git clone https://our_repo/path/to/k8s_cent7.git
The playbook files are in our git repo which is not public.
Go to k8s_cent7 directory. To check if minion is set up correctly, run ansible ping to minion from master.
(ansible)# cd k8s_cent7
(ansible)# ansible -i hosts ares68 -m ping
ares68 | SUCCESS => {
"changed": false,
"ping": "pong"
}
It’s good.
README.md
The playbook deploy the components for kubernetes cluster.
The playbook has the following roles in order:
- common role: common tasks (set up ntp, disable firewalld, etc.)
- etcd role: ETCD key-value datastore on the master
- flannel role: flannel overlay network on each minion
- master role: kubernetes master
- docker role: docker container engine
- registry role: a private docker registry on the master
- minion role: kubernetes minions
The order is important. The etcd role should be played before flannel role since flannel depends on etcd.
The default storage backend of docker on CentOS is loopback-lvm which is not production-ready. It is recommended to use direct-lvm. So the docker role sets up direct-lvm storage. Edit roles/docker/vars/main.yml to set up the partition to use for docker lvm.
To use, 1. copy ‘hosts.example’ to ‘hosts’ and edit the ‘hosts’ file. 2. Download and put the latest kubernetes binaries into the right places.
roles/minion/files/kubelet roles/minion/files/kube-proxy roles/master/files/kube-apiserver roles/master/files/kube-scheduler roles/master/files/kubectl roles/master/files/kube-controller-manager You can download the latest kubernetes binaries at https://github.com/kubernetes/kubernetes/releases/
Edit vars files in each role and global group_vars and host_vars.
Run the playbook:
ansible-playbook -i hosts site.yml
3.4. Verify the components
Now, verify the cluster is up and running.
To get the k8s version:
# su - ucim $ uc version Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}We can see the k8s version is 1.4.4.
To make sure master components are healthy:
$ uc get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health": "true"}The kubernetes master components(apiserver, scheduler, controller-manager, etcd) are all healthy.
To get the status of minions:
$ uc get nodes NAME STATUS AGE x.x.x.68 Ready 1m x.x.x.70 Ready 1m
Two minions are added to the master and ready to serve.
Conclusion
Ansible is a swiss-army knife for the devops. Our kubernetes cluster(1 master and 2 minions) setup using ansible took about 10 minutes.
- Save the time to set up the machines. No need to work on boiler-plate settings on every machine.
- Implement Infrastructure as Code(IaC) to manage machines.
- Simple rerun a playbook whenever the settings need to be changed. It will change the minions in the desired state.