First off, if you have a few spare physical machines; or machines with enough resources to run a few well resourced VMs it is fairly simple to install Kubernetes itself. I have a fairly old OpenStack stack deployment yaml file that I still use to throw-up/tear-down multicompute node environments under OpenStack if I need something more powerful than MiniKube, but MiniKube is all you really need for development/testing on a regular basis.
However this post is on minikube. MiniKube is the best tool for testing out Kubernetes for a home lab if you are comfortable running everything on one machine if you have one powerful enough. Minikube provides a working environment including mutiple nodes (or course on the local machine) if required.
What is a powerful enough machine is a matter of debate; for example to test istio it is recomended to use 6 cpus and 8Gb of memory; I had no trouble with 2 cpus and a physical machine with only 6Gb of memory and only a wireless interface for downloads, running all examples and kiali (it was slow, bit everything worked).
As a general rule you should probably allocate as much resource as you can, especially as minikube can run multiple nodes if you wish to by simply passing a command line flag to the start command.
One important thing to note about this post. I run minikube on a machine running docker using the docker driver. I strongly recomend you do the same so you can use docker commands to manage images in the cluster as discussed in some of the tips and tricks later on.
This post is about a few of the tips and tricks I have picked up using it.
For things I am testing (or even keeping) I prefer to keep them in their individual versioned directories where possible; for that reason I skip the steps some installers want of copying things to /usr/local/bin as you would only do that if you wanted every user on your machine to use them plus do not want aliases in your global profile. One advantage is that you can easily have multiple versions and just update the aliases.
Installing MiniKube and additional components
We will start off with configuring it to be useful. Note that I install everything under a directory ~/installs/kubernetes; you can place it in any directory of you choice.
# --- I keep everything under one directory and use aliases to run them
INST_DIR="~/installs/kuberernetes"
mkdir -p ${INST_DIR}
# --- get minikube
mkdir -p ${INST_DIR}/kuberernetes/minikube
cd ${INST_DIR}/kuberernetes/minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
# --- get istio
mkdir -p ${INST_DIR}/kuberernetes/istio
cd ${INST_DIR}/kuberernetes/istio
# check https://github.com/istio/istio/releases for the latest release available ---
# I am using 1.10.1 which is the latest at the time I write this
wget https://github.com/istio/istio/releases/download/1.10.1/istio-1.10.1-linux-amd64.tar.gz
tar -zxvf istio-1.10.1-linux-amd64.tar.gz
/bin/rm istio-1.10.1-linux-amd64.tar.gz
For things I am testing (or even keeping) I prefer to keep them in their individual versioned directories where possible; for that reason I skip the steps some installers want of copying things to /usr/local/bin as you would only do that if you wanted every user on your machine to use them plus do not want aliases in your global profile.
I just refer to the commands by aliases. So add the below lines to your ~/,bashrc (if using bash) or the profile file of whatever shell you use. Note the alias entry for ‘kubectl’, most documentation will recomend you download the latest copy of kubectl but as minikube has its own copy built in which is at the correct version for minikube you should use that copy, an example is the last of the three aliases shown below allowing the command ‘kubectl’ to be used at te terminal so copy/paste from websites you are interested in will work.
alias minikube="/home/mark/installs/kubernetes/minikube/minikube-linux-amd64" alias istioctl="/home/mark/installs/kubernetes/istio/istio-1.10.1/bin/istioctl" alias kubectl='minikube kubectl --'
Right, we are ready to start things up. Remember to ‘source ~/.bashrc’ (or start a new shell)
cd ~ minikube start --cpus 6 --memory 8192
At this point just occasionally use the command ‘kubectl get pod -A’. Wait until all pods are running before continuing.
Then you want istio installed
istioctl install
At this point just occasionally use the command ‘kubectl get pod -A’. Wait until all pods are running before continuing.
Lets add some of the whizzy-bang tools you will want to play with to monitor/visualize what you deploy now
kubectl apply -f istio/istio-1.10.1/samples/addons/grafana.yaml kubectl apply -f istio/istio-1.10.1/samples/addons/jaeger.yaml kubectl apply -f istio/istio-1.10.1/samples/addons/kiali.yaml kubectl apply -f istio/istio-1.10.1/samples/addons/prometheus.yaml
For istio to be injected into pods you must set a label on each namespace you want istio used in, for playing about you will probably use the ‘default’ namespace so enter
kubectl label namespace default istio-injection=enabled
At this point you will probably want to test some of your own deployments. One additional tool I would suggest is a very strict kubernetes yaml file checker. That can be installed into its own directory and aliased as were the other commands
mkdir -p ~/installs/kuberernetes/kube-score cd ~/installs/kuberernetes/kube-score # ---- check https://github.com/zegl/kube-score/releases for the latest release available --- wget https://github.com/zegl/kube-score/releases/download/v1.11.0/kube-score_1.11.0_linux_amd64.tar.gz tar -zxvf kube-score_1.11.0_linux_amd64.tar.gz alias kube-score="/home/mark/installs/kubernetes/kube-score/kube-score" # << and add to ~/.bashrc with the other aliases # usage kube-score score xxxx.yaml
Loading images into MikiKube
Now, you may want to use a local docker registry for images; good luck with that !.
There probably is a way to tell minikube to lookup local dns, its internal dns is perfectly able to resolve the internet addresses needed to download the images it needs to run, but it ignores the local host /etc/hsosts file and dns settings by default. Even if it could be overridden most 'local' docker registries are insecure so could not be used easily anyway.
However this is where the benefits of running minikube on a machine running docker come into play.
MiniKube has a 'minikube load xxx.tar' command where you can load into the cluster images you can manually save from your local docker repository and copy across to the machine running minikube to load; as an example (same machine running docker and minikube using that docker as the driver).
[mark@hawk ~]$ docker image list REPOSITORY TAG IMAGE ID CREATED SIZE gcr.io/k8s-minikube/kicbase v0.0.23 9fce26cb202e 10 days ago 1.09GB docker-local:5000/portainer-ce latest 96a1c6cc3d15 4 months ago 209MB portainer/portainer-ce latest 96a1c6cc3d15 4 months ago 209MB localhost/mvs38j latest 1df77f61cbed 6 months ago 787MB [mark@hawk ~]$ docker image save localhost/mvs38j > mvs38j.tar # <-- save from docker [mark@hawk ~]$ minikube image load mvs38j.tar # <-- load to minikube
Important: a image loaded with 'minikube load xxx.tar' will not be shown with a 'minikube image ls' command. It is available and will be used by your containers, the pod logs will show 'image already present on local machine' when the pod starts; it seems to be invisible in cache until then.
However if your machine runs docker you can easily switch it from managing the machines docker instance to the kubernetes docker instance with the simple command 'eval $(minikube docker-env)' which allows you to use normal docker commands directly against the image within the minikube cluster as shown below where I switch the environment.
[mark@hawk ~]$ docker image list # <--- local machine, not many REPOSITORY TAG IMAGE ID CREATED SIZE gcr.io/k8s-minikube/kicbase v0.0.23 9fce26cb202e 10 days ago 1.09GB docker-local:5000/portainer-ce latest 96a1c6cc3d15 4 months ago 209MB portainer/portainer-ce latest 96a1c6cc3d15 4 months ago 209MB localhost/mvs38j latest 1df77f61cbed 6 months ago 787MB [mark@hawk ~]$ [mark@hawk ~]$ eval $(minikube docker-env) # <--- switch to minikube environment [mark@hawk ~]$ docker image list # <--- and we see lots of images REPOSITORY TAG IMAGE ID CREATED SIZE istio/proxyv2 1.10.1 5c66e8ac89a7 2 weeks ago 282MB istio/pilot 1.10.1 07d6b563f74b 2 weeks ago 217MB quay.io/kiali/kiali v1.34 1d3ab1649f0b 5 weeks ago 194MB k8s.gcr.io/kube-proxy v1.20.7 ff54c88b8ecf 5 weeks ago 118MB k8s.gcr.io/kube-apiserver v1.20.7 034671b24f0f 5 weeks ago 122MB k8s.gcr.io/kube-controller-manager v1.20.7 22d1a2072ec7 5 weeks ago 116MB k8s.gcr.io/kube-scheduler v1.20.7 38f903b54010 5 weeks ago 47.3MB gcr.io/k8s-minikube/storage-provisioner v5 6e38f40d628d 2 months ago 31.5MB grafana/grafana 7.4.3 c9e576dccd68 3 months ago 198MB jimmidyson/configmap-reload v0.5.0 d771cc9785a1 4 months ago 9.99MB prom/prometheus v2.24.0 53fd5ed1cd48 5 months ago 173MB localhost/mvs38j latest 1df77f61cbed 6 months ago 787MB kubernetesui/dashboard v2.1.0 9a07b5b4bfac 6 months ago 226MB jaegertracing/all-in-one 1.20 84b5c715abd0 8 months ago 45.7MB k8s.gcr.io/etcd 3.4.13-0 0369cf4303ff 9 months ago 253MB k8s.gcr.io/coredns 1.7.0 bfe3a36ebd25 12 months ago 45.2MB kubernetesui/metrics-scraper v1.0.4 86262685d9ab 14 months ago 36.9MB k8s.gcr.io/pause 3.2 80d28bedfe5d 16 months ago 683kB [mark@hawk ~]$
You can use ordinary docker commands against images within the minikube kubernetes cluster at this point; for example 'docker image rm 83e6a8464b84' will remove the image; although you should probably use 'minikube image rm' and just use docker to check.
Important notes
Do not expect docker images you download from dockerhub to run under kubernetes without modification. There are design issues to take into consideration, personally all my containers get an environment variable passed to them to indicate which application startup login chain to take. You may be able to get them to run if you set the kubernetes parameters for the container runasuser/runasgroup to 0 (if kubernetes allows such a thing) but that's obviously not ideal.
So create your own containers, or stick to kubernetes repositories not dockerhub ones until you know how to customise them.
Cleaning it all up again
To remove everything again, another benefit of keeping everything under its own directory structure is how easy it is to remove.
- 'minikube stop' - shuts everything down in a state it can be restarted. It can be restarted from this state without losing any of your work with another 'minikube start --cpus 6 --memory 8192'
- 'minikube delete' - use only when stopped, will delete everything you have done from minikube, you must start again from scratch
- rm -rf the directory you installed all the downloads into, plus 'rm -rf ~/.minikube' as a lot of stuff is stored under your home directory in that folder
