Skip to content

Instantly share code, notes, and snippets.

@Nieto2018

Nieto2018/minikube_kvm2.md

Last active May 17, 2023 13:43
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Nieto2018/a282cc669b6542ffb29ae2bc13c18293 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Nieto2018/a282cc669b6542ffb29ae2bc13c18293 to your computer and use it in GitHub Desktop.
Download ZIP
Minikube - KVM2
Raw
minikube_kvm2.md

Minikube - KVM2

Create Minikube cluster

A cluster can be created with different configurations, but for testing in a local environment it is recommended to create the cluster with the kvm2 driver, as it allows us to modify parameters such as CPUs, RAM and volumes without having to delete and create clusters and also is the one that generates the least problems when using some tools, such as Ceph. The number of CPUs, RAM memory and volume size can be assigned as desired; in our case we will use 4 CPUs, 12Gb of RAM and 100Gb of storage volume.

minikube start --driver kvm2 --cpus 4 --memory=12g --disk-size=100g --extra-disks=1

Change CPUs assigned to Minikube cluster

Although there is no specific documentation and most of the information found on the internet indicates that it is not possible to apply the change without deleting the cluster, we will see that it is really possible, at least using the kvm2 driver.

See number of assigned CPUs

# virsh vcpucount "profile"
virsh vcpucount minikube

Assign number of CPUs

# Currently live CPUs
virsh setvcpus minikube 4 --live
# Current CPUs
virsh setvcpus minikube 4 --config
# Maximum CPUs
virsh setvcpus minikube 4 --maximum --config

Also change in config.json files in the following paths:

  • $HOME/.minikube/profiles/minikube
  • $HOME/.minikube/machines/minikube

Source: Modifying the number of virtual CPUs

Resize Minikube volume

It is possible that the Minikube volume will run short, for example due to the number of Docker images we use. Although there is no specific documentation and most of the information found on the internet indicates that it is not possible to apply the change without deleting the cluster, we will see that it is really possible, at least using the kvm2 driver.

The volume files of the cluster used by Minikube with the kvm2 driver have a rawdisk format and are found in the directory of their corresponding profile, $HOME/.minikube/machines/<profile>. We will apply examples to minikube's default profile, which is called minikube.

We stop the cluster, otherwise changes cannot be applied.

minikube stop

Get information about cluster volume

qemu-img info $HOME/.minikube/machines/minikube/minikube.rawdisk

Modify size of cluster volume (minikube cluster must be stopped)

sudo qemu-img resize $HOME/.minikube/machines/minikube/minikube.rawdisk 100g

Once you have launched this command you can check with qemu-img info that it has been assigned a new size. Now we are going to proceed to change its size within minikube's virtual machine so that it also uses this size; for this we will mount minikube's volume and use gparted tool to resize partition.

Mount cluster volume.

sudo losetup -Pf $HOME/.minikube/machines/minikube/minikube.rawdisk

Check path where volume has been mounted as loop file

sudo losetup --find --show $HOME/.minikube/machines/minikube/minikube.rawdisk

Open cluster volume in gparted to make modifications

# sudo gparted /dev/loopX (path returned by losetup --find... command)
sudo gparted /dev/loop32

Unmount loop file once modifications are made

# sudo losetup -d /dev/loopX (path returned by losetup --find... command)
sudo losetup -d /dev/loop32

Raise minikube cluster

minikube start

Check size of volume (/dev/vda) and partition (/dev/vda1) which should already have new size assigned

lsblk

Also change in config.json files in the following paths:

  • $HOME/.minikube/profiles/minikube
  • $HOME/.minikube/machines/minikube

Sources:

Create Docker network to connect with Minikube pods and vice versa

We check the IP used by Minikube

minikube ip
# 192.168.39.89

We get the network used by Minikube

ip -4 -br -o a s
# virbr1 UP 192.168.39.1/24

We create the Docker network that connects with the KVM2 network of Minikube

docker network create --driver=macvlan --subnet=192.168.39.0/24 -o parent=virbr1 kvm2-network

⚠️ This solution allows connecting from pods to containers by IP, it does not automatically resolve the container name

Source: Access KVM VM over the network from a Docker container

Solve network overlap problems in Ubuntu

KVM2 automatically creates network interfaces that usually overlap with those of other networks, preventing us from accessing machines on those networks. To check the network interfaces we launch the following command:

ip -4 -br -o a s

Returning the following

lo UNKNOWN 127.0.0.1/8 
wlp0s20f3 UP 192.168.1.146/24 
docker0 UP 172.17.0.1/16 
virbr1 UP 192.168.39.1/24 
virbr0 UP 192.168.122.1/24 
ppp0 UNKNOWN 10.120.20.108 peer 10.120.20.1/32

Since we have the following machine registered in our /etc/hosts file 192.168 .122 .237 container-registry.jimbo.local container-registry, the virbr0 interface prevents us from accessing that machine since both are on the 192 .168 .122.X subnet . To solve this we change the subnet of the virbr0 interface, for this we do the following:

We destroy the existing network:

virsh net-destroy default

We edit the network configuration:

virsh net-edit default

We create the network again:

virsh net-start default

After these steps we should have solved the problem.

Source: libvirt - virbr0 - setting the IP address

Deactivate subnet

ip link set down virbr0

Here is the translation of the text into English:

Configure Ingress DNS

To access services running on Minikube through Kubernetes' Ingress on our local machine.

Source: Ingress DNS

Solve problems

Solution 1

If you cannot access the service through its URL, make sure that the /etc/resolv.conf file is a symbolic link to the /run/NetworkManager/resolv.conf file and that it contains a single server name. Reported in the Minikube repository on Github.

sudo rm /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
sudo systemctl restart NetworkManager.service

Solution 2

If the previous solution did not work, check that in the /run/systemd/resolve/resolv.conf file there is the line nameserver 127.0.1.1, if it does not exist we create the following file /etc/systemd/resolved.conf.d/dns-localhost.conf (we create the folder structure if necessary) and add the following content.

[Resolve]
DNS=127.0.1.1

and we restart the systemd-resolved service.

sudo systemctl restart systemd-resolved

Source: https://unix.stackexchange.com/a/442599/289837

Solution 3

If the previous solution did not work either, create a copy of the /run/systemd/resolve/resolv.conf file, which we will call $HOME/custom.resolv.conf (we can call and locate it as we want) that we will make sure it has the line nameserver 127.0.1.1, and apply the following commands but using this file.

sudo rm /etc/resolv.conf
sudo ln -s $HOME/custom.resolv.conf /etc/resolv.conf
sudo systemctl restart NetworkManager.service

Configuration of private repositories

Insecure repositories

For minikube pods to be able to download images from private repositories, those repositories must be indicated within the ~/.minikube/machines/<minikube_profile>/config.json file (the default minikube_profile is “minikube”), within the InsecureRegistry section, as shown in example:

# ~/.minikube/machines/minikube/config.json
{
    ...
    "HostOptions": {
        ...
        "EngineOptions": {
            ...
            "InsecureRegistry": [
									...
                "container-registry.jimbo.local"
            ],
            ...
        }
        ...
    }
    ...
}

⚠️ If you use image repositories with HTTPS enabled but with self-signed certificates it may be necessary to also add them in the InsecureRegistry section to avoid certificate errors; for example, Gitlab's repository of Minikube cluster registry.gitlab.test has been necessary to add it during tests.

It may be necessary to add machine to minikube's /etc/hosts, in case driver Docker is used; driver KVM2 uses host machine's /etc/hosts:

mkdir -p ~/.minikube/files/etc
echo 127.0.0.1 localhost > ~/.minikube/files/etc/hosts
echo "192.168.122.237 container-registry.jimbo.local container-registry" >> ~/.minikube/files/etc/hosts
minikube stop && minikube start

Image cache repositories

For minikube pods to be able to download images from private repositories, those repositories must be indicated within the ~/.minikube/machines/<minikube_profile>/config.json file (the default minikube_profile is minikube) , within the RegistryMirror section. If cache repositories go through HTTPS and certificates are self-signed, they may also need to be added in the InsecureRegistry section.

Image repositories will be containers that can be running on Docker or Minikube, so if they are within same network as Minikube they can use their container name if running on Docker or Ingress URL if running on Minikube (in this case it may be necessary to configure local DNS to use Minikube's ingress-dns).

# ~/.minikube/machines/minikube/config.json
# In this example machine names refer to containers that are within same network as minikube,
# so they can reach them without need for extra configurations.
{
    ...
    "HostOptions": {
        ...
        "EngineOptions": {
            ...
            "InsecureRegistry": [
                "container-registry.jimbo.local",
                "docker-registry-mirror",
                "gitlab-jimbo-registry-mirror",
                "gitlab-nieto-registry-mirror",
                "gitlab-nieto:5050"
            ],
            ...
            "RegistryMirror": [
                "https://docker-registry-mirror",
                "https://gitlab-jimbo-registry-mirror",
                "https://gitlab-nieto-registry-mirror"
            ],
            ...
        }
        ...
    }
    ...
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment