NKP Post 2: Setting up a local registry with Harbor to avoid hitting rate limits at docker.io

Jonas Högman

Nov 1, 2024 • 9 min read

Welcome to part two of my series on how to deploy NKP on Nutanix. In this part, we will focus on setting up a local registry using Harbor.

The reason for this is that Docker has incorporated some rate limits on container pulls. You get 100 pull requests per 6 hours if you use an anonymous account, and 200 pull requests per 6 hours if you register a free Docker Hub account on their homepage.

Signup | Docker

Docker

In my lab, I found that I needed to look under the hood of the deployment process and did a lot of redeploys to test different features. As a result, I quickly hit Docker's rate limit.

Installation of Harbor VM

First, we need to set up a new VM for this purpose. I will not go into the specifics again on how to set up a new Rocky Linux VM on Nutanix. In my previous post, I covered how I set up the bastion/jump host as an installation and administration environment. If you need the detailed steps on how to set up a new Rocky Linux VM, please refer to my previous post.

I will also point out that this setup with a single VM running Harbor is not recommended for a production environment. For a production environment, set up Harbor in a Kubernetes cluster with some fault tolerance, etc. However, for the sake of this series, I will use a standalone host.

Once you have your Harbor VM up and running, connect to it using SSH.

The first thing we need to do is install Docker. Run the following commands:

# Add the Repo
sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

# Install using dnf
sudo dnf -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Enable docker
sudo systemctl --now enable docker

# Add the current user
sudo usermod -a -G docker $(whoami)

# Add a specific user
sudo usermod -a -G docker custom-user

Now we have Docker installed. Try the following command:

docker ps

It should give an output like this:

[administrator@joho-nkp-jumphost ~]$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Next, go to the Harbor GitHub page using the following link:

Scroll to the releases and find the latest stable release. At the time of writing this post, the latest release was 2.11.1. Click the release version to go to the release page. Right-click the online-installer.tgz and copy the link.

Head over to your Harbor VM and download the installer:

curl -Lo harbor-online-installer-v2.11.1.tgz "https://github.com/goharbor/harbor/releases/download/v2.11.1/harbor-online-installer-v2.11.1.tgz"

Untar the installer package and change the directory to the Harbor directory.

tar -zvxf harbor-online-installer-v2.11.1.tgz && cd harbor

Now, we should make some changes to the harbor.yaml file. First, copy the template to harbor.yml:

cp harbor.yml.tmpl harbor.yml

Edit harbor.yaml with your favorite editor. Change the hostname for your Harbor installation:

My hostname will be joho-registry.domain.local (where domain.local is our domain). Go ahead and create a DNS entry in your internal domain pointing to the IP of your Harbor Docker VM. Save the file.

Next, we need to create some certificates for our Harbor hostname. I will create the certificates from our issuing CA in my lab Active Directory.

First, I'll create a certs folder within the Harbor folder:

mkdir certs && cd certs

Then I create a file with the following content called req.conf:

[ req ]
default_bits	   = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
[ req_distinguished_name ]
countryName                 = Country Name (2 letter code)
stateOrProvinceName         = State or Province Name (full name)
localityName               = Locality Name (eg, city)
organizationName           = Organization Name (eg, company)
commonName                 = Common Name (e.g. server FQDN or YOUR name)
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = joho-registry.domain.local
DNS.2   = joho-registry
DNS.3   = 10.11.58.187

Let's go ahead and create a Certificate Request for our AD CA:

openssl req -new -out joho-registry.domain.local.csr -newkey rsa:2048 -nodes -sha256 -keyout joho-registry.domain.local.key -config req.conf

You will be presented with some questions. Add the information suitable for you to these questions.

Now, you should have a file tree that looks something like this:

Get the content of the .csr file using cat or your favorite text editor and copy that CSR request. The content should look something like this:

Take this CSR to your CA and make the request, and then copy the certificate that the CA issues to the folder. Also, make sure to download the CA-chain certificate from the CA as well. The certificate and chain should be in Base64 encoded format. Now you should have different cert files: a chain.cer, a certificate.cer, and a .key file.

Now, enter the harbor.yml file again and point out the full path to the cert files. In my case, it looks like this (make sure to adjust this to match your specific domain settings):

If you don't have a lab CA to issue the certificates, you can self-sign a certificate using the steps from Winson Sou's YouTube video on deploying Harbor with self-signed certs

1- Deploy harbor - Winson Sue Youtube channel

Now that we have the certificates in place, we need both the Linux machine and Docker to trust the CA. You do that by running the following commands:

# Make a folder with your domain name in dockers folder
sudo mkdir -p /etc/docker/certs.d/domain.local/

# Copy the files to that folder
sudo cp joho-registry.domain.local.cer /etc/docker/certs.d/domain.local/
sudo cp joho-registry.domain.local.key /etc/docker/certs.d/domain.local/

# Copy the CA to anchors folder for Linux host to trust the CA
sudo cp ca-chain.crt /etc/pki/ca-trust/source/anchors/

# Update the CA trust
sudo update-ca-trust

Now we're ready to install Harbor. Make sure that you're in the /harbor/ folder that we untarred earlier. Run the following commands:

sudo ./prepare

Docker should start pulling images down to your machine, and you'll get an output similar to this:

Let's install Harbor:

sudo ./install.sh --with-trivy

Harbor is now being started as containers in Docker.

Now you should be able to visit the Harbor installation using https://<your-fqdn>

You can log in with:

User: admin
Password: Harbor12345

Creating a Pull-Through Cache for Docker Hub

First, go to Administration -> Registries. Click "+New Endpoint."

Enter your details for the Docker Hub account you created in the first blog post of this series, and hit "Test Connection" followed by "OK."

Now, go back to Projects and click "+New Project." Fill out the form and hit "OK."

Uppload NKP Airgap bundle from Support portal

Now, we're almost done. There is only one thing left: uploading the NKP Airgap bundle from the Nutanix Support portal. You might wonder why you would do that if you are not performing an airgap install. If you want to experiment with deployment methods, it's good to have an offline repository of the NKP container binaries to avoid hitting Docker Hub's rate limits.

Start by connecting via SSH to your NKP Bastion host that we created in the first post of this series.

First, we need to ensure that we have the cert files that we created earlier on the NKP Bastion host. So, let's SCP the folder containing all the certs over to our bastion host:

scp administrator@<ip of harbor host>:/home/administrator/harbor/certs/* /home/administrator/certs/

You should get a similar output as below:

Get the local bastion machine to trust the CA by running the following command:

sudo cp ca-chain.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust

Then head into the Nutanix support portal again and click Downloads -> Nutanix Kubernetes Platform (NKP) and find the package called Nutanix Airgapped Bundle.

Click the three dots to the right of the "Download" button and copy the URL. From your NKP Bastion, run the following command:

curl -Lo nkp-air-gapped-bundle_v2.12.1_linux_amd64.tar.gz "<URL_FROM_SUPPORT_PORTAL>"

The file should now be downloading, which might take some time as the file is around 15 GB. Once the file is downloaded, untar the archive using the following command:

tar -zvxf nkp-air-gapped-bundle_v2.12.1_linux_amd64.tar.gz && cd nkp-v2.12.1

First, we need to load some files into Docker:

 docker load -i konvoy-bootstrap-image-v2.12.1.tar

# Should give the following output: 
0a2fd8cc81bd: Loading layer [==================================================>]  368.7MB/368.7MB
21260c944a11: Loading layer [==================================================>]  726.4MB/726.4MB
fc7e0e822c44: Loading layer [==================================================>]  4.373MB/4.373MB
9dfce09ca008: Loading layer [==================================================>]  7.168kB/7.168kB
2fbef85d524e: Loading layer [==================================================>]   2.56kB/2.56kB
1fdb11d6b213: Loading layer [==================================================>]  1.052GB/1.052GB
Loaded image: mesosphere/konvoy-bootstrap:v2.12.1

Run the follwoing commands:

docker tag mesosphere/konvoy-bootstrap:v2.12.1 

docker.io/mesosphere/konvoy-bootstrap:v2.12.1

Now that we have the extracted NKP bundle and tagged some things, let's upload the bundle to our /library project of our Harbor installation by executing the following commands:

# Push konvoy image:
nkp push bundle --bundle ./container-images/konvoy-image-bundle-v2.12.1.tar --to-registry='https://<FQDN_OR_IP_TO_HARBOR>/library' --to-registry-username='admin' --to-registry-password='<YOUR_PASSWORD>'

# Push kommander image:
nkp push bundle --bundle ./container-images/kommander-image-bundle-v2.12.1.tar --to-registry='https://<FQDN_OR_IP_TO_HARBOR>/library' --to-registry-username='admin' --to-registry-password='<YOUR_PASSWORD>'

# Push nkp-catalog-application image
nkp push bundle --bundle ./container-images/nkp-catalog-applications-image-bundle-v2.12.1.tar --to-registry='https://<FQDN_OR_IP_TO_HARBOR>/library' --to-registry-username='admin' --to-registry-password='<YOUR_PASSWORD>'

You should now see that the /library project of Harbor begins to populate with different images:

You should see an output similar to this:

You have now completed the setup of your Harbor registry. We're done for this post. Please keep reading my NKP series. In the next post, we'll go ahead and deploy the NKP management cluster that we will use to manage our K8s infrastructure.

The next post in the series you'll find here:

NKP Post 2: Setting up a local registry with Harbor to avoid hitting rate limits at docker.io

Jonas Högman

Installation of Harbor VM

Creating a Pull-Through Cache for Docker Hub

Uppload NKP Airgap bundle from Support portal

NKP Deployment Series Index:

Sign up to get the latest posts by email :)