Known Limitations and Issues

KUBECONFIG should already be set

We fake it for builds/tests, but note that KUBECONFIG generally must be set for things to work! Sadly, lots of tools will fail even simple invocations like helm version or even --help if this is undefined or set incorrectly, and it will often crash with errors that aren't very clear.

Dealing with per-project KUBECONFIGs is best, where you can set that inside your project Makefile and be assured of no interference with your main environment. Alternatively, use bashrc if you really want globals, or provide it directly as environment variable per invocation using something like KUBECONFIG=.. make <target_name>.

Working Directories

By default, tool containers share the working directory. Therefore, any files you're using inside the container should be inside or below the working directory, unless you're explicitly mounting that as another volume.

Compose files used with compose.import statements would typically be in the same project-root as your Makefile, but technically the compose file can be anywhere, like /opt or ~/.config. Note however that files mentioned inside compose files, say with dockerfile_inline blocks that are using COPY commands, are always relative to the compose file and not the Makefile.

Docker and File Permissions

The usual problem with root-user-in-containers vs normal-user on host and file permissions. The alpine base is a container using root, as are many other things. And there is a long-standing known bug in the compose spec that makes fixing this from the compose file hard.

Invoking compose exclusively from a Makefile actually helps with this though. By default with compose.mk, DOCKER_UID | DOCKER_GID| DOCKER_UGNAME variables are set and available for use in k8s-tools.yml. This works slightly differently for Linux and MacOS, based on what messes things up the least, but YMMV. With Linux, it looks something like this:

export DOCKER_UID?=$(shell id -u)
export DOCKER_GID?=$(shell getent group docker | cut -d: -f3 || id -g)
export DOCKER_UGNAME?=user

If you're not working with Makefiles at all, you can export appropriate values in .bashrc or .env files you use. If none of this is appealing, and you mix host-local and dockerized usage of things like helm, then you may end up with weird file ownership. You can fix this if it comes up using sudo chown -R $USER:$USER ..

MacOS, Docker Sockets, and DinD

As long as docker is working, any kind of setup (Docker Desktop, Rancher Desktop, Colima) can work with compose.mk for container-dispatch. But for working with k8s-tools.yml containers specifically, the docker-socket sharing must also be working. If you're having problems that might be related to this, first make sure that your setup can correctly run this command:

$ docker run -v /var/run/docker.sock:/var/run/docker.sock -ti docker ps

If the volume mount is working correctly, the result here should look the same as docker ps from your host. If your docker socket is in a different place (like ~/.rd/docker.sock for Rancher Desktop), you may need to symlink the file.

MacOS Docker desktop can be especially annoying here, and it seems likely the same is true for windows. YMMV, but as of 2024 sharing the socket may mean required changes from the UI preferences, and/or enabling/disabling virtualization backends. Another way the problem can manifest is an error like this:

If you want better parity with docker in Linux, you might like to checkout Colima/Rancher.

Pipes & Temp Files

Temporary files are generated with mktemp, and always start with .tmp.*, which should be added to .gitignore! Most temp files created via io.mktemp are removed using trap. But due to the fact that some temp files might need to be shared between host/tool-containers, due to crashes, magical stuff with containers, magical stuff with processes, due to author-error, etc, these temporary files might not always be removed. In some places, temporary files are kept just to aid in debugging, but pure streams would be better. As with any kind of state-file, be mindful of your secrets, and try to keep them off the filesystem completely.