Quickstart

Basic Installation

Just copy the compose.mk file and make it executable:

$ cd myproject 

# Download the compose.mk automation lib
$ curl -sL \
  https://raw.githubusercontent.com/Robot-Wranglers/compose.mk/master/compose.mk \
    > compose.mk

# Make it executable
$ chmod +x compose.mk

# Now equivalent to `make -f compose.mk ...`
$ ./compose.mk ...

If you have make and docker, that's all the setup you'll need for stand-alone tool mode usage.

Note that since various containers are only pulled on demand, initial usage might be slow. (For example, you might need a jq container before compose.mk can work with JSON). Subsequent runs however will take advantage of cache.

If you want to keep yet-another-file out of your project root, you could use a .cmk/ folder, and while it's not really necessary or recommended.. you could track upstream with a git submodule. (See the Plugins, Forks, and Versioning section for more details.)

Read on for details about tighter integration with projects, and see the compatibility notes for details about docker versions / possible gotchas in OSX.

Project Integration

If you're looking to integrate with existing automation or you brought your own tool containers, you'll probably be interested in setting up the Make/Compose Bridge or preparing for Container Dispatch. See that full documentation for more discussion, or if you already know what you're doing, read on for the copy/paste stuff.

Makefile Integration

A quick example of what your project Makefile should look like:

# Include compose.mk for access to target API and macros.
include compose.mk

# If you brought tool containers, you'll want to import them.
# Call the bridge-building macro to get targets for each service.
$(call compose.import, file=docker-compose.yml)
$(call docker.import, file=Dockerfile \
    namespace=docker.mycontainer  img=mycontainer)

# At this point, scaffolded targets are defined for 
# the tool containers mentioned in the compose file.
# Now you can dispatch any task to any tool container.
test: tool_container_name.dispatch/self.test

self.test:
    echo hello world from container `uname -n -v`

Docker Compose Integration

Working with docker compose is optional if you're just interested in stand-alone mode or the standard library offerings. And if you are using docker-compose, then simple access to tools-containers from your compose files will just work for many use-cases, via the Makefile integration in the last section and via the scaffolded targets.

However.. more complicated use-cases will need a volume-mount for your project directory so that tool containers can read/write files, and, using container dispatch means that your tool containers need make and bash.

Here is some fairly complete boilerplate that's usually a good starting place:

Summary 
##
# demos/data/docker-compose.yml: 
#  Typical compose file with container-bases that are preconfigured to 
#  work with target dispatch, default workspaces, volume sharing for
#  tools, yaml inheritance, etc
services:
  debian: &base
    hostname: debian
    entrypoint: bash
    working_dir: /workspace
    volumes:
      # optional, but required for docker-in-docker
      - ${DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock
      # standard, so tools can access project files
      - ${PWD}:/workspace
    build:
      context: .
      dockerfile_inline: |
        FROM ${IMG_DEBIAN_BASE:-debian:bookworm}
        RUN apt-get update -qq && apt-get install -qq -y make procps 

  ubuntu: 
    <<: *base
    hostname: ubuntu
    build:
      context: .
      dockerfile_inline: |
        FROM ${IMG_UBUNTU_BASE:-ubuntu:noble}
        RUN apt-get update -qq && apt-get install -y -qq make procps

  alpine:
    <<: *base
    hostname: alpine
    build:
      context: .
      dockerfile_inline: |
        FROM ${IMG_ALPINE_BASE:-alpine:3.21.2} 
        RUN apk add -q --update --no-cache coreutils build-base bash procps-ng

This example above is typical, but not necessarily minimal, so let's discuss a few things about it.

Regarding base images, there's a good chance your tool-container already includes make, and you don't need to add packages at all. But if you're looking for reasonable base-containers that include make and will help you avoid initial package updates with apt or apk.. you might consider things like golang containers, gcc containers, or debian/buildd:bookworm.

Tips for configuring docker-compose

  1. Use defaults-with-overrides in FROM lines to stay versioned but still keep some flexibility.
  2. Use dockerfile_inline to quickly extend a base-container, without adding an extra Dockerfile to your repository. Tool-containers rarely change once established, and having a separate file and separate git/docker repos for everything can get tedious.
  3. Use yaml inheritance with anchors, i.e. &base and <<: *base. This is a good trick to keep compose files DRY and as simple as possible.
  4. Provide a few optional keys just for clarity. Things like hostname clean up logging, and even if entrypoint isn't used by software it can help humans who are trying to debug.
  5. Correct config keys for working_dir and volumes are crucial for project file IO, which might include reading your project Makefile if you're using container dispatch.
  6. It's optional, but there are cases where you'll want docker to be available inside tool-containers and you also might want to share the hosts docker-socket.3 The boilerplate above goes half way, defaulting to using a volume mount but stopping short of actually installing docker.

Compatibility Notes

Platforms used in development for compose.mk include recentish docker, recentish make (3.8+) and bash (5+) on both Linux and MacOS. Testing done in CI only uses Linux and won't try every possible combination of versions, but the goal is to support most things you'll encounter in the wild.

For docker, recentish means a client at least 25+, engine at 20+ and while it's not a strict requirement, using docker-compose at 2.17.0+ is strongly recommended. Otherwise you won't have access to dockerfile_inline, which is inconvenient and also breaks many demos.

For MacOS, the standard library should mostly Just Work. For more advanced features though, there are the usual problems with BSD vs GNU tools, particularly sed and awk.

Typically /usr/local/bin comes before /usr/bin in PATH, so this should leave your base installation untouched and still prefer GNU by default:

# Install a GNU awk
$ brew install gawk
$ cp /usr/local/bin/gawk /usr/local/bin/awk

# Install a GNU sed
$ brew install gnu-sed
$ cp /usr/local/bin/gsed /usr/local/bin/sed

Usage of other tools like xargs should be BSD/GNU agnostic so please report any issues! Opting in to GNU tools on MacOS everywhere is a matter of taste, but might be worthwhile. See documentation for example here and here.

Another edge case is that docker desktop in OSX is increasingly problematic in general, and also officially unsupported these days if you're still on Intel instead of Apple silicon. Rancher-desktop is one possible work around.

For Developers

For development on compose.mk itself, here's how you can clone the project, build related containers, and run the test suites.

# Clone with SSH
$ git clone git@github.com:Robot-Wranglers/compose.mk.git

# Or clone with HTTP
$ git clone https://github.com/Robot-Wranglers/compose.mk.git

# Pull tool containers to do some early caching
$ make clean build 

# Run all demos (in Makefile)
$ make demos

# Run only CMK-lang demos
$ make demos/cmk

Plugins, Forks, & Versioning

Given the advice above to simply embed compose.mk directly into your project repositories, some people might be reading this and thinking what if compose.mk changes upstream?

Eventually compose.mk might get a pypi installer option1, but for now the main things you can do are:

Fork-and-Forget:
The standard library for compose.mk is approaching frozen, so especially if you're not using CMK-lang, this is a pretty reasonable option.
 
Git Submodule::
If you really need to track upstream, or if you want to track plugins, you can setup the .cmk/ repository (or your fork of it) as a git submodule inside your project folder.2

To check the version you're currently using:

$ ./compose.mk mk.stat

{
  "make_version": "4.3",
  "compose.mk": "d998bdc7598a4deeacf613bee2e9fddd"
}

References

  1. If you're interested, vote for this issue. ↩

  2. See also the docs at the .cmk plugins repo ↩

  3. For example, remote controlling the TUI relies on docker-sockets. Also tool wrappers fall back to docker, so using them with container dispatch means docker-in-docker. ↩