Import Services from Compose

One of the main features for compose.mk is a method for "importing" one or more docker compose files into your project Makefile, using the compose.import macro to automatically generate related make-targets for each docker-compose service.

Sometimes this "importing" process is referred to as automation scaffolding, or we might refer to it as the make/compose bridge, or simply.. the bridge.

For reference material, you can jump to details about the available Service Scaffolding, or skip over to details for Container Dispatch.

For project-automation, it's usually best to separate the details of automation / execution environment, as seen in the first example. If you are interested in prototyping systems or building self-contained tools/apps, you can skip the external compose file, and embed compose service definitions to generate scaffolding from that directly.

Basic Example

Let's walk through a minimal example, starting with a sample compose file.

We'll use images for debian and alpine just as examples, but these might be any tool containers that your workflow requires.

Summary 
# demos/data/docker-compose.minimal.yml: 
#  A minimal compose file with a few base images.  
services:
  debian:
    image: debian:bookworm
  ubuntu: 
    image: ubuntu:noble
  alpine:
    image: alpine

Next, the Makefile. To generate make-targets for every service in the given compose file, we just need to include compose.mk as a library, then call one of the compose.import.* functions.

The simplest way to use compose.import is passing in a single argument for the compose-file:

# Inside your project Makefile
include compose.mk
$(call compose.import, file=docker-compose.yml)

That's it for the boilerplate, but we already have lots of interoperability for the containers involved. This includes stuff like:

  • The ability to quickly shell into these containers
  • The usual docker compose verbs like stop/start/up/exec/logs
  • And crucially, capabilities for command-pipes and dispatch.

Target Scaffolding

Targets created as part of the automation scaffolding are can be top-level names, canonical names, or manually namespaced depending on exactly how you want to setup container import-statements and how verbose you want to be. Below you can see summaries for each type of scaffolding.

Top-level Container Handles

Top level container handles are targets that are created in the root namespace, basically mirroring the familiar verbs for docker compose ( up/down/build/stop/ps/logs ), plus adding conveniences for things like piping , dispatch, and shells.

So for our simple example so far, svc_name would be either debian or alpine.

Note that using compose.import creates top-level handles automatically, but there are other namespace management options to avoid collisions.

Also note that while the list above is mostly complete, variants are available for things like quiet dispatch, quiet build, etc. See the source for more details.

Canonical Names

Canonical scaffolding is sort of like an "absolute path" to the ones mentioned above, plus additional "group" targets that work with several (or all) of the tool container services at the same time.

For our example so far, compose_stem from docker-compose.ymlwould be docker-compose.

Canonical names are always created unconditionally with compose.import or any other kind of import statement.

There are several other scaffolded entrypoints! But they all follow a pattern like <compose_stem>/<svc_name>.<verb>, i.e., basically the same as the corresponding toplevel container handles we saw in the last section.

Dispatch Namespace

User-provided dispatch-namespaces are optional, but often a good idea. In addition to most of the "group" targets available as canonical names, you get syntactic sugar for target dispatch.

  • <namespace>/<svc>/<target>: -> roughly <svc_name>.dispatch/<target>

See the full docs for compose.import and the main container dispatch docs for more information.

Bridge API

❡ <svc_name>.shell

The svc_name.shell target drops to a containter shell for the named service, and is usually interactive.

# Interactive shell on debian container
$ make debian.shell

# Interactive shell on "alpine" container
$ make alpine.shell

❡ <svc_name>.build

The svc_name.build target explicitly builds containers. This is for convenience, but usually it's fine to let this happen implicitly or just in time.

# Build debian container (respects cache)
$ make debian.build

# Force-build debian container (busts cache)
$ force=1 make debian.build

❡ <svc_name>.ps

The svc_name.ps target is roughly docker compose -f .. ps <svc>.

# Shows results for running instances
$ make debian.ps

# Require results for running instances, failing otherwise
$ strict=1 make debian.ps

❡ <svc_name>.up

The svc_name.up and svc_name.up.detach targets are roughly docker compose -f .. up <svc> and docker compose -f .. up -d <svc> respectively, using the default entrypoints as usual.

# Foreground mode
$ make debian.up

# Daemon mode
$ make debian.up.detach

❡ <svc_name>.stop

The svc_name.stop target is roughly docker compose -f .. stop -t 1 <svc>. This does nothing if (up) has never been used.

# Foreground mode
$ make debian.stop

❡ <svc_name>.logs

The svc_name.logs target is roughly docker compose -f .. logs <svc>.

This requires that the container is already running, uses "follow" or "tail" mode by default and never exits. Under the hood this actually uses docker logs and not docker compose, which tends to avoid certain issues with pseudo ttys.

# Shows results for running instances
$ make debian.logs

# Require results for running instances, failing otherwise
$ strict=1 make debian.ps

❡ <svc_name>.dispatch

The svc_name.dispatch/ target is unary rather than nullary like our other examples. It accepts as an argument another target which will then be run in the given container. See the container dispatch docs for more details, but in this example we'll just pass in the trivial target flux.ok.

# Run the trivial `flux.ok` target inside the debian container
$ make debian.dispatch/flux.ok

❡ <svc_name>.exec

The svc_name.exec/ target is almost exactly the same as dispatch, except that it can only be used after svc_name.up or svc_name.up.detach has already run.

# Run the trivial `flux.ok` target inside a daemonized container.
# Won't work for containers that exit immediately!
$ make some_daemon.up.detach some_daemon.exec/flux.ok

❡ <svc_name>.shell.pipe

The svc_name.shell.pipe target allows streaming data:

# Stream commands into debian container
$ echo uname -n -v | make debian.shell.pipe

# Equivalent to above, because entrypoint for debian is bash
$ echo uname -n -v | make debian.pipe

# Streams command input / output between containers
echo echo echo hello-world | make alpine.pipe | make debian.pipe

❡ <svc_name>

The top-level svc_name targets are more generic and can be used without arguments, or with optional explicit overrides for the compose-service defaults. Usually this isn't used directly, but it's sometimes useful to call from automation. Indirectly, most other targets are implemented using this target.

# Overriding compose defaults:
#   Runs an arbitrary command on debian container 
$ entrypoint=ls cmd='-l' make debian

# Streams data into an arbitrary command on alpine container
$ echo hello world | pipe=yes entrypoint=cat cmd='/dev/stdin' make alpine

❡ <svc_name>/<special>

Besides targets for working with services there are targets for answering questions about services.

The <svc_name>.get_shell targets answers what shell can be used as an entrypoint for the container. Usually this is bash, sh, or an error, but when there's an answer you'll get it in the form of an absolute path.

$ make debian.get_shell

/bin/bash

The .get_config target dumps JSON-formatted config for the given tool container. This turns off most of the string-interpolation and path-resolution that happens by default.

Any additional arguments that are passed as <svc_name>.get_config/*<filter>* will be passed to jq for drilling down.

$ make alpine.get_config/build.context

.


$ make alpine.get_config

{
  "build": {
    "context": ".",
    "dockerfile_inline": "FROM ${IMG_ALPINE_BASE:-alpine:3.21.2} \nRUN apk add -q --update --no-cache coreutils build-base bash procps-ng"
  },
  "entrypoint": "bash",
  "hostname": "alpine",
  "networks": {
    "default": null
  },
  "volumes": [
    {
      "source": "${PWD}",
      "target": "/workspace",
      "type": "volume",
      "volume": {}
    },
    {
      "source": "${DOCKER_SOCKET",
      "target": "-/var/run/docker.sock}",
      "type": "volume",
      "volume": {}
    }
  ],
  "working_dir": "/workspace"
}

❡ <compose_stem>/<svc>

Namespaced aliases are also available, so that you can reference compose services by something like an "absolute path". Due to the file-stem of the compose file we imported, all of the stuff above will work on targets like you see below. 

$ make docker-compose/debian
$ make docker-compose/debian.shell

❡ <compose_stem>.<cmd>

Besides targets for working with compose-services, some targets work on the compose file itself. Assuming your compose file is named docker-compose.yml, the special targets work like this:

# Build 
# Equivalent to `docker compose -f docker-compose.yml build`
$ make docker-compose.build

# Build 
# Equivalent to `docker compose -f docker-compose.yml stop`
$ make docker-compose.stop

# Clean 
# Equivalent to `docker compose -f docker-compose.yml down --remove-orphans`
$ make docker-compose.clean

# List all services defined for file (Array of strings, xargs-friendly)
$ make docker-compose.services

Using the <compose_stem>.services target, it's easy to map a command onto every container. Try something like this:

$ make docker-compose.services \
  | xargs -I% sh -x -c "\
    echo uname -n \
      | make docker-compose/%.shell.pipe"

alpine
debian
ubuntu

Other Import Statements

So far we've used the simplest syntax for compose.import, but there's a few ways to do this.

Root + Namespace

Here's a variation that configures syntactic sugar for dispatch namespaces, by providing a 2nd argument to compose.import.

include compose.mk
$(call compose.import, \
  file=path/to/docker-compose.yml namespace=my_namespace)

Namespace Only

Recall that compose.import as used above defaults to creating top level targets.

To configure a user-provided dispatch-namespace and not import targets to the root.. use compose.import.as

include compose.mk
$(call compose.import.as, \
  file=path/to/docker-compose.yml namespace=my_namespace)

See also the platform demo.

Multiple Compose Files

The namespace-only style is perhaps the most useful for working with multiple compose files, and is an effective way to organize your tool containers into different "tool boxes" at the docker-compose level. Here's an example of how you could separate code-tools vs infracode-tools:

include compose.mk

# Load 1st compose file under paralleogram namespace,
$(call compose.import.as, \
  namespace=▰ file=my-compose-files/build-tools.yml)

# Load 2nd compose file under triangle namespace
$(call compose.import.as, \
  namespace=▲ file=my-compose-files/cluster-tools.yml)

# Top-level "build" target that dispatches subtargets
# "build-code" and "build-cluster" on different containers
build: ▰/maven/build.code ▲/kubectl/build.cluster
build.cluster:
  kubectl .. 
build.code:
  maven ...

There's lots of ways to use this, and if your service-names across several files do not collide, you are free to put everything under exactly the same namespace. It's only syntax, but if you choose the conventions wisely then it will probably help you to think and to read whatever you're writing.

See the container dispatch docs for a concrete demo.

Inlined / Embedded / From-String

So far, we've been working with files, but generating target-scaffolding can also work on strings, i.e. with embedded compose files. See that documentation for examples, but below you can find the basic usage. Note that "namespace" is implied, because it always the same as the define-block's name.

include compose.mk

define embedded_services
...
endef

$(call compose.import.string, def=embedded_services)

See the matrioshka demos for a concrete demo.

To avoid cluttering the root namespace, pass import_to_root=FALSE, and only the canonical names will be available.

$(call compose.import.string, \
  def=embedded_compose.yml import_to_root=FALSE)

But Why? 🤔

At first glance, maybe you're thinking this looks like lots of magic to save a little bit of typing and simply using docker compose -f .. from your Makefiles. Well, it's true that this method of "importing" compose services as make targets is often just paving the way for container dispatch later.

Even without dispatch though, native support for containers as primitives is useful, especially when you start to use actions on those containers as prerequisite tasks. So far we've seen that scaffolded targets effectively create an API over tool containers, but what's not emphasized yet is just how composable that API really is.

If you're unconvinced on this point, you might like to skip ahead to some more advanced stuff that is digging into this topic, like the TUI docs or the notebooking demo.

References

  1. https://make.mad-scientist.net/the-eval-function/ ↩