Signals and Supervisors

Every direct invocation of compose.mk optionally gets a "supervisor" process that allows for enhanced use (abuse?) of POSIX signals. Amongst other use-cases, you can use this pattern to yield control to another process in the middle of target-processing, or to implement entrance/exit handlers for make, in make.

Tool mode invocations implicitly involve supervisors and signals, and signals also enable certain experimental features, and in principle the supervisor can be extended with new entry/exit hooks.

Power users might be interested to understand the caveats and limitations around this, and anyone looking to work deeply on compose.mk itself might need to grok the implementation.

Background & Motivation

Without forking make, there's no simple method to get hooks into the default way that it handles interrupts and signals. But why would you want to anyway? Most of the time this occurs to people it's about cleanup, but compose.mk doesn't exactly look like traditional use-cases.

For compose.mk specifically, signals come up when we need to stop make from processing the command line input, and pass control over to a new process. But why? Well, short-circuiting the default CLI option parsing for make really useful. Especially since compose.mk allows us to "wrap" lots of containerized tools, we want to be able to proxy arguments over to those tools without make being greedy about parsing everything.

Here's a few typical examples of why you might want individual targets to essentially be able to consume the rest of the command line.

• The help command has an optional 2nd argument, a target, which must not be parsed by make because that would actually execute the target!
• The loadf command is similar: the 2nd argument, a filename, must not be parsed as a Makefile target because no such target exists.
• The wrappers for jq and jb are also examples where we need to read the whole command-line.

# Several example invocations where arguments are NOT targets.
$ ./compose.mk help docker.image.run
$ ./compose.mk loadf demos/data/docker-compose.yml
$ echo '{"foo":"bar"}' | ./compose.mk jq .foo
$ ./compose.mk jb foo=bar

Inheritance & Interpreters

Projects that are extending compose.mk might be interested in inheriting the capabilities for signals and supervisors. The simplest reason you might want to do this is just for the sake of bringing compose.mk-style CLI help to your own project, i.e. targets that actually support positional arguments, but the advanced examples on this page hint at other possibilities.

For technical reasons, there's unfortunately no way to inherit signal capabilities by simply using include compose.mk from your project Makefiles. But, there's a work-around for this if you're willing to setup compose.mk itself as the interpreter. (For information about getting an interpreter and a transpiler/preprocessor, see instead the compiler docs)

This pattern is an advanced topic, and although it is both stable and portable, the semantics are strange enough that the whole thing should probably be considered experimental!

Positional Arguments & Yield

Let's start with something simple. To demonstrate short-circuiting the default command line processing of make, we'll try to create a clone of the ls command.

In the code, the differences start right away. See the shebang at the top of the following file which uses mk.interpret ?

#!/usr/bin/env -S CMK_DISABLE_HOOKS=1 ./compose.mk mk.interpret
# Demonstrating compose.mk as an alternate interpreter for make.
# This is mostly used with extensions that want to inherit signals/supervisors.
#
# Main docs: https://robot-wranglers.github.io/compose.mk/signals/

# NB: explicit include is optional when the shebang sets `compose.mk`
# as the interpreter.  It's good to keep it anyway so the file can work
# as well with `make -f ..`
include compose.mk

ls:; $(call mk.yield, ls $${MAKE_CLI#* ls})

__main__:
    # this line works, but exits afterwards
    ${make} ls /var

    # lines afterwards will not run.
    this second line never runs!

Using compose.mk to "interpret" your makefile isn't as magical as it sounds. You can think of it as effectively inserting your makefile at the bottom of the compose.mk source code, then running the resulting temporary file as usual with make. As a result of this, there are two important side-effects for project automation:

1. Automation has access to compose.mk API implicitly (regardless of any include)
2. Automation inherits the compose.mk extensions to signals

At this point as long as the file is executable, then ./demos/interpreter.mk ls / works as expected. But the body of __main__ demonstrates something surprising, which is that mk.yield really does drastically change the rest of program flow control afterwards.

You can skip the usage of compose.mk as an interpreter by invoking the file as make -f demos/interpreter.mk ls /, which does work, but includes a Supervisor not found message at the end. And whereas ./demos/interpreter.mk runs __main__ and exits early with success, make -f ./demos/interpreter.mk fails because it does not exit early.

If you're only interested in proxy for positional arguments then this won't matter, meanwhile option-proxy works in some cases but must always be tested carefully.

Generic Option Proxy

The end of the last section hints at a basic problem with convenient but naive option-proxy. What if we want ls --help to work, or to put it another way: how can we stop make from eating our argv?

The solution for this is to use the end of options hint, aka ' -- '¹, which compose.mk has some built-in support for using the MAKE_CLI_EXTRA variable.²

#!/usr/bin/env -S CMK_DISABLE_HOOKS=1 ./compose.mk mk.interpret
# Demonstrating compose.mk as an alternate interpreter for make.
# This is mostly used for inheriting signals/supervisors.
#
# Main docs: https://robot-wranglers.github.io/compose.mk/signals/

include compose.mk

__main__:
    @# Print usage info and exit.
    $(call log, ${red}USAGE: ${__file__} ls)

ls:; $(call mk.yield, ls $${MAKE_CLI_EXTRA})

As seen above, this does require that we sacrifice the convenience of a really direct invocation. But now ./demos/interpreter.mk ls -- --help actually returns help for ls instead of help for make.

This works much more reliably for options compared with the last approach, but still has known issues. To see this, try passing optional args with and without =, i.e. ./demos/interpreter-2.mk ls -- --width=45 will break whereas ./demos/interpreter-2.mk ls -- --width 45 does not. ³

Advanced Examples

Of course, no one really cares about making an ls clone! But besides the simple use-cases with targets that might want to accept pos-args and halt further execution, there are some other applications.

The example below demonstrates an embedded python-script that we'll run inside a container, and uses real option parsing.

#!/usr/bin/env -S ./compose.mk mk.interpret
# Demonstrating compose.mk as an alternate interpreter for make.
# This is mostly used for inheriting signals/supervisors.  
#
# Main docs: https://robot-wranglers.github.io/compose.mk/signals/
#
# USAGE: 
#   ./demos/interpreter-3.mk

include compose.mk

python.img=python:3.11-slim-bookworm
python.interpreter=python

define python_script
from optparse import OptionParser
def main():
    parser = OptionParser(usage="usage: %prog [options] filename",version="%prog 1.0")
    parser.add_option("-f", "--foo", dest="foo", default='foo', help="set foo option")
    parser.add_option("-b", "--bar", 
        action="store_true", dest="bar", default=False, help="set bar option")
    (options, args) = parser.parse_args()
    if options.bar:
        print("bar is set")
    if options.foo:
        print(f"foo is {options.foo}")
    if len(args) > 0:
        print("Additional arguments:", args)
if __name__ == "__main__":
    print("hello python opt parse")
    main()
endef

# Target for running the script in a container, passing extra arguments in.
# This uses dense and idiomatic shorthand without explanation because 
# the container-dispatch is not the main point of this demo!
script.run:
    ${mk.def.to.file}/python_script \
    && cmd="python_script ${MAKE_CLI_EXTRA}" \
        ${docker.image.run}/${python.img},${python.interpreter} \
    && $(call mk.yield)

# Setting up a macro that uses ' -- ', we can abstract away the weird invocation
script=${make} script.run -- 

# Using the macro makes things tidy, and also completely abstracts the container.
__main__:; ${script} --foo my.foo --bar

my-normal-target:
    echo hello normal make target

Above we use $(call mk.yield) with no arguments, which is equivalent to $(call mk.yield, true), i.e. just stopping CLI parsing and exiting immediately. This is often sufficient and has the correct exit status, because if the line before it should fail, execution will stop anyway.

Anyway, using it looks like this:

$ ./demos/interpreter-3.mk my-normal-target

hello normal make target

$ ./demos/interpreter-3.mk 

hello python opt parse
bar is set
foo is my.foo

$ ./demos/interpreter-3.mk script.run -- --help

hello python opt parse
Usage: python_script [options] filename

Options:
  --version          show program's version number and exit
  -h, --help         show this help message and exit
  -f FOO, --foo=FOO  set foo option
  -b, --bar          set bar option

Pre & Post Hooks

If you're using the interpreter as seen in the previous examples, you'll also automatically be able to use pre and post hooks for every target that runs via the command line. Here's a simple example to illustrate:

#!/usr/bin/env -S ./compose.mk mk.interpret
# Demonstrating compose.mk as an alternate interpreter for make.
# This example shows automatic installation of pre/post hooks.
#
# USAGE: ./demos/interpreter-4.mk main_target
#
# See also: https://robot-wranglers.github.io/compose.mk/signals#pre-post-hooks

main_target.pre:; echo hello pre-hook
main_target:; echo hello main
main_target.post:; echo hello post-hook

flux.ok.pre:; echo look, hooking a builtin works too

__main__:; $(call log,${red} USAGE: ${__file__} main_target)

Running the code above works the way you'd expect, and we can get pre and post effects with no explicit plumbing. This is named-based with post-fixes as seen above, and extensions can also add hooks for compose.mk builtins.

Here's the output:

$ ./demos/interpreter-4.mk main_target

hello pre-hook
hello main
hello post-hook

This is partly just a style choice that an make certain code more readable, but it does also make possible a certain interesting kind of code reuse. Many targets may wish to share the same pre-conditions or post-conditions. This works because there's no need to define an explicit target body for the hooks actually, and you can just treat my.target.pre: common.precondition as an annotation, similar to function decorators in python, or lightweight "aspect oriented" programming.

It's very important to understand the limitation that was hinted at earlier though. What does it mean that hooks work for "every target used via command line" ? To put this in other words, hooks are only fired when the interpreter is used directly, i.e. recursive calls via ${make} or similar will not execute them! This makes the feature pretty useful for standalone tools and augmenting standard entrypoints like clean/build/test, and much less useful for libraries.

Hooks are enabled by default since they don't affect peformance much, but it's possible they might conflict with existing naming conventions or similar. To override this, you can set CMK_DISABLE_HOOKS=1 in your environment, or in the shebang lines at the top of your script. Always use CMK_DISABLE_HOOKS=1 if you are making proxy-wrappers, as with our ls clone earlier.

At-Exit Handlers

To setup one or more exit handlers, try using the CMK_AT_EXIT_TARGETS environment variable with space-separated target names. This should run if make dies for any reason and even if it is killed, but cannot run if the wrapper process is killed.

At-exit handlers defined this way are called whenever mk.supervisor.exit is called, which has the usual caveats that you must be using a interpreter-style invocation, and not running via make -f ...! See the next section for more details.

Implementation Details

If you're already very familiar with make and looking over the rest of this page, you might be wondering how this kind of semi-magical behavior can be achieved. And since the answer to that question is frankly kind of ridiculous, this should probably be considered experimental =) With that sternly worded warning out of the way..

Signals and supervisors are implemented using a shebang hack⁵ and a polyglot⁶, whereby compose.mk is simultaneously a Makefile, a Makefile library, and a bash script.

The way this works is as follows:

The bash script invokes make, but wraps it to catch any signals that are thrown. After throwing a supported signal causes make to exit, bash catches, and then passes the code for the caught signal back to another invocation of make for an exit handler.

As a consequence of this, signals can only be handled when compose.mk is invoked directly, i.e. with ./compose.mk ..., and cannot work for make -f compose.mk.

The pre/post hooks have exactly the same limitation, but a different implementation. Since the bash wrapper-script ultimately controls the initial invocation of make, it's possible to dynamically inject targets between any targets that are initially declared.

For Developers

All targets related to signals/supervisors can be found under the mk.interrupt and mk.super namespaces, respectively. These targets are for low-level internal use, and even hacking directly on compose.mk source probably won't require editing them, considering the other techniques available. Still, quick links to the relevant parts of the API are provided below for reference:

References