Self Extracting Executables

One of the strangest features of compose.mk is that it can create self-extracting executable archives from Makefiles (or anything else really) using a dockerized version of makeself.

Besides offering generic access to makeself, compose.mk offers simple support for bundling a project Makefile, plus any dependencies, into a single executable file. Dependencies might include things like compose.mk itself, data files, and any docker compose file you're using.

Packaging can remove the explicit dependency on the rest of the stack represented by your project filesystem state though, resulting in frozen 'release' where you can set a custom entrypoint.

Use-Cases & Motivation

There are two main reasons why native support for this is interesting:

• First, compose.mk is both a tool factory and an application factory, and wanting to "package" your way into a frozen release comes up naturally.
• Second, project automation and pipelines are also a big part of what compose.mk is used for, and compressing that down into a snapshotted "release" could be useful for stuff like checkpointing/rollbacks, or for topics in notebookes / reproducible research. See also the notebooking demo for a sketch of something like that.

Put simply, easy access to containers and multiple programming languages and workflows is usually stuff that we associate with microservices and distributed systems like Kubernetes, Jenkins or Airflow. What happens if it is easy to design undistributed applications, tools, and pipelines with similar architectures?

Another point of view is that what we're really talking about is the ability to export a project into a tool. Because let's be real: No one wants to look at your project, because it's a mess of docs/examples/tests/support data, and they have to clone it, then understand how it's all organized before they can even try to use it. Usually, even after a project is understood, it's not like you can use it directly, it must adapted or edited, or you go down a rabbit hole of dependencies. Self-extracting archives are far from new, but since they can actually help with some of this, they seem underutilized.

Example: Make-target to Executable

Shortcuts for packaging up targets-as-entrypoints use mk.pkg.

For the first example, we'll spin-off an individual utility from compose.mk as a stand-alone executable. Let's look at the stream.img target, which uses a dockerized version of the chafa tool for displaying images on console, comparable to imgcat or similar. It takes no arguments, and works with streams.

$ bin=./stream.img ./compose.mk mk.pkg/stream.img
$ cat img.png | ./stream.img

It works, and we use chafa without having chafa, because containers don't build or pull before they are used.

Example: TUI to Executable

While stream.img is just a small utility, the approach demonstrated above also works with "applications" implemented using compose.mk, i.e. full blown TUIs.

As an example, how about exporting the tux.demo?

$ bin=./tux.demo ./compose.mk mk.pkg/tux.demo

Example: Extension to Executable

In the examples so far, the ./compose.mk mk.pkg/<target> examples only use targets that are built-in. If you want to extend compose.mk with your own automation and export that, then you'll want to use mk.pkg from the extension instead.

As an example, let's export the tools built as part of the Custom Automation APIs Demo.

# create binary for the `ansible.adhoc.pipe` target
$ bin=./ansible.adhoc archive=demos \
    ./demos/ansible-adhoc.mk mk.pkg/ansible.adhoc.pipe

# confirm/inspect the generated executable
$ file ./ansible.adhoc 

./ansible.adhoc: POSIX shell script executable (binary data)

# use ansible without having ansible
$ echo msg=HelloWorld | module=debug ./ansible.adhoc

{
  "action": "debug",
  "changed": false,
  "msg": "HelloWorld"
}

Makeself Reference

Direct access to makeself is via mk.self. For those unfamiliar with makeself, this section is a quick tutorial, but you can use this to package up anything.

Let's look at an example of freezing the contents of a directory and an arbitrary script that uses that directory.

We only need to provide a space-separated list of stuff to archive, plus a name for the executable file, plus a script for the entrypoint. The script can be anything as long as it works properly relative to the archive root.. so for now we'll just use pwd; find . to demonstrate that we are indeed running in the archive.

$ archive=docs/demos bin=archive.run \
  script='sh -c "pwd; find ."' ./compose.mk mk.self

≣ docker.from.def // Dockerfile.makeself 
 ≣ docker.from.def // tag=compose.mk:makeself .. cached 
⇄ mk.self // Archive for docs/demos will be released as ./archive.run
⇄ mk.self // Total files: 13
⇄ mk.self // Entrypoint: sh -c "pwd; find ."
Header is 716 lines long
About to compress 100 KB of data...
Adding files to archive named "archive.run"...
./demos/README.md
CRC: ...
MD5: ...
Self-extractable archive "archive.run" successfully created.

So far so good. Running the exported archive.run bin looks like this:

$  ./archive.run 

/home/user/code/.tmp.oZWg1AYyc
./demos
./demos/README.md

Argument Passing

Argument passing hasn't been demonstrated so far, and this can get tricky.

Here's an example where the generated executable we want to build will take arguments in the form of a shell script, which script will run inside the archive directory. And in this case since we're passing in a script, we need to quote all the input too.

$ archive=demos/ bin=archive.run script='bash -x -c' ./compose.mk mk.self 
$ ./archive.run -- 'pwd; find .'

+ pwd
/home/user/.tmp.5idV3NHp6
+ find .
./demos
./demos/README.md

To pass arguments to makeself you can put stuff before the --. Mostly this isn't useful because compose.mk already uses makeself in such a way that --quiet --noprogress is implied for pipe-safety. For debugging workflows though, check out --keep to preserve the uncompressed archive, and --noexec to disable the script execution. See also the output of ./archive.run --help, which provides help for makeself and not for whatever you've packaged with it.