Compiler & Dialects


To avoid freaking out people who are just interested in tools or libraries, the fact that compose.mk is actually a programming language isn't emphasized in most docs, and the shocking secret truth is safe to ignore. If you are interested though, this section is hands-on with the language itself, and enough details to extend it.

Quick Version

  1. The compose.mk language is called CMK or CMK-lang.
  2. Files use the .cmk extension by convention.
  3. CMK is a superset of Makefile, compiled into legal classic Makefile.
  4. Makefile is always valid CMK; the reverse might not be true.
  5. The compiler is contained inside compose.mk, and there are no other tools.
  6. The compiler is extensible, enabling new CMK dialects with user-defined config.

You could say that CMK = Makefile + compose.mk standard library + optional extra syntax.

Crucially, writing code in the CMK language does not add new core capabilities to the stuff that compose.mk already does without CMK. At a high level, the point of CMK is to eliminate lots of pesky dollar-signs and curly brackets in pure Makefile, while it also elevates various existing idioms for containers and polyglots to native syntax.

Whether you love make or passionately hate it, you'll probably agree that default syntax can be very annoying. Since compose.mk fixes most of the other problems with practical general-purpose usage, syntax is the main the problem left! This actually matters a lot, not just for adoption or usage as a daily driver, but because notation is part of how we think.1

About CMK Demos


All examples written in CMK rather than pure Makefile can be found in the demos/cmk folder. Most CMK demos are direct ports from existing pure-Makefile demos, i.e. given demos/cmk/foo.cmk there's a corresponding vanilla Makefile demo at demos/foo.mk.

# Run some "foo" demo directly, using CMK interpreter via shebang
$ ./demos/cmk/foo.cmk

# Run demo explicitly using interpreter
$ ./compose.mk mk.interpret! ./demos/cmk/foo.cmk

# Test the compiler without running the result
$ cat ./demos/cmk/foo.cmk | ./compose.mk mk.compiler

Basic Example


Let's revisit the JSON emit/consume example that's part of the structured IO docs. The original code was simple, but rewriting this in CMK is much more pretty:

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang translation of demos/structured-io.mk

emit: 
    🡄 key=val

consume:
    🡆 .key

__main__:
    this.emit | this.consume
    cmk.log.target(hello world!)

We'll get to the odd arrow syntax at the end of this section, but start with most basic stuff first.

New Interpreter:
This example introduces mk.interpret! in a shebang. Not to be confused with mk.interpret, mk.interpret! includes additional steps for CMK->Makefile transpilation, allowing us to extend the base syntax and use the more exotic stuff you see in the rest of the file. As with other demos, this one is executable directly as ./demos/cmk/structured-io.cmk. Unlike other demos, we can't run it directly with make -f .., and would use instead ./compose.mk mk.interpret! ..
Implied Include + Standard Library:
When using the interpreter, a line like include compose.mk is always implied, and of course this brings with it full access to the standard library.
Function Calls Rewritten:
Awkward stuff like $(call log.target, ...) can be written as cmk.log.target(..) instead. Additionally, although we don't use it here, compose.import statements previously written as $(call ..) can be rewritten as simply compose.import(..)
Recursive Calls Rewritten:
Awkward stuff like ${make} foo can be written as this.foo instead. Recursive invocation of the current context is common, and absolutely critical for code-reuse.3 The choice of this for a new keyword is actually configurable too.. we'll get to that in the next section.

Finally, about that arrow syntax? These operators just help to make JSON IO feel more native, and are sugar to safely use jq and jb for parsing and pushing JSON, respectively, regardless of whether those tools are available on the host. If you glanced at the original code that's "ported" to CMK-lang here, then you might notice that the only code-transformation rules we really need to get from CMK -> Makefile in this case are just very simple string substititions*.

The next section explains the specific substitions that enable the code above at the same time as explaining "dialects" in general.

About Dialects


A dialect is just an array of paired substitition rules, and during compilation these rules take effect everywhere except for inside define-blocks. Using the reflection capabilities, we can show some info about the default dialect that the compiler supports like this:

$ ./compose.mk mk.def.read/cmk.default.dialect
[
  ["🡆", "${stream.stdin} | ${jq} -r"],
  ["🡄", "${jb}"],
  ["this.", "${make} "]
  ...
]

Matching/substitution for dialects is strict and simple. Regexes are not supported, and a preference for unicode symbols is an easy way to guarantee that new keywords will not collide with any other usage. But note that since substitution does not apply inside define-blocks, a new keyword like this cannot not effect polyglots, even if they also use the same keyword. You only need to think carefully about collision with shell-commands and their arguments that appear inside normal targets.

Full dialect details are available in the appendix, but the rules shown above are the ones relevant to the basic example we started with: all we've done so far is introduce new syntax for common stuff with jq and jb and calling make recursively.


It might not be obvious how useful and powerful that simple substition actually is here, so let's take a brief detour to peel back the layers. Above is the dialect literal, but below you can see a first approximation of variable expansion.

As discussed in structured IO docs, we try to use local tools if available, and fall back to docker.

$ ./compose.mk mk.get/cmk.default.dialect
[
  [
    "🡆",
    "cat /dev/stdin | /usr/bin/jq -r"
  ],
  [
    "🡄",
    "docker container run --rm ghcr.io/h4l/json.bash/jb:${JB_CLI_VERSION:-0.2.2}",
  ],
  [
    "this.",
    "make -rs --warn-undefined-variables -f ./compose.mk "
  ],
  ...
]

One layer of expansion reveals that fallback works as advertised; compose.mk knows that a local copy of jq is available already, but jb is missing and will have to use docker.

It also reveals that there is actually yet another layer of expansion, where JB_CLI_VERSION has a default value for tool versions that supports user-override. Therefore besides being visually easy to parse and clarifying intent, the usage of the arrow operators results in portable code with no extra tool installation.

Compose File Example


Revisiting the code from the inlined compose-file demo, here's what it looks like ported to CMK.

Like the original demo, this embeds a compose file, creates target-scaffolding for the containers involved, and demonstrates task-dispatch for them. But this time we introduce the ⋘ name ..body.. ⋙ syntax for compose files. (Unicode ahead: isn't <<< !)

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang translation of demos/inlined-composefile.mk
# USAGE: ./demos/cmk/inlined-composefile.cmk

 inlined.services
services:
  alice: &base
    hostname: alice
    build:
      context: .
      dockerfile_inline: |
        FROM ${IMG_DEBIAN_BASE:-debian:bookworm-slim}
        RUN apt-get update -qq && apt-get install -qq -y make procps
    entrypoint: bash
    working_dir: /workspace
    volumes:
      - ${PWD}:/workspace
      - ${DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock
  bob:
    <<: *base
    hostname: bob
    build:
      context: .
      dockerfile_inline: |
        FROM ${IMG_ALPINE_BASE:-alpine:3.21.2}
        RUN apk add -q --update --no-cache \
          coreutils build-base bash procps-ng


# A private target, intended to run inside containers 
self.internal_task:; cmk.log(Running inside ${red}`hostname`)

# Dispatch demo:
# Runs the private target inside containers, using 
# syntax that looks like function-invocation.
__main__: \
    alice.dispatch(self.internal_task) \
    bob.dispatch(self.internal_task)

alternate_style1:
    @# Equivalent to above, using target-body instead of prereqs;
    @# For this scenario, using the `this.` prefix is required.
    this.alice.dispatch(self.internal_task) 
    this.bob.dispatch(self.internal_task)

# More "classic" style in pure Makefile. 
# Equivalent to __main__, no target body required.
alternate_style2: \
    alice.dispatch/self.internal_task \
    bob.dispatch/self.internal_task

# More "classic" style in pure Makefile. 
# Equivalent to alternate_style1, using target body.
alternate_style3:
    @# Equivalent to style1, just trading `${make}` for `this`.
    ${make} alice.dispatch/self.internal_task 
    ${make} bob.dispatch/self.internal_task

Most of the differences with the original demo are pretty obvious, and alternate equivalents are shown to make what's happening more clear, but let's walk through it.

Closer Look

  1. As in the basic example, include compose.mk is implied.
  2. We skip the compose.import.string(..) the original code used to create target scaffolding for services
  3. After declaration of services with ⋘ name ..body.. ⋙ target handles are available immediately.
  4. We replaced a more naive "echo ..." command with a call to cmk.log that's using color and stderr.

Most interestingly though, <container_name>.dispatch(<target_name>) now looks like a normal function invocation. Also notice that alternate_style1 shows that double-rewrite rules for transpiler substition works the way you'd expect.

Compose-Context & Bind-Script


The last example inlines a compose file, and then dispatches targets. How about using an external compose file, then running script-dispatch with a tool-container? This example introduces two new idioms that can help. (Unicode ahead: isn't @!)

Below, examples for script1.sh and script2.sh are equivalent, but use two different idioms:

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
#
# CMK-lang equivalent of demos/bind-script.mk.
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# See also: http://robot-wranglers.github.io/compose.mk/compiler
#
# USAGE: ./demos/cmk/bind-script.cmk

compose.import(file=demos/data/docker-compose.yml)

# Script-in-container using With/As-idiom
 script1.sh
echo hello container `hostname`
 with alpine as compose_context

# Script-in-container using Decorator-Idiom
script2.sh: compose.bind.script(svc=debian)
define script2.sh
    echo hello container `hostname`
endef

# Target scripts always run from the container, regardless of
# whether they are called from the host or from inside the container.
__main__: \
    script1.sh alpine.dispatch/script1.sh \
    script2.sh debian.dispatch/script2.sh

So why have two idioms if they are equivalent? Well, both of these turn out to be pretty generic. Since we'll encounter other variants later, it makes sense to introduce them side by side for some compare and contrast.

  • The decorator-style idiom with compose.bind.script is most explicit, and one of several possible bind declarations that CMK supports. This actually looks like a traditional Makefile target, but also adds a piece that looks like a python decorator.

  • The ⨖ / with / as idiom is a more generic form. In this case since we're working with containers defined by compose, we use compose_context for the as-clause. But will come up again, because depending on the contents of the with-clause, it can work with any script, use any interpretter, and run with containers that aren't managed by docker compose.

Although we've switched to an external compose file rather than embedded one, both forms work with inlined services too.

Choosing between bind-declarations vs with / as is sometimes just a matter of taste, but there are some practical considerations too. One advantage of using bind-declarations is that they can be stacked. Meanwhile, the with/as idiom is easier to refactor later, for example if you're moving from compose-backed containers in external files to inlined-Dockerfiles, etc.


Supported arguments are the same for both the decorator style and the with/as style. Both forms optionally support forwarding environment variables, sending output to stderr instead of stdout, etc. See here 5 for additional explanation re: env and quiet.

Arguments for `compose.bind.script`
Name Required? Description Example
svc Service / container to use. debian
entrypoint Interpreter to use.
Defaults to bash.
entrypoint=bash
env Variables to pass through to container.
Defaults to value from environment
env='foo bar'
quiet Whether to silence announcements re: compose-file and service name.
Defaults to value from environment, or 0.
quiet=1

See this demo for a concrete example using multiple arguments.

Compose-Context & Bind-Target


Quick-and-dirty scripting is often convenient, but working with targets makes it much easier to leverage composition. For this you can use compose.bind.target.

This idiom binds together a given container, a "public" target and a "private" target. The public-target is the main interface; the "private" target has the same name but uses a "self." prefix and always runs inside the named container.

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
#
# CMK-lang equivalent of demos/bind-target.mk & demos/bind-script.mk,
# demonstrating some idioms similar to function-decorators.
#
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# See also: http://robot-wranglers.github.io/compose.mk/container-dispatch
#
# USAGE: ./demos/bind-target-2.mk

# Create scaffolding for debian container
compose.import(file=demos/data/docker-compose.yml)

# Decorator Idiom: Target-in-container
target_dispatch: compose.bind.target(debian)
self.target_dispatch:
    echo hello container `hostname`

# Main entrypoint: Drive the test-cases, ensuring
# that target still works whether it is called from 
# the host or from a container.
__main__: target_dispatch debian.dispatch/target_dispatch

Again we import services here instead of inlining them, but compose.bind.target works in either case.


Supported arguments for compose.bind.target include the ability to optionally forward environment variables, sending output to stderr instead of stdout, etc. See here 5 for additional explanation re: env and quiet.

Arguments for `compose.bind.target`
Name Required? Description Example
1st pos arg Service-name / Where to run target debian
prefix Prefix used to find private target.
Defaults to "self."
prefix=.
env Variables to pass through to container.
Defaults to value from environment
env='foo bar'
quiet Defaults to value from environment, or 0 quiet=1

About Sugar


Under the hood, examples using with/as involve something slightly more complicated than the dialects we saw earlier, and this is called "sugar".

Sugar is an array of triples representing [sugar_start, sugar_end, sugar_template]. Similar to dialects, we can dump the default sugar for the compiler by using mk.def.read. A complete reference for sugar is in the appendix, but the part that's relevant for inlined-compose files is this:

$ ./compose.mk mk.def.read/cmk.default.sugar
[
    ["⋘","⋙","$(call compose.import.string, def=__NAME__ import_to_root=TRUE)"],
    ...
]

Sugar transforms CMK source towards legal Makefile define-blocks while giving those blocks additional semantics. Since define-blocks have names, every sugar also requires a name.

For the sugar_template element of the triple, we render it and add it after the define-block. See the table below for an explanation of the variables that may appear inside this template.

Sugar Template-Variables
Name Description
__NAME__ The sugar-block's name
__WITH__ The contents of the with-clause, if one was used.
__AS__ The contents of the as-clause, if one was used.
__REST__ Everything that comes after `sugar_end` up to newline [^4]

Polyglot Examples


CMK particularly shines at cleaning up the idioms you can see in the polyglot index, and almost every example is more clear and concise.

Foreign Code Objects


The example below uses another form of with .. as and introduces different syntax for bound vs unbound code, i.e. whether or not an interpreter has been declared.

Bound code uses the syntax ⟦ name ..body.. ⟧ with .. as .., whereas unbound code uses 🞹 name ..body.. 🞹. Note that both syntaxes use unicode specials, and should not be confused with normal square-brackets or asterisks!

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang translation of demos/code-objects.mk
# USAGE: ./demos/cmk/code-objects.cmk

# Main entrypoint, showing how to preview and run embedded code.
__main__: \
    hello_world1.preview hello_world1.run \
    hello_world2.run \
    hello_world3.run \
    hello_world4.preview

# Code bound to a containerized interpreter using literals.
elixir.img=elixir:otp-27-alpine
elixir.interpreter=elixir
 hello_world1
import IO, only: [puts: 1]
puts("elixir World!")
System.halt(0)
 with ${elixir.img}, ${elixir.interpreter} as container

# Code bound to a containerized interpreter using variables.
# In this case, you can accept user-overrides from env-vars.
export python_interpreter=python
export python_img?=python:3.11-slim-bookworm
 hello_world3
print('hello world 2')
 with $${python_img}, $${python_interpreter} as container

# Code bound to a target-as-interpreter.
# Target must be parametric, then do something with the parameter!
 hello_world2
hello world2 !
 with my_interpreter as target

# A custom interpreter, as used above.
# This always takes a file argument as a parameter:
# we do nothing except for displaying the file-contents.
my_interpreter/%:; cat ${*}

# Unbound code block: only `.preview` target is available 
# unless you bind something else to `.interpreter` later
🞹 hello_world4
print('hello world 4')
🞹

The eagle-eyed reader will have noticed that our friend ⨖ / with / as is generic enough to already handle the use-case of working with polyglots by passing img=.. entrypoint=... Why add another syntax?

There are some differences, including the scaffolded preview target. But the main difference is that this approach to polyglots is intentionally pretty minimal.

  • Using .. as target tends to result in designs that could use a container, but doesn't strictly require one, and usually makes things more friendly for composition. And for example this allows you to use host tools if available and fall-back to containers, or make dynamic decisions about the image to use.

  • Using .. as container only supports container/interpreter arguments. Amongst other things, this disables support for environment-variable passthrough, and therefore encourages only pipe-based I/O if any is required.

Files as Targets with polyglot.import.file


For larger scripts, you'll want to use external files. Binding a file, container, and interpreter is a typical use-case.

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
#
# CMK-lang equivalent of demos/bind-script.mk.
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# See also: http://robot-wranglers.github.io/compose.mk/compiler
#
# USAGE: ./demos/cmk/bind-file.cmk

kwargs=\
    file=demos/data/test-file.py \
    img=python:3.11-slim-bookworm \
    entrypoint=python

__main__: polyglot.bind.file(${kwargs})


As an alternative to binding to an existing target, you can use an import statement, and create a target more dynamically. This example also demonstrates passing a few more of the supported arguments:

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
#
# CMK-lang equivalent of demos/import-file.mk.
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# See also: http://robot-wranglers.github.io/compose.mk/compiler
#
# USAGE: ./demos/cmk/import-file.cmk

export foo=val1
export bar=val2

kwargs= \
    file=demos/data/test-file.py \
    img=python:3.11-slim-bookworm \
    entrypoint=python cmd='-O' \
    namespace=testing \
    env='foo bar'

polyglot.import.file(${kwargs})

__main__: testing


Supported arguments for the bind-declarations and the import statements are mostly the same, but as you can see above, import requires an additional namespace argument to determine the name of the target that will be created.

Arguments for `polyglot.import.file` and `polyglot.bind.file`
Name Required? Description Example
file Filename relative/path/to/file.py
namespace Prefix used to find private target.
Defaults to "self."
prefix=.
img Defaults to value from environment, or 0 quiet=1
entrypoint Defaults to value from environment, or 0 quiet=1
env Variables to pass through to container.
Defaults to value from environment
env='foo bar'
cmd Variables to pass through to container.
Defaults to value from environment
env='foo bar'

Host-context with compose.import.script


The approach for running a shell script on the host is slightly different, and compose.import.script becomes built-in:

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang equivalent of demos/script-dispatch-host.mk.
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# See also: http://robot-wranglers.github.io/compose.mk/compiler
#
# USAGE: ./demos/cmk/script-dispatch-host.cmk

# Look, here's a simple shell script 
define script.sh
set -x
printf "multiline stuff\n"
for i in $(seq 2); do
    echo "Iteration $i"
done
endef

# Directly imports script to target of same name.
compose.import.script(def=script.sh)

__main__: script.sh

Host scripts inherit the parent environment automatically. This includes any variables passed or exported from the command line, as well as any variables exported globally in the main script.

Dockerfile Example


Let's revisit the code from the inlined Dockerfile demo, and port it to CMK. (Unicode ahead: is not ///!)

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang translation of demos/inlined-dockerfile.mk
# USAGE: ./demos/cmk/inlined-dockerfile.cmk

 Dockerfile.my_container
FROM alpine:3.21.2
RUN apk add -q --update --no-cache coreutils build-base bash procps-ng


__main__: test.scaffold test.runner test.raw

test.scaffold: \
    my_container.build \
    my_container.dispatch(self.dispatch_test)
self.dispatch_test:
    cmk.log(Testing target from inside the inlined-container)
    cmk.log(host details: `uname -a`)

test.runner:
    cmk.log.test_case(Low level access to run commands)
    entrypoint=sh cmd='-c "pwd"' this.my_container.run

test.raw:
    cmk.log.test_case(Low-level unmediated access to image)
    docker image inspect ${my_container.img} > /dev/null
    docker run --entrypoint sh ${my_container.img} \
        -x -c "true" > /dev/null

It's much cleaner than the plain Makefile now. Helper targets for dispatch and build are created automatically, and data like my_container.img is also scaffolded.

Tip

Notice that the Dockerfile. prefix as part of the block-name is required for declaration, but afterwards reference to the image/container won't use it.

Local-Context & Embedded Containers


The last example uses some raw docker commands just to show it's possible, but in most cases you'll want tighter integration that feels more "native". For this we can leverage the generic ⨖ / with / as idiom again, this time using with local_context for the as-clause.

Here's two examples, showing minimal usage first, then demonstrating passing some extra arguments.

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang equivalent of demos/script-dispatch-custom.mk.
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# See also: http://robot-wranglers.github.io/compose.mk/compiler
#
# USAGE: ./demos/script-dispatch-custom.mk

 Dockerfile.my_container
FROM debian/buildd:bookworm
# .. other customization here ..


 example-1.sh
echo hello `hostname` at `uname -a` 
 with my_container as local_context

export var1=hello-world

 example-2.sh
echo variable exported: ${var1:-not set}
 with img=my_container env='var1' quiet=1 as local_context

__main__: example-1.sh example-2.sh


Supported arguments for local_context are described below. See here 5 for additional explanation re: env and quiet.

Arguments for docker.bind.script & mk.docker.bind.script
Name Required? Description Example
img Image to use.
Defaults to 1st positional-arg if kwargs not present.
img=...
def Name of code-block.
Implied for CMK ⨖-syntax
def=..
entrypoint Interpreter to use.
Defaults to bash.
entrypoint=bash
cmd Arguments to pass to interpreter.
Defaults to empty string.
Filename is post-fixed to command.
cmd=-x
env Variables to pass through to container.
Defaults to value from environment.
env='foo bar'

Docker-Context & Stock Containers


Using stock images instead of inlined ones is similar to the last section, except that it uses docker_context in the as-clause.

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang equivalent of demos/script-dispatch-stock.mk.
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# See also: http://robot-wranglers.github.io/compose.mk/compiler
#
# USAGE: ./demos/cmk/script-dispatch-stock.cmk

 my_dockerized_script
echo hello `hostname` at `uname -a`
⨖ with img=debian/buildd:bookworm as docker_context

__main__: my_dockerized_script


Supported arguments for docker_context are exactly the same as the ones we saw earlier with local_context.

User-Defined Extensions


Both dialects and sugar can be defined at runtime on a per-file basis, basically allowing the code itself to describe aspects of the compiler that will be used on it.

Info

File hints must occur after the shebang, before the first non-comment line, and must be legal JSON that specifies the full dialect/sugar to use. This overwrites the default dialect/sugar, and does not append it!

Minified JSON is optional, and everything between the first and second ::: will be used, so multi-line JSON is allowed, but must remain comment-prefixed.

User-Defined Dialects


User-defined dialects require a hint at the top of the file. Building on the basic example, here's an example that changes the default jq/jb JSON operators to use different characters.

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# Demonstrating user-defined extensions for the CMK -> Makefile compiler.
#
# cmk_dialect ::: [ 
#  ["⏩️", "${stream.stdin} | ${jq} -r"],
#  ["⏪️", "${jb}"], ["this.", "${make} "] ] :::
#
emit: 
    ⏪️ key=val

consume:
    ⏩️ .key

__main__:
    this.emit | this.consume

To get the full current default dialect in a minified form that's ready for your own modifications, use ./compose.mk mk.def.read/cmk.default.dialect| jq -c

User-Defined Sugar


User-defined sugar requires a hint at the top of the file. Building on the compose file example, here's an example that changes that default syntax to use different characters.

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# Showing user-defined rules for CMK -> Makefile transpilation.
# You can define cmk_dialect and cmk_sugar individually or together.
#   
# cmk_dialect ::: [
#   ["cmk.io.","${make} io."], 
#   ["this.","${make} "]
# ] :::
# cmk_sugar ::: [
#   ["𒀹","𒀺","$(call compose.import.string, def=__NAME__ import_to_root=TRUE)"]
# ] :::
#
# USAGE: ./demos/cmk/user-sugar.cmk

𒀹 inlined.services
services:
  alice: &base
    hostname: alice
    build:
      context: .
      dockerfile_inline: |
        FROM ${IMG_DEBIAN_BASE:-debian:bookworm-slim}
        RUN apt-get update -qq && apt-get install -qq -y make procps
    working_dir: /workspace
    volumes: ['${PWD}:/workspace']
  bob:
    <<: *base
    hostname: bob
    build:
      context: .
      dockerfile_inline: |
        FROM ${IMG_ALPINE_BASE:-alpine:3.21.2}
        RUN apk add -q --update --no-cache \
          coreutils build-base bash procps-ng
𒀺

# A task that runs inside the containers.
self.internal_task:; cmk.log(Running inside ${red}`hostname`)

__main__: 
    cmk.log(Testing dialect substitutions)
    cmk.io.print.banner/testing_dialect_substitution
    cmk.log(Testing sugar and dispatching task to containers)
    this.alice.dispatch(self.internal_task)
    this.bob.dispatch(self.internal_task)

To get the full current default sugar in a minified form that's ready for your own modifications, use ./compose.mk mk.def.read/cmk.default.sugar | jq -c

Bind Declarations


Bind-declarations were introduced in the compose file example, but they show up in a few places. You can think of them as a type of "target decorator", similar to function-decorators in python. Bind declarations in CMK-Lang use (not @ !) for the usual reason of avoiding collisions with other languages and to simplify the compiler.

One use-case of bind-declarations is to bind variables to values, basically providing support for keyword-arguments, as seen elsewhere.

The equivalent idiom is much more readable in CMK-lang:

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang translation of ./demos/kwarg-parsing.mk
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# USAGE: ./demos/cmk/kwarg-parsing.cmk

emit: 
    🡄 shape=triangle color=red

consume: args.from_json(shape color=blue name=default)
    printf "shape=$${shape} color=$${color} name=$${name}\n"

__main__:; this.emit | this.consume

Stacking Bind Declarations


Bind-declarations can be stacked. For example, loading some variables from JSON and some from the environment looks like this:

Summary
#!/usr/bin/env -S ./compose.mk mk.interpret!
# CMK-lang translation of ./demos/kwarg-parsing.mk
# Part of the `compose.mk` repo. This file runs as part of the test-suite.  
# USAGE: ./demos/cmk/kwarg-parsing.cmk
export color=green

emit: 
    🡄 shape=square

consume: \
  ᝏargs.from_json(shape=default1) \
  ᝏargs.from_env(color=default name=Bob)
    @# Comments in the middle are ok.
    printf "shape=$${shape} color=$${color} name=$${name}\n"

__main__: flux.pipeline/emit,consume

Status Info, Limitations, & Other Advice


Compilation and interpreted CMK is even more experimental than the rest of the project! However if you're nervous about upstream changes due to depending on the default dialect, one option is to just make it the user-defined default.

That said, everything that's documented here is stable and unlikely to change. Mainly experimental just means that testing so far is limited, so please report bugs, but not until you've confirmed you're using GNU awk!.

Per the rest of the design philosophy, the goal is to freeze CMK after it's useful and capable of being extended and most future changes are probably optimizations rather than extensions. See also the docs re: Forks & Versioning and Contributing.

Appendix


Other related documentation includes: The more basic interpreter usage, which allows for inheritance of the entire compose.mk standard library and signals, but skips compilation and does not allow syntactic extensions. The packaging support generates executables, but isn't compilation. And there's also a few thoughts about style.

Data below is rendered from the latest source code.

Appendix: Default Dialect


$ ./compose.mk mk.def.read/cmk.default.dialect
[
    ["ᝏ","; cmk.bind."], ["ᝏ ","; cmk.bind."],
    ["⧐", ".dispatch/"],
    ["🡆", "${stream.stdin} | ${jq} -r"], 
    ["🡄", "${jb}"], 
    ["this.", "${make} "]
]

Appendix: Default Sugar


$ ./compose.mk mk.def.read/cmk.default.sugar
[
    ["⋘", "⋙", "$(call compose.import.string, def=__NAME__ import_to_root=TRUE)"],
    ["⫻",  "⫻",  "$(call dockerfile.import.string, def=__NAME__)"],
    ["⟦",  "⟧",  "$(call polyglot.__import__.__AS__,__NAME__,__WITH__)"],
    ["🞹",  "🞹", "$(call compose.import.code, def=__NAME__)"],
    ["⨖", "⨖", "__NAME__:; $(call __AS__,__WITH__)"]
]

References



  1. Notation is part of how we think 

  2. Note that during compilation for CMK code, this isn't just a naive a find/replace, because we do have to leave define-blocks alone 

  3. Actually compose.mk takes recursive invocation so seriously that a significant effort is invested in fixing problems with ${MAKE}. This results in the less shouty usage of ${make}, which is also more robust, reliable, and predictable. In compose.mk source code, there are more than 250 usages of ${make}, but extensions can avoid the pain. Literally if self was ${self} and this was ${this} everywhere, then the resulting RSI would be a public health crisis! 

  4. Line-feeds are allowed. 

  5. For multiple values with env, items must be single-quoted and space-delimited. As for quiet, pass 1 or 0 to control whether the target will run silently or announce context (like the compose filename/service, or the docker image being used).