Style & Idioms

A catch-all section for some documentation that doesn't fit anywhere else. This page is a work in progress, but here's a few things that seem worth writing down.

Style especially should be practical rather than prescriptive¹. Much of this advice is applicable only to usage of compose.mk specifically (or weird Makefiles in similar vein), but some advice here also applies to classical Makefile. CMK-lang is an optional superset of Makefile, and documented separately here, but much of this advice applies to that too.

• Naming Conventions
• Internal Style
• Style for Extensions
• Performance and Currying
• From Style to Dialect

Idioms are scattered throughout the documentation. Both the standard library walkthrough and the compiler docs are full of stuff that qualifies, but this page focuses on through-lines, macros, and make-functions, since other docs focuses on the more composable target API.

• Bind Declarations
• Import Statements
• Argument Parsing

Python Inspiration

Makefile is already a peculiar language because it handles 2 syntaxes simultaneously: one for shell, and one for Makefile-proper. If you're using compose.mk for polyglots or matrioshkas or other supported idioms, that can add even more syntax. This can make readability challenging, but strong conventions can help a lot.

With compose.mk, much inspiration is taken from python. A few random examples:

• Default target is __main__ unless .DEFAULT_GOAL is explicitly set
• Double-under "specials" are set by default: __file__, __interpreter__ & __interpreting__
• Usage of self conventionally indicates a non-public interface, e.g. with compose.bind.target
• Usage of make-prereqs as pseudo-functions resembles decorators
• Python-style docstrings for targets are supported
• Emulation of the atexit module semantics
• Support for keyword-args

General Naming Conventions

1. Dot-path style namespacing are a honking great idea (e.g. module.target).
2. Snake-case style target names are recommended when they will help with tab-completion (e.g. module.target_name).
3. Kebab-case for targets (e.g. my-target-name) is fairly common in Makefiles out in the wild, so it is explicitly reserved for project code, never used for targets in compose.mk, and is not recommended for library-style extensions of compose.mk.
4. Dot-paths should be used instead of snake-case if targets resemble keywords. i.e. if _ is just too onerous and . is faster. For example flux.if.then/<targets>.
5. Avoid custom functions in make, but if you need them, differentiate functions from vars using _ as a prefix to indicate args are required.
6. Differentiate "public" targets (i.e. the CLI's API) from internal helpers by using a self. prefix or simply .

Bind Declarations

Bind declarations for targets are similar to function-decorators, and actually look like decorators in CMK-lang. But what is being "bound" to what here?

Sometimes bind-declarations attach external variables as local defaults, allowing targets to support keyword-args

Sometimes bind-declarations bind existing code(s) or targets, to an existing runtime context like a target or a container.

Bind-declarations like in the 2nd category usually have an import-equivalent. You can think of these as import partials, i.e. you bring your own target for the sake of being explicit, whereas with an import statement the target name would usually be implied.

Import Statements

Whereas bind-declarations attach things, import-statements create (or "scaffold") new targets dynamically. All of this is documented elsewhere, but this section puts the related stuff in one place.

Targets As Functions

Targets-as-functions are pretty normal for lots of Makefile hackery, although many people would consider it an abuse! Others might argue that strong notational conventions can sometimes elevate a hack into an art form. ;)

For a lot of different reasons, compose.mk needs some way to simulate "functions" and "arguments" from mere "targets" without wrecking readability. This adds complexity, but is also very powerful, basically making the simple concept of flat DAGs like "clean -> build -> test" much more parametric. Besides adding function-calls, it effectively makes prerequisites more expressive, and more like function-decorators.

Extensions for compose.mk (or just random Makefiles) can potentially benefit from judicious application of this technique also, so here's a suggested approach.

For a more detailed look at advanced topics, see also the section on performance and currying, and a look at homoiconicity in compose.mk.

Arguments, Parameters, Pipes

Thinking of targets as functions raises the question of how we're going to pass data. See also the standard lib docs on argument-parsing.

Parametric targets should prefer a form like module.target_name/<arg>, i.e. use / to separate target-name from arguments. This works fine even for multiple file-arguments; parsing help is available from mk.unpack.arg(<arg_num>,<optional_default>). Since it's namespaced with a module-prefix, the target is unlikely to collide with your filesystem or require annoying usage of .PHONY everywhere.

In case your targets-as-functions have argument(s) that need to include spaces, newlines, etc.. you have no choice but to use env-vars or pipes. Prefer pipes where possible and consider passing/parsing structured data, because env-vars tend to be harder to organize and/or leak out of intended scope.

For clarity, targets that expect pipes should be considered honorary members of the stream module (e.g. stream.target_name), even if the implementation is part of extensions.

Runtime checks for required env-vars are good practice, and there is some support for this in the standard library too.

Internal Style

In terms of interface and in terms of internal style, compose.mk tends to avoid macros, function calls, and and other fancy Makefile'isms as much as possible. Compared with targets, they are often more magical, less readable, less reusable, and less composable. Still, there are many places where compromising on this makes sense.

1. For things like logging and internal implementation of autogenerated target scaffolding, etc.. we can't really avoid $(call fxn, args).²
2. Sometimes macros actually improve readability (for example writing ${stream.stdin} instead of cat /dev/stdin). We adopt those approaches for clarity in places where they seem self-documenting.
3. Macro expansion is used sometimes as an optimization, to avoid extra processes or recursive calls. Still.. macros are usually safe to ignore, because in most cases there's a target-equivalent available.

See the next section, and the section on performance and currying for more detailed discussion.

Style for Extensions

For code that's extending compose.mk, there's a range of options with most things. In the spirit of pragmatism over prescription.. you'll probably want to choose based on what/who you're already working with.

Depending on your appetite for magic and syntactic sugar, you could choose between:

• Explicit container dispatch (i.e. typical usage <container>.dispatch/<target>)
• Implicit container dispatch (i.e. namespace-style dispatch <ns>/<container>/<target>)

For polyglots you could choose between:

• Explicit polyglot invocation (i.e. less magic and manual control)
• Implicit polyglot invocation (i.e. more magic, using polyglot.import or CMK-lang)

For new development though and depending on how many tool containers you use, implicit style might be more idiomatic. (See the platform lifecycle demo for clean and practical example usage with a concrete problem.)

Revisiting the earlier idea that "sometimes macros improve readability" and knowing that many targets have equivalent macros, there's often a similar choice about implicit vs explicit style:

include compose.mk 

explicit:
    ${make} <target>/<argument>

implicit:
    ${target}/<argument>

When a target is used often enough, the explicit style becomes tedious to read and write. So whenever it won't confuse someone else an implicit style might be preferred, especially for larger programs.² See the next section for more detailed discussion.

Performance and Currying

In terms of performance, assignments and trivial expansions are very cheap⁶. This suggests that you're better off not using things like $(shell ..) in most expansions. Instead of doing actual work, the best way to use expansions is often about code-reuse, i.e. using snippets of shell across multiple targets, then letting targets call targets to do work. Doing this effectively is subtle and requires some understanding of flavors of Makefile variables.

Here's a quick example of a common pattern in compose.mk that uses a local tool if present, falling back to docker if necessary.

tool.docker=docker run ...

tool:=$(shell which tool 2>/dev/null || echo "${tool.docker}")

target1:
    ${tool} ...

target2:
    ${tool} ...

However, many targets in compose.mk also have macro-equivalents, which is more related to syntax than reusing computations or code. This is discussed elsewhere in this page, and is related to effectively using targets as "pseudo-functions", which is described in the section on arguments, pipes, and parameters. For related reasons, it's often a good idea to differentiate "nullary" expansions from ones that require parameters. Consider the following example:

# macro that accepts implicit arguments
foo=echo doing something with 

# target equivalent 
foo/%:
    ${foo}/${*}

# implicitly reusing foo, via macro 
implicit-style:
    ${foo}/bar

# explicitly reusing foo, via target
explicit-style:
    ${make} foo/bar

Above an implicit-style target saves a process and cleans syntax, but it only works because foo accepts zero arguments, or, an optional extra argument at the end.. essentially a coincidence based on the nature of echo.

Here's the naive way to use the argument somewhere in the middle of the expansion.. this is instructive and a useful trick, but we can do better.

# Private macro that requires explicit arguments
self.hello=printf "hello ${1}\n" 

# Primary target, available for API-style usage or from CLI
# this MUST use private macro to avoid infinite-recursion.
hello/%:
    $(call self.hello, ${*})

# public macro, cleaning syntax for library-style reuse of target.
hello=${make} hello/

# Implicitly reusing `hello/%` target, via macro. 
# this is basically using an internal API / library style call
implicit-style:
    ${hello}/world

# Explicitly reusing `hello/%` target, via recursion.
# this is basically using the "external" API via CLI.
explicit-style:
    ${make} hello/world

Above, implicit-style is only syntactic sugar that cleans up a bit but can't actually save a process. Ultimately it is the same as explicit-style. That's a shame since removing a recursive-make is potentially a significant performance optimization if your project is doing expensive stuff during bootstrap (like say, generating scaffolded targets for multiple compose files). In terms of argument-passing though the code above is fairly readable with consistent notation: both implicit/explicit style reuse are avoiding $(call ..) and the intent is fairly clear.

Unfortunately though the last version of the hello/% implementation is a bit ugly, plus we needed both a public version of the macro and a private helper version. One way to deal with this is similar to currying, ensuring that arguments can appear at the end even if they are used in the middle. Doing this with bash does make the macro itself more ugly, but cleans up both syntax and efficiency of implementations elsewhere:

# Public macro that accepts optional/implicit arguments.
# Accepts argument at the end, currying it towards middle
hello=bash -c 'printf "$${1\#/}\n"' -- 

# Parametric usage via CLI-API can use the 
# public macro for it's internal implementation
hello/%:
    ${hello}/${*}

# Implicit-style achieves code reuse using  
# a library-style call, but it still uses public macro.
implicit-style:
    ${hello}/alice

# Explicit-style reuses code via CLI API and recursion.
# Looks almost like implicit-style, thus refactoring between 
# implicit/explicit style is easy.
explicit-style:
    ${make} hello/bob

Two main things are happening now in the hello macro at the top. The first is an inlined script that accepts and uses an argument (hence the double-escaped $$1 because it's a bash-var and not a make-var). The second thing is using string-interpolation to strip the leading /, just to remain perfectly faithful to the original hello-world implementation.

Implicit-style is not only more pretty than explicit-style now, it's also significantly more efficient, and we eliminated not just one usage of $(call) but all of them in the future. When in doubt about whether an internal API exists, you can default to using the explicit approach with the CLI, and easily refactor later for any necessary performance optimizations.

To really emphasize the notational improvements, let's set aside our separate targets for implicit-style and explicit-style, add prerequisites, and put it all together: Now prerequisites, explicit style, and implicit style have a uniform style.

bonus: hello/alice hello/bob 
    ${hello}/charlie
    ${make} hello/world

Obviously, this hack is maybe not that appealing for something so simple. But by using define .. endef, of course the hello macro might be a large multiline, might be doing something much more complicated. To put it another way, eliminating $(call) enhances readability, and while replacing it with bash actually decreases efficiency, a clean invocation of bash is usually often much better than recursive calls to make.

A pretty good compromise overall.. balancing readability, code reuse, consistent notation, and ensuring public APIs remain accessible but private APIs are usable, all at the same time.

From Style to Dialect

Somewhere at the limit of style, syntax, and sugar you run into ideas like dialects³, extensible programming⁴, language-oriented programming⁵ and DSLs. To support this type of thing, CMK-lang allows for direct control of some aspects of the language syntax. See also the main docs for Compiler & Dialects.

References