Composable cljs build scripts

composable_build.md

Independent composable build scripts for cljs

There is a need more complex build scripts for ClojureScript. It would be nice to have a minmal interface and method for composing these scripts. Somehting like:

(-> (watch-cljs ["src"])
    (watch-sass [""])
    (compile-sass)
    (watch-css [""])
    (mark-macro-affected-source build-options)
    (compile-cljs build-options)
    (cljs-browser-repl)
    #_(fighweel-browser-connection)
    #_(figwheel-browser-repl)
    (ambly-repl))

It is not lost on me that this is similar to what boot currently does. I have nothing against Boot, it would just be nice to have something that was neither tied to lein or boot that allows us to create and share building tools.

This system would not require buy-in just adhearance to a simple interface.

What should this interface look like?

Right now I'm leaning towards a ring-like functional interace where individual components can use core.async or other methods to handle the nastiness of this type of async build work.

Things to consider.

1. It would be nice to be able to create these composable flows and perhaps explicitely start and stop them.

2. I am also leaning toward a message based flow. Where the current component either recieves and operates on the message eventually forwarding it or passes it on untouched.

Tiny example:

(defn compile-cljs [source-paths build-options]
  (fn [f]
    (fn [signal]
        (f (if (= [:cljs-changed] signal)
               (do (cljs.build.api/build source-paths build-options)
                   [:cljs.build/cljs-compiled {:data ...}])    
                signal)))))

I'm just putting this out there. This idea has been in my head for a year and I'd like to break figwheel down into parts that can be more easily reused.

Obviously I'm a huge fan of this approach, and the idea of a huge trans-tool ecosystem of middlewares is highly intriguing!

My experience with things like this is that the key thing is the value that passes between middlewares. The more context that can be loaded into this value, the more composable the middlewares are. In Ring, the only thing a middleware needs to know to participate in the pipeline and be useful is the shape of the request/response maps.

In Boot1 nil was threaded through, and the "pipeline" was really just an intricate metaphor for the order that the tasks needed to run in. All the actual coordination happened on the filesystem.

That's why in Boot2 we arrived at the FileSet idea - an immutable value middlewares could accept and return, just like with Ring. This way, in order to be useful and participate, middlewares only needed to know about the FileSet API - they didn't need to touch the actual input or output directories in order to function. This eliminates a class of problems related to running tasks in the correct order. It also provides all the context and capabilities middlewares need to cache work between runs, since they can safely retain old FileSets they've seen before. Users can still have problems related to middleware ordering, but at least with a threaded FileSet they are deterministic.

Long story short, if there was a single thing I could think of us to coordinate best around, it would be an immutable value type such as the FileSet. Once we have this, I bet we're off to the races.

Thanks for pushing your thoughts out on this! It's a sweet idea.

PS: This blog post about the Broccoli build tool is cool

Hi Bruce,

(defn compile-cljs [source-paths build-options]
  (fn [f]
    (fn [signal]
        (f (if (= [:cljs-changed] signal)
               (do (cljs.build.api/build source-paths build-options)
                   [:cljs.build/cljs-compiled {:data ...}])    
                signal)))))

This code snippet, that's the middleware pattern, right? Threading an entity through a series of functions designed to operate on it. This is Ring, sure, and Boot, for sure too.

Now the usage example, that pipeline:

(-> (watch-cljs ["src"])
    (watch-sass [""])
    (compile-sass)
    (watch-css [""])
    (mark-macro-affected-source build-options)
    (compile-cljs build-options)
    (cljs-browser-repl)
    #_(fighweel-browser-connection)
    #_(figwheel-browser-repl)
    (ambly-repl))

That is a middleware stack, right? That is, incidentally, how tasks are composed in Boot.

So that signal that you're threading through functions, what kind of entity would that be? If we shuffle things around a bit, we may come to a design where source-paths is the immutable fileset that Alan was mentioning, where build-options are task parameters, and we would have come full circle.

So, I am thinking about a cascading message bus. Where you react to, reduce, filter messages that you need and forward messages that you aren't interested in.

I am not thinking about a single entity like the FileSet getting passed down through the chain. That makes too many assumptions. Rather a message based approach. You can forward a message with a FileSet in a message and a component downstream can do something with it.

@bhauman the problem I've had with this approach is that upstream tasks must enumerate the signals they generate a-priori. Then, downstream tasks must be familiar with the names that indicate certain things must happen. This is as opposed to a FileSet, a context, the interpretation of which is open.

For example, the FileSet-ized cljs task could choose to do things when the FileSet it received contains .cljs files. In this way, upstream tasks don't need to know or care if cljs is downstream - their responsibility is only to (conditionally) pass the FileSet to the next handler.

The difference between an event indicating that certain work needs to be done, and a FileSet, a query over which might indicate the same, is subtle. However, the FileSet is better in my mind because it represents both an intent and all possible context required to act on that intent - without stating any particular intent. This fact is what saves users from the difficulties and inefficiencies inherent in managing both the order of events and the states they individually referred to at the time they were initiated.

Does that make sense? Datomic might be a good thing to compare to, with its idea of "the database as a value". With the FileSet, we get "the filesystem as a value" and all the immutable goodness along with it.

@danielsz I'm not sure all roads lead to the Ring/Boot/Bruce middleware style of composition. There are a lot of ways to arrange functions (monads, reducers, arrows, lenses, birds), most of which remain unexplored in this domain. What they'll all definitely need is a workable, immutable value to accept and return. If we can orient our tools around something as low level as an immutable associative type, all of these things, current and future, have the potential to compose functionally. Which would be unprecedented and awesome!

I forgot one possibly cool thing about FileSets! They can theoretically be reasonably efficiently shipped across the network.

So, we could maybe make build pipelines, some elements of which are on remote machines. Personally I'd love to have a warm cljs task in the sky. Maybe it could be a public service even?

So we may be looking at this differently. Upstream tasks are just firing events saying that "this happened" not intending or requiring further action, downstream tasks are just deciding to act if they need to.

I see FileSets as a state that says "now we have this data". Which is cool and interesting to think about. Everything that is expressed has to fit into this object. "Notes for the next guy." FileSets further arrange this around the idea of files that have changed, and this is not the only thing that I would like to express.

"Notes for the next guy" requires that you run predicates over the data to discern if such and such happened.

FileSets aside. This comes down to messages and/or state. I think I understand the generality of threading state down through the functions. But events and event listeners are a powerful means of general expression as well. Also, a single message chain where one can filter, reduce messages etc.

There is really no coupling difference in a threading a state (need to inspect and understand the shape of the state) or a message (need to know the name and content of the message). They are both general, meaning it is up to the next consumer to understand what is going on in this world.

I think messages have an advantage in that they express things that have occurred.

This is a common problem with an atom listener. In the listener we are trying to divine an occurrence that leaves an indistinct or difficult to trace mark on the state.

A compiler-component can emit "compile-start", "compile-warnings", "compile-errors", etc. These are better communicated as messages I think.

Say that we have something that is connected to the browser and a message that originates in a server-side component and all these messages are sent to the browser and on the browser we have a similar chain that extends the server side one.

But I can see how both worlds would be helpful. So maybe a full on Monad is a way to go. It's nice to have threaded messages and state.

But I do want something small, understandable, that invites people in. We aren't doing anything too crazy here. If things need to be expressed that can't be expressed this way nothing prevents folks from writing a script to do it.

Re: passing messages other than files and their content, it's worth noting that FileSets and the things in them (TmpFiles) can carry metadata. So, tasks can add non-file info to the FileSet by adding metadata to things. This is the direction we want to go in boot with accumulating error messages and warnings.

I didn't mean for anything to sound too crazy and I probably shouldn't have even mentioned the word monad 😺 I also totally see the familiarity and utility inherent in a messaging approach. I suppose I just feel that if tools were to share only one thing, in my mind an immutable data structure is a good bet.