Composable cljs build scripts

composable_build.md

Independent composable build scripts for cljs

There is a need more complex build scripts for ClojureScript. It would be nice to have a minmal interface and method for composing these scripts. Somehting like:

(-> (watch-cljs ["src"])
    (watch-sass [""])
    (compile-sass)
    (watch-css [""])
    (mark-macro-affected-source build-options)
    (compile-cljs build-options)
    (cljs-browser-repl)
    #_(fighweel-browser-connection)
    #_(figwheel-browser-repl)
    (ambly-repl))

It is not lost on me that this is similar to what boot currently does. I have nothing against Boot, it would just be nice to have something that was neither tied to lein or boot that allows us to create and share building tools.

This system would not require buy-in just adhearance to a simple interface.

What should this interface look like?

Right now I'm leaning towards a ring-like functional interace where individual components can use core.async or other methods to handle the nastiness of this type of async build work.

Things to consider.

1. It would be nice to be able to create these composable flows and perhaps explicitely start and stop them.

2. I am also leaning toward a message based flow. Where the current component either recieves and operates on the message eventually forwarding it or passes it on untouched.

Tiny example:

(defn compile-cljs [source-paths build-options]
  (fn [f]
    (fn [signal]
        (f (if (= [:cljs-changed] signal)
               (do (cljs.build.api/build source-paths build-options)
                   [:cljs.build/cljs-compiled {:data ...}])    
                signal)))))

I'm just putting this out there. This idea has been in my head for a year and I'd like to break figwheel down into parts that can be more easily reused.

So, I am thinking about a cascading message bus. Where you react to, reduce, filter messages that you need and forward messages that you aren't interested in.

I am not thinking about a single entity like the FileSet getting passed down through the chain. That makes too many assumptions. Rather a message based approach. You can forward a message with a FileSet in a message and a component downstream can do something with it.

@bhauman the problem I've had with this approach is that upstream tasks must enumerate the signals they generate a-priori. Then, downstream tasks must be familiar with the names that indicate certain things must happen. This is as opposed to a FileSet, a context, the interpretation of which is open.

For example, the FileSet-ized cljs task could choose to do things when the FileSet it received contains .cljs files. In this way, upstream tasks don't need to know or care if cljs is downstream - their responsibility is only to (conditionally) pass the FileSet to the next handler.

The difference between an event indicating that certain work needs to be done, and a FileSet, a query over which might indicate the same, is subtle. However, the FileSet is better in my mind because it represents both an intent and all possible context required to act on that intent - without stating any particular intent. This fact is what saves users from the difficulties and inefficiencies inherent in managing both the order of events and the states they individually referred to at the time they were initiated.

Does that make sense? Datomic might be a good thing to compare to, with its idea of "the database as a value". With the FileSet, we get "the filesystem as a value" and all the immutable goodness along with it.

@danielsz I'm not sure all roads lead to the Ring/Boot/Bruce middleware style of composition. There are a lot of ways to arrange functions (monads, reducers, arrows, lenses, birds), most of which remain unexplored in this domain. What they'll all definitely need is a workable, immutable value to accept and return. If we can orient our tools around something as low level as an immutable associative type, all of these things, current and future, have the potential to compose functionally. Which would be unprecedented and awesome!

I forgot one possibly cool thing about FileSets! They can theoretically be reasonably efficiently shipped across the network.

So, we could maybe make build pipelines, some elements of which are on remote machines. Personally I'd love to have a warm cljs task in the sky. Maybe it could be a public service even?

So we may be looking at this differently. Upstream tasks are just firing events saying that "this happened" not intending or requiring further action, downstream tasks are just deciding to act if they need to.

I see FileSets as a state that says "now we have this data". Which is cool and interesting to think about. Everything that is expressed has to fit into this object. "Notes for the next guy." FileSets further arrange this around the idea of files that have changed, and this is not the only thing that I would like to express.

"Notes for the next guy" requires that you run predicates over the data to discern if such and such happened.

FileSets aside. This comes down to messages and/or state. I think I understand the generality of threading state down through the functions. But events and event listeners are a powerful means of general expression as well. Also, a single message chain where one can filter, reduce messages etc.

There is really no coupling difference in a threading a state (need to inspect and understand the shape of the state) or a message (need to know the name and content of the message). They are both general, meaning it is up to the next consumer to understand what is going on in this world.

I think messages have an advantage in that they express things that have occurred.

This is a common problem with an atom listener. In the listener we are trying to divine an occurrence that leaves an indistinct or difficult to trace mark on the state.

A compiler-component can emit "compile-start", "compile-warnings", "compile-errors", etc. These are better communicated as messages I think.

Say that we have something that is connected to the browser and a message that originates in a server-side component and all these messages are sent to the browser and on the browser we have a similar chain that extends the server side one.

But I can see how both worlds would be helpful. So maybe a full on Monad is a way to go. It's nice to have threaded messages and state.

But I do want something small, understandable, that invites people in. We aren't doing anything too crazy here. If things need to be expressed that can't be expressed this way nothing prevents folks from writing a script to do it.

Re: passing messages other than files and their content, it's worth noting that FileSets and the things in them (TmpFiles) can carry metadata. So, tasks can add non-file info to the FileSet by adding metadata to things. This is the direction we want to go in boot with accumulating error messages and warnings.

I didn't mean for anything to sound too crazy and I probably shouldn't have even mentioned the word monad 😺 I also totally see the familiarity and utility inherent in a messaging approach. I suppose I just feel that if tools were to share only one thing, in my mind an immutable data structure is a good bet.