I've spent the last weekend making the first version of the MoonBit Golem SDK, a library and tool for writing "code-first" agents on the Golem platform.
While doing so I've ran into various problems / had some general thoughts that I'm trying to summarize in this document, with the intention of giving a constructive feedback.
(Note I'm not actively using MoonBit yet, just taking a look every few months)
The new Golem SDKs are fully code-driven, which means we need to capture some information (such as schema and interface) compile time for it to work. In the TS SDK it's done with a ts-morph based preprocessing step, and in the Rust SDK it's based on proc macros.
Based on help I received on Discord, I've created a separate preprocessor step in MoonBit using the moonbitlang/parser and moonbitlang/formatter packages. In this section I've collected some feedback about this part of the experience.
The obvious first one is that I have to do this at all - create a sepeate CLI project and integrate it in my (my SDK's users's) build flow is making this much harder than it should be.
The currently published latest parser and formatter are not compatible with each other, having an incompatible transitive dependency. I had to fork formatter and update it to depend on the latest parser to work on it.
Note: the current latest (unpublished) main branch of formatter also does not seeem to work - my coding agents says the git dependencies are not working and resolving to the published version, I did not verify this claim.
There was a step where I wanted to parse .mbti files but the mbti parser was not up-to-date. I've seen some updates to the parser package since last weekend so maybe this is now solved.
I couldn't find any package (or feature in the parser/formatter packages) to work on the new pkg format, so had to do some fragile text-based manipulation on them.
I was trying to integrate this deriver preprocessor tool using a pre-build hook, which almost worked but because of some reasons (will get to there) I also need to modify .pkg files, it turned out that the pre-build hook is too late in the build process. The packages are already resolved when it runs so its changes are not affecting the rest of the build.
Golem agents are compiled to WASM (non-gc) using WASI p2. They have to implement a (few) Goelem specific WIT interfaces and have access to some host functionality through other Golem specific WIT interfaces plus the "usual" WASI interfaces.
With the Golem SDK my goal is that users are not having to deal with this at all - they just use the MoonBit package(s) to define agents and the whole complexity of the component model, WIT files, bindings etc are hidden from them.
The most important missing feature to be a first-class citizen in Golem is async support. We'd need an async runtime implementation for MoonBit that runs on top of WASI Pollables, just like wstd for Rust.
It turned out that we can't just export the Golem WIT interfaces in our SDK, depend on the SDK as a library and then compile the user's code to WASM - it won't have the WIT exports. The user's code explicitly need to reexport every necessary export as a public function and also they have to be listed in the user's .pkg file's link section.
This is obviously something we can't expect our users to do (there are a lot of such exports and our SDK's goal is that they don't know about them - they use the higher level abstractions). So I had to add a reexport generator in the preprocessor tool, that generates an .mbt file exporting all the same functions that the SDK is exporting, delegating the calls to that, and also modifies the user's .pkg file by adding the link section to it.
With this, the resulting WASM is still not a WASM Component and need to be componentized with wasm-tools. These are post-build steps that our Golem CLI can easily hide from the users, but in general I think it would be nice to have not only pre-build but also post-build hooks in MoonBit.
I've found a bug in wit-bindgen, fixed it and it's already merged. Thanks! (bytecodealliance/wit-bindgen#1553)
The wit-bindgen mooonbit command generates a lot of separate directories to the root of the package and some of these are stubs that are going to be implemented so can't be just regeneratd later. Regenerating the stubs can be skipped with a CLI flag which is good, but in case the WIT interfaces change significantly, it is hard to regenerate everything - there will be lot of leftovers from the previous bindings that needs cleaning up.
(Compare this for example to the experience with Rust where it generates a single bindings.rs file that can be safely regenerated).
The wit-bindgen tool still generates .pkg.json files, and then moon fmt converts those to the new format and leaves the json files there too. As there are tons of packages generated by wit-bindgen, this is many files in many directories to clean up.
Maybe moon fmt should (optionally?) just remove the migrated files, or wit-bindgen should generate the new format.
This is probably more like a language feature request and not the binding generator's issue, but the generated wrappers for WIT resources need to be explicitly drop()ed which is very easy to forget / do right, and without some kind of destructors / finalizers the only way I can hide this from our users is to wrap everything in a callback-style API which is not nice.
Some random other thoughts / problems.
When I build my project every directory gets a generated .mbti file. I understand this is a feature but for me it feels quite noisy and makes it harder to see my actual changes. Maybe there could be an option to put them in the _build directory?
I've ran into an ICE with a wrong generated code, reported it it's probably already fixed, thanks! (moonbitlang/moonbit-docs#1137)
This one is quite weird and I did not investigate it further - can be something on the Golem side too; but when building a debug WASM of my example, when I'm trying to use it with Golem, the (wasmtime based) logic that invokes the metadata extraction in it goes to an infinite recursin and dies. Never seen this with any other WASM before.
I was looking for a generic logging library in the MoonBit ecosystem that I can extend with a Golem-specific (actually wasi-logging specific) logging sink. logr looked good but it is not open for extending with new log sinks, I think it would be nice if it would be.
When having multiple inter-depenent MoonBit packages, I can either use filesystem dependencies or published packages. When working on them I want the filesystem one, but when making a release I want to have the published ones. I ended up writing a "switch" tool to switch between them, but I think there should be some official help from the MoonBit tooling for this.