Datascript + xitdb: your humble, single-file, mini Datomic

datascript+xitdb.md

xitdb-clj is a new immutable database for Clojure. It can time travel like Datomic, and store data in a single file like SQLite. On its own, though, it has no query language. Enter Datascript...

Datascript is normally an in-memory database, but it lets you use anything as backing storage. What if we combined them?

Datascript's storage feature doesn't tell us exactly what changed, so I added one more trick: Editscript, the diffing library. I use it to figure out how to update the database with the minimal number of changes.

The end result is a sort of mini Datomic that writes to a single file. And yet, this setup can do something Datomic can't: it can branch off of any old copy of the database. The example below reverts to an older copy of the database.

Keep in mind that all reads and writes are incremental. The database is not being loaded into memory at once, so you can use it to work with larger-than-memory data.

To run this example:

git clone https://gist.github.com/663116fcd204f3f89a7e43f52fa676ef.git datascript+xitdb
cd datascript+xitdb
clj -M db.clj

...which will print this on the first run:

The old copy of the db, before retracting Alice:
#{[1 Alice 24] [2 Bob 45]}

The latest copy of the db:
#{[2 Bob 45]}

The latest copy of the db after reverting:
#{[1 Alice 24] [2 Bob 45]}

db.clj

(require '[xitdb.db :as xdb]
         '[datascript.core :as d]
         '[editscript.core :as e])
(import '[datascript.storage IStorage])

;; init xitdb
(def db (xdb/xit-db "main.db"))

;; tell datascript how to work with xitdb
(defn storage
  ([] (storage -1)) ;; the latest copy of the database
  ([history-index]
   (reify IStorage
     (-store [_ addr+data-seq]
       (swap! db (fn [db]
                   (reduce
                     (fn [m [addr data]]
                       ;; use editscript to make and apply a diff of the data.
                       ;; editscript's `patch` fn uses Clojure fns like `assoc` to
                       ;; apply the diff. since xitdb implements all the protocols,
                       ;; this works! it will update the data incrementally.
                       (->> (e/diff (get m addr) data)
                            (update m addr e/patch)))
                     (or db {})
                     addr+data-seq))))
     (-restore [_ addr]
       (-> db (xdb/deref-at history-index) (get addr))))))

;; init datascript
(def conn (or (d/restore-conn (storage))
              (d/create-conn {:name {:db/cardinality :db.cardinality/one}
                              :age  {:db/cardinality :db.cardinality/one}}
                             {:storage (storage)})))

;; add some entities
(d/transact! conn [{:name "Alice" :age 24}
                   {:name "Bob" :age 45}])

;; save the current history index so we time travel with it later
(def history-index (dec (count db)))

;; retract an entity
(def alice-id (ffirst (d/q '[:find ?e :where [?e :name "Alice"]] @conn)))
(d/transact! conn [[:db.fn/retractEntity alice-id]])

;; print the old copy of the db
(println "The old copy of the db, before retracting Alice:")
(println
  (d/q '[:find  ?e ?n ?a
         :where [?e :name ?n]
                [?e :age  ?a]]
       (d/restore (storage history-index))))
(println)

;; print the latest db
(println "The latest copy of the db:")
(println
  (d/q '[:find  ?e ?n ?a
         :where [?e :name ?n]
                [?e :age  ?a]]
       @conn))
(println)

;; revert to the old copy of the db
(reset! db (xdb/deref-at db history-index))
(def conn (d/restore-conn (storage)))

;; print the latest db
(println "The latest copy of the db after reverting:")
(println
  (d/q '[:find  ?e ?n ?a
         :where [?e :name ?n]
                [?e :age  ?a]]
       @conn))

deps.edn

{:deps {io.github.codeboost/xitdb-clj {:mvn/version "0.3.0"}
        datascript/datascript {:mvn/version "1.7.8"}
        juji/editscript {:mvn/version "0.7.0"}}}