Edamame syntax-quote optimization benchmarks

BENCHMARK_RESULTS.md

Syntax-Quote Optimization Benchmark Results

Implementation

Branch: https://github.com/yenda/edamame/tree/optimize-syntax-quote

The optimization avoids concat/sequence/seq scaffolding when no ~@ (unquote-splicing) is present, producing direct collection literals instead.

Form Size Comparison

Form	Upstream (chars)	Optimized (chars)	Reduction
`[]	83	2	97.6%
`[1 2 3 4 5]	193	11	94.3%
`[[1 2] [3 4]]	527	19	96.4%
`{:a 1 :b 2}	243	18	92.6%
`[{:a 1} {:b 2}]	605	22	96.4%
`[1 ~x 3]	149	7	95.3%
`[1 ~@xs 2]	130	130	0% (expected)

Runtime Eval Benchmark (criterium)

Benchmarks the cost of evaluating the generated code at runtime (not parsing).

Form	Upstream	Optimized	Speedup
`[]	44-78 ns	8-9 ns	5-9x
`[1 2 3 4 5]	598 ns	21 ns	28x
`[[1 2] [3 4]]	576-590 ns	18-26 ns	22-33x
`{:a 1 :b 2}	418-446 ns	15-16 ns	26-28x
`[{:a 1} {:b 2}]	539-543 ns	18 ns	30x
`[1 ~x 3]	435-448 ns	36 ns	12x
`[1 ~@xs 2]	407 ns	321-381 ns	~same

Parse Benchmark (criterium)

Parse time is essentially unchanged (~121 µs vs ~123 µs for 18 forms) since both versions do similar parsing work.

How to Run

# Eval benchmark
clj -Sdeps '{:paths ["src" "resources" "bench"] :deps {org.clojure/clojure {:mvn/version "1.12.0"} org.clojure/tools.reader {:mvn/version "1.5.2"} criterium/criterium {:mvn/version "0.4.6"}}}' -M -m eval-bench

# Parse benchmark
clj -Sdeps '{:paths ["src" "resources" "bench"] :deps {org.clojure/clojure {:mvn/version "1.12.0"} org.clojure/tools.reader {:mvn/version "1.5.2"} criterium/criterium {:mvn/version "0.4.6"}}}' -M -m syntax-quote-bench

Test Results

All 57 existing tests pass, plus 14 new tests for the optimization covering:

• Empty collections, literals, nested structures
• Unquote without splice, quoted symbols
• Eval equivalence, size reduction
• Metadata preservation, gensym, map types

eval_bench.clj

(ns eval-bench
  "Compare eval performance: optimized vs upstream syntax-quote."
  (:require [edamame.core :as e]
            [clojure.string :as str]
            [clojure.java.shell :as shell]
            [criterium.core :as crit]))

(def opts {:all true})

;; Key test forms - representative cases
(def test-cases
  [{:name "empty-vector"     :form "`[]"              :arg-syms [] :args []}
   {:name "literal-vector"   :form "`[1 2 3 4 5]"     :arg-syms [] :args []}
   {:name "nested-vectors"   :form "`[[1 2] [3 4]]"   :arg-syms [] :args []}
   {:name "literal-map"      :form "`{:a 1 :b 2}"     :arg-syms [] :args []}
   {:name "vector-of-maps"   :form "`[{:a 1} {:b 2}]" :arg-syms [] :args []}
   {:name "with-unquote"     :form "`[1 ~x 3]"        :arg-syms '[x] :args [42]}
   {:name "with-splice"      :form "`[1 ~@xs 2]"      :arg-syms '[xs] :args [[10 20]]}])

(defn make-test-fn [form-str arg-syms]
  (let [parsed (e/parse-string form-str opts)
        wrapper `(fn [~@arg-syms] ~parsed)]
    (eval wrapper)))

(defn run-benchmark []
  (println "\nRunning benchmarks...\n")

  (doseq [{:keys [name form arg-syms args]} test-cases]
    (let [f (make-test-fn form arg-syms)
          call-fn (case (count args)
                    0 #(f)
                    1 #(f (first args))
                    2 #(f (first args) (second args)))]
      (println (format "%-20s %s" name form))
      (println (format "  Parsed: %s" (pr-str (e/parse-string form opts))))
      (let [result (crit/quick-benchmark* call-fn {})]
        (println (format "  Mean: %.2f ns (±%.2f)\n"
                         (* 1e9 (first (:mean result)))
                         (* 1e9 (first (:variance result)))))))))

(defn -main [& args]
  (println "Eval Benchmark: Runtime Cost of Generated Code")
  (println "===============================================")
  (println "Branch:" (str/trim (:out (shell/sh "git" "rev-parse" "--abbrev-ref" "HEAD"))))
  (println "Commit:" (str/trim (:out (shell/sh "git" "rev-parse" "--short" "HEAD"))))

  (run-benchmark)

  (println "Done."))

(-main)

syntax_quote_bench.clj

(ns syntax-quote-bench
  "Benchmark syntax-quote optimization with criterium."
  (:require [edamame.core :as e]
            [clojure.string :as str]
            [clojure.java.shell :as shell]
            [criterium.core :as crit]))

(def opts {:all true})

;; Test cases - mix of forms that benefit from optimization
(def test-forms
  [;; Empty collections (optimization: direct literals)
   "`[]" "`{}" "`#{}" "`()"
   ;; Literals without splice (optimization: direct literals)
   "`[1 2 3 4 5]" "`{:a 1 :b 2 :c 3}" "`#{:a :b :c :d}" "`(foo bar baz)"
   ;; Nested without splice
   "`[[1 2] [3 4] [5 6]]" "`{:a {:b {:c 1}}}" "`[{:a 1} {:b 2} {:c 3}]"
   ;; With unquote but no splice
   "`[1 ~x 3]" "`{:a ~v :b 2}" "`[~a ~b ~c]"
   ;; With splice (NO optimization - needs concat machinery)
   "`[1 ~@xs 2]" "`[~@a ~@b ~@c]" "`#{1 ~@more 2}"
   ;; Complex
   "`(let [x# ~v] (inc x#))"])

(defn bench-form-size []
  (println "\n=== FORM SIZE COMPARISON ===")
  (doseq [f test-forms]
    (let [parsed (e/parse-string f opts)
          size (count (pr-str parsed))]
      (println (format "%-30s -> %4d chars" f size)))))

(defn -main [& args]
  (println "Edamame Syntax-Quote Benchmark (criterium)")
  (println "==========================================")
  (println "Branch:" (str/trim (:out (shell/sh "git" "rev-parse" "--abbrev-ref" "HEAD"))))
  (println "Commit:" (str/trim (:out (shell/sh "git" "rev-parse" "--short" "HEAD"))))

  (bench-form-size)

  (println "\n=== PARSE BENCHMARK ===")
  (println "Parsing all" (count test-forms) "forms per iteration...\n")
  (crit/quick-bench
   (doseq [f test-forms]
     (e/parse-string f opts)))

  (println "\nDone."))

(-main)