Skip to content

Instantly share code, notes, and snippets.

@tehranian

tehranian/json-migration-proposal.md

Created February 17, 2026 03:46
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save tehranian/3ea48bf42c43c3a5e8e44652dacfe4c5 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save tehranian/3ea48bf42c43c3a5e8e44652dacfe4c5 to your computer and use it in GitHub Desktop.
Download ZIP
AqualinkD: JSON Migration Proposal
Raw
json-migration-proposal.md

JSON Migration Research: AqualinkD

Analysis of JSON handling in AqualinkD and comparison of two migration approaches.

1. Current State: The Problem

AqualinkD builds and parses all JSON by hand using sprintf chains and character-by-character parsing. This is concentrated in one file: source/json_messages.c (1,577 lines), with secondary JSON handling in source/web_config.c.

How JSON is built today

Every JSON builder follows the same pattern: a fixed-size buffer, a running length offset, and a chain of sprintf(buffer+length, ...) calls that assemble JSON one fragment at a time.

// Real code from build_aqualink_status_JSON (json_messages.c:726)
length += sprintf(buffer+length, "{\"type\": \"status\"");
length += sprintf(buffer+length, ",\"status\":\"%s\"",getStatus(aqdata));
length += sprintf(buffer+length, ",\"panel_message\":\"%s\"",aqdata->last_message);
length += sprintf(buffer+length, ",\"version\":\"%s %s\"",aqdata->panel_cpu, aqdata->panel_rev);
// ... 80+ more sprintf calls in this one function ...
length += sprintf(buffer+length, "]}");

There are 224 sprintf/snprintf calls across json_messages.c (195 sprintf + 29 snprintf).

How JSON is parsed today

The main parser (parseJSONrequest, json_messages.c:1029) is a 56-line character-by-character state machine that destructively modifies the input buffer and sets pointers into it:

// Real code from parseJSONrequest (json_messages.c:1046-1081)
while (i < length) {
    switch (buffer[i]) {
    case '{':
    case '"':
    case '}':
    case ':':
    case ',':
    case ' ':
        if (reading == true && buffer[i] != ' ' && buffer[i] != ',' && buffer[i] != ':') {
            reading = false;
            buffer[i] = '\0';  // destructive modification
            found++;
        }
        break;
    default:
        if (reading == false) {
            reading = true;
            if (found % 2 == 0)
                request->kv[found / 2].key = &buffer[i];
            else
                request->kv[(found-1) / 2].value = &buffer[i];
        }
        break;
    }
}

Concrete problems

Problem Impact Example
No JSON escaping If aqdata->last_message contains " or \, the JSON output is invalid and breaks the web UI sprintf(buffer+length, ",\"panel_message\":\"%s\"", aqdata->last_message) — no escaping on string insertion
Buffer overflow risk sprintf has no size limit; the code relies on buffers being "big enough" build_device_JSON uses sprintf (not snprintf) for most calls, and only checks size at the very end (json_messages.c:607): if (length >= size) { LOG(..., "went over buffer size"); } — by then it's too late
Parser only handles flat key-value Can't parse nested JSON, arrays, or numbers parseJSONrequest only yields kv[i].key/kv[i].value string pairs — max 4 pairs
Massive duplication The same sprintf pattern repeated 285 times across 10+ builder functions Adding a new field to the device JSON requires careful surgery into a 270-line sprintf chain
Hard to audit Manually escaped quotes and commas make it very easy to introduce syntax errors The trailing-comma cleanup at json_messages.c:601 (if (buffer[length-1] == ',') length--) is a red flag
web_config.c string surgery save_web_config_json() uses find_nth_char() to strip a JSON wrapper by counting colons Breaks silently if the JSON structure changes

Inventory of functions that need migration

Function File Lines Complexity
build_logmsg_JSON json_messages.c:98 ~10 Simple
build_mqtt_status_JSON json_messages.c:186 ~15 Simple
build_mqtt_status_message_JSON json_messages.c:201 ~10 Simple
build_aqualink_error_status_JSON json_messages.c:213 ~5 Simple
build_aqualink_status_JSON json_messages.c:716 ~200 Medium (many fields, conditionals)
build_aux_labels_JSON json_messages.c (not shown) ~30 Medium (loops over buttons)
build_device_JSON json_messages.c:332 ~270 Complex (nested objects, arrays, homekit mode)
build_aqualink_aqmanager_JSON json_messages.c:645 ~80 Medium
build_aqualink_config_JSON json_messages.c:1307 ~270 Complex (config parameters, nested objects)
build_aqualink_simulator_packet_JSON json_messages.c ~50 Medium
parseJSONrequest json_messages.c:1029 ~56 Parsing (char-by-char)
save_web_config_json web_config.c:79 ~50 Parsing + file I/O
build_dynamic_webconfig_json web_config.c:46 ~10 Simple

2. Option A: Mongoose Built-in JSON API

Mongoose 7.19 (already vendored at source/mongoose.c / source/mongoose.h) includes a JSON API in two parts: building (via mg_snprintf with the %m/MG_ESC formatter) and parsing (via mg_json_get_* functions).

Building JSON with Mongoose

Mongoose doesn't have a DOM-style builder like cJSON. Instead, it extends its mg_snprintf with a %m format specifier and MG_ESC() macro for safe string escaping:

// How it would look using Mongoose's API
int build_aqualink_error_status_JSON(char* buffer, int size, const char *msg) {
    return (int)mg_snprintf(buffer, size,
        "{\"type\":\"status\",\"status\":\"%m\"}",
        MG_ESC(msg));
}

For the status builder (many fields):

int build_aqualink_status_JSON(struct aqualinkdata *aqdata, char* buffer, int size) {
    return (int)mg_snprintf(buffer, size,
        "{\"type\":\"status\""
        ",\"status\":\"%m\""
        ",\"panel_message\":\"%m\""
        ",\"version\":\"%m %m\""
        ",\"air_temp\":\"%d\""
        ",\"pool_temp\":\"%d\""
        // ... all other fields ...
        "}",
        MG_ESC(getStatus(aqdata)),
        MG_ESC(aqdata->last_message),
        MG_ESC(aqdata->panel_cpu), MG_ESC(aqdata->panel_rev),
        aqdata->air_temp,
        aqdata->pool_temp
        // ... all other args ...
    );
}

Parsing JSON with Mongoose

Mongoose's JSON parser uses JSONPath-style queries against a raw string — no tree allocation:

// How it would look using Mongoose's API
// Current: parseJSONrequest(buffer, &jsonkv) then linear scan
// Mongoose: direct named access
struct mg_str json = mg_str(buffer);
char *uri = mg_json_get_str(json, "$.uri");      // allocates, caller frees
char *value = mg_json_get_str(json, "$.value");   // allocates, caller frees

if (uri != NULL) {
    action_URI(uri, value, ...);
    free(uri);
}
if (value != NULL) free(value);

Iterating over JSON objects or arrays:

struct mg_str json = mg_str(buffer);
struct mg_str key, val;
size_t ofs = 0;
while ((ofs = mg_json_next(json, ofs, &key, &val)) > 0) {
    // key and val are mg_str (pointer + length, not null-terminated)
}

Pros

  • Zero new dependencies — already compiled into the binary; no additional .c/.h files to vendor
  • Zero additional memory overhead — parsing doesn't allocate a tree; building goes directly into a buffer
  • String escaping — MG_ESC() handles JSON escaping automatically
  • Size-bounded — mg_snprintf always respects the buffer size (unlike raw sprintf)
  • Stays in the Mongoose ecosystem — the project already depends on Mongoose for HTTP/WebSocket; using its JSON API is consistent

Cons

  • Building is still format-string based — the complex builders with loops and conditionals (like build_device_JSON with its 270 lines of device-array construction) are awkward. You'd need to track a running offset and call mg_snprintf(buffer+offset, size-offset, ...) in a loop, which is similar to what the code does today, just safer
  • No DOM/tree model — you can't build a JSON object incrementally, conditionally add/remove fields, or manipulate structure after creation. It's still "print it right the first time"
  • Parsing allocates per-call — mg_json_get_str() mallocs a new string each time. For parsing a message with 4 fields, that's 4 allocations + 4 frees
  • No array building helper — building JSON arrays in a loop requires manual comma management, similar to today's code
  • Less familiar API — the %m / MG_ESC pattern and JSONPath query style have a learning curve; fewer examples online than cJSON

What it looks like for the hardest case: build_device_JSON

// Sketch of build_device_JSON with Mongoose (simplified, showing the loop difficulty)
int build_device_JSON(struct aqualinkdata *aqdata, char* buffer, int size, bool homekit) {
    int len = 0;
    len += mg_snprintf(buffer+len, size-len, "{\"type\":\"devices\",\"devices\":[");

    for (int i = 0; i < aqdata->total_buttons; i++) {
        if (i > 0) len += mg_snprintf(buffer+len, size-len, ",");

        if (is_heater_button(&aqdata->aqbuttons[i])) {
            len += mg_snprintf(buffer+len, size-len,
                "{\"type\":\"setpoint_thermo\",\"id\":\"%m\",\"name\":\"%m\""
                ",\"state\":\"%s\",\"spvalue\":\"%d\",\"value\":\"%d\"}",
                MG_ESC(aqdata->aqbuttons[i].name),
                MG_ESC(aqdata->aqbuttons[i].label),
                aqdata->aqbuttons[i].led->state==ON ? "on" : "off",
                aqdata->pool_htr_set_point,
                aqdata->pool_temp);
        } else {
            len += mg_snprintf(buffer+len, size-len,
                "{\"type\":\"switch\",\"id\":\"%m\",\"name\":\"%m\""
                ",\"state\":\"%s\"}",
                MG_ESC(aqdata->aqbuttons[i].name),
                MG_ESC(aqdata->aqbuttons[i].label),
                aqdata->aqbuttons[i].led->state==ON ? "on" : "off");
        }
    }

    len += mg_snprintf(buffer+len, size-len, "]}");
    return len;
}

The loop-and-comma pattern is essentially the same as today, just with proper escaping and bounds checking.

3. Option B: cJSON Library

cJSON is a lightweight JSON library for C (MIT license). Two files: cJSON.c (~2,800 lines) and cJSON.h. It provides a DOM-style tree API for both building and parsing.

Building JSON with cJSON

cJSON lets you construct a JSON tree in memory, then serialize it into a buffer:

int build_aqualink_error_status_JSON(char* buffer, int size, const char *msg) {
    cJSON *root = cJSON_CreateObject();
    cJSON_AddStringToObject(root, "type", "status");
    cJSON_AddStringToObject(root, "status", msg);  // auto-escapes special chars

    cJSON_PrintPreallocated(root, buffer, size, 0);  // 0 = unformatted (compact)
    int len = (int)strlen(buffer);
    cJSON_Delete(root);
    return len;
}

For the status builder:

int build_aqualink_status_JSON(struct aqualinkdata *aqdata, char* buffer, int size) {
    cJSON *root = cJSON_CreateObject();
    char temp[16];

    cJSON_AddStringToObject(root, "type", "status");
    cJSON_AddStringToObject(root, "status", getStatus(aqdata));
    cJSON_AddStringToObject(root, "panel_message", aqdata->last_message);

    snprintf(temp, sizeof(temp), "%d", aqdata->pool_htr_set_point);
    cJSON_AddStringToObject(root, "pool_htr_set_pnt", temp);

    // ... other fields ...

    cJSON_PrintPreallocated(root, buffer, size, 0);
    int len = (int)strlen(buffer);
    cJSON_Delete(root);
    return len;
}

Note on number formatting: The current code formats numbers as JSON strings (e.g., "85" not 85) because the web UI JavaScript expects strings. To maintain backward compatibility during migration, you'd use cJSON_AddStringToObject with a formatted temp buffer. A future cleanup could switch to cJSON_AddNumberToObject after updating the web UI.

Parsing JSON with cJSON

cJSON parses a full JSON tree, then you access fields by name:

// Replace parseJSONrequest call site in net_services.c
cJSON *json = cJSON_Parse(buffer);
if (json != NULL) {
    cJSON *uri_item = cJSON_GetObjectItem(json, "uri");
    cJSON *value_item = cJSON_GetObjectItem(json, "value");

    if (cJSON_IsString(uri_item))
        uri = uri_item->valuestring;
    if (cJSON_IsString(value_item))
        value = value_item->valuestring;

    action_URI(uri, value, ...);
    cJSON_Delete(json);
}

Pros

  • DOM model eliminates structural bugs — you can't produce malformed JSON. No manual comma management, no mismatched braces, no forgotten closing brackets

  • Automatic string escaping — cJSON_AddStringToObject handles ", \, control characters automatically

  • Natural loop/conditional building — creating arrays of device objects is straightforward:

    cJSON *devices = cJSON_AddArrayToObject(root, "devices");
for (int i = 0; i < aqdata->total_buttons; i++) {
    cJSON *dev = cJSON_CreateObject();
    cJSON_AddStringToObject(dev, "id", aqdata->aqbuttons[i].name);
    cJSON_AddStringToObject(dev, "state", LED2text(aqdata->aqbuttons[i].led->state));
    // ... conditionally add more fields ...
    cJSON_AddItemToArray(devices, dev);
}

  • cJSON_PrintPreallocated — serializes into a caller-provided buffer (no allocation), matching the existing function signatures

  • Extremely well-known — the most popular C JSON library; abundant examples and documentation

  • Useful in tests — test code can use cJSON_Parse to validate builder output semantically (check field values) rather than exact string matching, making tests resilient to formatting changes

  • MIT license — no licensing issues

Cons

  • New dependency — adds ~2,800 lines of vendored code (cJSON.c + cJSON.h). Must be kept up to date or pinned
  • Memory overhead — each cJSON_CreateObject / cJSON_AddStringToObject call allocates memory (via malloc). A complex builder like build_device_JSON with ~30 device objects and ~10 fields each would do ~300+ allocations, then free them all after serialization. On an embedded-ish system (Raspberry Pi running the pool controller), this is fine in practice but is objectively more allocation-heavy than the current zero-allocation sprintf approach
  • Double-buffer overhead — cJSON builds a tree in memory, then serializes into the output buffer. The current code writes directly into the output buffer. For the largest messages (~12KB), the tree might temporarily use a comparable amount of memory
  • Numbers-as-strings workaround — because the web UI expects "85" not 85, you need an awkward snprintf(temp, ..., "%d", value) + cJSON_AddStringToObject pattern rather than the natural cJSON_AddNumberToObject

What it looks like for the hardest case: build_device_JSON

int build_device_JSON(struct aqualinkdata *aqdata, char* buffer, int size, bool homekit) {
    bool homekit_f = (homekit && aqdata->temp_units == FAHRENHEIT);
    cJSON *root = cJSON_CreateObject();
    char temp[16];

    cJSON_AddStringToObject(root, "type", "devices");
    cJSON_AddStringToObject(root, "aqualinkd_version", AQUALINKD_VERSION);
    cJSON_AddStringToObject(root, "date", aqdata->date);
    cJSON_AddStringToObject(root, "time", aqdata->time);

    cJSON *devices = cJSON_AddArrayToObject(root, "devices");

    for (int i = 0; i < aqdata->total_buttons; i++) {
        cJSON *dev = cJSON_CreateObject();

        if (strcmp(BTN_POOL_HTR, aqdata->aqbuttons[i].name) == 0) {
            cJSON_AddStringToObject(dev, "type", "setpoint_thermo");
            cJSON_AddStringToObject(dev, "id", aqdata->aqbuttons[i].name);
            cJSON_AddStringToObject(dev, "name", aqdata->aqbuttons[i].label);
            cJSON_AddStringToObject(dev, "state",
                aqdata->aqbuttons[i].led->state == ON ? JSON_ON : JSON_OFF);
            snprintf(temp, sizeof(temp), "%.*f",
                homekit ? 2 : 0,
                homekit_f ? degFtoC(aqdata->pool_htr_set_point) : aqdata->pool_htr_set_point);
            cJSON_AddStringToObject(dev, "spvalue", temp);
            // ... more fields ...
        } else {
            char aux_info_buf[AUX_BUFFER_SIZE];
            get_aux_information(&aqdata->aqbuttons[i], aqdata, aux_info_buf, homekit);
            cJSON_AddStringToObject(dev, "type", "switch");
            cJSON_AddStringToObject(dev, "id", aqdata->aqbuttons[i].name);
            // ... more fields, parse aux_info_buf into the object ...
        }

        cJSON_AddItemToArray(devices, dev);
    }

    // SWG, freeze protection, chiller, temperatures, etc.
    // ... each as a cJSON object added to the devices array ...

    cJSON_PrintPreallocated(root, buffer, size, 0);
    int len = (int)strlen(buffer);
    cJSON_Delete(root);
    return len;
}

The loop is cleaner: no manual comma tracking, no buffer[length-1] == ',' fixups.

4. Migration Strategy (Applies to Either Library)

Regardless of which library is chosen, the migration should follow a tests-first approach.

Phase 1: Test infrastructure

  1. Vendor a test framework (e.g., Unity — 3 files, MIT) and add a make test target to the Makefile
  2. Create test fixtures — populate struct aqualinkdata and _aqconfig_ with representative values
  3. Write tests for parseJSONrequest — the parser handles external input and is highest-risk
  4. Write tests for each build_*_JSON function — use cJSON (even if Mongoose is chosen for production) to parse the output and validate field values semantically. This avoids brittle exact-string matching that breaks when the library changes formatting

Phase 2: Migrate parsing

  1. Replace parseJSONrequest internals with the chosen library
  2. Replace web_config.c string surgery with real JSON parsing
  3. Verify all parse tests still pass

Phase 3: Migrate building (simple → complex)

  1. Migrate simple builders (error_status, logmsg, mqtt_status) — ~4 functions, ~50 lines each
  2. Migrate medium builders (aux_labels, status, simulator) — ~3 functions, including the 200-line status builder
  3. Migrate complex builders (device, config, aqmanager) — the 270-line device builder is the hardest
  4. Verify all build tests still pass after each function

Phase 4: Cleanup

  1. Remove dead code: json_chars(), printableChar(), old parse functions, unused structs
  2. Verify grep -n 'sprintf' source/json_messages.c returns zero hits (or only the serialization helper)
  3. Full build, full test suite

Risk mitigation

  • Migrate one function at a time with a commit after each. If something breaks in production, git bisect finds the exact function
  • Keep function signatures identical — callers don't change until all builders are migrated
  • Backward compatibility — format numbers as strings ("85" not 85) to match current web UI expectations. Change the API contract later in a separate PR
  • The existing buffer size constants (JSON_BUFFER_SIZE = 12288, JSON_STATUS_SIZE = 2048, etc.) should be sufficient since library-serialized output is typically the same size or smaller than hand-rolled sprintf output

5. Recommendation

Use cJSON.

The deciding factors:

Factor Mongoose JSON cJSON Winner
New files to vendor 0 2 (cJSON.c + cJSON.h) Mongoose
Building JSON with loops/conditionals Awkward (manual comma tracking) Natural (DOM tree) cJSON
Building simple flat objects Clean (mg_snprintf + MG_ESC) Clean (cJSON_AddStringToObject) Tie
JSON escaping MG_ESC() macro Automatic in all AddString calls Tie
Parsing JSONPath queries, allocates per field Full tree parse, field access by name Tie
Test utility Not useful for test assertions Ideal for parsing builder output in tests cJSON
Structural correctness guarantee Still possible to produce invalid JSON via format strings Impossible to produce invalid JSON cJSON
Memory usage Zero-alloc (prints directly) Allocates tree, then serializes Mongoose
Community / familiarity Niche API within Mongoose Most widely used C JSON library cJSON

The core argument: AqualinkD's biggest JSON problem is the 270-line build_device_JSON function that constructs a JSON array of polymorphic device objects inside a loop with complex conditionals. This is exactly the case where a DOM-style builder shines and a format-string builder doesn't help much. The Mongoose approach would improve escaping and bounds checking but would still leave the structural complexity intact.

cJSON's DOM model makes it impossible to produce structurally invalid JSON (mismatched braces, trailing commas, unclosed arrays). With Mongoose's mg_snprintf, those bugs are still possible — just less likely.

The extra ~2,800 lines of vendored code is a reasonable trade-off for a project that already vendors Mongoose's 15,000+ lines.

When Mongoose's JSON API is the right choice

If AqualinkD only had simple flat-object builders (like the MQTT status messages), Mongoose's built-in API would be the clear winner: zero new dependencies, zero extra memory, clean one-liner builders. For small format-string-friendly JSON, it's excellent.

It could also make sense to use both: Mongoose's mg_json_get_* for simple parsing (it's already linked), and cJSON for building complex objects. But mixing two JSON libraries adds cognitive overhead and isn't worth it when cJSON handles both well.

Appendix: cJSON Line Count Impact

Estimated lines of project code saved if migrating to cJSON, measured per-function.

JSON builders

Function File:Line Current cJSON Est. Delta Notes
build_device_JSON json_messages.c:332 290 ~120 -170 Biggest win: loop+array building, no comma tracking
build_aqualink_config_JSON json_messages.c:1307 269 ~130 -139 json_cfg_element (89 lines) becomes trivial
build_aqualink_status_JSON json_messages.c:716 209 ~90 -119 80+ sprintf calls become shorter AddString calls
json_cfg_element json_messages.c:1195 89 0 -89 Replaced by direct cJSON_AddXToObject at call sites
build_aqualink_aqmanager_JSON json_messages.c:645 70 ~35 -35
get_aux_information json_messages.c:266 64 ~35 -29 Nested object building simplifies
build_aqualink_simulator_packet_JSON json_messages.c:977 47 ~25 -22
json_cfg_element_OLD json_messages.c:1160 31 0 -31 Dead code, deleted
build_aux_labels_JSON json_messages.c:926 21 ~12 -9
logmaskjsonobject json_messages.c:623 16 ~8 -8
build_mqtt_status_JSON json_messages.c:186 14 ~6 -8
build_logmsg_JSON json_messages.c:98 11 ~7 -4
build_mqtt_status_message_JSON json_messages.c:201 11 ~5 -6
build_aqualink_error_status_JSON json_messages.c:213 6 ~4 -2
logleveljsonobject json_messages.c:639 5 ~4 -1

Parsing

Function File:Line Current cJSON Est. Delta Notes
parseJSONrequest json_messages.c:1029 56 ~15 -41 Or deleted entirely if call sites use cJSON directly

Deletable helpers (cJSON handles escaping)

Function File:Line Lines Notes
json_chars json_messages.c:64 33 cJSON handles JSON escaping automatically
printableChar json_messages.c:50 13 Only used by json_chars
parseJSONwebrequest json_messages.c:1087 71 Already commented out / dead code

web_config.c

Function File:Line Current cJSON Est. Delta Notes
save_web_config_json web_config.c:79 48 ~25 -23
fprintf_json web_config.c:137 46 0 -46 Replaced by cJSON_Print()
find_nth_char web_config.c:62 13 0 -13 No longer needed
fprint_indentation web_config.c:131 5 0 -5 No longer needed

Totals

Category Lines removed Lines added Net
Builder simplification ~672 ~0 -672
Deleted helpers ~117 ~0 -117
web_config.c cleanup ~87 ~0 -87
New json_serialize helper 0 ~10 +10
Project code total ~876 ~10 ~-866
cJSON vendored files 0 ~2,800 +2,800

Bottom line: ~866 lines of project code eliminated. json_messages.c drops from ~1,577 lines to ~700 lines. The code that remains is more readable — cJSON_AddStringToObject(root, "status", msg) is self-documenting compared to sprintf(buffer+length, ",\"status\":\"%s\"", msg).

The trade-off is vendoring cJSON's ~2,800 lines, but that's library code you never touch. The code you maintain shrinks by more than half.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment