RFC-001-completeness-based-streaming-validation.md

RFC-001: Completeness-Based Streaming Validation

Field	Value
RFC	001
Title	Completeness-Based Streaming Validation
Author	Thomas Norman
Status	Implemented
Created	2026-01-13
Branch	fix/partial-recursive-models
PR	#1997

Abstract

This RFC proposes a fundamental change to how Partial[T] handles validation during streaming. Instead of attempting to validate incomplete data on every chunk (and working around failures), we introduce completeness-based validation: only validate JSON structures that are structurally complete (closed with matching braces/brackets).

Motivation

Current Problems

The current streaming implementation has several fundamental issues:

1. Field constraints fail during streaming

   class User(BaseModel):
       name: str = Field(min_length=5)
   
   # During streaming, "Al" fails min_length validation
   # even though the final value "Alice" would pass

2. Model validators crash on incomplete data

   class Task(BaseModel):
       status: Literal["active", "inactive"]
       priority: int
   
       @model_validator(mode="after")
       def check(self):
           if self.status == "active" and self.priority < 5:
               raise ValueError("Active tasks need high priority")
           return self
   
   # Crashes when status arrives but priority is still None

3. Literal/Enum types fail on partial strings

   class Task(BaseModel):
       status: Literal["active", "inactive"]
   
   # Streaming chunk: {"status": "act
   # "act" is not a valid Literal value - validation fails

4. Workarounds are fragile

   • PartialLiteralMixin drops incomplete strings (loses data)
   • Wrapping validators with streaming context checks
   • Each validator type needs individual handling

The Whack-a-Mole Problem

The current approach fixes validators one at a time:

• Wrap model validators to check for streaming context
• Use PartialLiteralMixin to drop incomplete Literal strings
• Make fields optional to skip required field checks
• ... and so on for each validation type

This is unsustainable. Each new Pydantic feature or validator type requires new workarounds.

Specification

Core Principle

Only validate JSON structures that are "closed" (complete).

• Closed object: Starts with { and ends with }
• Closed array: Starts with [ and ends with ]
• Open/incomplete: Missing closing brace/bracket → no validation

Key Insight

If a JSON structure is syntactically complete, all its contents must also be complete:

{"status": "active"}   // Complete - "active" is a complete string
{"status": "act        // Incomplete - string not closed, JSON not closed

You cannot have a complete JSON object with an incomplete string inside it, because JSON strings require closing quotes.

Validation Flow

Stream: {"user": {"name": "Al
         ↳ Root object is OPEN (no closing })
         ↳ No validation, yield partial with name="Al"

Stream: {"user": {"name": "Alice"}}
         ↳ Root object is CLOSED
         ↳ Validate against original model immediately
         ↳ If validation fails → raise error (triggers retry)
         ↳ If validation passes → yield validated object

Stream: {"user": {"name": "Alice"}, "items": [1, 2
         ↳ Root object is OPEN (items array not closed)
         ↳ No validation on root
         ↳ "user" sub-object is CLOSED → could validate user separately
         ↳ Yield partial with validated user, raw items

Stream: {"user": {"name": "Alice"}, "items": [1, 2]}
         ↳ Root object is CLOSED
         ↳ Full validation against original model
         ↳ Yield final validated object

Implementation Components

1. JSON Completeness Tracker

New module instructor/dsl/json_tracker.py:

class JsonCompleteness:
    """Track which parts of accumulated JSON are complete."""

    def analyze(self, json_str: str) -> None:
        """Analyze JSON string and determine completeness of each path."""

    def is_path_complete(self, path: str) -> bool:
        """Check if sub-structure at path is complete (closed)."""

    def is_root_complete(self) -> bool:
        """Check if the root JSON structure is complete."""

The tracker:

• Handles strings containing braces: {"msg": "use {braces}"}
• Handles escaped quotes: {"msg": "say \"hello\""}
• Tracks nested structures at arbitrary depth
• Returns completeness status for any JSON path

2. Validation-Free Partial Models

The partial model created by get_partial_model() is now a pure data container:

• No Field constraints (min_length, max_length, ge, le, etc.)
• No field validators (@field_validator)
• No model validators (@model_validator)
• All fields are Optional with default=None

def get_partial_model(cls) -> type[T_Model]:
    # Create model with BaseModel base (no inherited validators)
    # Strip all Field constraints
    # Make all fields Optional with None default
    partial_model = create_model(
        model_name,
        __base__=BaseModel,  # Not the original class
        **{name: _make_field_optional(field, strip_constraints=True) ...}
    )
    partial_model._original_model = original_model
    return partial_model

3. Completeness-Based Validation in Streaming

def process_potential_object(potential_object, partial_mode, partial_model, **kwargs):
    json_str = potential_object.strip() or "{}"
    parsed = from_json(json_str.encode(), partial_mode=partial_mode)

    tracker = JsonCompleteness()
    tracker.analyze(json_str)

    original_model = getattr(partial_model, "_original_model", None)

    if tracker.is_root_complete() and has_data(parsed) and original_model:
        # Complete JSON → validate against original model
        return original_model.model_validate(parsed, **kwargs)
    else:
        # Incomplete JSON → build partial object without validation
        return _build_partial_object(parsed, partial_model, ...)

Validation Types Affected

All validation is now handled uniformly:

Validation Type	Incomplete JSON	Complete JSON
Field constraints (min_length, etc.)	Skipped	Enforced
Field validators (@field_validator)	Skipped	Run
Model validators (@model_validator)	Skipped	Run
Type validation (str vs int)	Skipped	Enforced
Required fields	Skipped	Enforced
Literal/Enum validation	Skipped	Enforced
Discriminated unions	Skipped	Enforced

Deprecation of PartialLiteralMixin

PartialLiteralMixin is no longer necessary and is deprecated:

class PartialLiteralMixin:
    """DEPRECATED: This mixin is no longer necessary."""

    def __init_subclass__(cls, **kwargs):
        warnings.warn(
            "PartialLiteralMixin is deprecated and no longer necessary. "
            "Completeness-based validation now handles Literal and Enum types "
            "automatically during streaming. You can safely remove this mixin.",
            DeprecationWarning,
        )

Rationale

Why Completeness-Based?

1. Natural boundary: JSON syntax provides a clear, unambiguous definition of "complete"
2. No special cases: All validation types handled uniformly
3. Predictable: Users can reason about when validation runs
4. Efficient: No wasted validation attempts on incomplete data

Why Validate Immediately on Completion?

When a JSON structure becomes complete, we validate immediately rather than waiting for the entire stream:

1. Fast feedback: Errors detected as soon as possible
2. Retry efficiency: Retries can start earlier
3. Partial progress: Valid sub-objects can be used while streaming continues

Why Strip All Validation from Partial Models?

1. Simplicity: One code path for all validation types
2. Safety: No risk of unexpected validation failures
3. Performance: No validation overhead during streaming
4. Correctness: Validation against original model preserves all constraints

Backwards Compatibility

Breaking Changes

1. PartialLiteralMixin behavior change: Previously dropped incomplete strings; now stores them. The mixin now emits a deprecation warning.

2. Streaming chunk content: Partial objects now contain incomplete string values (e.g., "act" for "active") instead of None.

3. Validation timing: Validation now runs immediately when JSON is complete, not at the end of streaming.

Migration Path

1. Remove PartialLiteralMixin: No longer needed; remove from model definitions.

2. Update chunk handling: If code checked for None values to detect incomplete fields, update to check JSON completeness or just wait for final validated object.

3. Validator assumptions: Model validators can now assume all fields have values when they run (since they only run on complete JSON).

Test Plan

Unit Tests

1. JSON completeness tracker (14 tests)

   • Empty/whitespace strings
   • Complete/incomplete objects and arrays
   • Nested structures
   • Strings containing braces
   • Escaped quotes
   • All JSON value types

2. Field constraints during streaming (7 tests)

   • min_length, max_length skipped on incomplete
   • ge, le enforced on complete
   • Field validators skipped/run appropriately
   • Nested model constraints

3. Literal/Enum streaming (10 tests)

   • Incomplete strings stored without validation
   • Complete strings validated
   • Invalid complete values rejected

4. Model validators (4 tests)

   • Skipped on incomplete JSON
   • Run on complete JSON
   • Multiple validators all run

5. Integration tests

   • Real OpenAI streaming
   • Discriminated unions (now work!)
   • Self-referential models

Results

All 60 tests pass after implementation.

Reference Implementation

• Branch: fix/partial-recursive-models
• PR: #1997
• Files changed:
  • instructor/dsl/json_tracker.py (new)
  • instructor/dsl/partial.py (modified)
  • tests/dsl/test_partial.py (modified)

Open Questions

1. Sub-object Validation

Should we validate sub-objects as they become complete, even if the root is incomplete?

{"user": {"name": "Alice"}, "items": [1, 2
          ^^^^^^^^^^^^^^^^
          This is complete - validate now?

Current decision: Only validate when root is complete. This is simpler and avoids partial validation complexity.

Future consideration: Could add opt-in sub-object validation for earlier error detection.

2. Streaming Retry Behavior

When validation fails on complete JSON, should we:

• Raise immediately (current behavior)
• Accumulate errors and retry at end
• Allow configurable behavior

Current decision: Raise immediately. This provides fastest feedback and matches non-streaming retry behavior.

Prior Art & Related Issues

GitHub Issues Addressed

This RFC addresses several long-standing issues in the instructor library:

Issue	Problem	How This RFC Addresses It
#1993	Field constraints (min_length, etc.) fail during streaming	Completeness-based validation skips constraints on incomplete JSON
#1995	Model validators crash on incomplete data	Validators only run when JSON is complete
#1991	Partial streaming issues with Literal types	No longer need PartialLiteralMixin workaround
#1736	Streaming validation timing issues	Clear semantics: validate on structural completion
#1873	Required fields during streaming	Fields only required when JSON is complete
#1962	Self-referential models cause recursion	Recursion tracking prevents infinite loops

Key Prior Work

The following PRs laid the foundation for this approach:

• PR #563 (jxnl): Introduced jiter for partial JSON parsing with partial_mode. This was the key insight that enabled streaming partial objects, but didn't address validation during streaming.

• PR #1996 (jxnl): Added recursion depth tracking for self-referential models. This fix is incorporated into the current implementation.

• PR #1877 (jxnl): Validation refactoring that set the foundation for separating partial model creation from validation.

Design Context

No previous RFC or design document existed for streaming validation. The community approach was incremental fixes:

1. PartialLiteralMixin: Dropped incomplete Literal strings (lost data)
2. Validator wrapping: Check for streaming context before running validators
3. Optional fields: Make all fields optional during streaming
4. Type-specific workarounds: Handle each Pydantic feature individually

This "whack-a-mole" approach was unsustainable. Each new Pydantic feature or validator type required new workarounds.

This RFC provides the first principled, unified solution by recognizing that JSON syntax itself defines completeness. If a JSON structure is syntactically complete (all braces/brackets matched), all its contents must also be complete—making validation safe.

Conclusion

Completeness-based validation provides a principled, uniform solution to streaming validation. By recognizing that JSON syntax itself defines completeness, we eliminate the need for type-specific workarounds and provide predictable, correct behavior for all validation types.

The implementation is backwards compatible for most use cases, with clear migration paths for code that depended on PartialLiteralMixin behavior.