EP Terraform Module URI-Based Ingest — Design Update

tf-module-uri-design.md

Terraform Module URI-Based Ingest — Design

Summary

Terraform modules are referenced as source URIs directly in toc.yaml. The URI is the single source of truth for where a module lives. During content sync, Vandoor resolves the URI, fetches the module using the GitHub App, and stores it with a commit SHA for change detection.

---

toc.yaml

- title: AWS
  terraform_module: github.com/replicatedhq/chartsmith-terraform//modules/aws?ref=v1.0.0
  visible_when:
    entitlements:
      - isAWSEnabled

- title: Google Cloud
  terraform_module: github.com/replicatedhq/chartsmith-terraform//modules/gcp?ref=v1.0.0
  visible_when:
    entitlements:
      - isGCPEnabled

- title: Azure
  terraform_module: github.com/other-org/azure-infra//modules/aks?ref=main
  visible_when:
    entitlements:
      - isAzureEnabled

The replicated/*.yaml files are eliminated. The URI follows Terraform's standard source syntax:

github.com/{owner}/{repo}//{path}?ref={tag|branch|commit}

---

URI Format

github.com/replicatedhq/chartsmith-terraform//modules/aws?ref=v1.0.0
│          │              │                            │              │
│          │              │                            │              └─ Git ref (tag, branch, or commit SHA)
│          │              │                            └─ Path within repo
│          │              └─ Repository name
│          └─ Owner (org or user)
└─ Host (github.com for now; extensible to other hosts later)

Parsed struct

type TerraformModuleSource struct {
    Host  string // "github.com"
    Owner string // "replicatedhq"
    Repo  string // "chartsmith-enterprise-portal"
    Path  string // "terraform/aws"
    Ref   string // "v1.0.0"
}

// Derived:
//   Name:    last segment of Path → "aws"
//   RepoURL: https://github.com/{Owner}/{Repo}

Rules

• // separates repo from path within the repo (standard Terraform source convention)
• ?ref= is required — we always want an explicit ref, not implicit default branch
• The module name is derived from the last path segment (e.g. terraform/aws → aws)
• The URI may point to the same repo as the content repo, or a different repo — both are valid
• If the URI points to a different repo, the GitHub App installation must have access to it; if not, the ingest fails with a clear error

---

Ingest Flow

Content sync (SyncContentFromRepo) gets a new step after syncing pages and toc:

Clone content repo (existing)
        │
        ▼
Parse toc.yaml (existing)
        │
        ▼
Sync markdown pages (existing)
        │
        ▼
NEW: Extract terraform_module URIs from parsed toc
        │
        ▼
For each URI:
        │
        ├─ Parse URI → owner, repo, path, ref
        │
        ├─ Resolve ref to commit SHA
        │  (GitHub API: GET /repos/{owner}/{repo}/commits/{ref})
        │  Uses GitHub App installation token
        │
        ├─ Compare resolved SHA to stored last_ingest_commit
        │  (unchanged?) → skip, module is current
        │  (changed or new?) ▼
        │
        ├─ Fetch module files at that commit
        │  (GitHub API: GET /repos/{owner}/{repo}/contents/{path}?ref={sha}
        │   or tarball download for the subtree)
        │
        ├─ Validate: must contain main.tf (or *.tf files)
        │
        ├─ Parse:
        │  ├─ variables.tf → extract variable blocks (name, type, default, description)
        │  ├─ outputs.tf   → extract output blocks (name, description)
        │  └─ README.md    → store raw markdown for EP display
        │
        ├─ Store in ep_terraform_module + ep_terraform_module_version
        │  (with resolved commit SHA for change detection)
        │
        └─ On error: store error in ep_terraform_module.last_ingest_error
           (e.g. "ref v1.0.0 not found", "no access to repo",
            "no .tf files found at terraform/aws")

Key details

• GitHub App token reuse: The same installation token fetched for cloning the content repo is passed to the TF module ingest step. If the module URI points to a repo the installation doesn't have access to, it fails with a stored error.
• Commit SHA tracking: Even for tags (which look immutable), we resolve to a commit SHA. If someone force-pushes a tag, the SHA changes and we re-ingest. This catches tag mutations.
• Cross-repo modules: The URI can point to any repo. The content repo might be acme/docs while a TF module lives in acme/terraform-aws. As long as the GitHub App installation has access to both, it works.
• Stale module cleanup: If a terraform_module URI is removed from toc.yaml, the module record is kept but marked as unlinked (not deleted — preserves pull event history).

---

DB Schema Changes

ep_terraform_module (updated)

-- Replace source_repository + source_path with source_uri
-- Add ingest tracking columns

ALTER TABLE ep_terraform_module
  ADD COLUMN source_uri       VARCHAR(2048) NOT NULL DEFAULT '' AFTER source_path,
  ADD COLUMN last_ingest_commit VARCHAR(40) NULL,
  ADD COLUMN last_ingest_at    TIMESTAMP NULL,
  ADD COLUMN last_ingest_error TEXT NULL;

-- source_repository and source_path become deprecated / removed
-- source_uri holds the raw toc.yaml terraform_module value

Updated SchemaHero YAML:

columns:
  - name: id
    type: char (36)
    constraints:
      notNull: true
  - name: app_id
    type: varchar (255)
    constraints:
      notNull: true
  - name: name
    type: varchar (255)
    constraints:
      notNull: true
  - name: provider
    type: varchar (100)
    constraints:
      notNull: true
  - name: source_uri
    type: varchar (2048)
    constraints:
      notNull: true
  - name: description
    type: text
    constraints:
      notNull: false
  - name: last_ingest_commit
    type: varchar (40)
    constraints:
      notNull: false
  - name: last_ingest_at
    type: timestamp
    constraints:
      notNull: false
  - name: last_ingest_error
    type: text
    constraints:
      notNull: false
  - name: created_at
    type: timestamp
    default: CURRENT_TIMESTAMP
    constraints:
      notNull: true
  - name: updated_at
    type: timestamp
    default: CURRENT_TIMESTAMP
    constraints:
      notNull: true

ep_terraform_module_version (no changes)

Already has source_ref, source_commit, variables_json, outputs_json, readme_html — all good.

ep_content_repo (no changes)

Sync errors for the content repo itself stay in last_sync_error.

---

Error Handling

Every error is stored in the database and surfaced in the ContentTab UI.

Error storage

Scope	Column	Examples
Content repo sync	ep_content_repo.last_sync_error	"git clone failed: authentication error", "invalid toc.yaml"
TF module ingest	ep_terraform_module.last_ingest_error	"ref v1.0.0 not found", "GitHub App does not have access to repo other-org/private-tf", "no .tf files found at terraform/aws", "failed to parse variables.tf"

Error behavior

• A failing TF module does not block the rest of the content sync. Pages and other modules still sync successfully.
• Each module's error is independent — one bad URI doesn't poison the others.
• Errors are cleared on successful ingest (last_ingest_error = NULL).
• The SyncResult is extended to include TF module results:

type SyncResult struct {
    Commit         string              `json:"commit"`
    PageCount      int                 `json:"page_count"`
    TocSynced      bool                `json:"toc_synced"`
    Error          string              `json:"error,omitempty"`
    TerraformModules []ModuleSyncResult `json:"terraform_modules,omitempty"`
}

type ModuleSyncResult struct {
    Name          string `json:"name"`
    SourceURI     string `json:"source_uri"`
    Status        string `json:"status"`         // "synced", "unchanged", "error"
    Commit        string `json:"commit,omitempty"`
    Error         string `json:"error,omitempty"`
}

---

ContentTab UI

The ContentTab becomes the single pane of glass for all synced content — pages, toc, TF modules, generated docs.

Connected state (repo linked)

┌─────────────────────────────────────────────────────────────────┐
│ Content Repository                                        Docs ↗│
│ Enterprise Portal content is synced from this repository.       │
│                                                                 │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ ◼ replicatedhq/chartsmith-enterprise-portal  [main]         │ │
│ │   ● Synced · 2 min ago · abc1234          [Sync Now] [Unlink]│ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                                                 │
│ Synced Content                                                  │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │                                                             │ │
│ │ Documentation Pages                                         │ │
│ │   21 pages synced from repo                        ● Synced │ │
│ │                                                             │ │
│ │ Table of Contents                                           │ │
│ │   toc.yaml parsed successfully                     ● Synced │ │
│ │                                                             │ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                                                 │
│ Terraform Modules                                               │
│ Modules referenced in toc.yaml are fetched and served through   │
│ the Replicated Terraform Registry.                              │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ aws                                                         │ │
│ │ github.com/replicatedhq/chartsmith-terraform//modules/aws        │ │
│ │ ref: v1.0.0 → a3f2c1e · Ingested 2 min ago        ● Synced │ │
│ │                                                             │ │
│ │ gcp                                                         │ │
│ │ github.com/replicatedhq/chartsmith-terraform//modules/gcp        │ │
│ │ ref: v1.0.0 → b7d4e2f · Ingested 2 min ago        ● Synced │ │
│ │                                                             │ │
│ │ azure                                                       │ │
│ │ github.com/other-org/azure-infra//modules/aks               │ │
│ │ ref: main → ——                                     ● Error  │ │
│ │ ⚠ GitHub App does not have access to other-org/azure-infra  │ │
│ │                                                             │ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                                                 │
│ Auto-Generated Reference                                        │
│ Generated from Helm charts at release promotion.                │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ chartsmith (Helm Values)                                    │ │
│ │ Generated from release seq 142 · 3 days ago       ● Complete│ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

API for ContentTab data

New vendor-api endpoint that aggregates everything:

GET /v3/app/{appId}/enterprise-portal/content-status

Response:
{
  "content_repo": {
    "id": "...",
    "repository_url": "...",
    "branch": "main",
    "github_repo_full_name": "replicatedhq/chartsmith-enterprise-portal",
    "last_sync_at": "2026-02-20T15:00:00Z",
    "last_sync_commit": "abc1234...",
    "sync_status": "success",
    "last_sync_error": null,
    "page_count": 21,
    "toc_synced": true
  },
  "terraform_modules": [
    {
      "id": "...",
      "name": "aws",
      "source_uri": "github.com/replicatedhq/chartsmith-terraform//modules/aws?ref=v1.0.0",
      "provider": "aws",
      "last_ingest_commit": "a3f2c1e...",
      "last_ingest_at": "2026-02-20T15:00:00Z",
      "last_ingest_error": null,
      "versions_count": 1
    },
    {
      "id": "...",
      "name": "azure",
      "source_uri": "github.com/other-org/azure-infra//modules/aks?ref=main",
      "provider": "azure",
      "last_ingest_commit": null,
      "last_ingest_at": null,
      "last_ingest_error": "GitHub App does not have access to other-org/azure-infra",
      "versions_count": 0
    }
  ],
  "generated_pages": [
    {
      "type": "chart-values",
      "chart_name": "chartsmith",
      "release_sequence": 142,
      "status": "complete",
      "generated_at": "2026-02-17T10:30:00Z"
    }
  ]
}

---

Implementation Order

1. URI parser — ParseTerraformModuleURI() in pkg/enterprise-portal/terraform_module_uri.go
2. DB schema — Update ep_terraform_module SchemaHero YAML (replace source_repository+source_path with source_uri, add ingest tracking columns)
3. Ingest logic — syncTerraformModules() in content_sync.go, called after syncPages()
4. Content status API — GET /v3/app/{appId}/enterprise-portal/content-status
5. ContentTab UI — Expand to show TF modules, errors, generated pages
6. Update example repo — Delete replicated/*.yaml files, update toc.yaml with URIs

---

What's Eliminated

• replicated/*.yaml files in vendor content repos
• apiVersion: replicated.com/v1beta1 kind: Terraform custom resource format
• Cross-referencing between toc.yaml keys and metadata file names
• source_repository and source_path columns (replaced by source_uri)

What's Added

• Terraform-native URI syntax in toc.yaml
• Per-module commit SHA tracking for change detection
• Per-module error storage and UI surfacing
• Aggregated content status API endpoint
• ContentTab shows full picture: pages, toc, TF modules, generated docs, errors