This document explains the major steps, script calls, and data flows in the sync-midstream.sh script.
─────────────────────────────────────────────────────────────────────────
Fetches upstream repos → Processes workspaces → Exports plugins →
Updates metadata → Generates catalog-index → Commits changes
─────────────────────────────────────────────────────────────────────────
upstream repos ──┤
├─> Fetch & rsync ──> overlay-repo/
├─> Fetch & rsync ──> workspaces/
overlay-repo/workspaces/*/source.json ────┤
▼
Clone workspaces (sparse checkout)
▼
Apply overlays & patches (override-sources.sh)
▼
Export plugins (batchExportPlugins.sh)
▼
workspaces/*/plugins/*/dist-dynamic/
──────────────────────────────────────────┴────────────────────
Update plugin_builds/*/*.json with registryReferences
──────────────────────────────────────────┬────────────────────
default.packages.yaml (from rhdh repo) ───┤
▼
generate_dpdy.py
▼
catalog-index/dynamic-plugins.default.yaml
plugin_builds/*/*.json ───────────────────┤
overlay-repo/catalog-entities/ ───────────┤
overlay-repo/workspaces/*/metadata/ ──────┤
▼
generateCatalog
Index.py
▼
catalog-index/index.json
catalog-index/catalog-entities/marketplace/
▼
Git commit
Git push
Setup
* Parse CLI Arguments
• Get DWNSTM_BRANCH
Read upstream_repos.yml to determine:
• RHDH_BRANCH
• RHDH_VERSION
• BACKSTAGE_VERSION
FOR each repo in upstream_repos.yml (NUM_REPOS iterations)
1. Parse repo info:
• repo URL (github.com/redhat-developer/rhdh-*)
• branch (branch0 or branch1)
• destination_folder
2. Clone repo to TMPDIR/repo{i}/
git clone $repo -b $branch "$TMPDIR/repo${i}" --depth=3
3. Rsync content to midstream destination:
rsync -aPzq $TMPDIR/repo${i}/* ${ROOTPATH}/${destination_folder}/
Common destinations:
• overlay-repo/ (rhdh-dynamic-plugin-export-overlays)
• workspaces/ (backstage, community-plugins)
──────────────────────────────────────────────────────────────────────
Download scripts to build/scripts/ from rhdh-plugin-export-utils
• override-sources.sh
• export-dynamic.sh
• shorten-component-name.sh
──────────────────────────────────────────────────────────────────────
FOR each overlay-repo/*/source.json file
─────────────────────────────────────────────────────────────────
1. Parse source.json
• repo: https://github.com/backstage/community-plugins
• repo-ref: @backstage-community/plugin-quay-common@1.5.0
• folder_name: workspaces/ocm
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
2. Check if plugin is in SUPPORTED_PACKAGES (overlay-repo/rhdh-supported-packages.txt)
• If not found → Skip or Remove
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
3. Sparse clone plugin workspace into workspaces/{workspace_name}/
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
4. Remove plugins not in rhdh-supported-packages.txt from the plugins-list.yaml file; not all plugins in overlays repo will be productized
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
5. Generate manifest.json with generate-workspace-manifest.js
• Captures all package versions in workspace
• Resolves workspace:^ references
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
6. Apply overlays and patches (if APPLY_OVERLAY_PATCHES=1)
override-sources.sh
• Copies overlay-repo/{workspace}/overlay/
to workspaces/{workspace}/
• Applies patches from
overlay-repo/{workspace}/patches/
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
7. Export plugins (if not skipped) with batchExportPlugins.sh
• Finds all plugins in workspace
• For each plugin:
export-dynamic.sh (from rhdh-plugin-export-utils)
• yarn install --no-immutable
• yarn tsc / yarn tsc:full
• yarn build:all
• Create dist-dynamic/ artifacts
• Update package.json & yarn.lock
• For backend plugins: include embedded/ folder
Output: workspaces/{ws}/plugins/*/dist-dynamic/
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
8. Cleanup
• Remove examples/ folders
• Remove packages/backend, packages/app folders
• Remove node_modules/
─────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
Apply sync-midstream.sh patches (if APPLY_SYNC_PATCHES=1)
• Apply all build/ci/*.patch files to the repo
• These are workspace-specific fixes downstream of the overlays repo; ideally this would be a no-op step
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
FOR each workspace with workspace: or backstage:^ refs
─────────────────────────────────────────────────────────────────
1. Copy update-workspace.js to workspace
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
2. Transform workspace dependencies
update-workspace.js --update $BS_VERSION
• Replace workspace:^ references
• Replace backstage:^ references
• Pin to specific Backstage version
update-workspace.js --delete $UNNEEDED_PACKAGES
• Remove unneeded packages like e2e-test,app,app-next,backend
update-workspace.js --remove-patches
• Remove yarn dependency patch resolutions (backstage only)
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
5. Fix .yarnrc.yml and package.json
• Set yarnPath
• Remove packageManager field (to support hermetic builds)
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
6. Run yarn install / tsc / build (if YARN_INSTALL=1) to validate changes above
• install_tsc_build() function
─────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
FOR each plugin in SUPPORTED_PACKAGES
─────────────────────────────────────────────────────────────────
1. Read plugin package.json
• Extract package_name
• Extract plugin_version
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
2. Update plugin_builds/{workspace}/{repo_name}.json
• Set registryReference:
quay.io/rhdh/{repo_name}:{RHDH_VERSION}--{plugin_version
• Set workspacePath: {workspace}/plugins/{plugin}
Example:
{
"backstage-community-plugin-ocm-backend": {
"registryReference":
"quay.io/rhdh/backstage-community-plugin-ocm-
backend:1.9.0--1.5.0",
"workspacePath": "ocm/plugins/ocm-backend"
}
}
─────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
Update build/containerfiles/ with metadata
─────────────────────────────────────────────────────────────────
1. Calculate next release number using getNextReleaseNum.sh
• Query for existing builds
• Increment release number in Containerfile
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
2. Update builder.Containerfile and index.Containerfile
• Add ENV variables (eg., RHDH_VERSION, BACKSTAGE_VERSION)
• Add LABEL metadata
• Set version and release
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
Generate dynamic-plugins.default.yaml
─────────────────────────────────────────────────────────────────
1. Download default.packages.yaml from rhdh repo
curl -sSLO "https://raw.githubusercontent.com/
redhat-developer/rhdh/refs/heads/${RHDH_BRANCH}/
default.packages.yaml"
─────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
2. Run generate_dpdy.py
Input:
-p catalog-index/default.packages.yaml
-d overlay-repo/
Process:
• Read all packages from default.packages.yaml
• For each package:
- Find metadata in overlay-repo/workspaces/
*/metadata/{package}.yaml
- Extract spec.dynamicArtifact
- Extract spec.appConfigExamples[0].content
→ becomes pluginConfig
- Add integrity field if present
• Sort: enabled first, then alphabetically
Output:
-o catalog-index/dynamic-plugins.default.yaml
Format:
plugins:
- package: "@scope/pkg@version" or "./path"
disabled: true/false
integrity: sha512-... (optional)
pluginConfig:
<content from appConfigExamples>
─────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
Generate catalog index
─────────────────────────────────────────────────────────────────
generateCatalogIndex.py (with optional --debug flag)
1. Move plugin definitions and static metadata (catalog-entities/marketplace)
From:
overlay-repo/catalog-entities/marketplace/
To:
catalog-index/catalog-entities/marketplace/
2. Copy workspace metadata files
Find all:
overlay-repo/workspaces/*/metadata/*.yaml
Copy to:
catalog-index/catalog-entities/marketplace/
packages/
These are Backstage catalog entity definitions
for individual plugins (Component kind)
3. Generate index.json from plugin_builds/
Read all:
plugin_builds/*/*.json
For each plugin:
• Extract registryReference
• Check if image exists (HTTP HEAD request)
• If exists, add to index
Combine digest into registryReference:
quay.io/rhdh/pkg@sha256:abc123...
Write to:
catalog-index/index.json
Format:
{
"plugin-name": {
"workspacePath": "ws/plugins/plugin",
"registryReference": "quay.io/rhdh/pkg@sha...",
"imageTag": "1.9.0--1.5.0",
"build-date": "...",
"vcs-ref": "...",
"upstream": "...",
"midstream": "..."
}
}
4. Update package YAML files with OCI references
For each plugin in index.json:
• Find matching YAML in:
- catalog-index/dynamic-plugins.default.yaml
- catalog-index/catalog-entities/marketplace/
packages/{plugin-name}.yaml
• Find existing entry:
- package: ./dynamic-plugins/dist/{plugin}
• Add OCI reference BEFORE the file path entry:
# Tag: 1.9.0--1.5.0, Build date: 2025-01-09
- package: oci://quay.io/rhdh/pkg@sha...!plugin
# new approach using oci images: to switch...
disabled: true
- package: ./dynamic-plugins/dist/{plugin}
pluginConfig:
...
This allows toggling between deprecated file-based wrappers and new
OCI-based plugin loading
5. Regenerate catalog-index/catalog-entities/marketplace/*/all.yaml files
─────────────────────────────────────────────────────────────────
Cleanup after generation - remove unneeded files:
• overlay-repo/catalog-entities/
• overlay-repo/workspaces/*/metadata/
• catalog-index/catalog-entities/marketplace/README.md
─────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
Update metadata in plugin_builds/*.json with generatePluginBuildInfo.py
• Adds additional metadata to plugin_builds/*.json
• Updates timestamps, git refs, etc.
───────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────
Final cleanup
• Remove install-state.gz, node_modules, dist, dist-*
• Remove .yarn/cache/*
• Remove packages/app, packages/app-next, packages/backend
• Keep only embedded/, package.json, yarn.lock in dist-dynamic/
• Restore .pnp.cjs and .config-schema.json files (ignore changes)
• Check if index.Containerfile has only trivial changes
Add, commit, and push changes
• git add -f ...
• ${destination_folders}
• overlay-repo/
• workspaces/
• build/
• plugin_builds/
• upstream_repos.yml
• catalog-index/
───────────────────────────────────────────────────────────────────────
The catalog-index content is built from multiple sources:
─────────────────────────────────────────────────────────────────────
CATALOG-INDEX REFRESH FLOW
─────────────────────────────────────────────────────────────────────
INPUT SOURCES:
──────────────────────────────────────────────────────────────────
1. overlay-repo/catalog-entities/marketplace/
• Static catalog entities (plugins, packages)
• Moved to catalog-index/catalog-entities/marketplace/
2. overlay-repo/workspaces/*/metadata/*.yaml
• Individual plugin metadata (Component kind)
• Contains: spec.packageName, spec.dynamicArtifact,
spec.version, spec.appConfigExamples
• Copied to catalog-index/catalog-entities/marketplace/
packages/
3. plugin_builds/*/*.json
• Contains registryReference for each plugin
• Format: quay.io/rhdh/{pkg}:{RHDH_VER}--{plugin_ver}
• Used to generate catalog-index/index.json
──────────────────────────────────────────────────────────────────
PROCESSING:
──────────────────────────────────────────────────────────────────
generateCatalogIndex.py performs:
1. Moves marketplace/ folder
2. Copies workspace metadata/*.yaml files
3. Checks if OCI images exist (HTTP HEAD to quay.io)
4. Generates index.json with digest-pinned references
5. Updates YAML files with OCI references alongside file paths
6. Regenerates all.yaml Location files
──────────────────────────────────────────────────────────────────
OUTPUT:
──────────────────────────────────────────────────────────────────
catalog-index/
├── index.json
── Maps plugin names to OCI image refs with digests
├── dynamic-plugins.default.yaml
── Updated with OCI references (disabled by default)
── catalog-entities/
── marketplace/
├── plugins/
├── all.yaml (Location kind)
── {plugin}.yaml
── packages/
├── all.yaml (Location kind)
── {package}.yaml (updated with OCI refs)
──────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────────
DYNAMIC-PLUGINS.DEFAULT.YAML UPDATE FLOW
─────────────────────────────────────────────────────────────────────
STEP 1: Download source file
──────────────────────────────────────────────────────────────────
curl -sSLO "https://raw.githubusercontent.com/
redhat-developer/rhdh/refs/heads/${RHDH_BRANCH}/
default.packages.yaml"
→ catalog-index/default.packages.yaml
Contains:
packages:
enabled:
- package: "@scope/package"
integrity: sha512-...
disabled:
- package: "another-package"
──────────────────────────────────────────────────────────────────
STEP 2: Generate DPDY with Python script
──────────────────────────────────────────────────────────────────
generate_dpdy.py
-p catalog-index/default.packages.yaml
-d overlay-repo/
-o catalog-index/dynamic-plugins.default.yaml
Process:
1. Scan overlay-repo/workspaces/*/metadata/*.yaml
2. Index by spec.packageName and spec.dynamicArtifact
3. For each package in default.packages.yaml:
• Find matching metadata file
• Extract spec.dynamicArtifact (package path/name)
• Extract spec.appConfigExamples[0].content
• Build plugin entry:
- package: {dynamicArtifact}
disabled: {from default.packages.yaml}
integrity: {from default.packages.yaml, if external}
pluginConfig:
{content from appConfigExamples}
4. Sort: enabled plugins first, then alphabetically
──────────────────────────────────────────────────────────────────
STEP 3: Add OCI references (via generateCatalogIndex.py Task 4)
──────────────────────────────────────────────────────────────────
For each plugin with valid OCI image:
• Find entry in dynamic-plugins.default.yaml
• Add OCI reference BEFORE the existing file-based entry:
# Tag: 1.9.0--1.5.0, Build date: 2025-01-09
- package: oci://quay.io/rhdh/plugin@sha256:abc...!plugin
# new approach using oci images: to switch...
disabled: true
- package: ./dynamic-plugins/dist/plugin
disabled: false
pluginConfig:
...
──────────────────────────────────────────────────────────────────
OUTPUT:
──────────────────────────────────────────────────────────────────
catalog-index/dynamic-plugins.default.yaml
• Contains all enabled and disabled plugins
• Each plugin has:
- OCI reference (disabled by default, for future use)
- File path reference (currently active)
- pluginConfig with example configuration
• Ready to be included in RHDH image builds
──────────────────────────────────────────────────────────────────
override-sources.sh Apply overlays and patches to workspaces sync-midstream.sh (Phase 4) Modified workspace files batchExportPlugins.sh Batch export all plugins in a workspace sync-midstream.sh (Phase 4) dist-dynamic/ folders export-dynamic.sh Export a single plugin (called by batchExportPlugins.sh) batchExportPlugins.sh dist-dynamic/package.json, yarn.lock, embedded/ generate-workspace-manifest.js Capture package versions for workspace:^ resolution sync-midstream.sh (Phase 4) manifest.json update-workspace.js Transform workspace dependencies sync-midstream.sh (Phase 5) Modified package.json, yarn.lock getNextReleaseNum.sh Query Konflux for next release number sync-midstream.sh (Phase 7) Release number for Containerfiles generate_dpdy.py Generate dynamic-plugins.default.yaml sync-midstream.sh (Phase 8) catalog-index/dynamic-plugins.default.yaml generateCatalogIndex.py Generate catalog index from plugin builds sync-midstream.sh (Phase 9) catalog-index/index.json, updated YAML files generatePluginBuildInfo.py Update plugin build metadata sync-midstream.sh (Phase 10) Updated plugin_builds/*.json
upstream_repos.yml- Defines repos to fetchoverlay-repo/rhdh-supported-packages.txt- List of plugins to includeoverlay-repo/workspaces/*/source.json- Source repo referencesoverlay-repo/workspaces/*/metadata/*.yaml- Plugin catalog entitiesoverlay-repo/workspaces/*/plugins-list.yaml- Plugins to exportoverlay-repo/workspaces/*/overlay/- Files to overlay onto workspacesoverlay-repo/workspaces/*/patches/- Patches to applyplugin_builds/*/*.json- Plugin registry references
workspaces/*/plugins/*/dist-dynamic/- Exported plugin artifactscatalog-index/index.json- Master plugin index with OCI refscatalog-index/dynamic-plugins.default.yaml- Plugin configuration filecatalog-index/catalog-entities/marketplace/packages/*.yaml- Individual plugin entitiescatalog-index/catalog-entities/marketplace/plugins/all.yaml- Location aggregatorcatalog-index/catalog-entities/marketplace/packages/all.yaml- Location aggregatorbuild/containerfiles/builder.Containerfile- Updated with metadatabuild/containerfiles/index.Containerfile- Updated with metadata