Claude Code/Sonnet 4's plan for a feature set to enable Hermeto to use a content proxy

content_proxy_plan_v3.md

Generic Artifact Proxy Integration Plan for Hermeto

Overview

Design and implement a flexible feature set that allows all Hermeto package managers to retrieve packages from various artifact proxy/repository systems.

This will enable hermetic builds in air-gapped environments using configurable content proxies like Pulp, Nexus, Artifactory, or custom solutions.

Core Components

1. Configuration System Extension

Extend hermeto/core/config.py:

• Add ProxyConfig model for generic proxy configuration
• Add ProxyProvider enum/registry for different proxy types
• Support multiple proxy instances with content type routing
• Include flexible authentication settings (username/password, client certs, tokens)

New config structure:

proxy:
  enabled: true
  default_instance: "main"
  instances:
    main:
      provider: "pulp"  # pulp, nexus, artifactory, generic
      base_url: "https://proxy.example.com"
      auth:
        type: "basic"
        username: "user"
        password: "pass"
    secondary:
      provider: "nexus"
      base_url: "https://nexus.example.com"
      auth:
        type: "client_cert"
        client_cert: "/path/to/cert.pem"
        client_key: "/path/to/key.pem"
    rpm_instance:
      provider: "artifactory"
      base_url: "https://artifactory.example.com"
      auth:
        type: "token"
        token: "art-token-123"
  content_mappings:
    npm: {instance: "main", repository: "npm-content"}
    pip: {instance: "main", repository: "python-content"}
    bundler: {instance: "main", repository: "gem-content"}
    rpm: {instance: "rpm_instance", repository: "rhel-content"}
    cargo: {instance: "main", repository: "cargo-content"}
    yarn: {instance: "main", repository: "npm-content"}
    gomod: {instance: "main", repository: "go-content"}

2. Abstract Proxy Client Framework

Create hermeto/core/proxy/client.py:

• AbstractProxyClient base class defining proxy interface
• ProxyClientFactory for creating provider-specific clients
• Common authentication and SSL handling
• Standardized error handling and fallback mechanisms

Create hermeto/core/proxy/providers/:

• base.py - Abstract base classes and interfaces
• pulp.py - Pulp-specific client implementation
• nexus.py - Nexus-specific client implementation
• artifactory.py - Artifactory-specific client implementation
• generic.py - Generic HTTP proxy client for simple URL rewriting

3. URL Transformation Framework

Create hermeto/core/proxy/url_transformer.py:

• AbstractUrlTransformer base class for URL transformation
• UrlTransformerRegistry for registering package-manager-specific transformers
• Provider-agnostic URL transformation pipeline
• Support for complex URL patterns and metadata preservation
• Handle special cases (git dependencies, file dependencies, etc.)

Package manager specific transformers:

• NpmUrlTransformer - handles npm registry URL patterns
• PipUrlTransformer - handles PyPI URL patterns
• BundlerUrlTransformer - handles RubyGems URL patterns
• CargoUrlTransformer - handles crates.io URL patterns
• RpmUrlTransformer - handles RPM repository URL patterns
• GomodUrlTransformer - handles Go module proxy URL patterns

URL transformation patterns:

• https://registry.npmjs.org/package/-/package-1.0.0.tgz → https://proxy.example.com/pulp/content/npm-content/package/-/package-1.0.0.tgz
• https://files.pythonhosted.org/packages/.../package-1.0.0.tar.gz → https://proxy.example.com/repository/pypi-proxy/.../package-1.0.0.tar.gz

4. Provider Abstraction Layer

Create hermeto/core/proxy/providers/base.py:

class AbstractProxyProvider:
    def transform_url(self, original_url: str, content_type: str, repository: str) -> str
    def get_download_url(self, package_info: dict) -> str
    def authenticate(self, request: HttpRequest) -> HttpRequest
    def validate_connection(self) -> bool

Specific implementations:

• Pulp Provider: Uses Pulp's distribution/content API patterns
• Nexus Provider: Uses Nexus repository API patterns
• Artifactory Provider: Uses Artifactory repository API patterns
• Generic Provider: Simple URL prefix replacement

5. Content Type Handlers

Create hermeto/core/proxy/content_types.py:

• Abstract base class for content type handlers
• Specific implementations for each package manager:
  • NpmContentHandler - maps npm registry URLs to proxy repositories
  • PipContentHandler - maps PyPI URLs to proxy repositories
  • BundlerContentHandler - maps RubyGems URLs to proxy repositories
  • CargoContentHandler - maps crates.io URLs to proxy repositories
  • RpmContentHandler - maps RPM repository URLs to proxy repositories
  • GomodContentHandler - maps Go module proxy URLs to proxy repositories

6. Package Manager Integration

Modify each package manager with provider-agnostic integration:

NPM (hermeto/core/package_managers/npm.py):

• Modify _get_npm_dependencies() to use proxy URLs when enabled
• Update registry detection logic to handle proxy URLs
• Use ProxyClient.transform_url() for registry URL transformation
• Preserve integrity checks and validation

Pip (hermeto/core/package_managers/pip/main.py):

• Modify process_package_distributions() to query proxy for packages
• Update index URL handling to use proxy repositories
• Transform index URLs through provider-specific logic
• Maintain requirement file parsing and hash validation

Bundler (hermeto/core/package_managers/bundler/main.py):

• Modify gem resolution to use proxy as source
• Update download logic in parser to use proxy URLs
• Preserve Gemfile.lock integrity

Cargo (hermeto/core/package_managers/cargo/main.py):

• Update crates.io registry URL handling to use proxy
• Maintain Cargo.lock integrity and validation

RPM (hermeto/core/package_managers/rpm/main.py):

• Update repository URL handling to use proxy repositories
• Preserve RPM metadata and signature validation

Yarn (hermeto/core/package_managers/yarn/main.py):

• Similar to NPM integration for registry URL handling
• Update yarn.lock processing to use proxy URLs

Gomod (hermeto/core/package_managers/gomod.py):

• Update GOPROXY environment variable to use proxy
• Handle Go module proxy protocol through proxy

7. Enhanced Download Layer

Modify hermeto/core/package_managers/general.py:

• Add proxy-aware download functions using provider abstraction
• Integrate authentication and SSL handling for proxy connections
• Implement provider-specific authentication injection
• Add configurable retry logic and error handling
• Maintain backward compatibility with direct downloads
• Maintain existing fallback mechanisms

8. CLI and Input Extensions

Extend CLI options:

• Add --proxy-config flag for custom proxy configuration
• Add --disable-proxy flag to bypass proxy routing
• Add --proxy-provider to override default provider
• Add proxy configuration validation

Input model extensions:

• Add optional proxy_override fields to package input models
• Allow per-package proxy instance/repository overrides

Implementation Phases

Phase 1: Core Infrastructure

1. Extend configuration system with proxy models
2. Implement abstract proxy client framework
3. Create provider registry and factory patterns
4. Create URL transformation pipeline framework
5. Add CLI configuration support

Phase 2: Core Providers

1. Implement Pulp provider
2. Implement Generic HTTP proxy provider
3. Add comprehensive testing framework
4. Integrate with NPM and Pip package managers

Phase 3: Extended Provider Support

1. Implement Nexus provider
2. Implement Artifactory provider
3. Add support for Bundler, Cargo, and RPM package managers
4. Comprehensive integration testing

Phase 4: Full Package Manager Support

1. Add Yarn and Gomod proxy integration
2. Add provider auto-detection capabilities
3. Comprehensive integration testing

Phase 5: Advanced Features

1. Add proxy content introspection and validation
2. Implement intelligent fallback strategies
3. Add performance optimizations and caching
4. Add detailed logging and monitoring

Provider-Specific Implementation Notes

Pulp Provider

• Use /pulp/content/{distribution}/ URL patterns
• Leverage Pulp's REST API for content discovery
• Handle Pulp-specific authentication (Basic, client certs, tokens)

Nexus Provider

• Use /repository/{repo-name}/ URL patterns
• Leverage Nexus REST API for package metadata
• Handle Nexus authentication (username/password, tokens)

Artifactory Provider

• Use /{repo-key}/ URL patterns
• Leverage Artifactory REST API
• Handle Artifactory authentication (API keys, tokens)

Generic Provider

• Simple URL prefix/pattern replacement
• Basic HTTP authentication support
• Minimal API requirements for maximum compatibility

Testing Strategy

1. Unit Tests: Test provider abstraction, URL transformation, configuration parsing
2. Integration Tests: Test with mock responses for each provider type
3. End-to-End Tests: Test with real instances of Pulp, Nexus, Artifactory
4. Compatibility Tests: Ensure existing workflows work unchanged

Backward Compatibility

• Proxy integration is completely opt-in via configuration
• Existing workflows continue unchanged when proxy is disabled
• No breaking changes to CLI or API interfaces
• Graceful degradation when proxy is unavailable

Security Considerations

• Provider-agnostic credential storage and transmission
• SSL/TLS certificate validation for all proxy connections
• Input validation for proxy URLs and configurations
• Audit logging for all proxy interactions
• Support for various authentication methods per provider
• Secure credential storage and transmission