Vault Secrets Not Passed to Workload Pods - Bug Fix

vault-secrets-bug-fix.md

Vault Secrets Not Passed to Workload Pods - Bug Fix

Problem

When using Vault Agent injection with ToolHive's Kubernetes operator, secrets are injected into the proxy runner pod at /vault/secrets, but they are not being passed to the actual workload StatefulSet pod.

Symptoms

• Vault agent successfully injects secrets into the proxy pod (visible in pod description)
• Secrets are mounted at /vault/secrets in the proxy pod
• Environment variables are missing in the workload StatefulSet pod
• Workload pod crashes with missing environment variables

Example Configuration

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: unsplash
spec:
  image: 937028213865.dkr.ecr.us-east-1.amazonaws.com/mcp-unsplash:test-6
  resourceOverrides:
    proxyDeployment:
      podTemplateMetadataOverrides:
        annotations:
          vault.hashicorp.com/agent-inject: "true"
          vault.hashicorp.com/role: "toolhive-mcp-workloads"
          vault.hashicorp.com/agent-inject-secret-unsplash-config: "toolhive/mcp-unsplash"
          vault.hashicorp.com/agent-inject-template-unsplash-config: |
            {{- with secret "toolhive/mcp-unsplash" -}}
            UNSPLASH_ACCESS_KEY="{{ .Data.data.UNSPLASH_ACCESS_KEY }}"
            {{- end -}}

Root Cause

The issue is in cmd/thv-proxyrunner/app/execution.go in the runWithFileBasedConfig function. This function loads the RunConfig from /etc/runconfig/runconfig.json (which includes EnvFileDir set to /vault/secrets by the operator), but it never processes the EnvFileDir field to actually read the environment files.

Code Flow

1. Operator detects Vault annotations (cmd/thv-operator/controllers/mcpserver_runconfig.go:216-218)

   • Sets EnvFileDir: "/vault/secrets" in the RunConfig
   • Serializes RunConfig to ConfigMap

2. Proxy runner loads config (cmd/thv-proxyrunner/app/execution.go:99-140)

   • Loads RunConfig from /etc/runconfig/runconfig.json
   • EnvFileDir field is present but never processed
   • Calls workloadManager.RunWorkload(ctx, config) directly

3. Workload manager (pkg/workloads/manager.go)

   • Doesn't process EnvFileDir either
   • Just passes config to runner

Solution

Add code to process EnvFileDir in runWithFileBasedConfig before deploying the workload:

// Process env file directory if specified (e.g., for Vault secrets)
if config.EnvFileDir != "" {
	var err error
	config, err = config.WithEnvFilesFromDirectory(config.EnvFileDir)
	if err != nil {
		return fmt.Errorf("failed to load environment files from directory %s: %w", config.EnvFileDir, err)
	}
}

Implementation

Add this code block in cmd/thv-proxyrunner/app/execution.go in the runWithFileBasedConfig function, after applying the k8s-pod-patch and before environment variable validation:

// runWithFileBasedConfig handles execution when a runconfig.json file is found.
// Uses config from file exactly as-is, ignoring all CLI configuration flags.
// Only uses essential non-configuration inputs: image, command args, and --k8s-pod-patch.
func runWithFileBasedConfig(
	ctx context.Context,
	cmd *cobra.Command,
	mcpServerImage string,
	cmdArgs []string,
	config *runner.RunConfig,
	rt runtime.Runtime,
	debugMode bool,
	envVarValidator runner.EnvVarValidator,
	imageMetadata *registry.ImageMetadata,
) error {
	// Use the file config directly with minimal essential overrides
	config.Image = mcpServerImage
	config.CmdArgs = cmdArgs
	config.Deployer = rt
	config.Debug = debugMode

	// Apply --k8s-pod-patch flag if provided (essential for K8s operation)
	if cmd.Flags().Changed("k8s-pod-patch") && runFlags.runK8sPodPatch != "" {
		config.K8sPodTemplatePatch = runFlags.runK8sPodPatch
	}

	// Process env file directory if specified (e.g., for Vault secrets)
	if config.EnvFileDir != "" {
		var err error
		config, err = config.WithEnvFilesFromDirectory(config.EnvFileDir)
		if err != nil {
			return fmt.Errorf("failed to load environment files from directory %s: %w", config.EnvFileDir, err)
		}
	}

	// Validate environment variables using the provided validator
	if envVarValidator != nil {
		validatedEnvVars, err := envVarValidator.Validate(ctx, imageMetadata, config, config.EnvVars)
		if err != nil {
			return fmt.Errorf("failed to validate environment variables: %v", err)
		}
		config.EnvVars = validatedEnvVars
	}

	// Apply image metadata overrides if needed (similar to what the builder does)
	if imageMetadata != nil && config.Name == "" {
		config.Name = imageMetadata.Name
	}

	workloadManager, err := workloads.NewManagerFromRuntime(rt)
	if err != nil {
		return fmt.Errorf("failed to create workload manager: %v", err)
	}
	return workloadManager.RunWorkload(ctx, config)
}

How It Works

1. Operator detects Vault → Sets EnvFileDir: "/vault/secrets" in RunConfig
2. Proxy runner loads config → Reads RunConfig from ConfigMap
3. NEW: Process EnvFileDir → Reads all files in /vault/secrets and merges them into config.EnvVars
4. Deploy workload → Environment variables are passed to the StatefulSet pod

The WithEnvFilesFromDirectory method reads all files in the directory, parses them as KEY=VALUE format, and merges them into the EnvVars map, which then gets passed to the workload pod.

Testing

After applying this fix:

1. Deploy an MCPServer with Vault annotations
2. Verify vault secrets are injected into proxy pod at /vault/secrets
3. Check that environment variables are present in the workload StatefulSet pod
4. Verify the workload pod starts successfully

Related Files

• cmd/thv-proxyrunner/app/execution.go - Main fix location
• cmd/thv-operator/controllers/mcpserver_runconfig.go - Operator sets EnvFileDir
• pkg/runner/config.go - RunConfig structure with EnvFileDir field
• pkg/runner/env_files.go - Environment file processing logic
• examples/operator/vault/mcpserver-github-with-vault.yaml - Example configuration

Status

Bug: File-based config loading doesn't process EnvFileDir Impact: Vault secrets injected into proxy pod but not passed to workload pod Fix: Add EnvFileDir processing in runWithFileBasedConfig function