Environment & Secrets Workflow Template

env-secret-spec.md

Environment & Secrets Workflow Template

This document is a generic specification for managing configuration, secrets, and workflows in Python-based projects.

It is designed to work with:

• Pure Python apps (CLI tools, workers, etc.)
• Web services (FastAPI, Django, Flask, etc.)
• Any stack using Docker, CI/CD (e.g. GitLab), and optionally Kubernetes

The document is intended both for humans (to discuss and agree on a workflow) and for AI coding agents (to implement it consistently across projects).

---

1. Goals

1. Single source of truth per environment
   Each environment (local, staging, production, etc.) is described exactly once via an encrypted .env.<name> file.

2. Same configuration model everywhere
   The same env files are used by:

   • Local development (host virtualenv)
   • Local development (Docker / docker-compose)
   • CI jobs
   • Deployment pipelines (e.g. Kubernetes)

3. dotenvx only at the edges

   • dotenvx is used to hydrate environment variables for commands.
   • Runtime containers and deployed apps do not depend on dotenvx or plaintext .env files.

4. No secret duplication
   Secrets are defined once in .env.<name> and then propagated through tools (Docker, CI, Kubernetes) as environment variables or secrets—not copy-pasted into multiple configs.

5. Support for multiple workflows

   • Local host workflow (virtualenv, uv or pip)
   • Container-based workflow (Docker / docker-compose)
   • Both use the same .env sources.

---

2. Environments and .env Files

2.1 Logical environments

Pick a small, explicit set of environment names for the project, e.g.:

• local – developer machine (pure host, no Docker required).
• dev – shared development environment (optional).
• staging – pre-production / UAT.
• production – live environment.

You can add more (e.g. preview, performance) as needed. Every environment name must map to a corresponding .env.<name> file.

2.2 .env file inventory

For each environment <name>, define:

• .env.<name> – encrypted, committed to git, managed by dotenvx.

In addition, define:

• .env – local overrides, not tracked in git.
• .env.keys – dotenvx key material, not tracked in git.
• .env.example – non-secret example values for documentation / onboarding.

Summary table:

File	Tracked	Encrypted	Purpose
.env	no	no	Developer-specific overrides (hostnames, local tweaks).
.env.local	yes	yes	Defaults for local dev / unit tests (fake or low-value secrets).
.env.dev	yes	yes	Shared development env (optional).
.env.staging	yes	yes	Staging environment configuration + secrets.
.env.production	yes	yes	Production configuration + secrets.
.env.keys	no	n/a	dotenvx key(s), used to decrypt encrypted env files.
.env.example	yes	no	Example values: for humans, not used by tooling.

Note: If you use a different naming convention (e.g. .env.sandbox instead of .env.production), adapt the names but keep the pattern: one encrypted file per environment.

2.3 Layering rules

When running commands, env files are applied in a deterministic order:

• Local development (host or Docker):

  1. .env (local overrides)
  2. .env.local (baseline dev config)

• Shared dev environment:

  1. .env (local overrides, if applicable)
  2. .env.dev

• Staging:
  .env.staging (no .env)

• Production:
  .env.production (no .env)

Command pattern (local dev):

dotenvx run -f .env -f .env.local -- <command>

Command pattern (staging, CI):

dotenvx run -f .env.staging -- <command>

Only these .env.* files are considered the source of truth for configuration and secrets.

---

3. Core Environment Variables & Naming Conventions

3.1 Deployment environment flag

Use a single top-level environment variable to describe the deployment environment:

• Name: APP_ENV (or APP_DEPLOYMENT_ENV if the project already uses that).
• Example values: local, dev, staging, production.

This variable is used by:

• Application configuration (e.g. to choose logging levels, toggle debug features).
• Docker builds (e.g. whether to install dev toolchains).
• CI/CD pipelines (e.g. to pick which env file to use).

3.2 Naming patterns

Follow these naming patterns for environment variables:

• Uppercase, snake_case names.
• Prefix by concern / subsystem, e.g.:
  • APP_ – app-level settings (APP_ENV, APP_DEBUG, APP_LOG_LEVEL).
  • DB_ – database connection (DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD).
  • CACHE_ – cache stores (CACHE_URL, CACHE_TIMEOUT).
  • BROKER_ – message broker (BROKER_URL).
  • AUTH_ – authentication-related (AUTH_JWT_SECRET, AUTH_JWT_ALGORITHM).
  • THIRD_PARTY_X_ – anything talking to external services.

These variables should be documented in .env.example with non-sensitive placeholder values.

Example:

APP_ENV=local
APP_DEBUG=true
APP_LOG_LEVEL=DEBUG

DB_HOST=localhost
DB_PORT=5432
DB_NAME=app
DB_USER=app
DB_PASSWORD=changeme

CACHE_URL=redis://localhost:6379/0

AUTH_JWT_SECRET=changeme
AUTH_JWT_ALGORITHM=HS256

.env.example is for humans and onboarding; it is not used at runtime.

---

4. Python Configuration Pattern (Generic)

Any Python project (FastAPI, Django, pure Python) can adopt a common pattern for configuration.

Example with Pydantic Settings (optional but recommended):

from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    app_env: str = 'local'
    app_debug: bool = False
    app_log_level: str = 'INFO'

    db_host: str
    db_port: int
    db_name: str
    db_user: str
    db_password: str

    cache_url: str | None = None

    auth_jwt_secret: str
    auth_jwt_algorithm: str = 'HS256'

    class Config:
        env_prefix = ''
        case_sensitive = False

Usage in application:

settings = Settings()

For a Django project, this pattern can be used inside settings.py (or settings/ package) to populate Django settings from environment variables.

For CLI or worker processes, reuse the same Settings class to ensure consistency.

---

5. Dotenvx Usage & Makefile Conventions

To avoid repeating long dotenvx run commands everywhere, define reusable macros in Makefile.

5.1 Makefile variables

At the top of Makefile:

ENV_FILES_LOCAL := .env .env.local
ENV_FILES_DEV := .env.dev
ENV_FILES_STAGING := .env.staging
ENV_FILES_PRODUCTION := .env.production

DOTENVX_LOCAL := dotenvx run $(addprefix -f ,$(ENV_FILES_LOCAL)) --
DOTENVX_DEV := dotenvx run $(addprefix -f ,$(ENV_FILES_DEV)) --
DOTENVX_STAGING := dotenvx run $(addprefix -f ,$(ENV_FILES_STAGING)) --
DOTENVX_PRODUCTION := dotenvx run $(addprefix -f ,$(ENV_FILES_PRODUCTION)) --

You can adjust which environments you actually use (e.g. drop dev if not relevant).

5.2 Typical Make targets (host workflows)

Examples for a Python app using uv and pytest (adapt as needed):

.PHONY: env-local
env-local:
	$(DOTENVX_LOCAL) uv sync --dev

.PHONY: run-local
run-local:
	$(DOTENVX_LOCAL) uv run python -m app

.PHONY: test-unit
test-unit:
	$(DOTENVX_LOCAL) uv run pytest -m 'not integration'

.PHONY: test-integration-staging
test-integration-staging:
	$(DOTENVX_STAGING) uv run pytest -m 'integration'

If the project uses pip instead of uv, replace the commands but keep the dotenvx pattern.

---

6. Docker & Images (Optional but Recommended)

For projects using Docker, adopt a single multi-stage Dockerfile with environment-controlled behavior.

6.1 General pattern

• base stage – installs Python, package manager (e.g. uv), and common OS deps.
• builder stage – installs dependencies, builds wheels, runs migrations or compile steps as needed.
• runtime stage – minimal image for running the application.

Use a build argument to control the environment behavior:

ARG APP_ENV=production
ENV APP_ENV=${APP_ENV}

In the runtime stage, you may optionally install extra tooling only for non-production builds:

RUN set -e;     if [ "$APP_ENV" != "production" ]; then         apt-get update &&         apt-get install -y --no-install-recommends build-essential python3-dev;         apt-get clean && rm -rf /var/lib/apt/lists/*;     fi

Key rules:

• The final runtime image must not contain:
  • dotenvx binaries
  • .env.* files (encrypted or plaintext)
  • secret key files for dotenvx
• The application reads configuration only from environment variables provided by Docker / Kubernetes.

6.2 Private package indexes

If the project uses private Python indexes (via pip or uv) or other package sources:

• Define credential env vars in .env.<name>, e.g.:
  • PIP_INDEX_URL, PIP_EXTRA_INDEX_URL, or
  • UV_INDEX_<INDEX_NAME>_USERNAME, UV_INDEX_<INDEX_NAME>_PASSWORD
• Pass them through as build args from the current environment.

Example build command pattern:

dotenvx run -f .env.production --   docker build     --build-arg UV_INDEX_INTERNAL_USERNAME     --build-arg UV_INDEX_INTERNAL_PASSWORD     -t app-image:production .

The Dockerfile declares corresponding ARG entries in the builder stage and uses them during dependency installation. The values do not persist as environment variables in the runtime image.

---

7. docker-compose (Local Development with Containers)

For projects using docker-compose locally, the compose file should:

• Define variable names, not secret values.
• Use ${VAR_NAME} interpolation to pull values from the environment.
• Avoid hardcoding real credentials; keep those in .env.*.

7.1 Example pattern (simplified)

services:
  app:
    build:
      context: .
      args:
        APP_ENV: local
    environment:
      APP_ENV: local

      DB_HOST: '${DB_HOST}'
      DB_PORT: '${DB_PORT}'
      DB_NAME: '${DB_NAME}'
      DB_USER: '${DB_USER}'
      DB_PASSWORD: '${DB_PASSWORD}'

      CACHE_URL: '${CACHE_URL}'
      AUTH_JWT_SECRET: '${AUTH_JWT_SECRET}'
      AUTH_JWT_ALGORITHM: '${AUTH_JWT_ALGORITHM}'

    volumes:
      - ./:/app
    ports:
      - '8000:8000'

When starting the stack for local dev, you always go through dotenvx:

dotenvx run -f .env -f .env.local -- docker compose up app

Compose reads environment variables from its own process environment (hydrated by dotenvx), interpolates ${VAR}, and injects them into containers.

---

8. Testing Strategy

Testing is split by dependency level, each mapped to specific environment files.

8.1 Unit tests

• Do not rely on external network services (DB, cache, message broker, etc.).
• Use .env.local for configuration (plus .env for local overrides if needed).
• Run frequently (pre-commit, CI on every push).

Example command (host):

dotenvx run -f .env -f .env.local -- uv run pytest -m 'not integration'

8.2 Integration tests

• Talk to real services (a real DB, real cache, real external APIs, etc.).
• Use environment-specific .env.<name> files (e.g. .env.staging).
• Run less frequently (on certain branches, nightly, or manually triggered).

Example command (host or CI):

dotenvx run -f .env.staging -- uv run pytest -m integration

The specifics of how tests are marked (@pytest.mark.integration, etc.) can be project-specific, but the environment mapping should always follow this pattern.

8.3 End-to-end tests (optional)

For full stack / browser tests, follow the same idea:

• .env.staging or .env.production for the deployed app.
• Additional env vars for test tooling as needed.

---

9. CI/CD and Secret Propagation

This section describes how to use dotenvx in continuous integration and how to propagate secrets into deployed environments.

9.1 CI environment

In your CI system (e.g. GitLab CI):

• Configure dotenvx key material as secure CI variables (e.g. DOTENVX_KEY or equivalent).
• The CI runner must be able to install and run dotenvx according to the project’s standard.

Each job that needs application configuration or secrets runs commands via dotenvx, using the appropriate .env.<name> file.

Examples:

• Unit tests job (local dev config):

  dotenvx run -f .env.local -- uv sync --dev
  dotenvx run -f .env.local -- uv run pytest -m 'not integration'

• Integration tests job (staging config):

  dotenvx run -f .env.staging -- uv sync
  dotenvx run -f .env.staging -- uv run pytest -m integration

• Build image job (production config for private package auth):

  dotenvx run -f .env.production --     docker build       --build-arg UV_INDEX_INTERNAL_USERNAME       --build-arg UV_INDEX_INTERNAL_PASSWORD       -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .

9.2 Kubernetes / Deployment targets

Deployment targets (like Kubernetes) do not receive encrypted .env files. Instead:

1. CI runs a deploy job with dotenvx for the target environment (e.g. .env.staging).
2. That job reads env vars from its own environment (already hydrated by dotenvx).
3. It maps those vars into:
   • Kubernetes Secrets (for sensitive values).
   • Kubernetes ConfigMaps or inline env vars (for non-sensitive config).

A simple pattern is to maintain an allowlist of environment variable names intended for the application, e.g. deploy/env.allowlist:

APP_ENV
APP_DEBUG
APP_LOG_LEVEL

DB_HOST
DB_PORT
DB_NAME
DB_USER
DB_PASSWORD

CACHE_URL

AUTH_JWT_SECRET
AUTH_JWT_ALGORITHM

A small script (bash or Python) can:

• Read deploy/env.allowlist.
• For each entry, read the variable from the current environment.
• Generate a Kubernetes Secret manifest or run kubectl create secret generic with --from-literal pairs.

The application Deployment then references that secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/app:{{ .Values.imageTag }}
          envFrom:
            - secretRef:
                name: app-env

The exact tooling (raw kubectl, Helm, Kustomize, etc.) can vary, but the rule remains: dotenvx only runs in CI, not in the cluster.

---

10. Bringing This Template into a Project

To apply this template to a new or existing project:

1. Choose environment names and create corresponding .env.<name> files.
2. Encrypt them with dotenvx and commit the encrypted versions.
3. Create .env.example listing all expected vars with safe placeholder values.
4. Introduce Makefile macros for dotenvx (DOTENVX_LOCAL, DOTENVX_STAGING, etc.).
5. Update local workflows (run server, tests, scripts) to go through dotenvx.
6. Update Dockerfile (if used) to:
   • Use APP_ENV as a build arg.
   • Avoid embedding dotenvx or .env.* in the runtime image.
7. Update docker-compose.yml (if used) to only reference ${VAR_NAME} and never hardcode secrets.
8. Update CI pipeline to:
   • Install dotenvx.
   • Run relevant jobs with dotenvx run -f .env.<name> -- ....
   • Build images using env-driven build args for private indexes.
9. Add deployment scripts/manifests that create/update Kubernetes Secrets or equivalent from CI env vars, using an allowlist.
10. Document the chosen environment names and patterns for the team so everyone understands the configuration model.

Once this template is instantiated, new services (web APIs, background workers, etc.) can be added without redesigning secrets and environment workflows—they simply subscribe to the same conventions.

Managing .env files and secrets across local / dev / staging / prod is where most teams quietly give up and turn their project into a config graveyard.

• Secrets duplicated in CI, Docker, and K8s

• .env files drifting out of sync

• “Just run this 200-char dotenvx command bro”

• Nobody remembers how to onboard a new dev without a 30-minute voice call

So I wrote a universal, ready-to-use spec for env & secrets management: one model that works for any Python project (FastAPI, Django, CLI), with:

• dotenvx

• uv

• docker compose

• make

• optional Kubernetes / CI integration

Curious: what is your way of managing .env files and secrets across environments?