Skip to content

Instantly share code, notes, and snippets.

@asears
Created March 6, 2026 12:41
Show Gist options
  • Select an option

  • Save asears/b2285d477610998f20e22d8112e51c63 to your computer and use it in GitHub Desktop.

Select an option

Save asears/b2285d477610998f20e22d8112e51c63 to your computer and use it in GitHub Desktop.
Testing Patterns for File Types

File Type Deployment, Validation, and Testing Guide

This guide maps file types found across the repos to practical deployment, schema checks, data checks, unit tests, and integration tests. It is written for Windows + PowerShell workflows.

Core Principles

  • Run fast static checks (lint/schema/type) on every PR.
  • Run data and integration checks on merge/main or release branches.
  • Keep unit tests isolated with mocks/stubs for network, DB, and cloud calls.
  • Use plan/preview first for infrastructure changes.
  • Fail fast in CI using non-zero exit codes.

File Type Matrix

File Type(s) How to Deploy Verify Schema Verify Data Unit Test Integration Test Key Tools
.yml, .yaml Apply via CI pipelines and environment-specific variable groups Lint structure and required keys Validate resolved values/paths/secrets references N/A (config-only) Execute pipeline template validation/dry-run in test project yamllint, yq, PowerShell key assertions
.json, .platform, .sample Package with app artifact or infra payloads Parse + JSON Schema validation Assert required fields are non-empty and valid N/A (config-only) Startup smoke test loads config without errors jq, ajv-cli, ConvertFrom-Json
.env Do not deploy raw files; map to secret stores and inject at runtime Validate required key set and formats Reject placeholders (CHANGEME, blank values) N/A Start app/container using injected secrets in test env dotenv-linter, PowerShell regex checks
.sql Versioned migrations via SQL deployment stage SQL lint + parse checks Seeded DB assertions (row counts, keys, null rules) Test SQL wrappers and query builders; use pgTAP for PostgreSQL assertions Run migration + rollback test against ephemeral DB sqlfluff, Invoke-Sqlcmd, tSQLt/pgTAP (where applicable)
.ps1 Deploy signed scripts/modules; run via controlled automation accounts Static script analysis and metadata checks Validate generated outputs and side effects in temp paths Pester tests with mocked cmdlets End-to-end runbook execution in test subscription/resource group PSScriptAnalyzer, Pester
.kql Deploy query/dashboard artifacts via workspace APIs Parse/compile query text Validate query outputs on seeded test datasets Test KQL construction helpers in host language Execute KQL against test Eventhouse/Kusto cluster Fabric/ADX APIs, PowerShell wrappers
.graphql Deploy schema/query artifacts with backend release GraphQL schema lint/introspection Validate result shape and required fields Test resolver/query helper functions Contract test against test GraphQL endpoint graphql-cli, spectaql/schema tooling
.py Build wheel/container; deploy via service release pipeline Type/lint/import checks Fixture-based output assertions and contract tests pytest with mocks for DB/HTTP/filesystem and architecture boundary tests pytest -m integration against test DB/services including Cosmos DB emulator/test account pytest, pytest-mock, unittest.mock, ruff, mypy, pytest-archon
.ts, .tsx, .js, .jsx Build static bundle or container and promote through envs Lint + type checks Validate API contract fixtures and component states Unit/component tests with mocked modules/network UI/API integration in test environment eslint, tsc --noEmit, vitest/jest, playwright/cypress
.java, .cs Build artifacts and deploy with service pipeline/container image Compile + static analysis Serialization and mapping checks on fixtures JUnit/xUnit tests with mock frameworks Service integration using ephemeral dependencies maven/gradle, dotnet test, Mockito, Moq, Testcontainers
.csv, .parquet, .geojson Publish to Lakehouse/warehouse versioned paths Enforce schema contract (columns/types/order) Data quality checks (nulls, uniqueness, range checks, geometry validity) Unit test transform functions with tiny fixtures Execute ingest pipeline on representative sample partition pandera, great_expectations, duckdb, pyarrow, ogrinfo
.ipynb Run parameterized notebooks in orchestrated jobs Validate notebook metadata/kernel/cell structure Assert output tables/metrics meet thresholds Unit-test extracted Python modules used by notebooks Execute notebooks end-to-end on test data papermill, nbqa, jupyter nbconvert --execute, pytest
.pdf, .png, .zip Publish as immutable release artifacts Manifest/signature/hash checks Size/hash/allowlist validation N/A (binary) Artifact download/open/readability smoke tests Get-FileHash, Expand-Archive, release artifact checks
.pbir, .pbism, .tmdl, .pbix Deploy semantic/report artifacts via Fabric/Power BI deployment pipeline Validate model metadata, bindings, and model rules via notebooks Refresh datasets and verify key measures/queries and memory profile Test model scripts/rules where supported End-to-end refresh + report render smoke tests + notebook analyzers Fabric/Power BI REST APIs, pbi-tools, XMLA scripts, service notebooks
.bicep (if used) Run plan-like preview, then staged deploy Compile and lint templates Assert planned resources (SKU, region, tags, names) Module-level assertions/scripts Deploy to ephemeral RG/subscription and run smoke tests bicep build, az deployment what-if, checkov
.tf, .tfvars (if used) plan in PR, apply in gated release Format/validate/providers/modules checks Inspect plan for policy/tag/naming compliance Module tests/terratest where implemented Apply to test workspace and verify provisioned resources terraform fmt -check, terraform validate, terraform plan, tfsec, checkov
.md, .txt (repo content) Publish as docs artifacts/site content Markdown and link linting Check generated docs links and anchors Spelling and glossary validation Docs site build and link crawl in preview env markdownlint-cli2, lychee, cspell

Tool Notes (Requested Focus)

yamllint

  • Use it for all .yml/.yaml files before pipeline execution.
  • Recommended command:
yamllint .

jq

  • Use it to parse and validate .json quickly in CI.
  • Recommended commands:
jq -e . path/to/file.json
jq -e '.requiredKey != null' path/to/file.json

pytest + Mocking

  • Use pytest for Python unit and integration layers.
  • Use pytest-mock or unittest.mock to isolate side effects.
  • Typical split:
pytest -m "not integration" -q
pytest -m integration -q

pytest-archon + Boundary Tools

  • Use architecture tests to enforce layering boundaries (for example, api -> services -> data, but not api -> data).
  • Keep boundary rules in test code and fail PRs when forbidden imports or dependency directions are introduced.
  • Complement pytest-archon with tools like import-linter/deptry when you need package-level dependency governance.

cspell

  • Use cspell to catch misspellings in docs, configs, and user-facing messages.
  • Maintain a project dictionary for domain-specific terms to minimize false positives.
  • Recommended commands:
cspell "**/*.{md,txt,yml,yaml,json,ts,tsx,py}" --no-progress
cspell "**/*" --config .cspell.json

bicep + terraform

  • Use preview/plan checks before any deploy step.
  • Bicep example:
bicep build .\infra\main.bicep
az deployment group what-if --resource-group <rg> --template-file .\infra\main.bicep
  • Terraform example:
terraform fmt -check
terraform validate
terraform plan -out tfplan

Semantic Model Verification with Service Notebooks

Use Power BI/Fabric service notebooks to validate semantic model quality after deploying .pbir, .pbism, .tmdl, or .pbix changes.

Recommended Verification Flow

  1. Open sample notebooks from the semantic model entry points in Power BI service.
  2. Run Best Practice Analyzer notebook to evaluate model design/performance rules.
  3. Run Model Memory Analyzer notebook to inspect table/column/partition memory usage.
  4. Record key findings (rule violations, high-memory objects) as release gates.
  5. Re-run after fixes and compare notebook outputs to ensure regression-free improvements.

What to Check

  • Model quality rules: Performance, DAX Expressions, Error Prevention, Maintenance, and Formatting categories.
  • Memory hotspots: oversized tables/columns, relationships, hierarchies, partitions.
  • Data correctness: sample DAX/query outputs for critical measures and dimensions after refresh.

Permissions and Preconditions

  • User must have Build permission on the semantic model.
  • Workspace must be on Fabric capacity and user should have Contributor role for notebook creation.
  • Avoid semantic model names ending with trailing whitespace (known notebook execution limitation).

Automation Pattern

  • Use deployment pipeline stage order: deploy model -> refresh -> run notebook analyzers -> evaluate thresholds.
  • Persist notebook outputs (JSON/CSV summary) to artifacts and fail CI/CD when configured thresholds are exceeded.
  • Keep baseline snapshots for BPA violations and memory metrics to detect regressions over time.

Reference: https://learn.microsoft.com/en-us/power-bi/transform-model/service-notebooks

Database Testing Details

PostgreSQL with pgTAP

  • Install and enable pgTAP in test databases, then run SQL-based unit tests as part of migration validation.
  • Use pg_prove in CI to execute tests under tests/sql/ (or your chosen folder).
  • Typical checks: table existence, constraint behavior, function outputs, view correctness, and permission grants.
  • Suggested flow: apply migrations -> run pgTAP tests -> run app integration tests -> optional rollback check.

Cosmos DB Testing

  • Unit tests: mock Cosmos DB SDK calls (ContainerProxy, query_items, upsert_item) with pytest-mock.
  • Integration tests: run against Cosmos DB Emulator (local) or a dedicated non-prod account/container.
  • Validate partition key behavior, RU-sensitive query paths, consistency expectations, and TTL/index policy behavior.
  • Add idempotent setup/teardown fixtures and isolate data by test run id to avoid cross-test contamination.

Fabric Permission Testing

  • Validate workspace role assignments (Viewer, Contributor, Member, Admin) for expected access paths.
  • Validate semantic model Build permission explicitly for notebook-based model verification flows.
  • Add negative tests (forbidden actions) to confirm least-privilege is enforced.
  • Test permission propagation after deployment: artifact access, refresh ability, notebook creation, and API execution.
  • In CI, run a post-deploy smoke suite using a restricted service principal identity in addition to elevated identities.

Naming Conventions by File Category

  • .py, .ps1, .sql: snake_case file names with action-oriented intent (for example, load_transactions.py, create_views.sql).
  • .ts, .tsx, .js, .jsx: app conventions (camelCase utilities, PascalCase React components) kept consistent per repo.
  • .yml/.yaml, .json: environment- and purpose-qualified names (for example, pipeline.pr.yml, settings.prod.json).
  • .ipynb: include purpose and stage in name (for example, qa_evaluation_notebook.ipynb, ingest_stage01.ipynb).
  • .pbir, .pbism, .tmdl, .pbix: include subject area + environment suffix where relevant to avoid deployment ambiguity.
  • Keep naming rules documented in a single source (README/CONTRIBUTING) and enforce with lint checks in CI.

Suggested CI Job Layout

  1. lint-yaml-config – run yamllint for pipeline/config files.
  2. validate-json-contracts – run jq parse checks + JSON Schema validation.
  3. check-env-contract – ensure required .env keys exist and are non-placeholder.
  4. sql-lint-migration-smoke – lint SQL and run migration smoke tests.
  5. powershell-unit-tests – run Pester with mocked external commands.
  6. python-unit-tests – run pytest unit tests + coverage.
  7. python-integration-tests – run pytest -m integration against test dependencies.
  8. frontend-quality-gate – lint, typecheck, unit tests for TS/JS.
  9. data-contract-quality – schema + quality checks for CSV/Parquet/GeoJSON.
  10. notebook-smoke – execute selected notebooks with papermill.
  11. iac-bicep-validatebicep build + az deployment what-if.
  12. iac-terraform-validateterraform fmt/validate/plan.
  13. postgres-pgtap-tests – run pg_prove test packs after migration in ephemeral PostgreSQL.
  14. cosmos-integration-tests – run integration tests against Cosmos emulator/test account with isolated containers.
  15. fabric-permission-smoke – verify Build/workspace role permissions with positive and negative checks.
  16. python-architecture-boundaries – enforce import/layer boundaries via pytest-archon (and optional import-linter).
  17. spelling-quality-gate – run cspell on docs/config/code comments.

Tooling Links Reference

This reference maps each tool/API mentioned in file-type-deploy-verify-test-guide.md to official docs/help, CLI/API references, GitHub repositories, and Microsoft Learn pages where available.

Linters, Parsers, and Validation

Tool Help / Docs CLI / API Reference GitHub Microsoft Learn
yamllint https://yamllint.readthedocs.io/ https://yamllint.readthedocs.io/en/stable/quickstart.html https://github.com/adrienverge/yamllint N/A
yq https://mikefarah.gitbook.io/yq/ https://mikefarah.gitbook.io/yq/commands https://github.com/mikefarah/yq N/A
jq https://jqlang.org/manual/ https://jqlang.org/manual/ https://github.com/jqlang/jq N/A
ajv-cli https://ajv.js.org/guide/getting-started.html https://ajv.js.org/packages/ajv-cli.html https://github.com/ajv-validator/ajv-cli N/A
dotenv-linter https://dotenv-linter.readthedocs.io/ https://dotenv-linter.readthedocs.io/en/latest/pages/usage/index.html https://github.com/dotenv-linter/dotenv-linter N/A
sqlfluff https://docs.sqlfluff.com/ https://docs.sqlfluff.com/en/stable/cli.html https://github.com/sqlfluff/sqlfluff N/A
pgTAP https://pgtap.org/documentation.html https://pgtap.org/documentation.html https://github.com/theory/pgtap N/A
pg_prove https://pgtap.org/documentation.html Part of pgTAP TAP harness workflow https://github.com/theory/pgtap N/A
graphql-cli https://www.graphql-cli.com/ https://www.graphql-cli.com/docs/commands https://github.com/Urigo/graphql-cli N/A
SpectaQL https://spectaql.com/docs https://spectaql.com/docs/getting-started https://github.com/anvilco/spectaql N/A
markdownlint-cli2 https://github.com/DavidAnson/markdownlint-cli2 https://github.com/DavidAnson/markdownlint-cli2#usage https://github.com/DavidAnson/markdownlint-cli2 N/A
lychee https://lychee.cli.rs/ https://lychee.cli.rs/usage/cli/ https://github.com/lycheeverse/lychee N/A
cspell https://cspell.org/ https://cspell.org/docs/api/cspell https://github.com/streetsidesoftware/cspell N/A
checkov https://www.checkov.io/ https://www.checkov.io/2.Basics/CLI%20Command%20Reference.html https://github.com/bridgecrewio/checkov N/A
tfsec https://aquasecurity.github.io/tfsec/latest/ https://aquasecurity.github.io/tfsec/latest/guides/usage/ https://github.com/aquasecurity/tfsec N/A

PowerShell and Script Testing

Tool Help / Docs CLI / API Reference GitHub Microsoft Learn
ConvertFrom-Json Get-Help ConvertFrom-Json -Full https://learn.microsoft.com/powershell/module/microsoft.powershell.utility/convertfrom-json Built into PowerShell https://learn.microsoft.com/powershell/
Invoke-Sqlcmd Get-Help Invoke-Sqlcmd -Full https://learn.microsoft.com/powershell/module/sqlserver/invoke-sqlcmd SQL Server PowerShell module https://learn.microsoft.com/sql/powershell/sql-server-powershell
PSScriptAnalyzer Get-Help Invoke-ScriptAnalyzer -Full https://learn.microsoft.com/powershell/utility-modules/psscriptanalyzer/overview https://github.com/PowerShell/PSScriptAnalyzer https://learn.microsoft.com/powershell/
Pester Get-Help Invoke-Pester -Full https://pester.dev/docs/quick-start https://github.com/pester/Pester N/A
Get-FileHash Get-Help Get-FileHash -Full https://learn.microsoft.com/powershell/module/microsoft.powershell.utility/get-filehash Built into PowerShell https://learn.microsoft.com/powershell/
Expand-Archive Get-Help Expand-Archive -Full https://learn.microsoft.com/powershell/module/microsoft.powershell.archive/expand-archive Built into PowerShell https://learn.microsoft.com/powershell/

Python Quality, Testing, and Notebook Tooling

Tool Help / Docs CLI / API Reference GitHub Microsoft Learn
pytest https://docs.pytest.org/ https://docs.pytest.org/en/stable/reference/reference.html https://github.com/pytest-dev/pytest N/A
pytest-mock https://pytest-mock.readthedocs.io/ https://pytest-mock.readthedocs.io/en/latest/usage.html https://github.com/pytest-dev/pytest-mock N/A
unittest.mock https://docs.python.org/3/library/unittest.mock.html https://docs.python.org/3/library/unittest.mock.html Python stdlib N/A
pytest-archon https://pypi.org/project/pytest-archon/ https://pypi.org/project/pytest-archon/ https://github.com/jwbargsten/pytest-archon N/A
import-linter https://import-linter.readthedocs.io/ https://import-linter.readthedocs.io/en/stable/ https://github.com/seddonym/import-linter N/A
deptry https://deptry.com/ https://deptry.com/usage/ https://github.com/fpgmaas/deptry N/A
ruff https://docs.astral.sh/ruff/ https://docs.astral.sh/ruff/configuration/ https://github.com/astral-sh/ruff N/A
mypy https://mypy.readthedocs.io/ https://mypy.readthedocs.io/en/stable/command_line.html https://github.com/python/mypy N/A
pandera https://pandera.readthedocs.io/ https://pandera.readthedocs.io/en/stable/ https://github.com/unionai-oss/pandera N/A
Great Expectations https://docs.greatexpectations.io/ https://docs.greatexpectations.io/docs/reference/learn/ https://github.com/great-expectations/great_expectations N/A
DuckDB https://duckdb.org/docs/ https://duckdb.org/docs/stable/clients/cli/overview.html https://github.com/duckdb/duckdb N/A
PyArrow https://arrow.apache.org/docs/python/ https://arrow.apache.org/docs/python/api.html https://github.com/apache/arrow N/A
papermill https://papermill.readthedocs.io/ https://papermill.readthedocs.io/en/latest/usage-cli.html https://github.com/nteract/papermill N/A
nbqa https://nbqa.readthedocs.io/ https://nbqa.readthedocs.io/en/latest/readme.html https://github.com/nbQA-dev/nbQA N/A
jupyter nbconvert https://nbconvert.readthedocs.io/ https://nbconvert.readthedocs.io/en/latest/usage.html https://github.com/jupyter/nbconvert N/A

JavaScript / TypeScript / UI Testing

Tool Help / Docs CLI / API Reference GitHub Microsoft Learn
ESLint https://eslint.org/docs/latest/ https://eslint.org/docs/latest/use/command-line-interface https://github.com/eslint/eslint N/A
TypeScript (tsc) https://www.typescriptlang.org/docs/ https://www.typescriptlang.org/docs/handbook/compiler-options.html https://github.com/microsoft/TypeScript https://learn.microsoft.com/training/modules/build-javascript-applications-typescript/
Vitest https://vitest.dev/guide/ https://vitest.dev/guide/cli.html https://github.com/vitest-dev/vitest N/A
Jest https://jestjs.io/docs/getting-started https://jestjs.io/docs/cli https://github.com/jestjs/jest N/A
Playwright https://playwright.dev/docs/intro https://playwright.dev/docs/test-cli https://github.com/microsoft/playwright https://learn.microsoft.com/training/modules/build-with-playwright/
Cypress https://docs.cypress.io/ https://docs.cypress.io/guides/guides/command-line https://github.com/cypress-io/cypress N/A

Build and Test Ecosystem (Java/.NET)

Tool Help / Docs CLI / API Reference GitHub Microsoft Learn
Apache Maven https://maven.apache.org/guides/ https://maven.apache.org/ref/current/maven-embedder/cli.html https://github.com/apache/maven N/A
Gradle https://docs.gradle.org/current/userguide/userguide.html https://docs.gradle.org/current/userguide/command_line_interface.html https://github.com/gradle/gradle N/A
dotnet test https://learn.microsoft.com/dotnet/core/tools/dotnet-test https://learn.microsoft.com/dotnet/core/tools/dotnet-test https://github.com/dotnet/sdk https://learn.microsoft.com/dotnet/
Mockito https://site.mockito.org/ https://javadoc.io/doc/org.mockito/mockito-core/latest/org/mockito/Mockito.html https://github.com/mockito/mockito N/A
Moq https://github.com/devlooped/moq/wiki/Quickstart https://github.com/devlooped/moq/wiki/Quickstart https://github.com/devlooped/moq N/A
Testcontainers https://testcontainers.com/guides/ https://java.testcontainers.org/ and https://dotnet.testcontainers.org/ https://github.com/testcontainers/testcontainers-java and https://github.com/testcontainers/testcontainers-dotnet N/A

IaC and Cloud Deployment

Tool Help / Docs CLI / API Reference GitHub Microsoft Learn
Bicep https://learn.microsoft.com/azure/azure-resource-manager/bicep/ https://learn.microsoft.com/azure/azure-resource-manager/bicep/bicep-cli https://github.com/Azure/bicep https://learn.microsoft.com/azure/azure-resource-manager/bicep/
Azure CLI (az deployment ... what-if) https://learn.microsoft.com/cli/azure/ https://learn.microsoft.com/cli/azure/deployment/group#az-deployment-group-what-if https://github.com/Azure/azure-cli https://learn.microsoft.com/cli/azure/
Terraform https://developer.hashicorp.com/terraform/docs https://developer.hashicorp.com/terraform/cli https://github.com/hashicorp/terraform N/A

Data and Query Platforms

Tool / API Help / Docs CLI / API Reference GitHub Microsoft Learn
OGR / ogrinfo (GDAL) https://gdal.org/ https://gdal.org/programs/ogrinfo.html https://github.com/OSGeo/gdal N/A
Kusto Query Language (KQL) https://learn.microsoft.com/azure/data-explorer/kusto/query/ https://learn.microsoft.com/azure/data-explorer/kusto/query/ N/A https://learn.microsoft.com/azure/data-explorer/kusto/query/
Azure Data Explorer APIs https://learn.microsoft.com/azure/data-explorer/ https://learn.microsoft.com/azure/data-explorer/kusto/api/ https://github.com/Azure/azure-kusto-python (and related SDK repos) https://learn.microsoft.com/azure/data-explorer/
Azure Cosmos DB (NoSQL) https://learn.microsoft.com/azure/cosmos-db/nosql/ https://learn.microsoft.com/azure/cosmos-db/nosql/sdk-python https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/cosmos/azure-cosmos https://learn.microsoft.com/azure/cosmos-db/nosql/
Cosmos DB Emulator https://learn.microsoft.com/azure/cosmos-db/how-to-develop-emulator Emulator setup and local endpoint usage in docs N/A https://learn.microsoft.com/azure/cosmos-db/how-to-develop-emulator
GraphQL (spec) https://graphql.org/learn/ https://graphql.org/learn/queries/ https://github.com/graphql/graphql-spec N/A

Fabric / Power BI Semantic Model and Reporting

Tool / API Help / Docs CLI / API Reference GitHub Microsoft Learn
Power BI Service Notebooks for semantic models https://learn.microsoft.com/power-bi/transform-model/service-notebooks Entry points described in article (Model health/Home/OneLake menu) N/A https://learn.microsoft.com/power-bi/transform-model/service-notebooks
Semantic Link https://learn.microsoft.com/fabric/data-science/semantic-link-overview Python APIs documented in Semantic Link docs N/A https://learn.microsoft.com/fabric/data-science/semantic-link-overview
Semantic Link Labs https://semantic-link-labs.readthedocs.io/ Notebook helper functions in docs https://github.com/microsoft/semantic-link-labs Mentioned in service notebook docs
Power BI REST API https://learn.microsoft.com/rest/api/power-bi/ https://learn.microsoft.com/rest/api/power-bi/ N/A https://learn.microsoft.com/rest/api/power-bi/
XMLA endpoint (Power BI/Fabric) https://learn.microsoft.com/power-bi/enterprise/service-premium-connect-tools XMLA-based model operations N/A https://learn.microsoft.com/power-bi/enterprise/service-premium-connect-tools
pbi-tools https://pbi.tools/ https://pbi.tools/cli/ https://github.com/pbi-tools/pbi-tools N/A
Power BI semantic model permissions https://learn.microsoft.com/power-bi/connect-data/service-datasets-permissions Permission model reference N/A https://learn.microsoft.com/power-bi/connect-data/service-datasets-permissions
Fabric workspace roles https://learn.microsoft.com/power-bi/collaborate-share/service-roles-new-workspaces Role capabilities by workspace role N/A https://learn.microsoft.com/power-bi/collaborate-share/service-roles-new-workspaces

Quick Notes

  • For PowerShell-native commands, use Get-Help <Command> -Full for offline/local help.
  • For CI/CD, pin tool versions in lockfiles or pipeline setup steps to avoid drift.
  • For Fabric/Power BI verification, combine REST/XMLA checks with service notebook analyzers for semantic model quality gates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment