StateCert deployment guideline

deployment.md

Deployment Guide

Developer Responsibility

Every developer is responsible for their code and actions. Before deploying any feature or bug fix:

• Ensure your code meets quality standards and follows best practices
• Test your changes thoroughly in development and staging environments
• Verify that your changes won't negatively impact other parts of the system
• Be prepared to support and fix any issues that arise from your changes
• Consider the security implications of your changes
• Document any necessary changes to configuration or dependencies

Remember: You are the first line of defense in maintaining system stability and reliability.

Feature Branch Deployment

1. For significant changes, create or find the GitHub issue:

   • Check if there's an existing issue for your feature/bug
   • If no issue exists, create one with a short, descriptive title
   • Add detailed description in the issue if needed
   • Note the issue number

   For small changes, you can skip issue creation and document the changes directly in the PR description.

2. Create your branch:

   • For issues: Use the format: {issue_number}-title-of-issue
   • For small changes: Use a descriptive name like fix-login-button or update-readme
   • You can create the branch directly from the GitHub issue using the "Create a branch" button
   • Or create it locally:

     git checkout main
     git pull origin main
     git checkout -b {branch-name}

3. Make your changes and commit them:

   git add .
   git commit -m "Description of your changes"

4. Push your branch to GitHub:

   git push origin {branch-name}

5. Create a Pull Request (PR) on GitHub:

   • Go to the repository on GitHub
   • Click "New Pull Request"
   • Select your feature branch as the source
   • Select main as the target
   • Add a descriptive title and description
   • For issues: Link the PR to the relevant issue
   • Request reviews from at least one team member

6. For very small, straightforward changes:

   • If the change is minimal (e.g., typo fixes, simple documentation updates)
   • Can be fully described in a single commit message
   • Has no risk of conflicts or side effects
   • You may push directly to main with a clear, descriptive commit message
   • Example: git commit -m "Fix typo in README.md" && git push origin main

7. Once approved, merge your PR into main

PR Review Apps

For PR reviews, you can create a review app in Heroku:

1. Go to the Heroku dashboard
2. In the left sidebar, find the "Review Apps" section
3. Click the "Create review app" button
4. The review app will be automatically created based on your PR branch
5. Use this review app to test and demonstrate your changes

PR Cooldown Policy

To ensure stability and prevent conflicts between PRs, we follow these cooldown rules:

1. Standard Cooldown (2 days):

   • Applies to PRs that could impact or conflict with other recent changes
   • Examples include:
     • Major feature changes
     • Database schema modifications
     • Core functionality updates
     • Changes to shared components or services
   • The 2-day cooldown allows time for:
     • Testing in staging
     • Identifying potential conflicts
     • Gathering feedback from other team members

2. Immediate Merge:

   • Applies to PRs that are:
     • Minor bug fixes
     • UI/UX improvements without core changes
     • Documentation updates
     • Non-conflicting feature additions
   • These can be merged immediately after approval

3. Cooldown Exceptions:

   • Critical security fixes
   • Emergency bug fixes
   • Time-sensitive business requirements
   • Must be approved by at least two team leads

Safe Database Schema Changes

When making database schema changes, follow these deployment rules to prevent issues with stagnant code:

1. First Deploy - Add New Columns

   • Add new columns with default values or nullable
   • No code changes that use the new columns
   • This allows the new columns to exist in production before any code uses them

2. Second Deploy - Code Changes

   • Update code to use the new columns
   • Ensure backward compatibility with old data
   • This deploy can safely use the new columns since they exist

3. Third Deploy - Remove Old Columns

   • Remove old columns only after confirming no code uses them
   • Ensure all data has been migrated if necessary
   • This is the safest time to remove columns as all code uses the new structure

This three-step process prevents issues where:

• Code tries to use columns that don't exist yet
• Code tries to access columns that have been removed
• Data migration issues occur due to timing mismatches

Production Deployment

Production deployments are handled through Heroku. Here's the process:

1. Ensure your changes are merged into main

2. Verify the build status on GitHub Actions

3. The deployment to staging will be automatically triggered if:

   • All tests pass in CI
   • All tests pass in the staging environment with data (coming soon)
   • The build is successful
   • The changes are on the main branch

4. Once staging is updated with the latest commit from main:

   • Go to the Heroku dashboard
   • Navigate to the staging app
   • Click the "Promote to production" button
   • Before promoting, click the "Compare on GitHub" button to verify all changes
   • Review the changes carefully to ensure everything is correct
   • Proceed with promoting to both production apps

5. Monitor the deployment:

   • Check Heroku dashboard for deployment status
   • Monitor error tracking services (Sentry, Honeybadger) for any issues
   • Verify the deployment in the production environment

Post-Deployment Checklist

After deployment, verify:

1. Error tracking services for any new errors
2. Background jobs are processing correctly
3. Email delivery is working (check Postmark)
4. Analytics are being collected
5. Feature flags are working as expected
6. Performance metrics are within normal ranges

Troubleshooting Deployments

If you encounter deployment issues:

1. Check the relevant service's dashboard for error messages
2. Review the deployment logs in Heroku
3. Verify environment variables and configurations
4. Check the status pages of the services (see External Services)