Change Management Process

Pre-Deployment Checklist

Before EVERY infrastructure change:

• 1. Review scope — What services are affected?
• 2. Check dependencies — Read context/inventory/services.md
• 3. Backup current state

  cp docker-compose.yml docker-compose.yml.backup-$(date +%Y%m%d-%H%M%S)

• 4. Check server resources — free -h and df -h
  • Refuse if RAM >85% or disk >85%
• 5. Test in non-production first (if possible)
• 6. Document expected behavior — What should happen after the change?
• 7. Prepare rollback plan — How to undo if it fails?

Deployment Steps

1. Implement change (one service at a time)
2. Validate immediately

   # Check container is running and healthy
   docker ps --filter name=<service>
   # Check logs for errors
   docker logs <container> --tail 50
   # Test connectivity
   docker exec traefik wget -qO- http://<service>:<port>

3. Monitor for 15 minutes — Watch logs and metrics
4. Document in changelog — Alfred/docs/changelog.md
5. Update inventory — context/inventory/docker-containers.md
6. Git commit with descriptive message
7. Send NTFY notification (if production change)

If Change Fails

1. Stop the failing service immediately
2. Restore backup config

   cp docker-compose.yml.backup-* docker-compose.yml
   docker compose up -d

3. Verify functionality restored
4. Document what failed in daily log
5. Send NTFY critical notification

   curl -s -H "Priority: urgent" \
     -d "FAILED: [service] change rolled back. Details: [error]" \
     https://ntfy.karetechsolutions.com/infrastructure-critical

6. Create incident report if production impact >5 minutes

Forbidden Without Explicit Approval

• Delete databases or data volumes
• Force push to git repositories
• Restart production services during business hours (8AM-6PM GYT)
• Modify Pangolin/Traefik/Gerbil config files directly
• Change network segmentation
• Disable monitoring or alerts
• Remove backup jobs or scripts

Change Types and Risk Levels

NTFY Notification Templates

# Before production change
curl -s -d "MAINTENANCE: Starting [change] on [service]. ETA: [time]" \
  https://ntfy.karetechsolutions.com/infrastructure
 
# After successful change
curl -s -d "COMPLETE: [change] on [service] successful. All services healthy." \
  https://ntfy.karetechsolutions.com/infrastructure
 
# After failed change
curl -s -H "Priority: urgent" \
  -d "FAILED: [change] on [service] rolled back. Error: [details]" \
  https://ntfy.karetechsolutions.com/infrastructure-critical