Docker Compose Deployment Guide

Complete guide for deploying the New Hires Reporting system using Docker Compose with pre-built images.

Prerequisites

Before deploying, ensure you have completed:

• [ ] Docker 20.10+ installed
• [ ] Docker Compose 2.0+ installed
• [ ] AWS account with Bedrock access (see AWS Setup)
• [ ] AWS IAM credentials with Bedrock invoke permissions
• [ ] ECR authentication configured
• [ ] Ports 8080 and 8000 available
• [ ] Minimum 4 GB RAM, 20 GB disk space

Quick Start (10 Minutes)

Step 1: Create Environment Configuration

Create your .env file:

# Copy example or create new
cp .env.example .env

# Edit with your settings
nano .env

Minimum required .env:

# AWS Bedrock Configuration (REQUIRED)
AWS_ACCESS_KEY_ID=your_aws_access_key_here
AWS_SECRET_ACCESS_KEY=your_aws_secret_key_here
AWS_REGION=us-east-1

# Docker Image Version (REQUIRED)
IMAGE_TAG=sha-1234567

# Database Configuration (Optional - defaults are fine)
POSTGRES_PASSWORD=change_this_in_production
POSTGRES_USER=newhires
POSTGRES_DB=newhires

See Environment Variables for complete reference.

Step 2: Authenticate to AWS ECR

# Install AWS CLI if not already installed
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Configure AWS credentials
aws configure
# Enter your AWS Access Key ID and Secret Access Key
# Region: us-east-1

# Authenticate Docker to ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  878796852397.dkr.ecr.us-east-1.amazonaws.com

Expected output: Login Succeeded

Step 3: Pull Docker Images

# Pull all production images
docker-compose -f docker-compose.prod.yml pull

This downloads four images: - PostgreSQL: ~150MB - Backend API: ~500MB - Workers: ~500MB - Frontend: ~100MB

Step 4: Start Services

# Start all services in detached mode
docker-compose -f docker-compose.prod.yml up -d

Expected output:

Creating network "newhires-reporting_default" ... done
Creating volume "newhires-reporting_postgres-data" ... done
Creating newhires-db       ... done
Creating newhires-backend  ... done
Creating newhires-workers  ... done
Creating newhires-frontend ... done

Step 5: Initialize Database

# Run database migrations
docker exec newhires-backend alembic upgrade head

Expected output:

INFO  [alembic.runtime.migration] Running upgrade  -> abcdef, Initial migration

Step 6: Verify Deployment

# Check all containers are running
docker ps

# Test backend health
curl http://localhost:8000/health

# Expected response:
# {"status":"healthy"}

# Access web interface
# Open browser to: http://your-server:8080

---

Detailed Production Deployment

Directory Structure

Organize your deployment files:

# Create deployment directory
mkdir -p /opt/newhires-reporting
cd /opt/newhires-reporting

# Directory structure:
# /opt/newhires-reporting/
# ├── docker-compose.prod.yml
# ├── .env
# └── backups/            (create for database backups)

Understanding docker-compose.prod.yml

The production compose file defines four services:

services:
  db:
    image: postgres:16-alpine
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}

  backend:
    image: 878796852397.dkr.ecr.us-east-1.amazonaws.com/newhires/backend:${IMAGE_TAG}
    ports:
      - "8000:8000"
    depends_on:
      - db
    environment:
      - DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}

  workers:
    image: 878796852397.dkr.ecr.us-east-1.amazonaws.com/newhires/workers:${IMAGE_TAG}
    depends_on:
      - db
    environment:
      - DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_REGION=${AWS_REGION}

  frontend:
    image: 878796852397.dkr.ecr.us-east-1.amazonaws.com/newhires/frontend:${IMAGE_TAG}
    ports:
      - "8080:80"
    depends_on:
      - backend

Key Points: - Uses pre-built ECR images (no local code needed) - Database persisted in Docker volume - Workers process jobs in background - Frontend proxies API requests to backend

Environment Configuration

Create a secure .env file:

# Create .env with secure permissions
touch .env
chmod 600 .env

Production .env template:

###############################
# Docker Image Configuration
###############################
IMAGE_TAG=sha-a85b396

###############################
# AWS Bedrock Configuration
###############################
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_REGION=us-east-1

# Optional: Override default model
# BEDROCK_MODEL_ID=us.anthropic.claude-sonnet-4-5-20250929-v1:0

###############################
# Database Configuration
###############################
POSTGRES_DB=newhires
POSTGRES_USER=newhires
POSTGRES_PASSWORD=CHANGE_THIS_SECURE_PASSWORD_123

###############################
# Worker Configuration
###############################
# Polling interval in seconds
POLL_INTERVAL=5

# Maximum Bedrock API calls at once
MAX_CONCURRENT_BEDROCK_CALLS=2

# Maximum retry attempts per job
MAX_AI_ATTEMPTS=5

###############################
# Frontend Configuration
###############################
# Backend API URL (leave default for same-host deployment)
VITE_API_URL=http://localhost:8000/api/v1

Security

• Never commit .env files to version control
• Use strong passwords for POSTGRES_PASSWORD
• Rotate AWS credentials every 90 days
• Set file permissions: chmod 600 .env

Deploy Services

Step-by-step deployment:

# 1. Ensure you're in the deployment directory
cd /opt/newhires-reporting

# 2. Verify .env file exists and is configured
ls -la .env
cat .env | grep AWS_ACCESS_KEY_ID  # Should show your key

# 3. Authenticate to ECR (credentials expire after 12 hours)
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  878796852397.dkr.ecr.us-east-1.amazonaws.com

# 4. Pull latest images
docker-compose -f docker-compose.prod.yml pull

# 5. Stop any running containers (if updating)
docker-compose -f docker-compose.prod.yml down

# 6. Start services
docker-compose -f docker-compose.prod.yml up -d

# 7. Wait for database to be ready (~10 seconds)
sleep 10

# 8. Run database migrations
docker exec newhires-backend alembic upgrade head

# 9. Verify all services are running
docker-compose -f docker-compose.prod.yml ps

# Expected output shows 4 containers "Up"

Verify Deployment

Check each service is healthy:

# 1. Check Docker containers
docker ps | grep newhires

# Should show 4 containers:
# newhires-frontend
# newhires-backend
# newhires-workers
# newhires-db

# 2. Check backend API health
curl http://localhost:8000/health
# Expected: {"status":"healthy"}

# 3. Check database connection
docker exec newhires-db psql -U newhires -d newhires -c "SELECT 1;"
# Expected: Shows "1" result

# 4. Check worker is polling
docker logs newhires-workers --tail=20
# Should see: "INFO: Worker polling for jobs..."

# 5. Check frontend is accessible
curl -I http://localhost:8080
# Expected: HTTP/1.1 200 OK

# 6. Open web interface
# Navigate to: http://your-server:8080

---

Common Operations

View Logs

# View all service logs (follow mode)
docker-compose -f docker-compose.prod.yml logs -f

# View specific service logs
docker-compose -f docker-compose.prod.yml logs -f backend
docker-compose -f docker-compose.prod.yml logs -f workers
docker-compose -f docker-compose.prod.yml logs -f frontend

# View last 100 lines
docker-compose -f docker-compose.prod.yml logs --tail=100 workers

# View logs with timestamps
docker-compose -f docker-compose.prod.yml logs -f -t workers

# Save logs to file
docker-compose -f docker-compose.prod.yml logs > deployment_logs.txt

Restart Services

# Restart all services
docker-compose -f docker-compose.prod.yml restart

# Restart specific service
docker-compose -f docker-compose.prod.yml restart backend
docker-compose -f docker-compose.prod.yml restart workers

# Full restart with rebuild
docker-compose -f docker-compose.prod.yml down
docker-compose -f docker-compose.prod.yml up -d

Stop Services

# Stop services (preserves data)
docker-compose -f docker-compose.prod.yml stop

# Stop and remove containers (preserves volumes)
docker-compose -f docker-compose.prod.yml down

# Stop and remove everything INCLUDING DATA (DANGEROUS!)
docker-compose -f docker-compose.prod.yml down -v

Scale Workers

To process more jobs concurrently, scale the workers service:

# Scale to 3 worker containers
docker-compose -f docker-compose.prod.yml up -d --scale workers=3

# Scale back to 1 worker
docker-compose -f docker-compose.prod.yml up -d --scale workers=1

# Check running workers
docker ps | grep workers

Worker Scaling

• Each worker can handle MAX_CONCURRENT_BEDROCK_CALLS concurrent Bedrock calls
• 3 workers with MAX_CONCURRENT_BEDROCK_CALLS=2 = 6 concurrent AI calls
• Monitor AWS Bedrock costs when scaling workers

Update to New Version

When a new version is released:

# 1. Update IMAGE_TAG in .env file
nano .env
# Change: IMAGE_TAG=sha-NEW_VERSION

# 2. Re-authenticate to ECR if needed (tokens expire after 12h)
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  878796852397.dkr.ecr.us-east-1.amazonaws.com

# 3. Pull new images
docker-compose -f docker-compose.prod.yml pull

# 4. Stop current containers
docker-compose -f docker-compose.prod.yml down

# 5. Start with new images
docker-compose -f docker-compose.prod.yml up -d

# 6. Run database migrations (important!)
docker exec newhires-backend alembic upgrade head

# 7. Verify health
curl http://localhost:8000/health
docker-compose -f docker-compose.prod.yml logs --tail=50

Database Operations

Backup Database

# Create backup directory
mkdir -p /opt/newhires-reporting/backups

# Create backup
docker exec newhires-db pg_dump -U newhires newhires > \
  /opt/newhires-reporting/backups/backup_$(date +%Y%m%d_%H%M%S).sql

# Create compressed backup
docker exec newhires-db pg_dump -U newhires newhires | \
  gzip > /opt/newhires-reporting/backups/backup_$(date +%Y%m%d_%H%M%S).sql.gz

# Automated daily backup (add to crontab)
# crontab -e
# 0 2 * * * cd /opt/newhires-reporting && docker exec newhires-db pg_dump -U newhires newhires | gzip > backups/backup_$(date +\%Y\%m\%d).sql.gz

Restore Database

# Stop workers to prevent job processing during restore
docker-compose -f docker-compose.prod.yml stop workers

# Restore from backup
cat backups/backup_20241124.sql | \
  docker exec -i newhires-db psql -U newhires -d newhires

# Or restore from compressed backup
gunzip -c backups/backup_20241124.sql.gz | \
  docker exec -i newhires-db psql -U newhires -d newhires

# Restart workers
docker-compose -f docker-compose.prod.yml start workers

Access Database CLI

# Connect to PostgreSQL
docker exec -it newhires-db psql -U newhires -d newhires

# Example queries:
# \dt                              -- List tables
# SELECT COUNT(*) FROM correction_jobs;
# SELECT * FROM job_queue WHERE status='pending';
# \q                               -- Quit

---

Monitoring

Check Service Status

# View container status
docker-compose -f docker-compose.prod.yml ps

# View resource usage
docker stats

# View network connectivity
docker network inspect newhires-reporting_default

Check Worker Processing

# View worker logs (live)
docker logs -f newhires-workers

# Check for Bedrock API calls
docker logs newhires-workers | grep "Bedrock"

# Check job processing
docker exec newhires-db psql -U newhires -d newhires -c \
  "SELECT status, COUNT(*) FROM correction_jobs GROUP BY status;"

Health Checks

# Backend health
curl http://localhost:8000/health

# Frontend health
curl -I http://localhost:8080

# Database health
docker exec newhires-db pg_isready -U newhires

# All services health
docker-compose -f docker-compose.prod.yml ps

---

Troubleshooting

Services Won't Start

# Check logs for errors
docker-compose -f docker-compose.prod.yml logs

# Common issues:

# 1. Missing or invalid .env file
ls -la .env
cat .env | head -5

# 2. ECR authentication expired (re-authenticate)
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  878796852397.dkr.ecr.us-east-1.amazonaws.com

# 3. Port conflicts
sudo netstat -tlnp | grep -E '8000|8080|5432'

# 4. Insufficient disk space
df -h
docker system df

# 5. Insufficient memory
free -h
docker stats --no-stream

Workers Not Processing Jobs

# Check worker logs
docker logs newhires-workers --tail=50

# Verify AWS credentials are set
docker exec newhires-workers env | grep AWS

# Test Bedrock access from worker
docker exec newhires-workers python3 -c "
import boto3
client = boto3.client('bedrock-runtime', region_name='us-east-1')
print('Bedrock connection successful')
"

# Check job queue
docker exec newhires-db psql -U newhires -d newhires -c \
  "SELECT * FROM job_queue WHERE status='pending';"

# Restart workers
docker-compose -f docker-compose.prod.yml restart workers

Database Connection Errors

# Check database is running
docker ps | grep newhires-db

# Check database logs
docker logs newhires-db --tail=50

# Test database connection
docker exec newhires-db psql -U newhires -d newhires -c "SELECT 1;"

# Verify database credentials in .env
grep POSTGRES .env

# Restart database (last resort)
docker-compose -f docker-compose.prod.yml restart db

Frontend Can't Connect to Backend

Symptom: Frontend shows error or blank page

Solutions:

# 1. Verify backend is running
curl http://localhost:8000/health

# 2. Check frontend logs
docker logs newhires-frontend

# 3. Verify frontend environment variable
docker exec newhires-frontend cat /etc/nginx/conf.d/default.conf | grep proxy_pass

# 4. Check network connectivity
docker exec newhires-frontend wget -O- http://backend:8000/health

# 5. Restart frontend
docker-compose -f docker-compose.prod.yml restart frontend

Out of Disk Space

# Check disk usage
df -h
docker system df

# Clean up unused Docker resources
docker system prune -a --volumes

# Remove old images
docker image prune -a

# Remove stopped containers
docker container prune

# Check volume sizes
docker volume ls
docker system df -v

---

Best Practices

Security

• ✅ Set secure POSTGRES_PASSWORD in .env
• ✅ File permissions: chmod 600 .env
• ✅ Rotate AWS credentials every 90 days
• ✅ Keep Docker and images updated
• ✅ Monitor logs for unauthorized access
• ✅ Use firewall rules to restrict port access
• ✅ Consider using HTTPS with reverse proxy

Operations

• ✅ Backup database daily (automated cron job)
• ✅ Monitor disk space and Docker volumes
• ✅ Review logs regularly for errors
• ✅ Test update procedures in staging first
• ✅ Keep IMAGE_TAG version documented
• ✅ Monitor AWS Bedrock costs
• ✅ Set up alerts for service failures

Performance

• ✅ Scale workers based on job volume
• ✅ Tune MAX_CONCURRENT_BEDROCK_CALLS for workload
• ✅ Monitor database size and performance
• ✅ Review Bedrock token usage
• ✅ Use docker stats to monitor resource usage

---

Next Steps

1. Configure Environment Variables - Complete configuration
2. Setup Health Monitoring - Monitor your deployment
3. Optional: Reverse Proxy - Add HTTPS and domain
4. Troubleshooting Guide - Fix common issues

---

Quick Reference

# Deployment
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 878796852397.dkr.ecr.us-east-1.amazonaws.com
docker-compose -f docker-compose.prod.yml pull
docker-compose -f docker-compose.prod.yml up -d
docker exec newhires-backend alembic upgrade head

# Operations
docker-compose -f docker-compose.prod.yml ps                      # Status
docker-compose -f docker-compose.prod.yml logs -f workers         # View logs
docker-compose -f docker-compose.prod.yml restart backend         # Restart service
docker-compose -f docker-compose.prod.yml down                    # Stop all
curl http://localhost:8000/health                                 # Health check

# Database
docker exec newhires-db pg_dump -U newhires newhires > backup.sql  # Backup
docker exec -it newhires-db psql -U newhires -d newhires          # CLI access

# Scaling
docker-compose -f docker-compose.prod.yml up -d --scale workers=3  # Scale workers