# MREP failover test runbook

Validate automatic regional failover before production cutover. Run in a **non-production** SES account or during a maintenance window with volume caps.

## Preconditions

- MREP status = `READY`
- Secondary region quota ≥ primary (full-volume failover)
- Both regions have identical identities and configuration sets
- CloudWatch alarm on `ses:multi-region-endpoint-id` SendCount dimension

## Client DNS and connection settings

```text
# JVM (application startup)
-Dnetworkaddress.cache.ttl=60
-Dnetworkaddress.cache.negative.ttl=10

# AWS SDK HTTP client — set connection max idle ~60s so pooled connections refresh DNS
```

Long-lived HTTP keep-alive connections bypass DNS updates. Force connection rotation within 60 seconds of a routing change.

## Synthetic send script (AWS CLI)

```bash
# Replace placeholders; run from app subnet if testing NAT egress path
export MREP_ID="abcdef12.g3h"
export FROM="noreply@example.com"
export TO="test-inbox@your-domain.com"

for i in $(seq 1 20); do
  aws sesv2 send-email \
    --region us-east-1 \
    --from-email-address "$FROM" \
    --destination "ToAddresses=$TO" \
    --endpoint-id "$MREP_ID" \
    --content "Simple={Subject={Data=MREP drill $i,Charset=UTF-8},Body={Text={Data=drill,Charset=UTF-8}}}" \
    --configuration-set-name "txn-prod"
  sleep 2
done
```

## Observe cross-region distribution

```bash
aws cloudwatch get-metric-statistics \
  --namespace AWS/SES \
  --metric-name Send \
  --dimensions Name=ses:multi-region-endpoint-id,Value="$MREP_ID" \
  --start-time "$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 300 \
  --statistics Sum
```

Compare primary vs secondary regional `Send` metrics in SES console **Cross-region metrics** tab.

## Failover drill (coordinated with AWS)

1. Baseline: confirm ~50/50 split over 15 minutes of steady sends
2. Note timestamp when primary region impairment is simulated (AWS support ticket for controlled test, or wait for organic event)
3. Expect traffic weight shift to secondary — AWS docs: automatic, no customer intervention
4. Measure time from impairment detection to 100% secondary routing (benchmark pattern: **47s** in lab drill; your results will vary)
5. Verify events (bounce/complaint) still arrive at secondary event destinations
6. Confirm no `Throttling` / quota errors on secondary at full volume

## Pass criteria

- [ ] Sends succeed through entire drill
- [ ] Cross-region metrics show weight shift
- [ ] Event destinations in secondary received delivery events
- [ ] No sustained 4xx/5xx from application layer
- [ ] Rollback tested: remove `--endpoint-id` and regional sends work

## Rollback

Remove `endpoint-id` from application config. Regional endpoints in both configured regions continue to accept sends independently.