​

Repo B Engine Cutover Runbook

​

Preconditions Checklist

• Parity Validation Complete
  • MTD validation passed for June, July, August, September 2025
  • MoM validation passed for Jul↔Jun, Aug↔Jul, Sep↔Aug
  • QTD validation passed for July, August, September 2025 (baseline = Jul 1)
  • All parity checks show [PASS] status
  • No critical diffs (counts exact, money within $0.05 tolerance)
• Shadow Datasets Populated
  • payroll_raw_shadow.stage1_snapshots contains data for June-September 2025
  • payroll_processed_shadow.stage3_snapshots contains data for June-September 2025
  • Row counts match production datasets for validation periods
• Shadow Views Created
  • payroll_analytics_shadow.ceo_snapshot view exists and queries successfully
  • payroll_analytics_shadow.top_businesses view exists and queries successfully
  • payroll_analytics_shadow.top_agents view exists and queries successfully
  • payroll_analytics_shadow.business_growth_loss view exists and queries successfully
• Environment Configuration
  • Repo B upload scripts tested in shadow mode (BQ_SHADOW_MODE=1)
  • Repo B can successfully write to shadow datasets
  • No production dataset writes occurred during shadow mode testing
• Backup & Rollback Plan
  • Production datasets backed up (or verified recent backups exist)
  • Rollback procedure documented and tested
  • Team notified of cutover window

​

Cutover Steps

​

Step 1: Disable Repo A Writers

• payroll_raw.stage1_snapshots
• payroll_processed.stage3_snapshots

-- Check last write timestamp for Repo A
SELECT MAX(created_at) as last_repo_a_write
FROM `payroll-bi-gauntlet.payroll_raw.stage1_snapshots`
WHERE source_file LIKE '%repo_a%' OR pipeline_version LIKE '%repo_a%'

​

Step 2: Enable Repo B Production Writers

1. Set environment variable:
   Copy

   Ask AI

   export BQ_SHADOW_MODE=0
   
   Or ensure BQ_SHADOW_MODE is unset/not equal to “1”.
2. Verify configuration:
   Copy

   Ask AI

   # In repo_b/upload_to_bq.py, resolve_dataset() should return prod datasets
   from repo_b.upload_to_bq import resolve_dataset
   assert resolve_dataset("payroll_raw") == "payroll_raw"
   assert resolve_dataset("payroll_processed") == "payroll_processed"
   
3. Run Repo B upload for current period:
   Copy

   Ask AI

   python -m repo_b.upload_to_bq --period 2025-10  # Example: current month
   

-- Check Repo B writes to production
SELECT MAX(created_at) as last_repo_b_write, COUNT(*) as repo_b_rows
FROM `payroll-bi-gauntlet.payroll_raw.stage1_snapshots`
WHERE source_file = 'repo_b_upload' OR pipeline_version = 'repo_b_v1'
  AND period_label >= DATE('2025-10-01')  -- Current period

​

Step 3: Post-Cutover Verification

# This validates Repo B shadow data matches Repo A production data
# Note: Validator always compares prod vs shadow datasets (no mode switch)
python tests/verify_dashboard_parity_shadow.py --tenant-id creative_benefit_strategies

1. CEO Dashboard:
   • Total businesses matches expected value
   • Total employees matches expected value
   • Gross payout matches expected value
   • Chargebacks match expected value
   • Net payout matches expected value
   • Agent commissions match expected value
2. Top Businesses Widget:
   • Top 10 businesses match expected list
   • Gross payout values match expected amounts
   • Employee counts match expected values
3. Top Agents Widget:
   • Top 10 agents match expected list
   • Commission totals match expected amounts
   • Employee counts match expected values
4. Growth/Loss Widget:
   • MoM changes match expected values
   • New/lost business counts match expected values
   • Employee deltas match expected values

-- Compare Repo B production vs shadow for current period (should match)
SELECT 
  'prod' as source,
  COUNT(*) as stage1_rows,
  SUM(total) as gross_payout
FROM `payroll-bi-gauntlet.payroll_raw.stage1_snapshots`
WHERE period_label = DATE('2025-10-01')  -- Current period
  AND tenant_id = 'creative_benefit_strategies'
  AND (source_file = 'repo_b_upload' OR pipeline_version = 'repo_b_v1')

UNION ALL

SELECT 
  'shadow' as source,
  COUNT(*) as stage1_rows,
  SUM(total) as gross_payout
FROM `payroll-bi-gauntlet.payroll_raw_shadow.stage1_snapshots`
WHERE period_label = DATE('2025-10-01')  -- Current period
  AND tenant_id = 'creative_benefit_strategies'

​

Rollback Procedure

​

Step 1: Re-enable Repo A Writers

SELECT MAX(created_at) as last_repo_a_write
FROM `payroll-bi-gauntlet.payroll_raw.stage1_snapshots`
WHERE source_file LIKE '%repo_a%' OR pipeline_version LIKE '%repo_a%'

​

Step 2: Disable Repo B Production Writers

export BQ_SHADOW_MODE=1  # Force shadow mode

SELECT MAX(created_at) as last_repo_b_write
FROM `payroll-bi-gauntlet.payroll_raw.stage1_snapshots`
WHERE source_file = 'repo_b_upload' OR pipeline_version = 'repo_b_v1'

​

Step 3: Clean Up Repo B Production Data (Optional)

-- Delete Repo B production data for affected periods
DELETE FROM `payroll-bi-gauntlet.payroll_raw.stage1_snapshots`
WHERE source_file = 'repo_b_upload' OR pipeline_version = 'repo_b_v1'
  AND period_label >= DATE('2025-10-01')  -- Affected periods

DELETE FROM `payroll-bi-gauntlet.payroll_processed.stage3_snapshots`
WHERE source_file = 'repo_b_upload' OR pipeline_version = 'repo_b_v1'
  AND period_label >= DATE('2025-10-01')  -- Affected periods

SELECT COUNT(*) as remaining_repo_b_rows
FROM `payroll-bi-gauntlet.payroll_raw.stage1_snapshots`
WHERE source_file = 'repo_b_upload' OR pipeline_version = 'repo_b_v1'
  AND period_label >= DATE('2025-10-01')

​

Post-Cutover Monitoring

1. Dashboard Load Times:
   • CEO dashboard loads within expected time (< 5 seconds)
   • No timeout errors in dashboard API
2. Data Freshness:
   • New period data appears within expected time window
   • No gaps in period coverage
3. Data Quality:
   • No unexpected null values
   • Financial totals match expected ranges
   • Business/employee counts match expected ranges
4. Error Logs:
   • No BigQuery query errors
   • No upload failures
   • No schema mismatches

​

Emergency Contacts

• On-Call Engineer: Contact Data Engineering team
• Data Engineering Lead: Contact Data Engineering team
• Product Owner: Contact Product team

​

Notes

• Shadow datasets (payroll_raw_shadow, payroll_processed_shadow, payroll_analytics_shadow) will remain for historical validation.
• Repo B shadow mode can be re-enabled at any time by setting BQ_SHADOW_MODE=1.
• Production view logic is unchanged; only data source (writers) changes.