​

AI Chatbot 403-for-Self Bug - Debug Guide

​

Symptom

• Question: “how much did kenny young make this month”
• Endpoint: POST https://payroll-backend-prod-evndxpcirq-uc.a.run.app/api/v1/ai/query-public
• Response: 403 Forbidden
• Expected: 200 OK with commission data

• Request includes valid Authorization: Bearer <token> header
• Token is valid (not expired)
• User role: agent
• Requested target: “Kenny Young” (self)

​

Hypotheses

​

Hypothesis A: agent_id Enrichment Missing/None

• Cloud Run logs for [RBAC-DEBUG] markers showing agent_id=None
• Verify get_user_by_email() is working
• Verify dim_users table has agent_id populated for Kenny’s email

• api/routes/ai_query.py lines 3325-3337: JWT enrichment with get_user_by_email()

• Ensure dim_users table has agent_id populated
• Long-term: Embed agent_id in JWT at login to avoid BigQuery dependency

​

Hypothesis B: Name→ID Resolution Mismatch

• Cloud Run logs for [RBAC] markers showing:
  • requested_target: “Kenny Young”
  • target_agent_id: resolved agent_id
  • user_agent_id: current_user.agent_id
• Compare resolved target_agent_id vs user_agent_id (should match)

• api/utils/rbac.py lines 338-345: resolve_agent_name_to_id() call
• api/utils/rbac.py lines 347-350: Self-check comparison

• Ensure name resolution uses exact match (case-insensitive, trimmed)
• Verify dim_agent_hierarchy has single active row for Kenny’s agent_id

​

Hypothesis C: Tenant ID Mismatch

• Cloud Run logs for [RBAC-DEBUG] markers showing:
  • User tenant_id: from JWT
  • Requested tenant_id: from request context
• Compare tenant_ids (should match)

• api/utils/rbac.py lines 316-321: Tenant isolation check

• Ensure JWT contains correct tenant_id
• Ensure request context uses same tenant_id as JWT

​

Hypothesis D: requested_target Extraction Issue

• Cloud Run logs for [RBAC-DEBUG] markers showing:
  • requested_target: extracted from question
  • User agent_name: from JWT/enrichment
• Compare extracted name vs user’s agent_name

• api/routes/ai_query.py intent resolution: Extracts requested_target from question
• api/utils/rbac.py lines 332-336: Defaults to self if no target specified

• Ensure intent resolution extracts full name (not partial)
• Or: Default to self if extracted name matches user’s agent_name (fuzzy match)

​

Required Logs/Values to Capture

​

1. Request Start Logs

[RBAC-DEBUG] User email: kenny@example.com
[RBAC-DEBUG] User role: agent
[RBAC-DEBUG] User agent_id: <should be non-null>
[RBAC-DEBUG] User agent_name: Kenny Young
[RBAC-DEBUG] Tenant ID: creative_benefit_strategies
[RBAC-DEBUG] Is demo: False
[RBAC-DEBUG] Question: how much did kenny young make this month...
[RBAC-DEBUG] Effective scope: self + downline (agent_id: <agent_id>)

• User agent_id: Should be non-null (if null → Hypothesis A)
• User agent_name: Should match extracted target (if different → Hypothesis D)
• Tenant ID: Should match request context (if different → Hypothesis C)

​

2. Intent Resolution Logs

[RBAC-DEBUG] AGENT_COMMISSION - requested_target: Kenny Young
[RBAC-DEBUG] AGENT_COMMISSION - authorized_agent_id: <should be non-null or DENIED>

• requested_target: Extracted agent name from question
• authorized_agent_id: Result of authorize_target_agent_id() (should be non-null for self)

​

3. RBAC Authorization Logs

[RBAC] Agent role: no target specified, allowing self: <agent_id>
# OR
[RBAC] Agent role: authorized self access: <agent_id>
# OR
[RBAC] Agent role: denied access - <agent_id> not in downline of <user_agent_id>

• Self-check result: Should allow self access
• Downline check result: Should allow if target is self

​

4. Name Resolution Logs

[RBAC] Resolving agent name to ID: name=Kenny Young, tenant=creative_benefit_strategies
[RBAC] Resolved agent_id: <agent_id>

• Resolved agent_id: Should match user_agent_id (if different → Hypothesis B)

​

Minimal Reproduction Checklist

​

Step 1: Reproduce the Bug

1. Login as Kenny Young (agent role)
2. Open AI Assistant in dashboard
3. Send Query: “how much did kenny young make this month”
4. Observe Response: Should get 403 Forbidden (bug)

​

Step 2: Capture Logs

1. Open Cloud Run Logs:
   Copy

   Ask AI

   gcloud logging read \
     "resource.type=cloud_run_revision AND \
      resource.labels.service_name=payroll-backend-prod AND \
      (textPayload=~'RBAC-DEBUG' OR textPayload=~'RBAC.*Kenny')" \
     --project payroll-bi-gauntlet \
     --limit 50 \
     --format=json
   
2. Extract Key Values:
   • current_user.agent_id
   • current_user.agent_name
   • current_user.tenant_id
   • requested_target (extracted from question)
   • target_agent_id (resolved from name)
   • authorized_agent_id (result of authorization)

​

Step 3: Compare Values

1. Compare agent_ids:
   • current_user.agent_id vs target_agent_id (should match)
   • If different → Hypothesis B
2. Compare tenant_ids:
   • current_user.tenant_id vs request context tenant_id (should match)
   • If different → Hypothesis C
3. Check agent_id enrichment:
   • current_user.agent_id is null → Hypothesis A
   • current_user.agent_id is non-null → Check other hypotheses
4. Check name extraction:
   • requested_target vs current_user.agent_name (should match or be similar)
   • If very different → Hypothesis D

​

Step 4: Verify Data

1. Check dim_users table:
   Copy

   Ask AI

   SELECT email, agent_id, agent_name
   FROM `payroll-bi-gauntlet.payroll_analytics.dim_users`
   WHERE email = 'kenny@example.com'  -- Replace with actual email
   
   • Verify agent_id is populated
   • Verify agent_name matches expected
2. Check dim_agent_hierarchy table:
   Copy

   Ask AI

   SELECT agent_id, agent_name, tenant_id, is_active, is_deleted
   FROM `payroll-bi-gauntlet.payroll_analytics.dim_agent_hierarchy`
   WHERE agent_name = 'Kenny Young'
     AND tenant_id = 'creative_benefit_strategies'
     AND is_active = TRUE
     AND is_deleted = FALSE
   
   • Verify single active row exists
   • Verify agent_id matches dim_users.agent_id

​

Debugging Commands

​

1. Query Cloud Run Logs

# Get all RBAC-DEBUG logs for recent requests
gcloud logging read \
  "resource.type=cloud_run_revision AND \
   resource.labels.service_name=payroll-backend-prod AND \
   textPayload=~'RBAC-DEBUG'" \
  --project payroll-bi-gauntlet \
  --limit 100 \
  --format="value(textPayload)" \
  --order=desc

# Get specific agent logs
gcloud logging read \
  "resource.type=cloud_run_revision AND \
   resource.labels.service_name=payroll-backend-prod AND \
   (textPayload=~'Kenny' OR textPayload=~'kenny')" \
  --project payroll-bi-gauntlet \
  --limit 50 \
  --format="value(textPayload)" \
  --order=desc

​

2. Test Name Resolution

# In Python REPL or test script
from api.utils.rbac import resolve_agent_name_to_id

tenant_id = "creative_benefit_strategies"
agent_name = "Kenny Young"

agent_id = resolve_agent_name_to_id(agent_name, tenant_id)
print(f"Resolved agent_id: {agent_id}")

​

3. Test Downline Check

# In Python REPL or test script
from api.utils.rbac import is_in_downline

tenant_id = "creative_benefit_strategies"
root_agent_id = "<kenny_agent_id>"  # From dim_users
requested_agent_id = "<kenny_agent_id>"  # Same (self)

is_allowed = is_in_downline(tenant_id, root_agent_id, requested_agent_id)
print(f"Self access allowed: {is_allowed}")  # Should be True

​

Expected Fixes

​

If Hypothesis A (agent_id enrichment missing)

• api/routes/auth.py (or login endpoint): Add agent_id to JWT payload
• Or: Fix get_user_by_email() to return agent_id

​

If Hypothesis B (name→id resolution mismatch)

• api/utils/rbac.py resolve_agent_name_to_id(): Verify exact match logic
• Or: Use fuzzy matching to handle name variations

​

If Hypothesis C (tenant_id mismatch)

• api/routes/ai_query.py: Verify tenant_id extraction from JWT matches request context

​

If Hypothesis D (requested_target extraction issue)

• api/utils/rbac.py authorize_target_agent_id(): Add fuzzy name matching for self-check

​

Related Documentation

• docs/AI_CHATBOT_INCIDENT_2025_12_24.md - Complete incident summary
• docs/AI_CHATBOT_SECURITY.md - Security features and API contract
• docs/DEPLOYMENT_RUNBOOK.md - Deployment procedures
• api/utils/rbac.py - RBAC implementation
• api/routes/ai_query.py - AI query endpoint implementation