Skip to Content
DevelopmentBackendEncryption Troubleshooting

Encryption Troubleshooting Guide

Quick Diagnostics

Check Encryption Status

# Verify DB_ENCRYPTION_KEY is set
echo $DB_ENCRYPTION_KEY
 
# Test encryption/decryption
python -c "from rhesis.backend.app.utils.encryption import encrypt, decrypt; print('✓ Working')"

Verify Database Encryption

-- Count encrypted vs plaintext values (plaintext_count should be 0 after migration)
SELECT
    'endpoint' as table_name,
    COUNT(*) FILTER (WHERE auth_token LIKE 'gAAAAA%') as encrypted,
    COUNT(*) FILTER (WHERE auth_token IS NOT NULL AND NOT auth_token LIKE 'gAAAAA%') as plaintext
FROM endpoint WHERE auth_token IS NOT NULL
UNION ALL
SELECT 'model',
    COUNT(*) FILTER (WHERE key LIKE 'gAAAAA%'),
    COUNT(*) FILTER (WHERE key IS NOT NULL AND NOT key LIKE 'gAAAAA%')
FROM model WHERE key IS NOT NULL
UNION ALL
SELECT 'token',
    COUNT(*) FILTER (WHERE token LIKE 'gAAAAA%'),
    COUNT(*) FILTER (WHERE token IS NOT NULL AND NOT token LIKE 'gAAAAA%')
FROM token WHERE token IS NOT NULL;

Common Issues

Issue: “DB_ENCRYPTION_KEY environment variable is not set”

Solution:

# Generate key
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
 
# Set in environment
export DB_ENCRYPTION_KEY="<generated-key>"
 
# For production: Update Kubernetes secret or GCP Secret Manager

Issue: “Invalid encrypted data or wrong encryption key”

Cause: Using different encryption key than when data was encrypted

Solution:

  • Verify same key across all instances: echo $DB_ENCRYPTION_KEY | md5sum
  • Check if key was recently changed - restore correct key from backup
  • Ensure Kubernetes secrets match across environments

Issue: Endpoint/Model API Calls Failing

Cause: Token not decrypting properly

Diagnosis:

# Check if value is encrypted in DB
from rhesis.backend.app.utils.encryption import is_encrypted
print(is_encrypted(endpoint.auth_token))  # Should be False (auto-decrypted by ORM)

Solution:

  • Verify encryption key is correct
  • Check logs for decryption errors: kubectl logs -l app=rhesis-backend | grep -i decrypt
  • Re-save the record to re-encrypt with current key

Issue: Migration Failed Partway Through

Solution:

  • Migration is idempotent - safe to re-run: alembic upgrade head
  • Already encrypted values will be skipped
  • Check database connection and timeout settings

Issue: Performance Degradation

Expected: ~1-2ms overhead per encrypt/decrypt operation

Check:

  • Monitor CPU usage: kubectl top pods -l app=rhesis-backend
  • Verify connection pool settings (pool_size=10, max_overflow=20)
  • Check for N+1 query patterns

Issue: “Found unencrypted value” Warnings

Solution:

  • Re-run migration: alembic upgrade head
  • Or manually re-save affected records to trigger encryption

Debug Script

#!/usr/bin/env python3
"""Quick encryption status check"""
from rhesis.backend.app.database import SessionLocal
from rhesis.backend.app.models.endpoint import Endpoint
from rhesis.backend.app.models.model import Model
from rhesis.backend.app.models.token import Token
from rhesis.backend.app.utils.encryption import is_encrypted
 
db = SessionLocal()
 
for table, model, field in [
    ("endpoint", Endpoint, "auth_token"),
    ("model", Model, "key"),
    ("token", Token, "token")
]:
    records = db.query(model).all()
    encrypted = sum(1 for r in records if getattr(r, field, None) and is_encrypted(getattr(r, field)))
    total = sum(1 for r in records if getattr(r, field, None))
    print(f"{table}: {encrypted}/{total} encrypted")

Verification Queries

-- After migration, all should return 0
SELECT COUNT(*) FROM endpoint
WHERE (auth_token IS NOT NULL AND NOT auth_token LIKE 'gAAAAA%')
   OR (client_secret IS NOT NULL AND NOT client_secret LIKE 'gAAAAA%')
   OR (last_token IS NOT NULL AND NOT last_token LIKE 'gAAAAA%');
 
SELECT COUNT(*) FROM model WHERE key IS NOT NULL AND NOT key LIKE 'gAAAAA%';
SELECT COUNT(*) FROM token WHERE token IS NOT NULL AND NOT token LIKE 'gAAAAA%';

Getting Help

Log Collection

# Collect encryption-related logs
kubectl logs -l app=rhesis-backend -n rhesis-prod --since=1h | grep -i "encrypt\|decrypt" > logs.txt

Support

Database Field EncryptionDevelopment Workflow