Encryption Troubleshooting

Diagnosing and fixing problems with encrypted database fields. For how encryption works, see Database Field Encryption.

Quick checks

# Key is set and valid (backend won't start otherwise)
echo $DB_ENCRYPTION_KEY

# Helpers import cleanly
python -c "from rhesis.backend.app.utils.encryption import encrypt, decrypt; print('ok')"

Count encrypted vs. plaintext rows (plaintext should be 0 after the migration):

SELECT COUNT(*) FROM endpoint
WHERE (auth_token IS NOT NULL AND NOT auth_token LIKE 'gAAAAA%')
   OR (client_secret IS NOT NULL AND NOT client_secret LIKE 'gAAAAA%')
   OR (last_token IS NOT NULL AND NOT last_token LIKE 'gAAAAA%');

SELECT COUNT(*) FROM model WHERE key IS NOT NULL AND NOT key LIKE 'gAAAAA%';
SELECT COUNT(*) FROM token WHERE token IS NOT NULL AND NOT token LIKE 'gAAAAA%';

Common issues

”DB_ENCRYPTION_KEY Field required” at startup

The key is missing. It is required with no default, and a missing or invalid key stops the backend from starting rather than silently disabling encryption. Generate one, set it in the environment (or the Kubernetes secret / GCP Secret Manager in deployed environments), and restart:

export DB_ENCRYPTION_KEY="$(python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"

DecryptionError / “Invalid encrypted data or wrong encryption key”

Reads fail loudly when a value cannot be decrypted. Two causes:

• The value is still plaintext because the data migration never ran — run alembic upgrade head, or re-save the affected records to encrypt them.
• The value was encrypted with a different key. Confirm the same key across all instances (echo $DB_ENCRYPTION_KEY | md5sum) and restore the original key from backup if it was changed.

Endpoint or model calls failing on the stored token

The token is not decrypting. Confirm the key is correct, then check backend logs:

kubectl logs -l app=rhesis-backend | grep -i decrypt

Re-saving the record re-encrypts it with the current key. In application code the value is already decrypted by the ORM (is_encrypted(endpoint.auth_token) should be False).

Migration failed partway through

alembic upgrade head is idempotent — already-encrypted values are skipped — so re-run it. If it keeps failing, check the database connection and statement timeout.

Debug script

Reports how many rows are encrypted per field:

from rhesis.backend.app.database import SessionLocal
from rhesis.backend.app.models.endpoint import Endpoint
from rhesis.backend.app.models.model import Model
from rhesis.backend.app.models.token import Token
from rhesis.backend.app.utils.encryption import is_encrypted

db = SessionLocal()
for table, model, field in [
    ("endpoint", Endpoint, "auth_token"),
    ("model", Model, "key"),
    ("token", Token, "token"),
]:
    records = db.query(model).all()
    encrypted = sum(1 for r in records if getattr(r, field, None) and is_encrypted(getattr(r, field)))
    total = sum(1 for r in records if getattr(r, field, None))
    print(f"{table}: {encrypted}/{total} encrypted")

Note: reading through the ORM decrypts values, so is_encrypted on an ORM attribute reflects decrypted plaintext, not the stored ciphertext. Use the SQL checks above to inspect what is actually stored.

Getting help

Collect recent encryption-related logs and open a GitHub issue with the encryption label:

kubectl logs -l app=rhesis-backend -n rhesis-prod --since=1h | grep -i "encrypt\|decrypt" > logs.txt