GKE Worker Troubleshooting Guide

This guide covers troubleshooting Celery workers running in Google Kubernetes Engine (GKE), including using the built-in debugging tools.

Quick Start: Connect to Your Cluster

1. Find Your Cluster

code.txt
gcloud container clusters list --format="table(name,location,status)"

2. Get Credentials

code.txt
gcloud container clusters get-credentials <cluster-name> --region=<region>

3. Install kubectl (if needed)

code.txt
sudo apt-get update
sudo apt-get install -y kubectl google-cloud-cli-gke-gcloud-auth-plugin

Health Check Endpoints

The worker includes several debugging endpoints:

Endpoint	Purpose	Use Case
`/ping`	Basic connectivity	Quick server test
`/health/basic`	Server health (no dependencies)	Readiness probe
`/health`	Lightweight health (Celery + Redis, no worker ping)	Liveness probe
`/debug`	Comprehensive system info	General debugging
`/debug/env`	Environment variables (sanitized)	Config issues
`/debug/redis`	Redis connectivity details	Connection problems
`/debug/detailed`	Slow health check with worker ping	Deep troubleshooting

Worker Registration Checking

To verify workers are registered with the broker from inside a pod, run the check_workers.py script from Troubleshooting:

code.txt
kubectl exec -it <pod-name> -n <namespace> -- python check_workers.py

Cluster Management Commands

Scale workers down (for debugging):

code.txt
kubectl scale deployment rhesis-worker --replicas=0 -n <namespace>

Scale workers back up:

code.txt
kubectl scale deployment rhesis-worker --replicas=2 -n <namespace>

Check current replica count:

code.txt
kubectl get deployment rhesis-worker -n <namespace>

Common Troubleshooting Commands

Check Pod Status

code.txt
kubectl get pods -n <namespace>

Expected Output:

code.txt
NAME                             READY   STATUS    RESTARTS   AGE
rhesis-worker-6d9bcd9c6f-6bxk8   2/2     Running   0          5m
rhesis-worker-6d9bcd9c6f-9kqwz   2/2     Running   0          3m

Problem Indicators:

1/2 Ready: Worker container failing, cloudsql-proxy working
0/2 Ready: Both containers failing
CrashLoopBackOff: Container repeatedly failing
High restart count: Ongoing issues

Check Pod Events

code.txt
kubectl describe pod <pod-name> -n <namespace>

Look for events section at the bottom:

Unhealthy: Health check failures
Failed: Container start failures
Killing: Pod being terminated

Test Basic Connectivity

code.txt
kubectl exec -it <pod-name> -n <namespace> -- curl http://localhost:8080/ping

Expected: pong

If this fails:

Health server not starting
Port 8080 not listening
Container networking issues

Test Health Endpoints

code.txt
# Basic health (no dependencies)
kubectl exec -it <pod-name> -n <namespace> -- curl http://localhost:8080/health/basic

# Full health (includes Celery)
kubectl exec -it <pod-name> -n <namespace> -- curl -m 10 http://localhost:8080/health

Get Debug Information

code.txt
# Comprehensive debug info
kubectl exec -it <pod-name> -n <namespace> -- curl http://localhost:8080/debug | jq

# Redis-specific debugging
kubectl exec -it <pod-name> -n <namespace> -- curl http://localhost:8080/debug/redis | jq

# Environment variables (sanitized)
kubectl exec -it <pod-name> -n <namespace> -- curl http://localhost:8080/debug/env | jq

# Detailed health check with worker ping (may be slow)
kubectl exec -it <pod-name> -n <namespace> -- curl -m 15 http://localhost:8080/debug/detailed | jq

Common Issues and Solutions

1. Pods Stuck at 1/2 Ready

Symptoms:

code.txt
NAME                             READY   STATUS    RESTARTS   AGE
rhesis-worker-586659994f-lldfn   1/2     Running   167        13h

Diagnosis:

code.txt
kubectl exec -it <pod-name> -n <namespace> -- curl http://localhost:8080/debug

Common Causes:

A. Redis Connection Issues

code.txt
{
  "redis_connectivity": "connection_failed",
  "environment": {
    "tls_detected": true,
    "broker_url_type": "rediss://"
  }
}

Solutions:

Check Redis URL format: rediss:// for TLS, redis:// for standard
Verify SSL parameters: ssl_cert_reqs=CERT_NONE
Check network policies allowing outbound connections
Verify Redis service is accessible from GKE

B. Health Check Timeouts

code.txt
{
  "celery_status": {
    "worker_state": "importable"
  },
  "redis_connectivity": "timeout"
}

The /health endpoint runs a lightweight check that does not ping workers, so timeouts there usually point to Redis rather than worker startup.

Solutions:

Check Redis connectivity specifically: curl localhost:8080/debug/redis
Use the detailed health check to test worker ping: curl localhost:8080/debug/detailed

C. Environment Configuration

code.txt
kubectl exec -it <pod-name> -n <namespace> -- curl http://localhost:8080/debug/env

Check for:

Missing environment variables
Incorrect secret references
Malformed URLs

2. CrashLoopBackOff

Diagnosis:

code.txt
kubectl logs <pod-name> -n <namespace> --previous

Common Causes:

A. Import Errors

code.txt
❌ Failed to import Celery app: No module named 'rhesis.backend.worker'

Solutions:

Check PYTHONPATH in deployment
Verify Docker image build process
Ensure all dependencies installed

B. Connection Failures

code.txt
❌ Broker connection failed: [SSL: CERTIFICATE_VERIFY_FAILED]

Solutions:

Check SSL certificate configuration
Verify ssl_cert_reqs=CERT_NONE parameter
Test Redis connectivity outside GKE

3. High Memory Usage

Diagnosis:

code.txt
kubectl top pods -n <namespace>
kubectl exec -it <pod-name> -n <namespace> -- free -h

Solutions:

Adjust CELERY_WORKER_MAX_TASKS_PER_CHILD
Increase memory limits in deployment
Monitor for memory leaks in tasks

4. Task Processing Issues

Diagnosis:

code.txt
# Check if worker is receiving tasks
kubectl logs <pod-name> -n <namespace> | grep "Received task"

# Check worker stats
kubectl exec -it <pod-name> -n <namespace> -- python -c "from rhesis.backend.worker import app; print(app.control.inspect().stats())"

Advanced Debugging

Interactive Shell Access

code.txt
kubectl exec -it <pod-name> -n <namespace> -- bash

From inside the container:

code.txt
# Test Redis connection manually
python -c "
import redis
import os
r = redis.Redis.from_url(os.getenv('BROKER_URL'))
print(r.ping())
"

# Test Celery import
python -c "
from rhesis.backend.worker import app
print(f'Tasks: {len(app.tasks)}')
print(f'Broker: {app.conf.broker_url}')
"

# Check network connectivity
nslookup <redis-hostname>
telnet <redis-hostname> 6378

Monitor Logs in Real-Time

code.txt
# Follow logs for all worker pods
kubectl logs -f deployment/rhesis-worker -n <namespace>

# Follow logs for specific container
kubectl logs -f <pod-name> -c worker -n <namespace>

Network Debugging

code.txt
# Check network policies
kubectl get networkpolicies -n <namespace>

# Test external connectivity
kubectl exec -it <pod-name> -n <namespace> -- nslookup google.com

# Check firewall rules (if applicable)
gcloud compute firewall-rules list --filter="direction=EGRESS"

Performance Monitoring

Resource Usage

code.txt
# Pod resource usage
kubectl top pods -n <namespace>

# Node resource usage
kubectl top nodes

# Detailed resource info
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 "Requests|Limits"

Health Check Performance

code.txt
# Time health check responses
kubectl exec -it <pod-name> -n <namespace> -- time curl http://localhost:8080/health

# Monitor health check frequency
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

Preventive Measures

1. Proper Resource Limits

code.txt
resources:
requests:
    memory: "1Gi"
    cpu: "500m"
limits:
    memory: "2Gi"
    cpu: "1000m"

2. Appropriate Health Check Timeouts

code.txt
livenessProbe:
httpGet:
    path: /health
    port: 8080
initialDelaySeconds: 120  # Allow for TLS startup
timeoutSeconds: 20        # Account for Redis delays
periodSeconds: 45
failureThreshold: 3

readinessProbe:
httpGet:
    path: /health/basic      # Fast, no dependencies
    port: 8080
initialDelaySeconds: 15
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3

3. Monitoring and Alerting

code.txt
# Set up monitoring for:
# - Pod restart frequency
# - Health check failure rates
# - Redis connection timeouts
# - Memory usage trends

Emergency Procedures

Force Pod Restart

code.txt
kubectl delete pod <pod-name> -n <namespace>

Scale Down/Up

code.txt
kubectl scale deployment rhesis-worker --replicas=0 -n <namespace>
kubectl scale deployment rhesis-worker --replicas=2 -n <namespace>

Emergency Debugging

code.txt
# Create debug pod with same network
kubectl run debug-pod --image=gcr.io/PROJECT_ID/rhesis-worker:latest --namespace=<namespace> --rm -it -- bash

# Test from debug pod
curl http://rhesis-worker-service:8080/debug

Getting Help

When reporting issues, include:

Cluster Information:
```
kubectl version
kubectl get nodes
```

Pod Status:


kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>

Debug Output:


kubectl exec -it <pod-name> -n <namespace> -- \
  curl http://localhost:8080/debug | jq

Recent Logs:


kubectl logs <pod-name> -n <namespace> --tail=100

Configuration:


kubectl get deployment rhesis-worker -n <namespace> -o yaml