Fixing a 20s Kubernetes API Lag in 15 Minutes with OpenTelemetry

Our Kubernetes API was suddenly taking 20 seconds to respond, but traditional pod metrics showed healthy CPU and memory. I needed true observability to find the root cause without rewriting code or paying Datadog's $0.05 per custom metric surcharge. I chose SigNoz for a unified metrics, logs, and traces backend, and the OpenTelemetry Operator to auto-instrument our cluster. The Setup First, I installed SigNoz directly into our cluster using Helm: helm repo add signoz https://charts.signoz.io kubectl create ns platform helm --namespace platform install my-release signoz/signoz Next, to get data flowing without touching our Java source code, I installed the OpenTelemetry Operator and simply added this annotation to our application's deployment manifest: template: metadata: annotations: instrumentation.opentelemetry.io/inject-java: "true" Enter fullscreen mode Exit fullscreen mode The Fix Almost instantly, traces populated the SigNoz dashboard. The trace proved the database was responding in single-digit milliseconds. The delay was happening before the database call. By clicking from the trace straight into the correlated JVM metrics, I saw application threads were deadlocking waiting for asynchronous external responses. We patched the thread pool logic, and response times dropped back to milliseconds. Takeaway Auto-instrumentation via Kubernetes annotations is a superpower. Standardizing on OpenTelemetry saves weeks of engineering time and stops vendor lock-in. Don't guess why your app is slow—trace it.