Most Kubernetes CI/CD pipelines look fine in staging and explode in production. After building and debugging pipelines across dozens of production clusters, these are the five mistakes I see engineers make repeatedly — and how to fix each one.

---

1. Deploying with kubectl apply instead of GitOps

If your pipeline runs kubectl apply -f manifests/ directly, you have no source of truth. When someone hotfixes production at 2 AM, your repo and your cluster diverge silently.

Fix: Use a GitOps operator like ArgoCD or Flux. Your Git repo becomes the single source of truth, and drift is detected automatically.

# ArgoCD Application manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/org/k8s-manifests.git
    targetRevision: main
    path: envs/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

2. No Readiness Gates on Deployments

Your pipeline says "deploy succeeded" because the rollout completed — but the new pods are crashing on their first real request. Without proper readiness probes, Kubernetes happily routes traffic to broken pods.

Fix: Always define readiness probes that actually test your application's critical path.

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

And in your pipeline, wait for the rollout to actually stabilize:

kubectl rollout status deployment/my-app -n production --timeout=300s

3. Hardcoding Secrets in Pipeline YAML

I still see DOCKER_PASSWORD and database credentials stored as plain-text environment variables in GitHub Actions workflows. One leaked log and you're compromised.

Fix: Use Kubernetes-native secrets management. External Secrets Operator pulls secrets from Vault, AWS Secrets Manager, or GCP Secret Manager directly into your cluster.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/production/db
        property: password

4. No Canary or Progressive Rollout

Shipping 100% of traffic to a new version instantly is gambling. One bad config change takes down your entire service.

Fix: Use Argo Rollouts for canary deployments. Route 10% of traffic first, watch error rates, then promote.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 30
        - pause: { duration: 5m }
        - setWeight: 60
        - pause: { duration: 5m }
      canaryMetadata:
        labels:
          role: canary

5. No Pipeline Observability

Your pipeline fails. The error says "exit code 1." You spend 45 minutes reading logs across three systems to find the root cause.

Fix: Instrument your pipeline. Push build duration, test pass rate, deployment frequency, and failure rate to Prometheus. Set up Grafana dashboards and alerts.

# Prometheus ServiceMonitor for pipeline metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cicd-metrics
spec:
  selector:
    matchLabels:
      app: pipeline-exporter
  endpoints:
    - port: metrics
      interval: 30s

Track the DORA metrics: deployment frequency, lead time, change failure rate, and mean time to recovery. These tell you if your pipeline is actually improving your engineering velocity.

---

These five patterns are covered in depth — with full hands-on labs — in the PipelineOps Academy course. Every lesson is built around real production scenarios, not toy clusters.

Stop debugging pipelines at 3 AM. Build them right the first time.