First-Time Kubernetes Deployment in a New Cloud

0) Problem Restatement

NVIDIA asked: you need to deploy your services on Kubernetes in a new cloud account for the first time. Explain everything from zero: account bootstrapping, networking, the cluster, security, deployments, observability, and how you'd keep it safe and repeatable.

Kubernetes (K8s) is a system that runs containers across a group of machines, restarts them if they fail, and scales them.

1) Step 1: Account and Foundations

• Account structure: separate accounts or projects for dev, staging and prod, so mistakes in dev can't touch prod. Use an organization with central billing and guardrails (policies that block public buckets, restrict regions).
• Identity: SSO for humans, no long-lived personal keys, and least-privilege roles.
• Infrastructure as Code (Terraform or Pulumi) from day one: everything below is code, reviewed and versioned, so environments are reproducible.

2) Step 2: Networking

• A VPC per environment, with private subnets for nodes (no public IPs) across 3 availability zones, and public subnets only for load balancers and NAT gateways.
• Plan IP ranges carefully (pods can use many IPs), and avoid overlap with offices or other VPCs you may connect later.
• Egress via NAT, and private endpoints for cloud services (storage, registry) so traffic doesn't cross the internet.

3) Step 3: The Cluster

• Use the cloud's managed Kubernetes (EKS/GKE/AKS), where the control plane is run for you.
• Node pools: a general pool (on-demand) for system and critical services, a spot/preemptible pool for batch work, and a GPU pool (with the NVIDIA device plugin and drivers) using taints, so only GPU workloads land there.
• Cluster autoscaler (or Karpenter) to add and remove nodes, and the Horizontal Pod Autoscaler for pods.
• Essential add-ons: ingress controller, cert-manager (TLS), external-dns, a metrics server, and a CSI storage driver.

flowchart LR
    DEV["Git push"] --> CI["CI - build, test, scan image"]
    CI --> REG[("Container registry")]
    CI --> CD["CD - GitOps (Argo CD)"]
    CD --> K8S["Kubernetes cluster - private subnets, 3 AZs"]
    LB["Cloud load balancer + ingress"] --> K8S
    K8S --> OBS["Observability - metrics, logs, traces"]
    SEC["Secrets manager"] --> K8S
    IAC["Terraform - VPC, cluster, IAM"] --> K8S

4) Step 4: Security

• Workload identity: pods get cloud permissions through service-account-to-IAM-role mapping, never node-wide credentials.
• Secrets in a secrets manager (synced into K8s), not in Git.
• RBAC in the cluster: teams can deploy only to their namespaces.
• Network policies: default-deny between namespaces, and allow only what's needed.
• Image security: scan images in CI, allow only signed images from our registry, and run as non-root with read-only file systems where possible.

5) Step 5: Deploying Services

• Containerize each service, write Helm charts or Kustomize manifests: a Deployment with resource requests and limits, readiness and liveness probes, a Service, and an Ingress.
• CI/CD: CI builds and tests the image and pushes it to the registry. CD (e.g., Argo CD, GitOps) applies the manifests from Git, so the cluster always matches the repo.
• Safe rollouts: rolling updates with health checks, and canary releases (Argo Rollouts) that watch error rates and roll back automatically.
• PodDisruptionBudgets and multiple replicas across zones for availability.

6) Step 6: Observability and Operations

• Metrics (Prometheus + Grafana), logs (Fluent Bit → a log store), traces (OpenTelemetry), and alerts on SLOs.
• Backups of cluster state and persistent volumes (Velero), and a disaster recovery plan (recreate from Terraform + GitOps in another region).
• Cost controls: resource requests right-sized, spot pools for batch, autoscaling, budgets and alerts, and namespace cost reports.

7) Wrap-Up

Start with separate accounts per environment, SSO and Infrastructure as Code, and a multi-AZ VPC with private subnets. Create a managed cluster with general, spot and GPU node pools plus autoscaling and core add-ons. Lock it down with workload identity, a secrets manager, RBAC, network policies and signed images. Ship services through CI plus GitOps CD with probes, canaries and disruption budgets, and finish with metrics, logs, traces, backups and cost controls.