kubernetes-conventions/comparison/stdlib-vs-kubernetes.md

Stdlib vs Kubernetes: Where K8s Does Things Differently

1. Concurrency: Channels vs. Condition Variables + Sets

Stdlib approach

Go stdlib idiom: communicate via channels. A worker pool typically uses a chan T for work items.

// Typical stdlib pattern:
jobs := make(chan Job, 100)
for i := 0; i < workers; i++ {
    go func() {
        for job := range jobs {
            process(job)
        }
    }()
}

Kubernetes approach

The workqueue uses sync.Cond + sets.Set instead of channels.

Source: staging/src/k8s.io/client-go/util/workqueue/queue.go (lines 190–300)

type Typed[t comparable] struct {
    queue      Queue[t]
    dirty      sets.Set[t]     // Items needing processing
    processing sets.Set[t]     // Items currently being processed
    cond       *sync.Cond      // Notification mechanism
}

Why K8s is different

Channels don't deduplicate. If you send the same key twice to a channel, it gets processed twice. K8s needs:

• Deduplication: same object modified 5 times → process once with latest state
• Re-entrant marking: if modified during processing, re-queue after Done()
• Inspection: can query queue length, processing count for metrics

A channel-based design would require a separate dedup layer anyway, losing its simplicity advantage.

---

2. Error Handling: Single Errors vs. Error Aggregation

Stdlib approach

Return a single error. Wrap with fmt.Errorf("...: %w", err).

Kubernetes approach

Source: staging/src/k8s.io/apimachinery/pkg/util/errors/errors.go (lines 34–100)

type Aggregate interface {
    error
    Errors() []error
    Is(error) bool
}

func NewAggregate(errlist []error) Aggregate {
    // Filters nils, deduplicates messages
}

Why K8s is different

A controller sync often does multiple operations (create 3 pods, update 2 services). You want to attempt all of them, not fail fast on the first error. Error aggregation collects all failures so the user sees the full picture.

Also: the Aggregate interface properly implements errors.Is() by checking if any contained error matches — which errors.Join didn't originally support well.

---

3. Retry: stdlib has nothing; K8s has structured retry

Stdlib approach

There's no retry utility in stdlib. You write your own loop.

Kubernetes approach

Source: staging/src/k8s.io/client-go/util/retry/util.go (lines 30–100)

var DefaultRetry = wait.Backoff{
    Steps:    5,
    Duration: 10 * time.Millisecond,
    Factor:   1.0,
    Jitter:   0.1,
}

func RetryOnConflict(backoff wait.Backoff, fn func() error) error {
    // Retries only on HTTP 409 Conflict
    return OnError(backoff, errors.IsConflict, fn)
}

Why K8s is different

In a distributed system, retries are the primary reliability mechanism. Stdlib doesn't provide them because stdlib targets single-machine programs. Kubernetes needs:

• Configurable backoff (steps, factor, jitter, cap)
• Condition-based retry (retry only on specific error types)
• Context-aware cancellation

---

4. Polling: time.Ticker vs. Contextual Loop With Crash Protection

Stdlib approach

ticker := time.NewTicker(interval)
defer ticker.Stop()
for range ticker.C {
    doWork()
}

Kubernetes approach

Source: staging/src/k8s.io/apimachinery/pkg/util/wait/backoff.go (lines 240–260), loop.go (lines 38–80)

func BackoffUntilWithContext(ctx context.Context, f func(ctx context.Context), backoff BackoffManager, sliding bool) {
    for {
        select {
        case <-ctx.Done():
            return
        default:
        }
        if !sliding { t = backoff.Backoff() }
        func() {
            defer runtime.HandleCrashWithContext(ctx)
            f(ctx)
        }()
        if sliding { t = backoff.Backoff() }
        // ... wait for timer or context cancellation
    }
}

Why K8s is different

• Crash protection: a panic in doWork() shouldn't kill the whole process
• Sliding vs non-sliding: controls whether interval includes execution time
• Context cancellation: allows clean shutdown
• Jitter: prevents thundering herd when many controllers sync at similar intervals
• Double-check for cancellation: Go's select is non-deterministic, so short timers can "win" against a cancelled context

---

5. Graceful Shutdown: http.Server.Shutdown vs. Multi-Phase Orchestration

Stdlib approach (net/http)

Source: /tmp/go-src/src/net/http/server.go (lines 3221–3260)

func (s *Server) Shutdown(ctx context.Context) error {
    s.inShutdown.Store(true)
    s.closeListenersLocked()
    // Poll for idle connections
    for {
        if s.closeIdleConns() { return nil }
        select {
        case <-ctx.Done(): return ctx.Err()
        case <-timer.C: timer.Reset(nextPollInterval())
        }
    }
}

Kubernetes approach

Source: pkg/controller/deployment/deployment_controller.go (lines 171–196)

func (dc *DeploymentController) Run(ctx context.Context, workers int) {
    defer utilruntime.HandleCrash()
    dc.eventBroadcaster.StartStructuredLogging(3)
    dc.eventBroadcaster.StartRecordingToSink(...)
    defer dc.eventBroadcaster.Shutdown()

    var wg sync.WaitGroup
    defer func() {
        dc.queue.ShutDown()  // Stop accepting new work
        wg.Wait()            // Wait for workers to finish
    }()

    // Gate: don't start until caches are synced
    if !cache.WaitForNamedCacheSyncWithContext(ctx, ...) { return }

    for i := 0; i < workers; i++ {
        wg.Go(func() {
            wait.UntilWithContext(ctx, dc.worker, time.Second)
        })
    }
    <-ctx.Done()  // Block until context cancelled
}

Why K8s is different

Stdlib's shutdown is reactive (wait for connections to drain). Kubernetes' shutdown is multi-phase orchestrated:

1. Stop accepting new events (close watch connections)
2. Drain the work queue (process remaining items)
3. Wait for in-flight syncs to complete
4. Shut down event recorders

The queue's ShutDownWithDrain() is the K8s-specific innovation: it waits until all in-flight items call Done().

---

6. Type Systems: interfaces vs. Runtime Type Registry

Stdlib approach

Interfaces for polymorphism. If you need to serialize, use encoding/json with struct tags.

Kubernetes approach

A full runtime type registry (Scheme) that maps between GVK strings and Go types.

Source: staging/src/k8s.io/apimachinery/pkg/runtime/scheme.go

Why K8s is different

Stdlib's encoding/json requires knowing the concrete type at compile time. Kubernetes must:

• Deserialize objects from the network without knowing their type in advance
• Convert between API versions (v1beta1.Deployment → v1.Deployment)
• Support third-party types (CRDs) that don't exist at compile time
• Apply defaulting and validation based on type metadata

This forces a runtime type system layered on top of Go's static types.

---

7. Testing: httptest vs. Fake Clients + Reactors

Stdlib approach

net/http/httptest provides a test server. You make real HTTP calls against it.

Kubernetes approach

Fake clientsets with reactor chains:

// Generated fake clients intercept API calls
fakeClient := fake.NewSimpleClientset(existingObjects...)
fakeClient.PrependReactor("create", "pods", func(action testing.Action) (bool, runtime.Object, error) {
    // Custom test behavior
    return true, nil, fmt.Errorf("simulated error")
})

Why K8s is different

Testing a controller doesn't require a running API server. The fake client + informer pattern lets you:

• Inject specific starting states
• Simulate failures at specific operations
• Run synchronously (no network delay)
• Test the controller logic in isolation

---

8. Lifecycle: main() returns vs. Infinite Reconciliation

Stdlib pattern

Programs start, do work, return.

Kubernetes pattern

Controllers start and run forever, continuously reconciling.

// The fundamental difference: stdlib programs terminate, controllers don't
func main() {
    // Stdlib: do work, exit
    result := compute()
    fmt.Println(result)
}

// Kubernetes: infinite loop with eventual consistency
func (c *Controller) Run(ctx context.Context) {
    // Run forever until context cancelled
    for i := 0; i < workers; i++ {
        go wait.UntilWithContext(ctx, c.worker, time.Second)
    }
    <-ctx.Done()
}

Why K8s is different

The real world is adversarial. Networks fail, nodes die, humans make mistakes. A one-shot program can't handle drift. The reconciliation loop is Kubernetes' answer to the CAP theorem: you can't guarantee consistency in a single call, but you can achieve it eventually through repetition.

---

9. Shared State: Package-Level vs. Shared Informer Cache

Stdlib approach

Package-level variables, or pass state through function parameters.

Kubernetes approach

The SharedInformerFactory creates a single in-memory cache per resource type, shared by all controllers in the process.

// All controllers share ONE watch and ONE cache per resource:
informerFactory := informers.NewSharedInformerFactory(client, resyncPeriod)
deployInformer := informerFactory.Apps().V1().Deployments()

// Controller A and B both get events from the same informer
deployInformer.Informer().AddEventHandler(controllerA)
deployInformer.Informer().AddEventHandler(controllerB)

Why K8s is different

Without sharing:

• 20 controllers × watch for Pods = 20 TCP connections to API server
• 20 copies of all Pods in memory

With SharedInformerFactory:

• 1 TCP connection for Pods
• 1 copy in memory
• Events fanned out to all registered handlers

---

10. Configuration: Flags/Env vs. Feature Gates

Stdlib approach

flag package, environment variables, config files.

Kubernetes approach

Feature gates: a versioned, lifecycle-aware configuration system.

Why K8s is different

Stdlib's flag package is for a single binary. Kubernetes has:

• Hundreds of features in various stages of maturity
• Features that must be consistent across control plane components
• Features that need to be enabled/disabled without redeployment
• Features with dependencies on other features
• Automated testing that exercises all combinations

Feature gates encode maturity (alpha/beta/GA) alongside the boolean value, something flag.Bool can never express.