docs: testing philosophy + API evolution strategies
Four testing models: defense-in-depth (CockroachDB), golden files (Prometheus), fake adapters (Ecto), testing modes (Oban). Three evolution strategies: version gates (distributed), numbered migrations (schema), compile-time deprecation (library).
This commit is contained in:
Rodin
@@ -0,0 +1,297 @@
|
||||
# Testing Philosophy & API Evolution
|
||||
|
||||
How codebases prove correctness and manage change over
|
||||
time reveals their deepest architectural commitments.
|
||||
|
||||
---
|
||||
|
||||
## Testing Philosophy: Four Models of Proof
|
||||
|
||||
### CockroachDB: Defense in Depth
|
||||
|
||||
**Levels of proof:**
|
||||
1. **Unit tests** — co-located in same package
|
||||
2. **Echotest/golden files** — snapshot expected output (209
|
||||
testdata directories, auto-rewrite with -rewrite flag)
|
||||
3. **Data-driven tests** — declarative test specs in txt files
|
||||
4. **KVNemesis** — chaos/fuzzing that generates random KV
|
||||
operations and checks linearizability
|
||||
5. **Leak detection** — goroutines, stoppers tracked globally
|
||||
|
||||
**The echotest pattern:**
|
||||
```go
|
||||
echotest.Require(t, output, filepath.Join("testdata", name+".txt"))
|
||||
```
|
||||
|
||||
Golden file says:
|
||||
```
|
||||
echo
|
||||
----
|
||||
result is ambiguous: boom with a secret
|
||||
result is ambiguous: boom with a ‹secret›
|
||||
```
|
||||
|
||||
The test produces output, compares against the golden file.
|
||||
Run with `-rewrite` to update. This means:
|
||||
- Tests are **self-documenting** (the golden file IS the spec)
|
||||
- Regressions are **visible in diffs** (the golden file changes)
|
||||
- No manual expected-value maintenance
|
||||
|
||||
**KVNemesis (chaos testing at ecosystem level):**
|
||||
Generates random sequences of KV operations (puts, gets,
|
||||
splits, merges, transfers) against a real cluster, then
|
||||
validates that results satisfy serializable isolation.
|
||||
|
||||
This isn't unit testing. This is proving the *system* is
|
||||
correct, not individual functions.
|
||||
|
||||
**Resource leak detection as CI gate:**
|
||||
```go
|
||||
// Every test file
|
||||
defer leaktest.AfterTest(t)()
|
||||
|
||||
// Every TestMain
|
||||
func init() {
|
||||
leaktest.PrintLeakedStoppers = PrintLeakedStoppers
|
||||
}
|
||||
```
|
||||
|
||||
If a test leaks a goroutine or Stopper, it **fails**. Not
|
||||
a warning. A failure. This means resource correctness is
|
||||
as enforceable as logic correctness.
|
||||
|
||||
### Prometheus: Golden Files + Goroutine Verification
|
||||
|
||||
**Testing DSL for PromQL:**
|
||||
```
|
||||
load 5m
|
||||
http_requests{job="api-server"} 0+10x10
|
||||
|
||||
eval instant at 50m SUM BY (group) (http_requests)
|
||||
{group="canary"} 700
|
||||
{group="production"} 300
|
||||
```
|
||||
|
||||
This is a custom test language. Load data, evaluate
|
||||
expressions, assert results. **205 test config files**
|
||||
in `config/testdata/` alone.
|
||||
|
||||
**Force:** PromQL is complex enough that example-based
|
||||
testing would be insufficient. The DSL lets you write
|
||||
hundreds of test cases concisely, covering edge cases
|
||||
that would require dozens of Go test functions.
|
||||
|
||||
**Goroutine leak detection:**
|
||||
```go
|
||||
func TolerantVerifyLeak(m *testing.M) {
|
||||
goleak.VerifyTestMain(m,
|
||||
goleak.IgnoreTopFunction("go.opencensus.io/..."),
|
||||
goleak.IgnoreTopFunction("k8s.io/klog/..."),
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
Explicit allowlist for known third-party leaks. Everything
|
||||
else is a test failure. Zero-tolerance with escape hatches
|
||||
for unfixable external dependencies.
|
||||
|
||||
### Ecto: Fake Adapter + Process Mailbox Assertions
|
||||
|
||||
```elixir
|
||||
defmodule Ecto.TestAdapter do
|
||||
@behaviour Ecto.Adapter
|
||||
@behaviour Ecto.Adapter.Queryable
|
||||
@behaviour Ecto.Adapter.Schema
|
||||
@behaviour Ecto.Adapter.Transaction
|
||||
|
||||
def execute(_, _, {:nocache, {:all, query}}, _, _) do
|
||||
send(self(), {:all, query})
|
||||
Process.get(:test_repo_all_results) || results_for_all_query(query)
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
**Ecto tests the entire query pipeline without a database.**
|
||||
The fake adapter:
|
||||
- Sends messages to `self()` on every operation
|
||||
- Tests assert on `receive {:insert, meta}` etc.
|
||||
- No network, no state, pure message-passing verification
|
||||
|
||||
**48 test files, 43 with `async: true`.** The test suite
|
||||
runs in parallel because there's no shared state — every
|
||||
test talks to its own process mailbox.
|
||||
|
||||
**Force:** Ecto is a *library*, not a service. It can't
|
||||
require Postgres in CI for every contributor. The fake
|
||||
adapter makes the entire query compilation + planning
|
||||
pipeline testable without external dependencies.
|
||||
|
||||
### Oban: Testing Modes as First-Class Feature
|
||||
|
||||
```elixir
|
||||
# In test config
|
||||
config :my_app, Oban, testing: :inline
|
||||
|
||||
# In test
|
||||
use Oban.Testing, repo: MyApp.Repo
|
||||
|
||||
test "job was enqueued" do
|
||||
assert_enqueued worker: MyWorker, args: %{id: 1}
|
||||
end
|
||||
|
||||
test "job executes correctly" do
|
||||
assert :ok = perform_job(MyWorker, %{id: 1})
|
||||
end
|
||||
```
|
||||
|
||||
Three modes:
|
||||
- **`:inline`** — jobs execute synchronously in the test
|
||||
process. No GenServers, no queues, no async.
|
||||
- **`:manual`** — jobs are enqueued but not executed.
|
||||
Use `assert_enqueued` to verify they were created.
|
||||
- **`:disabled`** — production behavior in tests.
|
||||
|
||||
**Force:** Background jobs are the #1 source of test
|
||||
flakiness. Oban eliminates it by making the execution
|
||||
model configurable. Tests never poll, never sleep, never
|
||||
race.
|
||||
|
||||
---
|
||||
|
||||
## API Evolution: Three Strategies
|
||||
|
||||
### CockroachDB: Version Gates (Distributed Migration)
|
||||
|
||||
```go
|
||||
const (
|
||||
V26_2_AddStatementStatisticsComputedColumns Key = iota
|
||||
V26_2_ChangefeedsStopReadingSpanLevelCheckpoints
|
||||
V26_2_ChangefeedsStopWritingSpanLevelCheckpoints
|
||||
)
|
||||
|
||||
// In code:
|
||||
if settings.Version.IsActive(ctx, clusterversion.V26_2) {
|
||||
// use new behavior
|
||||
}
|
||||
```
|
||||
|
||||
**The pattern:** Every change to observable behavior gets
|
||||
a version constant. The feature is only enabled when ALL
|
||||
nodes in the cluster have been upgraded past that version.
|
||||
|
||||
**Two-phase deprecation for distributed changes:**
|
||||
```
|
||||
V26_2_ChangefeedsStopReadingSpanLevelCheckpoints
|
||||
V26_2_ChangefeedsStopWritingSpanLevelCheckpoints
|
||||
V26_2_ChangefeedsNoLongerHaveSpanLevelCheckpoints
|
||||
```
|
||||
|
||||
Three versions for one removal:
|
||||
1. Stop reading (new code doesn't depend on old format)
|
||||
2. Stop writing (old format no longer produced)
|
||||
3. Clean up (safe to remove the old code)
|
||||
|
||||
**Force:** In a distributed database, you can't change
|
||||
behavior atomically. Some nodes will be old, some new.
|
||||
The version gate ensures new behavior only activates
|
||||
when it's safe — when all nodes understand it.
|
||||
|
||||
**Pruning:** Once MinSupported advances past a version
|
||||
constant, it's deleted. The code path is always active
|
||||
so the `IsActive` check becomes dead code. Regular
|
||||
pruning keeps the codebase from accumulating gates.
|
||||
|
||||
### Oban: Numbered Migrations (Schema Evolution)
|
||||
|
||||
```elixir
|
||||
lib/oban/migrations/postgres/
|
||||
├── v01.ex # Initial schema (job table, state enum)
|
||||
├── v02.ex # Add columns
|
||||
├── v03.ex # Index optimization
|
||||
...
|
||||
├── v14.ex # Latest
|
||||
```
|
||||
|
||||
Each migration is:
|
||||
- **Idempotent** (safe to run twice)
|
||||
- **Prefix-aware** (multi-tenant schemas)
|
||||
- **Bidirectional** (up + down)
|
||||
- **Database-specific** (postgres/, sqlite/, myxql/)
|
||||
|
||||
**Consumer usage:**
|
||||
```elixir
|
||||
defmodule MyApp.Repo.Migrations.AddOban do
|
||||
use Ecto.Migration
|
||||
def up, do: Oban.Migrations.up(version: 14)
|
||||
def down, do: Oban.Migrations.down(version: 14)
|
||||
end
|
||||
```
|
||||
|
||||
**Force:** Oban owns a database table but lives inside
|
||||
the consumer's migration system. Numbered versions let
|
||||
consumers upgrade incrementally without knowing Oban
|
||||
internals.
|
||||
|
||||
### Ecto: Compile-Time Deprecation + Semver
|
||||
|
||||
```elixir
|
||||
# In changeset.ex
|
||||
IO.warn(
|
||||
"passing a list of binaries to cast/3 is deprecated..."
|
||||
)
|
||||
```
|
||||
|
||||
Ecto deprecates at **compile time**. When you compile
|
||||
code that uses a deprecated API, you get a warning.
|
||||
At runtime, everything still works.
|
||||
|
||||
**CHANGELOG as contract:**
|
||||
```
|
||||
## v3.14.0-dev
|
||||
### Enhancements
|
||||
### Bug fixes
|
||||
|
||||
## v3.13.5 (2025-11-09)
|
||||
### Enhancements
|
||||
```
|
||||
|
||||
The changelog is the API evolution document. Breaking
|
||||
changes require a major version bump (hasn't happened
|
||||
in years because the adapter pattern provides
|
||||
extensibility without breakage).
|
||||
|
||||
---
|
||||
|
||||
## What This Teaches for Code Review
|
||||
|
||||
### Testing Questions:
|
||||
1. Is this testable **without standing up the system**?
|
||||
(Ecto's fake adapter, Oban's inline engine)
|
||||
2. Are resources **tracked and leak-detected**?
|
||||
(CockroachDB's stopper/goroutine tracking)
|
||||
3. Are test assertions **deterministic**? No sleep, no
|
||||
poll, no "eventually consistent" in unit tests.
|
||||
4. Could this be a **golden file test**? If the output
|
||||
is deterministic, snapshot it. Regression = visible diff.
|
||||
5. Is there **chaos/property testing** for invariants?
|
||||
(KVNemesis for linearizability)
|
||||
|
||||
### Evolution Questions:
|
||||
1. Can this change be deployed **gradually**? Or does it
|
||||
require all consumers to upgrade atomically?
|
||||
2. Is there a **two-phase** path? (Stop reading → stop
|
||||
writing → remove)
|
||||
3. Is the deprecation **visible at compile time**? Or
|
||||
will consumers only discover it at runtime?
|
||||
4. Is the migration **idempotent**? Can it be run twice
|
||||
safely?
|
||||
|
||||
### Red Flags:
|
||||
- Tests that require a running database for unit-level logic
|
||||
- No resource leak detection in concurrent code
|
||||
- `time.Sleep` / `Process.sleep` in tests instead of
|
||||
deterministic signals
|
||||
- Breaking changes without version gates or migration path
|
||||
- Deprecation that only appears in docs, not in tooling
|
||||
|
||||
<!-- PATTERN_COMPLETE -->
|
||||
Reference in New Issue
Block a user