From 74101b513c371585328e7caf548ccd8c7de3190c Mon Sep 17 00:00:00 2001 From: Rodin Date: Thu, 7 May 2026 18:01:42 -0700 Subject: [PATCH] chore: merge elixir-conventions and oban-conventions into sources/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Absorbed content from rodin/elixir-conventions and rodin/oban-conventions into a sources/ directory. These are reference material — descriptive, not prescriptive. Patterns that prove broadly applicable get promoted into patterns/. Part of taxonomy cleanup (issue #4): - Pattern = prescriptive, follow these - Convention/Source = reference, study for ideas The original repos can now be archived. --- sources/README.md | 15 ++ sources/elixir-lang-analysis.md | 303 +++++++++++++++++++++++++++ sources/elixir-lang.md | 266 +++++++++++++++++++++++ sources/oban.md | 359 ++++++++++++++++++++++++++++++++ 4 files changed, 943 insertions(+) create mode 100644 sources/README.md create mode 100644 sources/elixir-lang-analysis.md create mode 100644 sources/elixir-lang.md create mode 100644 sources/oban.md diff --git a/sources/README.md b/sources/README.md new file mode 100644 index 0000000..5c98a76 --- /dev/null +++ b/sources/README.md @@ -0,0 +1,15 @@ +# Sources + +Reference material extracted from specific projects. Study for ideas, don't copy blindly. + +These are **descriptive** — they document what a project does and why. +The `patterns/` directory is **prescriptive** — it tells you what to do. + +Patterns that prove broadly applicable get promoted from here into `patterns/`. +The rest stays as reference for understanding how mature projects solve specific problems. + +## Files + +- `elixir-lang.md` — conventions from the elixir-lang/elixir source +- `elixir-lang-analysis.md` — deeper analysis of elixir-lang source architecture +- `oban.md` — patterns from oban-bg/oban (job processing, plugin system) diff --git a/sources/elixir-lang-analysis.md b/sources/elixir-lang-analysis.md new file mode 100644 index 0000000..62d2db9 --- /dev/null +++ b/sources/elixir-lang-analysis.md @@ -0,0 +1,303 @@ +# Elixir Language Source: Architectural Conventions + +How does José Valim and the Elixir core team build Elixir itself? +What does the language source reveal about conventions that aren't +documented anywhere else? + +**Repo:** [elixir-lang/elixir](https://github.com/elixir-lang/elixir) + +--- + +## 1. Repo Shape + +| Metric | Value | +|--------|-------| +| Size | 92M | +| Source files | 567 .ex/.exs | +| Erlang bootstrap | 33 .erl files | +| Commits | 22,032 | +| Contributors | 1,578 | +| Test files | 208 | +| Production files | 248 | +| Test ratio | 1:1.2 | +| TODOs (non-test) | 127 (all version-gated) | + +### Organizational Philosophy + +``` +lib/ +├── elixir/ # The language core (compiler + stdlib) +│ ├── src/ # 33 Erlang files (bootstrap) +│ └── lib/ # Elixir stdlib + compiler +├── eex/ # Templating (independent OTP app) +├── ex_unit/ # Testing framework (independent OTP app) +├── iex/ # Interactive shell (independent OTP app) +├── logger/ # Logging (independent OTP app) +└── mix/ # Build tool (independent OTP app) +``` + +Each component is a separate OTP application. They could theoretically +be released independently. This is Elixir eating its own dog food — +the umbrella project convention that Phoenix apps use comes directly +from how the language itself is organized. + +--- + +## 2. What the Codebase Values + +### By size (what gets the most lines) + +| Module | Lines | Role | +|--------|-------|------| +| `Kernel` | 7,102 | The implicit language surface | +| `Module.Types.Descr` | 6,301 | Set-theoretic type descriptions | +| `Enum` | 5,242 | Collection operations | +| `String` | 3,263 | First-class string concept | +| `Macro` | 3,102 | Metaprogramming foundation | +| `Exception` | 2,720 | Error taxonomy | +| `Code.Formatter` | 2,605 | Code formatting as library | + +**The surprise:** The type system (`types/descr.ex` at 6,301 lines) is +nearly as large as Kernel (7,102 lines). It's the newest and +fastest-growing module — 504 commits, 96% written by José Valim. This +is where the investment is going. + +### By authorship (who shapes the language) + +Type system: 396/504 commits from José, 32 from Eric Meadows-Jönsson, +31 from Guillaume Duboc. This is auteur-driven development — one person +holds the architectural vision for the most complex subsystem. + +--- + +## 3. The Bootstrap Problem + +**How does Elixir compile itself?** + +The answer is 33 Erlang files in `lib/elixir/src/`: + +``` +elixir_bootstrap.erl — minimal Kernel for self-compilation +elixir_compiler.erl — the compiler entry point +elixir_tokenizer.erl — lexer (in Erlang for speed) +elixir_expand.erl — macro expansion +elixir_erl.erl — Elixir AST → Erlang AST +elixir_erl_pass.erl — code generation pass +elixir_env.erl — compilation environment +elixir_clauses.erl — pattern matching compilation +``` + +**Convention:** The tokenizer and core compiler remain in Erlang +permanently. This isn't technical debt — it's a deliberate choice. +The tokenizer benefits from Erlang's binary pattern matching +performance. The compiler needs to exist before Elixir does. + +**Origin:** The bootstrap file dates to Nov 22, 2013 (commit +`260be7c8e`: "Start porting elixir_macros to pure elixir"). Before +this, MORE of the compiler was in Erlang. The trajectory is clear: +minimize Erlang over time, but keep it where it provides genuine value. + +--- + +## 4. TODO Culture: Version-Gated Deadlines + +```elixir +# TODO: Remove me on v2.0 — 16 occurrences +# TODO: Deprecate me on Elixir v1.23 — 6 occurrences +# TODO: Remove this clause on Elixir v2.0 once single-quoted charlists are removed +# TODO: Make an error on Elixir v2.0 — 3 occurrences +# TODO: Deprecate on Elixir v1.22 — 3 occurrences +``` + +**Convention:** Every TODO has a version target. No "someday" TODOs +exist. When a version ships, grep for that version's TODOs and resolve +them all. + +**127 total TODOs** across 567 files. Contrast with Go's 3,428 TODOs +across 11K files — the Elixir team treats TODOs as time-bombs, not +documentation. + +--- + +## 5. Unique Patterns + +### 5.1 Protocol Consolidation + +Protocols dispatch dynamically at runtime by default (checking each +struct's implementation). **Protocol consolidation** compiles all known +implementations into a single dispatch module at build time. + +From `lib/elixir/lib/protocol.ex`: +> "Consolidation directly links the protocol to its implementations. +> Invoking a consolidated protocol is equivalent to invoking two remote +> functions." + +**Convention:** Mix enables consolidation by default in production. The +`@callback __protocol__(:consolidated?)` exists so code can check at +runtime whether fast-path dispatch is active. + +**When NOT to use:** Tests often disable consolidation (`consolidate_ +protocols: false`) so new protocol implementations added during tests +are discoverable without recompilation. + +### 5.2 Parallel Type Checker + +`Module.ParallelChecker` (introduced July 2019, PR #9203 by Eric +Meadows-Jönsson as "Add ExCk chunk") enables concurrent type checking +across modules. + +The type system itself (13,034 lines across 7 files in +`lib/elixir/lib/module/types/`) is set-theoretic — types are sets, and +operations are set operations (union, intersection, difference). + +**Key files:** +- `descr.ex` (6,301 lines) — type descriptions and set operations +- `apply.ex` — function application typing +- `expr.ex` — expression typing +- `pattern.ex` — pattern match typing +- `of.ex` — type inference +- `helpers.ex` — shared utilities +- `traverse.ex` — AST traversal + +### 5.3 Code.Formatter as Library Function + +The code formatter (2,605 lines) is a library function, not a CLI tool. +You can call `Code.format_string!/2` from any Elixir code. + +**Introduced:** Oct 7, 2017 (PR #6639 by José Valim). **Zero review +comments. Merged in 1 hour.** José opened and merged his own formatter +with no external review. This is the BDFL model — the language author +ships foundational infrastructure by authority. + +**Convention:** The formatter uses `Inspect.Algebra` (Wadler-Lindig +pretty-printing) for layout decisions. It defines all operators and +their associativity as module attributes: + +```elixir +@pipeline_operators [:|>, :~>>, :<<~, :~>, :<~, :<~>, :"<|>"] +@right_new_line_before_binary_operators [:|, :when] +@required_parens_logical_binary_operands [:|||, :||, :or, :&&&, :&&, :and] +``` + +### 5.4 Mix Tasks as Single-File Modules + +55 Mix tasks, each in its own file. Convention: +- One task = one file +- Module name determines task name: `Mix.Tasks.Deps.Clean` → `deps.clean` +- `@shortdoc` for brief help, `@moduledoc` for full docs +- `@recursive true` for umbrella traversal + +### 5.5 ExUnit CaseTemplate (Extension Pattern) + +The `ExUnit.CaseTemplate` is how Elixir's test framework supports +extension — you define a module that `use`s `CaseTemplate`, and test +modules `use YourModule` to inherit setup callbacks and helpers. + +This is the same pattern Phoenix uses for `ConnCase` and `DataCase`. +It originates from ExUnit itself — the framework demonstrates its own +extension point. + +### 5.6 Logger: Erlang Integration Done Right + +PR #9333 (Sep 2019, merged Nov 2019): "Use Erlang's logger as main +logging implementation." The Elixir Logger was rewritten to sit on top +of Erlang's `:logger` module rather than reimplementing log dispatch. + +**Convention:** When OTP provides infrastructure, wrap it rather than +replace it. The compatibility layer translates Erlang log messages to +Elixir format, but dispatch/filtering/handlers are OTP's. + +--- + +## 6. PR Discussion Patterns + +### JSON.Encoder (PR #14021, Dec 2024) + +38 review comments, 13 days to merge. Key debate: + +**sabiwara** asked: "What is the reason we went with a different API +than Jason?" — questioning why the stdlib JSON module doesn't mirror +the dominant community library. + +**michalmuskala** (Jason author): "Once 1.18 is released with the new +JSON module, I plan to make a new release of Jason with some small +fixes and then effectively deprecate it." + +**Lesson:** When stdlib absorbs community library functionality, the +community library author participates in the review. Jason's author +blessed the replacement and planned deprecation. This is how healthy +ecosystem evolution works. + +### Duration (PR #13385, Mar-Apr 2024) + +75 comments + 116 review comments. The most debated PR in recent +Elixir history. + +**Pattern:** Community contributor (@tfiedlerdejanze) opened a PR +adding `Date.shift/2`. José redirected to a broader `Duration` type. +The contributor iterated through multiple designs. + +**José's key intervention:** "I would rather prefer to pass a Duration +to Calendar.ISO.shift_date, if we ever have such a type, rather than a +keyword list." — refusing a simpler PR because it would lock in a +suboptimal API before the full design was clear. + +**Lesson:** The BDFL model means one person can say "this is the wrong +abstraction" and redirect months of work. The PR took 33 days and +several complete rewrites. The result was better because someone held +the line on "solve the whole problem, not just the immediate pain." + +### Formatter (PR #6639, Oct 2017) + +Zero comments. Merged in 1 hour. 2,605 lines of new code. + +**Lesson:** BDFL-driven projects can ship massive foundational changes +with no review. José was both the author and the authority. This is the +opposite of CockroachDB's Handle PR (2.5 months, extensive debate). +Neither model is wrong — it depends on team structure and trust level. + +--- + +## 7. Cross-Ecosystem Comparisons + +| Aspect | Elixir | Go | +|--------|--------|-----| +| TODOs | 127, all version-gated | 3,428, all owner-attributed | +| Formatter origin | BDFL ships in 1hr, no review | `gofmt` shipped with language | +| Bootstrap | Erlang (33 files, permanent) | Assembly + Go (self-hosting since 1.5) | +| Extension | 6 protocols + CaseTemplate | `internal/` packages (61 of them) | +| Type system | Set-theoretic, 13K lines, growing | Static, mature, compile-time only | +| Test ratio | 1:1.2 (file per file) | 1:3.3 (package-level tests) | +| Governance | BDFL (José) | Committee (Russ Cox + team) | + +--- + +## 8. What This Teaches + +1. **BDFL projects can move faster on foundational infrastructure** — + the formatter, type system, and JSON module all shipped because one + person had authority. But Duration took 33 days because community + contribution required iteration with the BDFL's vision. + +2. **Version-gated TODOs are a superior cleanup strategy** for + projects with regular release cycles. You never have to decide "is + this worth fixing?" — the version bump forces the question. + +3. **Keep the minimum viable bootstrap in the host language.** 33 + Erlang files is the floor, not a ceiling. The trajectory is always + toward more Elixir, less Erlang — but the tokenizer stays in Erlang + because binary matching is genuinely faster there. + +4. **The type system's growth rate predicts the language's future.** + 504 commits, 96% from José, nearly as large as Kernel. Elixir's + next 5 years will be defined by gradual typing. + +5. **Community library authors should bless stdlib absorption.** The + Jason → JSON.Encoder transition worked because michalmuskala + participated in the review and planned deprecation. + +6. **Each OTP app is an independent unit** — this convention flows + directly into how Phoenix projects are organized. The language + teaches its own architectural pattern by example. + + diff --git a/sources/elixir-lang.md b/sources/elixir-lang.md new file mode 100644 index 0000000..57c8365 --- /dev/null +++ b/sources/elixir-lang.md @@ -0,0 +1,266 @@ +# Elixir Language Source: Convention Reference + +Quick-reference for conventions extracted from the elixir-lang/elixir +source code. Each entry: pattern name, location, example, when to use, +when NOT to use, origin. + +--- + +## Version-Gated TODOs + +**Location:** Throughout `lib/` + +```elixir +# TODO: Remove me on v2.0 +# TODO: Deprecate me on Elixir v1.23 +# TODO: Make an error on Elixir v2.0 +``` + +**When to use:** Any backward-compatible code that should be removed at +a known future version. Deprecation paths, compatibility shims, feature +flags. + +**When NOT to use:** Performance improvements ("make this faster +someday"), refactoring desires, or anything without a clear version +boundary. Those belong in issues, not TODOs. + +**Origin:** Consistent throughout history. The pattern predates the +repo's earliest commits — José's convention from the start. + +--- + +## Independent OTP Applications + +**Location:** `lib/` (6 top-level dirs) + +``` +lib/elixir/mix.exs # The language core +lib/ex_unit/mix.exs # Testing framework +lib/mix/mix.exs # Build tool +lib/logger/mix.exs # Logging +lib/iex/mix.exs # Interactive shell +lib/eex/mix.exs # Templates +``` + +**When to use:** When components have distinct lifecycle, deployment, or +dependency requirements. When you want components to be independently +testable. + +**When NOT to use:** Small projects where the overhead of multiple +applications exceeds the organizational benefit. If components always +deploy together and never independently, a single app is simpler. + +**Origin:** Elixir 0.x — the language was always structured this way. + +--- + +## Erlang for Performance-Critical Paths + +**Location:** `lib/elixir/src/` (33 .erl files) + +```erlang +%% elixir_tokenizer.erl — binary pattern matching is faster in Erlang +%% elixir_erl_pass.erl — code generation benefits from proximity to BEAM +``` + +**When to use:** When Erlang's binary pattern matching or NIF interface +provides measurable performance advantage. When code must exist before +Elixir's compiler is available (bootstrap). + +**When NOT to use:** For new features that could be written in Elixir. +The trajectory is always toward less Erlang. Don't write new Erlang +unless profiling proves it's necessary. + +**Origin:** The entire language started as Erlang (2011). Bootstrap file +(`elixir_bootstrap.erl`) formalized as distinct from "Elixir written +in Erlang" in 2013 (commit `260be7c8e`). + +--- + +## Protocol Consolidation + +**Location:** `lib/elixir/lib/protocol.ex` + +```elixir +# In mix.exs (default for prod): +consolidate_protocols: true + +# Disable for tests: +consolidate_protocols: Mix.env() != :test +``` + +**When to use:** Always in production (it's the default). Consolidated +protocols dispatch in two function calls instead of dynamic lookup. + +**When NOT to use:** In tests where you define new protocol +implementations at runtime. In dev if you're frequently recompiling +protocol implementations. + +**Origin:** PR adding consolidation to escript.build (2014, issue +#2699). Later made default for Mix projects. + +--- + +## Code.Formatter via Inspect.Algebra + +**Location:** `lib/elixir/lib/code/formatter.ex` (2,605 lines) + +```elixir +# The formatter is a library function: +Code.format_string!("def foo( x,y),do: x+y") +# => "def foo(x, y), do: x + y" + +# Internally uses Wadler-Lindig algebra: +import Inspect.Algebra, except: [format: 2, surround: 3, surround: 4] +``` + +**When to use:** When building code generation tools, macros that +produce source, or custom formatting rules. The algebra is available +to any Elixir code. + +**When NOT to use:** Don't reimplement formatting logic. Use +`Code.format_string!/2` or the `mix format` task. The algebra is for +building formatters, not for end-user formatting. + +**Origin:** Oct 7, 2017 (PR #6639, José Valim). Merged in 1 hour, zero +review comments. + +--- + +## Mix Task Convention + +**Location:** `lib/mix/lib/mix/tasks/` (55 files) + +```elixir +defmodule Mix.Tasks.Deps.Clean do + @moduledoc "Removes the given dependencies' build artifacts." + @shortdoc "Deletes generated files and artifacts for dependencies" + + use Mix.Task + @recursive true + + @impl true + def run(args) do + # ... + end +end +``` + +**When to use:** Any command-line operation that should be available via +`mix `. One file per task, module name determines task name. + +**When NOT to use:** Tasks that require interactive user input (use +escript or IEx helpers instead). Tasks that need to run without Mix +loaded. + +**Origin:** Part of Mix since its creation. The one-file convention +is strict — all 55 stdlib tasks follow it. + +--- + +## ExUnit.CaseTemplate + +**Location:** `lib/ex_unit/lib/ex_unit/case_template.ex` (162 lines) + +```elixir +defmodule MyApp.DataCase do + use ExUnit.CaseTemplate + + setup tags do + :ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo) + unless tags[:async], do: Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, {:shared, self()}) + :ok + end +end + +# Usage: +defmodule MyApp.SomeTest do + use MyApp.DataCase, async: true +end +``` + +**When to use:** When multiple test modules share setup logic, +assertions, or helper functions. The inheritance model allows +composition of test concerns. + +**When NOT to use:** For test helpers that don't need lifecycle +callbacks. Simple `import` or `alias` is cleaner for utility functions +that don't need `setup`/`setup_all`. + +**Origin:** Commit `1f491dfbe` — "Add support to case templates." The +pattern Phoenix adopted for `ConnCase`/`DataCase` comes directly from +ExUnit. + +--- + +## Logger Wrapping OTP + +**Location:** `lib/logger/` + +```elixir +# Elixir's Logger dispatches to Erlang's :logger +# Translation layer converts Erlang messages → Elixir format +# Filters and handlers use OTP's infrastructure +``` + +**When to use:** When OTP provides the infrastructure you need. +Wrap it — don't replace it. Provide Elixir-idiomatic API on top. + +**When NOT to use:** When OTP's solution has fundamental architectural +limitations you can't work around with a wrapper (rare). + +**Origin:** PR #9333 (Sep-Nov 2019). Rewrote Logger from custom +dispatch to wrapping Erlang's `:logger`. ~2 month implementation. + +--- + +## BDFL Merge Pattern + +**Location:** Throughout git history + +``` +PR #6639 (Formatter): 0 comments, merged in 1 hour, 2,605 new lines +PR #14021 (JSON): 38 review comments, 13 days, community input +PR #13385 (Duration): 75+116 comments, 33 days, community redirected +``` + +**When to use:** Foundational infrastructure where one person holds the +architectural vision. The BDFL can ship without review when the change +is self-evidently correct (formatter) or redirect community work when +the abstraction isn't right (Duration). + +**When NOT to use:** In team-maintained projects without a clear +authority. In projects where consensus is required for API decisions. +The BDFL model fails when the BDFL is wrong and no one can override. + +**Origin:** José Valim has been sole authority since Elixir's creation +(2011). The core team exists but José has final say on language design. + +--- + +## Type System Architecture (Set-Theoretic) + +**Location:** `lib/elixir/lib/module/types/` (13,034 lines, 7 files) + +```elixir +# Types are sets. Operations are set operations. +# descr.ex (6,301 lines) defines the algebra: +# - Union of types +# - Intersection of types +# - Difference (negation types) +# - Subtype checking via set inclusion +``` + +**When to use:** Understanding how Elixir's gradual type system works +internally. The set-theoretic approach means types compose naturally — +`integer() | String.t()` is literally a set union. + +**When NOT to use:** This is internal compiler infrastructure. Don't +depend on `Module.Types` internals — they change frequently (504 +commits and counting). + +**Origin:** Aug 2019 (PR #9270 by Eric Meadows-Jönsson). Originally +just function clause exhaustiveness checking, now growing into full +gradual typing. 96% of subsequent work by José. + + diff --git a/sources/oban.md b/sources/oban.md new file mode 100644 index 0000000..2ceab57 --- /dev/null +++ b/sources/oban.md @@ -0,0 +1,359 @@ +# Patterns Extracted from oban-bg/oban + +## Pattern: Plugin as Behaviour + GenServer + +**Source:** `lib/oban/plugin.ex` +**Category:** plugin + +**What:** Define a plugin interface as a behaviour with +`start_link/1` and `validate/1` callbacks. Plugins must be +OTP-compliant (GenServer/Agent). The host supervises them. + +**Why:** Extensibility without coupling. Oban can start any +module that satisfies the behaviour — pruning, cron, +lifeline — without knowing implementation details. The +`validate/1` callback ensures misconfigured plugins fail at +startup, not at runtime. + +**Example:** + +```elixir +@callback start_link([option()]) :: GenServer.on_start() +@callback validate([option()]) :: :ok | {:error, String.t()} +@optional_callbacks [format_logger_output: 2] +``` + +**When to use:** When your application needs a plugin +system where third parties add behavior. The behaviour +ensures type safety; supervision ensures fault isolation. + +**When NOT to use:** Internal modules that you control. +Behaviours add ceremony — if there is only one +implementation, use a module directly. + +--- + +## Pattern: Structured Telemetry Spans + +**Source:** `lib/oban/telemetry.ex` +**Category:** telemetry + +**What:** Emit telemetry events as spans with +start/stop/exception structure. Every operation (job +execution, engine calls, plugin work) follows the same +three-event pattern with consistent metadata shapes. + +**Why:** Uniform observability. Any monitoring tool +(AppSignal, Datadog, custom logger) can hook into the same +event structure. The span pattern (start → stop|exception) +enables latency tracking, error rates, and resource usage +measurement without custom instrumentation per feature. + +**Example:** + +```elixir +# Event names follow: [:oban, :component, :action, :phase] +[:oban, :job, :start] +[:oban, :job, :stop] # measurements: duration, memory +[:oban, :job, :exception] # + kind, reason, stacktrace + +[:oban, :engine, :fetch_jobs, :start] +[:oban, :engine, :fetch_jobs, :stop] +[:oban, :engine, :fetch_jobs, :exception] +``` + +**When to use:** Any library or application that wants +observability without coupling to a specific monitoring +backend. The pattern works for database queries, HTTP +requests, background jobs, cache operations. + +**When NOT to use:** Ultra-hot paths where telemetry +overhead matters (millions of events/second). Use sampling +or skip entirely. + +--- + +## Pattern: Engine Abstraction for Backend Swap + +**Source:** `lib/oban/engine.ex` +**Category:** engine + +**What:** Define a behaviour (`Engine`) with callbacks for +all database operations (insert, fetch, complete, etc.). +Ship multiple implementations (Basic/Inline/Lite) that swap +at config time. + +**Why:** Different environments need different backends: +Postgres for production, SQLite for development, inline +(in-memory) for testing. The engine abstraction lets you +swap without changing application code. + +**Example:** + +```elixir +@callback init(conf, opts) :: {:ok, meta} | {:error, term} +@callback insert_job(conf, changeset, opts) :: {:ok, Job.t()} +@callback fetch_jobs(conf, meta, opts) :: {:ok, {meta, [Job.t()]}} +@callback complete_job(conf, Job.t()) :: :ok +``` + +**When to use:** When your system needs to support multiple +storage backends, or when testing requires a fundamentally +different execution model (synchronous vs async). + +**When NOT to use:** Single-backend applications. The +abstraction layer adds complexity that is only justified +when you actually swap implementations. + +--- + +## Pattern: Keyword Validation with Reduce-While + +**Source:** `lib/oban/validation.ex` +**Category:** config + +**What:** Validate keyword options by iterating with +`Enum.reduce_while/3` and a validator function. Stop at +first error. Return `:ok` or `{:error, reason}`. + +**Why:** Keyword lists are the standard Elixir config +format. Validating them procedurally (nested if/case) gets +messy. The reduce-while + validator pattern is composable: +each option validates independently, errors short-circuit, +and the validator function can be swapped or extended. + +**Example:** + +```elixir +def validate(opts, validator) when is_list(opts) do + Enum.reduce_while(opts, :ok, fn opt, acc -> + case validator.(opt) do + :ok -> {:cont, acc} + {:error, _} = error -> {:halt, error} + end + end) +end +``` + +**When to use:** Any public API that accepts keyword +options from users. Libraries, GenServer init, plugin +configs. + +**When NOT to use:** Internal functions where the caller +is trusted. Also avoid for deeply nested configs — use +schema-based validation (NimbleOptions, Ecto embedded +schemas) instead. + +--- + +## Pattern: Testing Mode Toggle + +**Source:** `lib/oban/testing.ex`, `lib/oban/config.ex` +**Category:** testing + +**What:** Support a `testing:` config option that switches +execution mode: `:disabled` (production), `:inline` +(execute immediately in caller process), `:manual` (enqueue +but don't execute — assert on DB state). + +**Why:** Background job systems are inherently async, which +makes testing hard. The mode toggle gives you: (1) inline +for unit tests that need synchronous execution, (2) manual +for integration tests that verify enqueueing without +side effects. + +**Example:** + +```elixir +# In test config: +config :my_app, Oban, testing: :manual + +# In tests: +use Oban.Testing, repo: MyApp.Repo + +perform_job(MyWorker, %{id: 1}) +assert_enqueued worker: MyWorker, args: %{id: 1} +``` + +**When to use:** Any async system that needs deterministic +testing — job queues, event buses, notification systems. +The testing mode replaces "sleep and hope" with explicit +control. + +**When NOT to use:** Synchronous systems that are already +deterministic. Also avoid if the mode toggle leaks into +production code paths (keep it config-only, not conditional +logic scattered through business code). + +--- + +## Pattern: Stopper for Goroutine Lifecycle (CockroachDB) + +**Source:** `pkg/util/stop/stopper.go` (cockroachdb) +**Category:** concurrency + +**What:** A dedicated struct that manages the lifecycle of +all goroutines in a component: tracks active tasks, refuses +new work during shutdown (quiesce), waits for completion, +then runs closers. + +**Why:** In distributed systems, clean shutdown is critical. +You need to: (1) stop accepting new work, (2) finish +in-flight work, (3) release resources in order. The Stopper +centralizes this instead of scattering shutdown logic across +every goroutine. + +**Example:** + +```go +type Stopper struct { + quiescer chan struct{} // closed when quiescing + stopped chan struct{} // closed when fully stopped + mu struct { + syncutil.RWMutex + _numTasks int32 + quiescing, stopping bool + closers []Closer + } +} + +// RunAsyncTask refuses new work during quiesce +func (s *Stopper) RunAsyncTask(ctx context.Context, + taskName string, f func(context.Context)) error { + if !s.addTask() { + return ErrUnavailable + } + go func() { + defer s.decTask() + f(ctx) + }() + return nil +} +``` + +**When to use:** Any server or subsystem that spawns +goroutines and needs graceful shutdown. Especially in +long-running services where leaked goroutines cause +resource exhaustion. + +**When NOT to use:** Simple programs with a single main +goroutine. Or when `errgroup` with context cancellation +suffices for the shutdown coordination. + +--- + +## Pattern: Atomic File Operations with Suffix Convention + +**Source:** `tsdb/db.go` (prometheus) +**Category:** storage + +**What:** Use directory suffixes (`.tmp-for-creation`, +`.tmp-for-deletion`) to make multi-step file operations +crash-safe. On startup, clean up any dirs with these +suffixes (they represent incomplete operations). + +**Why:** Database storage needs atomicity. If the process +crashes between creating a block and finalizing it, you +need to know the block is incomplete. The suffix convention +makes incomplete state visible at the filesystem level +without requiring a separate journal. + +**Example:** + +```go +const ( + tmpForDeletionBlockDirSuffix = ".tmp-for-deletion" + tmpForCreationBlockDirSuffix = ".tmp-for-creation" +) + +// On startup: remove any .tmp-* dirs (incomplete ops) +// On create: write to dir.tmp-for-creation, then rename +// On delete: rename to dir.tmp-for-deletion, then remove +``` + +**When to use:** Any system that manages files/directories +and needs crash consistency without a full WAL. Simpler +than a write-ahead log for coarse-grained operations. + +**When NOT to use:** When you already have a WAL or +transaction log. Or for fine-grained operations where +rename semantics are insufficient. + +--- + +## Pattern: Options as DefaultOptions() + Override + +**Source:** `tsdb/db.go` (prometheus) +**Category:** configuration + +**What:** Provide a `DefaultOptions()` function returning a +fully-populated config struct. Users copy and override only +what they need. No nil-means-default ambiguity. + +**Why:** Large config structs (20+ fields) are unwieldy. +By providing sane defaults as a function (not a package- +level var), you avoid mutation bugs and make it clear what +"normal" looks like. Users only specify deviations. + +**Example:** + +```go +func DefaultOptions() *Options { + return &Options{ + WALSegmentSize: wlog.DefaultSegmentSize, + RetentionDuration: int64(15 * 24 * time.Hour / ...), + MinBlockDuration: DefaultBlockDuration, + MaxBlockDuration: DefaultBlockDuration, + SamplesPerChunk: DefaultSamplesPerChunk, + // ... 20 more fields with sane defaults + } +} + +// Usage: +opts := tsdb.DefaultOptions() +opts.RetentionDuration = 30 * 24 * time.Hour +db, err := tsdb.Open(dir, nil, nil, opts, nil) +``` + +**When to use:** Config structs with many fields where most +users want defaults. Especially when zero-value semantics +would be confusing (e.g., 0 retention = infinite? or off?). + +**When NOT to use:** Small configs (3-4 fields) where +struct literal with zero-means-default is clear enough. + +--- + +## Pattern: Scrape Loop with Aligned Timestamps + +**Source:** `scrape/scrape.go` (prometheus) +**Category:** concurrency + +**What:** Periodic scrape loops that align timestamps to +intervals with a small tolerance, enabling better storage +compression downstream. + +**Why:** Time-series databases compress better when +timestamps are regular. A 2ms tolerance on alignment +means scraped data aligns to the expected grid while +accommodating real-world jitter. + +**Example:** + +```go +var ScrapeTimestampTolerance = 2 * time.Millisecond +var AlignScrapeTimestamps = true + +// In scrape loop: if scrape finishes within tolerance +// of the expected timestamp, snap to the grid +``` + +**When to use:** Any periodic data collection where +downstream storage benefits from timestamp regularity. +Metrics, heartbeats, polling loops. + +**When NOT to use:** Event-driven data where timestamps +must reflect actual occurrence time. Audit logs, user +actions, financial transactions. + +