chore: merge elixir-conventions and oban-conventions into sources/
Absorbed content from rodin/elixir-conventions and rodin/oban-conventions into a sources/ directory. These are reference material — descriptive, not prescriptive. Patterns that prove broadly applicable get promoted into patterns/. Part of taxonomy cleanup (issue #4): - Pattern = prescriptive, follow these - Convention/Source = reference, study for ideas The original repos can now be archived.
This commit is contained in:
Rodin
@@ -0,0 +1,15 @@
|
||||
# Sources
|
||||
|
||||
Reference material extracted from specific projects. Study for ideas, don't copy blindly.
|
||||
|
||||
These are **descriptive** — they document what a project does and why.
|
||||
The `patterns/` directory is **prescriptive** — it tells you what to do.
|
||||
|
||||
Patterns that prove broadly applicable get promoted from here into `patterns/`.
|
||||
The rest stays as reference for understanding how mature projects solve specific problems.
|
||||
|
||||
## Files
|
||||
|
||||
- `elixir-lang.md` — conventions from the elixir-lang/elixir source
|
||||
- `elixir-lang-analysis.md` — deeper analysis of elixir-lang source architecture
|
||||
- `oban.md` — patterns from oban-bg/oban (job processing, plugin system)
|
||||
@@ -0,0 +1,303 @@
|
||||
# Elixir Language Source: Architectural Conventions
|
||||
|
||||
How does José Valim and the Elixir core team build Elixir itself?
|
||||
What does the language source reveal about conventions that aren't
|
||||
documented anywhere else?
|
||||
|
||||
**Repo:** [elixir-lang/elixir](https://github.com/elixir-lang/elixir)
|
||||
|
||||
---
|
||||
|
||||
## 1. Repo Shape
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Size | 92M |
|
||||
| Source files | 567 .ex/.exs |
|
||||
| Erlang bootstrap | 33 .erl files |
|
||||
| Commits | 22,032 |
|
||||
| Contributors | 1,578 |
|
||||
| Test files | 208 |
|
||||
| Production files | 248 |
|
||||
| Test ratio | 1:1.2 |
|
||||
| TODOs (non-test) | 127 (all version-gated) |
|
||||
|
||||
### Organizational Philosophy
|
||||
|
||||
```
|
||||
lib/
|
||||
├── elixir/ # The language core (compiler + stdlib)
|
||||
│ ├── src/ # 33 Erlang files (bootstrap)
|
||||
│ └── lib/ # Elixir stdlib + compiler
|
||||
├── eex/ # Templating (independent OTP app)
|
||||
├── ex_unit/ # Testing framework (independent OTP app)
|
||||
├── iex/ # Interactive shell (independent OTP app)
|
||||
├── logger/ # Logging (independent OTP app)
|
||||
└── mix/ # Build tool (independent OTP app)
|
||||
```
|
||||
|
||||
Each component is a separate OTP application. They could theoretically
|
||||
be released independently. This is Elixir eating its own dog food —
|
||||
the umbrella project convention that Phoenix apps use comes directly
|
||||
from how the language itself is organized.
|
||||
|
||||
---
|
||||
|
||||
## 2. What the Codebase Values
|
||||
|
||||
### By size (what gets the most lines)
|
||||
|
||||
| Module | Lines | Role |
|
||||
|--------|-------|------|
|
||||
| `Kernel` | 7,102 | The implicit language surface |
|
||||
| `Module.Types.Descr` | 6,301 | Set-theoretic type descriptions |
|
||||
| `Enum` | 5,242 | Collection operations |
|
||||
| `String` | 3,263 | First-class string concept |
|
||||
| `Macro` | 3,102 | Metaprogramming foundation |
|
||||
| `Exception` | 2,720 | Error taxonomy |
|
||||
| `Code.Formatter` | 2,605 | Code formatting as library |
|
||||
|
||||
**The surprise:** The type system (`types/descr.ex` at 6,301 lines) is
|
||||
nearly as large as Kernel (7,102 lines). It's the newest and
|
||||
fastest-growing module — 504 commits, 96% written by José Valim. This
|
||||
is where the investment is going.
|
||||
|
||||
### By authorship (who shapes the language)
|
||||
|
||||
Type system: 396/504 commits from José, 32 from Eric Meadows-Jönsson,
|
||||
31 from Guillaume Duboc. This is auteur-driven development — one person
|
||||
holds the architectural vision for the most complex subsystem.
|
||||
|
||||
---
|
||||
|
||||
## 3. The Bootstrap Problem
|
||||
|
||||
**How does Elixir compile itself?**
|
||||
|
||||
The answer is 33 Erlang files in `lib/elixir/src/`:
|
||||
|
||||
```
|
||||
elixir_bootstrap.erl — minimal Kernel for self-compilation
|
||||
elixir_compiler.erl — the compiler entry point
|
||||
elixir_tokenizer.erl — lexer (in Erlang for speed)
|
||||
elixir_expand.erl — macro expansion
|
||||
elixir_erl.erl — Elixir AST → Erlang AST
|
||||
elixir_erl_pass.erl — code generation pass
|
||||
elixir_env.erl — compilation environment
|
||||
elixir_clauses.erl — pattern matching compilation
|
||||
```
|
||||
|
||||
**Convention:** The tokenizer and core compiler remain in Erlang
|
||||
permanently. This isn't technical debt — it's a deliberate choice.
|
||||
The tokenizer benefits from Erlang's binary pattern matching
|
||||
performance. The compiler needs to exist before Elixir does.
|
||||
|
||||
**Origin:** The bootstrap file dates to Nov 22, 2013 (commit
|
||||
`260be7c8e`: "Start porting elixir_macros to pure elixir"). Before
|
||||
this, MORE of the compiler was in Erlang. The trajectory is clear:
|
||||
minimize Erlang over time, but keep it where it provides genuine value.
|
||||
|
||||
---
|
||||
|
||||
## 4. TODO Culture: Version-Gated Deadlines
|
||||
|
||||
```elixir
|
||||
# TODO: Remove me on v2.0 — 16 occurrences
|
||||
# TODO: Deprecate me on Elixir v1.23 — 6 occurrences
|
||||
# TODO: Remove this clause on Elixir v2.0 once single-quoted charlists are removed
|
||||
# TODO: Make an error on Elixir v2.0 — 3 occurrences
|
||||
# TODO: Deprecate on Elixir v1.22 — 3 occurrences
|
||||
```
|
||||
|
||||
**Convention:** Every TODO has a version target. No "someday" TODOs
|
||||
exist. When a version ships, grep for that version's TODOs and resolve
|
||||
them all.
|
||||
|
||||
**127 total TODOs** across 567 files. Contrast with Go's 3,428 TODOs
|
||||
across 11K files — the Elixir team treats TODOs as time-bombs, not
|
||||
documentation.
|
||||
|
||||
---
|
||||
|
||||
## 5. Unique Patterns
|
||||
|
||||
### 5.1 Protocol Consolidation
|
||||
|
||||
Protocols dispatch dynamically at runtime by default (checking each
|
||||
struct's implementation). **Protocol consolidation** compiles all known
|
||||
implementations into a single dispatch module at build time.
|
||||
|
||||
From `lib/elixir/lib/protocol.ex`:
|
||||
> "Consolidation directly links the protocol to its implementations.
|
||||
> Invoking a consolidated protocol is equivalent to invoking two remote
|
||||
> functions."
|
||||
|
||||
**Convention:** Mix enables consolidation by default in production. The
|
||||
`@callback __protocol__(:consolidated?)` exists so code can check at
|
||||
runtime whether fast-path dispatch is active.
|
||||
|
||||
**When NOT to use:** Tests often disable consolidation (`consolidate_
|
||||
protocols: false`) so new protocol implementations added during tests
|
||||
are discoverable without recompilation.
|
||||
|
||||
### 5.2 Parallel Type Checker
|
||||
|
||||
`Module.ParallelChecker` (introduced July 2019, PR #9203 by Eric
|
||||
Meadows-Jönsson as "Add ExCk chunk") enables concurrent type checking
|
||||
across modules.
|
||||
|
||||
The type system itself (13,034 lines across 7 files in
|
||||
`lib/elixir/lib/module/types/`) is set-theoretic — types are sets, and
|
||||
operations are set operations (union, intersection, difference).
|
||||
|
||||
**Key files:**
|
||||
- `descr.ex` (6,301 lines) — type descriptions and set operations
|
||||
- `apply.ex` — function application typing
|
||||
- `expr.ex` — expression typing
|
||||
- `pattern.ex` — pattern match typing
|
||||
- `of.ex` — type inference
|
||||
- `helpers.ex` — shared utilities
|
||||
- `traverse.ex` — AST traversal
|
||||
|
||||
### 5.3 Code.Formatter as Library Function
|
||||
|
||||
The code formatter (2,605 lines) is a library function, not a CLI tool.
|
||||
You can call `Code.format_string!/2` from any Elixir code.
|
||||
|
||||
**Introduced:** Oct 7, 2017 (PR #6639 by José Valim). **Zero review
|
||||
comments. Merged in 1 hour.** José opened and merged his own formatter
|
||||
with no external review. This is the BDFL model — the language author
|
||||
ships foundational infrastructure by authority.
|
||||
|
||||
**Convention:** The formatter uses `Inspect.Algebra` (Wadler-Lindig
|
||||
pretty-printing) for layout decisions. It defines all operators and
|
||||
their associativity as module attributes:
|
||||
|
||||
```elixir
|
||||
@pipeline_operators [:|>, :~>>, :<<~, :~>, :<~, :<~>, :"<|>"]
|
||||
@right_new_line_before_binary_operators [:|, :when]
|
||||
@required_parens_logical_binary_operands [:|||, :||, :or, :&&&, :&&, :and]
|
||||
```
|
||||
|
||||
### 5.4 Mix Tasks as Single-File Modules
|
||||
|
||||
55 Mix tasks, each in its own file. Convention:
|
||||
- One task = one file
|
||||
- Module name determines task name: `Mix.Tasks.Deps.Clean` → `deps.clean`
|
||||
- `@shortdoc` for brief help, `@moduledoc` for full docs
|
||||
- `@recursive true` for umbrella traversal
|
||||
|
||||
### 5.5 ExUnit CaseTemplate (Extension Pattern)
|
||||
|
||||
The `ExUnit.CaseTemplate` is how Elixir's test framework supports
|
||||
extension — you define a module that `use`s `CaseTemplate`, and test
|
||||
modules `use YourModule` to inherit setup callbacks and helpers.
|
||||
|
||||
This is the same pattern Phoenix uses for `ConnCase` and `DataCase`.
|
||||
It originates from ExUnit itself — the framework demonstrates its own
|
||||
extension point.
|
||||
|
||||
### 5.6 Logger: Erlang Integration Done Right
|
||||
|
||||
PR #9333 (Sep 2019, merged Nov 2019): "Use Erlang's logger as main
|
||||
logging implementation." The Elixir Logger was rewritten to sit on top
|
||||
of Erlang's `:logger` module rather than reimplementing log dispatch.
|
||||
|
||||
**Convention:** When OTP provides infrastructure, wrap it rather than
|
||||
replace it. The compatibility layer translates Erlang log messages to
|
||||
Elixir format, but dispatch/filtering/handlers are OTP's.
|
||||
|
||||
---
|
||||
|
||||
## 6. PR Discussion Patterns
|
||||
|
||||
### JSON.Encoder (PR #14021, Dec 2024)
|
||||
|
||||
38 review comments, 13 days to merge. Key debate:
|
||||
|
||||
**sabiwara** asked: "What is the reason we went with a different API
|
||||
than Jason?" — questioning why the stdlib JSON module doesn't mirror
|
||||
the dominant community library.
|
||||
|
||||
**michalmuskala** (Jason author): "Once 1.18 is released with the new
|
||||
JSON module, I plan to make a new release of Jason with some small
|
||||
fixes and then effectively deprecate it."
|
||||
|
||||
**Lesson:** When stdlib absorbs community library functionality, the
|
||||
community library author participates in the review. Jason's author
|
||||
blessed the replacement and planned deprecation. This is how healthy
|
||||
ecosystem evolution works.
|
||||
|
||||
### Duration (PR #13385, Mar-Apr 2024)
|
||||
|
||||
75 comments + 116 review comments. The most debated PR in recent
|
||||
Elixir history.
|
||||
|
||||
**Pattern:** Community contributor (@tfiedlerdejanze) opened a PR
|
||||
adding `Date.shift/2`. José redirected to a broader `Duration` type.
|
||||
The contributor iterated through multiple designs.
|
||||
|
||||
**José's key intervention:** "I would rather prefer to pass a Duration
|
||||
to Calendar.ISO.shift_date, if we ever have such a type, rather than a
|
||||
keyword list." — refusing a simpler PR because it would lock in a
|
||||
suboptimal API before the full design was clear.
|
||||
|
||||
**Lesson:** The BDFL model means one person can say "this is the wrong
|
||||
abstraction" and redirect months of work. The PR took 33 days and
|
||||
several complete rewrites. The result was better because someone held
|
||||
the line on "solve the whole problem, not just the immediate pain."
|
||||
|
||||
### Formatter (PR #6639, Oct 2017)
|
||||
|
||||
Zero comments. Merged in 1 hour. 2,605 lines of new code.
|
||||
|
||||
**Lesson:** BDFL-driven projects can ship massive foundational changes
|
||||
with no review. José was both the author and the authority. This is the
|
||||
opposite of CockroachDB's Handle PR (2.5 months, extensive debate).
|
||||
Neither model is wrong — it depends on team structure and trust level.
|
||||
|
||||
---
|
||||
|
||||
## 7. Cross-Ecosystem Comparisons
|
||||
|
||||
| Aspect | Elixir | Go |
|
||||
|--------|--------|-----|
|
||||
| TODOs | 127, all version-gated | 3,428, all owner-attributed |
|
||||
| Formatter origin | BDFL ships in 1hr, no review | `gofmt` shipped with language |
|
||||
| Bootstrap | Erlang (33 files, permanent) | Assembly + Go (self-hosting since 1.5) |
|
||||
| Extension | 6 protocols + CaseTemplate | `internal/` packages (61 of them) |
|
||||
| Type system | Set-theoretic, 13K lines, growing | Static, mature, compile-time only |
|
||||
| Test ratio | 1:1.2 (file per file) | 1:3.3 (package-level tests) |
|
||||
| Governance | BDFL (José) | Committee (Russ Cox + team) |
|
||||
|
||||
---
|
||||
|
||||
## 8. What This Teaches
|
||||
|
||||
1. **BDFL projects can move faster on foundational infrastructure** —
|
||||
the formatter, type system, and JSON module all shipped because one
|
||||
person had authority. But Duration took 33 days because community
|
||||
contribution required iteration with the BDFL's vision.
|
||||
|
||||
2. **Version-gated TODOs are a superior cleanup strategy** for
|
||||
projects with regular release cycles. You never have to decide "is
|
||||
this worth fixing?" — the version bump forces the question.
|
||||
|
||||
3. **Keep the minimum viable bootstrap in the host language.** 33
|
||||
Erlang files is the floor, not a ceiling. The trajectory is always
|
||||
toward more Elixir, less Erlang — but the tokenizer stays in Erlang
|
||||
because binary matching is genuinely faster there.
|
||||
|
||||
4. **The type system's growth rate predicts the language's future.**
|
||||
504 commits, 96% from José, nearly as large as Kernel. Elixir's
|
||||
next 5 years will be defined by gradual typing.
|
||||
|
||||
5. **Community library authors should bless stdlib absorption.** The
|
||||
Jason → JSON.Encoder transition worked because michalmuskala
|
||||
participated in the review and planned deprecation.
|
||||
|
||||
6. **Each OTP app is an independent unit** — this convention flows
|
||||
directly into how Phoenix projects are organized. The language
|
||||
teaches its own architectural pattern by example.
|
||||
|
||||
<!-- PATTERN_COMPLETE -->
|
||||
@@ -0,0 +1,266 @@
|
||||
# Elixir Language Source: Convention Reference
|
||||
|
||||
Quick-reference for conventions extracted from the elixir-lang/elixir
|
||||
source code. Each entry: pattern name, location, example, when to use,
|
||||
when NOT to use, origin.
|
||||
|
||||
---
|
||||
|
||||
## Version-Gated TODOs
|
||||
|
||||
**Location:** Throughout `lib/`
|
||||
|
||||
```elixir
|
||||
# TODO: Remove me on v2.0
|
||||
# TODO: Deprecate me on Elixir v1.23
|
||||
# TODO: Make an error on Elixir v2.0
|
||||
```
|
||||
|
||||
**When to use:** Any backward-compatible code that should be removed at
|
||||
a known future version. Deprecation paths, compatibility shims, feature
|
||||
flags.
|
||||
|
||||
**When NOT to use:** Performance improvements ("make this faster
|
||||
someday"), refactoring desires, or anything without a clear version
|
||||
boundary. Those belong in issues, not TODOs.
|
||||
|
||||
**Origin:** Consistent throughout history. The pattern predates the
|
||||
repo's earliest commits — José's convention from the start.
|
||||
|
||||
---
|
||||
|
||||
## Independent OTP Applications
|
||||
|
||||
**Location:** `lib/` (6 top-level dirs)
|
||||
|
||||
```
|
||||
lib/elixir/mix.exs # The language core
|
||||
lib/ex_unit/mix.exs # Testing framework
|
||||
lib/mix/mix.exs # Build tool
|
||||
lib/logger/mix.exs # Logging
|
||||
lib/iex/mix.exs # Interactive shell
|
||||
lib/eex/mix.exs # Templates
|
||||
```
|
||||
|
||||
**When to use:** When components have distinct lifecycle, deployment, or
|
||||
dependency requirements. When you want components to be independently
|
||||
testable.
|
||||
|
||||
**When NOT to use:** Small projects where the overhead of multiple
|
||||
applications exceeds the organizational benefit. If components always
|
||||
deploy together and never independently, a single app is simpler.
|
||||
|
||||
**Origin:** Elixir 0.x — the language was always structured this way.
|
||||
|
||||
---
|
||||
|
||||
## Erlang for Performance-Critical Paths
|
||||
|
||||
**Location:** `lib/elixir/src/` (33 .erl files)
|
||||
|
||||
```erlang
|
||||
%% elixir_tokenizer.erl — binary pattern matching is faster in Erlang
|
||||
%% elixir_erl_pass.erl — code generation benefits from proximity to BEAM
|
||||
```
|
||||
|
||||
**When to use:** When Erlang's binary pattern matching or NIF interface
|
||||
provides measurable performance advantage. When code must exist before
|
||||
Elixir's compiler is available (bootstrap).
|
||||
|
||||
**When NOT to use:** For new features that could be written in Elixir.
|
||||
The trajectory is always toward less Erlang. Don't write new Erlang
|
||||
unless profiling proves it's necessary.
|
||||
|
||||
**Origin:** The entire language started as Erlang (2011). Bootstrap file
|
||||
(`elixir_bootstrap.erl`) formalized as distinct from "Elixir written
|
||||
in Erlang" in 2013 (commit `260be7c8e`).
|
||||
|
||||
---
|
||||
|
||||
## Protocol Consolidation
|
||||
|
||||
**Location:** `lib/elixir/lib/protocol.ex`
|
||||
|
||||
```elixir
|
||||
# In mix.exs (default for prod):
|
||||
consolidate_protocols: true
|
||||
|
||||
# Disable for tests:
|
||||
consolidate_protocols: Mix.env() != :test
|
||||
```
|
||||
|
||||
**When to use:** Always in production (it's the default). Consolidated
|
||||
protocols dispatch in two function calls instead of dynamic lookup.
|
||||
|
||||
**When NOT to use:** In tests where you define new protocol
|
||||
implementations at runtime. In dev if you're frequently recompiling
|
||||
protocol implementations.
|
||||
|
||||
**Origin:** PR adding consolidation to escript.build (2014, issue
|
||||
#2699). Later made default for Mix projects.
|
||||
|
||||
---
|
||||
|
||||
## Code.Formatter via Inspect.Algebra
|
||||
|
||||
**Location:** `lib/elixir/lib/code/formatter.ex` (2,605 lines)
|
||||
|
||||
```elixir
|
||||
# The formatter is a library function:
|
||||
Code.format_string!("def foo( x,y),do: x+y")
|
||||
# => "def foo(x, y), do: x + y"
|
||||
|
||||
# Internally uses Wadler-Lindig algebra:
|
||||
import Inspect.Algebra, except: [format: 2, surround: 3, surround: 4]
|
||||
```
|
||||
|
||||
**When to use:** When building code generation tools, macros that
|
||||
produce source, or custom formatting rules. The algebra is available
|
||||
to any Elixir code.
|
||||
|
||||
**When NOT to use:** Don't reimplement formatting logic. Use
|
||||
`Code.format_string!/2` or the `mix format` task. The algebra is for
|
||||
building formatters, not for end-user formatting.
|
||||
|
||||
**Origin:** Oct 7, 2017 (PR #6639, José Valim). Merged in 1 hour, zero
|
||||
review comments.
|
||||
|
||||
---
|
||||
|
||||
## Mix Task Convention
|
||||
|
||||
**Location:** `lib/mix/lib/mix/tasks/` (55 files)
|
||||
|
||||
```elixir
|
||||
defmodule Mix.Tasks.Deps.Clean do
|
||||
@moduledoc "Removes the given dependencies' build artifacts."
|
||||
@shortdoc "Deletes generated files and artifacts for dependencies"
|
||||
|
||||
use Mix.Task
|
||||
@recursive true
|
||||
|
||||
@impl true
|
||||
def run(args) do
|
||||
# ...
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
**When to use:** Any command-line operation that should be available via
|
||||
`mix <task>`. One file per task, module name determines task name.
|
||||
|
||||
**When NOT to use:** Tasks that require interactive user input (use
|
||||
escript or IEx helpers instead). Tasks that need to run without Mix
|
||||
loaded.
|
||||
|
||||
**Origin:** Part of Mix since its creation. The one-file convention
|
||||
is strict — all 55 stdlib tasks follow it.
|
||||
|
||||
---
|
||||
|
||||
## ExUnit.CaseTemplate
|
||||
|
||||
**Location:** `lib/ex_unit/lib/ex_unit/case_template.ex` (162 lines)
|
||||
|
||||
```elixir
|
||||
defmodule MyApp.DataCase do
|
||||
use ExUnit.CaseTemplate
|
||||
|
||||
setup tags do
|
||||
:ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)
|
||||
unless tags[:async], do: Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, {:shared, self()})
|
||||
:ok
|
||||
end
|
||||
end
|
||||
|
||||
# Usage:
|
||||
defmodule MyApp.SomeTest do
|
||||
use MyApp.DataCase, async: true
|
||||
end
|
||||
```
|
||||
|
||||
**When to use:** When multiple test modules share setup logic,
|
||||
assertions, or helper functions. The inheritance model allows
|
||||
composition of test concerns.
|
||||
|
||||
**When NOT to use:** For test helpers that don't need lifecycle
|
||||
callbacks. Simple `import` or `alias` is cleaner for utility functions
|
||||
that don't need `setup`/`setup_all`.
|
||||
|
||||
**Origin:** Commit `1f491dfbe` — "Add support to case templates." The
|
||||
pattern Phoenix adopted for `ConnCase`/`DataCase` comes directly from
|
||||
ExUnit.
|
||||
|
||||
---
|
||||
|
||||
## Logger Wrapping OTP
|
||||
|
||||
**Location:** `lib/logger/`
|
||||
|
||||
```elixir
|
||||
# Elixir's Logger dispatches to Erlang's :logger
|
||||
# Translation layer converts Erlang messages → Elixir format
|
||||
# Filters and handlers use OTP's infrastructure
|
||||
```
|
||||
|
||||
**When to use:** When OTP provides the infrastructure you need.
|
||||
Wrap it — don't replace it. Provide Elixir-idiomatic API on top.
|
||||
|
||||
**When NOT to use:** When OTP's solution has fundamental architectural
|
||||
limitations you can't work around with a wrapper (rare).
|
||||
|
||||
**Origin:** PR #9333 (Sep-Nov 2019). Rewrote Logger from custom
|
||||
dispatch to wrapping Erlang's `:logger`. ~2 month implementation.
|
||||
|
||||
---
|
||||
|
||||
## BDFL Merge Pattern
|
||||
|
||||
**Location:** Throughout git history
|
||||
|
||||
```
|
||||
PR #6639 (Formatter): 0 comments, merged in 1 hour, 2,605 new lines
|
||||
PR #14021 (JSON): 38 review comments, 13 days, community input
|
||||
PR #13385 (Duration): 75+116 comments, 33 days, community redirected
|
||||
```
|
||||
|
||||
**When to use:** Foundational infrastructure where one person holds the
|
||||
architectural vision. The BDFL can ship without review when the change
|
||||
is self-evidently correct (formatter) or redirect community work when
|
||||
the abstraction isn't right (Duration).
|
||||
|
||||
**When NOT to use:** In team-maintained projects without a clear
|
||||
authority. In projects where consensus is required for API decisions.
|
||||
The BDFL model fails when the BDFL is wrong and no one can override.
|
||||
|
||||
**Origin:** José Valim has been sole authority since Elixir's creation
|
||||
(2011). The core team exists but José has final say on language design.
|
||||
|
||||
---
|
||||
|
||||
## Type System Architecture (Set-Theoretic)
|
||||
|
||||
**Location:** `lib/elixir/lib/module/types/` (13,034 lines, 7 files)
|
||||
|
||||
```elixir
|
||||
# Types are sets. Operations are set operations.
|
||||
# descr.ex (6,301 lines) defines the algebra:
|
||||
# - Union of types
|
||||
# - Intersection of types
|
||||
# - Difference (negation types)
|
||||
# - Subtype checking via set inclusion
|
||||
```
|
||||
|
||||
**When to use:** Understanding how Elixir's gradual type system works
|
||||
internally. The set-theoretic approach means types compose naturally —
|
||||
`integer() | String.t()` is literally a set union.
|
||||
|
||||
**When NOT to use:** This is internal compiler infrastructure. Don't
|
||||
depend on `Module.Types` internals — they change frequently (504
|
||||
commits and counting).
|
||||
|
||||
**Origin:** Aug 2019 (PR #9270 by Eric Meadows-Jönsson). Originally
|
||||
just function clause exhaustiveness checking, now growing into full
|
||||
gradual typing. 96% of subsequent work by José.
|
||||
|
||||
<!-- PATTERN_COMPLETE -->
|
||||
+359
@@ -0,0 +1,359 @@
|
||||
# Patterns Extracted from oban-bg/oban
|
||||
|
||||
## Pattern: Plugin as Behaviour + GenServer
|
||||
|
||||
**Source:** `lib/oban/plugin.ex`
|
||||
**Category:** plugin
|
||||
|
||||
**What:** Define a plugin interface as a behaviour with
|
||||
`start_link/1` and `validate/1` callbacks. Plugins must be
|
||||
OTP-compliant (GenServer/Agent). The host supervises them.
|
||||
|
||||
**Why:** Extensibility without coupling. Oban can start any
|
||||
module that satisfies the behaviour — pruning, cron,
|
||||
lifeline — without knowing implementation details. The
|
||||
`validate/1` callback ensures misconfigured plugins fail at
|
||||
startup, not at runtime.
|
||||
|
||||
**Example:**
|
||||
|
||||
```elixir
|
||||
@callback start_link([option()]) :: GenServer.on_start()
|
||||
@callback validate([option()]) :: :ok | {:error, String.t()}
|
||||
@optional_callbacks [format_logger_output: 2]
|
||||
```
|
||||
|
||||
**When to use:** When your application needs a plugin
|
||||
system where third parties add behavior. The behaviour
|
||||
ensures type safety; supervision ensures fault isolation.
|
||||
|
||||
**When NOT to use:** Internal modules that you control.
|
||||
Behaviours add ceremony — if there is only one
|
||||
implementation, use a module directly.
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Structured Telemetry Spans
|
||||
|
||||
**Source:** `lib/oban/telemetry.ex`
|
||||
**Category:** telemetry
|
||||
|
||||
**What:** Emit telemetry events as spans with
|
||||
start/stop/exception structure. Every operation (job
|
||||
execution, engine calls, plugin work) follows the same
|
||||
three-event pattern with consistent metadata shapes.
|
||||
|
||||
**Why:** Uniform observability. Any monitoring tool
|
||||
(AppSignal, Datadog, custom logger) can hook into the same
|
||||
event structure. The span pattern (start → stop|exception)
|
||||
enables latency tracking, error rates, and resource usage
|
||||
measurement without custom instrumentation per feature.
|
||||
|
||||
**Example:**
|
||||
|
||||
```elixir
|
||||
# Event names follow: [:oban, :component, :action, :phase]
|
||||
[:oban, :job, :start]
|
||||
[:oban, :job, :stop] # measurements: duration, memory
|
||||
[:oban, :job, :exception] # + kind, reason, stacktrace
|
||||
|
||||
[:oban, :engine, :fetch_jobs, :start]
|
||||
[:oban, :engine, :fetch_jobs, :stop]
|
||||
[:oban, :engine, :fetch_jobs, :exception]
|
||||
```
|
||||
|
||||
**When to use:** Any library or application that wants
|
||||
observability without coupling to a specific monitoring
|
||||
backend. The pattern works for database queries, HTTP
|
||||
requests, background jobs, cache operations.
|
||||
|
||||
**When NOT to use:** Ultra-hot paths where telemetry
|
||||
overhead matters (millions of events/second). Use sampling
|
||||
or skip entirely.
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Engine Abstraction for Backend Swap
|
||||
|
||||
**Source:** `lib/oban/engine.ex`
|
||||
**Category:** engine
|
||||
|
||||
**What:** Define a behaviour (`Engine`) with callbacks for
|
||||
all database operations (insert, fetch, complete, etc.).
|
||||
Ship multiple implementations (Basic/Inline/Lite) that swap
|
||||
at config time.
|
||||
|
||||
**Why:** Different environments need different backends:
|
||||
Postgres for production, SQLite for development, inline
|
||||
(in-memory) for testing. The engine abstraction lets you
|
||||
swap without changing application code.
|
||||
|
||||
**Example:**
|
||||
|
||||
```elixir
|
||||
@callback init(conf, opts) :: {:ok, meta} | {:error, term}
|
||||
@callback insert_job(conf, changeset, opts) :: {:ok, Job.t()}
|
||||
@callback fetch_jobs(conf, meta, opts) :: {:ok, {meta, [Job.t()]}}
|
||||
@callback complete_job(conf, Job.t()) :: :ok
|
||||
```
|
||||
|
||||
**When to use:** When your system needs to support multiple
|
||||
storage backends, or when testing requires a fundamentally
|
||||
different execution model (synchronous vs async).
|
||||
|
||||
**When NOT to use:** Single-backend applications. The
|
||||
abstraction layer adds complexity that is only justified
|
||||
when you actually swap implementations.
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Keyword Validation with Reduce-While
|
||||
|
||||
**Source:** `lib/oban/validation.ex`
|
||||
**Category:** config
|
||||
|
||||
**What:** Validate keyword options by iterating with
|
||||
`Enum.reduce_while/3` and a validator function. Stop at
|
||||
first error. Return `:ok` or `{:error, reason}`.
|
||||
|
||||
**Why:** Keyword lists are the standard Elixir config
|
||||
format. Validating them procedurally (nested if/case) gets
|
||||
messy. The reduce-while + validator pattern is composable:
|
||||
each option validates independently, errors short-circuit,
|
||||
and the validator function can be swapped or extended.
|
||||
|
||||
**Example:**
|
||||
|
||||
```elixir
|
||||
def validate(opts, validator) when is_list(opts) do
|
||||
Enum.reduce_while(opts, :ok, fn opt, acc ->
|
||||
case validator.(opt) do
|
||||
:ok -> {:cont, acc}
|
||||
{:error, _} = error -> {:halt, error}
|
||||
end
|
||||
end)
|
||||
end
|
||||
```
|
||||
|
||||
**When to use:** Any public API that accepts keyword
|
||||
options from users. Libraries, GenServer init, plugin
|
||||
configs.
|
||||
|
||||
**When NOT to use:** Internal functions where the caller
|
||||
is trusted. Also avoid for deeply nested configs — use
|
||||
schema-based validation (NimbleOptions, Ecto embedded
|
||||
schemas) instead.
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Testing Mode Toggle
|
||||
|
||||
**Source:** `lib/oban/testing.ex`, `lib/oban/config.ex`
|
||||
**Category:** testing
|
||||
|
||||
**What:** Support a `testing:` config option that switches
|
||||
execution mode: `:disabled` (production), `:inline`
|
||||
(execute immediately in caller process), `:manual` (enqueue
|
||||
but don't execute — assert on DB state).
|
||||
|
||||
**Why:** Background job systems are inherently async, which
|
||||
makes testing hard. The mode toggle gives you: (1) inline
|
||||
for unit tests that need synchronous execution, (2) manual
|
||||
for integration tests that verify enqueueing without
|
||||
side effects.
|
||||
|
||||
**Example:**
|
||||
|
||||
```elixir
|
||||
# In test config:
|
||||
config :my_app, Oban, testing: :manual
|
||||
|
||||
# In tests:
|
||||
use Oban.Testing, repo: MyApp.Repo
|
||||
|
||||
perform_job(MyWorker, %{id: 1})
|
||||
assert_enqueued worker: MyWorker, args: %{id: 1}
|
||||
```
|
||||
|
||||
**When to use:** Any async system that needs deterministic
|
||||
testing — job queues, event buses, notification systems.
|
||||
The testing mode replaces "sleep and hope" with explicit
|
||||
control.
|
||||
|
||||
**When NOT to use:** Synchronous systems that are already
|
||||
deterministic. Also avoid if the mode toggle leaks into
|
||||
production code paths (keep it config-only, not conditional
|
||||
logic scattered through business code).
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Stopper for Goroutine Lifecycle (CockroachDB)
|
||||
|
||||
**Source:** `pkg/util/stop/stopper.go` (cockroachdb)
|
||||
**Category:** concurrency
|
||||
|
||||
**What:** A dedicated struct that manages the lifecycle of
|
||||
all goroutines in a component: tracks active tasks, refuses
|
||||
new work during shutdown (quiesce), waits for completion,
|
||||
then runs closers.
|
||||
|
||||
**Why:** In distributed systems, clean shutdown is critical.
|
||||
You need to: (1) stop accepting new work, (2) finish
|
||||
in-flight work, (3) release resources in order. The Stopper
|
||||
centralizes this instead of scattering shutdown logic across
|
||||
every goroutine.
|
||||
|
||||
**Example:**
|
||||
|
||||
```go
|
||||
type Stopper struct {
|
||||
quiescer chan struct{} // closed when quiescing
|
||||
stopped chan struct{} // closed when fully stopped
|
||||
mu struct {
|
||||
syncutil.RWMutex
|
||||
_numTasks int32
|
||||
quiescing, stopping bool
|
||||
closers []Closer
|
||||
}
|
||||
}
|
||||
|
||||
// RunAsyncTask refuses new work during quiesce
|
||||
func (s *Stopper) RunAsyncTask(ctx context.Context,
|
||||
taskName string, f func(context.Context)) error {
|
||||
if !s.addTask() {
|
||||
return ErrUnavailable
|
||||
}
|
||||
go func() {
|
||||
defer s.decTask()
|
||||
f(ctx)
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
**When to use:** Any server or subsystem that spawns
|
||||
goroutines and needs graceful shutdown. Especially in
|
||||
long-running services where leaked goroutines cause
|
||||
resource exhaustion.
|
||||
|
||||
**When NOT to use:** Simple programs with a single main
|
||||
goroutine. Or when `errgroup` with context cancellation
|
||||
suffices for the shutdown coordination.
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Atomic File Operations with Suffix Convention
|
||||
|
||||
**Source:** `tsdb/db.go` (prometheus)
|
||||
**Category:** storage
|
||||
|
||||
**What:** Use directory suffixes (`.tmp-for-creation`,
|
||||
`.tmp-for-deletion`) to make multi-step file operations
|
||||
crash-safe. On startup, clean up any dirs with these
|
||||
suffixes (they represent incomplete operations).
|
||||
|
||||
**Why:** Database storage needs atomicity. If the process
|
||||
crashes between creating a block and finalizing it, you
|
||||
need to know the block is incomplete. The suffix convention
|
||||
makes incomplete state visible at the filesystem level
|
||||
without requiring a separate journal.
|
||||
|
||||
**Example:**
|
||||
|
||||
```go
|
||||
const (
|
||||
tmpForDeletionBlockDirSuffix = ".tmp-for-deletion"
|
||||
tmpForCreationBlockDirSuffix = ".tmp-for-creation"
|
||||
)
|
||||
|
||||
// On startup: remove any .tmp-* dirs (incomplete ops)
|
||||
// On create: write to dir.tmp-for-creation, then rename
|
||||
// On delete: rename to dir.tmp-for-deletion, then remove
|
||||
```
|
||||
|
||||
**When to use:** Any system that manages files/directories
|
||||
and needs crash consistency without a full WAL. Simpler
|
||||
than a write-ahead log for coarse-grained operations.
|
||||
|
||||
**When NOT to use:** When you already have a WAL or
|
||||
transaction log. Or for fine-grained operations where
|
||||
rename semantics are insufficient.
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Options as DefaultOptions() + Override
|
||||
|
||||
**Source:** `tsdb/db.go` (prometheus)
|
||||
**Category:** configuration
|
||||
|
||||
**What:** Provide a `DefaultOptions()` function returning a
|
||||
fully-populated config struct. Users copy and override only
|
||||
what they need. No nil-means-default ambiguity.
|
||||
|
||||
**Why:** Large config structs (20+ fields) are unwieldy.
|
||||
By providing sane defaults as a function (not a package-
|
||||
level var), you avoid mutation bugs and make it clear what
|
||||
"normal" looks like. Users only specify deviations.
|
||||
|
||||
**Example:**
|
||||
|
||||
```go
|
||||
func DefaultOptions() *Options {
|
||||
return &Options{
|
||||
WALSegmentSize: wlog.DefaultSegmentSize,
|
||||
RetentionDuration: int64(15 * 24 * time.Hour / ...),
|
||||
MinBlockDuration: DefaultBlockDuration,
|
||||
MaxBlockDuration: DefaultBlockDuration,
|
||||
SamplesPerChunk: DefaultSamplesPerChunk,
|
||||
// ... 20 more fields with sane defaults
|
||||
}
|
||||
}
|
||||
|
||||
// Usage:
|
||||
opts := tsdb.DefaultOptions()
|
||||
opts.RetentionDuration = 30 * 24 * time.Hour
|
||||
db, err := tsdb.Open(dir, nil, nil, opts, nil)
|
||||
```
|
||||
|
||||
**When to use:** Config structs with many fields where most
|
||||
users want defaults. Especially when zero-value semantics
|
||||
would be confusing (e.g., 0 retention = infinite? or off?).
|
||||
|
||||
**When NOT to use:** Small configs (3-4 fields) where
|
||||
struct literal with zero-means-default is clear enough.
|
||||
|
||||
---
|
||||
|
||||
## Pattern: Scrape Loop with Aligned Timestamps
|
||||
|
||||
**Source:** `scrape/scrape.go` (prometheus)
|
||||
**Category:** concurrency
|
||||
|
||||
**What:** Periodic scrape loops that align timestamps to
|
||||
intervals with a small tolerance, enabling better storage
|
||||
compression downstream.
|
||||
|
||||
**Why:** Time-series databases compress better when
|
||||
timestamps are regular. A 2ms tolerance on alignment
|
||||
means scraped data aligns to the expected grid while
|
||||
accommodating real-world jitter.
|
||||
|
||||
**Example:**
|
||||
|
||||
```go
|
||||
var ScrapeTimestampTolerance = 2 * time.Millisecond
|
||||
var AlignScrapeTimestamps = true
|
||||
|
||||
// In scrape loop: if scrape finishes within tolerance
|
||||
// of the expected timestamp, snap to the grid
|
||||
```
|
||||
|
||||
**When to use:** Any periodic data collection where
|
||||
downstream storage benefits from timestamp regularity.
|
||||
Metrics, heartbeats, polling loops.
|
||||
|
||||
**When NOT to use:** Event-driven data where timestamps
|
||||
must reflect actual occurrence time. Audit logs, user
|
||||
actions, financial transactions.
|
||||
|
||||
<!-- PATTERN_COMPLETE -->
|
||||
Reference in New Issue
Block a user