chore: merge elixir-conventions and oban-conventions into sources/
Absorbed content from rodin/elixir-conventions and rodin/oban-conventions into a sources/ directory. These are reference material — descriptive, not prescriptive. Patterns that prove broadly applicable get promoted into patterns/. Part of taxonomy cleanup (issue #4): - Pattern = prescriptive, follow these - Convention/Source = reference, study for ideas The original repos can now be archived.
This commit is contained in:
Aaron Weiker
@@ -0,0 +1,303 @@
|
||||
# Elixir Language Source: Architectural Conventions
|
||||
|
||||
How does José Valim and the Elixir core team build Elixir itself?
|
||||
What does the language source reveal about conventions that aren't
|
||||
documented anywhere else?
|
||||
|
||||
**Repo:** [elixir-lang/elixir](https://github.com/elixir-lang/elixir)
|
||||
|
||||
---
|
||||
|
||||
## 1. Repo Shape
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Size | 92M |
|
||||
| Source files | 567 .ex/.exs |
|
||||
| Erlang bootstrap | 33 .erl files |
|
||||
| Commits | 22,032 |
|
||||
| Contributors | 1,578 |
|
||||
| Test files | 208 |
|
||||
| Production files | 248 |
|
||||
| Test ratio | 1:1.2 |
|
||||
| TODOs (non-test) | 127 (all version-gated) |
|
||||
|
||||
### Organizational Philosophy
|
||||
|
||||
```
|
||||
lib/
|
||||
├── elixir/ # The language core (compiler + stdlib)
|
||||
│ ├── src/ # 33 Erlang files (bootstrap)
|
||||
│ └── lib/ # Elixir stdlib + compiler
|
||||
├── eex/ # Templating (independent OTP app)
|
||||
├── ex_unit/ # Testing framework (independent OTP app)
|
||||
├── iex/ # Interactive shell (independent OTP app)
|
||||
├── logger/ # Logging (independent OTP app)
|
||||
└── mix/ # Build tool (independent OTP app)
|
||||
```
|
||||
|
||||
Each component is a separate OTP application. They could theoretically
|
||||
be released independently. This is Elixir eating its own dog food —
|
||||
the umbrella project convention that Phoenix apps use comes directly
|
||||
from how the language itself is organized.
|
||||
|
||||
---
|
||||
|
||||
## 2. What the Codebase Values
|
||||
|
||||
### By size (what gets the most lines)
|
||||
|
||||
| Module | Lines | Role |
|
||||
|--------|-------|------|
|
||||
| `Kernel` | 7,102 | The implicit language surface |
|
||||
| `Module.Types.Descr` | 6,301 | Set-theoretic type descriptions |
|
||||
| `Enum` | 5,242 | Collection operations |
|
||||
| `String` | 3,263 | First-class string concept |
|
||||
| `Macro` | 3,102 | Metaprogramming foundation |
|
||||
| `Exception` | 2,720 | Error taxonomy |
|
||||
| `Code.Formatter` | 2,605 | Code formatting as library |
|
||||
|
||||
**The surprise:** The type system (`types/descr.ex` at 6,301 lines) is
|
||||
nearly as large as Kernel (7,102 lines). It's the newest and
|
||||
fastest-growing module — 504 commits, 96% written by José Valim. This
|
||||
is where the investment is going.
|
||||
|
||||
### By authorship (who shapes the language)
|
||||
|
||||
Type system: 396/504 commits from José, 32 from Eric Meadows-Jönsson,
|
||||
31 from Guillaume Duboc. This is auteur-driven development — one person
|
||||
holds the architectural vision for the most complex subsystem.
|
||||
|
||||
---
|
||||
|
||||
## 3. The Bootstrap Problem
|
||||
|
||||
**How does Elixir compile itself?**
|
||||
|
||||
The answer is 33 Erlang files in `lib/elixir/src/`:
|
||||
|
||||
```
|
||||
elixir_bootstrap.erl — minimal Kernel for self-compilation
|
||||
elixir_compiler.erl — the compiler entry point
|
||||
elixir_tokenizer.erl — lexer (in Erlang for speed)
|
||||
elixir_expand.erl — macro expansion
|
||||
elixir_erl.erl — Elixir AST → Erlang AST
|
||||
elixir_erl_pass.erl — code generation pass
|
||||
elixir_env.erl — compilation environment
|
||||
elixir_clauses.erl — pattern matching compilation
|
||||
```
|
||||
|
||||
**Convention:** The tokenizer and core compiler remain in Erlang
|
||||
permanently. This isn't technical debt — it's a deliberate choice.
|
||||
The tokenizer benefits from Erlang's binary pattern matching
|
||||
performance. The compiler needs to exist before Elixir does.
|
||||
|
||||
**Origin:** The bootstrap file dates to Nov 22, 2013 (commit
|
||||
`260be7c8e`: "Start porting elixir_macros to pure elixir"). Before
|
||||
this, MORE of the compiler was in Erlang. The trajectory is clear:
|
||||
minimize Erlang over time, but keep it where it provides genuine value.
|
||||
|
||||
---
|
||||
|
||||
## 4. TODO Culture: Version-Gated Deadlines
|
||||
|
||||
```elixir
|
||||
# TODO: Remove me on v2.0 — 16 occurrences
|
||||
# TODO: Deprecate me on Elixir v1.23 — 6 occurrences
|
||||
# TODO: Remove this clause on Elixir v2.0 once single-quoted charlists are removed
|
||||
# TODO: Make an error on Elixir v2.0 — 3 occurrences
|
||||
# TODO: Deprecate on Elixir v1.22 — 3 occurrences
|
||||
```
|
||||
|
||||
**Convention:** Every TODO has a version target. No "someday" TODOs
|
||||
exist. When a version ships, grep for that version's TODOs and resolve
|
||||
them all.
|
||||
|
||||
**127 total TODOs** across 567 files. Contrast with Go's 3,428 TODOs
|
||||
across 11K files — the Elixir team treats TODOs as time-bombs, not
|
||||
documentation.
|
||||
|
||||
---
|
||||
|
||||
## 5. Unique Patterns
|
||||
|
||||
### 5.1 Protocol Consolidation
|
||||
|
||||
Protocols dispatch dynamically at runtime by default (checking each
|
||||
struct's implementation). **Protocol consolidation** compiles all known
|
||||
implementations into a single dispatch module at build time.
|
||||
|
||||
From `lib/elixir/lib/protocol.ex`:
|
||||
> "Consolidation directly links the protocol to its implementations.
|
||||
> Invoking a consolidated protocol is equivalent to invoking two remote
|
||||
> functions."
|
||||
|
||||
**Convention:** Mix enables consolidation by default in production. The
|
||||
`@callback __protocol__(:consolidated?)` exists so code can check at
|
||||
runtime whether fast-path dispatch is active.
|
||||
|
||||
**When NOT to use:** Tests often disable consolidation (`consolidate_
|
||||
protocols: false`) so new protocol implementations added during tests
|
||||
are discoverable without recompilation.
|
||||
|
||||
### 5.2 Parallel Type Checker
|
||||
|
||||
`Module.ParallelChecker` (introduced July 2019, PR #9203 by Eric
|
||||
Meadows-Jönsson as "Add ExCk chunk") enables concurrent type checking
|
||||
across modules.
|
||||
|
||||
The type system itself (13,034 lines across 7 files in
|
||||
`lib/elixir/lib/module/types/`) is set-theoretic — types are sets, and
|
||||
operations are set operations (union, intersection, difference).
|
||||
|
||||
**Key files:**
|
||||
- `descr.ex` (6,301 lines) — type descriptions and set operations
|
||||
- `apply.ex` — function application typing
|
||||
- `expr.ex` — expression typing
|
||||
- `pattern.ex` — pattern match typing
|
||||
- `of.ex` — type inference
|
||||
- `helpers.ex` — shared utilities
|
||||
- `traverse.ex` — AST traversal
|
||||
|
||||
### 5.3 Code.Formatter as Library Function
|
||||
|
||||
The code formatter (2,605 lines) is a library function, not a CLI tool.
|
||||
You can call `Code.format_string!/2` from any Elixir code.
|
||||
|
||||
**Introduced:** Oct 7, 2017 (PR #6639 by José Valim). **Zero review
|
||||
comments. Merged in 1 hour.** José opened and merged his own formatter
|
||||
with no external review. This is the BDFL model — the language author
|
||||
ships foundational infrastructure by authority.
|
||||
|
||||
**Convention:** The formatter uses `Inspect.Algebra` (Wadler-Lindig
|
||||
pretty-printing) for layout decisions. It defines all operators and
|
||||
their associativity as module attributes:
|
||||
|
||||
```elixir
|
||||
@pipeline_operators [:|>, :~>>, :<<~, :~>, :<~, :<~>, :"<|>"]
|
||||
@right_new_line_before_binary_operators [:|, :when]
|
||||
@required_parens_logical_binary_operands [:|||, :||, :or, :&&&, :&&, :and]
|
||||
```
|
||||
|
||||
### 5.4 Mix Tasks as Single-File Modules
|
||||
|
||||
55 Mix tasks, each in its own file. Convention:
|
||||
- One task = one file
|
||||
- Module name determines task name: `Mix.Tasks.Deps.Clean` → `deps.clean`
|
||||
- `@shortdoc` for brief help, `@moduledoc` for full docs
|
||||
- `@recursive true` for umbrella traversal
|
||||
|
||||
### 5.5 ExUnit CaseTemplate (Extension Pattern)
|
||||
|
||||
The `ExUnit.CaseTemplate` is how Elixir's test framework supports
|
||||
extension — you define a module that `use`s `CaseTemplate`, and test
|
||||
modules `use YourModule` to inherit setup callbacks and helpers.
|
||||
|
||||
This is the same pattern Phoenix uses for `ConnCase` and `DataCase`.
|
||||
It originates from ExUnit itself — the framework demonstrates its own
|
||||
extension point.
|
||||
|
||||
### 5.6 Logger: Erlang Integration Done Right
|
||||
|
||||
PR #9333 (Sep 2019, merged Nov 2019): "Use Erlang's logger as main
|
||||
logging implementation." The Elixir Logger was rewritten to sit on top
|
||||
of Erlang's `:logger` module rather than reimplementing log dispatch.
|
||||
|
||||
**Convention:** When OTP provides infrastructure, wrap it rather than
|
||||
replace it. The compatibility layer translates Erlang log messages to
|
||||
Elixir format, but dispatch/filtering/handlers are OTP's.
|
||||
|
||||
---
|
||||
|
||||
## 6. PR Discussion Patterns
|
||||
|
||||
### JSON.Encoder (PR #14021, Dec 2024)
|
||||
|
||||
38 review comments, 13 days to merge. Key debate:
|
||||
|
||||
**sabiwara** asked: "What is the reason we went with a different API
|
||||
than Jason?" — questioning why the stdlib JSON module doesn't mirror
|
||||
the dominant community library.
|
||||
|
||||
**michalmuskala** (Jason author): "Once 1.18 is released with the new
|
||||
JSON module, I plan to make a new release of Jason with some small
|
||||
fixes and then effectively deprecate it."
|
||||
|
||||
**Lesson:** When stdlib absorbs community library functionality, the
|
||||
community library author participates in the review. Jason's author
|
||||
blessed the replacement and planned deprecation. This is how healthy
|
||||
ecosystem evolution works.
|
||||
|
||||
### Duration (PR #13385, Mar-Apr 2024)
|
||||
|
||||
75 comments + 116 review comments. The most debated PR in recent
|
||||
Elixir history.
|
||||
|
||||
**Pattern:** Community contributor (@tfiedlerdejanze) opened a PR
|
||||
adding `Date.shift/2`. José redirected to a broader `Duration` type.
|
||||
The contributor iterated through multiple designs.
|
||||
|
||||
**José's key intervention:** "I would rather prefer to pass a Duration
|
||||
to Calendar.ISO.shift_date, if we ever have such a type, rather than a
|
||||
keyword list." — refusing a simpler PR because it would lock in a
|
||||
suboptimal API before the full design was clear.
|
||||
|
||||
**Lesson:** The BDFL model means one person can say "this is the wrong
|
||||
abstraction" and redirect months of work. The PR took 33 days and
|
||||
several complete rewrites. The result was better because someone held
|
||||
the line on "solve the whole problem, not just the immediate pain."
|
||||
|
||||
### Formatter (PR #6639, Oct 2017)
|
||||
|
||||
Zero comments. Merged in 1 hour. 2,605 lines of new code.
|
||||
|
||||
**Lesson:** BDFL-driven projects can ship massive foundational changes
|
||||
with no review. José was both the author and the authority. This is the
|
||||
opposite of CockroachDB's Handle PR (2.5 months, extensive debate).
|
||||
Neither model is wrong — it depends on team structure and trust level.
|
||||
|
||||
---
|
||||
|
||||
## 7. Cross-Ecosystem Comparisons
|
||||
|
||||
| Aspect | Elixir | Go |
|
||||
|--------|--------|-----|
|
||||
| TODOs | 127, all version-gated | 3,428, all owner-attributed |
|
||||
| Formatter origin | BDFL ships in 1hr, no review | `gofmt` shipped with language |
|
||||
| Bootstrap | Erlang (33 files, permanent) | Assembly + Go (self-hosting since 1.5) |
|
||||
| Extension | 6 protocols + CaseTemplate | `internal/` packages (61 of them) |
|
||||
| Type system | Set-theoretic, 13K lines, growing | Static, mature, compile-time only |
|
||||
| Test ratio | 1:1.2 (file per file) | 1:3.3 (package-level tests) |
|
||||
| Governance | BDFL (José) | Committee (Russ Cox + team) |
|
||||
|
||||
---
|
||||
|
||||
## 8. What This Teaches
|
||||
|
||||
1. **BDFL projects can move faster on foundational infrastructure** —
|
||||
the formatter, type system, and JSON module all shipped because one
|
||||
person had authority. But Duration took 33 days because community
|
||||
contribution required iteration with the BDFL's vision.
|
||||
|
||||
2. **Version-gated TODOs are a superior cleanup strategy** for
|
||||
projects with regular release cycles. You never have to decide "is
|
||||
this worth fixing?" — the version bump forces the question.
|
||||
|
||||
3. **Keep the minimum viable bootstrap in the host language.** 33
|
||||
Erlang files is the floor, not a ceiling. The trajectory is always
|
||||
toward more Elixir, less Erlang — but the tokenizer stays in Erlang
|
||||
because binary matching is genuinely faster there.
|
||||
|
||||
4. **The type system's growth rate predicts the language's future.**
|
||||
504 commits, 96% from José, nearly as large as Kernel. Elixir's
|
||||
next 5 years will be defined by gradual typing.
|
||||
|
||||
5. **Community library authors should bless stdlib absorption.** The
|
||||
Jason → JSON.Encoder transition worked because michalmuskala
|
||||
participated in the review and planned deprecation.
|
||||
|
||||
6. **Each OTP app is an independent unit** — this convention flows
|
||||
directly into how Phoenix projects are organized. The language
|
||||
teaches its own architectural pattern by example.
|
||||
|
||||
<!-- PATTERN_COMPLETE -->
|
||||
Reference in New Issue
Block a user