Refine Python pattern boundaries and Pydantic guidance

PROCESS.md

@@ -45,6 +45,7 @@ Selection criteria:
 
 - `sources/` = raw evidence notes, one file per upstream repo
 - `patterns/` = synthesized guidance from repeated evidence
+- `comparison/` = short notes for places where a library or framework bends the generic Python rule
 - `smells/` = anti-patterns derived from the same evidence base
 
 ## Step-by-step workflow
@@ -68,6 +69,8 @@ Important: source notes are not polished guidance. They are reusable evidence.
 ### 4) Synthesize only after evidence exists
 Turn strong repeated signals into `patterns/*.md` docs.
 
+Use `comparison/*.md` only when the generic Python rule still matters, but a specific library or framework changes how it should be applied.
+
 Each pattern doc should usually include:
 - what the pattern is
 - why it exists
@@ -95,16 +98,17 @@ Instead:
 - rewrite source-note files to be denser and more reusable
 - reduce duplicated guidance across docs
 
-That was the right move for this repo after the first bootstrap wave.
+## Handling mixed-concern source code
 
-## Fresh-context refinement pattern
+Upstream code will often mix generic Python design with framework-specific behavior in the same function, block, or line.
 
-A good refinement split is:
+When that happens:
-- one fresh pass over `patterns/*.md`
+- do not classify the whole snippet by repo name or file path alone
-- one fresh pass over `sources/*.md`
+- split the evidence into **Python-level signal** and **framework-owned signal**
-- one fresh pass doing citation audit and anti-vagueness cleanup
+- keep only the reusable language/library-design lesson in `python-patterns`
+- move the framework-owned lesson into the framework repo or comparison note
 
-The point is to avoid simply echoing earlier wording.
+If the claim depends on request parsing, dependency injection, response modeling, or HTTP error semantics, it no longer belongs here as a base Python rule.
 
 ## Review checklist
 
@@ -129,16 +133,6 @@ When the repo is ready for human review:
 
 This repo intentionally avoids pushing or creating remotes unless explicitly requested.
 
-## How to repeat this process next time
-
-1. Define the scope split first.
-2. Pick a small, high-signal upstream set.
-3. Build `sources/` before `patterns/`.
-4. Synthesize only the strongest topics first.
-5. Add smell docs after positive patterns exist.
-6. Run a fresh-context refinement wave.
-7. Initialize git only once the repo is reviewable.
-
 ## What to avoid
 
 - writing docs from memory

README.md

@@ -6,10 +6,7 @@ This repo captures reusable Python patterns extracted from mature upstream codeb
 
 A good pattern doc here includes:
 - **Why** — the reasoning, not just the rule
-- **When to use** — the trigger conditions
+- **When to use / not use** — where the pattern helps or causes harm
-- **When NOT to use** — where the pattern causes harm
-- **Preferred shapes** — examples of the intended form
-- **Counterexamples** — what to avoid and why
 - **Source citations** — verified `file:line` anchors from real codebases
 
 These docs should be derived from what strong Python codebases actually do, not from generic style-blog advice.
@@ -17,6 +14,7 @@ These docs should be derived from what strong Python codebases actually do, not
 ## Structure
 
 - `patterns/` — what to do
+- `comparison/` — where framework or library choices bend the generic Python guidance
 - `smells/` — what to avoid
 - `sources/` — extracted source-study notes and upstream references
 - `PROCESS.md` — the repeatable extraction/refinement workflow used to build this repo
@@ -32,15 +30,6 @@ Primary upstreams mined so far:
 - `pallets/click`
 - `sqlalchemy/sqlalchemy`
 
-Why this mix works:
-- CPython: API boundaries, exceptions, context managers, typing surface design
-- HTTPX: package facade design, sync/async split, transport seams, error taxonomy
-- pytest: fixture lifetime, parametrization, test ergonomics
-- Pydantic: validation and serialization boundaries
-- attrs: lightweight value-object/data-model patterns
-- Click: CLI-facing exception and public-surface patterns
-- SQLAlchemy: explicit persistence/session lifetime and sync-vs-async caveats
-
 ## Current pattern set
 
 - `patterns/module-design.md`
@@ -49,22 +38,39 @@ Why this mix works:
 - `patterns/async-boundaries.md`
 - `patterns/testing.md`
 - `patterns/data-models.md`
+- `comparison/pydantic-vs-python.md`
 - `smells/common-mistakes.md`
 
-## Reviewing this repo
+## Boundary rubric: does this belong in `python-patterns`?
 
-Recommended review order:
+Use this rubric both when writing docs here and when deciding where a real code example belongs.
-1. `README.md`
-2. `PROCESS.md`
-3. `sources/*.md` for evidence quality
-4. `patterns/*.md` for synthesis quality
-5. `smells/*.md` for anti-pattern coverage
 
-Questions to ask during review:
+Put the guidance here when most of the rule is about:
-- Is each claim grounded in repeated upstream evidence?
+- module and package structure
-- Are caveats preserved instead of flattened away?
+- API shape that would still make sense in a library, CLI, worker, or batch job
-- Does the pattern stay at the Python level rather than drifting into framework guidance?
+- internal data modeling and state boundaries
-- Are citations specific enough to be re-checked quickly?
+- exception design independent of HTTP or framework behavior
+- typing, protocols, sync/async design, and test structure that do not depend on a web framework
+
+Push the guidance out to a framework repo when most of the rule is about:
+- request or response handling owned by FastAPI
+- dependency injection through framework signatures
+- HTTP exception translation, status codes, or response envelopes
+- OpenAPI-facing schema behavior as part of a service contract
+- app lifespan and other framework-owned resource wiring
+
+Tie-break:
+- if the guidance could reasonably live in multiple places and a clean split would make it harder for someone writing code to find, prefer the **higher-level abstraction** and cross-link to the narrower guidance instead of over-optimizing repo purity
+
+### How to handle sloppy mixed examples
+
+Real code often mixes Python and framework concerns in the same file, function, or even line. Do **not** classify by file path alone.
+
+Instead:
+- extract the **Python-level rule** from the mixed example and keep only that part here
+- move or link the **framework-owned rule** to the framework repo
+- if one line expresses both concerns, write down which part is reusable without FastAPI and which part depends on FastAPI being present
+- when in doubt, prefer a narrower Python rule plus a comparison note instead of smuggling framework behavior into the base pattern
 
 ## Extraction rule
 

comparison/pydantic-vs-python.md

@@ -0,0 +1,142 @@
+# Pydantic vs Python
+
+Use this file when a rule *sounds* like ordinary Python data-model advice but changes once you choose Pydantic as the boundary type.
+
+For the underlying extracted evidence, see `sources/pydantic.md`.
+
+## Quick decision rule
+
+Put the guidance in `python-patterns` when it is mainly about:
+- small internal value objects
+- plain dataclasses or attrs-style state carriers
+- module and API design unrelated to validation frameworks
+- typing contracts that static tools can understand without runtime schema machinery
+
+Put the guidance in this comparison doc when it is mainly about:
+- validating untrusted input at runtime
+- coercing raw input into typed fields
+- explicit serialization rules
+- field or model validators
+- aliasing, defaults, and boundary-specific schema behavior
+
+## Where Pydantic deliberately bends the generic Python advice
+
+### 1) Construction: ordinary initialization vs validation-at-construction
+
+Generic Python often assumes object construction mostly means assigning already-good values.
+
+Pydantic changes that on purpose: model construction is a validation boundary.
+
+**Python default:**
+- constructors usually assume the caller is mostly honest
+- invariants may live in `__post_init__`, factories, or explicit validation code
+- field annotations are mostly for humans, editors, and type checkers
+
+**Pydantic adjustment:**
+- constructing a model from external data is itself the validation step
+- raw inputs may be coerced into the declared field types
+- bad input should fail immediately with structured validation errors
+- use `model_validate(...)` when you want the boundary-crossing step to be explicit
+
+See:
+- `python-patterns/patterns/data-models.md`
+- `python-patterns/sources/pydantic.md`
+
+### 2) Annotations: static contract hints vs runtime schema contract
+
+In ordinary Python, type annotations mostly describe intent and power static tooling.
+
+In Pydantic, annotations are also runtime instructions for parsing and schema behavior.
+
+**Python default:**
+- annotations are descriptive unless other machinery reads them
+- changing a type hint usually should not silently change runtime semantics
+
+**Pydantic adjustment:**
+- field annotations participate directly in parsing and validation
+- changing an annotation may change accepted input, coercion, and generated schema
+- treat model field types as part of the runtime boundary contract, not just editor sugar
+
+See:
+- `python-patterns/patterns/typing.md`
+- `python-patterns/sources/pydantic.md`
+
+### 3) Models: internal state carriers vs explicit boundary objects
+
+Generic Python guidance says to keep internal state simple and avoid one class doing every job.
+
+Pydantic sharpens that into a practical split: models are strongest at I/O boundaries, not automatically everywhere.
+
+**Python default:**
+- use small value-like classes for internal state
+- split transport, persistence, and domain concerns when they diverge
+
+**Pydantic adjustment:**
+- treat `BaseModel` primarily as a boundary model unless you have a reason not to
+- use Pydantic aggressively for request payloads, config, parsed files, or other untrusted input
+- be cautious about making every internal object a `BaseModel` just because the framework makes it convenient
+
+See:
+- `python-patterns/patterns/data-models.md`
+- `python-patterns/sources/pydantic.md`
+- `fastapi-conventions/patterns/pydantic-boundaries.md`
+
+### 4) Serialization: ad hoc dict building vs explicit dump methods
+
+Generic Python allows almost endless freedom in how objects become dictionaries or JSON.
+
+Pydantic narrows that on purpose by giving you named serialization APIs.
+
+**Python default:**
+- serialization may be custom and local
+- simple internal objects do not need a universal dump method
+
+**Pydantic adjustment:**
+- serialize with `model_dump()` or the corresponding explicit API, not hand-built dicts scattered around the codebase
+- keep the conversion step visible when leaving the boundary
+- let the model own field-level serialization behavior where that is the point of the model
+
+See:
+- `python-patterns/patterns/data-models.md`
+- `python-patterns/sources/pydantic.md`
+
+### 5) Validation hooks: general object methods vs narrow phase-aware validators
+
+In ordinary Python, invariants might live in constructors, factory functions, helper functions, or mutator methods.
+
+Pydantic offers validator hooks, but they are not a license to hide arbitrary business logic in schema classes.
+
+**Python default:**
+- put validation where the owning abstraction is obvious
+- prefer straightforward code over magical hook chains
+
+**Pydantic adjustment:**
+- keep validators narrow and local to the field or model concern they actually own
+- choose validator phase deliberately: `before` for raw input shaping, `after` for parsed-value checks
+- treat `plain` validators as an escape hatch because they can bypass the normal type-validation path
+- do not turn validators into a second invisible service layer
+
+See:
+- `python-patterns/sources/pydantic.md`
+- `fastapi-conventions/patterns/pydantic-boundaries.md`
+
+### 6) Error behavior: normal exceptions vs aggregated boundary errors
+
+Generic Python error design usually prefers exception types that map cleanly to caller decisions.
+
+Pydantic adds a boundary-specific pattern: multiple field failures can be reported together as structured validation output.
+
+**Python default:**
+- raise exceptions that match the semantic failure
+- keep error taxonomies useful to the caller
+
+**Pydantic adjustment:**
+- expect invalid boundary input to become `ValidationError`
+- treat that as part of the parsing contract, not as an incidental implementation detail
+- do not flatten that structure into vague strings unless the outer boundary has a reason to translate it
+
+See:
+- `python-patterns/patterns/error-handling.md`
+- `python-patterns/sources/pydantic.md`
+- `fastapi-conventions/patterns/errors.md`
+

patterns/data-models.md

@@ -11,16 +11,7 @@ A lot of messy Python comes from making one class do four jobs at once:
 - serialize output
 - own persistence or business behavior
 
-Mature codebases usually split those concerns.
+Mature codebases usually split those concerns:
-
-The recurring pattern is:
-
-- use explicit boundary models for validated input and output
-- keep serialization explicit
-- use lightweight value-like classes for simple internal state
-- avoid turning every data container into a god object
-
-## The pattern
 
 1. Use dedicated boundary models where data enters or leaves the system.
 2. Validate and serialize explicitly.
@@ -43,9 +34,7 @@ Use small value-like objects when:
 
 ## When not to use
 
-Do not make every internal object a heavyweight validation model.
+Do not treat a boundary model as proof that every internal object should also become a heavyweight validation model.
-
-Do not scatter custom `to_dict()` logic across the codebase.
 
 Do not let one model become schema, ORM wrapper, validator, service object, and side-effect manager at the same time.
 
@@ -117,9 +106,9 @@ That is not a model. That is a junk drawer.
 
 - `pydantic/main.py:253-264` says `BaseModel.__init__` parses and validates input data and raises `ValidationError` on bad input.
 - `pydantic/main.py:455-519` defines `model_dump(...)` as an explicit serialization step with caller-controlled include/exclude behavior.
-- `pydantic/main.py:721-768` defines `model_validate(...) -> Self` as a named boundary-crossing API.
+- `pydantic/main.py:721-768` defines `model_validate(...) -> Self` as a named boundary-crossing API with explicit validation options.
-- `docs/index.md:82-107` pairs model creation with `model_dump()` instead of treating instances as already wire-ready.
+- `docs/index.md:68-107` shows raw external data being coerced into typed fields and then dumped back out explicitly.
-- `docs/index.md:109-124` shows invalid external input failing loudly with `ValidationError`.
+- `docs/index.md:109-152` shows invalid external input failing with structured per-field validation errors instead of ad hoc strings.
 
 ### Attrs
 
@@ -131,6 +120,10 @@ That is not a model. That is a junk drawer.
 
 - `src/_pytest/timing.py:24-64` models `Instant` and `Duration` as frozen dataclasses for simple internal timing state.
 
+## Related comparison
+
+- `comparison/pydantic-vs-python.md`
+
 ## Bottom line
 
 Boundary models should validate and serialize cleanly.

patterns/typing.md

@@ -106,9 +106,14 @@ If runtime safety matters, attribute-presence checks are not enough. Protocols d
 
 ### Pydantic
 
-- `pydantic/main.py:156-205` exposes typed `ClassVar[...]` metadata for config, fields, serializer, and validator state even though framework internals are dynamic.
+- `pydantic/main.py:156-205` exposes typed `ClassVar[...]` metadata for config, schema, fields, serializer, and validator state even though framework internals are dynamic.
+- `pydantic/main.py:167-168` documents a synthesized `__init__` signature, which is another signal that annotations shape the runtime boundary.
 - `pydantic/main.py:253-264` makes model construction validate `**data: Any` immediately instead of pretending arbitrary inputs are already safe.
-- `pydantic/main.py:721-768` gives `model_validate(...) -> Self` an explicit typed boundary contract.
+- `pydantic/main.py:721-768` gives `model_validate(...) -> Self` an explicit typed boundary contract with runtime validation options.
+
+## Related comparison
+
+- `comparison/pydantic-vs-python.md`
 
 ## Bottom line
 

sources/pydantic.md

@@ -1,56 +1,106 @@
 # Pydantic source notes
 
 Repo: `pydantic/pydantic`
-Local checkout: `/home/ubuntu/repos/rodin-sources/pydantic`
+Local checkout: `/home/ubuntu/repos/rodin-bootstrap/upstream/pydantic`
 
 ## Why this repo is useful
-- Pydantic is a strong source for boundary-object patterns: validating incoming data, preserving typed state, and serializing back out explicitly.
+- Pydantic is a strong source for runtime boundary-object patterns: validating incoming data, coercing raw values into typed fields, and serializing back out explicitly.
-- It is also useful for validation-hook design because the docs distinguish several validator phases and call out their tradeoffs clearly.
+- It is also useful for validation-hook design because the docs distinguish validator phases and call out where those phases change the guarantees readers should expect.
+- It is *not* a generic argument that every Python data object should become a `BaseModel`. The strongest repeated signals are boundary-oriented.
 
-## Models are explicit validation + serialization boundaries
+## Model construction is a validation boundary, not plain attribute assignment
 
 ### Repeated evidence
-- `pydantic/main.py:119-145` defines `BaseModel` as the central abstraction and documents that models carry schema, field metadata, and decorator metadata.
+- `pydantic/main.py:253-264` says `BaseModel.__init__` creates a model by parsing and validating input data and raises `ValidationError` if the input cannot form a valid model.
-- `pydantic/main.py:201-205` explicitly exposes model-level serializer and validator machinery as core parts of the abstraction.
+- `pydantic/main.py:263-264` routes construction through `self.__pydantic_validator__.validate_python(...)`, which is a much stronger runtime contract than normal Python object initialization.
-- `docs/index.md:68-82` shows external data entering through model construction.
+- `pydantic/main.py:721-768` exposes `model_validate(...) -> Self` as an explicit alternative validation entrypoint with knobs for `strict`, `extra`, `from_attributes`, `by_alias`, and `by_name`.
-- `docs/index.md:82-89` immediately turns the model back into a plain data structure with `model_dump()`.
+- `docs/index.md:68-107` shows external data entering through model construction, being coerced into typed fields, and then becoming a typed model instance.
-- `docs/index.md:109-152` shows invalid boundary data raising `ValidationError` with structured per-field errors instead of silently degrading.
+- `docs/index.md:93-107` explicitly explains coercions such as strings to integers, strings to datetimes, and bytes keys to strings.
 
 ### Why it matters
-Repeated signal: Pydantic models are meant to sit at I/O boundaries. Input is validated/coerced at construction time; output is serialized through an explicit dump step.
+Repeated signal: once you choose Pydantic, object construction is no longer just “assign the fields.” It is a boundary-crossing operation that parses, coerces, and validates raw input.
 
 ### Caveat / counterexample
-The strong pattern is not "models are your whole domain model." The evidence here is boundary-oriented: construct from external data, then call `model_dump()` when leaving the boundary again.
+That makes Pydantic great for untrusted or external data. It does **not** automatically make it the right default for every small internal value object.
+
+## Field annotations are runtime parsing and schema instructions, not just static hints
+
+### Repeated evidence
+- `pydantic/main.py:156-205` exposes typed class-level metadata such as `model_config`, `__pydantic_core_schema__`, `__pydantic_serializer__`, `__pydantic_validator__`, and `__pydantic_fields__`.
+- `pydantic/main.py:167-168` documents a synthesized `__init__` signature for the model.
+- `docs/index.md:93-104` explains field annotations in runtime terms: requiredness, accepted/coerced input shapes, and typed container expectations.
+- `docs/index.md:99-100` ties `PositiveInt` directly to an annotated constrained type, showing that type declarations are part of the runtime contract.
+
+### Why it matters
+Repeated signal: in Pydantic, changing a field annotation can change runtime acceptance, coercion, and emitted schema behavior. These annotations are not mere editor decoration.
+
+## Models are strongest as explicit boundary objects
+
+### Repeated evidence
+- `docs/index.md:68-82` starts from `external_data` and immediately feeds it into a model.
+- `docs/index.md:82-89` then immediately uses `model_dump()` to cross back out of the model into plain data.
+- `pydantic/main.py:455-519` defines `model_dump(...)` as an explicit serialization API with include/exclude, alias, unset/default/none filtering, and error-handling controls.
+- `pydantic/main.py:521-569` provides `model_dump_json(...)` as the corresponding JSON-mode serialization boundary.
+
+### Why it matters
+Repeated signal: Pydantic models are designed to sit at runtime boundaries where input validation and output shaping matter. The repo keeps showing “raw external data in, explicit dump back out,” not “all domain state everywhere should live inside `BaseModel` forever.”
+
+### Caveat / counterexample
+The strong pattern is boundary ownership, not model monoculture. If an internal object only needs simple state and no runtime parsing or schema behavior, generic Python types may be clearer.
+
+## Serialization is explicit and configurable
+
+### Repeated evidence
+- `docs/index.md:82-89` uses `model_dump()` as the normal way to convert a model to a dictionary.
+- `pydantic/main.py:455-519` gives `model_dump(...)` explicit controls for aliasing, partial output, omission of unset/default/none values, round-tripping, and serialization error handling.
+- `pydantic/main.py:493-496` shows that serialization errors are configurable behavior, not an afterthought.
+
+### Why it matters
+Repeated signal: Pydantic wants serialization to be named and explicit. That is stronger and safer than scattering hand-built dict shaping around the codebase.
+
+## Validation errors are structured boundary output
+
+### Repeated evidence
+- `pydantic/main.py:253-257` documents `ValidationError` on invalid model construction.
+- `pydantic/main.py:745-749` documents `ValidationError` on `model_validate(...)` as well.
+- `docs/index.md:109-152` shows one bad input producing a list of per-field errors with `type`, `loc`, `msg`, `input`, and documentation URL.
+
+### Why it matters
+Repeated signal: Pydantic treats invalid input as a structured parsing result, not just a plain exception string. That is part of the contract callers and outer boundaries can build on.
 
 ## Validators are narrow and phase-aware
 
 ### Repeated evidence
-- `docs/concepts/validators.md:91-114` shows an `after` field validator that checks one parsed field and must return the validated value.
+- `docs/concepts/validators.md:94-114` shows an `after` field validator checking one parsed field and returning the validated value.
-- `docs/concepts/validators.md:160-167` explains that `before` validators run prior to internal parsing and therefore receive raw input.
+- `docs/concepts/validators.md:160-167` says `before` validators run before internal parsing and validation.
-- `docs/concepts/validators.md:220-252` demonstrates a `before` validator that reshapes raw input and then lets normal item validation continue.
+- `docs/concepts/validators.md:177-209` shows a `before` validator reshaping raw input while Pydantic still performs normal type validation afterward.
+- `docs/concepts/validators.md:201-206` explicitly warns that `before` validators receive arbitrary raw input and therefore must account for more cases.
 
 ### Why it matters
 Repeated signal: the best validator hooks are small in scope and explicit about phase:
 - `before` for raw-input normalization
-- `after` for post-parse invariants
+- `after` for parsed-value invariants
 
-This prevents validation logic from becoming an opaque second parser.
+This keeps validator logic from becoming a second opaque parser.
 
-## Validator mode choice has real behavioral consequences
+## Validator mode choice changes guarantees
 
 ### Repeated evidence
-- `docs/concepts/validators.md:160-164` warns that `before` validators should avoid careless mutation when raising later, especially with unions.
+- `docs/concepts/validators.md:160-164` warns against careless mutation in `before` validators, especially when later raising errors and when unions are involved.
 - `docs/concepts/validators.md:254-255` states that `plain` validators terminate validation immediately.
 - `docs/concepts/validators.md:273-283` shows the consequence directly: a `PlainValidator` can return `'invalid'` for a field annotated as `int`, and Pydantic will accept it.
+- `docs/concepts/validators.md:296-308` repeats the same consequence in decorator form.
 
 ### Why it matters
-Repeated signal: validator mode is not just an implementation detail. It changes whether core type validation still runs.
+Repeated signal: validator mode is not a cosmetic option. It changes whether core type validation still runs.
 
 ### Caveat / counterexample
-This is the sharpest anti-pattern in the repo: `plain` validators are powerful, but they can bypass the type guarantee a reader expects from the annotation. Use them only when terminating validation is the actual goal.
+This is the sharpest anti-pattern in the repo: `plain` validators are powerful, but they can bypass the type guarantee a reader expects from the annotation. Use them only when terminating validation is the real goal.
 
 ## Pattern candidates supported by this repo
-- use typed models at I/O boundaries
+- use Pydantic models at runtime input/output boundaries
-- serialize explicitly with `model_dump()`
+- treat model construction as a validation step, not plain assignment
+- treat field annotations as runtime parsing contracts when using `BaseModel`
+- serialize explicitly with `model_dump()` / `model_dump_json()`
 - keep validators field-scoped and phase-aware
-- treat `plain` validators as an escape hatch, not the default
+- treat `plain` validators as an escape hatch, not the default