rodin/review-bot
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Improve test coverage and add end-to-end integration tests in CI #32

New Issue
Closed
opened 2026-05-02 16:45:36 +00:00 by rodin · 1 comment
rodin commented 2026-05-02 16:45:36 +00:00
Owner
Copy Link
Copy Source

Problem

Current unit test coverage is 13.9%. The utility functions (hasSharedToken, findOwnReview, reviewUnchanged, extractSentinelName) are well-tested, but the core run() flow — which does all the real work (API calls, file I/O, AI invocation, review posting) — has zero test coverage.

This means we only catch logic bugs in isolated helpers. Misconfigurations, API contract changes, or integration failures between components are invisible until they hit production.

Goals

1. Unit test coverage ≥ 60%

  • Refactor run() to inject dependencies (HTTP client, file reader, AI caller)
  • Test the orchestration logic with mocked dependencies
  • Cover error paths: API failures, malformed responses, missing env vars, rate limits

2. End-to-end integration tests in CI

Run a real review-bot cycle against a test repo during CI:

  • Setup: Create a test PR on a dedicated test repo (or use the review-bot repo itself with a known diff)
  • Execute: Run review-bot with real Gitea API credentials (CI secrets) but a test model/prompt that produces deterministic output
  • Assert:
    • Review was posted with correct sentinel
    • Review state matches expected verdict
    • Inline comments land on correct lines
    • Update-in-place works (run twice, verify edit not duplicate)
    • Shared-token detection skips PATCH when misconfigured
    • Cleanup removes stale reviews correctly

3. CI workflow additions

  • Add a test-integration job that runs after unit tests pass
  • Use a dedicated test repo or test branch with a fixed diff
  • Gate merges on integration test pass
  • Consider a --dry-run flag that exercises the full pipeline but skips the final API POST (for cheaper CI runs)

Approach ideas

  • Testcontainers or docker-compose: Spin up a local Gitea instance for hermetic integration tests (no external deps)
  • Golden file testing: Compare posted review body against expected output for a known diff
  • Separate CI secret: INTEGRATION_TEST_TOKEN with write access to a test repo only

Acceptance criteria

  • Unit coverage ≥ 60% (reported in CI)
  • At least one integration test exercises the full post-review flow
  • CI fails if integration test fails
  • Coverage report visible in PR (badge or comment)
## Problem Current unit test coverage is **13.9%**. The utility functions (`hasSharedToken`, `findOwnReview`, `reviewUnchanged`, `extractSentinelName`) are well-tested, but the core `run()` flow — which does all the real work (API calls, file I/O, AI invocation, review posting) — has zero test coverage. This means we only catch logic bugs in isolated helpers. Misconfigurations, API contract changes, or integration failures between components are invisible until they hit production. ## Goals ### 1. Unit test coverage ≥ 60% - Refactor `run()` to inject dependencies (HTTP client, file reader, AI caller) - Test the orchestration logic with mocked dependencies - Cover error paths: API failures, malformed responses, missing env vars, rate limits ### 2. End-to-end integration tests in CI Run a real review-bot cycle against a test repo during CI: - **Setup:** Create a test PR on a dedicated test repo (or use the review-bot repo itself with a known diff) - **Execute:** Run review-bot with real Gitea API credentials (CI secrets) but a test model/prompt that produces deterministic output - **Assert:** - Review was posted with correct sentinel - Review state matches expected verdict - Inline comments land on correct lines - Update-in-place works (run twice, verify edit not duplicate) - Shared-token detection skips PATCH when misconfigured - Cleanup removes stale reviews correctly ### 3. CI workflow additions - Add a `test-integration` job that runs after unit tests pass - Use a dedicated test repo or test branch with a fixed diff - Gate merges on integration test pass - Consider a `--dry-run` flag that exercises the full pipeline but skips the final API POST (for cheaper CI runs) ## Approach ideas - **Testcontainers or docker-compose:** Spin up a local Gitea instance for hermetic integration tests (no external deps) - **Golden file testing:** Compare posted review body against expected output for a known diff - **Separate CI secret:** `INTEGRATION_TEST_TOKEN` with write access to a test repo only ## Acceptance criteria - [ ] Unit coverage ≥ 60% (reported in CI) - [ ] At least one integration test exercises the full post-review flow - [ ] CI fails if integration test fails - [ ] Coverage report visible in PR (badge or comment)
rodin commented 2026-05-02 19:10:59 +00:00
Author
Owner
Copy Link
Copy Source

Closing as sufficiently addressed.

Current coverage:

  • budget: 91.8%
  • review: 100%
  • llm: 88.6%
  • gitea: 80.5%
  • cmd/review-bot: 42.4%

Integration test scaffold exists (//go:build integration) and can be run manually via make test-integration. The remaining uncovered code in cmd/review-bot is the main orchestration flow which is effectively integration-tested by real usage on every PR.

CI already runs go test ./... on every push.

Closing as sufficiently addressed. Current coverage: - `budget`: 91.8% - `review`: 100% - `llm`: 88.6% - `gitea`: 80.5% - `cmd/review-bot`: 42.4% Integration test scaffold exists (`//go:build integration`) and can be run manually via `make test-integration`. The remaining uncovered code in `cmd/review-bot` is the main orchestration flow which is effectively integration-tested by real usage on every PR. CI already runs `go test ./...` on every push.
rodin closed this issue 2026-05-02 19:10:59 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ai-review
bug
needs-review
ready
Ready for human review
 self-reviewed
Self-review passed since last push
 wip
Work in progress - do not start new work on this PR
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
security-review-bot rodin (Rodin)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rodin/review-bot#32