docs: TESTING.md + TEST_RULES.md for the isolation/domain model
Document the two-target layout, per-test Kratos account isolation, the seed-before-login precondition rules (requiresResidence / seedAccountPreconditions), how to run the phased runner, and how to add a suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,85 @@
|
|||||||
|
# HoneyDue iOS Tests — How it works
|
||||||
|
|
||||||
|
Two test targets:
|
||||||
|
|
||||||
|
| Target | Kind | What it covers | Speed |
|
||||||
|
|--------|------|----------------|-------|
|
||||||
|
| **HoneyDueUITests** | XCUITest | Drives the real app via accessibility IDs. Organized by domain. | slow (per-test app launch + account) |
|
||||||
|
| **HoneyDueAPITests** | standalone unit-test | Pure-API/contract tests (no UI, no app launch). | seconds |
|
||||||
|
|
||||||
|
## Running
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd iosApp
|
||||||
|
|
||||||
|
# Full run: Smoke gate -> Seed -> API -> Parallel UI (8 workers) -> Sweep
|
||||||
|
./run_ui_tests.sh
|
||||||
|
|
||||||
|
# Variants
|
||||||
|
./run_ui_tests.sh "iPhone Air" 6 # device + worker count
|
||||||
|
./run_ui_tests.sh --smoke # just the smoke gate
|
||||||
|
./run_ui_tests.sh --only-api # just the fast API target
|
||||||
|
./run_ui_tests.sh --only-parallel # just the UI suites
|
||||||
|
```
|
||||||
|
|
||||||
|
The parallel phase runs the **whole UI target minus** the four phase-managed
|
||||||
|
suites (`SmokeUITests`, `AppLaunchUITests`, `AAA_SeedTests`,
|
||||||
|
`SuiteZZ_CleanupTests`) via `-skip-testing`. **New suites are picked up
|
||||||
|
automatically** — there is no hand-maintained include list to forget to update.
|
||||||
|
|
||||||
|
## Per-test account isolation (the core idea)
|
||||||
|
|
||||||
|
Every UI test that needs to be logged in subclasses `AuthenticatedUITestCase`,
|
||||||
|
which in `setUp`:
|
||||||
|
|
||||||
|
1. mints a **unique, pre-verified Kratos identity** —
|
||||||
|
`uit_<domain>_<uuid>@test.honeydue.local` (see `Core/Fixtures/TestAccount.swift`),
|
||||||
|
2. logs the app in as that account,
|
||||||
|
3. exposes `session` / `cleaner` / `account` for seeding under its **own** token,
|
||||||
|
|
||||||
|
and in `tearDown` calls `account.delete()`, which **cascades all of the
|
||||||
|
account's data and clears the Kratos identity** in one call.
|
||||||
|
|
||||||
|
No test shares state with any other, so suites run in parallel at high worker
|
||||||
|
counts with zero data races. (This replaced the old shared-`testuser` model
|
||||||
|
that capped the suite at 2 workers and forced Suite6 to run isolated.)
|
||||||
|
|
||||||
|
### Seeding data the UI must SEE
|
||||||
|
|
||||||
|
A fresh account is **empty at login**. Anything you seed via API *after* login
|
||||||
|
is invisible to the app until a manual refresh. So:
|
||||||
|
|
||||||
|
- **Creating** a thing through the UI → no setup needed (but note: task and
|
||||||
|
document creation are gated on a residence existing — set
|
||||||
|
`override var requiresResidence: Bool { true }`, which seeds one *before*
|
||||||
|
login and exposes it as `seededResidence`).
|
||||||
|
- **Viewing / editing / deleting an existing** thing → seed it **before login**:
|
||||||
|
|
||||||
|
```swift
|
||||||
|
override func seedAccountPreconditions(_ account: TestAccount) {
|
||||||
|
super.seedAccountPreconditions(account) // seeds seededResidence if requiresResidence
|
||||||
|
let r = seededResidence ?? account.seedResidence()
|
||||||
|
myTask = account.seedTask(residenceId: r.id, title: "Edit me")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Adding a suite
|
||||||
|
|
||||||
|
1. Pick the domain folder (`Task/`, `Residence/`, …). The folder is an Xcode
|
||||||
|
synchronized group, so a new `.swift` file is compiled into the target
|
||||||
|
automatically — no `.pbxproj` editing.
|
||||||
|
2. Name it `<Domain><Aspect>UITests`; subclass `BaseUITestCase` (logged-out
|
||||||
|
surface: login/registration/onboarding) or `AuthenticatedUITestCase`
|
||||||
|
(logged-in feature work).
|
||||||
|
3. Use accessibility IDs from `iosApp/Helpers/AccessibilityIdentifiers.swift`
|
||||||
|
(shared with the app — the single source of truth).
|
||||||
|
4. It runs automatically in the parallel phase. Done.
|
||||||
|
|
||||||
|
For a **pure-API** test (no UI): add it to `HoneyDueAPITests/` instead — it'll
|
||||||
|
run in the fast API phase.
|
||||||
|
|
||||||
|
## Known trade-off
|
||||||
|
|
||||||
|
Per-test account creation costs ~1–2s of setup (Kratos create + login). For a
|
||||||
|
big suite that adds up; if a full run drags, the planned mitigation is a
|
||||||
|
recycled account pool keyed by worker id. Correctness-first today.
|
||||||
@@ -15,7 +15,7 @@ These rules are non-negotiable. Every test, every suite, every helper must follo
|
|||||||
|
|
||||||
## Independence
|
## Independence
|
||||||
8. **Every suite runs alone, in combination, or in parallel** — no ordering dependencies
|
8. **Every suite runs alone, in combination, or in parallel** — no ordering dependencies
|
||||||
9. **Every test creates its own data in setUp, cleans up in tearDown**
|
9. **Every test gets its OWN isolated account** — `AuthenticatedUITestCase` mints a fresh Kratos identity per test (`uit_<domain>_<uuid>@test.honeydue.local`), seeds under its own token, and deletes it in tearDown (cascading all data). Never share an account across tests.
|
||||||
10. **No shared mutable state** — no `static var`, no class-level properties mutated across tests
|
10. **No shared mutable state** — no `static var`, no class-level properties mutated across tests
|
||||||
|
|
||||||
## Clarity
|
## Clarity
|
||||||
@@ -29,4 +29,4 @@ These rules are non-negotiable. Every test, every suite, every helper must follo
|
|||||||
16. **Target: each individual test completes in under 15 seconds** (excluding setUp/tearDown)
|
16. **Target: each individual test completes in under 15 seconds** (excluding setUp/tearDown)
|
||||||
|
|
||||||
## Preconditions
|
## Preconditions
|
||||||
17. **Every test assumption is validated before the test runs** — if a task test assumes a residence exists, verify via API in setUp. If the precondition isn't met, create it via API. Preconditions are NOT what the test is testing — they're infrastructure. Use API, not UI, to establish them.
|
17. **Seed UI-gated preconditions BEFORE login** — a fresh account is empty at login, so data the UI must display has to be seeded before the app loads it. Use `requiresResidence` (seeds a residence) or override `seedAccountPreconditions(_:)` to seed a full scenario via API. Preconditions are infrastructure, not what the test is testing — establish them via API, never UI.
|
||||||
|
|||||||
Reference in New Issue
Block a user