mirror of
https://github.com/vxcontrol/pentagi.git
synced 2026-09-03 07:25:50 +08:00
chore(e2e): scope Tier 2 networks and gate pentagi on a healthy mock LLM
The base compose file pins fixed network names, so any network left un-overridden in this e2e override joins the stack to the developer's live observability and langfuse networks. Give all three networks e2e-scoped names. Add a healthcheck to the mock-LLM service and make pentagi depend on service_healthy, so a crash-looping mock can never read as a started dependency and silently mask itself. Update frontend/docs/e2e.md to match reality: the stand tier runs @stand only (flow-run drives a real paid agent and stays local), every mock run rebuilds its own bundle by design, and the trend/affected-routes notes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -11,9 +11,16 @@
|
||||
# developer's .env and the stack inherits live provider keys and DOCKER_HOST.
|
||||
# (frontend/e2e/tools/run-local-tier.sh wraps all of this.)
|
||||
|
||||
# All three base networks get e2e names — the base file gives them fixed
|
||||
# `name:`s, so leaving any un-overridden would join this stack to the
|
||||
# developer's live observability/langfuse networks.
|
||||
networks:
|
||||
pentagi-network:
|
||||
name: pentagi-e2e-network
|
||||
observability-network:
|
||||
name: pentagi-e2e-observability-network
|
||||
langfuse-network:
|
||||
name: pentagi-e2e-langfuse-network
|
||||
|
||||
services:
|
||||
mock-llm:
|
||||
@@ -25,6 +32,13 @@ services:
|
||||
- ./frontend/e2e/mock-llm:/app:ro
|
||||
networks:
|
||||
- pentagi-network
|
||||
# Keeps a crash-looping mock (restart would silently mask it) from ever
|
||||
# reading as a started dependency.
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -qO /dev/null http://127.0.0.1:8080/v1/models"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
pentagi:
|
||||
container_name: pentagi-e2e
|
||||
@@ -32,7 +46,7 @@ services:
|
||||
pgvector:
|
||||
condition: service_healthy
|
||||
mock-llm:
|
||||
condition: service_started
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- LLM_SERVER_URL=http://mock-llm:8080/v1
|
||||
# The langchaingo client refuses to construct without a token even though
|
||||
|
||||
@@ -28,14 +28,18 @@ CI=1 pnpm e2e # byte-identical reproduction of a CI run
|
||||
|
||||
./e2e/tools/run-local-tier.sh # Tier 2: branch image + isolated docker
|
||||
# stack + mock LLM; runs specs/real/**
|
||||
E2E_TIER=stand E2E_BASE_URL=https://… pnpm e2e # against a live stand
|
||||
E2E_TIER=stand E2E_BASE_URL=https://… pnpm e2e # live stand: @stand smoke only
|
||||
# (flow-run drives a real paid agent run
|
||||
# and is scoped to the local tier)
|
||||
|
||||
pnpm e2e:visual # visual snapshots (pinned container)
|
||||
pnpm e2e:visual:update # regenerate baselines after a UI change
|
||||
```
|
||||
|
||||
Tips for the mock tier: `vite preview` is reused between runs if you keep it
|
||||
running (`reuseExistingServer`), so only the first run pays the build.
|
||||
Every mock-tier run rebuilds the bundle and starts its own `vite preview` —
|
||||
deliberately: reusing a server already listening on the port would silently
|
||||
test a previous commit's dist. A stray listener on 8100 fails the run loudly;
|
||||
kill it and rerun.
|
||||
|
||||
## Tiers
|
||||
|
||||
@@ -105,7 +109,7 @@ Key conventions:
|
||||
## Stand tier (Tier 3)
|
||||
|
||||
Runs the LLM-independent `@stand` smoke against a real deployment. Label a PR
|
||||
`e2e:stand` (or dispatch the workflow); the job runs in a protected
|
||||
`e2e:stand` (or dispatch the workflow with `tier: stand`); the job runs in a protected
|
||||
`e2e-stand` Environment whose required reviewers approve before any secret is
|
||||
exposed, so even a mislabeled run blocks on a human. Fork PRs never reach it.
|
||||
|
||||
@@ -119,10 +123,11 @@ prefix `NODE_TLS_REJECT_UNAUTHORIZED=0`; a real stand has a valid cert).
|
||||
|
||||
## Trends and selective runs
|
||||
|
||||
- **Trend:** `node e2e/tools/trend.mjs` turns a run's `results.json` into one
|
||||
JSONL record (p50/p95 spec duration, slowest three, pass/flaky/fail counts).
|
||||
CI appends it to the retained `e2e-trend` artifact so duration creep and flake
|
||||
are visible, not just green/red.
|
||||
- **Trend:** `node e2e/tools/trend.mjs` turns a run's `results.json` (written
|
||||
under `CI=1`) into one JSONL record (p50/p95 spec duration, slowest three,
|
||||
pass/flaky/fail counts). Each CI run uploads its own `e2e-trend` artifact;
|
||||
aggregate across runs offline (`gh run download`) to see duration creep and
|
||||
flake, not just green/red.
|
||||
- **Affected routes:** `pnpm exec tsx e2e/tools/affected.ts <base>` prints the
|
||||
manifest routes a diff touches (backed by each route's owning `sources` in
|
||||
`e2e/routes.ts`). Empty output = no frontend route changed. This is the
|
||||
@@ -153,6 +158,3 @@ bespoke bot:
|
||||
|
||||
This section is the recipe, not yet wired — the deterministic tiers above are
|
||||
the foundation it plugs into.
|
||||
|
||||
The full design (tiers, CI topology, mocking contract, roadmap) lives in the
|
||||
team's E2E testing plan; scenario coverage maps 1:1 to the scenario catalog IDs.
|
||||
|
||||
Reference in New Issue
Block a user