Improve e2e testing posture to make it sliiiightly less unfriendly for others to get working

2026-05-03 12:03:04 +02:00 · 2026-04-10 11:36:26 -07:00
parent c0fc5fbba2
commit 43c5e0f67d
6 changed files with 193 additions and 48 deletions
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -70,17 +70,111 @@ npm run test:run
 npm run build
 ```

+## Quality + Publishing Scripts
+
+<details>
+<summary>scripts/quality/</summary>
+
+| Script | Purpose |
+|--------|---------|
+| `all_quality.sh` | Repo-standard gate: autofix (ruff, eslint, prettier), then pyright, pytest, vitest, and frontend build. Run before finishing any code change. |
+| `extended_quality.sh` | `all_quality.sh` plus e2e tests and Docker build matrix. Used for release validation. |
+| `e2e.sh` | Thin wrapper that runs Playwright e2e tests from `tests/e2e/`. |
+| `docker_ci.sh` | Builds the Docker image and runs a smoke test against it. |
+| `test_aur_package.sh` | Builds the AUR package in an Arch container, then installs and boots it in a second container with port 8000 exposed (hang finish). |
+| `run_aur_with_radio.sh` | Like `test_aur_package.sh` but passes through the host serial device for testing with a real radio (hang finish). |
+
+</details>
+
+<details>
+<summary>scripts/build/</summary>
+
+| Script | Purpose |
+|--------|---------|
+| `publish.sh` | Full release ceremony: quality gate, version bump, changelog, frontend build, Docker multi-arch push, GitHub release. |
+| `release_common.sh` | Shared shell helpers (version validation, formatting) sourced by other build scripts. |
+| `package_release_artifact.sh` | Builds the prebuilt-frontend release zip attached to GitHub releases. |
+| `push_docker_multiarch.sh` | Builds and pushes multi-arch Docker images (amd64 + arm64). |
+| `create_github_release.sh` | Creates a GitHub release with changelog notes and the release artifact. |
+| `extract_release_notes.sh` | Extracts the latest version's notes from `CHANGELOG.md` for the release body. |
+| `collect_licenses.sh` | Gathers third-party license attributions into `LICENSES.md`. |
+| `print_frontend_licenses.cjs` | Helper that extracts frontend npm dependency licenses. |
+| `dump_api_specs.py` | Dumps the OpenAPI spec from the running backend (developer utility). |
+
+</details>
+
 ## E2E Testing

-E2E coverage exists, but it is intentionally not part of the normal development path.
+E2E tests exercise the full stack (backend + frontend + real radio hardware) via Playwright.

-These tests are only guaranteed to run correctly in a narrow subset of environments; they require a busy mesh with messages arriving constantly, an available autodetect-able radio, and a contact in the test database (which you can provide in `tests/e2e/.tmp/e2e-test.db` after an initial run). E2E tests are generally not necessary to run for normal development work.
+> [!WARNING]
+> E2E tests are **not part of the normal development path** — most contributors will never need to run them. They exist to catch integration issues that unit tests can't and generally only need to be run by maintainers.
+
+### Hardware requirements
+
+- A MeshCore radio connected via serial (auto-detected, or set `MESHCORE_SERIAL_PORT`)
+- The radio must be powered on and past its startup sequence before tests begin
+
+### Running

 ```bash
 cd tests/e2e
 npm install
-npx playwright test # headless
-npx playwright test --headed # you can probably guess
+npx playwright install chromium  # first time only
+npx playwright test              # headless
+npx playwright test --headed     # watch it run
+```
+
+The test harness starts its own uvicorn instance on port 8001 with a fresh temporary database. Your development server (port 8000) is unaffected.
+
+### Test tiers
+
+**Most tests (22 of 28) are fully self-contained.** They seed their own data via API calls or direct DB writes and need only a connected radio. These cover messaging, pagination, search, favorites, settings, fanout integrations, historical decryption, and all UI-only views.
+
+**Mesh-traffic tests (tagged `@mesh-traffic`)** wait up to 3 minutes for an incoming message from another node on the network. If no traffic arrives, they fail with an advisory that the failure may be RF conditions, not a bug. These are: `incoming-message` and `packet-feed` (second test only).
+
+**The partner-radio DM ACK test (tagged `@partner-radio`)** validates direct-route learning by sending a DM and waiting for an ACK. It requires a second radio in range that has your test radio in its contacts. Configure the partner node's public key and name via `E2E_PARTNER_RADIO_PUBKEY` and `E2E_PARTNER_RADIO_NAME`.
+
+### Making mesh-traffic tests reliable: the echo bot
+
+The most practical way to guarantee incoming traffic is to run an **echo bot on a second radio** monitoring a known channel. When the test suite starts a `@mesh-traffic` test, it sends a trigger message to that channel. If a bot on another radio is listening, it replies — generating the incoming RF packet the test needs within seconds instead of waiting for organic mesh traffic.
+
+The test suite sends `!echo please give incoming message` to the echo channel (default `#flightless`) at the start of each `@mesh-traffic` test. The trigger message is configurable via `E2E_ECHO_TRIGGER_MESSAGE`.
+
+Setup:
+1. Set up a second MeshCore radio within RF range of your test radio
+2. Run a RemoteTerm instance on the second radio
+3. Configure a bot on the second radio that monitors the echo channel and replies when it sees the trigger. Example bot code:
+   ```python
+   def bot(sender_name, sender_key, message_text, is_dm,
+           channel_key, channel_name, sender_timestamp, path):
+       if "!echo" in message_text.lower():
+           return f"[ECHO] {message_text}"
+       return None
+   ```
+4. The test suite calls `nudgeEchoBot()` automatically — no manual intervention needed
+
+Without the echo bot, `@mesh-traffic` tests rely on organic traffic from other nodes. In a quiet RF environment they will time out.
+
+### Environment variables
+
+All E2E environment configuration is centralized in `tests/e2e/helpers/env.ts` with defaults that work for the maintainer's test rig. Override via environment variables:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `MESHCORE_SERIAL_PORT` | auto-detect | Serial port for the test radio |
+| `E2E_ECHO_CHANNEL` | `#flightless` | Channel the echo bot monitors for traffic generation |
+| `E2E_ECHO_TRIGGER_MESSAGE` | `!echo please give incoming message` | Message sent to nudge the echo bot |
+| `E2E_PARTNER_RADIO_PUBKEY` | *(maintainer's test node)* | 64-char hex public key of a node that will ACK DMs from your radio |
+| `E2E_PARTNER_RADIO_NAME` | *(maintainer's test node)* | Display name of that node (used in UI assertions) |
+
+Example for a contributor with their own two-radio setup:
+
+```bash
+E2E_ECHO_CHANNEL="#mytest" \
+E2E_PARTNER_RADIO_PUBKEY="abcd1234...full64charhexkey..." \
+E2E_PARTNER_RADIO_NAME="MyTestNode" \
+npx playwright test
 ```

 ## Pull Request Expectations
--- a/scripts/quality/run_aur_with_radio.sh
+++ b/scripts/quality/run_aur_with_radio.sh
--- a/tests/e2e/helpers/api.ts
+++ b/tests/e2e/helpers/api.ts
@@ -282,14 +282,19 @@ export function deleteFanoutConfig(id: string): Promise<{ deleted: boolean }> {
 // --- Helpers ---

 /**
- * Ensure #flightless channel exists, creating it if needed.
+ * Ensure a channel exists by name, creating it if needed.
 * Returns the channel object.
 */
-export async function ensureFlightlessChannel(): Promise<Channel> {
+export async function ensureChannel(name: string): Promise<Channel> {
  const channels = await getChannels();
-  const existing = channels.find((c) => c.name === '#flightless');
+  const existing = channels.find((c) => c.name === name);
  if (existing) return existing;
-  return createChannel('#flightless');
+  return createChannel(name);
+}
+
+/** Convenience alias — ensures #flightless exists. */
+export async function ensureFlightlessChannel(): Promise<Channel> {
+  return ensureChannel('#flightless');
 }

 /**
--- a/tests/e2e/helpers/env.ts
+++ b/tests/e2e/helpers/env.ts
@@ -0,0 +1,46 @@
+/**
+ * Centralized E2E environment configuration.
+ *
+ * All environment-dependent values live here with sensible defaults that
+ * match the maintainer's test rig. Contributors can override any of these
+ * via environment variables to match their own hardware setup.
+ *
+ * See CONTRIBUTING.md § "E2E Testing" for what each variable means and
+ * how to set up a test environment from scratch.
+ */
+
+/**
+ * Channel used to trigger echo-bot traffic generation.
+ *
+ * The echo bot (running on a second "partner" radio) should monitor this
+ * channel and reply to any message, generating incoming RF traffic that
+ * mesh-traffic tests can observe. The channel is created automatically if
+ * it doesn't exist in the test database.
+ */
+export const E2E_ECHO_CHANNEL =
+  process.env.E2E_ECHO_CHANNEL ?? '#flightless';
+
+/**
+ * Message sent to the echo channel to nudge the bot into replying.
+ * The bot just needs to see *any* message and respond; the exact text
+ * doesn't matter as long as the bot doesn't filter it out.
+ */
+export const E2E_ECHO_TRIGGER_MESSAGE =
+  process.env.E2E_ECHO_TRIGGER_MESSAGE ?? '!echo please give incoming message';
+
+/**
+ * Public key (64-char hex) of a nearby node that will ACK direct messages
+ * sent by the test radio. This node must have the test radio's public key
+ * in its contact list. Used only by the partner-radio DM ACK test.
+ */
+export const E2E_PARTNER_RADIO_PUBKEY =
+  process.env.E2E_PARTNER_RADIO_PUBKEY ??
+  'ae92577bae6c269a1da3c87b5333e1bdb007e372b66e94204b9f92a6b52a62b1';
+
+/**
+ * Display name for the partner radio node above. Used in UI assertions
+ * (searching the sidebar, verifying the conversation header, etc.).
+ */
+export const E2E_PARTNER_RADIO_NAME =
+  process.env.E2E_PARTNER_RADIO_NAME ?? 'FlightlessDt\u{1F95D}';
+
--- a/tests/e2e/helpers/meshTrafficTest.ts
+++ b/tests/e2e/helpers/meshTrafficTest.ts
@@ -17,7 +17,8 @@
 * long polling timeout for environments without the bot.
 */
 import { test as base, expect } from '@playwright/test';
-import { ensureFlightlessChannel, sendChannelMessage } from './api';
+import { ensureChannel, sendChannelMessage } from './api';
+import { E2E_ECHO_CHANNEL, E2E_ECHO_TRIGGER_MESSAGE } from './env';

 export { expect };

@@ -26,15 +27,18 @@ const TRAFFIC_ADVISORY =
  'network. Failure may indicate insufficient mesh traffic rather than a bug.';

 /**
- * Best-effort: send a message to #flightless that triggers a remote echo
- * bot. If the bot is within radio range it will reply, generating the
- * incoming traffic the test needs. Failures are silently ignored — the
- * test will fall back to waiting for organic mesh traffic.
+ * Best-effort: send a message to the echo channel that triggers a remote
+ * echo bot on a partner radio. If the bot is within radio range it will
+ * reply, generating the incoming traffic the test needs. Failures are
+ * silently ignored — the test will fall back to waiting for organic mesh
+ * traffic.
+ *
+ * Configure the channel via E2E_ECHO_CHANNEL (default: #flightless).
 */
 export async function nudgeEchoBot(): Promise<void> {
  try {
-    const channel = await ensureFlightlessChannel();
-    await sendChannelMessage(channel.key, '!echo please give incoming message');
+    const channel = await ensureChannel(E2E_ECHO_CHANNEL);
+    await sendChannelMessage(channel.key, E2E_ECHO_TRIGGER_MESSAGE);
  } catch {
    // Best-effort — bot may not be reachable
  }
--- a/tests/e2e/specs/dev-flightless-direct-route.spec.ts
+++ b/tests/e2e/specs/dev-flightless-direct-route.spec.ts
@@ -6,50 +6,46 @@ import {
  getMessages,
  setContactRoutingOverride,
 } from '../helpers/api';
+import {
+  E2E_PARTNER_RADIO_PUBKEY,
+  E2E_PARTNER_RADIO_NAME,
+} from '../helpers/env';

-const DEV_ONLY_ENV = 'MESHCORE_ENABLE_DEV_FLIGHTLESS_ROUTE_E2E';
-const FLIGHTLESS_NAME = 'FlightlessDt🥝';
-const FLIGHTLESS_PUBLIC_KEY =
-  'ae92577bae6c269a1da3c87b5333e1bdb007e372b66e94204b9f92a6b52a62b1';
-const DEVELOPER_ONLY_NOTICE =
-  `Developer-only hardware test. This scenario assumes ${FLIGHTLESS_NAME} ` +
-  `(${FLIGHTLESS_PUBLIC_KEY.slice(0, 12)}...) is a nearby reachable node for the author's test radio. ` +
-  `Set ${DEV_ONLY_ENV}=1 to run it intentionally.`;
+const PARTNER_RADIO_NOTICE =
+  `Partner-radio hardware test. Requires a nearby node "${E2E_PARTNER_RADIO_NAME}" ` +
+  `(${E2E_PARTNER_RADIO_PUBKEY.slice(0, 12)}...) that will ACK DMs from this radio. ` +
+  `Set E2E_USE_PARTNER_RADIO_FOR_DM_ACK_TEST=1 to run, and override ` +
+  `E2E_PARTNER_RADIO_PUBKEY / E2E_PARTNER_RADIO_NAME to match your hardware.`;

 function escapeRegex(value: string): string {
  return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
 }

-test.describe('Developer-only direct-route learning for FlightlessDt🥝', () => {
-  test('zero-hop adverts then DM ACK learns a direct route', { tag: '@developer-only' }, async ({
+test.describe('Partner-radio direct-route learning via DM ACK', () => {
+  test('zero-hop adverts then DM ACK learns a direct route', { tag: '@partner-radio' }, async ({
    page,
  }, testInfo) => {
-    testInfo.annotations.push({ type: 'notice', description: DEVELOPER_ONLY_NOTICE });
-    if (process.env[DEV_ONLY_ENV] !== '1') {
-      test.skip(true, DEVELOPER_ONLY_NOTICE);
-    }
-
+    testInfo.annotations.push({ type: 'notice', description: PARTNER_RADIO_NOTICE });
    test.setTimeout(180_000);
-    console.warn(`[developer-only e2e] ${DEVELOPER_ONLY_NOTICE}`);

    try {
-      await deleteContact(FLIGHTLESS_PUBLIC_KEY);
+      await deleteContact(E2E_PARTNER_RADIO_PUBKEY);
    } catch {
      // Best-effort reset; the contact may not exist yet in the temp E2E DB.
    }

-    await createContact(FLIGHTLESS_PUBLIC_KEY, FLIGHTLESS_NAME);
-    await setContactRoutingOverride(FLIGHTLESS_PUBLIC_KEY, '');
+    await createContact(E2E_PARTNER_RADIO_PUBKEY, E2E_PARTNER_RADIO_NAME);
+    await setContactRoutingOverride(E2E_PARTNER_RADIO_PUBKEY, '');

    await expect
      .poll(
        async () => {
-          const contact = await getContactByKey(FLIGHTLESS_PUBLIC_KEY);
+          const contact = await getContactByKey(E2E_PARTNER_RADIO_PUBKEY);
          return contact?.direct_path_len ?? null;
        },
        {
          timeout: 10_000,
-          message: 'Waiting for recreated FlightlessDt contact to start in flood mode',
+          message: 'Waiting for recreated partner contact to start in flood mode',
        }
      )
      .toBe(-1);
@@ -74,22 +70,22 @@ test.describe('Developer-only direct-route learning for FlightlessDt🥝', () =>
    });

    const searchInput = page.getByLabel('Search conversations');
-    await searchInput.fill(FLIGHTLESS_PUBLIC_KEY.slice(0, 12));
-    await expect(page.getByText(FLIGHTLESS_NAME, { exact: true })).toBeVisible({
+    await searchInput.fill(E2E_PARTNER_RADIO_PUBKEY.slice(0, 12));
+    await expect(page.getByText(E2E_PARTNER_RADIO_NAME, { exact: true })).toBeVisible({
      timeout: 15_000,
    });
-    await page.getByText(FLIGHTLESS_NAME, { exact: true }).click();
+    await page.getByText(E2E_PARTNER_RADIO_NAME, { exact: true }).click();
    await expect
      .poll(() => page.url(), {
        timeout: 15_000,
-        message: 'Waiting for FlightlessDt conversation route to load',
+        message: 'Waiting for partner contact conversation route to load',
      })
-      .toContain(`#contact/${encodeURIComponent(FLIGHTLESS_PUBLIC_KEY)}`);
+      .toContain(`#contact/${encodeURIComponent(E2E_PARTNER_RADIO_PUBKEY)}`);
    await expect(
-      page.getByPlaceholder(new RegExp(`message\\s+${escapeRegex(FLIGHTLESS_NAME)}`, 'i'))
+      page.getByPlaceholder(new RegExp(`message\\s+${escapeRegex(E2E_PARTNER_RADIO_NAME)}`, 'i'))
    ).toBeVisible({ timeout: 15_000 });

-    const text = `dev-flightless-direct-${Date.now()}`;
+    const text = `dm-ack-route-test-${Date.now()}`;
    const input = page.getByPlaceholder(/message/i);
    await input.fill(text);
    await page.getByRole('button', { name: 'Send', exact: true }).click();
@@ -100,7 +96,7 @@ test.describe('Developer-only direct-route learning for FlightlessDt🥝', () =>
        async () => {
          const messages = await getMessages({
            type: 'PRIV',
-            conversation_key: FLIGHTLESS_PUBLIC_KEY,
+            conversation_key: E2E_PARTNER_RADIO_PUBKEY,
            limit: 25,
          });
          const match = messages.find((message) => message.outgoing && message.text === text);
@@ -108,7 +104,7 @@ test.describe('Developer-only direct-route learning for FlightlessDt🥝', () =>
        },
        {
          timeout: 90_000,
-          message: 'Waiting for FlightlessDt DM ACK',
+          message: 'Waiting for partner radio DM ACK',
        }
      )
      .toBeGreaterThan(0);
@@ -116,17 +112,17 @@ test.describe('Developer-only direct-route learning for FlightlessDt🥝', () =>
    await expect
      .poll(
        async () => {
-          const contact = await getContactByKey(FLIGHTLESS_PUBLIC_KEY);
+          const contact = await getContactByKey(E2E_PARTNER_RADIO_PUBKEY);
          return contact?.direct_path_len ?? null;
        },
        {
          timeout: 90_000,
-          message: 'Waiting for FlightlessDt route to update from flood to direct',
+          message: 'Waiting for partner radio route to update from flood to direct',
        }
      )
      .toBe(0);

-    const learnedContact = await getContactByKey(FLIGHTLESS_PUBLIC_KEY);
+    const learnedContact = await getContactByKey(E2E_PARTNER_RADIO_PUBKEY);
    expect(learnedContact?.direct_path ?? '').toBe('');

    await page.locator('[title="View contact info"]').click();