diff --git a/.planning/phases/08-schema-foundation-and-signing-page-safety/08-RESEARCH.md b/.planning/phases/08-schema-foundation-and-signing-page-safety/08-RESEARCH.md new file mode 100644 index 0000000..6bdf508 --- /dev/null +++ b/.planning/phases/08-schema-foundation-and-signing-page-safety/08-RESEARCH.md @@ -0,0 +1,417 @@ +# Phase 8: Schema Foundation and Signing Page Safety - Research + +**Researched:** 2026-03-21 +**Domain:** TypeScript discriminated union schema extension, Drizzle ORM migration, Next.js API route filtering, React client component type branching +**Confidence:** HIGH + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|-----------------| +| FIELD-01 | Agent can place text field markers on a PDF (for typed content like names, addresses, prices) | Schema type discriminant enables safe introduction of all new field types; signing page filter prevents non-client-sig types from surfacing as required unsigned; migration ensures no data loss on v1.0 documents | + + +--- + +## Summary + +Phase 8 is a pure safety gate — no user-visible feature is added. It makes the codebase safe to receive the new field types that Phases 9-13 will introduce. The single most dangerous change in v1.1 is adding new values to `SignatureFieldData` while the client signing page and the `/api/sign/[token]` GET route still treat every field as a required client-signature. Two concrete failure modes exist: (1) a client opens a signing session and sees an `agent-signature` field as a required unsigned overlay, panics, and cannot complete signing; (2) the `SigningPageClient` calls `setModalOpen(true)` for a non-drawable field type (text/checkbox/date), producing broken UX before any modal code for those types exists. Both failures are HIGH recovery cost in production — the signing session is broken and cannot self-heal. + +The technical work is narrow and well-scoped: add a `type` discriminant to the `SignatureFieldData` TypeScript interface, generate and run a Drizzle migration (the migration is a no-op in SQL terms — JSONB column already exists, no SQL change required for the type discriminant — but a migration for the `agentSignatureData` column on `users` IS needed), apply a backward-compatible runtime fallback (`field.type ?? 'client-signature'`) in every path that reads `signatureFields`, add a type filter in the `/api/sign/[token]` GET route before returning `signatureFields` to the client, and branch `SigningPageClient.tsx` so only `client-signature` types open the signature modal. None of these changes alter any existing user-visible behavior on v1.0 documents because all existing fields have no `type` property and will read as `'client-signature'` via the fallback. + +The Drizzle ORM discriminated union type annotation is applied to `.$type()` on the `signatureFields` JSONB column — this is a TypeScript-only annotation with no database DDL change. The only SQL migration needed for Phase 8 is `ALTER TABLE "users" ADD COLUMN "agent_signature_data" TEXT` (needed by Phase 11, but adding it now keeps the schema extension atomic). However, examining the roadmap plan files, the `agentSignatureData` column is planned for Phase 11's migration. Phase 8's migration may therefore be ZERO SQL statements (the discriminant lives in the TypeScript interface only) unless the team decides to pre-add the users column. + +**Primary recommendation:** Extend `SignatureFieldData` with `type` discriminant in `schema.ts`, add a backward-compat fallback in all read paths, filter agent-signature fields from the sign GET route, and branch `SigningPageClient.tsx` by field type — all in a single atomic deployment. The Drizzle migration for Phase 8 is likely a zero-SQL migration (TypeScript type change only); generate it anyway to ensure the meta snapshot stays in sync. + +--- + +## Standard Stack + +### Core (All Existing — No New Dependencies) + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| Drizzle ORM | ^0.45.1 | TypeScript schema definition, JSONB type annotation, migration generation | Already in project; `.$type()` method annotates JSONB column TypeScript type without DDL change | +| drizzle-kit | ^0.31.10 | Migration generation (`npm run db:generate`) and migration execution (`npm run db:migrate`) | Already in project; generates 0006_*.sql diff | +| Next.js | 16.2.0 | API route (server-side filter) and React client component (type branching) | Already in project | +| TypeScript | (project version) | Discriminated union type definition | Already in project | +| postgres | ^3.4.8 | Database driver | Already in project | + +### No New Dependencies + +Phase 8 adds zero new npm packages. All changes are to TypeScript type definitions, runtime logic in existing files, and a Drizzle migration. + +**Installation:** +```bash +# No new packages needed +``` + +--- + +## Architecture Patterns + +### Recommended Project Structure + +No new files or directories are created in Phase 8. All changes are modifications to existing files: + +``` +src/lib/db/schema.ts # Modified: SignatureFieldData type + discriminant +src/app/api/sign/[token]/route.ts # Modified: filter signatureFields before GET response +src/app/sign/[token]/_components/SigningPageClient.tsx # Modified: type branching in field render + click handler +drizzle/0006_*.sql # New: migration file (may be zero-SQL or users column add) +``` + +### Pattern 1: TypeScript Discriminated Union on JSONB .$type() + +**What:** Add a `type` literal union to the `SignatureFieldData` interface. Drizzle's `.$type()` annotation changes only the TypeScript type of the column — it does NOT generate any SQL DDL change. The JSONB column remains unchanged in the database. + +**When to use:** Any time a JSONB column stores an array of objects that need type discrimination at the TypeScript layer without a DB schema change. + +**Example:** +```typescript +// src/lib/db/schema.ts — AFTER Phase 8 + +export type SignatureFieldType = + | 'client-signature' + | 'initials' + | 'text' + | 'checkbox' + | 'date' + | 'agent-signature'; + +export interface SignatureFieldData { + id: string; + page: number; // 1-indexed + x: number; // PDF user space, bottom-left origin, points + y: number; // PDF user space, bottom-left origin, points + width: number; // PDF points (default: 144 — 2 inches) + height: number; // PDF points (default: 36 — 0.5 inches) + type?: SignatureFieldType; // Optional for backward compat with v1.0 documents +} + +// Helper: always use this to read field type — never read .type directly +export function getFieldType(field: SignatureFieldData): SignatureFieldType { + return field.type ?? 'client-signature'; +} + +// Type predicates for common checks +export function isClientVisibleField(field: SignatureFieldData): boolean { + const t = getFieldType(field); + return t !== 'agent-signature'; +} + +// In the documents table — .$type annotation only (no SQL change) +signatureFields: jsonb("signature_fields").$type(), +``` + +**Key insight:** `type` is declared as `type?: SignatureFieldType` (optional). This is backward-compatible: existing documents with no `type` property in their JSONB fields will NOT fail JSON parsing, and the `getFieldType()` helper coalesces `undefined` to `'client-signature'`. + +### Pattern 2: Server-Side Filter Before Response + +**What:** Before returning `signatureFields` in the GET `/api/sign/[token]` response, filter to client-visible types only. + +**When to use:** Any API route that returns field data to an unauthenticated client (the signing page). + +**Exact location:** `/src/app/api/sign/[token]/route.ts` — the current line 88: +```typescript +// BEFORE (line 88 — current, unguarded): +signatureFields: doc.signatureFields ?? [], + +// AFTER (Phase 8): +signatureFields: (doc.signatureFields ?? []).filter(isClientVisibleField), +``` + +This single line is the security-critical change. After this filter, `agent-signature` fields are permanently invisible to the client signing page regardless of what field types future phases add. + +### Pattern 3: Type-Branching in SigningPageClient + +**What:** The `handleFieldClick` callback and the field overlay render must branch on field type. Non-client-signature types must not open the signature modal. + +**When to use:** Any UI component that renders `SignatureFieldData[]` received from the server. + +**Example pattern (Phase 8 minimum — no UI changes needed yet):** +```typescript +// In SigningPageClient.tsx + +// handleFieldClick — guard against non-modal types +const handleFieldClick = useCallback( + (fieldId: string) => { + const field = signatureFields.find((f) => f.id === fieldId); + if (!field) return; + const fieldType = getFieldType(field); + // Only client-signature opens the modal in Phase 8 + // Other types will be handled in Phase 10 (initials, text, etc.) + if (fieldType !== 'client-signature') return; + if (signedFields.has(fieldId)) return; + setActiveFieldId(fieldId); + setModalOpen(true); + }, + [signatureFields, signedFields] +); + +// handleSubmit — only require signatures for client-signature fields +const handleSubmit = useCallback(async () => { + const clientSigFields = signatureFields.filter( + (f) => getFieldType(f) === 'client-signature' + ); + if (signedFields.size < clientSigFields.length || submitting) return; + // ... rest of submit unchanged +}, [signatureFields, signedFields.size, submitting, token]); +``` + +**Note on progress bar:** `SigningProgressBar` receives `total` and `signed` counts. After Phase 8, `total` must be `signatureFields.filter(f => getFieldType(f) === 'client-signature').length` — not `signatureFields.length`. Because the API route now filters out agent-signature fields before sending, the client will never receive them. This is handled transparently. The filter in the GET route means `signatureFields` on the client already only contains client-visible types — so the `SigningPageClient` type branching in Phase 8 is a defense-in-depth guard, not the primary protection. + +### Pattern 4: Drizzle Migration for Phase 8 + +**What:** The TypeScript discriminant requires no SQL DDL change — `signatureFields` is already `jsonb`. The migration generated by `drizzle-kit generate` will reflect the TypeScript change but produce an empty SQL file (or just a comment). + +**Important:** Run `npm run db:generate` after updating `schema.ts`. Even if the migration is empty SQL, it updates the Drizzle meta snapshot (the JSON file in `drizzle/meta/`) to stay in sync with the schema. Skipping this causes future `db:generate` runs to produce incorrect diffs. + +**The migration SQL will be:** +```sql +-- No DDL change needed for Phase 8 discriminant (JSONB type annotation only) +-- This is an intentionally empty migration to advance the Drizzle snapshot +``` + +If the team decides to pre-add `agent_signature_data` to `users` (for Phase 11 prep), the migration would be: +```sql +ALTER TABLE "users" ADD COLUMN "agent_signature_data" text; +``` + +**DECISION NEEDED:** The roadmap plan files assign `agentSignatureData` to Phase 11. Do NOT pre-add it in Phase 8 unless explicitly decided. Keep Phase 8 minimal. + +### Anti-Patterns to Avoid + +- **Reading `field.type` directly without fallback:** Always use `getFieldType(field)` or `field.type ?? 'client-signature'`. Direct access produces `undefined` on v1.0 documents and TypeScript will not catch this at all entry points. +- **Filtering on the client instead of the server:** The `agent-signature` filter MUST happen in the API route before the response is sent. Filtering only in `SigningPageClient.tsx` is insufficient — an attacker can call the API directly and see field coordinates. +- **Changing `type` from optional to required:** Making `type` required in `SignatureFieldData` breaks TypeScript compilation on every existing call site that constructs a `SignatureFieldData` object (including `FieldPlacer.tsx` `handleDragEnd`). Keep `type` optional; add `getFieldType()` helper for safe reading. +- **Modifying `embed-signature.ts` or the POST `/api/sign/[token]` route:** Phase 8 does NOT touch the signing POST handler or PDF embedding pipeline. Those are working correctly and only handle client-signature fields today. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| TypeScript type narrowing for JSONB union | Custom type guard per field | `getFieldType()` helper + type predicate functions exported from `schema.ts` | Single source of truth; all consumers import the same fallback logic | +| API field filtering | Field visibility logic inline in route | `isClientVisibleField()` pure function exported from `schema.ts` | Testable, consistent across all routes that return fields to clients | +| Drizzle meta sync | Manual edits to meta JSON | `npm run db:generate` after every schema.ts change | Drizzle meta is machine-generated; manual edits corrupt future diffs | +| Backward-compat runtime coercion | Migration that sets `type = 'client-signature'` on all existing JSONB rows | Optional TypeScript field with fallback in `getFieldType()` | No data migration risk; existing rows work immediately; all new rows get explicit type | + +**Key insight:** The entire Phase 8 safety guarantee comes from two pure TypeScript functions (`getFieldType` and `isClientVisibleField`) that are easy to unit-test and can be imported by every consumer. Don't scatter the fallback logic across callsites. + +--- + +## Common Pitfalls + +### Pitfall 1: Breaking Active Signing Sessions +**What goes wrong:** Any document that was prepared and sent BEFORE Phase 8 ships has `signatureFields` JSONB with no `type` property on each field object. If Phase 8 changes the TypeScript interface to require `type` (non-optional), TypeScript may compile but the `getFieldType()` fallback handles this correctly — however, if any code path branches on `field.type === 'client-signature'` with a strict equality check and `field.type` is `undefined`, the branch is never taken. +**Why it happens:** Confusion between optional TypeScript field and runtime undefined value. +**How to avoid:** Always coerce via `getFieldType(field)` which returns `field.type ?? 'client-signature'`. Export this function from `schema.ts` so there is only one implementation. +**Warning signs:** TypeScript strict mode warnings on `field.type` comparisons; unit tests on v1.0-style field objects (no type property) failing. + +### Pitfall 2: Drizzle Meta Snapshot Out of Sync +**What goes wrong:** After updating `SignatureFieldData` in `schema.ts`, the developer skips `npm run db:generate` because "there's no SQL change." The next `db:generate` run (in Phase 9 or later) produces a diff against the stale meta snapshot and generates incorrect SQL. +**Why it happens:** Drizzle stores the TypeScript schema representation as a JSON snapshot in `drizzle/meta/`. Even TypeScript-only changes to `.$type()` annotations update this snapshot. +**How to avoid:** Always run `npm run db:generate` after any `schema.ts` change, even if you expect zero SQL output. Commit the resulting migration file (even if it has no SQL statements). +**Warning signs:** `drizzle-kit generate` produces a migration that re-adds columns that already exist. + +### Pitfall 3: The Filter Is in the Wrong Place +**What goes wrong:** Developer adds the `agent-signature` filter only in `SigningPageClient.tsx` (client component) but leaves the GET `/api/sign/[token]` route returning unfiltered `signatureFields`. The filter works in the browser, but the field coordinates and IDs of agent-signature fields are still leaked to the network. +**Why it happens:** The signing page is the visible failure point, so developers focus on it. +**How to avoid:** Filter in the API route (server-side, line 88) as the primary protection. The `SigningPageClient` type guard is defense-in-depth only. +**Warning signs:** Network tab in browser DevTools shows agent-signature field objects in the GET response JSON. + +### Pitfall 4: Breaking the Submit Logic +**What goes wrong:** `handleSubmit` checks `signedFields.size < signatureFields.length`. If an `initials` or other non-signature type is returned in `signatureFields` (which Phase 8's server filter prevents), this check would require the client to "sign" non-signable fields before enabling Submit. Because the filter is in the API route, this is already safe — but if the filter is ever removed or incomplete, Submit becomes permanently disabled. +**Why it happens:** The submit guard counts total fields, not just client-signature fields. +**How to avoid:** Update `handleSubmit` to count only `client-signature` fields in its completeness check: `signatureFields.filter(f => getFieldType(f) === 'client-signature').length`. Even though Phase 8's server filter makes this moot today, it makes the component resilient to future API changes. +**Warning signs:** Submit button stays disabled even after all visible signature fields are completed. + +### Pitfall 5: Forgetting the `SigningProgressBar` Total +**What goes wrong:** `SigningProgressBar` is passed `total={signatureFields.length}` as a prop. If for any reason a non-client-signature field appears in the array, the total is inflated and "X of Y signed" shows a count the client can never reach. +**Why it happens:** The progress bar total is passed from `SigningPageClient` which receives the full array. +**How to avoid:** Calculate `total` as `signatureFields.filter(f => getFieldType(f) === 'client-signature').length`. This is currently moot because the API route filters, but is correct defensive coding. +**Warning signs:** Progress bar shows "0 of 3" when only 2 signature fields are visible. + +--- + +## Code Examples + +Verified patterns from codebase inspection (2026-03-21): + +### Current SignatureFieldData (schema.ts — to be extended) +```typescript +// Source: /src/lib/db/schema.ts (current — no type discriminant) +export interface SignatureFieldData { + id: string; + page: number; // 1-indexed + x: number; // PDF user space, bottom-left origin, points + y: number; // PDF user space, bottom-left origin, points + width: number; // PDF points (default: 144 — 2 inches) + height: number; // PDF points (default: 36 — 0.5 inches) +} +``` + +### Extended SignatureFieldData with Discriminant +```typescript +// Source: Phase 8 target schema +export type SignatureFieldType = + | 'client-signature' + | 'initials' + | 'text' + | 'checkbox' + | 'date' + | 'agent-signature'; + +export interface SignatureFieldData { + id: string; + page: number; + x: number; + y: number; + width: number; + height: number; + type?: SignatureFieldType; // Optional — v1.0 docs have no type; fallback = 'client-signature' +} + +export function getFieldType(field: SignatureFieldData): SignatureFieldType { + return field.type ?? 'client-signature'; +} + +export function isClientVisibleField(field: SignatureFieldData): boolean { + return getFieldType(field) !== 'agent-signature'; +} +``` + +### Current Unguarded Line in sign GET route (the critical line) +```typescript +// Source: /src/app/api/sign/[token]/route.ts line 83-92 (current) +return NextResponse.json({ + status: 'pending', + document: { + id: doc.id, + name: doc.name, + signatureFields: doc.signatureFields ?? [], // ← UNGUARDED — returns all fields including agent-signature + preparedFilePath: doc.preparedFilePath, + }, + expiresAt: new Date(payload.exp * 1000).toISOString(), +}); +``` + +### Fixed sign GET route (Phase 8 target) +```typescript +// Source: Phase 8 target — /src/app/api/sign/[token]/route.ts +import { isClientVisibleField } from '@/lib/db/schema'; + +// Line 88 replacement: +signatureFields: (doc.signatureFields ?? []).filter(isClientVisibleField), +``` + +### Current SigningPageClient handleFieldClick (to be guarded) +```typescript +// Source: /src/app/sign/[token]/_components/SigningPageClient.tsx (current) +const handleFieldClick = useCallback( + (fieldId: string) => { + if (signedFields.has(fieldId)) return; + setActiveFieldId(fieldId); + setModalOpen(true); // ← opens modal for ANY field, no type check + }, + [signedFields] +); +``` + +### Fixed SigningPageClient handleFieldClick (Phase 8 target) +```typescript +// Phase 8 target — defense-in-depth guard (primary protection is the server filter) +import { getFieldType } from '@/lib/db/schema'; + +const handleFieldClick = useCallback( + (fieldId: string) => { + const field = signatureFields.find((f) => f.id === fieldId); + if (!field) return; + if (getFieldType(field) !== 'client-signature') return; // type guard + if (signedFields.has(fieldId)) return; + setActiveFieldId(fieldId); + setModalOpen(true); + }, + [signatureFields, signedFields] +); +``` + +### Drizzle Migration Pattern (additive, no data loss) +```typescript +// After schema.ts change, run: npm run db:generate +// Expected generated SQL (0006_*.sql) — may be empty or only a comment: +// --> statement-breakpoint +// (no DDL change for JSONB type annotation) + +// If pre-adding agentSignatureData (Phase 11 scope — decision needed): +// ALTER TABLE "users" ADD COLUMN "agent_signature_data" text; +``` + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| Flat interface, no type discriminant | Discriminated union with optional `type` field | Phase 8 | Enables safe multi-type field arrays without breaking existing data | +| All fields sent to signing page | Server-side filter for client-visible types only | Phase 8 | Agent-signature fields permanently invisible to clients | +| Modal opens for every field | Type-branched click handler | Phase 8 | Non-drawable field types cannot accidentally trigger the signature modal | + +**Deprecated/outdated after Phase 8:** +- `signatureFields: doc.signatureFields ?? []` (unguarded) in sign GET route: replaced by filtered version + +--- + +## Open Questions + +1. **Should Phase 8 add the `agentSignatureData` column to `users`?** + - What we know: The roadmap assigns this to Phase 11. Phase 8 migration is otherwise zero-SQL. + - What's unclear: Is there value in pre-adding it (avoids a second migration later) vs. keeping Phase 8 minimal? + - Recommendation: Keep Phase 8 minimal. Do NOT pre-add `agentSignatureData`. Add it in Phase 11 where it belongs. Phase 8's migration is zero-SQL (JSONB type annotation change only) — that is fine and expected. + +2. **Do `initials` fields need any treatment in Phase 8's signing page?** + - What we know: The roadmap says Phase 8 filters `agent-signature` from the client. Phase 10 handles `initials` end-to-end (including the signing page UI for initials capture). + - What's unclear: Should Phase 8's `handleFieldClick` guard against ALL non-`client-signature` types, or just `agent-signature`? + - Recommendation: Guard against everything except `client-signature` in Phase 8 (`if (getFieldType(field) !== 'client-signature') return`). This is more future-proof and makes Phase 10 easier — Phase 10 will expand the click handler to handle `initials` explicitly. + +3. **Is the Drizzle meta snapshot the only artifact to update?** + - What we know: `drizzle/meta/` contains JSON snapshot files. The schema change affects the snapshot even with no SQL output. + - What's unclear: Whether `drizzle-kit generate` produces a meaningful SQL file or a completely empty one when only a `.$type()` annotation changes. + - Recommendation: Run `npm run db:generate` and commit whatever it produces. If it's empty, commit it anyway to keep the snapshot current. + +--- + +## Validation Architecture + +> `workflow.nyquist_validation` is not present in `.planning/config.json` (config has `workflow.research`, `workflow.plan_check`, `workflow.verifier` — no `nyquist_validation` key). Skipping this section. + +--- + +## Sources + +### Primary (HIGH confidence) +- `/Users/ccopeland/temp/red/teressa-copeland-homes/src/lib/db/schema.ts` (codebase inspection 2026-03-21) — `SignatureFieldData` interface confirmed to have no `type` field; JSONB annotation via `.$type()` confirmed +- `/Users/ccopeland/temp/red/teressa-copeland-homes/src/app/api/sign/[token]/route.ts` lines 83-92 (codebase inspection 2026-03-21) — unguarded `doc.signatureFields ?? []` on line 88 confirmed +- `/Users/ccopeland/temp/red/teressa-copeland-homes/src/app/sign/[token]/_components/SigningPageClient.tsx` (codebase inspection 2026-03-21) — `handleFieldClick` opens modal for every field with no type check confirmed; `handleSubmit` checks `signedFields.size < signatureFields.length` confirmed +- `/Users/ccopeland/temp/red/.planning/research/SUMMARY.md` (project v1.1 pre-research 2026-03-21) — Phase 8 rationale, critical pitfalls #1 and #3, exact file/line references confirmed +- `/Users/ccopeland/temp/red/.planning/research/STACK.md` (project v1.1 pre-research 2026-03-21) — existing stack confirmed; no new dependencies for Phase 8 +- `/Users/ccopeland/temp/red/.planning/ROADMAP.md` (2026-03-21) — Phase 8 plan structure confirmed (08-01-PLAN.md + 08-02-PLAN.md) +- Drizzle ORM docs (`.$type()` on jsonb) — TypeScript-only annotation, no DDL generated — HIGH confidence (confirmed via codebase usage pattern) + +### Secondary (MEDIUM confidence) +- Drizzle ORM GitHub/docs — `db:generate` updates meta snapshot even for TypeScript-only changes (standard Drizzle behavior; verified via project migration pattern in `drizzle/meta/`) + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — all libraries already in use; no new dependencies +- Architecture: HIGH — based on actual codebase files; specific line numbers identified; minimal surface area +- Pitfalls: HIGH — grounded in actual file inspection; failure modes are deterministic, not speculative + +**Research date:** 2026-03-21 +**Valid until:** 2026-04-20 (stable stack; no external dependencies that could change)