docs: complete project research
This commit is contained in:
Chandler Copeland
@@ -1,185 +1,206 @@
|
||||
# Project Research Summary
|
||||
|
||||
**Project:** Teressa Copeland Homes — v1.1 Smart Document Preparation
|
||||
**Domain:** Real estate agent website + PDF document signing portal
|
||||
**Researched:** 2026-03-21
|
||||
**Project:** teressa-copeland-homes
|
||||
**Domain:** Real estate agent website + AI-assisted document signing portal
|
||||
**Researched:** 2026-04-03
|
||||
**Milestone scope:** v1.1 (AI field placement, expanded field types, agent signature, filled preview) + v1.2 (multi-signer support, Docker production deployment)
|
||||
**Confidence:** HIGH
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This is a v1.1 feature expansion of an existing, working Next.js 15 real estate document signing app. The v1.0 codebase is already validated — it uses Drizzle ORM, local PostgreSQL, `@cantoo/pdf-lib` for PDF writing, `react-pdf` for client-side rendering, Auth.js v5, and `signature_pad` for canvas signatures. The v1.1 additions are: AI-assisted field placement via GPT-4o-mini, five new field types (text, checkbox, initials, date, agent-signature), agent saved signature with a draw-once-reuse workflow, and a filled document preview before sending. The minimal dependency delta is two new packages: `openai@^6.32.0` and optionally `unpdf@^1.4.0` — though `pdfjs-dist` is already installed as a transitive dependency of `react-pdf` and can serve the server-side text extraction role via its legacy build.
|
||||
This is a solo real estate agent's custom document signing portal built on Next.js 15 + Drizzle ORM + PostgreSQL (Neon). The core v1.0 feature set — PDF upload, drag-drop field placement, email-link signing, presigned download — is complete. Two milestones of new capability are being planned. The v1.1 milestone adds AI-assisted field detection (GPT-4.1 + PDF text extraction), five new field types (checkbox, initials, date, agent signature, text), a saved agent signature, and a filled-document preview before send. The v1.2 milestone extends single-signer to multi-signer (parallel, any-order) and adds production Docker deployment with SMTP fix. Both milestones require zero new npm packages beyond `openai` (already installed for v1.1) — all capability is extension of the existing stack.
|
||||
|
||||
The recommended build order is anchored by a schema-first phase. The `SignatureFieldData` type currently has no `type` discriminant — every field is treated identically as a client signature. Adding new field types without simultaneously updating both the schema AND the client signing page would break any in-flight signing session. The architecture research maps out an explicit 8-step dependency chain. For AI field placement, the correct approach uses `pdfjs-dist` for server-side text extraction (not vision), then GPT-4o-mini for semantic label classification — raw vision-based bounding box inference returns accurate coordinates less than 3% of the time. The OpenAI integration must use a manually defined JSON schema for structured output; the `zodResponseFormat` helper is broken with Zod v4 (confirmed open bug).
|
||||
The most important architectural constraint for both milestones: the current signing flow has load-bearing single-signer assumptions throughout (one token per document, first token claim marks the document Signed, `documents.status` tracks one signer's journey). Multi-signer requires a deliberate schema-first approach — add the `signers` JSONB column, `signerEmail` to `signingTokens`, and a completion detection rewrite before touching any UI. Building the send route or signing UI before the schema is solid will create hard-to-unwind bugs. For Docker, the single non-obvious pitfall is that `NEXT_PUBLIC_BASE_URL` is baked at build time — signing URLs will point to localhost unless the variable is renamed to a server-only name before the production image is built.
|
||||
|
||||
The key risk cluster is around the AI coordinate pipeline and signing page integrity. OpenAI returns percentage-based coordinates; `@cantoo/pdf-lib` expects PDF user-space points with a bottom-left origin — a Y-axis inversion that will silently produce wrong field positions without a dedicated conversion utility and unit test. A second risk is that agent-signature fields must be filtered from the `signatureFields` array sent to clients — the exact unguarded line (`/src/app/api/sign/[token]/route.ts` line 88) is identified in pitfalls research. Preview PDFs must use versioned paths separate from the final prepared PDF to maintain legal integrity between what the agent reviewed and what the client signs.
|
||||
For v1.1, AI field placement must use the text-extraction approach (pdfjs-dist `getTextContent()` → GPT-4.1 for label classification → coordinates from text bounding boxes). Vision-based coordinate inference from PDF images has under 3% accuracy in published benchmarks and is not production-viable. The hybrid text+AI approach is the pattern used by Apryse, Instafill, and DocuSign Iris. On Utah standard forms (REPC, listing agreements), which have consistent label patterns, accuracy should be 90%+. The fallback — scanned/image-based PDFs — should degrade gracefully to manual drag-drop with a clear agent-facing message.
|
||||
|
||||
---
|
||||
|
||||
## Key Findings
|
||||
|
||||
### Recommended Stack
|
||||
|
||||
The v1.0 stack is unchanged and validated. See `STACK.md` for full version details.
|
||||
The existing stack handles every v1.1 and v1.2 requirement without new packages. `openai@6.32.0` (already installed) covers AI field placement. `@cantoo/pdf-lib@2.6.3` handles all five new field types and PDF assembly for both single-signer and multi-signer workflows. `signature_pad@5.1.3` (already installed) handles agent signature capture. `react-pdf@10.4.1` handles filled document preview. Docker deployment requires no npm changes — it is a Dockerfile + docker-compose.yml infrastructure concern.
|
||||
|
||||
**New dependencies for v1.1:**
|
||||
- `openai@^6.32.0`: Official SDK, TypeScript-native structured output for GPT-4o-mini — use manual `json_schema` response_format, NOT `zodResponseFormat` (broken with Zod v4, confirmed open GitHub issues #1540, #1602, #1709)
|
||||
- `pdfjs-dist` legacy build (already installed): Server-side PDF text extraction via `pdfjs-dist/legacy/build/pdf.mjs` — no new dependency needed if using this path
|
||||
**Core technologies:**
|
||||
- `openai@6.32.0`: GPT-4.1 structured extraction via manual JSON schema — not `zodResponseFormat`, which is broken with Zod v4 (confirmed open bug)
|
||||
- `pdfjs-dist` (legacy build, already installed): PDF text extraction with bounding boxes for AI field placement pipeline
|
||||
- `@cantoo/pdf-lib@2.6.3`: All new field types (text, checkbox, initials, date); agent signature via `embedPng()` + `drawImage()`; coordinate system: y=0 at page bottom (transform from pdfjs-dist y=0-at-top)
|
||||
- `signature_pad@5.1.3`: Agent signature canvas — use as plain class with `useRef<HTMLCanvasElement>` in React, not a React wrapper
|
||||
- `react-pdf@10.4.1`: Filled document preview from `ArrayBuffer`; always copy buffer before passing (detachment bug in v7+)
|
||||
- `@vercel/blob`: File storage — installed but currently unused (dead dependency risk; see Gaps)
|
||||
- `nodemailer@7.0.13`: SMTP email for signing links and completion notifications
|
||||
- `node:20-slim` (Debian-based): Docker base image — required for `@napi-rs/canvas` glibc native binary compatibility; do NOT use Alpine (musl libc incompatible)
|
||||
|
||||
**Existing stack components covering all v1.1 needs:**
|
||||
- `@cantoo/pdf-lib@2.6.3`: All five new field types (text, checkbox, initials, date, agent-signature) supported natively via `createTextField`, `createCheckBox`, `drawImage` APIs
|
||||
- `signature_pad@5.1.3`: Agent signature canvas — use `useRef<HTMLCanvasElement>` + `useEffect` pattern directly; do NOT add `react-signature-canvas` (alpha wrapper)
|
||||
- `react-pdf@10.4.1`: Filled preview rendering — pass `ArrayBuffer` directly; copy the buffer before passing to avoid detachment issue (known bug #1657)
|
||||
- `@vercel/blob@2.3.1` + Drizzle ORM: Agent signature storage — architecture research recommends TEXT column on `users` table for 2-8KB base64 PNG; no new file storage needed
|
||||
**Critical version constraint:** `zodResponseFormat` from `openai/helpers/zod` throws runtime exceptions with Zod v4.x. Use `response_format: { type: "json_schema", ... }` with a hand-written schema.
|
||||
|
||||
### Expected Features
|
||||
|
||||
All v1.1 features are P1 (must-have for launch). Research confirms the full feature set is aligned with industry standard behavior across DocuSign, dotloop, and SkySlope DigiSign.
|
||||
**Must have (table stakes — all major real estate e-sig tools provide these):**
|
||||
- Initials field type — every Utah standard form (REPC, listing agreement) has per-page initials lines
|
||||
- Date field (auto-stamp only, not a calendar picker) — records actual signing timestamp; client cannot type a date
|
||||
- Checkbox field type — boolean elections throughout Utah REPC and addenda
|
||||
- Agent signature field type — pre-filled during agent signing flow; read-only for client
|
||||
- Agent saved signature (draw once, reuse) — DocuSign "Adopted Signature" pattern; re-drawing per document is a daily friction point
|
||||
- Agent signs before sending (routing order: agent first, client second) — industry convention in real estate
|
||||
- Filled document preview before send — prevents wrong-version sends; agents expect to see what client sees
|
||||
|
||||
**Must have (table stakes):**
|
||||
- Initials field type — every Utah standard form (REPC, listing agreement, addenda) has per-page initials lines; missing this makes the app unusable for standard Utah workflows
|
||||
- Date field (auto-stamp, read-only) — "Date Signed" pattern; auto-populated at signing session completion; client never types a date; legally important
|
||||
- Checkbox field type — Utah REPC uses boolean checkboxes throughout (mediation clauses, contingency elections, disclosure acknowledgments)
|
||||
- Agent saved signature — draw once, reuse across documents; the "Adopted Signature" pattern in every major real estate e-sig tool
|
||||
- Agent signs first workflow — industry convention: agent at routing order 1, client at routing order 2; confirmed by DocuSign community docs
|
||||
- Filled document preview with Send gating — prevents the most-cited mistake (sending wrong document version); Send button lives in preview
|
||||
**Should have (differentiators for this solo-agent, Utah-forms workflow):**
|
||||
- AI field placement (text extraction + GPT-4.1 label classification) — eliminates manual drag-drop for known Utah forms; 90%+ accuracy on standard forms
|
||||
- AI pre-fill from client profile data (name, property address, date) — populates obvious fields; agent reviews in PreparePanel
|
||||
- Property address field on client profile — enables AI pre-fill to be transaction-specific
|
||||
|
||||
**Should have (differentiators):**
|
||||
- AI field placement via gpt-4o-mini + text extraction — eliminates manual drag-drop session; accuracy 90%+ on structured Utah forms with predictable label patterns ("Buyer's Signature", "Date", "Initial Here")
|
||||
- AI pre-fill from client profile — maps client name, email, property address to text fields; low hallucination risk (structured profile data, not free-text inference)
|
||||
- Property address field on client profile — enables AI pre-fill to be property-specific; simple schema addition
|
||||
**Defer to v2+:**
|
||||
- AI confidence scores surfaced to agent — adds UI complexity; preview step catches gaps instead
|
||||
- Template save from AI placement — high value but requires template management UI; validate AI accuracy first
|
||||
- Per-document agent signature redraw — adds decision fatigue; profile settings only is the right UX
|
||||
|
||||
**Defer to v1.2+:**
|
||||
- AI confidence display to agent — adds UI noise; agent can see and correct in preview instead
|
||||
- Template save from AI placement — high value but requires template management UI; defer until AI accuracy is validated
|
||||
- Multiple agent signature fields per document — needs UX design; defer
|
||||
**Multi-signer (v1.2 must-haves per PROJECT.md):**
|
||||
- Parallel signing in any order — no sequencing enforcement
|
||||
- Per-signer field isolation — each signer sees only fields tagged to their email
|
||||
- Completion detection after all signers finish — agent notified only when last signer completes
|
||||
- Final PDF distributed to all parties on completion
|
||||
|
||||
### Architecture Approach
|
||||
|
||||
The v1.1 architecture is an incremental extension of the existing system — not a rewrite. Seven new files are created (two server-only AI lib files, three API routes, two client components). Eight existing files are modified with targeted additions. The critical architectural constraint: the existing client signing flow (`embed-signature.ts`, signing token route, `SignatureModal.tsx`) must not be altered. Agent-sig and text/checkbox/date fields are baked into the prepared PDF before the client opens the signing link. The client signing page handles only `client-signature` and `initials` field types.
|
||||
|
||||
See `ARCHITECTURE.md` for complete component boundaries, data flow diagrams, and the full 8-step build order.
|
||||
The system is a clean single-signer flow extended to multi-signer via additive schema changes without breaking existing documents. The key architectural principle: `signingTokens` is the source of truth for signer identity and completion state — never derive this from the `signatureFields` JSONB array. The per-signer PDF accumulation pattern (each signer embeds into the running `signedFilePath`, protected by a Postgres advisory lock) eliminates the need for a separate merge step and works with the existing `embedSignatureInPdf()` function unchanged.
|
||||
|
||||
**Major components:**
|
||||
1. `lib/ai/extract-text.ts` + `lib/ai/field-placement.ts` (NEW, server-only) — pdfjs-dist legacy build for text extraction; GPT-4o-mini structured output with manual JSON schema; `server-only` import guard prevents accidental client bundle inclusion
|
||||
2. `POST /api/documents/[id]/ai-prepare` (NEW) — orchestrates extract + AI call + coordinate conversion (percentage to PDF points using actual page dimensions)
|
||||
3. `GET/PUT /api/agent/signature` (NEW) — stores agent signature as base64 PNG TEXT column on `users` table; always auth-gated
|
||||
4. `POST /api/documents/[id]/preview` (NEW) — reuses existing `preparePdf` in preview mode; writes to versioned `_preview_{timestamp}.pdf`; streams bytes directly; never overwrites final prepared PDF
|
||||
5. Extended `FieldPlacer.tsx` palette — five new draggable tokens; existing drag/move/resize/persist mechanics unchanged
|
||||
6. Extended `prepare-document.ts` — type-aware rendering switch for all six field types; existing `client-signature` path unchanged
|
||||
1. `SignatureFieldData` JSONB (on `documents`) — field placement data; extended with optional `signerEmail` for multi-signer field routing; `signerEmail` absent = legacy single-signer
|
||||
2. `signingTokens` table — one row per signer per document; extended with `signerEmail` (nullable) and `viewedAt` columns; source of truth for who needs to sign and who has signed
|
||||
3. `documents.signers` JSONB column (new) — ordered list of `{ email, name, tokenJti, signedAt }` per document; coexists with legacy `assignedClientId` for backward compatibility
|
||||
4. `documents.completionTriggeredAt` column (new) — one-time-set guard preventing duplicate final PDF assembly on concurrent last-signer submissions
|
||||
5. POST `/api/sign/[token]` (modified) — atomic token claim, advisory lock PDF accumulation, completion detection via unclaimed-token count, `document_completed` event + notifications when all done
|
||||
6. `signing-mailer.tsx` (modified) — multi-signer path: `Promise.all()` over signers; new `sendAllSignersCompletionEmail()`
|
||||
7. `PreparePanel` + `FieldPlacer` (modified) — signer list entry UI, per-field signer assignment, color-coded markers, send-block validation
|
||||
|
||||
### Critical Pitfalls
|
||||
|
||||
1. **Breaking the signing page with new field types** — `SigningPageClient.tsx` opens the signature modal for every field in `signatureFields` with no type branching. Adding new field types without updating the signing page in the same deployment breaks active signing sessions. Ship schema + signing page filter as one atomic deployment, before any other v1.1 work.
|
||||
1. **First signer marks document Signed prematurely** — the current POST `/api/sign/[token]` unconditionally sets `documents.status = 'Signed'` on any token claim. With two signers, Signer A's completion fires the agent notification and marks the document complete before Signer B has signed. Fix: replace with a count of unclaimed tokens for the document; only set Signed when count reaches zero.
|
||||
|
||||
2. **AI coordinate Y-axis inversion** — AI returns percentages from top-left; `@cantoo/pdf-lib` uses PDF user-space with Y=0 at bottom. Storing AI coordinates without conversion inverts every field position. Write a `aiCoordsToPagePdfSpace()` conversion utility with a unit test asserting known PDF-space x/y values against a real Utah REPC before any OpenAI call is made.
|
||||
2. **Race condition — two simultaneous last signers both trigger PDF assembly** — atomic token claim does not prevent two concurrent handlers from both detecting zero remaining tokens. Fix: add `completionTriggeredAt TIMESTAMP` to `documents`; use `UPDATE WHERE completionTriggeredAt IS NULL RETURNING` guard — same pattern as the existing token claim. Zero rows returned means another handler already won; skip assembly.
|
||||
|
||||
3. **Agent-signature field sent unfiltered to client** — `/src/app/api/sign/[token]/route.ts` line 88 returns `doc.signatureFields ?? []` without type filtering. When `agent-signature` fields are in that array, the client sees them as required unsigned fields. Add type filter before any agent-signed document is sent.
|
||||
3. **`NEXT_PUBLIC_BASE_URL` baked at Docker build time, not runtime** — signing URLs will contain `localhost:3000` in production if this variable is not renamed before building the image. Fix: rename to `SIGNING_BASE_URL` (no `NEXT_PUBLIC_` prefix); inject at container runtime via `env_file:`; the send route is server-side only and never needs a public variable.
|
||||
|
||||
4. **Stale preview after field changes** — preview PDF written to a deterministic path gets cached; agent sends a document based on a stale preview. Use versioned preview paths (`{docId}_preview_{timestamp}.pdf`) and disable Send when fields have changed since last preview generation.
|
||||
4. **`@napi-rs/canvas` native binary incompatible with Alpine Docker images** — `node:alpine` uses musl libc; `@napi-rs/canvas` only ships glibc prebuilt binaries. The module fails with `invalid ELF header` at runtime. Fix: use `node:20-slim` (Debian); build with explicit `--platform linux/amd64` when deploying to x86 from an ARM Mac.
|
||||
|
||||
5. **OpenAI token limits on multi-page Utah forms** — Utah standard forms are 10-30 pages; full text extraction fits in ~2,000-8,000 tokens (within gpt-4o-mini's 128k context). Risk: testing only with 2-3 page PDFs in development. Prevention: test AI pipeline with the full Utah REPC (20+ pages) before shipping.
|
||||
5. **Uploads directory ephemeral in Docker** — all PDFs are written to `process.cwd()/uploads` inside the container's writable layer. Container recreation (deployment, crash) permanently deletes all documents including legally executed signed copies. Fix: mount a named Docker volume at `/app/uploads` in `docker-compose.yml` before the first production upload.
|
||||
|
||||
6. **SMTP env vars absent in container** — `CONTACT_SMTP_HOST` etc. exist in `.env.local` on the dev host but are invisible to Docker unless explicitly injected. Nodemailer `createTransporter()` is called at send time, not startup, so the missing vars produce no startup error — only silent email failure at first send. Fix: `env_file: .env.production` in docker-compose.yml. Verify with `docker exec <container> printenv CONTACT_SMTP_HOST`.
|
||||
|
||||
7. **Legacy `signingTokens` rows break if `signerEmail` added as NOT NULL** — existing rows have no signer email. Fix: add column as nullable (`TEXT`); backfill existing rows in the migration using a JOIN to the client email; all signing code handles the null legacy case.
|
||||
|
||||
8. **Neon connection pool exhaustion** — `postgres(url)` defaults to 10 connections, which saturates Neon free tier entirely. Fix: `postgres(url, { max: 5, idle_timeout: 20, connect_timeout: 10 })` in `db/index.ts`.
|
||||
|
||||
---
|
||||
|
||||
## Implications for Roadmap
|
||||
|
||||
The architecture research provides an explicit 8-step build order based on hard dependencies. This maps directly to 5 phases.
|
||||
Based on combined research, a five-phase structure is recommended:
|
||||
|
||||
### Phase 1: Schema Foundation + Signing Page Safety
|
||||
### Phase 1: Complete v1.1 — AI Field Placement and Expanded Field Types
|
||||
|
||||
**Rationale:** The single most dangerous change in v1.1 is adding field types to a schema the client signing page does not handle. Any document with mixed field types sent before the signing page is updated is a HIGH-recovery-cost production incident. Must be first, before any other v1.1 work.
|
||||
**Delivers:** Extended `DocumentField` discriminated union in `schema.ts` with backward-compatible fallback for v1.0 documents (`type ?? 'client-signature'`); two new nullable DB columns (`agentSignatureData` on users, `propertyAddress` on clients); Drizzle migration; updated `SigningPageClient.tsx` and `POST /api/sign/[token]` with type-based field filtering.
|
||||
**Addresses:** Foundation for all expanded field types; agent-signature client exposure risk
|
||||
**Avoids:** Pitfall 1 (signing page crash on new field types), Pitfall 10 (agent-sig field shown to client as required unsigned field)
|
||||
**Research flag:** None needed — Drizzle discriminated union and nullable column additions are well-documented; two-line ALTER TABLE migration.
|
||||
**Rationale:** v1.1 work is in progress (MEMORY.md confirms active debugging of AI field classification). This phase finishes in-flight work before adding multi-signer complexity on top of an unstable foundation.
|
||||
**Delivers:** Working AI field detection pipeline (text extraction + GPT-4.1 classification + post-processing rules), all five field types (checkbox, initials, date, agent signature, text), agent saved signature, filled document preview, agent signs first workflow, property address on client profile, AI pre-fill from client profile.
|
||||
**Addresses:** All FEATURES.md P1 items.
|
||||
**Avoids:** Vision-based coordinate inference (use text extraction only); `zodResponseFormat` with Zod v4 (use manual JSON schema).
|
||||
**Research flag:** No additional research needed — patterns are implemented and being debugged.
|
||||
|
||||
### Phase 2: Agent Saved Signature + Agent Signing Workflow
|
||||
### Phase 2: Multi-Signer Schema Foundation
|
||||
|
||||
**Rationale:** Agent signature is a prerequisite for the agent-signs-first workflow, which is a prerequisite for the filled preview (preview only makes sense after agent has signed). Agent signature embed also establishes the PNG embed pattern in `prepare-document.ts` that informs how other field types are handled.
|
||||
**Delivers:** `GET/PUT /api/agent/signature` routes; `AgentSignaturePanel` component (draw + save + thumbnail); extended `prepare-document.ts` to embed agent-sig PNG at field coordinates; `FieldPlacer` palette token for agent-signature type; supersede-and-resend flow guard preventing re-preparation of sent/viewed documents without user confirmation.
|
||||
**Uses:** `signature_pad@5.1.3` (existing), `@cantoo/pdf-lib@2.6.3` (existing), `users.agentSignatureData TEXT` column (Phase 1)
|
||||
**Avoids:** Pitfall 5 (signature stored as dataURL in DB is correct — TEXT column is right for 2-8KB), Pitfall 6 (race condition on re-preparation), Pitfall 10 (agent-sig filtered from client fields via Phase 1 foundation)
|
||||
**Research flag:** None needed — draw-save-reuse pattern is identical to v1.0 client signature; only new pieces are DB column and API route.
|
||||
**Rationale:** Schema changes are additive and backward-compatible. They must be deployed and validated against production Neon before any multi-signer sending or signing code is written. This phase has no user-visible effect — it is purely infrastructure.
|
||||
**Delivers:** Migration `drizzle/0010_multi_signer.sql`: `signer_email` on `signingTokens` (nullable), `signers` JSONB on `documents`, `completionTriggeredAt` on `documents`, `viewedAt` on `signingTokens`, three new `auditEventTypeEnum` values. TypeScript additions: `DocumentSigner` interface, `signerEmail` on `SignatureFieldData`, `getSignerEmail()` helper.
|
||||
**Avoids:** Legacy token breakage (nullable column), race condition setup (completionTriggeredAt column ready), enum ADD VALUE transaction issues (statement-breakpoint between each ALTER).
|
||||
**Research flag:** Standard Drizzle migration patterns — no additional research needed.
|
||||
|
||||
### Phase 3: Expanded Field Types End-to-End
|
||||
### Phase 3: Multi-Signer Backend (Token Creation, Signing Flow, Completion)
|
||||
|
||||
**Rationale:** Phase 1 made the schema and signing page safe. Phase 2 established the PNG embed pattern in `prepare-document.ts`. Now extend the field placement UI and prepare pipeline to handle all five new field types. Completing this phase gives the agent a fully functional field system without any AI dependency.
|
||||
**Delivers:** Five new draggable palette tokens in `FieldPlacer.tsx` (text, checkbox, initials, date, agent-signature); type-aware rendering in `prepare-document.ts` (text stamp, checkbox embed, date auto-stamp, initials placeholder); `propertyAddress` field in `ClientModal` and clients server action; field type coverage from placement through to embedded PDF.
|
||||
**Addresses:** All P1 table stakes: initials, date, checkbox, text field types
|
||||
**Avoids:** Pitfall 1 (signing page hardened in Phase 1 before these types can be placed and sent)
|
||||
**Research flag:** None needed — all APIs are in existing `@cantoo/pdf-lib@2.6.3`.
|
||||
**Rationale:** Backend-complete before UI. The signing POST rewrite is the highest-complexity change and must be independently testable via API calls before any UI work begins.
|
||||
**Delivers:** Updated `createSigningToken(docId, signerEmail?)`, signer-aware field filtering in signing GET (null signerEmail = legacy path), accumulator PDF pattern + advisory lock in signing POST, completion detection via unclaimed-token count + `completionTriggeredAt` guard, `document_completed` audit event, `sendAllSignersCompletionEmail()`, updated send route per-signer token loop. All changes preserve the legacy single-signer path.
|
||||
**Uses:** `pg_advisory_xact_lock(hashtext(documentId))` for concurrent PDF write protection, existing `embedSignatureInPdf()` unchanged, existing `logAuditEvent()` and `sendAgentNotificationEmail()`.
|
||||
**Avoids:** Premature completion (token count check), race condition (completionTriggeredAt guard), audit trail gap (signerEmail in event metadata), partial-send failure (individual email failure must not void already-created tokens).
|
||||
**Research flag:** The advisory lock interaction with Drizzle transactions may warrant a focused research pass if the developer is unfamiliar with Postgres advisory locks.
|
||||
|
||||
### Phase 4: Filled Document Preview
|
||||
### Phase 4: Multi-Signer UI (PreparePanel + FieldPlacer)
|
||||
|
||||
**Rationale:** Preview depends on the fully extended `preparePdf` from Phase 3 and agent signing from Phase 2. It is a composition of previous phases — build it after those foundations are solid.
|
||||
**Delivers:** `POST /api/documents/[id]/preview` route; `PreviewModal` component with in-app react-pdf rendering; versioned preview path with staleness detection; Send button disabled when fields changed since last preview; Back-to-edit flow; prepared PDF hashed at prepare time (extend existing `pdfHash` pattern).
|
||||
**Uses:** Existing `preparePdf` (reused unchanged), `react-pdf@10.4.1` (existing), ArrayBuffer copy pattern for react-pdf detachment bug
|
||||
**Avoids:** Pitfall 7 (stale preview), Pitfall 8 (OOM — generate-once, serve-cached pattern), Pitfall 9 (client signs different doc than agent previewed — hash verification)
|
||||
**Research flag:** Deployment target should be confirmed before implementation — the write-to-local-`uploads/` preview pattern fails on Vercel serverless (ephemeral filesystem). If deployed to Vercel, preview must write to Vercel Blob instead.
|
||||
**Rationale:** UI changes come last — they have no downstream dependencies and are safe to build once the backend is fully operational and tested.
|
||||
**Delivers:** Multi-signer list entry in PreparePanel (name + email rows, add/remove, replaces single email textarea), `PUT /api/documents/[id]/signers` endpoint, per-field signer email selector in FieldPlacer, color-coded field markers by signer, send-block validation (all client-visible fields must have signerEmail before send is allowed), agent dashboard showing per-signer status.
|
||||
**Avoids:** Chicken-and-egg ordering problem (signer list is saved to `documents.signers` before FieldPlacer is loaded; enforce in UI).
|
||||
**Research flag:** Standard React and Next.js API patterns — no additional research needed.
|
||||
|
||||
### Phase 5: AI Field Placement + Pre-fill
|
||||
### Phase 5: Docker Production Deployment + SMTP Fix
|
||||
|
||||
**Rationale:** AI is the highest-complexity feature and depends on field types being fully placeable (Phase 3) and the FieldPlacer accepting `DocumentField[]` from an external source. Building last means the agent can use manual placement throughout earlier phases. AI placement is an enhancement of the field system, not a replacement.
|
||||
**Delivers:** `lib/ai/extract-text.ts` (pdfjs-dist legacy build, server-only); `lib/ai/field-placement.ts` (GPT-4o-mini structured output, manual JSON schema, `server-only` guard); `POST /api/documents/[id]/ai-prepare` route with coordinate conversion utility + unit test; "AI Auto-place" button in PreparePanel with loading state and agent review step; AI pre-fill of text fields from client profile data.
|
||||
**Uses:** `openai@^6.32.0` (new install), pdfjs-dist legacy build (existing), gpt-4o-mini (sufficient for structured label extraction; ~15x cheaper than gpt-4o)
|
||||
**Avoids:** Pitfall 2 (coordinate mismatch — unit-tested conversion utility against known Utah REPC before shipping), Pitfall 3 (token limits — full-form test required), Pitfall 4 (hallucination — Zod validation of AI response before any field is stored; explicit enum for field types in JSON schema)
|
||||
**Research flag:** Requires integration test with real 20-page Utah REPC before shipping. Also validate that gpt-4o-mini text extraction accuracy on Utah standard forms (which have predictable label patterns) meets the 90%+ threshold claimed in research.
|
||||
**Rationale:** Deployment comes after all application features are complete and tested locally. The SMTP fix and `NEXT_PUBLIC_BASE_URL` rename must be verified before multi-signer completion emails are tested in production.
|
||||
**Delivers:** Three-stage Dockerfile (`node:20-slim`, Next.js `output: 'standalone'`), `docker-compose.yml` with named uploads volume + Neon connection via `env_file:`, `NEXT_PUBLIC_BASE_URL` renamed to `SIGNING_BASE_URL`, Neon connection pool limits (`max: 5`) in `db/index.ts`, `/api/health` endpoint, `.dockerignore`, SMTP transporter consolidated to shared utility, deployment runbook.
|
||||
**Avoids:** All Docker pitfalls: ephemeral uploads (named volume), wrong base image (`node:20-slim` not alpine), build-time URL baking (rename variable), SMTP failure (env_file + verification step), connection exhaustion (explicit max), startup order failure (healthcheck + depends_on if using local Postgres).
|
||||
**Research flag:** Standard patterns — official Next.js Docker documentation covers the three-stage standalone pattern precisely.
|
||||
|
||||
### Phase Ordering Rationale
|
||||
|
||||
- Phase 1 is a safety gate — deploy it before any document with new field types can be created or sent
|
||||
- Phase 2 before Phase 3 because `prepare-document.ts` needs the agent-sig embed pattern established before adding the full type-aware rendering switch
|
||||
- Phase 3 before Phase 4 because preview calls `preparePdf` — incomplete field type handling in prepare means an incomplete preview
|
||||
- Phase 5 last because it enhances a complete field system; agents can use manual placement throughout all earlier phases; no blocking dependency
|
||||
- The agent-signature field filtering (Pitfall 10) is addressed in Phase 1, not Phase 2 — this is deliberate; the signing route must be hardened before the first agent-sig field can be placed and sent
|
||||
- v1.1 must complete before v1.2 begins: multi-signer adds schema complexity on top of the field placement and signing pipeline; debugging both simultaneously is high-risk.
|
||||
- Schema before backend: multi-signer schema changes are additive and can deploy to production Neon independently; backend code written against the old schema cannot be safely run.
|
||||
- Backend before UI: the signing POST rewrite is the load-bearing change; FieldPlacer signer assignment has no effect until token and completion logic is correct.
|
||||
- Deployment last: Docker work is independent of application features but should be done with a complete, tested application to avoid conflating feature bugs with deployment bugs.
|
||||
|
||||
### Research Flags
|
||||
|
||||
**Needs deeper research during planning:**
|
||||
- **Phase 5 (AI):** The coordinate conversion from percentage to PDF user-space points needs a concrete unit test against a known Utah REPC before implementation. Validate pdfjs-dist legacy build text extraction works correctly in the project's actual Node 20 / Next.js 16.2 environment.
|
||||
- **Phase 4 (Preview):** Deployment target (Vercel serverless vs. self-hosted container) determines whether preview files can use the local `uploads/` filesystem or must use Vercel Blob. Confirm before writing the preview route.
|
||||
Phases likely needing `/gsd:research-phase` during planning:
|
||||
- **Phase 3 (Multi-Signer Backend):** Advisory lock pattern and Drizzle transaction interaction may need a targeted research pass for developers unfamiliar with `pg_advisory_xact_lock`.
|
||||
|
||||
**Standard patterns (skip research-phase):**
|
||||
- **Phase 1 (Schema):** Drizzle discriminated union extension and nullable column additions are well-documented; two-line ALTER TABLE migration.
|
||||
- **Phase 2 (Agent Signature):** The draw-save-reuse pattern is identical to v1.0 client signature; only new pieces are a DB column and API route.
|
||||
- **Phase 3 (Field Types):** All field type APIs are in existing `@cantoo/pdf-lib@2.6.3`; no new library research needed.
|
||||
Phases with standard patterns (skip research-phase):
|
||||
- **Phase 1 (v1.1):** Already researched and partially implemented.
|
||||
- **Phase 2 (Schema):** Drizzle migration patterns are routine.
|
||||
- **Phase 4 (Multi-Signer UI):** Standard React + API patterns.
|
||||
- **Phase 5 (Docker):** Officially documented Next.js standalone Docker pattern.
|
||||
|
||||
---
|
||||
|
||||
## Confidence Assessment
|
||||
|
||||
| Area | Confidence | Notes |
|
||||
|------|------------|-------|
|
||||
| Stack | HIGH | All versions verified via npm registry; OpenAI Zod v4 incompatibility confirmed via open GitHub issues #1540, #1602, #1709; pdfjs-dist server-side usage confirmed via actual codebase inspection |
|
||||
| Features | HIGH for field types and signing flows; MEDIUM for AI field detection accuracy | Field behavior confirmed against DocuSign, dotloop, SkySlope docs; AI coordinate accuracy confirmed via Feb 2025 benchmarks (< 3% pixel accuracy from vision); actual accuracy on Utah forms is untested |
|
||||
| Architecture | HIGH | Based on actual v1.0 codebase review (not speculative); specific file names, function names, and line numbers cited throughout; build order confirmed by dependency analysis |
|
||||
| Pitfalls | HIGH | All pitfalls grounded in actual codebase inspection; specific file paths and line numbers identified (e.g., sign route line 88); no speculative claims |
|
||||
| Stack | HIGH | All decisions code-verified against `package.json` and running source files. Zero speculative library choices. Zod v4 / zodResponseFormat incompatibility confirmed via open GitHub issues. |
|
||||
| Features | HIGH | Industry analysis of DocuSign, dotloop, SkySlope DigiSign confirms field type behavior and agent-signs-first convention. AI coordinate accuracy from February 2025 published benchmarks. |
|
||||
| Architecture | HIGH | Multi-signer design based on direct inspection of schema.ts, signing route, send route. Advisory lock pattern is Postgres-standard. OpenSign reference confirms the data model approach. |
|
||||
| Pitfalls | HIGH | All pitfalls cite specific file paths and code from the v1.1 codebase. NEXT_PUBLIC bake-at-build confirmed against Next.js official docs. No speculative claims. |
|
||||
|
||||
**Overall confidence:** HIGH
|
||||
|
||||
### Gaps to Address
|
||||
|
||||
- **AI coordinate accuracy on real Utah forms:** Research confirms the text-extraction + label-matching approach is correct, but accuracy on actual Utah REPC and listing agreement forms is untested. Phase 5 must include an integration test with real forms before the feature ships.
|
||||
- **Preview file lifecycle in production:** The `_preview_{timestamp}.pdf` pattern creates unbounded file growth in `uploads/`. A cleanup strategy (delete previews older than 24 hours, or delete on document send) needs to be decided before Phase 4 implementation.
|
||||
- **Deployment target for preview writes:** The write-to-disk preview pattern silently fails on Vercel serverless (ephemeral filesystem). Confirm whether the app runs on Vercel serverless or a persistent container before implementing Phase 4.
|
||||
- **`@vercel/blob` dead dependency:** Installed in `package.json` but not used anywhere in the codebase. Risks accidental use in future code that would silently fail in a non-Vercel Docker deployment. Decision needed before Phase 5: remove the package, or document it as unused and ensure it stays that way.
|
||||
- **Nodemailer port mismatch between mailers:** `signing-mailer.tsx` defaults to port 465 (implicit TLS) and `contact-mailer.ts` defaults to port 587 (STARTTLS) when `CONTACT_SMTP_PORT` is not set. A shared `createSmtpTransporter()` utility is needed; this should be addressed in Phase 5 before the first production email test.
|
||||
- **AI accuracy on non-standard forms:** The 90%+ accuracy expectation applies to Utah standard forms with consistent label patterns. The system's behavior on commercial addenda, non-standard addenda, or scanned PDFs is untested. The graceful fallback (manual drag-drop with a clear message) handles the failure case, but real-world accuracy across Teressa's full form library should be validated in Phase 1 before committing to the AI-first workflow.
|
||||
- **Local Postgres vs. Neon in Docker:** The research and architecture assume Neon as the production database (external managed service). If a local `postgres` Docker service is substituted, the `depends_on: service_healthy` pattern documented in ARCHITECTURE.md applies. The current research covers the Neon path only.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- `src/lib/db/schema.ts` (actual codebase, inspected 2026-03-21) — `SignatureFieldData` has no `type` field confirmed
|
||||
- `src/app/api/sign/[token]/route.ts` line 88 (actual codebase) — unfiltered `signatureFields` sent to client confirmed
|
||||
- `src/app/portal/(protected)/documents/[docId]/_components/FieldPlacer.tsx` (actual codebase) — single "Signature" token; `screenToPdfCoords` Y-inversion pattern confirmed
|
||||
- [openai npm](https://www.npmjs.com/package/openai) — v6.32.0 confirmed, Node 20 requirement
|
||||
- [OpenAI Structured Outputs docs](https://platform.openai.com/docs/guides/structured-outputs) — manual json_schema format confirmed
|
||||
- Direct code audit: `src/lib/ai/extract-text.ts`, `src/lib/ai/field-placement.ts`, `src/lib/db/schema.ts`, `src/lib/signing/signing-mailer.tsx`, `src/lib/db/index.ts`, `next.config.ts`, `package.json` — architecture verification and pitfall specifics
|
||||
- [OpenAI Structured Outputs docs](https://platform.openai.com/docs/guides/structured-outputs) — manual JSON schema format confirmed
|
||||
- [openai-node Issue #1540](https://github.com/openai/openai-node/issues/1540) — zodResponseFormat broken with Zod v4
|
||||
- [openai-node Issue #1602](https://github.com/openai/openai-node/issues/1602) — zodTextFormat broken with Zod v4
|
||||
- [openai-node Issue #1709](https://github.com/openai/openai-node/issues/1709) — Zod 4.1.13+ discriminated union break
|
||||
- [@cantoo/pdf-lib npm](https://www.npmjs.com/package/@cantoo/pdf-lib) — v2.6.3; createTextField, createCheckBox, drawImage APIs confirmed
|
||||
- [react-pdf ArrayBuffer detach issue #1657](https://github.com/wojtekmaj/react-pdf/issues/1657) — ArrayBuffer copy workaround confirmed
|
||||
- [Vercel Serverless Function Limits](https://vercel.com/docs/functions/runtimes/node-js#memory-and-compute) — 256MB default memory, 60s max execution on Pro
|
||||
- [Utah Division of Real Estate — State Approved Forms](https://realestate.utah.gov/real-estate/forms/state-approved/) — REPC form structure context
|
||||
- [@cantoo/pdf-lib npm page](https://www.npmjs.com/package/@cantoo/pdf-lib) — v2.6.3 field type API
|
||||
- [signature_pad GitHub](https://github.com/szimek/signature_pad) — v5.1.3 canvas API
|
||||
- [Docker Official Next.js Containerize Guide](https://docs.docker.com/guides/nextjs/containerize/) — three-stage Dockerfile
|
||||
- [Next.js with-docker official example](https://github.com/vercel/next.js/blob/canary/examples/with-docker/Dockerfile) — standalone mode pattern
|
||||
- [Next.js env var classification](https://nextjs.org/docs/pages/guides/environment-variables) — NEXT_PUBLIC_ bake-at-build-time behavior
|
||||
- [Docker Compose Secrets — Official Docs](https://docs.docker.com/compose/how-tos/use-secrets/) — env_file vs secrets decision
|
||||
- [OpenSign GitHub Repository](https://github.com/OpenSignLabs/OpenSign) — multi-signer reference implementation (signers array in document record)
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- [Edge AI and Vision Alliance — SAM 2 + GPT-4o (Feb 2025)](https://www.edge-ai-vision.com/2025/02/sam-2-gpt-4o-cascading-foundation-models-via-visual-prompting-part-2/) — GPT-4o returns accurate bounding box coordinates in < 3% of attempts
|
||||
- [Instafill.ai — Real estate law flat PDF form automation (Feb 2026)](https://blog.instafill.ai/2026/02/18/case-study-real-estate-law-flat-pdf-form-automation/) — hybrid text-extraction + LLM approach confirmed as production pattern
|
||||
- [DocuSign community — routing order for real estate](https://community.docusign.com/esignature-111/prefill-fields-before-sending-envelope-for-signature-180) — agent order 1, client order 2 confirmed
|
||||
- [Dotloop support — date auto-stamp behavior](https://support.dotloop.com/hc/en-us/articles/217936457-Adding-Signatures-or-Initials-to-Locked-Templates) — date field auto-stamp pattern confirmed
|
||||
- [DocuSign community — Date Signed field](https://community.docusign.com/esignature-111/am-i-able-to-auto-populate-the-date-field-2271) — read-only auto-populated date confirmed
|
||||
- [Next.js Standalone Docker Mode — DEV Community (2025)](https://dev.to/angojay/optimizing-nextjs-docker-images-with-standalone-mode-2nnh) — image size benchmarks (~300 MB vs ~7 GB)
|
||||
- [Docker DNS EAI_AGAIN — dev.to (2025)](https://dev.to/ameer-pk/beyond-it-works-on-my-machine-solving-docker-networking-dns-bottlenecks-4f3m) — DNS resolution behavior in Docker bridge network
|
||||
- [Docker Compose Secrets: What Works, What Doesn't](https://www.bitdoze.com/docker-compose-secrets/) — plain Compose vs Swarm secrets security comparison
|
||||
- Published AI bounding box accuracy benchmarks (February 2025) — under 3% accuracy for vision-based coordinate inference from PDF images
|
||||
- [Nodemailer Docker EAI_AGAIN — Docker Forums](https://forums.docker.com/t/not-able-to-send-email-using-nodemailer-within-docker-container-due-to-eai-again/40649) — root cause confirmed as env var injection, not DNS
|
||||
|
||||
---
|
||||
*Research completed: 2026-03-21*
|
||||
|
||||
*Research completed: 2026-04-03*
|
||||
*Milestone coverage: v1.1 (AI field placement + expanded field types) + v1.2 (multi-signer + Docker)*
|
||||
*Ready for roadmap: yes*
|
||||
|
||||
Reference in New Issue
Block a user