ccopeland/red
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: complete project research

Browse Source
This commit is contained in:
Chandler Copeland
2026-03-21 11:28:42 -06:00
parent 8c69deeb68
commit e36c6c8ee2
5 changed files with 1436 additions and 1689 deletions
Download Patch File Download Diff File Expand all files Collapse all files

298
.planning/research/SUMMARY.md
View File

@@ -1,251 +1,185 @@
# Project Research Summary
**Project:** Teressa Copeland Homes
**Domain:** Real estate agent marketing site + custom PDF document signing portal (Utah/WFRMLS)
**Researched:** 2026-03-19
**Project:** Teressa Copeland Homes — v1.1 Smart Document Preparation
**Domain:** Real estate agent website + PDF document signing portal
**Researched:** 2026-03-21
**Confidence:** HIGH
## Executive Summary
This is a dual-product build: a public real estate marketing site for a solo Utah agent, and a private document-signing portal that replaces per-month third-party tools (DocuSign, HelloSign) with a fully branded, custom implementation. Research across all four domains converges on a single-repo Next.js 15 application deployed to Vercel with a Neon PostgreSQL database and Vercel Blob storage for PDFs. The stack is unambiguous: `pdfjs-dist` for browser PDF rendering, `@pdfme/pdf-lib` (the maintained fork) for server-side PDF modification, `signature_pad` for canvas signature capture, `better-auth` for agent authentication, and `resend` for email delivery of signing links. This combination is well-documented, actively maintained, and sized exactly right for a solo-agent workflow — no enterprise licensing, no per-user cost, and full brand control throughout.
This is a v1.1 feature expansion of an existing, working Next.js 15 real estate document signing app. The v1.0 codebase is already validated — it uses Drizzle ORM, local PostgreSQL, `@cantoo/pdf-lib` for PDF writing, `react-pdf` for client-side rendering, Auth.js v5, and `signature_pad` for canvas signatures. The v1.1 additions are: AI-assisted field placement via GPT-4o-mini, five new field types (text, checkbox, initials, date, agent-signature), agent saved signature with a draw-once-reuse workflow, and a filled document preview before sending. The minimal dependency delta is two new packages: `openai@^6.32.0` and optionally `unpdf@^1.4.0` — though `pdfjs-dist` is already installed as a transitive dependency of `react-pdf` and can serve the server-side text extraction role via its legacy build.
The recommended architecture cleanly separates two subsystems that share only the Next.js shell: the public marketing site (listings, bio, contact) and the protected agent portal (clients, documents, signing requests). These can be built in parallel after the foundation is in place and are only loosely coupled — the listings page has no dependency on the signing flow, and vice versa. The five-stage PDF pipeline (import → parse fields → fill text → send for signing → embed signature) maps directly to a sequential build order where each stage unblocks the next. The architecture research prescribes a 7-phase build order that aligns exactly with this dependency chain.
The recommended build order is anchored by a schema-first phase. The `SignatureFieldData` type currently has no `type` discriminant — every field is treated identically as a client signature. Adding new field types without simultaneously updating both the schema AND the client signing page would break any in-flight signing session. The architecture research maps out an explicit 8-step dependency chain. For AI field placement, the correct approach uses `pdfjs-dist` for server-side text extraction (not vision), then GPT-4o-mini for semantic label classification — raw vision-based bounding box inference returns accurate coordinates less than 3% of the time. The OpenAI integration must use a manually defined JSON schema for structured output; the `zodResponseFormat` helper is broken with Zod v4 (confirmed open bug).
The most serious risks in this domain are legal, not technical. A custom e-signature implementation that lacks a tamper-evident PDF hash, an incomplete audit trail, or replayable signing links is legally indefensible under ESIGN/UETA regardless of how well the rest of the app works. Pitfalls research is unambiguous: audit logging and one-time token enforcement must be built in from the first signing ceremony — they cannot be retrofitted. A second cluster of risk sits in the WFRMLS integration: scraping the utahrealestate.com forms library violates Terms of Service (the platform partners with SkySlope; no public API exists for forms), and displaying IDX listing data without required broker attribution and NAR-mandated disclaimer text risks fines and loss of MLS access. Both risks are avoidable with correct design decisions made before code is written.
The key risk cluster is around the AI coordinate pipeline and signing page integrity. OpenAI returns percentage-based coordinates; `@cantoo/pdf-lib` expects PDF user-space points with a bottom-left origin — a Y-axis inversion that will silently produce wrong field positions without a dedicated conversion utility and unit test. A second risk is that agent-signature fields must be filtered from the `signatureFields` array sent to clients — the exact unguarded line (`/src/app/api/sign/[token]/route.ts` line 88) is identified in pitfalls research. Preview PDFs must use versioned paths separate from the final prepared PDF to maintain legal integrity between what the agent reviewed and what the client signs.
## Key Findings
### Recommended Stack
The stack is cohesive and all components are current-stable. Next.js 15.5.x (not 16 — too many breaking changes just landed) with React 19 provides the full-stack framework, App Router route groups for clean public/agent separation, and Server Components for the listings page without client-side JavaScript. Neon PostgreSQL with Prisma 6.x is the data layer — serverless-compatible, branches per PR, and generous free tier. Authentication uses `better-auth` 1.5.x (Auth.js/NextAuth is merging with better-auth and remains in beta; Clerk has per-MAU pricing that makes no sense for a single agent). PDF processing requires three distinct libraries that serve non-overlapping roles: `pdfjs-dist` for browser rendering and field detection (cannot modify PDFs), `@pdfme/pdf-lib` for server-side modification and signature embedding (cannot render to screen), and `signature_pad` for canvas capture. The original `pdf-lib` is unmaintained (4 years since last publish) — use the `@pdfme/pdf-lib` fork exclusively.
The v1.0 stack is unchanged and validated. See `STACK.md` for full version details.
**Core technologies:**
- **Next.js 15.5.x**: Full-stack framework — App Router, Server Components, API routes in one repo; Vercel-native; skip v16 until ecosystem stabilizes
- **React 19**: Ships with Next.js 15; Server Components and form actions stable
- **TypeScript 5.x**: Required; all major libraries ship full types
- **Neon PostgreSQL + Prisma 6.x**: Serverless-compatible DB; best-in-class DX for TypeScript; Vercel Marketplace integration
- **better-auth 1.5.x**: No per-MAU cost; built-in rate limiting; first-class Next.js App Router support
- **pdfjs-dist 5.5.x**: Browser-side PDF rendering and AcroForm field detection only
- **@pdfme/pdf-lib 5.5.x**: Server-side PDF modification — fill fields, embed signature PNG, flatten; actively maintained fork
- **signature_pad 5.1.3**: Canvas signature capture; Bezier smoothing; touch-normalized; use directly (not the alpha React wrapper)
- **resend + @react-email/components**: Transactional email for signing links; 3k/month free tier; React email templates
- **@vercel/blob**: Zero-config PDF storage for Vercel; S3-backed; abstract behind `storage.ts` to allow future swap
- **Playwright**: Required for utahrealestate.com forms (if used); run via separate service or Browserless.io — not inline in Vercel serverless
- **Tailwind CSS v4 + shadcn/ui**: Styling; Tailwind v4 is CSS-native, no config file required
**New dependencies for v1.1:**
- `openai@^6.32.0`: Official SDK, TypeScript-native structured output for GPT-4o-mini — use manual `json_schema` response_format, NOT `zodResponseFormat` (broken with Zod v4, confirmed open GitHub issues #1540, #1602, #1709)
- `pdfjs-dist` legacy build (already installed): Server-side PDF text extraction via `pdfjs-dist/legacy/build/pdf.mjs` — no new dependency needed if using this path
**Critical version note:** Do NOT use `next@16` — breaking changes (async APIs, middleware renamed to proxy, edge runtime dropped) make it incompatible with the ecosystem today. Use `next@15.5.x` and revisit in 6-12 months.
**Existing stack components covering all v1.1 needs:**
- `@cantoo/pdf-lib@2.6.3`: All five new field types (text, checkbox, initials, date, agent-signature) supported natively via `createTextField`, `createCheckBox`, `drawImage` APIs
- `signature_pad@5.1.3`: Agent signature canvas — use `useRef<HTMLCanvasElement>` + `useEffect` pattern directly; do NOT add `react-signature-canvas` (alpha wrapper)
- `react-pdf@10.4.1`: Filled preview rendering — pass `ArrayBuffer` directly; copy the buffer before passing to avoid detachment issue (known bug #1657)
- `@vercel/blob@2.3.1` + Drizzle ORM: Agent signature storage — architecture research recommends TEXT column on `users` table for 2-8KB base64 PNG; no new file storage needed
### Expected Features
Research identifies a clear MVP boundary. The signing portal is the novel differentiator — all the marketing site features are table stakes for any professional real estate presence. The key UX insight from feature research: no client account creation, ever. The signing link token is the client's identity. Every friction point between "email received" and "signature captured" is an abandonment driver.
All v1.1 features are P1 (must-have for launch). Research confirms the full feature set is aligned with industry standard behavior across DocuSign, dotloop, and SkySlope DigiSign.
**Must have for v1 launch (P1):**
- Marketing site: hero with agent photo, bio, contact form, testimonials
- Active listings display from WFRMLS (with full IDX compliance: broker attribution, disclaimer text, last-updated timestamp)
- Agent login (better-auth, credentials)
- Client management: create/view clients with name and email
- PDF upload and browser rendering (pdfjs-dist + react-pdf)
- Signature field placement UI: drag-and-drop on PDF canvas with coordinate storage in PDF user space
- Email delivery of unique, tokenized signing link (resend)
- Token-based anonymous client signing page: no account, no login
- Canvas signature capture (signature_pad) — mobile-first, touch-action:none, iOS Safari tested
- Audit trail: IP, timestamp, user-agent, consent acknowledgment — all server-side timestamps
- PDF signature embed + form flatten with fonts embedded before flattening (@pdfme/pdf-lib)
- Tamper-evident PDF hash: SHA-256 of final signed PDF bytes stored in DB
- One-time signing token enforcement: used_at column, DB transaction, 72-hour TTL
- Signed document storage (Vercel Blob, private ACLs, presigned URLs only)
- Agent dashboard: document status at a glance (Draft / Sent / Viewed / Signed)
**Must have (table stakes):**
- Initials field type — every Utah standard form (REPC, listing agreement, addenda) has per-page initials lines; missing this makes the app unusable for standard Utah workflows
- Date field (auto-stamp, read-only) — "Date Signed" pattern; auto-populated at signing session completion; client never types a date; legally important
- Checkbox field type — Utah REPC uses boolean checkboxes throughout (mediation clauses, contingency elections, disclosure acknowledgments)
- Agent saved signature — draw once, reuse across documents; the "Adopted Signature" pattern in every major real estate e-sig tool
- Agent signs first workflow — industry convention: agent at routing order 1, client at routing order 2; confirmed by DocuSign community docs
- Filled document preview with Send gating — prevents the most-cited mistake (sending wrong document version); Send button lives in preview
**Should have, add after v1 (P2):**
- Forms library import from utahrealestate.com — add only as manual agent upload, not scraping
- Heuristic AcroForm field detection on Utah standard forms — manual placement fallback always present
- Document view/open tracking (link opened audit event)
- Signed document confirmation email to client
- Multiple field types: initials, auto-date, checkbox, text inputs
- Neighborhood guide / SEO content pages
**Should have (differentiators):**
- AI field placement via gpt-4o-mini + text extraction — eliminates manual drag-drop session; accuracy 90%+ on structured Utah forms with predictable label patterns ("Buyer's Signature", "Date", "Initial Here")
- AI pre-fill from client profile — maps client name, email, property address to text fields; low hallucination risk (structured profile data, not free-text inference)
- Property address field on client profile — enables AI pre-fill to be property-specific; simple schema addition
**Defer to v2+ (P3):**
- Automated unsigned-document reminder system (requires scheduling infrastructure)
- Client portal with document history (requires client accounts — explicitly an anti-feature in v1)
- Multi-agent / brokerage support (role/permissions model doubles auth complexity)
- Bulk document send
- Native mobile app (responsive web signing works; 50%+ of e-signatures happen on mobile web)
**Anti-features to avoid entirely:**
- Scraping utahrealestate.com forms library (ToS violation; use manual upload)
- DocuSign/HelloSign integration (monthly cost, third-party branding — defeats the purpose)
- In-app PDF content editing (real estate contracts have legally mandated language; editing creates liability)
- WebSockets for signing (one-shot action, not a live session)
- Client account creation for signing (friction kills completion rates)
**Defer to v1.2+:**
- AI confidence display to agent — adds UI noise; agent can see and correct in preview instead
- Template save from AI placement — high value but requires template management UI; defer until AI accuracy is validated
- Multiple agent signature fields per document — needs UX design; defer
### Architecture Approach
The system is a single Next.js monorepo with two distinct route groups: `(public)` for the unauthenticated marketing site and `(agent)` for the protected portal. These share the Prisma/Neon data layer and Vercel Blob storage but have no UI coupling. The PDF pipeline is entirely server-side (API routes / Server Actions) except for browser rendering and signature capture, which require Client Components loaded with `dynamic(() => import(...), { ssr: false })`. Authentication uses three defense-in-depth layers — edge middleware, layout Server Component, and per-route handler — because the CVE-2025-29927 middleware bypass (disclosed March 2025) demonstrated that middleware alone is not sufficient. The signing flow uses JWT tokens (HMAC-SHA256) stored server-side with one-time enforcement via a `used_at` column, providing both stateless verification and replay protection.
The v1.1 architecture is an incremental extension of the existing system — not a rewrite. Seven new files are created (two server-only AI lib files, three API routes, two client components). Eight existing files are modified with targeted additions. The critical architectural constraint: the existing client signing flow (`embed-signature.ts`, signing token route, `SignatureModal.tsx`) must not be altered. Agent-sig and text/checkbox/date fields are baked into the prepared PDF before the client opens the signing link. The client signing page handles only `client-signature` and `initials` field types.
**Major components and responsibilities:**
See `ARCHITECTURE.md` for complete component boundaries, data flow diagrams, and the full 8-step build order.
1. **`middleware.ts`** — Edge auth gate for `/agent/*`; redirects to login; NOT the only auth layer (CVE-2025-29927 requires defense in depth)
2. **`(agent)/layout.tsx`** — Server Component second auth layer; calls `verifySession()` on every render
3. **`lib/pdf/parse.ts`** — pdfjs-dist server-side: extracts AcroForm field names, types, and bounding boxes from PDF bytes
4. **`lib/pdf/fill.ts`** — @pdfme/pdf-lib server-side: writes agent-supplied text into named form fields; embeds fonts before flattening
5. **`lib/pdf/sign.ts`** — @pdfme/pdf-lib server-side: embeds signature PNG at stored coordinates; flattens and seals the form; computes SHA-256 hash
6. **`components/agent/PDFFieldMapper.tsx`** — Client Component: renders PDF page via canvas; drag-to-define signature zones; converts viewport coordinates to PDF user space (origin bottom-left) before storing
7. **`components/sign/SignatureCanvas.tsx`** — Client Component: signature_pad with touch-action:none; exports PNG for submission
8. **`app/sign/[token]/page.tsx`** — Public signing page: validates JWT, streams prepared PDF, renders field overlays, submits canvas PNG
9. **`lib/wfrmls/client.ts`** — WFRMLS RESO OData client with ISR revalidate=3600 for listings
10. **`lib/storage/s3.ts`** — Vercel Blob abstraction: upload, download, presigned URL (5-minute TTL); PDFs never served with public ACLs
**Database schema highlights:** `User` → `Client` → `SigningRequest` → `SignatureAuditLog` chain. `Document` stores the raw PDF template (reusable across multiple `SigningRequest`s). `SigningRequest` holds `fieldValues`, `signatureFields` (coordinates in PDF space), `preparedS3Key`, `signedS3Key`, `token`, `tokenJti`, `usedAt`, and status enum (DRAFT/SENT/VIEWED/SIGNED/EXPIRED/CANCELLED). `SignatureAuditLog` is append-only and records every ceremony event with server-side timestamps.
**PDF coordinate conversion is mandatory:** Browser canvas coordinates (origin top-left, Y down) must be converted to PDF user space (origin bottom-left, Y up) using `viewport.convertToPdfPoint(x, y)` from pdfjs-dist before storage. This conversion must be unit-tested against actual Utah real estate forms before the field placement UI ships.
**Major components:**
1. `lib/ai/extract-text.ts` + `lib/ai/field-placement.ts` (NEW, server-only) — pdfjs-dist legacy build for text extraction; GPT-4o-mini structured output with manual JSON schema; `server-only` import guard prevents accidental client bundle inclusion
2. `POST /api/documents/[id]/ai-prepare` (NEW) — orchestrates extract + AI call + coordinate conversion (percentage to PDF points using actual page dimensions)
3. `GET/PUT /api/agent/signature` (NEW) — stores agent signature as base64 PNG TEXT column on `users` table; always auth-gated
4. `POST /api/documents/[id]/preview` (NEW) — reuses existing `preparePdf` in preview mode; writes to versioned `_preview_{timestamp}.pdf`; streams bytes directly; never overwrites final prepared PDF
5. Extended `FieldPlacer.tsx` palette — five new draggable tokens; existing drag/move/resize/persist mechanics unchanged
6. Extended `prepare-document.ts` — type-aware rendering switch for all six field types; existing `client-signature` path unchanged
### Critical Pitfalls
1. **No tamper-evident PDF hash** — Compute SHA-256 of the complete signed PDF bytes immediately after embedding the signature, before storing. Store the hash in `SigningRequest`. This is the difference between a legally defensible document and "a drawing on a page."
1. **Breaking the signing page with new field types** — `SigningPageClient.tsx` opens the signature modal for every field in `signatureFields` with no type branching. Adding new field types without updating the signing page in the same deployment breaks active signing sessions. Ship schema + signing page filter as one atomic deployment, before any other v1.1 work.
2. **Incomplete audit trail** — Log six ceremony events server-side: document prepared, email sent, link opened (with IP/UA/timestamp), document viewed, signature submitted, final PDF hash computed. Timestamps must be server-side — client-reported timestamps are legally worthless. This must be wired in before the first signing ceremony, not added later.
2. **AI coordinate Y-axis inversion** — AI returns percentages from top-left; `@cantoo/pdf-lib` uses PDF user-space with Y=0 at bottom. Storing AI coordinates without conversion inverts every field position. Write a `aiCoordsToPagePdfSpace()` conversion utility with a unit test asserting known PDF-space x/y values against a real Utah REPC before any OpenAI call is made.
3. **Replayable signing token** — Signing tokens must have a `used_at` column set atomically on successful submission (DB transaction to prevent race conditions). After use, the link must return "already signed" — never the canvas. 72-hour TTL is appropriate for real estate (clients don't check email instantly; 15-minute magic-link windows are too short).
3. **Agent-signature field sent unfiltered to client** — `/src/app/api/sign/[token]/route.ts` line 88 returns `doc.signatureFields ?? []` without type filtering. When `agent-signature` fields are in that array, the client sees them as required unsigned fields. Add type filter before any agent-signed document is sent.
4. **PDF coordinate system mismatch** — PDF user space uses bottom-left origin with Y increasing upward; browser canvas uses top-left with Y downward. Embedding a signature without the conversion inverts the Y position. Write a coordinate conversion unit test against a real Utah purchase agreement form before building the drag-and-drop UI. Also handle page rotation (`Rotate` key in PDF) — Utah forms may be rotated 90 degrees.
4. **Stale preview after field changes** — preview PDF written to a deterministic path gets cached; agent sends a document based on a stale preview. Use versioned preview paths (`{docId}_preview_{timestamp}.pdf`) and disable Send when fields have changed since last preview generation.
5. **utahrealestate.com forms scraping violates ToS** — The platform partners with SkySlope and has no public forms API. Storing Teressa's credentials to automate downloads violates the WFRMLS data licensing agreement. Use manual PDF upload as the document source. State-approved Utah DRE forms at commerce.utah.gov are public domain and can be embedded directly.
6. **IDX compliance violations** — Every listing page (card and detail) must display: listing broker/office name (`ListOfficeName`), WFRMLS disclaimer text verbatim, last-updated timestamp from the feed, and buyer's agent compensation disclosure per 2024 NAR settlement. Missing any of these risks fines up to $15,000 and loss of MLS access. Treat these as acceptance criteria, not post-launch polish.
7. **PDF font flattening failure** — AcroForm fields reference fonts by name (e.g., "Helvetica"). On Vercel serverless, no system fonts are installed. Calling `form.flatten()` without first embedding fonts produces blank text fields in the downloaded PDF. Explicitly embed standard fonts from pdf-lib's built-in set on every form field before flattening. Validate in production environment, not just local Mac with system fonts.
8. **Mobile canvas scrolling instead of signing** — Without `touch-action: none` on the canvas element, iOS Safari and Android Chrome intercept touch gestures as page scroll. The client tries to sign and the page scrolls instead. This must be tested on physical devices before any client is sent a signing link.
5. **OpenAI token limits on multi-page Utah forms** — Utah standard forms are 10-30 pages; full text extraction fits in ~2,000-8,000 tokens (within gpt-4o-mini's 128k context). Risk: testing only with 2-3 page PDFs in development. Prevention: test AI pipeline with the full Utah REPC (20+ pages) before shipping.
## Implications for Roadmap
Based on the dependency chain identified in architecture research and the feature priority matrix from feature research, the natural build order is 7 phases:
The architecture research provides an explicit 8-step build order based on hard dependencies. This maps directly to 5 phases.
### Phase 1: Foundation
**Rationale:** Everything else depends on this. Database schema, auth system, and storage infrastructure must exist before any feature can be built. Auth has a well-documented security pattern (three-layer defense-in-depth due to CVE-2025-29927) that must be established upfront — retrofitting auth is expensive.
**Delivers:** Working Next.js project, Prisma schema + Neon DB, Vercel Blob bucket, better-auth credentials login, middleware + layout guard, agent can log in and reach a blank dashboard.
**Addresses:** Agent login (P1 feature), tamper-resistant auth architecture.
**Avoids:** Middleware-only auth bypass (CVE-2025-29927 defense-in-depth established from day one).
**Research flag:** None needed — well-documented patterns for all components.
### Phase 1: Schema Foundation + Signing Page Safety
### Phase 2: Public Marketing Site
**Rationale:** Independent of the document workflow; provides immediate business value; unblocks IDX integration testing early. Can be built by a different person in parallel with Phase 3 if needed.
**Delivers:** Public-facing site with hero, bio, contact form, and WFRMLS listings display with full IDX compliance (broker attribution, disclaimer, last-updated, NAR 2024 compensation disclosure).
**Addresses:** Hero/bio/contact (P1), listings display (P1), IDX compliance (legal requirement), delta-sync listings refresh.
**Avoids:** Stale off-market listings (hourly delta sync with `ModificationTimestamp` filter); IDX attribution violations (treated as acceptance criteria, not polish).
**Research flag:** WFRMLS RESO API vendor enrollment takes 2-4 weeks — start this process immediately, in parallel with Phase 1. Do not block Phase 2 on this; build with mock data while approval is pending.
**Rationale:** The single most dangerous change in v1.1 is adding field types to a schema the client signing page does not handle. Any document with mixed field types sent before the signing page is updated is a HIGH-recovery-cost production incident. Must be first, before any other v1.1 work.
**Delivers:** Extended `DocumentField` discriminated union in `schema.ts` with backward-compatible fallback for v1.0 documents (`type ?? 'client-signature'`); two new nullable DB columns (`agentSignatureData` on users, `propertyAddress` on clients); Drizzle migration; updated `SigningPageClient.tsx` and `POST /api/sign/[token]` with type-based field filtering.
**Addresses:** Foundation for all expanded field types; agent-signature client exposure risk
**Avoids:** Pitfall 1 (signing page crash on new field types), Pitfall 10 (agent-sig field shown to client as required unsigned field)
**Research flag:** None needed — Drizzle discriminated union and nullable column additions are well-documented; two-line ALTER TABLE migration.
### Phase 3: Agent Portal Shell
**Rationale:** Client management and the agent dashboard are prerequisites for the document workflow. These are straightforward CRUD operations with standard patterns.
**Delivers:** Agent dashboard (skeleton), client list + create/edit, agent login page polish, navigation shell.
**Addresses:** Client management (P1), document status dashboard (P1).
**Avoids:** No pitfalls specific to this phase — standard CRUD patterns.
**Research flag:** None needed — well-documented patterns.
### Phase 2: Agent Saved Signature + Agent Signing Workflow
### Phase 4: PDF Ingest and Storage
**Rationale:** Must exist before field mapping UI can be built. The storage abstraction and PDF parsing pipeline are the lowest layer of the document workflow.
**Delivers:** PDF upload (manual agent upload — no scraping), Vercel Blob storage pipeline (original/prepared/signed versions), pdfjs-dist AcroForm field extraction, document create + detail pages.
**Addresses:** PDF upload + rendering (P1), signed document storage (P1).
**Avoids:** Local filesystem storage (serverless ephemeral filesystem); utahrealestate.com scraping (manual upload only, no credentials stored, no headless browser automation in core app); PDF coordinate detection tested with actual Utah forms.
**Research flag:** May need deeper research on pdfjs-dist Node.js legacy build for server-side parsing — uses `pdfjs-dist/legacy/build/pdf.mjs` without a worker, which is distinct from the browser build.
**Rationale:** Agent signature is a prerequisite for the agent-signs-first workflow, which is a prerequisite for the filled preview (preview only makes sense after agent has signed). Agent signature embed also establishes the PNG embed pattern in `prepare-document.ts` that informs how other field types are handled.
**Delivers:** `GET/PUT /api/agent/signature` routes; `AgentSignaturePanel` component (draw + save + thumbnail); extended `prepare-document.ts` to embed agent-sig PNG at field coordinates; `FieldPlacer` palette token for agent-signature type; supersede-and-resend flow guard preventing re-preparation of sent/viewed documents without user confirmation.
**Uses:** `signature_pad@5.1.3` (existing), `@cantoo/pdf-lib@2.6.3` (existing), `users.agentSignatureData TEXT` column (Phase 1)
**Avoids:** Pitfall 5 (signature stored as dataURL in DB is correct — TEXT column is right for 2-8KB), Pitfall 6 (race condition on re-preparation), Pitfall 10 (agent-sig filtered from client fields via Phase 1 foundation)
**Research flag:** None needed — draw-save-reuse pattern is identical to v1.0 client signature; only new pieces are DB column and API route.
### Phase 5: PDF Fill and Field Mapping
**Rationale:** Depends on Phase 4 (storage + parse infrastructure). The field mapper UI and the fill API are the agent's core document preparation workflow.
**Delivers:** PDFFieldMapper.tsx (drag-to-place signature zones on PDF canvas), coordinate conversion from viewport space to PDF user space with unit tests, @pdfme/pdf-lib fill API (text fields, font embedding before flatten), document editor form.
**Addresses:** Signature field placement UI (P1), agent-fills-then-client-signs workflow (differentiator).
**Avoids:** PDF coordinate mismatch (unit-tested against actual Utah purchase agreement form before UI ships); font flattening failure (fonts embedded explicitly; tested in production serverless environment); heuristic-only detection (manual placement fallback is the primary flow; auto-detect is an enhancement).
**Research flag:** Research the specific pdfjs-dist `viewport.convertToPdfPoint()` API and pdf-lib `StandardFonts` embedding before implementation — these are narrow but critical APIs.
### Phase 3: Expanded Field Types End-to-End
### Phase 6: Signing Flow — End to End
**Rationale:** The highest-complexity phase; depends on all previous phases. Contains all the legally critical components. Should be built as a complete vertical slice in one phase to ensure the audit trail is woven through from start to finish.
**Delivers:** JWT token generation (HMAC-SHA256, 72-hour TTL, one-time enforcement with `used_at` DB column), resend email delivery, `/sign/[token]` public signing page, SignatureCanvas.tsx (mobile-first, touch-action:none, iOS Safari + Android Chrome tested on physical devices), @pdfme/pdf-lib signature PNG embed, SHA-256 hash of final signed PDF, complete SignatureAuditLog (6 ceremony events with server-side timestamps), one-time token invalidation with race-condition-safe DB transaction, "already signed" page for expired/used tokens.
**Addresses:** Email delivery (P1), client signing page (P1), canvas capture (P1), audit trail (P1), signed document storage (P1), tamper-evident hash (P1).
**Avoids:** Replayable signing tokens; incomplete audit trail; mobile canvas scroll-instead-of-sign; blank-canvas submission; font flattening blank text; unsigned PDF served publicly.
**Research flag:** This phase warrants a `/gsd:research-phase` — the intersection of JWT one-time enforcement, PDF hash storage, ESIGN/UETA audit requirements, and mobile touch handling has enough edge cases that a focused spike before implementation reduces rework risk.
**Rationale:** Phase 1 made the schema and signing page safe. Phase 2 established the PNG embed pattern in `prepare-document.ts`. Now extend the field placement UI and prepare pipeline to handle all five new field types. Completing this phase gives the agent a fully functional field system without any AI dependency.
**Delivers:** Five new draggable palette tokens in `FieldPlacer.tsx` (text, checkbox, initials, date, agent-signature); type-aware rendering in `prepare-document.ts` (text stamp, checkbox embed, date auto-stamp, initials placeholder); `propertyAddress` field in `ClientModal` and clients server action; field type coverage from placement through to embedded PDF.
**Addresses:** All P1 table stakes: initials, date, checkbox, text field types
**Avoids:** Pitfall 1 (signing page hardened in Phase 1 before these types can be placed and sent)
**Research flag:** None needed — all APIs are in existing `@cantoo/pdf-lib@2.6.3`.
### Phase 7: Audit Trail, Status Tracking, and Download
**Rationale:** Completes the agent-facing visibility layer. The underlying data is already being written in Phase 6; this phase surfaces it in the UI.
**Delivers:** Document status tracking in agent dashboard (Draft/Sent/Viewed/Signed with last-activity timestamp), presigned Vercel Blob URLs for agent PDF download (5-minute TTL, authenticated route only), confirmation screen for client after signing, optional signed document email to client.
**Addresses:** Agent dashboard status (P1 polish), signed document retrieval (P1), document view tracking (P2).
**Avoids:** Signed PDFs accessible via guessable URL (all downloads gated behind authenticated presigned URLs).
**Research flag:** None needed — well-documented patterns; the audit data infrastructure is already in place from Phase 6.
### Phase 4: Filled Document Preview
**Rationale:** Preview depends on the fully extended `preparePdf` from Phase 3 and agent signing from Phase 2. It is a composition of previous phases — build it after those foundations are solid.
**Delivers:** `POST /api/documents/[id]/preview` route; `PreviewModal` component with in-app react-pdf rendering; versioned preview path with staleness detection; Send button disabled when fields changed since last preview; Back-to-edit flow; prepared PDF hashed at prepare time (extend existing `pdfHash` pattern).
**Uses:** Existing `preparePdf` (reused unchanged), `react-pdf@10.4.1` (existing), ArrayBuffer copy pattern for react-pdf detachment bug
**Avoids:** Pitfall 7 (stale preview), Pitfall 8 (OOM — generate-once, serve-cached pattern), Pitfall 9 (client signs different doc than agent previewed — hash verification)
**Research flag:** Deployment target should be confirmed before implementation — the write-to-local-`uploads/` preview pattern fails on Vercel serverless (ephemeral filesystem). If deployed to Vercel, preview must write to Vercel Blob instead.
### Phase 5: AI Field Placement + Pre-fill
**Rationale:** AI is the highest-complexity feature and depends on field types being fully placeable (Phase 3) and the FieldPlacer accepting `DocumentField[]` from an external source. Building last means the agent can use manual placement throughout earlier phases. AI placement is an enhancement of the field system, not a replacement.
**Delivers:** `lib/ai/extract-text.ts` (pdfjs-dist legacy build, server-only); `lib/ai/field-placement.ts` (GPT-4o-mini structured output, manual JSON schema, `server-only` guard); `POST /api/documents/[id]/ai-prepare` route with coordinate conversion utility + unit test; "AI Auto-place" button in PreparePanel with loading state and agent review step; AI pre-fill of text fields from client profile data.
**Uses:** `openai@^6.32.0` (new install), pdfjs-dist legacy build (existing), gpt-4o-mini (sufficient for structured label extraction; ~15x cheaper than gpt-4o)
**Avoids:** Pitfall 2 (coordinate mismatch — unit-tested conversion utility against known Utah REPC before shipping), Pitfall 3 (token limits — full-form test required), Pitfall 4 (hallucination — Zod validation of AI response before any field is stored; explicit enum for field types in JSON schema)
**Research flag:** Requires integration test with real 20-page Utah REPC before shipping. Also validate that gpt-4o-mini text extraction accuracy on Utah standard forms (which have predictable label patterns) meets the 90%+ threshold claimed in research.
### Phase Ordering Rationale
- **Foundation first (Phase 1):** Auth, DB, and storage have zero optional dependencies. Every subsequent phase builds on these.
- **Public site early (Phase 2):** Independent of the signing workflow; provides immediate value and allows WFRMLS API approval process to complete while portal phases are underway. The 2-4 week vendor enrollment timeline makes early start critical.
- **Portal CRUD before PDF (Phase 3):** Clients and documents are the entities that the PDF pipeline operates on. Establishing the data model and basic CRUD before building the complex pipeline reduces schema churn.
- **Storage before fill before sign (Phases 4-6):** Each phase adds one layer of the PDF pipeline. Building them in dependency order means each phase can ship independently and be tested in isolation.
- **Audit/download last (Phase 7):** The underlying data is written by Phase 6. Phase 7 is surfacing and visibility only — no new data model needed.
- **Scraping architecture decision made before Phase 4:** The decision to use manual upload (no utahrealestate.com scraping) must be explicit in Phase 4 so no scraping infrastructure is ever built.
- Phase 1 is a safety gate — deploy it before any document with new field types can be created or sent
- Phase 2 before Phase 3 because `prepare-document.ts` needs the agent-sig embed pattern established before adding the full type-aware rendering switch
- Phase 3 before Phase 4 because preview calls `preparePdf` — incomplete field type handling in prepare means an incomplete preview
- Phase 5 last because it enhances a complete field system; agents can use manual placement throughout all earlier phases; no blocking dependency
- The agent-signature field filtering (Pitfall 10) is addressed in Phase 1, not Phase 2 — this is deliberate; the signing route must be hardened before the first agent-sig field can be placed and sent
### Research Flags
**Needs `/gsd:research-phase` during planning:**
- **Phase 6 (Signing Flow):** Legal compliance intersection (ESIGN/UETA audit requirements + JWT one-time enforcement + PDF hash + mobile touch) has enough edge cases and gotchas that a pre-implementation research spike is warranted.
- **Phase 2 (Listings):** WFRMLS vendor enrollment process and exact IDX compliance requirements (disclaimer text, 2024 NAR settlement fields) should be confirmed with WFRMLS directly before the listings UI is built — the requirements change with NAR policy updates.
**Needs deeper research during planning:**
- **Phase 5 (AI):** The coordinate conversion from percentage to PDF user-space points needs a concrete unit test against a known Utah REPC before implementation. Validate pdfjs-dist legacy build text extraction works correctly in the project's actual Node 20 / Next.js 16.2 environment.
- **Phase 4 (Preview):** Deployment target (Vercel serverless vs. self-hosted container) determines whether preview files can use the local `uploads/` filesystem or must use Vercel Blob. Confirm before writing the preview route.
**Standard patterns (skip research-phase):**
- **Phase 1 (Foundation):** Next.js + Prisma + better-auth + Vercel Blob are all well-documented with official guides.
- **Phase 3 (Portal Shell):** Standard Next.js CRUD patterns; no novel integration.
- **Phase 7 (Audit/Status):** The data model is established; surfacing it is standard UI work.
- **Phase 1 (Schema):** Drizzle discriminated union extension and nullable column additions are well-documented; two-line ALTER TABLE migration.
- **Phase 2 (Agent Signature):** The draw-save-reuse pattern is identical to v1.0 client signature; only new pieces are a DB column and API route.
- **Phase 3 (Field Types):** All field type APIs are in existing `@cantoo/pdf-lib@2.6.3`; no new library research needed.
## Confidence Assessment
| Area | Confidence | Notes |
|------|------------|-------|
| Stack | HIGH | All library versions verified via npm. next@15.5 vs 16 verified against official upgrade guide. pdf-lib unmaintained status confirmed (npm publish date). @pdfme/pdf-lib fork activity confirmed. better-auth + NextAuth merge confirmed via GitHub discussion. |
| Features | HIGH | Feature list cross-referenced against multiple industry sources (DocuSign, HelloSign, SkySlope/Authentisign competitor analysis). ESIGN/UETA requirements sourced from legal and engineering references. Utah-specific requirements sourced from Utah DRE and WFRMLS. |
| Architecture | HIGH | CVE-2025-29927 middleware bypass is a real, documented vulnerability (Vercel postmortem + ProjectDiscovery analysis). PDF coordinate system documented by multiple sources. Build order derived from dependency analysis, not assumption. |
| Pitfalls | HIGH | Pitfalls sourced from court cases (e-signature audit trail failures), WFRMLS vendor FAQ (ToS prohibition on scraping), NAR IDX policy statement 7.58 (attribution requirements), and confirmed GitHub issues (pdf-lib coordinate bugs, font flattening). |
| Stack | HIGH | All versions verified via npm registry; OpenAI Zod v4 incompatibility confirmed via open GitHub issues #1540, #1602, #1709; pdfjs-dist server-side usage confirmed via actual codebase inspection |
| Features | HIGH for field types and signing flows; MEDIUM for AI field detection accuracy | Field behavior confirmed against DocuSign, dotloop, SkySlope docs; AI coordinate accuracy confirmed via Feb 2025 benchmarks (< 3% pixel accuracy from vision); actual accuracy on Utah forms is untested |
| Architecture | HIGH | Based on actual v1.0 codebase review (not speculative); specific file names, function names, and line numbers cited throughout; build order confirmed by dependency analysis |
| Pitfalls | HIGH | All pitfalls grounded in actual codebase inspection; specific file paths and line numbers identified (e.g., sign route line 88); no speculative claims |
**Overall confidence:** HIGH across all research areas. Sources are primary (official docs, official policies, CVE disclosures) with consistent cross-referencing. The main area of uncertainty is not technical but logistical: WFRMLS vendor enrollment timeline (2-4 weeks, outside developer control).
**Overall confidence:** HIGH
### Gaps to Address
- **WFRMLS vendor enrollment timeline:** The RESO OData API requires a vendor contract, background check, and compliance review — 2-4 weeks. Start this process on day one. Build the listings page with mock data or a dev-environment token while waiting. Do not block Phase 2 on this.
- **Exact IDX disclaimer text:** WFRMLS provides required disclaimer text that must appear on every listing page. This text changes with NAR policy updates (2024 settlement changed required fields). Obtain the current required text directly from WFRMLS before the listings feature ships — do not copy from another agent's site.
- **Which Utah standard forms Teressa uses most frequently:** The manual upload workflow (replacing planned scraping) needs a curated set of base forms pre-loaded. Teressa should identify the 5-10 forms she uses in 90% of transactions so they can be manually uploaded and stored as reusable document templates in the app. This is a product/content decision, not a technical one.
- **Playwright deployment strategy (if forms import is added in v1.x):** Vercel serverless functions cannot run a full Playwright browser due to size limits. If the forms library import feature is added after v1, the Playwright scraping job must run on a separate service (Railway, Render, or a $5/month VPS with Browserless.io). This architecture decision must be made before any scraping code is written — and the ToS decision must be re-evaluated at that time.
- **SPF/DKIM/DMARC for teressacopelandhomes.com:** Signing link emails sent from an unverified sender domain will go to spam. DNS records must be configured before any signing link is sent to a real client. This is a DNS/infrastructure task, not a code task — it must be in the Phase 6 acceptance criteria.
- **AI coordinate accuracy on real Utah forms:** Research confirms the text-extraction + label-matching approach is correct, but accuracy on actual Utah REPC and listing agreement forms is untested. Phase 5 must include an integration test with real forms before the feature ships.
- **Preview file lifecycle in production:** The `_preview_{timestamp}.pdf` pattern creates unbounded file growth in `uploads/`. A cleanup strategy (delete previews older than 24 hours, or delete on document send) needs to be decided before Phase 4 implementation.
- **Deployment target for preview writes:** The write-to-disk preview pattern silently fails on Vercel serverless (ephemeral filesystem). Confirm whether the app runs on Vercel serverless or a persistent container before implementing Phase 4.
## Sources
### Primary (HIGH confidence)
- [Next.js 15.5 Release Notes](https://nextjs.org/blog/next-15-5) — framework version and stable features
- [Next.js 16 Upgrade Guide](https://nextjs.org/docs/app/guides/upgrading/version-16) — breaking changes confirming 15 is correct choice
- [CVE-2025-29927 — Next.js Middleware Bypass (Vercel)](https://nextjs.org/blog/cve-2025-29927) — middleware auth bypass requiring defense-in-depth
- [@pdfme/pdf-lib npm page](https://www.npmjs.com/package/@pdfme/pdf-lib) — v5.5.8, active maintenance confirmed
- [pdfjs-dist npm](https://www.npmjs.com/package/pdfjs-dist) — v5.5.207 current
- [signature_pad npm](https://www.npmjs.com/package/signature_pad) — v5.1.3
- [better-auth npm](https://www.npmjs.com/package/better-auth) — v1.5.5; Auth.js merge confirmed
- [UtahRealEstate.com Vendor Data Services](https://vendor.utahrealestate.com/) — RESO OData API and forms ToS confirmed
- [WFRMLS RESO OData API Examples](https://www.reso.org/web-api-examples/mls/utah-mls/) — API field names and query syntax
- [NAR IDX Policy Statement 7.58](https://www.nar.realtor/handbook-on-multiple-listing-policy/advertising-print-and-electronic-section-1-internet-data-exchange-idx-policy-policy-statement-7-58) — listing attribution requirements
- [ESIGN/UETA Audit Trail Schema — Anvil Engineering](https://www.useanvil.com/blog/engineering/e-signature-audit-trail-schema-events-json-checklist/) — legal audit requirements
- [pdf-lib Field Coordinates Issue — GitHub #602](https://github.com/Hopding/pdf-lib/issues/602) — coordinate API gap confirmed
- [Utah DRE State-Approved Forms](https://commerce.utah.gov/realestate/real-estate/forms/state-approved/) — public domain forms source
- [Vercel Blob documentation](https://vercel.com/docs/vercel-blob) — storage strategy confirmed
- [Prisma 6.19.0 announcement](https://www.prisma.io/blog/announcing-prisma-6-19-0) — version confirmed
- [Neon + Vercel integration](https://vercel.com/marketplace/neon) — serverless DB strategy
- `src/lib/db/schema.ts` (actual codebase, inspected 2026-03-21) — `SignatureFieldData` has no `type` field confirmed
- `src/app/api/sign/[token]/route.ts` line 88 (actual codebase) — unfiltered `signatureFields` sent to client confirmed
- `src/app/portal/(protected)/documents/[docId]/_components/FieldPlacer.tsx` (actual codebase) — single "Signature" token; `screenToPdfCoords` Y-inversion pattern confirmed
- [openai npm](https://www.npmjs.com/package/openai) — v6.32.0 confirmed, Node 20 requirement
- [OpenAI Structured Outputs docs](https://platform.openai.com/docs/guides/structured-outputs) — manual json_schema format confirmed
- [openai-node Issue #1540](https://github.com/openai/openai-node/issues/1540) — zodResponseFormat broken with Zod v4
- [openai-node Issue #1602](https://github.com/openai/openai-node/issues/1602) — zodTextFormat broken with Zod v4
- [openai-node Issue #1709](https://github.com/openai/openai-node/issues/1709) — Zod 4.1.13+ discriminated union break
- [@cantoo/pdf-lib npm](https://www.npmjs.com/package/@cantoo/pdf-lib) — v2.6.3; createTextField, createCheckBox, drawImage APIs confirmed
- [react-pdf ArrayBuffer detach issue #1657](https://github.com/wojtekmaj/react-pdf/issues/1657) — ArrayBuffer copy workaround confirmed
- [Vercel Serverless Function Limits](https://vercel.com/docs/functions/runtimes/node-js#memory-and-compute) — 256MB default memory, 60s max execution on Pro
- [Utah Division of Real Estate — State Approved Forms](https://realestate.utah.gov/real-estate/forms/state-approved/) — REPC form structure context
### Secondary (MEDIUM confidence)
- [NextAuth vs Clerk vs better-auth comparison (supastarter)](https://supastarter.dev/blog/better-auth-vs-nextauth-vs-clerk) — auth selection rationale (third-party analysis, but findings align with primary sources)
- [JavaScript PDF Libraries Comparison 2025 (Nutrient)](https://www.nutrient.io/blog/javascript-pdf-libraries/) — PDF library selection (vendor-written but technically accurate)
- [MLS Listing Data Freshness — MLSImport](https://mlsimport.com/fix-outdated-listings-on-your-wordpress-real-estate-site/) — delta sync approach
- [Playwright vs Puppeteer 2025 (BrowserStack)](https://www.browserstack.com/guide/playwright-vs-puppeteer) — scraping tool selection
### Tertiary (LOW confidence — validate during implementation)
- [E-Signature UX Best Practices (various vendors)](https://www.esignglobal.com/blog/best-practices-embedded-signing-user-experience-ux) — UX recommendations sourced from vendor blogs; general patterns are sound but specific metrics need validation against real user behavior
- WFRMLS exact required disclaimer text — must be obtained directly from WFRMLS; text changes with NAR policy and cannot be reliably sourced from third parties
- [Edge AI and Vision Alliance — SAM 2 + GPT-4o (Feb 2025)](https://www.edge-ai-vision.com/2025/02/sam-2-gpt-4o-cascading-foundation-models-via-visual-prompting-part-2/) — GPT-4o returns accurate bounding box coordinates in < 3% of attempts
- [Instafill.ai — Real estate law flat PDF form automation (Feb 2026)](https://blog.instafill.ai/2026/02/18/case-study-real-estate-law-flat-pdf-form-automation/) — hybrid text-extraction + LLM approach confirmed as production pattern
- [DocuSign community — routing order for real estate](https://community.docusign.com/esignature-111/prefill-fields-before-sending-envelope-for-signature-180) — agent order 1, client order 2 confirmed
- [Dotloop support — date auto-stamp behavior](https://support.dotloop.com/hc/en-us/articles/217936457-Adding-Signatures-or-Initials-to-Locked-Templates) — date field auto-stamp pattern confirmed
- [DocuSign community — Date Signed field](https://community.docusign.com/esignature-111/am-i-able-to-auto-populate-the-date-field-2271) — read-only auto-populated date confirmed
---
*Research completed: 2026-03-19*
*Research completed: 2026-03-21*
*Ready for roadmap: yes*