Files

Chandler Copeland 622ca3dc21 docs: complete project research

2026-04-03 14:47:06 -06:00

22 KiB

Raw Blame History

Project Research Summary

Project: teressa-copeland-homes Domain: Real estate agent website + AI-assisted document signing portal Researched: 2026-04-03 Milestone scope: v1.1 (AI field placement, expanded field types, agent signature, filled preview) + v1.2 (multi-signer support, Docker production deployment) Confidence: HIGH

Executive Summary

This is a solo real estate agent's custom document signing portal built on Next.js 15 + Drizzle ORM + PostgreSQL (Neon). The core v1.0 feature set — PDF upload, drag-drop field placement, email-link signing, presigned download — is complete. Two milestones of new capability are being planned. The v1.1 milestone adds AI-assisted field detection (GPT-4.1 + PDF text extraction), five new field types (checkbox, initials, date, agent signature, text), a saved agent signature, and a filled-document preview before send. The v1.2 milestone extends single-signer to multi-signer (parallel, any-order) and adds production Docker deployment with SMTP fix. Both milestones require zero new npm packages beyond openai (already installed for v1.1) — all capability is extension of the existing stack.

The most important architectural constraint for both milestones: the current signing flow has load-bearing single-signer assumptions throughout (one token per document, first token claim marks the document Signed, documents.status tracks one signer's journey). Multi-signer requires a deliberate schema-first approach — add the signers JSONB column, signerEmail to signingTokens, and a completion detection rewrite before touching any UI. Building the send route or signing UI before the schema is solid will create hard-to-unwind bugs. For Docker, the single non-obvious pitfall is that NEXT_PUBLIC_BASE_URL is baked at build time — signing URLs will point to localhost unless the variable is renamed to a server-only name before the production image is built.

For v1.1, AI field placement must use the text-extraction approach (pdfjs-dist getTextContent() → GPT-4.1 for label classification → coordinates from text bounding boxes). Vision-based coordinate inference from PDF images has under 3% accuracy in published benchmarks and is not production-viable. The hybrid text+AI approach is the pattern used by Apryse, Instafill, and DocuSign Iris. On Utah standard forms (REPC, listing agreements), which have consistent label patterns, accuracy should be 90%+. The fallback — scanned/image-based PDFs — should degrade gracefully to manual drag-drop with a clear agent-facing message.

Key Findings

Recommended Stack

The existing stack handles every v1.1 and v1.2 requirement without new packages. openai@6.32.0 (already installed) covers AI field placement. @cantoo/pdf-lib@2.6.3 handles all five new field types and PDF assembly for both single-signer and multi-signer workflows. signature_pad@5.1.3 (already installed) handles agent signature capture. react-pdf@10.4.1 handles filled document preview. Docker deployment requires no npm changes — it is a Dockerfile + docker-compose.yml infrastructure concern.

Core technologies:

openai@6.32.0: GPT-4.1 structured extraction via manual JSON schema — not zodResponseFormat, which is broken with Zod v4 (confirmed open bug)
pdfjs-dist (legacy build, already installed): PDF text extraction with bounding boxes for AI field placement pipeline
@cantoo/pdf-lib@2.6.3: All new field types (text, checkbox, initials, date); agent signature via embedPng() + drawImage(); coordinate system: y=0 at page bottom (transform from pdfjs-dist y=0-at-top)
signature_pad@5.1.3: Agent signature canvas — use as plain class with useRef<HTMLCanvasElement> in React, not a React wrapper
react-pdf@10.4.1: Filled document preview from ArrayBuffer; always copy buffer before passing (detachment bug in v7+)
@vercel/blob: File storage — installed but currently unused (dead dependency risk; see Gaps)
nodemailer@7.0.13: SMTP email for signing links and completion notifications
node:20-slim (Debian-based): Docker base image — required for @napi-rs/canvas glibc native binary compatibility; do NOT use Alpine (musl libc incompatible)

Critical version constraint: zodResponseFormat from openai/helpers/zod throws runtime exceptions with Zod v4.x. Use response_format: { type: "json_schema", ... } with a hand-written schema.

Expected Features

Must have (table stakes — all major real estate e-sig tools provide these):

Initials field type — every Utah standard form (REPC, listing agreement) has per-page initials lines
Date field (auto-stamp only, not a calendar picker) — records actual signing timestamp; client cannot type a date
Checkbox field type — boolean elections throughout Utah REPC and addenda
Agent signature field type — pre-filled during agent signing flow; read-only for client
Agent saved signature (draw once, reuse) — DocuSign "Adopted Signature" pattern; re-drawing per document is a daily friction point
Agent signs before sending (routing order: agent first, client second) — industry convention in real estate
Filled document preview before send — prevents wrong-version sends; agents expect to see what client sees

Should have (differentiators for this solo-agent, Utah-forms workflow):

AI field placement (text extraction + GPT-4.1 label classification) — eliminates manual drag-drop for known Utah forms; 90%+ accuracy on standard forms
AI pre-fill from client profile data (name, property address, date) — populates obvious fields; agent reviews in PreparePanel
Property address field on client profile — enables AI pre-fill to be transaction-specific

Defer to v2+:

AI confidence scores surfaced to agent — adds UI complexity; preview step catches gaps instead
Template save from AI placement — high value but requires template management UI; validate AI accuracy first
Per-document agent signature redraw — adds decision fatigue; profile settings only is the right UX

Multi-signer (v1.2 must-haves per PROJECT.md):

Parallel signing in any order — no sequencing enforcement
Per-signer field isolation — each signer sees only fields tagged to their email
Completion detection after all signers finish — agent notified only when last signer completes
Final PDF distributed to all parties on completion

Architecture Approach

The system is a clean single-signer flow extended to multi-signer via additive schema changes without breaking existing documents. The key architectural principle: signingTokens is the source of truth for signer identity and completion state — never derive this from the signatureFields JSONB array. The per-signer PDF accumulation pattern (each signer embeds into the running signedFilePath, protected by a Postgres advisory lock) eliminates the need for a separate merge step and works with the existing embedSignatureInPdf() function unchanged.

Major components:

SignatureFieldData JSONB (on documents) — field placement data; extended with optional signerEmail for multi-signer field routing; signerEmail absent = legacy single-signer
signingTokens table — one row per signer per document; extended with signerEmail (nullable) and viewedAt columns; source of truth for who needs to sign and who has signed
documents.signers JSONB column (new) — ordered list of { email, name, tokenJti, signedAt } per document; coexists with legacy assignedClientId for backward compatibility
documents.completionTriggeredAt column (new) — one-time-set guard preventing duplicate final PDF assembly on concurrent last-signer submissions
POST /api/sign/[token] (modified) — atomic token claim, advisory lock PDF accumulation, completion detection via unclaimed-token count, document_completed event + notifications when all done
signing-mailer.tsx (modified) — multi-signer path: Promise.all() over signers; new sendAllSignersCompletionEmail()
PreparePanel + FieldPlacer (modified) — signer list entry UI, per-field signer assignment, color-coded markers, send-block validation

Critical Pitfalls

First signer marks document Signed prematurely — the current POST /api/sign/[token] unconditionally sets documents.status = 'Signed' on any token claim. With two signers, Signer A's completion fires the agent notification and marks the document complete before Signer B has signed. Fix: replace with a count of unclaimed tokens for the document; only set Signed when count reaches zero.
Race condition — two simultaneous last signers both trigger PDF assembly — atomic token claim does not prevent two concurrent handlers from both detecting zero remaining tokens. Fix: add completionTriggeredAt TIMESTAMP to documents; use UPDATE WHERE completionTriggeredAt IS NULL RETURNING guard — same pattern as the existing token claim. Zero rows returned means another handler already won; skip assembly.
NEXT_PUBLIC_BASE_URL baked at Docker build time, not runtime — signing URLs will contain localhost:3000 in production if this variable is not renamed before building the image. Fix: rename to SIGNING_BASE_URL (no NEXT_PUBLIC_ prefix); inject at container runtime via env_file:; the send route is server-side only and never needs a public variable.
@napi-rs/canvas native binary incompatible with Alpine Docker images — node:alpine uses musl libc; @napi-rs/canvas only ships glibc prebuilt binaries. The module fails with invalid ELF header at runtime. Fix: use node:20-slim (Debian); build with explicit --platform linux/amd64 when deploying to x86 from an ARM Mac.
Uploads directory ephemeral in Docker — all PDFs are written to process.cwd()/uploads inside the container's writable layer. Container recreation (deployment, crash) permanently deletes all documents including legally executed signed copies. Fix: mount a named Docker volume at /app/uploads in docker-compose.yml before the first production upload.
SMTP env vars absent in container — CONTACT_SMTP_HOST etc. exist in .env.local on the dev host but are invisible to Docker unless explicitly injected. Nodemailer createTransporter() is called at send time, not startup, so the missing vars produce no startup error — only silent email failure at first send. Fix: env_file: .env.production in docker-compose.yml. Verify with docker exec <container> printenv CONTACT_SMTP_HOST.
Legacy signingTokens rows break if signerEmail added as NOT NULL — existing rows have no signer email. Fix: add column as nullable (TEXT); backfill existing rows in the migration using a JOIN to the client email; all signing code handles the null legacy case.
Neon connection pool exhaustion — postgres(url) defaults to 10 connections, which saturates Neon free tier entirely. Fix: postgres(url, { max: 5, idle_timeout: 20, connect_timeout: 10 }) in db/index.ts.

Implications for Roadmap

Based on combined research, a five-phase structure is recommended:

Phase 1: Complete v1.1 — AI Field Placement and Expanded Field Types

Rationale: v1.1 work is in progress (MEMORY.md confirms active debugging of AI field classification). This phase finishes in-flight work before adding multi-signer complexity on top of an unstable foundation. Delivers: Working AI field detection pipeline (text extraction + GPT-4.1 classification + post-processing rules), all five field types (checkbox, initials, date, agent signature, text), agent saved signature, filled document preview, agent signs first workflow, property address on client profile, AI pre-fill from client profile. Addresses: All FEATURES.md P1 items. Avoids: Vision-based coordinate inference (use text extraction only); zodResponseFormat with Zod v4 (use manual JSON schema). Research flag: No additional research needed — patterns are implemented and being debugged.

Phase 2: Multi-Signer Schema Foundation

Rationale: Schema changes are additive and backward-compatible. They must be deployed and validated against production Neon before any multi-signer sending or signing code is written. This phase has no user-visible effect — it is purely infrastructure. Delivers: Migration drizzle/0010_multi_signer.sql: signer_email on signingTokens (nullable), signers JSONB on documents, completionTriggeredAt on documents, viewedAt on signingTokens, three new auditEventTypeEnum values. TypeScript additions: DocumentSigner interface, signerEmail on SignatureFieldData, getSignerEmail() helper. Avoids: Legacy token breakage (nullable column), race condition setup (completionTriggeredAt column ready), enum ADD VALUE transaction issues (statement-breakpoint between each ALTER). Research flag: Standard Drizzle migration patterns — no additional research needed.

Phase 3: Multi-Signer Backend (Token Creation, Signing Flow, Completion)

Rationale: Backend-complete before UI. The signing POST rewrite is the highest-complexity change and must be independently testable via API calls before any UI work begins. Delivers: Updated createSigningToken(docId, signerEmail?), signer-aware field filtering in signing GET (null signerEmail = legacy path), accumulator PDF pattern + advisory lock in signing POST, completion detection via unclaimed-token count + completionTriggeredAt guard, document_completed audit event, sendAllSignersCompletionEmail(), updated send route per-signer token loop. All changes preserve the legacy single-signer path. Uses: pg_advisory_xact_lock(hashtext(documentId)) for concurrent PDF write protection, existing embedSignatureInPdf() unchanged, existing logAuditEvent() and sendAgentNotificationEmail(). Avoids: Premature completion (token count check), race condition (completionTriggeredAt guard), audit trail gap (signerEmail in event metadata), partial-send failure (individual email failure must not void already-created tokens). Research flag: The advisory lock interaction with Drizzle transactions may warrant a focused research pass if the developer is unfamiliar with Postgres advisory locks.

Phase 4: Multi-Signer UI (PreparePanel + FieldPlacer)

Rationale: UI changes come last — they have no downstream dependencies and are safe to build once the backend is fully operational and tested. Delivers: Multi-signer list entry in PreparePanel (name + email rows, add/remove, replaces single email textarea), PUT /api/documents/[id]/signers endpoint, per-field signer email selector in FieldPlacer, color-coded field markers by signer, send-block validation (all client-visible fields must have signerEmail before send is allowed), agent dashboard showing per-signer status. Avoids: Chicken-and-egg ordering problem (signer list is saved to documents.signers before FieldPlacer is loaded; enforce in UI). Research flag: Standard React and Next.js API patterns — no additional research needed.

Phase 5: Docker Production Deployment + SMTP Fix

Rationale: Deployment comes after all application features are complete and tested locally. The SMTP fix and NEXT_PUBLIC_BASE_URL rename must be verified before multi-signer completion emails are tested in production. Delivers: Three-stage Dockerfile (node:20-slim, Next.js output: 'standalone'), docker-compose.yml with named uploads volume + Neon connection via env_file:, NEXT_PUBLIC_BASE_URL renamed to SIGNING_BASE_URL, Neon connection pool limits (max: 5) in db/index.ts, /api/health endpoint, .dockerignore, SMTP transporter consolidated to shared utility, deployment runbook. Avoids: All Docker pitfalls: ephemeral uploads (named volume), wrong base image (node:20-slim not alpine), build-time URL baking (rename variable), SMTP failure (env_file + verification step), connection exhaustion (explicit max), startup order failure (healthcheck + depends_on if using local Postgres). Research flag: Standard patterns — official Next.js Docker documentation covers the three-stage standalone pattern precisely.

Phase Ordering Rationale

v1.1 must complete before v1.2 begins: multi-signer adds schema complexity on top of the field placement and signing pipeline; debugging both simultaneously is high-risk.
Schema before backend: multi-signer schema changes are additive and can deploy to production Neon independently; backend code written against the old schema cannot be safely run.
Backend before UI: the signing POST rewrite is the load-bearing change; FieldPlacer signer assignment has no effect until token and completion logic is correct.
Deployment last: Docker work is independent of application features but should be done with a complete, tested application to avoid conflating feature bugs with deployment bugs.

Research Flags

Phases likely needing /gsd:research-phase during planning:

Phase 3 (Multi-Signer Backend): Advisory lock pattern and Drizzle transaction interaction may need a targeted research pass for developers unfamiliar with pg_advisory_xact_lock.

Phases with standard patterns (skip research-phase):

Phase 1 (v1.1): Already researched and partially implemented.
Phase 2 (Schema): Drizzle migration patterns are routine.
Phase 4 (Multi-Signer UI): Standard React + API patterns.
Phase 5 (Docker): Officially documented Next.js standalone Docker pattern.

Confidence Assessment

Area	Confidence	Notes
Stack	HIGH	All decisions code-verified against `package.json` and running source files. Zero speculative library choices. Zod v4 / zodResponseFormat incompatibility confirmed via open GitHub issues.
Features	HIGH	Industry analysis of DocuSign, dotloop, SkySlope DigiSign confirms field type behavior and agent-signs-first convention. AI coordinate accuracy from February 2025 published benchmarks.
Architecture	HIGH	Multi-signer design based on direct inspection of schema.ts, signing route, send route. Advisory lock pattern is Postgres-standard. OpenSign reference confirms the data model approach.
Pitfalls	HIGH	All pitfalls cite specific file paths and code from the v1.1 codebase. NEXT_PUBLIC bake-at-build confirmed against Next.js official docs. No speculative claims.

Overall confidence: HIGH

Gaps to Address

@vercel/blob dead dependency: Installed in package.json but not used anywhere in the codebase. Risks accidental use in future code that would silently fail in a non-Vercel Docker deployment. Decision needed before Phase 5: remove the package, or document it as unused and ensure it stays that way.
Nodemailer port mismatch between mailers: signing-mailer.tsx defaults to port 465 (implicit TLS) and contact-mailer.ts defaults to port 587 (STARTTLS) when CONTACT_SMTP_PORT is not set. A shared createSmtpTransporter() utility is needed; this should be addressed in Phase 5 before the first production email test.
AI accuracy on non-standard forms: The 90%+ accuracy expectation applies to Utah standard forms with consistent label patterns. The system's behavior on commercial addenda, non-standard addenda, or scanned PDFs is untested. The graceful fallback (manual drag-drop with a clear message) handles the failure case, but real-world accuracy across Teressa's full form library should be validated in Phase 1 before committing to the AI-first workflow.
Local Postgres vs. Neon in Docker: The research and architecture assume Neon as the production database (external managed service). If a local postgres Docker service is substituted, the depends_on: service_healthy pattern documented in ARCHITECTURE.md applies. The current research covers the Neon path only.

Sources

Primary (HIGH confidence)

Direct code audit: src/lib/ai/extract-text.ts, src/lib/ai/field-placement.ts, src/lib/db/schema.ts, src/lib/signing/signing-mailer.tsx, src/lib/db/index.ts, next.config.ts, package.json — architecture verification and pitfall specifics
OpenAI Structured Outputs docs — manual JSON schema format confirmed
openai-node Issue #1540 — zodResponseFormat broken with Zod v4
openai-node Issue #1602 — zodTextFormat broken with Zod v4
@cantoo/pdf-lib npm page — v2.6.3 field type API
signature_pad GitHub — v5.1.3 canvas API
Docker Official Next.js Containerize Guide — three-stage Dockerfile
Next.js with-docker official example — standalone mode pattern
Next.js env var classification — NEXT_PUBLIC_ bake-at-build-time behavior
Docker Compose Secrets — Official Docs — env_file vs secrets decision
OpenSign GitHub Repository — multi-signer reference implementation (signers array in document record)

Secondary (MEDIUM confidence)

Next.js Standalone Docker Mode — DEV Community (2025) — image size benchmarks (~300 MB vs ~7 GB)
Docker DNS EAI_AGAIN — dev.to (2025) — DNS resolution behavior in Docker bridge network
Docker Compose Secrets: What Works, What Doesn't — plain Compose vs Swarm secrets security comparison
Published AI bounding box accuracy benchmarks (February 2025) — under 3% accuracy for vision-based coordinate inference from PDF images
Nodemailer Docker EAI_AGAIN — Docker Forums — root cause confirmed as env var injection, not DNS

Research completed: 2026-04-03 Milestone coverage: v1.1 (AI field placement + expanded field types) + v1.2 (multi-signer + Docker) Ready for roadmap: yes

22 KiB Raw Blame History