From f83dba5e69c77608317e72582489fbf582877158 Mon Sep 17 00:00:00 2001 From: Chandler Copeland Date: Mon, 6 Apr 2026 11:54:40 -0600 Subject: [PATCH] docs: complete project research --- .../.planning/research/ARCHITECTURE.md | 331 ++++++++++++++++++ .../.planning/research/FEATURES.md | 183 ++++++++++ .../.planning/research/PITFALLS.md | 256 ++++++++++++++ .../.planning/research/STACK.md | 216 ++++++++++++ .../.planning/research/SUMMARY.md | 177 ++++++++++ 5 files changed, 1163 insertions(+) create mode 100644 teressa-copeland-homes/.planning/research/ARCHITECTURE.md create mode 100644 teressa-copeland-homes/.planning/research/FEATURES.md create mode 100644 teressa-copeland-homes/.planning/research/PITFALLS.md create mode 100644 teressa-copeland-homes/.planning/research/STACK.md create mode 100644 teressa-copeland-homes/.planning/research/SUMMARY.md diff --git a/teressa-copeland-homes/.planning/research/ARCHITECTURE.md b/teressa-copeland-homes/.planning/research/ARCHITECTURE.md new file mode 100644 index 0000000..de2923a --- /dev/null +++ b/teressa-copeland-homes/.planning/research/ARCHITECTURE.md @@ -0,0 +1,331 @@ +# Architecture Patterns: Document Templates + +**Domain:** Real-estate agent portal — PDF signing with field placement +**Researched:** 2026-04-06 +**Confidence:** HIGH (based on direct code inspection) + +--- + +## Context: What Was Inspected + +| File | Purpose | +|------|---------| +| `src/lib/db/schema.ts` | Full Drizzle schema — all tables, types, helpers | +| `src/app/api/documents/route.ts` | Document creation (two paths: library copy + custom upload) | +| `src/app/api/documents/[id]/fields/route.ts` | GET/PUT for `signatureFields` JSONB | +| `src/app/api/forms-library/route.ts` | Lists `form_templates` rows | +| `documents/[docId]/_components/DocumentPageClient.tsx` | State owner for document editing page | +| `documents/[docId]/_components/FieldPlacer.tsx` | Drag-drop field placement (822 lines) | +| `documents/[docId]/_components/PreparePanel.tsx` | Prepare + send side panel | +| `portal/_components/AddDocumentModal.tsx` | "Add Document" modal — uses form_templates | +| `portal/_components/ClientProfileClient.tsx` | Client profile page with document list | +| `drizzle/` | 12 migration files (0000–0011) | + +--- + +## Current Architecture + +### Schema Summary + +``` +form_templates (id, name, filename, createdAt, updatedAt) + — raw PDF registry only, no field data + +documents (id, name, clientId, status, formTemplateId FK, + filePath, signatureFields JSONB, textFillData JSONB, + signers JSONB, preparedFilePath, signedFilePath, + emailAddresses, pdfHash, sentAt, signedAt, + completionTriggeredAt, assignedClientId) + — per-client working copy, carries all field placement state + +clients (id, name, email, contacts JSONB, propertyAddress, + createdAt, updatedAt) + +signing_tokens (jti, documentId FK, signerEmail, expiresAt, usedAt) +audit_events (id, documentId FK, eventType, metadata JSONB, ...) +users (id, email, passwordHash, agentSignatureData, + agentInitialsData) +``` + +### Document Creation Flow (current) + +``` +AddDocumentModal + → POST /api/documents + ├── (library path) formTemplateId provided + │ → copy seed PDF to uploads/clients/{clientId}/{docId}.pdf + │ → INSERT documents (no signatureFields — empty, to be placed) + └── (upload path) custom file + → write file to uploads/clients/{clientId}/{docId}.pdf + → INSERT documents (no signatureFields) +``` + +After creation, agent opens `/portal/documents/{docId}`, uses FieldPlacer to place fields +(drag-drop or AI auto-place), then uses PreparePanel to fill text values and send. + +### FieldPlacer Integration Point + +`FieldPlacer` is tightly coupled to `docId`: + +- Fetches fields from `GET /api/documents/{docId}/fields` +- Persists changes to `PUT /api/documents/{docId}/fields` +- The `aiPlacementKey` prop triggers a re-fetch after AI auto-place + +The component has **no concept of a "template source"** — it only reads/writes document fields. +That is the integration seam where template pre-loading must be injected. + +--- + +## Schema Decision: New Table vs Extend `form_templates` + +**Decision: New `document_templates` table. Do not extend `form_templates`.** + +### Reasoning + +`form_templates` is currently a **file registry** — a pointer to a PDF filename on disk. +Every part of the system treats it that way: + +- `forms-library/route.ts` returns `{ id, name, filename }` only +- `documents/route.ts` uses it only to resolve `template.filename` for `copyFile()` +- `AddDocumentModal` renders it as a name list for file selection +- `documents.formTemplateId` is a FK used only for audit/traceability + +Adding `signatureFields` + `signers` JSONB to `form_templates` would: +1. Break the clear single-responsibility of that table +2. Force every consumer that reads `form_templates` to handle nullable field data +3. Make the "template has fields" state conflated with "template is just a PDF" +4. Create confusion: which `form_templates` rows have fields? All? Some? + +A separate `document_templates` table is clean: +- Represents an intentionally-prepared template (PDF + fields + signer role slots) +- Can be soft-deleted, versioned, or duplicated without touching `form_templates` +- `form_templates` continues working exactly as it does today +- The new table has an optional FK to `form_templates` for traceability + +### Proposed Schema Addition + +```typescript +// New interface — signer role slot (no email yet, filled at document creation) +export interface TemplateSignerSlot { + role: string; // e.g. "Buyer", "Seller", "Agent" + color: string; // display color, matches FieldPlacer SIGNER_COLORS +} + +// SignatureFieldData already has signerEmail — for templates, signerEmail +// holds the ROLE string (e.g. "Seller") rather than an actual email. +// This is a deliberate dual-use: roles are resolved to emails at apply time. + +export const documentTemplates = pgTable("document_templates", { + id: text("id").primaryKey().$defaultFn(() => crypto.randomUUID()), + name: text("name").notNull(), + // Optional link back to the form_templates raw PDF + formTemplateId: text("form_template_id").references(() => formTemplates.id), + // Stored field layout — same shape as documents.signatureFields + // signerEmail values are ROLE strings at this stage + signatureFields: jsonb("signature_fields").$type(), + // Signer role slots — defines which roles exist and their colors + signerSlots: jsonb("signer_slots").$type(), + createdAt: timestamp("created_at").defaultNow().notNull(), + updatedAt: timestamp("updated_at").defaultNow().notNull(), +}); +``` + +**Backwards compatibility:** Zero impact on existing documents. `form_templates` is unchanged. +All existing documents with `signatureFields: null` continue working (FieldPlacer handles empty arrays). + +--- + +## Component Boundaries + +### New Components + +| Component | Location | Responsibility | +|-----------|----------|---------------| +| `TemplatesPageClient` | `portal/_components/` | List templates, launch editor, delete | +| `TemplateEditorPage` (server) | `portal/(protected)/templates/[templateId]/page.tsx` | Loads template data, renders editor shell | +| `TemplateEditorClient` | `portal/(protected)/templates/[templateId]/_components/` | Thin wrapper — composes FieldPlacer + TemplatePanel | +| `TemplatePanel` | same directory | Sidebar: signer slot management, save, AI auto-place | +| `TemplatePicker` | `portal/_components/` | Dropdown/modal used in AddDocumentModal to choose a document_template | + +### Modified Components + +| Component | Change | +|-----------|--------| +| `AddDocumentModal` | Add "Start from template" option that calls a new `POST /api/templates/{id}/apply` endpoint | +| `FieldPlacer` | Add optional `templateId` prop (or make `docId` accept either); add `onSaveAsTemplate` callback. Alternatively: keep FieldPlacer document-only and build a thin TemplateFieldPlacer wrapper that remaps the API calls. | + +**Recommendation on FieldPlacer:** Keep FieldPlacer document-only. Build `TemplateFieldPlacer` as a thin wrapper that intercepts the `persistFields` call (currently hardcoded to `/api/documents/{docId}/fields`) by accepting an `onPersist` callback prop, or by adding a `persistUrl` override prop. This is a 5-line change to FieldPlacer and avoids branching logic inside the 822-line component. + +--- + +## New API Routes + +| Route | Method | Purpose | +|-------|--------|---------| +| `GET /api/templates` | GET | List all document_templates | +| `POST /api/templates` | POST | Create new document_template (blank or from form_template) | +| `GET /api/templates/[id]` | GET | Fetch single template (with signatureFields) | +| `PATCH /api/templates/[id]` | PATCH | Save field layout + signer slots | +| `DELETE /api/templates/[id]` | DELETE | Soft or hard delete | +| `POST /api/templates/[id]/apply` | POST | Apply template to a new document (see flow below) | + +--- + +## "Apply Template to Document" Operation + +This is the core integration between templates and the existing creation flow. + +``` +POST /api/templates/{templateId}/apply +Body: { clientId, name, signerRoleMap: { "Seller": "email@client.com", ... } } + +Server: +1. Fetch document_template row (fields + signerSlots) +2. Fetch form_templates row for the PDF filename (via template.formTemplateId) +3. Copy PDF file to uploads/clients/{clientId}/{docId}.pdf +4. Deep-clone signatureFields array +5. For each field: replace signerEmail (role string) with actual email from signerRoleMap +6. INSERT documents with: clientId, name, formTemplateId, filePath, signatureFields (resolved), + signers (built from signerRoleMap + slot colors) +7. Return created document + +Client (AddDocumentModal): +- Shows template picker (list from GET /api/templates) +- Shows role-to-email assignment UI (one row per signerSlot, pre-populated from client.contacts) +- On submit: calls POST /api/templates/{id}/apply +- On 201: router.refresh() + onClose() — same pattern as current flow +``` + +The document created this way is indistinguishable from a hand-placed document. +The agent can still edit fields after creation — FieldPlacer works identically. + +--- + +## Template Editor Architecture + +``` +/portal/(protected)/templates/[templateId]/page.tsx (Server Component) + → fetches document_template row + → renders TemplateEditorClient + +TemplateEditorClient + → state: fields (from template.signatureFields), signerSlots, dirty flag + → layout: same 2-column grid as DocumentPageClient (PDF left, panel right) + + Left: TemplateFieldPlacer + → wraps FieldPlacer with persistUrl="/api/templates/{templateId}/fields" + → OR: FieldPlacer gets onPersist callback prop (cleaner) + + Right: TemplatePanel + → Signer slot list (add/remove roles, assign colors) + → Save button (PATCH /api/templates/{templateId}) + → AI Auto-Place button (reuses existing /api/documents/{docId}/ai-prepare pattern, + needs a parallel /api/templates/{id}/ai-prepare endpoint) +``` + +**AI auto-place for templates:** The existing `ai-prepare` route writes to `documents.signatureFields`. A parallel `POST /api/templates/{id}/ai-prepare` writes to `document_templates.signatureFields` instead. The extraction + classification logic in `src/lib/ai/` is document-agnostic and can be called from either route without modification. + +--- + +## Data Flow Diagram + +``` +form_templates (PDF registry) + │ + │ formTemplateId FK (optional) + ▼ +document_templates (field layout + signer slots) + │ + │ POST /api/templates/{id}/apply + │ → resolves role → email + ▼ +documents (working copy, per client) + │ + ├── signatureFields JSONB ← FieldPlacer reads/writes + ├── signers JSONB ← PreparePanel reads/writes + └── textFillData JSONB ← PreparePanel reads/writes +``` + +--- + +## Scalability Considerations + +| Concern | Now (one agent) | Future (multi-agent SaaS) | +|---------|-----------------|--------------------------| +| Template ownership | All templates belong to one agent | Add `userId` FK to `document_templates` | +| Template sharing | N/A | Separate `shared_templates` or `userId: null` convention | +| Template versioning | Not needed | Add `version` int or duplicate-on-edit pattern | +| PDF storage | Local filesystem (uploads/) | Already abstracted in route — swap `copyFile` for S3 copy | + +--- + +## Build Order + +Dependencies flow downward — each step enables the next. + +### Step 1: Schema migration +- Add `document_templates` table to `schema.ts` +- Add `TemplateSignerSlot` interface +- Write drizzle migration (`0012_document_templates.sql`) +- **No changes to existing tables. Safe to deploy alongside live system.** + +### Step 2: Template CRUD API +- `GET /api/templates` — list +- `POST /api/templates` — create (blank shell, optionally linked to form_template) +- `GET /api/templates/[id]` — fetch +- `PATCH /api/templates/[id]` — save fields + signerSlots +- `DELETE /api/templates/[id]` — delete +- These routes follow the exact same auth + db pattern as `forms-library/route.ts` and `documents/[id]/route.ts` + +### Step 3: FieldPlacer persistence abstraction +- Add `onPersist?: (fields: SignatureFieldData[]) => Promise` prop to FieldPlacer +- When prop is provided, use it instead of the hardcoded `persistFields(docId, ...)` call +- Fallback: keep existing behavior when prop is absent +- **This is a non-breaking change — all existing consumers pass no prop, behavior unchanged** + +### Step 4: Template editor UI +- `TemplateEditorClient` + `TemplatePanel` components +- `/portal/(protected)/templates/[templateId]/page.tsx` server page +- `/portal/(protected)/templates/page.tsx` list page +- Portal nav link ("Templates") + +### Step 5: Apply template API + AddDocumentModal update +- `POST /api/templates/[id]/apply` — the apply operation (step 5 depends on step 1+2) +- Update `AddDocumentModal` to add "Start from template" path +- Role-to-email mapping UI (pre-seeded from `client.contacts`) + +### Step 6: AI auto-place for templates (optional, can defer) +- `POST /api/templates/[id]/ai-prepare` — mirrors existing route, writes to `document_templates` +- Reuse `extractBlanks()` and `classifyFieldsWithAI()` from `src/lib/ai/` unchanged + +--- + +## Anti-Patterns to Avoid + +### Anti-Pattern 1: Extending `form_templates` with field data +**What goes wrong:** `form_templates` becomes overloaded. Some rows have fields, some don't. Every consumer must handle both states. The forms-library UI must differentiate "raw PDF" from "template with fields". +**Instead:** New `document_templates` table with an optional FK back to `form_templates`. + +### Anti-Pattern 2: FieldPlacer knows about templates +**What goes wrong:** 822-line component gets template-vs-document branching throughout. Hard to test, easy to regress. +**Instead:** FieldPlacer gets a single `onPersist` callback. The caller decides what to do with the fields. + +### Anti-Pattern 3: Storing actual signer emails in template fields +**What goes wrong:** Templates must be edited when a client's email changes. Template becomes client-specific, defeating the purpose. +**Instead:** Store role strings ("Seller", "Buyer") in `signerEmail`. Resolve to actual emails at apply time. + +### Anti-Pattern 4: Applying template by mutating the template's PDF +**What goes wrong:** Templates get corrupted or have prepared PDF artifacts baked in. +**Instead:** Apply = copy raw PDF + clone + resolve fields. Template's PDF path is never touched. + +### Anti-Pattern 5: Building a new field placer from scratch for templates +**What goes wrong:** Two diverging implementations of the same complex drag-drop logic. +**Instead:** Reuse FieldPlacer via the `onPersist` prop abstraction in Step 3. + +--- + +## Sources + +- Direct code inspection of all listed files (HIGH confidence — current state) +- Schema introspection of all 12 drizzle migrations +- No external sources needed — this is pure architecture derivation from existing patterns diff --git a/teressa-copeland-homes/.planning/research/FEATURES.md b/teressa-copeland-homes/.planning/research/FEATURES.md new file mode 100644 index 0000000..cb141c5 --- /dev/null +++ b/teressa-copeland-homes/.planning/research/FEATURES.md @@ -0,0 +1,183 @@ +# Feature Research + +**Domain:** Document template system for real estate e-signing app — v1.3 Document Templates +**Researched:** 2026-04-06 +**Confidence:** HIGH for DocuSign/PandaDoc/OpenSign template model (multiple sources); HIGH for real estate re-use patterns (domain confirmed by project context); MEDIUM for edge case behaviors (version-skew risk in competitor analysis) + +--- + +## Scope Note + +This file covers only the v1.3 milestone features. Prior features (forms library, drag-drop field placement, AI auto-place, agent signature, multi-signer document sending, signing flow) are already built and validated. + +**New feature under research: Document Templates** + +A template is a saved "shell" of a prepared document: PDF source + field layout + signer role assignments + text fill hints (label + blank, no filled values). When Teressa starts a document for a new client, she picks a template instead of a blank PDF. The document is created pre-loaded with all fields; she fills in client-specific text values and assigns signer emails, then sends. + +--- + +## Feature Landscape + +### Table Stakes (Users Expect These) + +Features a real estate agent expects from any template system. Missing these makes v1.3 feel like it does not solve the re-use problem. + +| Feature | Why Expected | Complexity | Notes | +|---------|--------------|------------|-------| +| Save a prepared document as a template | Core premise — if you cannot save the field layout you just placed, there is no product | MEDIUM | Agent places fields on a PDF (manually or via AI auto-place), names the template, clicks Save. Stores: formTemplateId (source PDF), signatureFields JSONB (field layout with types + signer roles), textFillData JSONB (hint labels only — no client values), template name. No new PDF copy needed — template references the existing form_templates record. | +| Template list in portal (browse, rename, delete) | DocuSign, PandaDoc, OpenSign all provide a template library as a first-class section. Agents cannot maintain templates without list management. | LOW | Separate "Templates" nav section. List shows template name, source form name, created date, field count. Rename (modal), delete (confirm dialog). Matches the existing ClientCard/ConfirmDialog UI pattern. | +| Apply template when adding a document to a client | This is the entire value delivery — one click to get a pre-loaded document instead of starting blank | MEDIUM | When agent creates a document for a client, offer: "Start from template" vs "Start from blank PDF." Picking a template copies the field layout to the new document record. Text fill hints show as placeholder/label in PreparePanel — not as filled values. Agent fills client-specific text and assigns signer emails, then sends as normal. | +| Field layout is copied, not referenced | A template is a snapshot, not a live link. Editing the template later must not retroactively change documents already in flight. | LOW | On apply: deep-copy signatureFields and textFillData hint labels into the new documents record. The template row is just a source — documents are independent after creation. This is the industry standard (DocuSign, OpenSign both copy on apply). | +| Signer roles (not emails) stored in template | Template cannot store a specific client's email — that changes every time. Templates store a role label ("Buyer", "Seller", "Agent") on each field. Agent assigns real emails to roles when applying the template. | MEDIUM | The existing SignatureFieldData already has `signerEmail?: string`. Templates will store role labels (e.g., "signer:buyer", "signer:seller") in this field instead of real emails. On apply, agent maps roles to real emails. For solo-agent, single-signer use case the mapping step can default to the client's email automatically. | +| Text fill hints (not filled values) | Template stores the label a field should be pre-filled with ("Property Address", "Client Name") but NOT the actual values. Values come from the client record on apply. | LOW | textFillData in the template stores `{ fieldId: "hint:propertyAddress" }` or similar hint keys. On apply, known hints (propertyAddress, clientName) auto-populate from the client record. Unknown hint labels surface as PreparePanel prompts. | +| Agent can re-edit template (update the field layout) | Agents refine templates over time. DocuSign supports template editing as a standard operation. | LOW | Template editor is the same FieldPlacer UI used for document preparation, opened against the template record instead of a live document. Save overwrites the template's signatureFields and textFillData. No versioning needed at v1. | + +--- + +### Differentiators (Competitive Advantage) + +Features specific to this app's real estate context that go beyond generic template systems. + +| Feature | Value Proposition | Complexity | Notes | +|---------|-------------------|------------|-------| +| AI auto-place works in template editor | Teressa can create a new listing agreement template by clicking "Auto-Place Fields" — AI places all fields on the Utah UAR form in one step, then she saves as template. She never manually places again. | LOW | This is the existing AI auto-place feature run against a template context. No new AI code. The template editor reuses the same FieldPlacer + AI auto-place pipeline that already exists for document preparation. | +| Known hints auto-fill from client record on apply | When applying a template, fields with hint keys `clientName`, `propertyAddress`, `clientEmail`, `todayDate` are silently pre-populated from the client record and system date. Agent sees them already filled in PreparePanel. | MEDIUM | Maps hint keys to client record fields. Extensible as the client schema grows. Removes manual re-typing of the same values across every document for a given client/transaction. | +| "Create from existing document" | If Teressa already has a well-prepared document for a past client, she can promote it to a template — snapshotting its field layout (minus client-specific filled values) into a new template record. | LOW | API: `POST /templates { sourceDocumentId }`. Copy `signatureFields`, convert `textFillData` values to hint labels (or blank them), let agent rename and save. Avoids the need to re-place fields from scratch when a good example already exists. | +| Template name includes form name | Template list shows both the agent-assigned name ("Listing Agreement — Standard") and the underlying form name ("Exclusive Right to Sell Listing Agreement - UAR"). Disambiguates when agent has multiple templates for the same base form. | LOW | UI display only — no schema change. Template record already links to `formTemplateId` which joins to `form_templates.name`. | + +--- + +### Anti-Features (Commonly Requested, Often Problematic) + +| Feature | Why Requested | Why Problematic | Alternative | +|---------|---------------|-----------------|-------------| +| Template versioning | "I want to know what changed between v1 and v2 of my listing agreement template" | Adds schema complexity (version rows, diffs, rollback UI) with near-zero value for a single-agent shop managing ~10 templates. Real change history is in git for the app, not in the product for the agent. | v1: overwrite saves. If agent needs rollback, they re-create from a document that used the old layout. Add versioning in v2 only if specifically requested. | +| Template sharing / team templates | "Multiple agents at the brokerage should share templates" | PROJECT.md explicitly scopes this as a solo-agent tool. Multi-agent support is Out of Scope at v1. Adding sharing requires user roles, permissions, org scoping — a major architectural expansion. | v1: all templates belong to the authenticated agent (single user). | +| Template-level AI pre-fill with actual values | "The template should store the filled values, not just hints" | Templates must be client-agnostic. Storing actual values (Teressa's own name pre-filled, etc.) creates confusion when applied to a new client who needs different values. | Store only hint labels in the template. Actual values always come from the client record at apply time. | +| Template locking (prevent field edits after apply) | "Once I apply a template, the fields should be immutable" | Prevents legitimate corrections before sending. A wrong field position or wrong signer assignment discovered in PreparePanel requires fix-it access. | Documents created from a template are full editable drafts until sent. No locks. | +| Template categories / folder organization | "I need folders for Buyer templates, Seller templates, Addenda" | Premature hierarchy. Teressa has ~10-20 templates at most. A flat list with good names is sufficient. Folders add UI complexity for no meaningful benefit at this scale. | Rely on clear naming conventions ("Buyer — Exclusive Buyer Broker Agreement", "Seller — Listing Agreement"). Add filtering/folders in v2 if template count grows beyond ~20. | +| Public/shareable template links | "Send a template link for clients to self-initiate" | OpenSign supports this but it requires anonymous sessions, role self-assignment by the signer, and a completely different signing flow. It is a distinct product surface. | Out of scope for v1. All documents are still agent-initiated. | + +--- + +## Feature Dependencies + +``` +[formTemplates table (existing — PDF library)] + └──referenced by──> [document_templates table (new)] + └──enables──> [Template List UI] + └──enables──> [Template Editor (save/update)] + └──enables──> [Apply Template to Document] + +[FieldPlacer + AI auto-place (existing v1.1)] + └──reused by──> [Template Editor] + └──enables──> [AI auto-place in template context] + +[documents table (existing)] + └──extended by──> [templateId FK (nullable, new)] + └──enables──> [Apply Template to Document] + └──enables──> [Create Template from Existing Document] + +[clients table (existing — propertyAddress, name, email)] + └──feeds──> [Hint auto-fill on template apply] + +[PreparePanel (existing)] + └──receives──> [Pre-populated fields from template apply] + └──receives──> [Hint-resolved values from client record] +``` + +### Dependency Notes + +- **Template editor reuses FieldPlacer entirely:** No new field-placement UI is needed. The template editor opens FieldPlacer in a "template mode" context where the target is a `document_templates` record instead of a `documents` record. The only new code is the route and the save action. + +- **Signer role mapping is required before fields can be copied:** On apply, each field's `signerEmail` role label ("signer:buyer") must be replaced with a real email address before the document is usable. For single-signer documents, this defaults to the client's email automatically. For multi-signer documents (v1.2 feature), agent maps roles to emails in a quick assignment step. + +- **textFillData hint keys must be defined before auto-fill works:** The known hint key set (`clientName`, `clientEmail`, `propertyAddress`, `todayDate`) must be documented and stable. The PreparePanel must recognize hint keys and resolve them from the client record. + +--- + +## MVP Definition + +### Launch With (v1.3) + +Minimum viable template system — must match what DocuSign and PandaDoc offer for the core save/reuse loop. + +- [ ] `document_templates` table — stores template name, `formTemplateId` FK, `signatureFields` JSONB (with signer role labels, not emails), `textFillData` JSONB (hint keys only), `createdAt`, `updatedAt` +- [ ] Template list page in portal (`/portal/templates`) — shows name, source form, field count, created date; rename and delete actions +- [ ] Template editor — FieldPlacer opened in template context; Save button writes to `document_templates`; AI auto-place works the same as in document prep +- [ ] Apply template when creating a document — "Start from template" option in the new-document flow; copies fields to new document record; replaces signer role labels with real emails; resolves known hint keys from client record +- [ ] Agent can update a template — re-open template in editor, save overwrites (no versioning) + +### Add After Validation (v1.x) + +- [ ] "Create template from existing document" — promote a prepared document to a template after the agent confirms it; useful when good examples already exist in the system +- [ ] Template preview — read-only PDF view of the source form with field overlays, so agent can verify field layout before applying to a new client + +### Future Consideration (v2+) + +- [ ] Template versioning — change log, rollback — defer until agent has >10 templates and reports that overwrite saves cause pain +- [ ] Template sharing / team access — requires multi-agent architecture; out of scope for solo agent +- [ ] Template categories / folders — defer until template count makes flat list unwieldy (unlikely before ~20 templates) +- [ ] Public template links (client self-initiate) — distinct product surface; requires anonymous session handling +- [ ] Template usage analytics — "this template has been used 12 times" — low value for single-agent + +--- + +## Feature Prioritization Matrix + +| Feature | User Value | Implementation Cost | Priority | +|---------|------------|---------------------|----------| +| Save template (editor + save action) | HIGH | MEDIUM | P1 | +| Template list (browse, rename, delete) | HIGH | LOW | P1 | +| Apply template to new document | HIGH | MEDIUM | P1 | +| Field copy on apply (not live reference) | HIGH | LOW | P1 | +| Signer role labels in template | HIGH | LOW | P1 | +| Hint key auto-fill from client record | HIGH | MEDIUM | P1 | +| AI auto-place in template editor | HIGH | LOW (reuse) | P1 | +| Template update / re-edit | MEDIUM | LOW | P1 | +| Create template from existing document | MEDIUM | LOW | P2 | +| Template preview (read-only) | MEDIUM | LOW | P2 | +| Template versioning | LOW | HIGH | P3 | +| Template sharing | LOW | HIGH | P3 | + +**Priority key:** +- P1: Must have for launch — defines whether v1.3 solves the re-use problem +- P2: Should have — reduces friction once core works +- P3: Future — defer until explicitly requested + +--- + +## Competitor Feature Analysis + +| Feature | DocuSign | PandaDoc | OpenSign (open source) | Our Approach | +|---------|----------|----------|------------------------|--------------| +| Template library | Dedicated "Templates" section, flat or folder organization | Dedicated "Templates" section, folder support | Dedicated "Templates" section, flat list | Flat list (`/portal/templates`), named by agent | +| Signer roles | Named roles ("Signer 1", "Agent") assigned at template creation; real emails assigned at send time | Variables + roles; variables can be pre-populated or left blank for sender | Named roles assigned at template creation; emails mapped on send | Role labels in `signerEmail` field ("signer:buyer"); mapped to real emails on apply | +| Field layout | Saved with template; editor is full drag-drop interface | Saved with template; drag-drop editor | Saved with template; same editor as document prep | Saved in `signatureFields` JSONB; template editor reuses existing FieldPlacer | +| AI field placement | Available in some tiers (2025) | Available via AI agents (separate product tier) | Not built-in; community feature requests | Existing AI auto-place pipeline reused directly in template editor | +| Apply to document | Copy fields to new envelope; sender fills recipient-specific data | "Use template" creates a new document; variables prompt for values | "Use template" creates a new document; signer emails assigned per send | Deep copy `signatureFields` + resolve hints from client record | +| Template editing | Full editor; overwrites existing template | Full editor; no versioning in standard tier | Edit available; overwrites | Overwrite save; no versioning at v1 | +| Versioning | Enterprise tier only | Not in standard tier | Not implemented in open source | v2+ if needed | +| Template sharing | Team/org templates in Business tier | Team templates with permission scoping | Shared templates supported | v2+ only; not in scope for solo agent | +| "Use existing doc as template" | Not standard; must recreate from scratch | "Save as template" from a document | Not documented in open source | P2 feature: "Create template from document" | + +--- + +## Sources + +- [DocuSign: Working with Templates](https://support.docusign.com/en/guides/ndse-user-guide-working-with-templates) — HIGH confidence +- [DocuSign: Learn Basics of How to Use a Template](https://www.docusign.com/en-gb/blog/learn-basics-of-e-signature-how-to-use-document-templates) — HIGH confidence +- [PandaDoc: Save Time with a Reusable Template](https://support.pandadoc.com/en/articles/9714616-save-time-with-a-reusable-template) — HIGH confidence +- [PandaDoc: Creating Dynamic Document Templates](https://www.pandadoc.com/blog/creating-dynamic-document-templates/) — MEDIUM confidence +- [OpenSign: Create Templates](https://docs.opensignlabs.com/docs/help/Templates/create-template/) — HIGH confidence +- [OpenSign: Use Template to Create Documents](https://docs.opensignlabs.com/docs/help/Templates/use-template/) — HIGH confidence +- [OpenSign: Manage Templates](https://docs.opensignlabs.com/docs/help/Templates/manage-templates/) — HIGH confidence +- [Xodo Sign: Document Templates for E-Signatures](https://eversign.com/features/templates) — MEDIUM confidence +- [BoldSign: Create eSignature Templates for Unlimited Reuse](https://boldsign.com/electronic-signature-features/templates/) — MEDIUM confidence +- [Adobe Sign: Apply a Form Field Template](https://helpx.adobe.com/sign/authoring/apply-field-template.html) — HIGH confidence +- Codebase audit: `src/lib/db/schema.ts` — confirmed `SignatureFieldData` interface, `documents.signatureFields` JSONB, `formTemplates` table, `textFillData` JSONB — HIGH confidence +- Codebase audit: `.planning/PROJECT.md` — confirmed solo-agent scope, v1.3 template goals, out-of-scope items — HIGH confidence + +--- +*Feature research for: Document template system — real estate signing portal* +*Researched: 2026-04-06* diff --git a/teressa-copeland-homes/.planning/research/PITFALLS.md b/teressa-copeland-homes/.planning/research/PITFALLS.md new file mode 100644 index 0000000..76b2517 --- /dev/null +++ b/teressa-copeland-homes/.planning/research/PITFALLS.md @@ -0,0 +1,256 @@ +# Pitfalls Research + +**Domain:** Document signing app — adding template management to existing system +**Researched:** 2026-04-06 +**Confidence:** HIGH (derived from direct codebase analysis + domain knowledge) + +--- + +## Critical Pitfalls + +### Pitfall 1: Field ID Collision Between Template Fields and Document Fields + +**What goes wrong:** +When a template is applied to create a document, the template's pre-authored `SignatureFieldData` is copied into `documents.signatureFields`. If template fields carry stable/predictable IDs (e.g., sequential strings, hash-of-position, or any ID format that could be reused), two documents created from the same template end up with the same field IDs. Everything works until multi-document views, audit correlation, or `textFillData` lookups appear — then the wrong fill values surface on the wrong document. + +**Why it happens:** +Developers author template fields by hand or via an AI-prepare step and assign IDs at template-save time rather than at document-creation time. They assume fields belong to a document and never reconsider that templates are shared parents. + +**How to avoid:** +Assign fresh `crypto.randomUUID()` IDs to every field at the moment a document is created from a template, not when the template is saved. The template stores field geometry/type/hint but carries NO IDs. The document creation route (`POST /api/documents`) performs the ID assignment during the copy step. + +In the current codebase `documents.signatureFields` is `SignatureFieldData[]` with a required `id: string` on each field (schema.ts line 17). The template table (`formTemplates`) has no `signatureFields` column today. When that column is added, store fields as a type without IDs, and stamp fresh UUIDs on copy. + +**Warning signs:** +- `textFillData` (keyed by field ID) returns the wrong prefill values for a document +- Agent edits a field on document A and document B's prepare screen shows the change +- Duplicate `id` values found in `signatureFields` arrays across two documents in the DB + +**Phase to address:** +The phase that adds `templateFields` to the `formTemplates` table and the document-creation route that copies them. + +--- + +### Pitfall 2: Template Deletion Breaks Existing Documents' Audit / Reference Integrity + +**What goes wrong:** +`documents.formTemplateId` is a nullable FK to `formTemplates.id` (schema.ts line 116). If a template is deleted, all documents that were created from it lose their lineage reference silently — the FK becomes a dangling null. In the current schema there is no `ON DELETE` constraint on `formTemplateId`, so Postgres will reject the delete with a FK violation, not silently nullify it. A naive delete route will throw a 500. + +More subtly: even with `ON DELETE SET NULL`, the signed PDF and audit trail remain intact (they reference `documents.id`, not `formTemplateId`), but the agent loses the ability to see which form was used, which matters for compliance and support. + +**Why it happens:** +Template management UIs typically have a "Delete" button that calls a simple `DELETE FROM form_templates WHERE id = ?`. No one thinks about the 47 existing documents that were created from that template. + +**How to avoid:** +- Never hard-delete templates. Soft-delete only: add `archivedAt timestamp` to `formTemplates`. Archived templates are hidden from the library but remain in the DB. +- If hard-delete is required, block it when `documents` rows reference the template: check `SELECT COUNT(*) FROM documents WHERE form_template_id = ?` before allowing deletion. +- Do NOT use `ON DELETE CASCADE` on `documents.formTemplateId` — that would delete all documents created from a template when the template is removed. + +**Warning signs:** +- 500 errors from a template delete route +- Agent reports that document history no longer shows which form was used +- `SELECT * FROM documents WHERE form_template_id IS NULL AND created_at > [template_launch_date]` returns unexpected rows + +**Phase to address:** +The phase that adds template CRUD (create, list, archive/delete). Must precede any phase that exposes a "Delete template" button. + +--- + +### Pitfall 3: Template Edit Retroactively Changes Fields on Existing Documents + +**What goes wrong:** +If `documents.signatureFields` is stored as a reference to (or derived live from) the template's fields rather than as an independent copy, editing the template — adding a new field, removing a date field, changing a field type — retroactively alters the field layout of every document already created from that template. A document that was prepared and sent with 12 fields suddenly has 14. Signers on active signing sessions see a different document than the one the agent reviewed. + +**Why it happens:** +Developers store only `formTemplateId` in the document and resolve fields from the template at runtime to avoid duplication. This seems efficient but breaks document immutability. + +**How to avoid:** +Fields must be snapshotted at document creation. `documents.signatureFields` holds the authoritative field list for that document for life. The `formTemplateId` is metadata (lineage), not a live pointer. Today this is already the correct pattern — `signatureFields` lives on the `documents` row — and adding templates must not change it. + +Do not add any code path that reads `formTemplates.signatureFields` to resolve a document's fields after document creation. + +**Warning signs:** +- A route that does `JOIN form_templates ON documents.form_template_id = form_templates.id` and reads template fields instead of document fields +- A prepare screen that fetches fields from the template endpoint rather than `GET /api/documents/[id]/fields` +- "My client's signing link broke after I updated the template" reports + +**Phase to address:** +Template authoring phase. The snapshot rule must be written into the document-creation route and into code review guidelines for all future template-related routes. + +--- + +### Pitfall 4: Agent-Typed Hint Values Stored in textFillData Instead of hint Field + +**What goes wrong:** +`SignatureFieldData.hint` is the label shown to a signer for `client-text` / `client-checkbox` fields (schema.ts line 24). `documents.textFillData` is a `Record` keyed by field ID that stores agent-typed fill values for `text` fields that are burned into the prepared PDF. If a developer uses `textFillData[fieldId] = "Enter your address here"` to store what is actually a signer hint, the prepared PDF will have the hint text printed as content rather than displayed as a placeholder. The signer sees a field already filled with the instruction text, not an empty field with a hint. + +**Why it happens:** +Both `hint` and `textFillData` are string values associated with a field ID. The naming is close enough that a developer authoring a "template defaults" feature stores instructional text in the wrong bucket. + +**How to avoid:** +- `textFillData` = agent fills agent-typed values at prepare time; these are burned into the PDF as visible text. +- `hint` on `SignatureFieldData` = a label shown in the signing UI as a placeholder; never written to the PDF. +- Template defaults (pre-seeded agent fill values) go into `textFillData`. Signer instructions go into `SignatureFieldData.hint`. +- When a template stores default field configurations, store `hint` on the field struct, not in a separate `textFillData`-like map on the template. + +**Warning signs:** +- The prepared PDF contains placeholder text like "Enter address here" as printed content +- The signer sees a field that appears pre-filled with an instruction rather than empty +- The AI classify step (`classifyFieldsWithAI`) returns `prefillValue` for a field and a developer copies it directly into `hint` instead of `textFillData` + +**Phase to address:** +Template field authoring phase AND any phase that adds signer-instruction hints to fields. + +--- + +### Pitfall 5: Signer Role Slots Mixed Up With Actual Signer Emails + +**What goes wrong:** +Templates define roles ("Seller 1", "Seller 2", "Agent"). Documents bind actual emails to those roles at prepare time. If role assignment is skipped or the binding is applied to the wrong field, `field.signerEmail` ends up with a role label string ("Seller 1") instead of an email address. The signing flow uses `signerEmail` to route tokens and color-code fields. A role label string passes string validation silently but breaks every `signerEmail` comparison downstream: token routing fails, the wrong signer is asked to sign agent fields, and `getSignerEmail()` (schema.ts line 52) returns the role string as the fallback email. + +**Why it happens:** +Template fields need to reference a role without knowing the actual email. Developers use the role label as a placeholder value and forget to replace it during the document-creation step. Or they add a `roleName` property to `SignatureFieldData` but never strip it before saving to the document. + +**How to avoid:** +- Templates store `signerRole: string` (not `signerEmail`) on field definitions. This is a separate property, never the same field as `signerEmail`. +- At document creation, a role-to-email map is required and validated. If any role has no email, block document creation — do not fall through to writing a role label into `signerEmail`. +- `SignatureFieldData` in the document must have `signerEmail` as a real email or undefined (legacy fallback). Validate with a regex or `z.string().email()` before DB write. +- The `getSignerEmail(field, fallbackEmail)` helper (schema.ts line 52) is the right pattern for legacy fallback, but is not a safe guard against role-label corruption. + +**Warning signs:** +- `field.signerEmail` values contain spaces or capital letters (e.g., "Seller 1") +- Signing tokens are created with a non-email `signerEmail` value +- The multi-signer signing page routes a client to sign agent-owned fields + +**Phase to address:** +Template field definition phase AND the document-creation route that binds roles to emails. Role validation must be an explicit step, not an afterthought. + +--- + +### Pitfall 6: Template PDF File Gone But DB Row Persists (or Vice Versa) + +**What goes wrong:** +`formTemplates.filename` maps to a file in `seeds/forms/`. If the file is renamed, moved, or deleted on disk while the DB row exists (or the row is inserted with the wrong filename), `copyFile(srcPath, destPath)` in `POST /api/documents` (documents/route.ts line 64) throws `ENOENT`. The agent gets a 500 with no explanation. Because `seeds/forms/` is under version control (per gitStatus), this happens most often during a migration that renames form files. + +**Why it happens:** +The DB row and the file are not managed as a unit. A migration script inserts rows, the filename in the row has a typo, or someone renames a PDF without updating the DB. + +**How to avoid:** +- On startup (or in a health-check route), validate that every `formTemplates` row's `filename` resolves to an existing file. Log mismatches loudly. +- The `POST /api/documents` route should return a 422 with a meaningful message when `ENOENT` is caught, not fall through to a 500. +- When seeding templates (or adding new forms), use a migration that inserts the DB row and checks file existence atomically in a script, never separate steps. + +**Warning signs:** +- Unhandled `ENOENT` errors in the documents creation API +- Agent reports "template not found" but the template appears in the library list +- `formTemplates` rows exist with filenames that differ from actual files in `seeds/forms/` + +**Phase to address:** +Template seeding / library management phase. + +--- + +## Technical Debt Patterns + +| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable | +|----------|-------------------|----------------|-----------------| +| Store field IDs in template at save time | Skip ID-generation step at doc creation | ID collisions across documents; corrupted textFillData | Never | +| Use `ON DELETE CASCADE` on `formTemplateId` | No orphan rows | Deleting a template deletes all documents created from it | Never | +| Live-resolve fields from template at runtime | Single source of truth for field layout | Template edits retroactively break sent/signed documents | Never | +| Store role labels ("Seller 1") in `signerEmail` | Simpler template field schema | Silent routing failures in signing flow | Never | +| Skip snapshot of textFillData defaults when creating doc from template | Less code at creation time | Agent's fill values come from template, diverge when template is edited | Never acceptable after Sent status | +| Hard-delete templates | Smaller DB | Agent can't trace which form a historical document used | MVP only if template count is tiny and no compliance need | + +--- + +## Integration Gotchas + +| Integration | Common Mistake | Correct Approach | +|-------------|----------------|------------------| +| pdfjs-dist + template field storage | Extract blanks from template PDF at save time and store AI-classified fields permanently on the template | Store only the template PDF filename; always re-run `extractBlanks` + `classifyFieldsWithAI` at document creation, or store field geometry without IDs for later stamping | +| OpenAI AI-prepare + templates | Run AI-prepare once, save result to template, assume it applies to all documents | AI-prepare personalizes prefill values (client name, property address) — these must be per-document. Template may store field geometry hints but prefill values must be computed at document creation time | +| Drizzle FK + soft-delete | Add `archivedAt` column but forget to filter it in all `formTemplates` queries | Add a `where(isNull(formTemplates.archivedAt))` default filter or use a query helper that always excludes archived rows | +| `documents.signers` + template roles | Populate `signers` from template role definitions without binding real emails | `signers` must contain real `{ email, color }` pairs; roles are template-side only and must be resolved to emails before the `signers` array is written to the DB | + +--- + +## Performance Traps + +| Trap | Symptoms | Prevention | When It Breaks | +|------|----------|------------|----------------| +| AI-prepare on every document created from template | Slow doc creation (1-3s OpenAI call); high API cost | Cache AI-classified field geometry on the template (without IDs or client-specific prefill); run AI-prepare only for field type/geometry, run prefill substitution locally | At ~50+ documents/day created from same template | +| Loading all `signatureFields` JSONB for every document list query | Portal page slow as document count grows | Never SELECT signatureFields in list queries; only fetch when opening a specific document | At ~500+ documents | +| Copying template PDF for every new document (current pattern) | Disk fills with identical PDFs | Store the template PDF once; only copy when the document will be mutated (prepare step). Pre-prepare creates the mutated `_prepared.pdf` from the canonical seed | At ~1000+ documents from same template | + +--- + +## Security Mistakes + +| Mistake | Risk | Prevention | +|---------|------|------------| +| Allowing agents to upload arbitrary PDFs as "templates" | Malicious PDF embedded in template library, served to all clients | Keep template PDFs in `seeds/forms/` under version control, not in `uploads/`. Agent-uploaded custom PDFs go to `uploads/clients/` only and are not promoted to templates | +| Template field `hint` containing HTML/script content | XSS in the signing UI if hint is rendered as HTML | Render hint as plain text only (`textContent`, never `dangerouslySetInnerHTML`) | +| Missing auth on template CRUD routes | Any authenticated user (in a multi-agent system) can modify shared templates | Add ownership or role check on template write routes; read-only access is fine for all authenticated agents | +| Path traversal via template `filename` field | Agent-crafted filename reads arbitrary files from server | Validate that `path.join(SEEDS_DIR, template.filename)` starts with `SEEDS_DIR` — same guard pattern already used in `documents/route.ts` line 47 | + +--- + +## UX Pitfalls + +| Pitfall | User Impact | Better Approach | +|---------|-------------|-----------------| +| Template library shows archived/deleted templates | Agent adds form from a stale template; document creation fails | Filter `archivedAt IS NULL` in library API; show "archived" badge on template management screen, not in the add-document flow | +| No preview of template fields before creating a document | Agent creates document, runs AI-prepare, finds wrong fields, deletes and recreates | Show a field-count and type summary on the template card in the library; optional: thumbnail of first page | +| Prepare screen pre-populates client data from template defaults instead of the assigned client | Wrong name/address in text fields for documents with multiple clients sharing a template | Prefill values are always derived from the assigned `clientId` at prepare time, never from the template | +| Role slot UI during prepare requires understanding "Seller 1 = email" mapping | Agent assigns wrong signer to wrong role; wrong party signs | Show the client's name (from `clients` table) next to each role slot during preparation, not just the email address | + +--- + +## "Looks Done But Isn't" Checklist + +- [ ] **Template field IDs:** Verify field IDs in `documents.signatureFields` are freshly generated at doc creation — not copied from template. Query two documents from same template and confirm no shared field IDs. +- [ ] **Template deletion guard:** Verify DELETE endpoint returns 4xx (not 5xx and not 2xx) when documents reference the template. Test by creating a document from a template, then attempting to delete the template. +- [ ] **Snapshot independence:** Edit a template's fields after creating a document from it. Verify the existing document's `signatureFields` is unchanged. +- [ ] **textFillData vs hint:** Open a prepared PDF for a `client-text` field that has a hint. Confirm the PDF does not contain the hint string as printed text. +- [ ] **Role-to-email binding:** Confirm that after document creation no `signerEmail` value on any field contains a space or non-email string. +- [ ] **ENOENT handling:** Rename a template PDF, attempt to create a document from it, verify a 4xx with a clear message (not a 500). +- [ ] **Archived templates:** Archive a template, confirm it disappears from the library API response but its historical documents remain accessible. + +--- + +## Recovery Strategies + +| Pitfall | Recovery Cost | Recovery Steps | +|---------|---------------|----------------| +| Field ID collision discovered post-launch | HIGH | Write a migration that assigns fresh UUIDs to all `signatureFields` on every document; cross-reference with `textFillData` keys and rekey them atomically | +| Template deletion cascaded to documents (if wrongly configured) | CRITICAL | Restore from backup; no in-place recovery possible. Prevention is the only option. | +| Hint text burned into prepared PDFs as content | MEDIUM | Re-run prepare step (only possible if document is still Draft); for Sent/Signed documents, manual intervention required | +| Role label stored in signerEmail | MEDIUM | Write a one-time migration: query documents where any `signatureFields[].signerEmail` does not match email pattern; surface them to the agent for manual correction | +| Template PDF and DB row out of sync | LOW | Add a validation script that walks `formTemplates` rows and checks file existence; run during deploys | + +--- + +## Pitfall-to-Phase Mapping + +| Pitfall | Prevention Phase | Verification | +|---------|------------------|--------------| +| Field ID collision | Template fields schema design phase (adding `signatureFields` to `formTemplates`) | Assert no duplicate `id` values across two documents created from same template | +| Template deletion breaking documents | Template CRUD phase | Attempt delete of template with associated documents; expect 409 or soft-delete only | +| Template edit retroactively changing documents | Template authoring phase + document creation route | Edit template fields; confirm existing document `signatureFields` unchanged in DB | +| Hint vs textFillData confusion | Template field authoring phase | Inspect prepared PDF bytes; confirm hint text absent from PDF content stream | +| Signer role vs email confusion | Template field definition + document creation binding phase | Inspect `signatureFields` after doc creation; confirm all `signerEmail` values pass email regex | +| Template file/DB desync | Template seeding / library management phase | Remove a seed file; confirm creation attempt returns 422 not 500 | + +--- + +## Sources + +- Direct codebase analysis: `src/lib/db/schema.ts`, `src/app/api/documents/route.ts`, `src/app/api/documents/[id]/prepare/route.ts`, `src/app/api/documents/[id]/ai-prepare/route.ts`, `src/lib/ai/field-placement.ts`, `src/lib/ai/extract-text.ts` +- Schema observation: `documents.formTemplateId` has no `ON DELETE` action specified (schema.ts line 116) — Postgres defaults to `RESTRICT` +- `SignatureFieldData` interface (schema.ts lines 15-25) — `hint`, `signerEmail`, `type` fields and their optionality +- `textFillData: Record` (schema.ts line 119) — keyed by field ID +- Milestone context: template field IDs, deletion safety, hint vs fill, role vs email + +--- +*Pitfalls research for: document signing app — template management addition* +*Researched: 2026-04-06* diff --git a/teressa-copeland-homes/.planning/research/STACK.md b/teressa-copeland-homes/.planning/research/STACK.md new file mode 100644 index 0000000..03c35e6 --- /dev/null +++ b/teressa-copeland-homes/.planning/research/STACK.md @@ -0,0 +1,216 @@ +# Technology Stack — Document Templates Feature + +**Project:** teressa-copeland-homes +**Feature:** Document Templates (reusable PDF + pre-placed fields + signer role assignments) +**Researched:** 2026-04-06 +**Overall confidence:** HIGH — analysis is based on direct inspection of the live codebase + +--- + +## Verdict: Zero New Packages Required + +The existing stack handles document templates entirely. This feature is a schema extension +plus UI reuse — no new runtime dependencies are needed. + +--- + +## Existing Stack (Validated — Do Not Re-research) + +| Layer | Technology | Version | +|-------|-----------|---------| +| Framework | Next.js App Router | 16.2.0 | +| Language | TypeScript | ^5 | +| ORM | Drizzle ORM | ^0.45.1 | +| Database | PostgreSQL (local dev / Neon prod) | — | +| PDF manipulation | @cantoo/pdf-lib | ^2.6.3 | +| PDF rendering | react-pdf | ^10.4.1 | +| Drag-drop fields | @dnd-kit/core + @dnd-kit/utilities | ^6.3.1 / ^3.2.2 | +| AI field placement | openai (GPT-4.1) | ^6.32.0 | +| Validation | zod | ^4.3.6 | +| Auth | next-auth | 5.0.0-beta.30 | + +--- + +## Schema Change: New `document_templates` Table + +The current `form_templates` table stores only raw PDFs (id, name, filename). It has no +field data. A new table is needed — it must NOT be confused with `form_templates`, which +is a forms library (source PDFs only). + +### New table: `document_templates` + +```typescript +// src/lib/db/schema.ts addition + +export interface TemplateSignerRole { + role: string; // e.g. "Seller", "Buyer", "Agent" + color: string; // hex — matches signer color convention from DocumentSigner +} + +export const documentTemplates = pgTable("document_templates", { + id: text("id").primaryKey().$defaultFn(() => crypto.randomUUID()), + name: text("name").notNull(), + // FK to form_templates — the source PDF this template was built from + formTemplateId: text("form_template_id") + .notNull() + .references(() => formTemplates.id, { onDelete: "restrict" }), + // Pre-placed fields — same SignatureFieldData[] shape used everywhere else + // signerEmail on each field will hold a ROLE string here (e.g. "Seller"), + // not a real email. Resolved to actual emails at apply-time. + signatureFields: jsonb("signature_fields").$type(), + // Ordered list of signer roles with their assigned colors + signerRoles: jsonb("signer_roles").$type(), + // Pre-filled static text hints for text/client-text fields + textHints: jsonb("text_hints").$type>(), + createdAt: timestamp("created_at").defaultNow().notNull(), + updatedAt: timestamp("updated_at").defaultNow().notNull(), +}); +``` + +### Why a new table, not extending `form_templates` + +`form_templates` is a seeded forms library (120+ Utah/CCIM PDF forms). It is read-only +catalog data. Adding mutable agent-authored field data to it would mix concerns: catalog +vs. prepared template. A clean FK relationship (`document_templates.formTemplateId → +form_templates.id`) keeps them separated and allows one form to have multiple named +templates (e.g. "Standard Listing Agreement", "Listing Agreement + Short Sale Addendum"). + +### Why not extend `documents` + +`documents` is a per-client instance with a concrete `clientId`, real signer emails, and +a lifecycle (Draft → Sent → Signed). Templates are agent-owned, have no client, and have +no signing lifecycle. Mixing them would require nullable foreign keys and conditional +logic throughout the signing flow. + +--- + +## Apply-Template Operation: Pure Data Transform, No New Packages + +When an agent applies a template to a new document, the operation is: + +1. Load `document_templates` row (fields, signerRoles, textHints) +2. Create a new `documents` row with `formTemplateId` copied from the template's source +3. Deep-clone `signatureFields` array; substitute role strings → real signer emails + supplied by the agent at apply-time (e.g. "Seller" → "alice@example.com") +4. Copy `textHints` into `documents.textFillData` as the initial fill data +5. `documents.signers` is populated from the resolved email+color pairs + +This is entirely Drizzle + TypeScript. No transformation library needed. + +--- + +## JSONB Performance: No Concern at This Scale + +**Field for concern:** `signatureFields` JSONB arrays in both `document_templates` and +`documents`. Typical Utah real estate form: 10–60 fields. Maximum realistic: ~150 fields +on a complex addendum stack. + +**PostgreSQL JSONB behavior:** JSONB is stored binary, fully indexed when needed. At +< 200 fields per document and < 1000 total templates, there is no performance risk with +unindexed JSONB. The existing `documents.signatureFields` already uses this pattern at +production with no issues. + +**If GIN indexing were ever needed** (e.g. querying "all templates that have a field of +type X"), `drizzle-kit` supports it via `index()`. Not needed for the MVP. + +--- + +## Migration Path + +This is an additive migration — no existing tables are altered. + +```sql +-- Generated by: npx drizzle-kit generate +CREATE TABLE "document_templates" ( + "id" text PRIMARY KEY, + "name" text NOT NULL, + "form_template_id" text NOT NULL REFERENCES "form_templates"("id") ON DELETE RESTRICT, + "signature_fields" jsonb, + "signer_roles" jsonb, + "text_hints" jsonb, + "created_at" timestamp DEFAULT now() NOT NULL, + "updated_at" timestamp DEFAULT now() NOT NULL +); +``` + +Follow existing migration pattern: add to schema.ts, run `npm run db:generate`, then +`npm run db:migrate`. + +--- + +## UI Reuse + +All FieldPlacer / DocumentPageClient / PreparePanel / PdfViewerWrapper components can be +reused verbatim for the template editor. The template editor is functionally identical to +the document prepare flow — same drag-drop, same AI auto-place endpoint — except: + +- No `clientId` context (no real signer emails yet; use role names as placeholder emails) +- No "Send" action (templates are saved, not sent) +- Signer panel shows roles ("Seller", "Buyer") not real email addresses + +This means the template editor is a new route +(`/portal/templates/[templateId]/edit`) that instantiates the existing components with +role-mode props, not a new component tree. + +--- + +## Integration Points + +| Existing File | Change Needed | +|---------------|--------------| +| `src/lib/db/schema.ts` | Add `documentTemplates` table + `TemplateSignerRole` interface | +| `src/app/api/documents/route.ts` | Accept optional `documentTemplateId` on POST; apply-template transform | +| `src/app/portal/_components/AddDocumentModal.tsx` | Add "From template" tab alongside "From forms library" | +| `drizzle/` | New migration file (auto-generated by drizzle-kit) | + +No changes needed to: signing flow, PDF preparation, email sending, audit events. + +--- + +## Alternatives Considered + +| Option | Why Rejected | +|--------|-------------| +| Extend `form_templates` with field columns | Mixes catalog data with agent-authored data; blocks multi-template-per-form | +| Store template fields inside `documents` with a `isTemplate` flag | Breaks document lifecycle invariants; complicates all document queries | +| New npm package for template merge (e.g. json-merge-patch) | Not needed — deep clone + email substitution is trivial TypeScript | +| Separate template file storage (copy PDF per template) | Wasteful; templates share the original `form_templates` PDF via FK | + +--- + +## Confidence Assessment + +| Area | Confidence | Basis | +|------|------------|-------| +| Zero new packages | HIGH | Direct schema/code inspection; transform is pure TypeScript | +| New table over extending existing | HIGH | `form_templates` is seeded catalog; concerns are clearly different | +| JSONB performance | HIGH | Existing pattern already in production; field counts well within limits | +| UI reuse via props | MEDIUM | Components not yet inspected for hardcoded assumptions about signer emails vs roles — confirm before implementation | +| Migration is additive | HIGH | No existing column changes; FK is to read-only catalog table | + +--- + +## Open Questions for Implementation Phase + +1. **Role-string convention in `signerEmail` field:** The `SignatureFieldData.signerEmail` + field currently holds real email addresses. Using it for role strings ("Seller") during + template editing requires the FieldPlacer and PreparePanel components to handle both + modes. Confirm these components have no email-format validation before proceeding. + +2. **Template apply: partial field substitution.** If a template has 3 signer roles but + the agent assigns only 2 real signers at apply-time, the behavior needs to be defined + (error, or leave unassigned fields with no signer). + +3. **`updatedAt` auto-update:** The current schema uses `.defaultNow()` for `updatedAt` + but does not use a DB trigger for auto-update on mutation. Existing tables share this + pattern — use the same convention (set `updatedAt` explicitly in update queries). + +--- + +## Sources + +- Direct inspection: `/Users/ccopeland/temp/red/teressa-copeland-homes/src/lib/db/schema.ts` +- Direct inspection: `/Users/ccopeland/temp/red/teressa-copeland-homes/package.json` +- Direct inspection: `/Users/ccopeland/temp/red/teressa-copeland-homes/src/app/portal/_components/AddDocumentModal.tsx` +- Direct inspection: `/Users/ccopeland/temp/red/teressa-copeland-homes/src/app/api/documents/[id]/fields/route.ts` +- Direct inspection: `/Users/ccopeland/temp/red/teressa-copeland-homes/drizzle/` (migrations 0000–0011) diff --git a/teressa-copeland-homes/.planning/research/SUMMARY.md b/teressa-copeland-homes/.planning/research/SUMMARY.md new file mode 100644 index 0000000..924b84a --- /dev/null +++ b/teressa-copeland-homes/.planning/research/SUMMARY.md @@ -0,0 +1,177 @@ +# Project Research Summary + +**Project:** teressa-copeland-homes +**Domain:** Real estate agent portal — PDF e-signing with document template management +**Researched:** 2026-04-06 +**Confidence:** HIGH — all research based on direct codebase inspection + verified competitor analysis + +## Executive Summary + +This milestone (v1.3) adds a document template system to an existing, working PDF e-signing portal. The core product is already built: forms library, drag-drop field placement, AI auto-place, agent signature, multi-signer document sending, and the full signing flow are all live. The template feature is a focused extension — a new `document_templates` database table, a template editor that reuses the existing `FieldPlacer` component with a minor callback abstraction, and an "apply template" path in the new-document flow. No new npm packages are required. The entire feature is a schema extension plus UI reuse. + +The recommended approach follows the industry-standard template model used by DocuSign, PandaDoc, and OpenSign: templates store a field layout with signer role labels (not actual emails), and roles are resolved to real signer emails at apply time. Template fields must be snapshotted — deep-copied with fresh UUIDs — into each new document at creation. Templates are never live references. This is the correct pattern for document immutability and is already the implicit design of the existing `documents.signatureFields` JSONB column. The implementation must not break this invariant. + +The primary risks are all data-integrity issues that are easy to introduce and hard to recover from: field ID collisions if template fields carry stable IDs into documents, signer role labels leaking into the signing flow as fake email addresses, and hint text being burned into prepared PDFs as printed content. These risks are fully preventable if the document-creation route enforces ID re-stamping, role-to-email validation, and clear separation of `hint` vs `textFillData`. Build order matters: schema and CRUD API before editor UI, editor UI before apply flow, and the `FieldPlacer` persistence abstraction before the template editor. + +--- + +## Key Findings + +### Recommended Stack + +The existing stack handles this feature entirely. See `.planning/research/STACK.md` for full detail. + +**Core technologies:** +- **Drizzle ORM + PostgreSQL:** New `document_templates` table via additive migration — zero changes to existing tables +- **JSONB columns:** `signatureFields` + `signerSlots` on `document_templates`, matching the existing pattern on `documents` +- **@cantoo/pdf-lib + react-pdf:** No changes needed — template editor renders the same PDF, no new PDF manipulation +- **FieldPlacer (existing component):** Reused via a single `onPersist` callback prop addition — not duplicated +- **openai / GPT-4.1:** Existing AI auto-place pipeline reused via a parallel `/api/templates/[id]/ai-prepare` route + +One schema distinction is critical: `form_templates` is a read-only seeded catalog of 120+ Utah/CCIM PDF forms. `document_templates` is agent-authored field layouts. These are different concerns and must remain separate tables with a FK from new to old. + +### Expected Features + +See `.planning/research/FEATURES.md` for full detail and competitor analysis. + +**Must have (table stakes — P1):** +- Save a prepared document as a template (name, source form FK, field layout with role labels, text hints) +- Template list in portal (`/portal/templates`) with rename and delete actions +- Apply template when adding a document to a client (one-click pre-loaded document creation) +- Field layout is copied (snapshot), never referenced live +- Signer roles ("Seller", "Buyer", "Agent") stored in template; real emails assigned at apply time +- Text fill hints stored in template; actual values resolved from client record at apply time +- Agent can re-edit template (overwrite save, no versioning at v1) +- AI auto-place works in template editor (reuses existing pipeline) + +**Should have (competitive — P2, after core validates):** +- "Create template from existing document" — promote a prepared document to a template +- Template preview — read-only PDF view with field overlays before applying + +**Defer (v2+):** +- Template versioning — not needed until agent has >10 templates and reports pain +- Template sharing / team access — requires multi-agent architecture; out of scope +- Template categories / folders — flat list with good names is sufficient at <20 templates +- Public/shareable template links — distinct product surface requiring anonymous sessions + +### Architecture Approach + +The architecture is a clean extension of the existing document flow. The `document_templates` table holds field layouts and signer role slots. A `POST /api/templates/[id]/apply` route performs the document creation: copy PDF, deep-clone fields with fresh UUIDs, resolve role labels to emails from the agent-supplied map, insert document row. The template editor reuses `FieldPlacer` via an `onPersist` callback prop (a 5-line addition to the 822-line component). See `.planning/research/ARCHITECTURE.md` for full data flow and component boundaries. + +**Major components:** +1. **`document_templates` table** — new schema; stores name, formTemplateId FK, signatureFields JSONB (role labels, not emails), signerSlots JSONB, timestamps +2. **Template CRUD API** (`/api/templates`, `/api/templates/[id]`) — list, create, fetch, save fields, soft-delete +3. **Template editor UI** (`TemplateEditorClient` + `TemplatePanel`) — thin wrapper over existing `FieldPlacer` with template-mode save; shares layout with `DocumentPageClient` +4. **Apply operation** (`POST /api/templates/[id]/apply`) — the core integration: PDF copy + field snapshot + role-to-email resolution + document INSERT +5. **`AddDocumentModal` update** — adds "Start from template" tab; shows `TemplatePicker` and role-to-email assignment UI +6. **`/api/templates/[id]/ai-prepare`** — mirrors existing `ai-prepare` route; writes to `document_templates.signatureFields` + +### Critical Pitfalls + +See `.planning/research/PITFALLS.md` for full detail, recovery strategies, and verification checklist. + +1. **Field ID collision** — Template fields must NOT carry IDs into documents. Stamp fresh `crypto.randomUUID()` IDs on every field at document creation time. Never copy field IDs from the template row. Recovery is expensive (DB migration + textFillData rekeying). + +2. **Template edit retroactively changing existing documents** — Never resolve a document's fields from the template at runtime. `documents.signatureFields` is the authoritative field list for the life of that document. The `formTemplateId` FK is lineage metadata only. Violating this breaks active signing sessions. + +3. **Signer role labels leaking into `signerEmail`** — Templates use role strings ("Seller") where emails go. The document-creation route must enforce a complete role-to-email map before writing to DB. Validate all `signerEmail` values pass email format before INSERT. Role labels in `signerEmail` silently break signing token routing downstream. + +4. **`hint` vs `textFillData` confusion** — `hint` is a placeholder shown in the signing UI; it must never be written to the PDF. `textFillData` holds agent-typed values that are burned into the prepared PDF. Storing hints in `textFillData` prints instruction text as PDF content. Distinction must be enforced in the template authoring phase. + +5. **Template deletion breaking document history** — No hard deletes. Use soft-delete (`archivedAt` column). If hard delete is required, block it when `documents` rows reference the template. Never use `ON DELETE CASCADE` on `documents.formTemplateId` — that would destroy client documents. + +--- + +## Implications for Roadmap + +Based on the dependency graph in ARCHITECTURE.md and the pitfall-to-phase mapping in PITFALLS.md, four phases are recommended. Each phase is independently deployable and leaves the existing system working. + +### Phase 1: Schema + Template CRUD API +**Rationale:** Everything else depends on the table existing and the API working. Schema is additive — no existing tables change. Safe to deploy to production immediately. +**Delivers:** `document_templates` table with migration; `GET/POST /api/templates`; `GET/PATCH/DELETE /api/templates/[id]`; `TemplateSignerSlot` interface in schema.ts +**Addresses:** Table stakes "save a prepared template", "template list with rename/delete" +**Avoids:** Template deletion cascade (establish soft-delete pattern here); file/DB desync (validate filename at create time) +**Research flag:** Standard patterns (follows existing `forms-library/route.ts` and `documents/[id]/route.ts` exactly — skip phase research) + +### Phase 2: FieldPlacer Abstraction + Template Editor UI +**Rationale:** FieldPlacer must have the `onPersist` callback before the template editor can reuse it. This is a non-breaking change — all existing document consumers pass no prop, behavior unchanged. Template editor then becomes a thin configuration layer. +**Delivers:** `onPersist` prop on `FieldPlacer`; `TemplateEditorClient` + `TemplatePanel`; `/portal/(protected)/templates/[templateId]/page.tsx`; portal nav link +**Addresses:** "Agent can re-edit template", signer role slot management in editor +**Avoids:** Building a separate FieldPlacer implementation (diverging drag-drop logic); branching inside the 822-line component +**Research flag:** Standard patterns (direct component extension, well-understood — skip phase research) + +### Phase 3: Apply Template Flow +**Rationale:** The apply operation is the value-delivery moment — it depends on Phases 1+2. This phase requires the most careful implementation due to three critical pitfalls (field ID collision, role-to-email validation, snapshot independence). +**Delivers:** `POST /api/templates/[id]/apply`; updated `AddDocumentModal` with "Start from template" tab; `TemplatePicker` component; role-to-email assignment UI; known hint keys auto-resolved from client record +**Addresses:** Core table stakes "apply template when adding a document"; signer role resolution; text fill hint auto-population +**Avoids:** Field ID collision (stamp UUIDs at apply time, never copy from template); role labels in signerEmail (validate before INSERT); live field resolution (snapshot is the rule) +**Research flag:** Needs careful implementation review — the three never-acceptable shortcuts in PITFALLS.md all live here. No external research needed, but this phase warrants the most internal code review. + +### Phase 4: Template List UI + Portal Integration +**Rationale:** The list page can be built after the API exists (Phase 1), but is sequenced last because it depends on the full round-trip working (create template in editor, list it, apply it). Completing this phase closes the v1.3 loop. +**Delivers:** `TemplatesPageClient`; `/portal/(protected)/templates/page.tsx`; template cards with field count, source form name, rename/delete actions; archived template filtering +**Addresses:** "Template list in portal" table stakes; "template name includes form name" differentiator +**Avoids:** Archived templates surfacing in library (filter `archivedAt IS NULL` in list API); no preview confusion (show field count summary on card) +**Research flag:** Standard patterns (matches existing `ClientCard` + `ConfirmDialog` + `ClientsPageClient` pattern — skip phase research) + +### Phase Ordering Rationale + +- Schema must precede all API work (no table = no routes) +- FieldPlacer abstraction must precede template editor (can't reuse a component that isn't abstracted) +- Apply flow depends on both the API (template data) and the editor (template creation) being in place +- List UI is independently testable but benefits from the full create-apply round-trip being available for manual verification +- All phases avoid touching the signing flow, PDF preparation, email sending, and audit events — this isolation is deliberate and must be maintained + +### Research Flags + +Phases with standard patterns (skip research-phase): +- **Phase 1:** Schema migration follows established drizzle-kit pattern (migrations 0000-0011 already in place); CRUD API follows `forms-library/route.ts` and `documents/[id]/route.ts` exactly +- **Phase 2:** FieldPlacer abstraction is a well-understood prop injection; template editor composes existing components +- **Phase 4:** List UI follows existing `ClientsPageClient` + `ClientCard` + `ConfirmDialog` pattern verbatim + +Phases needing implementation-time attention (not external research, but internal discipline): +- **Phase 3:** Three critical pitfalls converge here. Recommend a peer review checklist against PITFALLS.md "Looks Done But Isn't" section before marking phase complete. + +--- + +## Confidence Assessment + +| Area | Confidence | Notes | +|------|------------|-------| +| Stack | HIGH | Direct codebase inspection; zero new packages; existing JSONB pattern already in production | +| Features | HIGH | DocuSign, PandaDoc, OpenSign all confirmed; competitor analysis corroborates feature set; solo-agent scope confirmed from PROJECT.md | +| Architecture | HIGH | Based on direct inspection of 10 source files and 12 drizzle migrations; no external assumptions | +| Pitfalls | HIGH | Derived from schema.ts type analysis and existing route code; all pitfalls trace to specific lines in the codebase | + +**Overall confidence:** HIGH + +### Gaps to Address + +- **FieldPlacer `signerEmail` format validation:** STACK.md flags that FieldPlacer and PreparePanel components may have email-format validation that would reject role strings. Confirm no `z.string().email()` or similar check runs on `signerEmail` during template editing before implementing Phase 2. If validation exists, introduce a `mode: "template" | "document"` prop alongside `onPersist`. + +- **Role-to-email mapping UX for single-signer documents:** FEATURES.md notes that for solo-agent, single-signer use case the role-to-email mapping step can default to the client's email automatically. The exact trigger condition (when to auto-default vs when to prompt) needs to be defined in Phase 3 planning. Suggested rule: if the template has exactly one non-agent signer role, pre-populate it with the assigned client's email. + +- **`updatedAt` auto-update convention:** Existing tables set `updatedAt` explicitly in update queries (no DB trigger). `document_templates` must follow the same convention. Flag this in Phase 1 implementation notes. + +--- + +## Sources + +### Primary (HIGH confidence) +- Direct inspection: `src/lib/db/schema.ts` — full table schema, `SignatureFieldData` interface, `textFillData` type +- Direct inspection: `src/app/api/documents/route.ts` — document creation flow (library path + upload path) +- Direct inspection: `src/app/api/documents/[id]/fields/route.ts` — GET/PUT pattern for signatureFields +- Direct inspection: `src/app/portal/_components/AddDocumentModal.tsx` — modal flow for new document creation +- Direct inspection: `drizzle/` migrations 0000-0011 — confirmed additive migration pattern +- Direct inspection: `package.json` — confirmed zero new packages needed +- [DocuSign: Working with Templates](https://support.docusign.com/en/guides/ndse-user-guide-working-with-templates) +- [OpenSign: Create Templates](https://docs.opensignlabs.com/docs/help/Templates/create-template/) +- [OpenSign: Use Template to Create Documents](https://docs.opensignlabs.com/docs/help/Templates/use-template/) + +### Secondary (MEDIUM confidence) +- [PandaDoc: Creating Dynamic Document Templates](https://www.pandadoc.com/blog/creating-dynamic-document-templates/) — template variable and role model +- [BoldSign: Create eSignature Templates for Unlimited Reuse](https://boldsign.com/electronic-signature-features/templates/) — competitor feature comparison + +--- +*Research completed: 2026-04-06* +*Ready for roadmap: yes*