← Back to Assay

Assay Architecture

Event-Driven Serverless Design for Document Intelligence

Assay transforms document processing through an event-driven, serverless architecture that scales automatically, processes documents asynchronously, and delivers intelligent summaries in real-time. Beyond the web interface, Assay provides programmatic access through MCP (Model Context Protocol) and CLI/API interfaces, enabling users to interact directly with the document substrate for deeper analysis, synthesis, and custom workflows. This document outlines the architectural decisions and design patterns that make Assay both powerful and scalable.

1. Architecture Philosophy

Assay is built on a foundation of event-driven, serverless architecture using Google Cloud Platform. This design choice addresses the core challenges of document processing at scale: unpredictable workloads, the need for parallel processing, and the requirement for real-time user feedback. The architecture also supports programmatic access through MCP (Model Context Protocol) and CLI/API interfaces, allowing users to treat Assay as a document substrate for custom analysis, synthesis, and integration into larger workflows.

Why Event-Driven?
  • Decoupling: Each processing stage operates independently, allowing us to modify or extend functionality without affecting other components. Workers communicate through Pub/Sub topics, creating a loosely coupled system.
  • Scalability: Workers scale automatically based on message volume, handling spikes in upload activity without manual intervention. Google Cloud Functions scale from zero to thousands of concurrent executions based on demand.
  • Fault Tolerance: Failed messages can be retried automatically through Pub/Sub's built-in retry mechanism. Each worker is designed to be idempotent, ensuring safe re-execution without side effects.
  • Observability: The event flow provides clear visibility into processing stages, making debugging and monitoring straightforward. Each stage logs structured events with document IDs, user IDs, and processing metadata.
  • Extensibility: New processing stages can be added by simply subscribing to relevant Pub/Sub topics, without modifying existing code. This enables rapid feature development and experimentation.
  • Cost Efficiency: Serverless architecture means we only pay for actual processing time. During quiet periods, functions scale to zero, eliminating idle costs.

2. System Overview

At its core, Assay follows a simple but powerful pattern: upload triggers processing, processing publishes events, events trigger workers, workers update state, state updates appear in real-time. 

┌─────────────────────────────────────────────────────────────────┐
│                         ASSAY SYSTEM                           │
│                                                                 │
│  ┌──────────────┐         ┌──────────────┐                    │
│  │   Frontend   │────────▶│ Cloud Storage│                    │
│  │  (Next.js)   │  Upload │   (PDFs)     │                    │
│  └──────────────┘         └──────┬───────┘                    │
│         │                        │                             │
│         │                        │ Storage Trigger             │
│         │                        │ (onFileUpload)              │
│         │                        ▼                             │
│         │              ┌─────────────────┐                     │
│         │              │  Upload Trigger │                     │
│         │              │  (Entry Point)  │                     │
│         │              └────────┬────────┘                    │
│         │                        │                             │
│         │                        │ Pub/Sub Topics              │
│         │                        │ (Event Bus)                 │
│         │                        ▼                             │
│         │              ┌─────────────────┐                     │
│         │              │  Pub/Sub Topics  │                     │
│         │              │  (Message Queue) │                     │
│         │              └────────┬────────┘                    │
│         │                        │                             │
│         │              ┌─────────┴─────────┐                  │
│         │              │                   │                   │
│         │        ┌──────▼──────┐    ┌──────▼──────┐          │
│         │        │   Workers    │    │   Workers    │          │
│         │        │ (Parallel    │    │ (Parallel    │          │
│         │        │  Processing) │    │  Processing) │          │
│         │        └──────┬──────┘    └──────┬──────┘          │
│         │               │                   │                  │
│         │               └─────────┬─────────┘                 │
│         │                         │                            │
│         │                         ▼                            │
│         │              ┌─────────────────┐                    │
│         │              │   Firestore      │                    │
│         │              │  (Document State│                    │
│         │              │   & Summaries)   │                    │
│         │              └─────────────────┘                    │
│         │                        │                             │
│         │                        │ Real-time Listeners         │
│         │                        │ (onSnapshot)                │
│         │                        ▼                             │
│         │              ┌─────────────────┐                    │
│         │              │   Frontend UI     │                    │
│         │              │  (Live Updates)  │                    │
│         │              └─────────────────┘                    │
└─────────┴─────────────────────────────────────────────────────┘

Key Components

Frontend (Next.js)

  • React-based web application
  • Real-time Firestore listeners for live updates
  • Client-side validation and upload management
  • Theme-based search and document discovery

Cloud Storage

  • PDF file storage: uploads/{userId}/{documentId}/original.pdf
  • Summary storage: AI-generated summaries stored in Firestore
  • Original PDFs deleted after processing (only summaries retained, no extracted text kept)

Storage Trigger (`onFileUpload`)

  • Automatically fires when PDF is uploaded
  • Validates PDF signature and file size (5MB limit)
  • Creates Firestore document record
  • Publishes to pdf-uploaded Pub/Sub topic

Pub/Sub Topics (Event Bus)

  • pdf-uploaded - Triggers text extraction
  • pdf-text-extracted - Triggers parallel metadata and signals extraction
  • pdf-metadata-extracted - Metadata extraction complete
  • pdf-signals-extracted - Triggers theme matching and strategy selection
  • pdf-themes-match-requested - Triggers canonical theme matching
  • pdf-themes-matched - Triggers FAQ generation
  • pdf-strategy-selected - Triggers summary generation
  • pdf-summary-requested - Triggers single-pass or hierarchical summarization
  • pdf-faq-summary-requested - Triggers FAQ generation
  • pdf-summary-completed - Processing complete
  • pdf-processing-failed - Error tracking

Worker Functions (Cloud Functions)

  • All workers run in a single region for low latency
  • Auto-scaling based on message volume
  • Idempotent design for safe retries
  • Structured logging with document/user context

Firestore Database

  • processing/ collection - Processing state and raw data
  • display/ collection - UI-optimized display data
  • Real-time listeners enable live progress updates
Processing & Design →