Architecture

This section covers the technical details of how textrawl works under the hood.

Architecture Overview

textrawl is built on a modern stack optimized for semantic search:

• Runtime: Node.js 22+ with ES modules
• Database: Supabase PostgreSQL with pgvector extension
• Transport: HTTP (REST API, MCP over StreamableHTTP, WebSocket events)
• Embeddings: OpenAI, Google AI, or Ollama (configurable)

Request Flow

 MCP Client       Dashboard      REST Client       CLI
     │                │               │              │
     ▼                ▼               ▼              ▼
 POST /mcp      GET/POST /api   GET/POST /api   POST /api
     └────────────────┼───────────────┘──────────────┘
                      ▼
               ┌─────────────┐
               │   Express   │
               │   Server    │
               └──────┬──────┘
                      │
                      ▼
               ┌─────────────┐
               │    Tool     │
               │  Handlers   │
               └──────┬──────┘
                      │
                ┌─────┴─────┐
                ▼           ▼
           ┌─────────┐ ┌─────────┐
           │ Supabase│ │ OpenAI/ │
           │   DB    │ │ Ollama  │
           └─────────┘ └─────────┘

Core Components

Hybrid Search

The search system combines full-text keyword matching with semantic vector similarity using Reciprocal Rank Fusion (RRF).

Learn about hybrid search →

Text Chunking

Documents are split into overlapping chunks to fit within embedding model context limits while preserving semantic coherence.

• 512 tokens (~2048 characters) per chunk
• 50 token overlap for context preservation
• Paragraph-aware splitting on \n\n

Learn about chunking →

Embeddings

Vector representations that capture semantic meaning, enabling similarity search.

Learn about embeddings →

Database Schema

Core Tables

Memory Tables (Optional)

Key Indexes

Transport Layer

textrawl exposes three transport interfaces: MCP (for AI assistants), REST API (for programmatic access), and WebSocket (for real-time events).

MCP

Uses StreamableHTTPServerTransport for stateless HTTP operation:

• Stateless: Each request creates a fresh MCP server instance
• Serverless-compatible: Works with Cloud Run, Lambda, etc.
• Single endpoint: POST /mcp handles all MCP operations

REST API

Standard HTTP endpoints for direct integration:

• Search: GET /api/search?q=query&limit=10
• Documents: GET /api/documents, GET /api/documents/:id
• Upload: POST /api/upload

WebSocket

Real-time event streaming for document ingestion, insight discovery, and other background operations.

Rate Limiting

Security

• Bearer token authentication (API_BEARER_TOKEN)
• Row Level Security (RLS) on all tables
• Service role key bypasses RLS (single-tenant design)

See Security Hardening for production setup.

Next Steps

• Hybrid Search - Deep dive into RRF
• Chunking Strategy - How documents are split
• Embeddings - Vector representations