Batch Upload

Upload converted markdown files to your knowledge base with automatic chunking and embedding.

Usage

pnpm upload -- <path> [options]

Options

Example

# Upload directory
pnpm upload -- ./converted/emails
 
# Force re-upload
pnpm upload -- ./converted/emails --force
 
# Dry run
pnpm upload -- ./converted/ --dry-run

Upload Process

1. Scan Directory: Find all .md files
2. Check Manifest: Skip already-uploaded files
3. Parse Frontmatter: Extract metadata
4. Chunk Content: Split into ~512 token chunks
5. Generate Embeddings: Via OpenAI or Ollama
6. Store in Postgres: Documents + chunks + embeddings
7. Update Manifest: Track uploaded files

Manifest File

A .manifest.json file tracks uploaded files:

{
  "version": 1,
  "files": {
    "email-001.md": {
      "hash": "abc123",
      "documentId": "550e8400-...",
      "uploadedAt": "2024-03-20T14:22:00Z"
    }
  }
}

Use --force to ignore the manifest and re-upload everything.

Rate Limiting

Uploads are rate-limited to avoid API throttling:

• Embedding API: 60 requests/minute
• Server API: 100 requests/minute

Error Handling

Partial Failures

If some files fail, the upload continues. Failed files are logged and can be retried.

Resume After Error

Re-run the same command. Successfully uploaded files are skipped via manifest.

# Retry after failure
pnpm upload -- ./converted/emails

Next Steps

• Search your knowledge using MCP tools