CLI Tools
Batch Upload
Upload converted files to your knowledge base with automatic embedding
Upload converted markdown files to your knowledge base with automatic chunking and embedding.
Usage
pnpm upload -- <path> [options]Options
| Option | Default | Description |
|---|---|---|
--force | false | Re-upload even if already uploaded |
-v, --verbose | false | Enable verbose logging |
--dry-run | false | Preview without uploading |
--batch-size <n> | 10 | Files per batch |
Example
# Upload directory
pnpm upload -- ./converted/emails
# Force re-upload
pnpm upload -- ./converted/emails --force
# Dry run
pnpm upload -- ./converted/ --dry-runUpload Process
- Scan Directory: Find all
.mdfiles - Check Manifest: Skip already-uploaded files
- Parse Frontmatter: Extract metadata
- Chunk Content: Split into ~512 token chunks
- Generate Embeddings: Via OpenAI or Ollama
- Store in Postgres: Documents + chunks + embeddings
- Update Manifest: Track uploaded files
Manifest File
A .manifest.json file tracks uploaded files:
{
"version": 1,
"files": {
"email-001.md": {
"hash": "abc123",
"documentId": "550e8400-...",
"uploadedAt": "2024-03-20T14:22:00Z"
}
}
}Use --force to ignore the manifest and re-upload everything.
Rate Limiting
Uploads are rate-limited to avoid API throttling:
- Embedding API: 60 requests/minute
- Server API: 100 requests/minute
Error Handling
Partial Failures
If some files fail, the upload continues. Failed files are logged and can be retried.
Resume After Error
Re-run the same command. Successfully uploaded files are skipped via manifest.
# Retry after failure
pnpm upload -- ./converted/emailsNext Steps
- Search your knowledge using MCP tools