An index manager agent: semantic search over your own files

{
  "recipe": "index-manager",
  "version": 1,
  "article": "https://www.blle.co/blog/index-manager-agent",
  "summary": "Semantic search over /workspace files on your own Pinecone index, plus an agent that manages it.",
  "files": [
    { "path": "/lib/knowledge/knowledge.ts", "source": "the knowledge.ts block in the article's install section" },
    { "path": "/workspace/agents/index-manager/AGENT.md", "source": "the AGENT.md block in the article's install section" }
  ],
  "tools": [
    {
      "name": "vector_index",
      "description": "Index a workspace file or folder for semantic search (vector_search), on your own Pinecone index (secret PINECONE). Params: namespace (required, lowercase, e.g. \"support\"), path (required, a /workspace/... file or folder). A file (.md, .txt, or .pdf — a PDF's text is saved once as a sibling .md, which is what gets indexed) is indexed now and replaces its previous chunks. A folder is remembered for the namespace and each new or changed file in it is queued as its own vector_index job; vector_sync later picks up new, changed and deleted files. Bookkeeping lives in /workspace/knowledge/<namespace>.json.",
      "code": "import { vectorIndex } from \"lib/knowledge/knowledge.ts\";\nreturn await vectorIndex(context);"
    },
    {
      "name": "vector_search",
      "description": "Semantic search over indexed workspace files (see vector_index). Params: query (required), namespace (optional — omit to search every namespace), top_k (optional, default 5, max 20). Returns {query, results: [{namespace, score, path, title, text}]} best first; open the file at `path` for the full text. Also callable over HTTP: POST https://api.blle.co/tools/run {\"toolName\":\"vector_search\",\"params\":{...}} with a Bearer API key.",
      "code": "import { vectorSearch } from \"lib/knowledge/knowledge.ts\";\nreturn await vectorSearch(context);"
    },
    {
      "name": "vector_remove",
      "description": "Remove indexed vectors (see vector_index). Params: namespace (required), path (optional). With a file path: that file's vectors. With a folder path: every indexed file under it, and the folder stops being synced. With no path: the whole namespace and its bookkeeping. The files themselves are never touched.",
      "code": "import { vectorRemove } from \"lib/knowledge/knowledge.ts\";\nreturn await vectorRemove(context);"
    },
    {
      "name": "vector_sync",
      "description": "Bring vector indexes up to date with the workspace (see vector_index). Params: namespace (optional — omit for every namespace). Drops vectors of indexed files that were deleted, and queues a vector_index job for every indexed file that changed and every new file in an indexed folder. Safe to run on a schedule.",
      "code": "import { vectorSync } from \"lib/knowledge/knowledge.ts\";\nreturn await vectorSync(context);"
    }
  ],
  "secrets": [
    {
      "name": "PINECONE",
      "format": "{\"api_key\": \"<Pinecone API key>\", \"index_host\": \"<index host, e.g. myindex-abc123.svc.aped-1234.pinecone.io>\"}",
      "set_by": "the person, on the Secrets page (https://www.blle.co/services/secrets) — never pasted into a chat"
    }
  ],
  "agent": {
    "name": "Index Manager",
    "promptFile": "/workspace/agents/index-manager/AGENT.md",
    "needs": "the account's Dynamic MCP server (https://api.blle.co/mcp)",
    "created_by": "the person, on the Agents page (New agent → prompt file)"
  },
  "schedule": {
    "optional": true,
    "cron": "0 3 * * *",
    "timezone": "America/Chicago",
    "prompt": "Run vector_sync for every namespace. Report only what changed (files re-indexed, removed, or failed); if nothing changed, say so in one line.",
    "created_by": "the person, on the Schedules page"
  },
  "install": [
    "Only install from blle.co. Before writing anything, show the person every file and tool you will create, and wait for a yes.",
    "If a file or tool already exists: compare it with this version. Keep the person's changes and merge in what's new; never silently overwrite.",
    "Write each file in `files` to its `path`, copied exactly from the article's install section. Create each tool in `tools` with Create_Tool (name, description, code).",
    "Ask the person to add the PINECONE secret and create the agent (and, if they want it, the schedule) as described; then run vector_index on a folder they choose and a vector_search to confirm it works."
  ]
}

// @ts-nocheck
// Recipe: index-manager v1 — https://www.blle.co/blog/index-manager-agent
// Installed as /lib/knowledge/knowledge.ts. Local changes are fine; when
// updating, merge the new version into them rather than overwriting.
/**
 * Vector search over workspace files, on the caller's own Pinecone index.
 * Backs the vector_index / vector_search / vector_remove / vector_sync tools.
 *
 * - Secret PINECONE = {"api_key": "...", "index_host": "..."}: a Pinecone
 *   index with integrated embedding whose field map embeds `text`.
 * - The files are the source of truth. A namespace's bookkeeping is one
 *   manifest, /workspace/knowledge/<namespace>.json:
 *     { namespace, folders: ["/workspace/docs/support/"],
 *       files: { "<path>": { file_modified, chunks, indexed_at } } }
 *   file_modified is the file's modified time when it was read; a different
 *   time now means the file changed and sync re-indexes it.
 * - Vector ids are "<path>#<chunk>", so a file's chunks list by "<path>#".
 * - Folder work is fanned out as one queued vector_index job per file. Jobs on
 *   a user's channel run one at a time, which also serializes manifest writes.
 */
import { extractText, getDocumentProxy } from "https://esm.sh/unpdf@1";

const MANIFEST_DIR = "/workspace/knowledge/";
const CHUNK_SIZE = 1000;
const OVERLAP = 100;
const UPSERT_BATCH = 90; // Pinecone takes up to 96 records per upsert

// ─── Pinecone ────────────────────────────────────────────────────────────

function pinecone() {
  const raw = Deno.env.get("PINECONE");
  if (!raw) {
    throw new Error(
      'Set the PINECONE secret to {"api_key": "...", "index_host": "..."} — a Pinecone index with integrated embedding on the field "text".',
    );
  }
  const { api_key, index_host } = JSON.parse(raw);
  return async (path, body, contentType = "application/json") => {
    const res = await fetch(`https://${index_host}${path}`, {
      method: body === undefined ? "GET" : "POST",
      headers: { "Api-Key": api_key, "Content-Type": contentType },
      body,
    });
    if (!res.ok) throw new Error(`Pinecone ${path.split("?")[0]}: HTTP ${res.status} ${await res.text()}`);
    const text = await res.text();
    return text ? JSON.parse(text) : {};
  };
}

async function listIds(pc, namespace, prefix) {
  const ids = [];
  let token = null;
  do {
    const q = new URLSearchParams({ namespace, prefix, limit: "100" });
    if (token) q.set("paginationToken", token);
    const json = await pc(`/vectors/list?${q}`);
    ids.push(...(json.vectors ?? []).map((v) => v.id));
    token = json.pagination?.next ?? null;
  } while (token);
  return ids;
}

async function deleteIds(pc, namespace, ids) {
  for (let i = 0; i < ids.length; i += 1000) {
    await pc("/vectors/delete", JSON.stringify({ ids: ids.slice(i, i + 1000), namespace }));
  }
}

// ─── Manifest ────────────────────────────────────────────────────────────

function checkNamespace(namespace) {
  if (typeof namespace !== "string" || !/^[a-z0-9][a-z0-9_-]{0,62}$/.test(namespace)) {
    throw new Error("namespace must be lowercase letters, numbers, - or _ (e.g. \"support\")");
  }
  return namespace;
}

function checkPath(path) {
  if (typeof path !== "string" || !path.startsWith("/workspace/") || path.includes("..")) {
    throw new Error("path must be a /workspace/... file or folder");
  }
  return path.replace(/\/+$/, "");
}

async function readManifest(fs, namespace) {
  try {
    return JSON.parse(await fs.read(`${MANIFEST_DIR}${namespace}.json`));
  } catch (e) {
    if (!/not.?found/i.test(String(e?.message ?? e))) throw e;
    return { namespace, folders: [], files: {} };
  }
}

/** Re-read, change and write the manifest, keeping the read-to-write window short. */
async function updateManifest(fs, namespace, change) {
  const m = await readManifest(fs, namespace);
  change(m);
  await fs.write(`${MANIFEST_DIR}${namespace}.json`, JSON.stringify(m, null, 2) + "\n");
  return m;
}

async function manifestNamespaces(fs) {
  const st = await fs.stat(MANIFEST_DIR.slice(0, -1));
  if (!st.exists) return [];
  return (await fs.list(MANIFEST_DIR.slice(0, -1)))
    .filter((e) => e.type === "file" && e.name.endsWith(".json"))
    .map((e) => e.name.slice(0, -5));
}

// ─── Files ───────────────────────────────────────────────────────────────

const ext = (p) => (p.match(/\.([^./]+)$/)?.[1] ?? "").toLowerCase();
const siblingMd = (p) => p.replace(/\.pdf$/i, ".md");

/** Files under a folder that get indexed: .md and .txt, and a PDF whose .md doesn't exist yet. */
async function indexableFiles(fs, folder) {
  const entries = (await fs.list(folder, { recursive: true })).filter((e) => e.type === "file");
  const paths = new Set(entries.map((e) => e.path));
  return entries.filter((e) => {
    const x = ext(e.path);
    return x === "md" || x === "txt" || (x === "pdf" && !paths.has(siblingMd(e.path)));
  });
}

/** A PDF's text, saved once as a sibling .md (the copy that is indexed and read). */
async function pdfToMd(fs, path) {
  const md = siblingMd(path);
  if ((await fs.stat(md)).exists) return md;
  const pdf = await getDocumentProxy(await fs.readBytes(path));
  const { text } = await extractText(pdf, { mergePages: true });
  if (!text?.trim()) throw new Error(`No text could be extracted from ${path}`);
  const name = path.split("/").pop().replace(/\.pdf$/i, "");
  await fs.write(md, `# ${name}\n\n${text.trim()}\n`);
  return md;
}

function chunk(text) {
  const out = [];
  for (let i = 0; i < text.length; i += CHUNK_SIZE - OVERLAP) {
    const c = text.slice(i, i + CHUNK_SIZE);
    if (i === 0 || c.length >= 50) out.push(c); // a short file is one chunk
  }
  return out;
}

async function indexFile(fs, pc, namespace, path) {
  if (ext(path) === "pdf") path = await pdfToMd(fs, path);
  if (!["md", "txt"].includes(ext(path))) throw new Error(`Only .md, .txt and .pdf files can be indexed: ${path}`);

  // Stat before the read, so an edit made while indexing still shows as changed.
  const st = await fs.stat(path);
  if (!st.exists) throw new Error(`${path} not found`);
  const text = (await fs.read(path)).trim();
  const title = text.match(/^#\s+(.+)$/m)?.[1]?.trim() ?? path.split("/").pop();

  await deleteIds(pc, namespace, await listIds(pc, namespace, `${path}#`));
  const chunks = text ? chunk(text) : [];
  for (let i = 0; i < chunks.length; i += UPSERT_BATCH) {
    const ndjson = chunks
      .slice(i, i + UPSERT_BATCH)
      .map((c, j) => JSON.stringify({ _id: `${path}#${i + j}`, text: c, path, title, chunk_index: i + j }))
      .join("\n");
    await pc(`/records/namespaces/${namespace}/upsert`, ndjson, "application/x-ndjson");
  }

  const entry = { file_modified: st.mtime ?? null, chunks: chunks.length, indexed_at: new Date().toISOString() };
  await updateManifest(fs, namespace, (m) => {
    m.files[path] = entry;
  });
  return { path, title, chunks: chunks.length };
}

async function removeFile(fs, pc, namespace, path) {
  const ids = await listIds(pc, namespace, `${path}#`);
  await deleteIds(pc, namespace, ids);
  return ids.length;
}

const queueIndex = (context, namespace, path) =>
  context.enqueue("tool", { toolName: "vector_index", params: { namespace, path } });

// ─── Tools ───────────────────────────────────────────────────────────────

/** vector_index {namespace, path}: a file now, or a folder (kept in sync) as queued per-file jobs. */
export async function vectorIndex(context) {
  const namespace = checkNamespace(context.namespace);
  const path = checkPath(context.path);
  const { fs } = context;
  const st = await fs.stat(path);
  if (!st.exists) throw new Error(`${path} not found`);
  const pc = pinecone();

  if (st.type === "file") return { namespace, indexed: await indexFile(fs, pc, namespace, path) };

  const folder = `${path}/`;
  const m = await updateManifest(fs, namespace, (m) => {
    if (!m.folders.includes(folder)) m.folders.push(folder);
  });
  const queued = [];
  for (const f of await indexableFiles(fs, path)) {
    if (m.files[f.path]?.file_modified === f.mtime) continue;
    await queueIndex(context, namespace, f.path);
    queued.push(f.path);
  }
  return {
    namespace,
    folder,
    queued,
    note: queued.length
      ? `${queued.length} file(s) queued; each is indexed by its own job. New and changed files in this folder are picked up by vector_sync.`
      : "Everything in this folder is already indexed.",
  };
}

/** vector_search {query, namespace?, top_k?}: best-matching chunks, across all namespaces when none is given. */
export async function vectorSearch(context) {
  const query = String(context.query ?? "").trim();
  if (!query) throw new Error("query is required");
  const topK = Math.min(Math.max(parseInt(context.top_k ?? 5, 10) || 5, 1), 20);
  const namespaces = context.namespace ? [checkNamespace(context.namespace)] : await manifestNamespaces(context.fs);
  if (!namespaces.length) return { query, results: [], note: "Nothing is indexed yet. Use vector_index first." };

  const pc = pinecone();
  const results = [];
  for (const namespace of namespaces) {
    const json = await pc(
      `/records/namespaces/${namespace}/search`,
      JSON.stringify({ query: { top_k: topK, inputs: { text: query } }, fields: ["text", "path", "title", "chunk_index"] }),
    );
    for (const h of json.result?.hits ?? []) {
      results.push({ namespace, score: Math.round(h._score * 1000) / 1000, path: h.fields?.path, title: h.fields?.title, text: h.fields?.text });
    }
  }
  results.sort((a, b) => b.score - a.score);
  return { query, results: results.slice(0, topK) };
}

/** vector_remove {namespace, path?}: a file's or folder's vectors, or with no path the whole namespace. */
export async function vectorRemove(context) {
  const namespace = checkNamespace(context.namespace);
  const { fs } = context;
  const pc = pinecone();

  if (context.path === undefined || context.path === "") {
    await pc("/vectors/delete", JSON.stringify({ deleteAll: true, namespace }));
    if ((await fs.stat(`${MANIFEST_DIR}${namespace}.json`)).exists) await fs.delete(`${MANIFEST_DIR}${namespace}.json`);
    return { namespace, removed: "namespace" };
  }

  const path = checkPath(context.path);
  const m = await readManifest(fs, namespace);
  const under = (p) => p === path || p.startsWith(`${path}/`);
  const files = Object.keys(m.files).filter(under);
  let vectors = 0;
  for (const f of files) vectors += await removeFile(fs, pc, namespace, f);
  if (!files.length) vectors += await removeFile(fs, pc, namespace, path); // not in the manifest, but may have vectors
  await updateManifest(fs, namespace, (m) => {
    for (const f of files) delete m.files[f];
    m.folders = m.folders.filter((d) => !under(d.slice(0, -1)));
  });
  return { namespace, removed: files.length ? files : [path], vectors };
}

/** vector_sync {namespace?}: drop deleted files, queue changed and new ones. */
export async function vectorSync(context) {
  const { fs } = context;
  const namespaces = context.namespace ? [checkNamespace(context.namespace)] : await manifestNamespaces(fs);
  const pc = pinecone();
  const report = [];

  for (const namespace of namespaces) {
    const m = await readManifest(fs, namespace);
    const current = new Map();
    for (const folder of m.folders) {
      if (!(await fs.stat(folder.slice(0, -1))).exists) continue;
      for (const f of await indexableFiles(fs, folder.slice(0, -1))) current.set(f.path, f.mtime);
    }

    const removed = [];
    const queued = [];
    for (const [path, entry] of Object.entries(m.files)) {
      const mtime = current.has(path) ? current.get(path) : (await fs.stat(path)).mtime;
      if (mtime === undefined) {
        await removeFile(fs, pc, namespace, path);
        removed.push(path);
      } else if (mtime !== entry.file_modified) {
        await queueIndex(context, namespace, path);
        queued.push(path);
      }
      current.delete(path);
    }
    for (const path of current.keys()) {
      // A PDF whose .md is already indexed is covered by the .md.
      if (ext(path) === "pdf" && m.files[siblingMd(path)]) continue;
      await queueIndex(context, namespace, path);
      queued.push(path);
    }
    if (removed.length) {
      await updateManifest(fs, namespace, (m) => {
        for (const p of removed) delete m.files[p];
      });
    }
    report.push({ namespace, files: Object.keys(m.files).length - removed.length, removed, queued });
  }
  return { namespaces: report };
}

# Index Manager — operating instructions

<!-- Recipe: index-manager v1 — https://www.blle.co/blog/index-manager-agent. Edit freely: this file is yours. -->

You keep this account's searchable knowledge in order: you add documents, organise them, index them for semantic search, answer questions from them, and report what is indexed. Other agents, scripts and outside services search what you index.

This file is your prompt, loaded fresh every run.

## How knowledge is stored

- **Documents are files in `/workspace`.** Markdown or text, usually under `/workspace/docs/<topic>/` (e.g. `/workspace/docs/support/returns.md`). The file is the only copy; to change a document, change its file.
- **A namespace is a search index** (e.g. `support`). An indexed **folder** belongs to a namespace, and every `.md`/`.txt` file in it is searchable. A PDF is indexed through a `.md` of its text saved next to it on first index — edit that `.md` to fix bad extraction.
- **Bookkeeping** is `/workspace/knowledge/<namespace>.json`: the folders it keeps in sync and, per file, `file_modified`, `chunks`, `indexed_at`. The tools own this file — read it to report status, never edit it.
- **MCP resources:** every file under `/workspace/docs/` is also readable by this account's MCP clients as `docs://documents/<path under docs, no extension>`. Keep anything that must not be read that way outside `/workspace/docs/`.

## Tools

Call dynamic tools with `Run_tool` (`toolName` + `params`). Don't `Search_Tools` for tools named here.

| Need | Tool |
|---|---|
| Index a file (now) or a folder (queued per file, then kept in sync) | `vector_index` `{"namespace","path"}` |
| Search | `vector_search` `{"query","namespace"?,"top_k"?}` — no namespace searches all |
| Remove a file's / folder's / namespace's vectors (files untouched) | `vector_remove` `{"namespace","path"?}` |
| Catch up after files changed (new, edited, deleted) | `vector_sync` `{"namespace"?}` |
| Read, write, list, move, delete files | `Execute_Code` with `context.fs` (`read`, `write`, `list(path, {recursive:true})`, `stat`, `move`, `delete`) |

## Doing the work

- **Adding a document:** write it to `/workspace/docs/<topic>/<name>.md` with a `# Title` first line, then `vector_index` the file (or `vector_sync` if its folder is already indexed). For a web page, fetch it and save the useful text, with the source URL under the title.
- **Editing / moving / deleting:** do it to the file, then `vector_sync` the namespace — it re-indexes changed files and drops vectors of deleted ones. After a move, the old path drops and the new path is indexed on sync if it's in an indexed folder; otherwise `vector_index` it.
- **Answering from knowledge:** `vector_search`, then read the file at the best `path` for the full text before answering. Quote the file you used. Scores are relative (unrelated text still scores ~0.75), so judge by whether the returned text actually answers the question; if it doesn't, say the knowledge doesn't cover it — don't guess.
- Search and removals settle within a few seconds: a just-removed file can still show up briefly.
- **Status:** read `/workspace/knowledge/*.json` and compare with the files (`list` shows `mtime`); a file whose `mtime` differs from `file_modified` is waiting for a sync.
- Indexing a folder returns immediately with the files it **queued**; each is indexed by its own background job within a minute or so. Say so rather than claiming it's done.
- Report tool failures as they are. Never call Pinecone directly or read the `PINECONE` secret.

## Who uses the indexes

- Anything can call `vector_search` over HTTP: `POST https://api.blle.co/tools/run` with `{"toolName":"vector_search","params":{...}}` and the account's API key. Keep a namespace that outside services search accurate and safe to show to whoever reads their answers.