Prompt: semantic search and dedup with ird embeddings

Have an agent build a small semantic search / near-duplicate pipeline on ird jobs embed: your texts embedded once each, vectors cached on disk under a hash of the text so a rerun only pays for what changed, cosine ranking and duplicate pairs in a few lines of Python, the cost read from the price book before anything runs, and a mode that works unattended from a script with an API key.

Needs: the ird CLI, logged in or with IRD_API_KEY (ird jobs is CLI-only); Python 3 with numpy; a folder for the texts and the cache Costs: per embedding job (credits_prices → jobs.embedding: per_mb_in, per_mb_out, per_minute, multiplier); job status and result reads are not separately listed Group: developer workflow

Paste everything below the line into a fresh conversation. Fill in the FILL IN block first.

Task

Embed every text in INPUT with the ird embedding job, keep the vectors in CACHE_DIR, then answer QUERY by cosine similarity and list pairs above DUP_THRESHOLD. Show the estimated cost and wait for my yes before the first job.

FILL IN

INPUT:          ./texts/*.md          # files, one document each — or a CSV/JSONL with a text column
QUERY:          how do I cancel a subscription   # blank = dedup only
TOP_K:          10
DUP_THRESHOLD:  0.92                  # cosine similarity above this = near-duplicate
MODEL:          default               # default = what `ird jobs embed --help` shows (BAAI/bge-m3), or a model name
CACHE_DIR:      ./.embeddings
UNATTENDED:     no                    # yes = script for cron/CI with IRD_API_KEY, no prompts

Non-negotiable rules

1. Price before the first job. credits_prices → jobs.embedding: size the input (bytes of every uncached text ÷ 1 MB × per_mb_in), the output (vector bytes × per_mb_out), compute time (per_minute) and apply multiplier (how it composes with the three rates is undocumented — label the figure an estimate); show it and the job count, then wait for a yes. Re-estimate if INPUT changes.
2. Read the job output before writing the loop. The help doesn't say whether ird jobs embed prints the vector or a job id: run it once on a short text with --json, and branch on what comes back (a vector → done; a job id → ird jobs status <job-id>, then ird jobs result <job-id>). Poll gently — seconds apart, not a tight loop.
3. Results live 24 hours. ird jobs result keeps a result for 24 hours; the disk cache is the durable copy. Never rerun a job for a hash already in CACHE_DIR.
4. One model per cache. The key hashes MODEL plus the text; switching MODEL is a full re-embed — say so and re-price.

Steps

A. Collect texts. Read INPUT into a list of (id, text). Strip nothing that carries meaning; normalise only whitespace before hashing (sha256(MODEL + "\n" + text) with MODEL resolved to the real model name, not "default"). Report count, total bytes, and how many hashes already exist in CACHE_DIR.

B. Price. Step per rule 1 with the uncached count. Stop for confirmation.

C. Embed. For each uncached text, one call: ird jobs embed "$text" --json (--model MODEL unless default; the text is one argument — quote it, or pass a file as "$(cat file)"). Follow rule 2 for the result, then write CACHE_DIR/<hash>.json holding {model, dim, vector}. Retry a failed call once after a pause, then log and skip.

D. Rank. Load every cached vector into a matrix, L2-normalise, and:

import json, glob, numpy as np
vs = {f: json.load(open(f))["vector"] for f in glob.glob(CACHE_DIR + "/*.json")}
ids, M = list(vs), np.array(list(vs.values()), dtype=float)
M /= np.linalg.norm(M, axis=1, keepdims=True)
q = np.array(embed(QUERY), dtype=float)  # same call and cache as step C
q /= np.linalg.norm(q)
top = np.argsort(-(M @ q))[:TOP_K]
S = M @ M.T; np.fill_diagonal(S, 0)
dups = [(ids[i], ids[j], S[i, j]) for i, j in zip(*np.where(np.triu(S) > DUP_THRESHOLD))]

Map ids back to INPUT names; skip the query if QUERY is blank.

E. Unattended mode. If UNATTENDED: produce embed.sh that runs with IRD_API_KEY exported from the job's secret store (never in the script; --api-key would land in the process list), every ird call with --json --no-input so a prompt fails the job rather than hanging it, set -euo pipefail, the cache as the only state, and an exit code separating "nothing new" from "a job failed". If a call insists on confirmation, add --yes to that call only — it can spend credits, so it belongs nowhere else. The key comes from the api-keys-automation prompt.

F. Verify. Embed one text twice; the second run must hit the cache (no job). Check dim matches across files; run QUERY and eyeball the top three — they must read as related. Compare credits spent (credits_balance before and after; CLI: ird credits) with step B's estimate and report the ratio.

Deliverable

Counts (texts, embedded now, from cache), model and dimension, the cost estimate against credits actually spent, the TOP_K hits with scores, the duplicate pairs above DUP_THRESHOLD, the cache layout, and — if UNATTENDED — embed.sh plus the secrets it expects.