Semantic Video Search

Suppose you remember a speaker comparing HNSW with IVFFlat, but you cannot remember the video title or the speaker's exact words. Keyword search works best when the question and transcript share terms; a paraphrase can be harder to find. This experiment searches the available transcripts by meaning and answers with quotes whose timestamps can seek the player to the relevant moment.

The example question on this page is illustrative. The architecture, defaults, commands, and test counts were checked against the public repository on 1 October 2026. This is a working learning project with known gaps. The page does not claim measured search quality.

The concepts behind one search#

A YouTube caption is a short piece of timed text. All captions together form the transcript. The ingester joins captions into chunks, passages that are long enough to carry context and small enough to search. Adjacent chunks repeat some text; that overlap keeps a sentence from disappearing at a boundary.

An embedding turns text into a list of numbers learned from patterns in language. A question and a passage about the same idea should end up close together even when their wording differs. PostgreSQL stores those lists through pgvector. Its HNSW index is a graph that follows promising neighbours instead of comparing the question with every stored vector. This makes search faster, with the trade-off that approximate search can miss a neighbour an exhaustive scan would find.

An agent is a model that can choose named tools, read their results, and choose another step. Here, each tool is ordinary Python around an embedding call or a database query. A citation is the final quote plus its start time in seconds, which the interface can turn into a seek action.

The map below separates ingestion, which happens once per video, from search, which happens for every question.

Ingesting a video#

The public implementation uses four stages. Look at which information is preserved at each hand-off: the text, its timing, and its vector must stay aligned.

The database index in the current public source is:

CREATE INDEX IF NOT EXISTS idx_transcript_chunks_embedding_hnsw
ON transcript_chunks USING hnsw (embedding vector_cosine_ops);

vector_cosine_ops makes pgvector order chunks by cosine distance. Smaller distance means the directions of the two vectors are more alike. The number is a ranking signal; it is not a probability that the chunk is correct.

Where overlap breaks the timestamp#

The overlap preserves words, but the current code assigns the next new caption's start time to the whole new chunk. This illustrative two-chunk trace makes the mismatch visible.

At 138 seconds, a new caption pushed the candidate text beyond 1,500 characters. The code finalised chunk 12, copied the last 200 characters into chunk 13, appended the new caption, and set current_start = 138.0. The copied words retain no mapping to their earlier caption, so a citation using chunk 13 can seek six seconds late in this example. A fix needs overlap at caption boundaries, or enough character-to-caption provenance to recover the first copied word's time.

Answering one question#

The question “What trade-off did the speaker give for HNSW?” becomes an embedding and enters a broad nearest-neighbour search. The table below is an illustrative trace that matches the current chunk fields. Distances are included to explain ordering; the current repository query orders by cosine distance but returns chunk objects without selecting that distance into the tool payload.

The initial tool groups those rows by clip and sends only each clip ID and its chunk text to the model. Start times and distances are absent from that first result. The agent can then search inside a promising clip and finally request cached quotes, where start times reappear.

The trace below uses made-up content but preserves the public request and response shapes. It shows the mechanics without presenting the example as a real search result.

{
  "call": {
    "name": "examine_clip_deeper",
    "args": {
      "clip_id": "vector-index-talk",
      "question": "What trade-off did the speaker give for HNSW?"
    }
  },
  "tool_result": {
    "clip_id": "vector-index-talk",
    "title": "Vector index talk",
    "description": "",
    "transcript": "HNSW uses more memory but keeps query latency low..."
  }
}

After the quote tool returns text and timestamps, the model must produce the SearchResponse shape below. The instruction says the answer must use only tool information and stay under 300 characters.

{
  "results": [
    {
      "clip_id": "vector-index-talk",
      "question": "What trade-off did the speaker give for HNSW?",
      "answer": "The speaker trades higher memory use for low query latency.",
      "relevant_quotes": [
        {
          "quote": "HNSW uses more memory but keeps query latency low.",
          "quote_description": "The stated index trade-off",
          "quote_timestamp": 138.0
        }
      ],
      "related_questions": []
    }
  ]
}

The timestamp makes the answer inspectable. It is the retrieved chunk's start time, not the exact time of the quoted sentence inside that chunk. It does not prove that the quote entails the answer or that a better chunk was missed. Those are evaluation problems covered by the linked research note.

Why use an agent here#

A fixed retrieval pipeline could embed the question, take the top chunks, and generate an answer in one route. This agent can broaden the search, inspect selected clips, and fetch quotes after it has chosen the evidence. That flexibility costs model round trips and creates more states to test.

The eight-turn cap bounds one part of that cost and prevents an endless tool loop. It does not guarantee a useful answer at turn eight. In the current graph, hitting the cap routes the latest message to the response parser; if that message contains a tool call rather than final JSON, the parser falls back to a raw-text response with no citations. A production version would need a separate, explicit “budget exhausted” result or one forced final-answer turn.

There is another boundary worth noticing. The initial tool discards start times and distances when it groups chunks, while the deeper tool concatenates its selected moments into one string. The quote tool restores timestamps from _clip_cache, which is shared by the Python process. Concurrent requests can clear or overwrite one another's cached clip entries. This keeps the model input small, but makes ranking decisions harder to inspect and makes citation quality depend on mutable process-wide state.

What it does not do yet#

The gaps below describe the public implementation as checked on 1 October 2026. They are questions to measure and repairs to verify, rather than promised improvements.

The research note Evaluating Retrieval for Transcript Search turns those gaps into a test plan. No improvement is claimed until that plan produces results.

Running the public project#

The current full-stack path uses Docker Compose. After cloning the repository, copy the example configuration, provide an OpenAI API key, and run:

cp .env.example .env
make up

The Streamlit interface then listens on port 8501. The public Makefile also provides make test, make test-unit, and make test-integration. The repository currently documents 36 tests: 32 unit tests across the chunker, transcriber, tools, graph and schemas, plus four HTTP integration tests. Those counts were checked from the public test listing; this portfolio review did not execute the separate repository.

Sources#

• Public repository: README and system overview, checked 1 October 2026
• Public source: chunk construction, embedding batches, and three agent tools
• Public source: agent graph and eight-turn cap, vector queries, and developer commands
• pgvector: HNSW and distance operators