Finding the Right Passage: Information Retrieval and Text Mining

You search for “How do you stop a model from fitting the training data too closely?” The lecture instead says, “Regularization reduces overfitting.” A search based only on matching words can miss the connection. At the same time, a search for a specific model number should not favor vaguely similar terms.

Information retrieval investigates how to find relevant information for a particular need. Text mining extracts value from texts through classification, extraction and other analytical methods. In our example media library, both begin with the transcript but lead to different results: relevant passages on one side and additional information about the collection on the other.

Word-based search remains an important foundation

An inverted index maps terms to the documents or positions where they occur. A query therefore does not have to reread every text. Ranking methods score the candidates found. BM25 accounts for factors including term frequency, a term’s rarity in the collection and document length. Precision and recall are fundamental to evaluation: the proportion of returned results that are relevant, and the proportion of all relevant results that were found. Manning, Raghavan and Schütze, 2008: Introduction to Information Retrieval

An illustrative calculation: the evaluation collection contains 20 passages relevant to a question. Search returns ten results, eight of which are relevant. Precision is 8/10, or 80 percent. Recall is 8/20, or 40 percent. The list contains mostly useful results but misses many relevant passages.

Which property matters more depends on the task. Someone seeking a single suitable explanation primarily wants a good result near the top. Someone documenting every mention of a topic needs as complete a collection as possible.

A transcript is often the wrong unit of retrieval

For our library, a whole lecture transcript would be too coarse. A result should contain a time range with enough context. At the same time, splitting must not destroy meaning: an explanation spanning two paragraphs should not become two barely understandable fragments.

One possible design aligns segments with pauses and topic sections, storing a start, end and media version for each. Whether overlapping excerpts help would need to be tested. Too much overlap could fill the top result positions with variations of the same passage.

Segmentation is therefore part of the search method. Better ranking cannot magically restore missing context. The data model in Part 4 keeps timestamps and versions connected.

Learned vectors extend search

Dense retrieval maps queries and passages to learned vectors. A similarity function selects candidates. This allows different phrasings to match. The original Dense Passage Retrieval paper demonstrates this approach for open-domain question answering. Its results apply to the datasets studied there and do not automatically transfer to every specialist collection. Karpukhin et al., 2020: Dense Passage Retrieval for Open-Domain Question Answering

For the example library, a comparison would therefore make sense: first word-based search, then vector search, followed by a combination. A hybrid variant could merge result lists and rerank them. This would be an additional development decision, not an automatically superior method.

Separate results for technical terms, proper names, paraphrases and transcription errors would be especially revealing. An average across all questions might hide a substantial weakness in an important group. Part 6 on machine learning explains why suitable evaluation conditions matter.

Text mining produces additional clues

Named entity recognition can identify names of people or organizations. Classification methods assign categories to texts. Information extraction investigates relationships and events, for example. A recognized name string does not yet uniquely identify a real person; context and ambiguity must be considered. Jurafsky and Martin: Speech and Language Processing, online manuscript

In the library design, these methods could suggest topic filters. Suggestions would be stored with provenance and review status. A term such as “Java” would need to be considered in context before becoming a subject assignment. An automatically obtained category should remain correctable too.

Evaluate search and generated answers separately

Retrieval-augmented generation, RAG, combines finding external content with generative processing. The original paper combines a neural retriever with a generative model. External passages provide an additional information source; this does not guarantee error-free reproduction of their content. Lewis et al., 2020: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

For our library, we would therefore check three questions separately: was a suitable source found? Does the answer accurately represent what it says? Does the source reference lead to the correct passage? An incorrect result and an incorrect summary require different corrections.

An answer feature would be added only after search demonstrably works. Until then, a good result list with text context and direct playback is a complete, useful feature. Part 3 on UX evaluation investigates how people actually use it. The project article describes the shared evaluation collection for these comparisons.