Keyword search vs. concept search: where each one fails

Search methodology is a substantive discovery decision, not merely a technical preference. Keyword search asks whether documents contain specified strings. Concept and similarity search ask whether documents express meaning associated with a concept. Each method can retrieve useful material that the other misses, but each can also produce failures that become apparent only after review.

Neither method is self-validating. A defensible search process documents the proposed method, tests the results, records revisions, and permits the producing party to explain how responsive information was located. That process matters more than any single term, ranking system, or combination of features.

What keyword and Boolean search actually match#

Boolean/keyword search operates on strings. A term identifies documents containing that exact text, subject to the operators and options selected by the reviewer. Quoted phrases, AND, OR, NOT, proximity operators, wildcards, and field restrictions give the reviewer different ways to combine those strings.

Those mechanics make keyword search familiar and explainable. A search for “Project Bluebird” AND invoice can be described as requiring both the quoted project name and the word “invoice.” A reviewer can inspect the expression, reproduce it in another system, and reason about why a document appeared in the result set.

The same mechanics also explain keyword search’s characteristic weaknesses. It matches the form of the indexed text, not the concept behind the text. Its results therefore depend heavily on the words people happened to use and on the processing decisions that made those words searchable.

Over-inclusion: one term, several meanings#

A term can produce over-inclusion when it has more than one meaning. Suppose the proposed term is “board.” It may refer to a corporate board, a physical board, a board document, or an editorial board. A Boolean expression containing that term can retrieve relevant records while also retrieving records that use the word in an unrelated sense.

Over-inclusion can also arise from common words in signature blocks, routing text, or repeated document furniture. A term appearing in every message from a particular custodian may identify correspondence with that custodian without adding information about the subject of the dispute. The result is volume rather than responsiveness.

The problem compounds when a term is used as a culling mechanism. If irrelevant documents are removed only because they contain a term, a concept expressed without that term may never reach the review stage. The reviewer must distinguish a term that identifies a subject from a term that merely identifies a participant, document type, or event.

Under-inclusion: the relevant document never uses the term#

Under-inclusion occurs when a responsive document lacks the selected strings. People may describe an arrangement without using its formal name, refer to a project by an internal code, misspell a product name, or adopt language intended to avoid an obvious term. An attachment may contain the relevant language while the parent email contains none of it.

Consider a hypothetical negotiation discussed in emails as “the arrangement,” even though the supporting documents repeatedly refer to a definitive agreement. A term list centered on the formal agreement name may miss the conversational messages that establish what the parties understood. Another document may identify the project by a code that appears only in its filename or attachment content rather than in the searchable body text.

Near-duplicate detection, email threading, and attachment processing can mitigate some retrieval problems, but they do not eliminate semantic variation. A relevant message still may not contain the expected term. Keyword recall is therefore constrained by language choice, not merely by document volume.

Concept and similarity search#

Concept search represents meaning through patterns of language co-occurrence. Words that appear in similar contexts can receive related representations even when they do not share an obvious synonym. Similarity search uses those representations to identify documents whose content appears close to a query concept or to an already-reviewed document.

This approach can retrieve related language that a term list misses. It may connect a discussion of an agreement with documents using terms such as “transaction,” “contract,” or “approval,” without requiring the reviewer to decide in advance which vocabulary should appear in the responsive set. The same mechanism can connect a project name with internal code names when their language occurs in comparable contexts.

Predictive coding is related but not identical to a simple concept query. In a TAR workflow, a system can learn from reviewer judgments and apply a model to rank or classify documents by likely responsiveness. CAL, or continuous active learning, is a TAR method in which reviewer feedback is incorporated during the review process rather than treating the model as a fixed search executed at the outset.

Why concept results are harder to explain#

Concept and similarity methods reach beyond exact strings, but that reach complicates explanation. A keyword expression can be stated directly: it requires the listed terms and the listed operators. A concept result may depend on learned relationships, the text used to construct representations, and the similarity of an entire passage to a query or exemplar.

The distinction does not make either method superior. It changes the type of question an opponent may ask. For keyword search, the question may concern term selection, fielding, processing, or the treatment of synonyms. For concept search, the question may concern how the system derived relevance and why a document received a particular score or classification.

A review record should therefore explain concept search in operationally meaningful terms. It may describe the responsiveness definition, the examples used, the treatment of false positives and false negatives, the sampling plan, and the revisions made during review. This is different from asserting that a semantic method simply “finds meaning.”

The negotiated reality of search terms#

Search terms are often negotiated between the parties rather than imposed unilaterally. Rule 26(f) requires the parties to confer before the scheduling conference, and Rule 16 governs the scheduling order. In that conferral process, the parties can address the proposed search methodology, sources, custodians, date ranges, format, testing, and any agreed iteration.

The defensibility of keyword search does not come from the term list alone. It comes from testing how the expression performs against known responsive documents and against a sample of documents expected to be nonresponsive. Reviewers can record hit counts, inspect the retrieved set, identify irrelevant results, and revise the expression.

Iteration is part of the evidentiary record. A term may appear too broadly at the outset and narrow after testing. A phrase may prove too rigid and expand after reviewers identify relevant documents that use different language. A code may enter the list after a known document is missed. The important question is not whether the first list was perfect. It is whether the process exposed defects and incorporated what the testing showed.

Rule 26(b)(1) places proportionality factors within the definition of the scope of discovery and adds the parties’ relative access to relevant information. Those factors can frame a discussion about search burden, source selection, and method, although their application remains matter-specific. A negotiated plan is not automatically defensible merely because both parties agreed to it.

Concrete failure modes#

Several common failures can be tested without attributing them to any actual matter:

• A signature-block term. The term “consultant” appears in the signature block of every email from a proposed witness. The expression retrieves unrelated exchanges because the term identifies a role, not the subject under review.
• An attachment-only term. A document discusses the disputed transaction, but the search expression is applied only to indexed body text. The email retrieves nothing because the decisive term appears in an attachment that was not processed for the intended text.
• A conversation without the formal term. Participants repeatedly call a proposed transaction “the deal.” None uses the project’s formal name, and the term-based expression misses the exchange entirely.
• A deliberate euphemism. The participants describe approval of a document as “signing off.” A concept-oriented query may connect that language to the approval process, while a narrow keyword query may not.
• A code-name mismatch. The relevant documents use an internal code, but the negotiated terms list only the customer-facing name. Similarity search may bridge the vocabulary if the language distributions connect them.

These examples show why hit counts alone are insufficient. A large result set can reflect a noisy term. A small result set can reflect precise terminology. A zero-hit query can reveal a vocabulary mismatch rather than the absence of responsive information.

Combining methods: cull, then rank#

Many workflows use keyword search and TAR together rather than treating them as competing alternatives. In a common sequence, terms identify or cull a candidate population, and TAR ranks that population by likely responsiveness. Reviewers concentrate on the highest-ranked documents and return lower-ranked material for subsequent evaluation.

The sequence can be efficient because term culling reduces the population that the ranking model must process. It can also be dangerous. A relevant document excluded during culling cannot be rescued by later ranking. The culling step therefore requires its own testing, particularly when the expression relies on a formal term, a code, or vocabulary that a custodian may not use.

A more defensible workflow can preserve both channels. The producing party can begin with an expression, review known responsive documents to identify missed language, and adjust the expression before culling. TAR can then rank the expanded candidate set. Reviewers can separately search for responsive documents that resemble known examples and compare those results with the term-based results.

The method should be scaled to the matter rather than treated as a fixed recipe. A relatively compact collection, a large custodial collection, and a mixed set of structured and unstructured data present different search problems. The record should explain the workflow actually used, including any culling, ranking, active learning, and review cycles.

The null set and the search for what terms missed#

The “null set” is the portion that matters most for testing a hybrid workflow: responsive material found through TAR or another review method but absent from the keyword-culled population. Sampling that set shows what the terms missed and whether the culling expression needs revision.

A null-set sample can reveal recurring language, missing code names, attachment-only references, or communication patterns that the initial expression did not capture. Reviewers can then determine whether the issue is vocabulary, processing, data-source coverage, or the culling design itself. The sample provides evidence of the gap; it does not by itself prove the prevalence of that gap throughout the collection.

This is why hit counts, sampling, and iteration form a connected validation process. Hit counts show the scale and noise of retrieval. Sampling shows what the result contains and omits. Iteration converts those observations into changes. A term list standing alone has none of those properties.

The producing party’s search choice#

Hyles v. City of New York addresses the allocation of control over search methodology. Judge Peck held that a court will not force a producing party to use TAR when it prefers keyword searching. The reasoning aligns with Sedona Principle 6, which places the responding party in the best position to decide how to search for and produce its own ESI.

The practical consequence is that a requesting party generally cannot dictate the producing party’s method merely by preferring a different technology. The parties can negotiate search terms, test expressions, propose sampling, and define agreed procedures. Disputes remain possible when a particular method produces a demonstrable deficiency or when the parties disagree about the agreed process.

Some language in Rio Tinto PLC v. Vale S.A. and Hyles is dicta rather than holding. That caveat matters. Those discussions signal judicial attitudes toward technology-assisted review; they are not binding rules that select a particular search method for every matter.

Da Silva Moore v. Publicis Groupe and Rio Tinto nevertheless show a different point: courts can accept TAR as a search method in appropriate circumstances. Agreement between the parties was important in Da Silva Moore, while Rio Tinto described judicial acceptance of technology-assisted review as black-letter law and later appointed a special master to resolve TAR disputes. The negotiated and documented process remains central.

Building a search record that can withstand challenge#

A defensible record identifies more than the final expression. It documents the information sources considered, the processing performed, the search terms and their alternatives, known responsive documents used for testing, the review protocol, and the changes made after sampling. It also distinguishes documents excluded by a search expression from documents excluded during review.

Objections to production must be stated with specificity under Rule 34(b)(2)(B), and an objection must state whether responsive material is withheld on that basis under Rule 34(b)(2)(C). Those requirements make the search and review record relevant to disputes about whether the process was reasonable and whether withheld material exists.

A weak search should not be confused with the loss of ESI. Rule 37(e) addresses electronically stored information that should have been preserved and was lost. Under its first tier, a finding of prejudice permits measures no greater than necessary to cure the prejudice. Under its second tier, intent to deprive another party is required before a court may presume the information was unfavorable, instruct the jury accordingly, or dismiss the action or enter default judgment. Search failure and preservation failure are different problems and should not be conflated.

The strongest position is neither an unquestioned keyword list nor an unexplained semantic model. It is a transparent process that uses each method where its mechanics are useful, tests both retrieval and culling, records what changed, and ties the negotiated methodology to the discovery record.