Document review: how the largest line item is actually run

Document review is not a single event that begins when a production set is uploaded and ends when privileged documents are logged. It is an operating system for deciding what must be examined, who examines it, how the work is sequenced, and how the result can be defended. The financial exposure is often discussed only after the review population has been defined, but the largest cost decisions usually occur before a reviewer opens a document.

That makes review an operations problem rather than a software problem. A document-review platform may provide batching, coding, search, analytics, and production support, but the quality of the result still depends on population design, instructions, reviewer management, and quality-control sampling. Technology changes the mechanics of review; governance determines whether those mechanics produce a defensible record.

Review begins with the population, not the reviewer#

The review population should be defined and reconciled before substantive coding begins. Custodian collections, load-file records, date ranges, data sources, and known exceptions need to be tied to the preservation record and the issues in the litigation. If the population cannot be explained, neither the review metrics nor the final production can be explained.

De-duplication should occur before documents are distributed to reviewers. The same file may appear in several custodial collections, and common attachments may be produced again for each custodian. Removing exact duplicates reduces repeated reading without removing distinct custodial context. Metadata still matters: the same document can have different custodial, temporal, or issue relevance depending on where it was found.

Email threading provides a related reduction. Rather than assigning every message in a thread for separate review, the reviewer can examine an inclusive message containing the entire exchange, with child messages available as needed. The operation must preserve enough context to identify participants, attachments, and responses. A threaded design that hides relevant content or makes it difficult to retrieve a missing message shifts work backward into later investigation.

Near-duplicate grouping offers a more selective lever. Groups may contain materially similar drafts, forms, or standard documents that differ in limited ways. A reviewer can identify the group structure, assess the variations, and make a reasoned coding decision. The danger is treating similarity as substantive equivalence without reviewing the fields that differ.

Domain and date culling are especially powerful when the search for responsive material can be narrowed through information from the matter. A custodian’s role, the systems used, and the relevant chronology can support reasoned limits. The instruction should describe the evidentiary basis for the limit, the exceptions, and the person responsible for confirming it. A cull that merely reflects an intuitive sense of relevance is difficult to defend and can conceal responsive material.

The workflow has distinct decision points#

A workable review workflow separates collection and population preparation from responsiveness review, privilege review, issue coding, and quality control. Each stage needs an owner and a defined output. Combining all judgments into one coding pass may reduce interface steps, but it also makes errors harder to isolate and can encourage reviewers to guess when instructions are ambiguous.

Batching organizes documents for coherent review. Batches can be built by custodian, date, issue, data source, or some combination. The best structure depends on the review design, but each batch should have a clear question and known boundaries. Custodian-based batching supports context; issue-based batching supports consistent treatment of similar material. Mixed batches can work when the batch definition remains understandable to reviewers and later auditors.

Coding layouts and tag panels should reflect the decisions reviewers must make. A responsive/nonresponsive decision may need to remain separate from privilege, confidentiality, issue, and merits tags. If every possible value appears on every screen, reviewers spend attention interpreting the taxonomy instead of applying it. Tag panels should present the applicable sequence, use unmistakable labels, and distinguish a substantive code from an exception or workflow status.

First-pass responsiveness review asks whether the document is relevant to the defined matters and time periods. Instructions should define relevance by content and context rather than by keywords alone. Reviewers need to know which issues are under review, what information is considered within scope, and how near misses should be coded. A reviewer should not have to infer whether an incomplete communication, an attachment without a covering message, or a document outside the principal issue remains responsive.

Second-pass privilege and issue coding performs different work. Privilege review requires attention to the basis for withholding and the document’s communication context. Issue coding maps content to the theories and questions being developed. The second pass may identify documents that should return to responsiveness review because the first pass did not recognize their significance. That return path should be designed rather than left to ad hoc requests from reviewers.

For issue coding, the controlled vocabulary should reflect the decisions the litigation team expects to make. Codes should not duplicate an overly broad responsiveness tag, and a code should mean something different from every adjacent code. Legal-hold status and privilege treatment should remain auditable workflow events, not informal notes attached to a document without explanation.

Quality control should test the review rather than repeat it. Re-reviewing the entire population can consume the savings expected from targeted controls. Sampling concentrates review effort on a defined subset whose findings can test both responsiveness decisions and the application of instructions.

Staffing models carry different risks#

Three common staffing models place the review function in different hands. The choice should reflect the subject matter, volume, required speed, existing staffing, and the degree of judgment the matter demands.

Managed review can organize contract personnel, coding infrastructure, and quality-control reporting while preserving legal decisions with the matter team. The provider does not replace the need for a party to understand the workflow. Someone must approve the coding taxonomy, answer instruction questions, investigate exceptions, and determine how quality-control results affect the production.

Contract reviewers are not interchangeable. Their work should be managed through clear assignments, calibration examples, escalation paths, and feedback tied to documented instructions. Counsel reviewing directly still benefits from batch design, coding support, and independent sampling. The absence of a large external reviewer pool does not eliminate the need for structured quality control.

Quality control must produce evidence, not reassurance#

A defensible quality-control program identifies what it is testing, selects the sample, defines the decision standard, records disagreements, and explains what happens after the results are known. The sample should reflect the review population rather than consist only of documents already flagged as sensitive or contested. Reviewers should understand that participation in sampling is part of the process, not a judgment on individual performance.

Responsiveness quality is commonly tested through reviewer agreement: the degree to which independent reviewers make the same decision on the same document. Agreement is not identical to correctness, but persistent disagreement can expose unclear instructions, uncontrolled tag definitions, or differences in how reviewers interpret the review scope. Those findings are more useful when converted into revised instructions and targeted retraining.

A sampled null set matters as much as the responsive sample. A responsive set tests whether reviewers find material that should be produced. The null set tests documents expected to be nonresponsive and asks whether reviewers wrongly flagged them for production. Without that comparison, a program can measure the ability to recognize obvious responsiveness while missing overinclusive coding decisions.

The sampling plan should also address false negatives and false positives without pretending that a small sample reveals every error. Reviewers may examine the sampled documents independently, compare decisions, and record the reason for any disagreement. The goal is to identify systematic defects, not merely accumulate a raw agreement number that no one interprets.

Privilege review and issue coding need their own checks. Privilege samples should test whether the asserted basis is consistently understood and whether documents are being withheld for a stated reason. Issue-code samples can show whether similarly described communications receive similarly useful classifications. Because these judgments serve different purposes, combining them into a single review metric can conceal a problem in one stage.

Instructions are the center of the production#

Many disputes about a production are disputes about coding instructions. The producing party may describe a document as nonresponsive because the instruction excluded a subject; the requesting party may treat the same document as plainly responsive under the discovery scope. When the instruction is vague, reviewers resolve that ambiguity individually. The resulting inconsistency appears later as an unexplained difference between the production and the requesting party’s expectation.

Instructions should separate elements of relevance from examples and exceptions. Elements tell reviewers what to consider. Examples illustrate how elements operate, but an example should not silently become a new element. Exceptions should address recurring edge cases, such as incomplete threads, duplicate drafts, or communications that mention a topic without conveying relevant information.

Reviewer questions should be answered in a controlled way. Prompts that reveal that an answer may affect other documents should be logged, interpreted by the responsible decision-maker, and incorporated into the instructions. If a question is narrow and fact-specific, it may be resolved individually without changing the general rule. If it exposes ambiguity, the instruction should change. Reviewers need to see the resulting clarification, whether through an updated tag definition, written guidance, or a controlled communication from the matter team.

Quality depends on clarity, but even clear instructions do not eliminate variation. Reviewer fatigue can affect consistency, especially across long assignments or repetitive batches. Rotating difficult material, limiting unproductive work, and checking early batches can help reveal whether the design is sustainable. These are management choices, not substitutes for written standards.

Technology changes sequencing, not obligation#

TAR, predictive coding, CAL, and related methods can assist in identifying relevant ESI. In Da Silva Moore v. Publicis Groupe, the court accepted computer-assisted review as an appropriate search method in the circumstances presented, including the parties’ agreement to use it. In Rio Tinto PLC v. Vale S.A., Judge Peck described judicial willingness to permit technology-assisted review as black-letter law and appointed a special master to resolve TAR disputes. The language in Rio Tinto and Hyles includes dicta, so it signals judicial attitude rather than a binding rule.

Hyles v. City of New York also reflects that a producing party may retain keyword searching rather than use TAR. The responding party is generally best situated to decide how to search and produce its own ESI, consistent with Sedona Principle 6 as described in the decision. These authorities do not make review technology optional for every purpose; they frame the dispute around method, context, and the producing party’s selection.

Prioritized review changes the order in which documents are seen, not the obligation to make a defensible responsiveness determination. Whether a document receives priority because a model scored it highly or a reviewer selected it does not remove the need to apply the governing scope and instructions. The audit trail should connect prioritization to review decisions, especially when an error appears in the selected population.

Technology-assisted methods also make validation and documentation central. The training set, relevance judgments, error analysis, and treatment of unassessed documents should be explainable to the court and the opposing party. A platform may support the process, but the defensibility of the result rests on the design choices, the judgment applied, and the record created while conducting it.

The 2015 Federal Rules add an important outer discipline. Rule 26(b)(1) places proportionality factors within the definition of the scope of discovery and includes the parties’ relative access to relevant information. Rules 34(b)(2)(B) and 34(b)(2)(C) require specific objections and a statement whether responsive material is being withheld on that basis. Rule 37(e) addresses lost ESI that should have been preserved, separating curative measures from the more severe consequences available when intent to deprive is found. These provisions do not replace operational judgment, but they shape what a defensible process must document.

The practical standard is straightforward: establish the population, remove justified duplication, organize review into coherent batches, write instructions that separate legal elements from examples, staff the work with clear accountability, and test the result through independent sampling. The goal is not a claim that every reviewer will make every decision identically. It is a process that can explain differences, correct systematic errors, and show why the production reflects the governing scope rather than the habits of individual reviewers.