TAR, predictive coding and CAL: what the courts have actually said

TAR, predictive coding and CAL: what the courts have actually said

Technology-assisted review, or TAR, applies machine-learning techniques to the review of electronically stored information, or ESI. In a conventional predictive-coding workflow, a reviewer codes a set of documents as responsive, nonresponsive, privileged, or otherwise material. The system uses those judgments to rank the remaining documents according to their likely responsiveness. Reviewers then work through the population in that order, with the objective of reaching a point at which the remaining unreviewed documents have sufficiently low expected value to stop reviewing them.

TAR does not make the legal decision. Reviewers supply the training judgments, and the system generalizes from them. That distinction matters because the output is only as meaningful as the coding instructions, the training set, the review population, and the controls used to determine when review can stop. A high-performing ranker cannot compensate for an incomplete collection, inconsistent coding, or a poorly defined issue.

One-shot predictive coding and continuous active learning, or CAL, use the same central mechanism but differ in how the model interacts with the review. One-shot predictive coding trains on a seed set and then applies the resulting model once. CAL continues to retrain as reviewers code additional documents, incorporating those judgments and returning the highest-ranked unreviewed documents for further assessment. In either approach, the model is a prioritization and review-management tool rather than a substitute for review.

What the federal cases establish#

Da Silva Moore v. Publicis Groupe is the foundational decision. In the Southern District of New York in 2012, Magistrate Judge Andrew Peck held that computer-assisted review was an acceptable way to search for relevant ESI in appropriate cases. The agreement of the parties to use the technology mattered to that determination. District Judge Andrew Carter affirmed in April 2012. The decision is widely described as the first judicial opinion approving predictive coding.

The significance of Da Silva Moore is narrower than a general endorsement of any technology-assisted process. The court considered computer-assisted review in the setting before it and attached importance to the parties’ agreement. The case therefore supplies a useful foundation for the proposition that TAR can be acceptable, but it should not be converted into a proposition that TAR is required or that any model and protocol will be accepted.

Rio Tinto PLC v. Vale S.A., decided in the Southern District of New York in 2015, moved the discussion forward. Judge Peck treated a producing party’s choice to use technology-assisted review as uncontroversial and described the proposition that courts will permit it as black-letter law. Maura Grossman was later appointed special master in the case to resolve disputes concerning TAR. That appointment illustrates a separate practical point: disagreement about a TAR process may become a technical dispute requiring a neutral decision-maker rather than a general argument about whether TAR is permissible.

Hyles v. City of New York, decided on 1 August 2016, addressed the question from the opposite direction. Judge Peck refused to force a producing party to use TAR when that party preferred keyword searching. The ruling was consistent with Sedona Principle 6, which places the responding party in the best position to decide how to search for and produce its own ESI. The case confirms an important limit on the prior decisions: judicial acceptance of TAR does not mean that a producing party can be required to abandon another reasonable method.

Some of the language in Rio Tinto and Hyles is dicta rather than holding. That language still signals judicial attitude toward the technology and the allocation of decision-making authority, but it should not be treated as a binding rule in every jurisdiction. The distinctions should be stated precisely. A court allowing TAR is not a court requiring it. A producing party selecting TAR is not automatically entitled to use TAR without a defensible process. A party that prefers keyword search is not automatically relieved of showing that its process produces the required information.

The defensibility record comes before the algorithm#

A defensible TAR workflow documents how the model was trained, how its performance was evaluated, and how reviewers used it. The documentation is the defensibility. The algorithm is not. Courts and opposing parties can evaluate a well-described process, a defined population, coding instructions, and a reproducible record. They cannot readily evaluate a vendor assertion that a proprietary system is accurate or effective without a record showing what the system did and what reviewers did with its output.

The process begins before any model is configured. The matter team defines the issues to be coded, identifies the relevant custodians and data sources, and states the collection and processing assumptions that shape the population. That may include how email is preserved, how attachments and families are handled, what de-duplication means, how email threading is applied, and how near-duplicate detection is treated. Processing choices do not merely prepare data for the model; they determine what the model is being asked to rank.

A written protocol should also explain the division between responsiveness and privilege. Reviewers need coding instructions that define the issue, describe examples and counterexamples, and address documents that may qualify under more than one category. The protocol should explain how conflicting codes are escalated, how revised instructions are communicated, and how the effect of later changes is assessed. A record that shows only a final sample total does not reveal whether the underlying judgments remained consistent.

The training set should be described with enough specificity to permit evaluation. The record can identify who selected the documents, which custodians or sources they came from, how they were sampled, and why the set was considered suitable for training. It should also explain how responsive, nonresponsive, privileged, and other coded documents were allocated. If reviewers were given a common set of documents, that avoids two independent coders producing incompatible training examples. If judgments were compared through an independent process, the protocol should identify how differences were reconciled.

Training-set size does not substitute for training-set quality. A small set may omit important document types, and a larger set may encode inconsistent judgments. The protocol should therefore explain how relevant variations were represented, including different custodians, business functions, document formats, and communication types. It should also record whether documents with little visible text were included, whether duplicates or near-duplicates were removed, and whether unusual attachments or encoded content affected the population.

Stabilization is the point at which the review team concludes that additional manual review is unlikely to change the production materially. The defensibility record should explain how that conclusion was reached. Relevant questions include how performance was measured, what documents were reviewed after each model update, whether the reviewed sample included lower-ranked documents, and what threshold or other decision criterion was used. The point is not to suggest a universal measurement. It is to show that the stopping decision was tied to an articulated process rather than to convenience or an unexplained change in workflow.

Quality control should be separately documented. A common structure is to have additional reviewers assess a sample of documents, including documents near the relevant ranking boundary and examples drawn from different portions of the population. The record should identify who performed the quality-control review, what the reviewers were shown, and how disagreements were handled. It should also distinguish a validation exercise from a second training set. The same documents cannot serve as both an independent check on the model and evidence that the model learned from the reviewers’ answers.

The record should address exclusions. If certain files, custodians, sources, document types, or languages were excluded from the population, the reason should be stated and tied to a stated assumption. A claim that an exclusion is immaterial is less useful than an explanation of what was excluded, how the exclusion arose, and how the team assessed its effect. The same principle applies to corrupted files, unsupported image content, encrypted materials, and documents that could not be processed into a reviewable form.

How the model operates during review#

Once training begins, the operational record should show how the model and reviewers worked together. For one-shot predictive coding, the record should identify the training set, the resulting model run, and the ranked review population. For CAL, it should show the feedback loop: which documents were coded, how those judgments were incorporated, how the rankings changed, and how the team assessed stabilization. The distinction is not semantic. One-shot coding and CAL create different audit trails and different questions for a party challenging the process.

The model’s rankings should also be understood in context. A document ranked near the top may receive immediate attention, but the most productive review may involve both highly ranked and boundary documents. A production set should not be treated as a direct output of the algorithm without showing the reviewer decisions applied to the ranked population. The load file, coding decisions, privilege log, production records, and model records should permit the production path to be reconstructed.

Where a workflow moves from TAR to keyword search, or uses both methods, the protocol should identify the point at which the method changes and explain the reason. Keyword search remains useful for targeted inquiries, such as locating a specific term or testing whether a concept appears in particular custodians’ documents. A hybrid workflow is easier to evaluate when the roles of each method are defined rather than left to the reviewing team’s unrecorded judgment.

Procedural treatment should be considered alongside the technical workflow. Rule 26(f) requires the parties to confer before the scheduling conference, and Rule 16(b) governs the scheduling order. A TAR proposal can be discussed through that process, including the proposed method, the review population, validation steps, and how disputes will be addressed. Rule 26(b)(1) places proportionality factors within the definition of the scope of discovery and adds the parties’ relative access to relevant information. Those provisions do not select a search technology, but they frame the proposal and the parties’ respective positions.

Dispute procedures also matter. The record should identify how privilege disputes, coding disagreements, and challenges to the validation method will be handled. Rule 34(b)(2)(B) requires a responding party to state the grounds for an objection with specificity, and Rule 34(b)(2)(C) requires the objection to state whether responsive materials are withheld on that basis. TAR does not alter those obligations. Nor does use of TAR remove the need to explain why a withheld category is responsive, nonresponsive, privileged, or otherwise outside the production.

Where TAR is weak and keyword search remains suitable#

TAR is weak on very small collections. When the full population is limited and economical to review, the cost of training, validating, and documenting a model may outweigh any review benefit. The fact that a model can rank documents does not make it worthwhile to rank them in a collection that can be reviewed directly.

Foreign-language material presents another limit unless the system and review process have the appropriate support. A model trained on English-language responsiveness judgments may not transfer reliably to documents whose language, terminology, or context differs. The defensible record should address the languages in the population, the training examples for each relevant language, and whether separate models or review approaches were used. Unsupported language capability should not be treated as an answer to the issue.

Image-only documents can present the same problem in a different form. If material is not visible as usable text to the model or the review platform, it may not influence ranking in a predictable way. The workflow should distinguish optical character recognition, image analysis, and ordinary text-based classification. It should also state how reviewers assessed documents that the system could not read or interpret through text. A limitation in the model’s input is not cured simply by describing the document population accurately.

TAR is also weak where responsiveness is not a text-visible concept. Some issues depend on context that cannot be inferred from the document alone, such as the significance of an event known to be true, the relationship among records separated across sources, or the practical operation of a business process. If the model cannot observe the necessary features, no amount of ranking precision resolves the issue. Those matters may call for targeted keyword searches, issue-specific review, or closer examination of a defined subset.

Keyword search remains the right tool when the target is a known term, the population is small, the documents are image-based, language support is absent, or the legal issue cannot be expressed through document content. Boolean or keyword search can also provide a useful complement to TAR by testing a focused hypothesis or investigating a narrow group of records. The choice should follow the review problem, not the label attached to the technology.

Generative AI raises a separate defensibility question#

Generative AI is being added to review tooling, but it should not be folded into the settled TAR discussion. A generative system may assist with document summarization, issue coding, query formulation, or other review tasks. Its outputs and failure modes require separate evaluation. A document summary is not the same as a responsiveness classification, and an AI-generated issue list is not a training set with validated reviewer judgments.

The defensibility record for generative AI is thinner than the record commonly built for TAR. The underlying TAR cases establish that a form of computer-assisted review can be acceptable; they do not establish that any generative-AI feature is acceptable for a particular task. The documentation burden for generative AI is the same as for TAR or higher because the workflow must explain the model’s role, the instructions and context provided to it, the handling of document content, the review of outputs, and the record of corrections.

Relevant questions include whether a reviewer can distinguish model-generated text from source text, whether the output is treated as a suggestion or a substantive determination, whether privilege and responsiveness judgments remain attributable to reviewers, and whether the system’s behavior can be reconstructed for a disputed document. The workflow should also document what information was made available to the system and how that information was protected within the review environment.

For TAR, the core record remains the training set, coding instructions, review decisions, validation approach, stabilization analysis, and production trail. Generative AI adds questions about prompts, retrieved context, generated outputs, human corrections, and version-specific system behavior. Neither technology shifts the need to define the population, explain the review method, and show how the produced material was identified and evaluated.

The cases support a measured proposition. Courts in the Southern District of New York have accepted computer-assisted review in appropriate circumstances and have not required a producing party that prefers keyword search to use TAR. Those decisions make TAR a credible review method, not a self-executing defense. Its defensibility continues to rest on the process record: how the population was built, how the model was trained, how reviewers interacted with it, how performance and stabilization were assessed, and how the resulting production can be explained.