AI for lawyers: separating the four things being sold

Commercial uses of artificial intelligence often collapse distinct systems into a single category labeled “legal AI.” That label obscures more than it explains. A system that searches a company’s documents for relevant language, a system that answers questions from reported case law, a system that extracts obligations from contracts, and a system that classifies documents for discovery review operate on different information and create different risks.

The four categories may share software architecture, but they do not share a definition of accuracy, a typical failure mode, or an appropriate verification process. A buyer evaluating legal AI should begin by identifying the task, the source material, the person accountable for the result, and the mechanism that records review.

Retrieval and drafting assistants over internal documents#

Retrieval and drafting assistants work with documents supplied for a particular purpose: a matter file, a contract set, a policy collection, or another body of material held by the organization. They can search that corpus, identify passages that appear responsive to a prompt, summarize records, or draft text grounded in retrieved material. Some may provide an interface for asking questions about the collection rather than merely returning search results.

Accuracy for this category has several layers. Retrieval accuracy concerns whether the system surfaced the records needed to answer the question. Grounding accuracy concerns whether each assertion in the answer can be traced to the retrieved source. Drafting accuracy concerns whether the generated text accurately reflects the source’s terms and does not introduce unsupported facts. A fluent answer with a defective citation is not an accurate answer, regardless of how polished it appears.

The characteristic failure mode is unsupported synthesis. The assistant may combine language from separate documents, present a proposal as an established position, or fabricate a citation that resembles a genuine authority. Another failure is omission: the answer may sound complete while the retrieval step never surfaced the document that contained the controlling language.

A buyer should ask whether the system returns links or source excerpts with each answer, how citations are generated, and whether the model can distinguish quoted source text from generated analysis. The demonstration should include a source set containing deliberately difficult material, such as conflicting versions of an agreement or records with similar names. Buyers should also ask how permissions are enforced across custodians and matter folders, and what happens when a source document is removed or superseded.

Evaluation should test the whole retrieval-and-drafting chain rather than the text-generation layer alone. That means reviewing both the answer and the materials the system selected. A technically correct paragraph is not a reliable work product if the evidence for it was missing from the retrieval set or the cited source does not say what the answer claims.

Legal research tools over a case-law corpus#

Legal research tools answer questions using a defined corpus of judicial decisions and related legal materials. The key distinction is that the corpus is external to the requesting matter, and the task is to locate and explain legal authority. The system may identify relevant decisions, summarize a holding, compare treatment, or generate an answer with citations.

Accuracy here should not be reduced to whether a response sounds like legal reasoning. It includes whether the research system retrieved the relevant legal authority, whether it represented the procedural posture and scope of a decision correctly, and whether the citations resolve to the stated propositions. It also includes whether the system distinguishes binding authority from material that is merely relevant or distinguishable.

The characteristic failure mode is false or misplaced authority. A tool may cite a real decision for a proposition the decision does not support. It may omit a limitation, confuse the treatment of a decision, or give an answer that is sound under one set of facts but not the facts presented in the query. Generic statements that resemble established law are particularly difficult to detect without checking the cited material.

A buyer should ask how the corpus is defined, whether the system reports its retrieval date, and whether every citation can be inspected at the source. The tool should distinguish a quotation or paraphrase from an inference generated by the system. Buyers should also ask how the tool handles decisions with negative treatment, competing authorities, or different procedural postures.

A meaningful evaluation should include questions where a keyword search produces a plausible but incomplete set of results and where the answer depends on a specific distinction. Reviewers should open the cited decisions and compare the tool’s characterization with the actual text. The relevant control is not simply citation presence, but a documented check of authority, context, and treatment.

Contract review and analysis tools#

Contract review and analysis tools extract defined fields, compare clauses, identify departures from a playbook, and produce changes for review. Unlike a general drafting assistant, the central task is usually structured. The system works from a defined agreement type and a defined set of legal or business criteria, such as liability language, termination rights, payment terms, or notice obligations.

Accuracy for contract analysis depends on the schema. Extraction accuracy concerns whether the tool identifies the correct party, date, amount, or defined term. Classification accuracy concerns whether a clause falls into the expected category. Comparison accuracy concerns whether the tool identifies the actual difference between two versions rather than merely changing their wording. Negotiation accuracy concerns whether a proposed revision expresses the intended legal position.

The characteristic failure mode is a technically correct extraction attached to the wrong legal meaning. A tool may identify a liability cap but overlook an exclusion, fail to recognize a carve-out, or treat an obligation in the counterparty’s form as materially equivalent to the preferred form. It may also miss changes expressed through defined terms, incorporated schedules, or cross-references.

A buyer should ask how the system represents exceptions and dependencies, whether a result identifies the clause and surrounding text supporting the analysis, and how a reviewer can correct an extraction without erasing the audit history. The demonstration should use agreements containing amendments, schedules, and nonstandard language. Buyers should also determine whether the tool flags uncertainty or treats a clause as fitting a category when its language is ambiguous.

Verification should focus on the contractual unit as a whole. Reviewing a highlighted clause is not enough if the tool ignored a definition elsewhere in the agreement. The preferred control is a traceable output: each extracted value, classification, and proposed change should connect to source language, with the reviewer’s decision recorded separately.

Classification technology used in discovery#

Classification technology used in discovery is commonly described as technology-assisted review, or TAR. Predictive coding is one method within that category, and continuous active learning, or CAL, is another approach to identifying documents for review. These systems can rank or classify documents for responsiveness, privilege, or another defined issue. TAR predates the current commercial interest in generative AI and has its own body of judicial treatment.

Accuracy in discovery classification is usually evaluated in terms of the error the matter can tolerate. A missed responsive document and a nonresponsive document sent for unnecessary review are different errors. The assessment also depends on how well the classification categories fit the information, the quality of the document population, and the training or review process. For a specific issue, the relevant question is whether the system separates the defined category from everything else.

The characteristic failure mode is a systematic classification error that is difficult to see from a polished output. A document may be excluded because its format was not processed as expected, because the information appeared in an unusual location, or because the training examples did not represent the issue adequately. A classifier may also perform differently across custodians, data sources, or document types.

A buyer should ask how the system handles email attachments, native files, encrypted data, and data that requires processing before classification. The evaluation should explain how training examples were selected, how results were validated, and how the system identifies errors. It should also explain whether reviewers can inspect the document population and the system’s reasoning signals, and whether a human escalation path exists for uncertain material.

Discovery buyers should distinguish classification from extraction. A system that identifies likely responsive documents does not necessarily determine why a document is responsive, while a tool that extracts a contract term does not necessarily evaluate whether the contract is relevant to the preservation or collection issue. The controls must match the classification task and the record reviewed.

The discovery case law does not extend to generative drafting#

The clearest judicial treatment among these categories concerns TAR in discovery. That treatment should not be generalized to legal research assistants, contract analysis products, or generative drafting tools. The cited decisions address the use of technology-assisted review in identifying relevant electronically stored information; they do not establish that any generative tool produces reliable legal work product.

In Da Silva Moore v. Publicis Groupe, Magistrate Judge Andrew Peck held that computer-assisted review was an acceptable way to search for relevant electronically stored information in appropriate cases. The parties’ agreement to use it mattered. District Judge Andrew Carter affirmed in April 2012, and the decision is widely described as the first judicial opinion approving predictive coding.

In Rio Tinto PLC v. Vale S.A., Judge Peck described permission for a producing party to use technology-assisted review as black-letter law. Maura Grossman was later appointed special master in that case to resolve TAR disputes. These authorities support the acceptability of TAR in appropriate discovery settings; they do not decide whether an AI assistant should draft a brief, answer a legal research question, or revise a contract.

Hyles v. City of New York, decided on 1 August 2016, went further in a practical direction: a court will not force a producing party to use TAR when it prefers keyword searching. That conclusion is consistent with Sedona Principle 6, under which the responding party is best situated to decide how to search for and produce its own electronically stored information. Some language in Rio Tinto and Hyles is dicta rather than holding, so it signals judicial attitude rather than a binding rule.

The discovery framework remains relevant to how AI-assisted review is governed. Under Rule 26(b)(1), the 2015 amendment moved proportionality factors into the definition of the scope of discovery itself and added a factor for the parties’ relative access to relevant information. Rule 34(b)(2)(B) requires a responding party to state grounds for objecting to a request with specificity, while Rule 34(b)(2)(C) requires an objection to state whether responsive materials are withheld on that basis.

Rule 26(f) requires the parties to confer before the scheduling conference, and Rule 16(b) governs the scheduling order. Those provisions do not make a particular technology mandatory. They preserve room for a defensible discovery process, including decisions about technology-assisted review.

Preservation and production issues also remain separate from model performance. Rule 37(e) was rewritten in 2015 into a two-tier framework for electronically stored information that should have been preserved and was lost. Under (e)(1), a court may order measures no greater than necessary to cure the prejudice upon a finding of prejudice. Under (e)(2), the adverse presumption, instruction, dismissal, or default judgment options require a finding that a party acted with the intent to deprive another party of the information.

Controls that apply across all four products#

Confidentiality and training use. A buyer should determine what information is transmitted for processing, where it is stored, how long it is retained, and whether the information is used to train or improve a model. The answer should address both the organization’s own documents and information belonging to opposing parties, third parties, or individual employees. A contractual prohibition on using uploaded material for training should be evaluated separately from general claims that the vendor does not train on customer data.

Access controls should match the sensitivity of the material. The relevant questions include whether authentication is required, whether permissions travel with a document, whether an administrator can export or delete the collection, and whether a user can use one matter’s material in another. Confidentiality analysis should include the people and systems outside the legal team that may have access.

Verification workflow. Every category needs a named person responsible for checking the output, but the verification method depends on the task. A legal research answer should be checked against the cited authority. A contract extraction should be checked against the source clause and related definitions. A TAR result should be sampled and reviewed through a documented process. A drafting answer should be checked for both source support and the absence of unsupported propositions.

The workflow should also record how an error was corrected. A reviewer’s approval, a revised conclusion, and the reason for the change should be distinguishable from the system’s original output. That record makes later review possible without treating an unexplained edit as equivalent to independent verification.

Auditability. Auditability means being able to reconstruct what information was available, what the system retrieved, what it produced, and what the reviewer changed. Depending on the product, that may involve source excerpts, retrieval logs, generated output, version history, classification decisions, validation results, and the identity of the reviewer. A system that provides only a final answer may still be useful, but it offers less evidence for explaining or repeating the underlying judgment.

Professional competence. A lawyer remains responsible for the output regardless of how it was produced. The use of AI does not transfer responsibility for a citation, a contract interpretation, a discovery decision, or an advice communication. This is especially important where the output is plausible, fast, and embedded in a workflow that makes verification easy to overlook.

A practical evaluation sequence#

Begin by naming the category. Do not accept “legal AI” as a sufficient description of the task. Identify whether the system retrieves and drafts over internal documents, answers questions over a case-law corpus, extracts and compares contract provisions, or classifies discovery material. Then define the relevant error: an unsupported statement, an omitted authority, a missed contractual exception, or a missed responsive document.

Next, specify the source population and its controls. Ask how source documents enter the system, how outdated or conflicting versions are handled, and whether users can see the materials supporting an output. In discovery, connect the system to the collection and processing decisions reflected in the preservation, custodian, and collection record. The Electronic Discovery Reference Model provides a useful vocabulary for that work: information governance, identification, preservation, collection, processing, review, analysis, production, and presentation. It is a reference model, not a mandatory sequence, and matters often loop between stages.

Then test controls rather than demonstrations. Use a representative and adversarial sample, require the system to expose its sources and uncertainty, and require a reviewer to explain a sample of the results. Measure the error appropriate to the category rather than treating a single notion of accuracy as universal.

Finally, document the operating decision. Record the selected tool, its permitted use, the categories of material it may process, the review standard, the escalation path, and the person accountable for the result. A privilege log, load file, production format, redaction, clawback, legal hold, and custodian may all become part of the surrounding record, but no product substitutes for a coherent process that a lawyer can defend.

These four products can be useful without being interchangeable. Treating them as separate categories makes evaluation more concrete: define the task, identify the failure that matters, verify against the source, preserve an audit trail, and retain professional responsibility for the result.

Common questions#

What are the four categories of legal AI?

They are retrieval and drafting assistants over internal documents, legal research tools over case law, contract review and analysis tools, and classification technology used in discovery.

What are the main accuracy layers for retrieval and drafting assistants?

Retrieval accuracy concerns surfacing needed records, grounding accuracy concerns traceability of assertions, and drafting accuracy concerns faithful reflection of source terms.

How should legal research tools be evaluated?

Reviewers should open cited decisions and compare the tool's characterization with the actual text, checking authority, context, and treatment.

What has the cited discovery case law established about generative AI?

It addressed TAR in discovery and did not establish that generative drafting, legal research, or contract-analysis tools produce reliable legal work product.