Proof of Concept
Using Deep Learning to Prevent Litigation
Nelson E. (Nick) Brestoff
© Intraspexion Inc. 2016 All Rights Reserved.
Attorneys who work in the legal departments of businesses understand the high value of less litigation. In another work, cited below, I showed that one of the principal advantages of less litigation was financial, and that, in fact, the average cost of avoiding having to make payouts in settlements and verdict, attorneys’ fees, and administrative costs, was about $350,000 per case.
And that result means that learning how to avoid or prevent only three average lawsuits would preserve just over $1 million.
Today, in-house attorneys are stuck with “door law,” meaning that they must deal with lawsuits as they “come in the door.” They cannot see the risks in their own internal communications with enough time or clarity to engage in prevention or enable their executives to be proactive.
But in the future, to achieve the end of less litigation, we believe that inhouse counsel will be turning to the latest advance in artificial intelligence, Deep Learning. [For a 20-minute overview of Deep Learning, I recommend Jeremy Howard’s TED talk, accessible via YouTube.] We have learned how to apply Deep Learning to the legal profession, and we report our initial results here, what we consider to be a Proof of Concept for an “early warning system” to find the signals of a potential lawsuit in an enterprise’s internal communications.
The signals of risk, which can be assessed soon after the enterprise internal emails are generated, enable in-house counsel to conduct an investigation, before the damage is done. In-house attorneys are, after all, closest to the data. With this approach, in-house counsel can help to prevent litigation, not just manage it.
To see if Deep Learning would enable an early warning system, we trained a Deep Learning algorithm with unstructured text related to employment discrimination torts. The algorithm was not hand-crafted or pre-programmed in any way. There were no emails in the training data.
We decided to test a Deep Learning system on email data it had never ingested before: a subset of the well-known Enron corpus. We obtained one 3 month’s worth of email messages to and from Ken Lay, the Chairman and CEO of Enron from the Electronic Discovery Reference Model (www.edrm.net).
We began by researching the federal court litigation database, known as “Public Access to Court Electronic Records” (PACER) for statistics about Enron. (A step-by-step tutorial to explain how we collected these statistics on Enron using PACER is posted under the heading “How to Build A Litigation Risk Profile.”)
For the five-year period 1997-2001, the chances of finding a workplace discrimination case against an Enron company was only about one percent. During that five year timeframe, an Enron company was named in litigation 1,339 times, and was named in an employment discrimination case only 13 times. For our experiments, we did not expect to find an employment discrimination case in Ken Lay’s emails because, aside from the fact that we had selected only one month of emails, and did so at random, we also knew that employees were unlikely to reach out to Ken Lay. After all, he was not the director of HR.
Next, we data-mined PACER to create a set of training documents in this silo, and trained a Deep Learning algorithm. With apologies, this aspect of our work is a trade secret, and we will not discuss what documents we used to train the algorithm or further describe the algorithm itself.
Once the Deep Learning algorithm was trained, we presented it with test documents, i.e., one month of Ken Lay’s emails (which we did not read in advance), and conducted our experiments in baby steps. As recounted in Corporate Counsel Magazine’s January 2016 issue, we struck gold immediately. We found 22 emails which scored at 0.90 or above for accuracy out of 6,352 emails,4 and two of them expressed a risk of employment discrimination.
(By accuracy, we mean a number which indicates how close a test document comes to hitting an imaginary bulls-eye for the training documents.)
A distribution of the scores output to us by the now-trained Deep Learning algorithm appears below. The distribution provides context for the highest scoring emails, which are the emails we would expect an in-house attorney to review first. But, in addition, we think any investigating in-house counsel would want to see the range of results. With a distribution, a reviewing attorney could spot-check the lower-scoring emails to decide whether he or she had seen enough.
In the distribution bar graph below, where the y-axis runs from 0 to 25, we show only the scores ranging from 0.80 to 1.00.
This bar graph shows that there were 129 emails scoring at 0.80 or above, but only 22 emails at 0.90 or above. We reviewed the top-scoring 22 emails. Most of them were false positives, but two emails stood out. Scoring at 0.94, both of 5 them presented a discrimination risk, but it was the same risk, because one email was a “forward” of the initial email. The subject of that email was “[M]y unfair treatment at Enron--Please HELP.” (See Exhibit A.) The first paragraph of that email began:
Dear Mr. Lay, [M]y employment with Enron is to be terminated, the reason given by HR, for not meeting performance standards. However, I firmly believe that this is not the real reason, the real one being for defying the wishes of my manager …, who, I believe was acting in a discriminatory way towards me….
As we increased the number of our training documents, added in documents relating to the Holy Roman Empire (to create a negative alternative or, if you will, “noise”), and resolved some technical impediments which were due to data processing safeguards, the Ken Lay email subset grew to 7,665.
In our last experiment before writing this whitepaper (our eighth), only four emails scored at or above 0.80 for accuracy with respect to the training data. Two emails scored above 0.90 for accuracy, while the other two scored 0.88 and 0.86. In the distribution bar graph below, note that the y-axis runs from 0 to only 12, which shows that the results were now much more focused.
Now, imagine you are an in-house attorney and the distribution above has been sent to you as an “alert.”
Imagine further that you could move your cursor over a one of the four top-scoring rows on a spreadsheet or any of the four top-scoring bars on the graph.
Then imagine that, with a mouse click, the emails for a particular score appear on your screen, along with a menu of options. You can keep an email, forward it to someone else, or decide that it is a false positive. (False positives are not necessarily useless. Indeed, they may be useful for recalibrating (tuning) the algorithm.) You might highlight some portion of the email or simply archive it.
If you keep it, you can link to the other compliance, investigation, or case management platforms (and tools) that may be available to you
Let’s look at the other high-scoring emails. The email which scored 0.92 was slightly higher than the risky employment discrimination emails (see Exhibit B), but it was about a conflict of interest involving investments during the employee’s time at Enron. As a result of the allegation, the employee lost a bonus. This email presented a risk, but not a risk of a potential employment discrimination lawsuit. A reviewing in-house counsel might forward this email to an attorney tasked with investigating conflicts of interest.
The highest scoring email, the one scoring at 0.97, was also from an Enron employee in India. (See Exhibit C.) In part, it read:
Subsequently, I was forced upon a cheque of severance amount on 27th August 2001 which I
received under protest and the notice of such protest was served by me on firm in the interest of justice. I submit such a termination is illegal, bad in law and void ab-initio and accordingly when such an action was not corrected by firm, I [was] constrained to approach the Court of law.
Thus, the discrimination risk had already resulted in a lawsuit. A reviewing attorney might annotate and archive this email.
In our experiments, since the emails scoring 0.86 and 0.88 were the same, our early warning system found one email out of 7,665 emails which signaled a risk of an employment discrimination claim. We had found a needle in a haystack.
We also saw a steady increase in algorithm “sharpness” when we added the “negative” training documents and as we increased the number of training documents.
Now suppose that such an early warning system had been installed by the legal department. An attorney receiving an alert would still have needed to review the results, but the system would have given that person something to review. Currently, in-house attorneys are blind to such risks.
And if the reviewing attorney agreed that the system had surfaced a risk, he or she would have had a starting point for an internal investigation, that allimportant first clue. If that investigation warranted further action, the attorney could conduct an internal investigation and could thereafter report to a control group executive.
In other words, when the legal department installs the system, the attorney’s work will be in anticipation of litigation, and should be confidential under the 9 work-product doctrine. Furthermore, the reviewing attorney’s report to a control group executive should be confidential under the attorney-client privilege.
So enabled, that executive could be proactive, and either avoid or mitigate the risk.
The key innovation is to enable in-house counsel to be able to receive an alert in the first place, and to see whether a high-scoring email signals the risk of a potential lawsuit of a specific type, here an employment discrimination risk.
Given our results, we believe that a Deep Learning system, trained to score internal communications, e.g., emails and attachments, is a potential game-changer for the legal profession. Avoiding a potential lawsuit is far less expensive (and far less stressful) than having to deal with lawsuit that is filed and served.
We therefore conclude with a prediction: with Artificial Intelligence in the form Deep Learning, the role of in-house counsel will eventually move from managing lawsuits to avoiding them.
# # #
Nelson E. (Nick) Brestoff was a California litigator for 38 years, after earning two degrees in engineering at UCLA and Caltech, and attending law school at the University of Southern California. He is the principal author of 10 Preventing Litigation: An Early Warning System to Get Big Value Out of Big Data (Business Expert Press, 2015), and is the Founder and CEO of Intraspexion Inc.
This email scored 0.86 and 0.88. You will find “my unfair treatment at Enron—Please HELP” in the Subject line. The first paragraph in the message alleged that performance issues were a pretext for a manager’s “acting in a discriminatory way towards me.”
This email scored 0.92 but concerned a “conflict of interest” regarding an investment. Both the Subject line and the message contain the phrase “Conflict of interest.”
This email scored 0.97 but indicated that a lawsuit had already been filed: “I constrained to approach the Court of law.” (In the message field, see the last two lines.)