for the corporate law departments of the future,
Intraspexion Is the New Sonar
By Nick Brestoff, Founder and CEO and Jagannath Rajagopal, Chief Data Scientist
On January 29, 2018, in the Artificial Lawyer blog by the UK's Richard Tromans, Ari Weinstein (CEO of Confident Contracts) reported on a panel at LTNY's Legal AI Bootcamp: "The 'kumbaya moment' of the session," he reported, "was when all agreed on the panel that AI not only helps legal teams to be more efficient, but it helps you to do what you could not do before." (Italics added.)
So let's start off by describing what it is, as a member of Corporate Law Department of the Present, that you can't do now.
Consider your daily flow of internal communications. All that data used to be called a data lake. Now it's an ocean of data, and you're closer to it than any outside counsel or eDiscovery vendor. But you can't see into that ocean. You don't know what's in those communications.
Wouldn't you like to see the risky or "smoking gun" emails? You know, the ones lurking below the surface of your ocean; the ones that will show up as being material in some future lawsuit. Would you like to know about them before you have to manage the lawsuit?
Of course you would. If you could do that, you could be proactive about the risks.
But you can't be proactive now, can you? You can't see these risks.
That's where we come in. Our system is like Sonar, a device that emits sound waves and let users see below the surface of the water.
Without Sonar, you can't see below the surface. But with Sonar, you can: you can locate fish and shipwrecks, as below.
And with Sonar, you can also find threats like underwater mines, except that your underwater mines are the risky emails.
Our system is like Sonar. Intraspexion is a new tool for corporate law departments. It's a litigation "early warning system."
NEEDLES IN A HAYSTACK
The form of Artificial Intelligence we use is called Deep Learning.
in this White Paper, and without getting into the math of Deep Learning, we'll explain how our patented software system works. (Our patents are listed under About Us in the 7 Patents sub-page.)
To better explain why Intraspexion is the New Sonar, we'll also use the analogy of finding the needles in the haystack. We know you've heard that one.
For starters, here's a little bit of our history. The term "first light" refers, generally, to the first use of a new instrument. It's then that you see what needs further attention. In Q4 of 2017, we completed a pilot project with a company whose identity and confidential information we’re not permitted to disclose.
However, in a non-confidential telephone communication, a company attorney reported that our system had found, in a now-closed discrimination case, a risky email that the company already knew about. That was good news, but it was not exciting news.
But we were also told that our system had also found a risky email the company had later determined was material, but also that, previously, the company had not known about it.
Now that was compelling. How’d that happen?
Deep Learning (for text) requires two basic ingredients: a classification of data (i.e., a label) and lots of examples of the classification (a "positive" set) and, for contrast, examples of text we wouldn't want to find (a "negative" set).
When we put the pieces together, we've created a "model."
Our first classification is “employment discrimination.”
Now for "positive" and "negative" examples.
We created a “positive” set of examples from the factual allegations in hundreds of previously filed discrimination complaints in the federal court litigation database called PACER, and in the classification for “Civil Rights-Jobs," which (in PACER) is Nature of Suit code 442.
"Civil Rights-Jobs" is the PACER category. The less formal name is "employment discrimination."
Now, for our purposes, we didn't care about the legalese of "jurisdiction and venue," the names of the parties, or specific claims that were being made. And it didn't matter whether the discrimination was for age, race, sex, and any other sub-category of discrimination. PACER has no sub-classifications for them.
Second, we created a “negative” set of examples that was “unrelated” to “employment discrimination.” This negative set consisted of newspaper and Wikipedia articles and other text, including emails.
But to the best of our knowledge, there were no "discrimination" articles or emails in these sources for our "negative" examples.
After that, we looked at Enron emails and, to make a long story short, found four (4) examples of true risks for employment discrimination. We found them in the subsets for Lay, Kenneth (Ken Lay was the Chairman and CEO of Enron); Derrick, J.; and a few other former Enron employees.
Now, having found four "true risks," we knew that we had something special. Enron is known for fraud, not employment discrimination. And, as far as we know, no one before us had previously surfaced emails that were about "discrimination."
Thus, we had successfully trained our Deep Learning model to "learn" the pattern for "discrimination."
Our third step was the pilot project we can't discuss.
Then, after that "first light" pilot project, we added 10,000 Enron non-discrimination emails to the unrelated set, so the model could “understand” English in the context of emails.
Then we looked at a held-out set of 20,401 Enron emails that our system had never analyzed previously.
Result: Our "model" called out 25 emails as being "related" to discrimination, and our 4 "needles" were among the 25.
That's 25 out of 20,401 emails, a fraction of 0.001225, which is a little less than one-eighth of one percent.
Given that result, we knew we had a very sharp "model" for employment discrimination. (Note: We can create a model for any business-relevant PACER classification.)
But how do we know that any model is any good? The answer is that there is a standard way of visualizing data patterns. The technical name is “t-stochastic neighbor embedding,” but the abbreviation is “t-sne,” and it's pronounced "tee-snee."
In the image below, you’ll see a t-sne visualization. Here, whites = training documents unrelated to discrimination (e. g., news and Wikipedia articles, emails, etc.); while reds (lower left hand corner) = are training documents that are related to discrimination, e.g., the factual allegations in previously filed and publicly available lawsuits).
See the separation between the whites and the reds? That's a clear decision boundary. There are no red documents in the cluster of whites, and no white documents in the cluster of reds. If the colors are mixed, the parameters in the Deep Learning "engine" need to be ajusted to eliminate the mixing. There's some "art" in this science, after all.
Next, when the "model" is asked to assess text in emails, which the system has never seen before, it can "read" the text in each email and indicate whether an email matches up with the pattern of reds, the documents related to discrimination, and very significantly, to what degree.
WHAT'S GOING ON?
The "reds" are documents consisting of factual allegations that were drawn from hundreds of discrimination complaints after they were filed in PACER. It didn't matter who the defendant was. We think of this level of training now as "generic."
Later we realized that we can augment the "generic" training by using factual allegations in discrimination complaints that have been previously filed against a specific company. When we do that, the level of training is "company-specific." If you're in the Law Department of a potential customer, we augment our model with publicly available data about your company.
In addition, our system includes a patented feedback feature. Our software allows a user to accept a "related to the risk" email as a True Positive or reject it as a False Positive.
After there's enough company-specific feedback data like that, we can augment both the positive and negative training sets.
So, currently (and as we go to market), our best model for employment discrimination is a super binary filter. It splits out the emails "related" to the risk from the "unrelated" emails. We show you only the small number of risky emails related to the risk for which the model has been trained.
This filtering makes it possible for a human reviewer to see a relatively small subset of emails "related" to the risk, and then that person splits out the True Positives from the False Positives.
So the human reviewer--a corporate paralegal or attorney--is the Gold Standard. A human decides which high scoring email to escalate to a second reviewer, if need be, and a human decides whether an internal investigation should take place.
And that is why AI here is not frightening in any way. It means Augmented Intelligence.
Moreover, with this new tool at their disposal, corporate counsel will be more valuable to the company than ever before.
So, returning to our analogy of mines below the surface of your ocean of data, now you can see the underwater mines in time for the captain of the ship to take evasive action.
Now we'd like you to see one Power Point slide that depicts the architecture and, in a sense, the work flow of our system. We've covered the training step.
Once your Law Department deploys our system, the work flow begins with your company's emails and ends with a User Interface (UI).
Using the Administrative Console, you can designate who gets an alert about what "use case," and you can schedule an automated review of daily emails, or manually set up a date range for a special selection of emails to review.
Your emails then flow into and through one (or more) of our models and then into the UI Viewer.
Here's the Architecture / work flow slide:
And here's an example of our UI.
For a demo of our UI's other functionalities, please contact Sales for a demo.
What's all this really doing for you?
Now let’s consider a large set of emails.
We remember an attorney for an NYSE company who told us that it was typical for their company to handle two million emails per month.
We were going to analyze emails daily and report "early warnings" on a near real-time basis, so we did the math:
Assume 2,000,000 emails per month; now, when
divided by 4.3 weeks per month, the result is 465,116 emails per week;
and when 465,116 emails per week is divided by 5 days per week, the result is 93,023 emails per day.
At that point, we realized that we were being asked to look at 93,023 emails per day! Yikes.
OK, back to the haystack analogy. If that's the size of the daily haystack, a call for volunteers will be unavailing.
So, understandably, without a way to surface the needles from a haystack that large, no one even bothers to look.
The size of the task makes the work just impossible.
Let's continue the calculation, but only in an informal way. Remember that when we ran our discrimination model against a held-out set of 20,401 Enron emails, it surfaced 25 emails related to the risk, a fraction of about one-eighth of one percent, i.e., 0.0012.
So the number of emails the system would surface as "related to the risk," when presented with 93,023 emails per day, is:
93,023 multiplied by 0.0012 = about 112 emails per day.
Ah ha! Now that's doable.
Our hypothetical is just an illustration, but it shows how to turn the impossible into possible.
Wait. How do we know. We know because, in an "Email Statistics Report, 2011-2015," the Radicati Group reported (at p. 3) that business users sent and received 121 emails per day in 2014 (on average), and expected the number to grow to 140 emails per day in 2018.
So, for a reviewer, 112 emails per day is a slightly below-average amount, and, assuming a 7-hour workday, turns out to be about 16 “related” emails per hour, which is one email about every four (4) minutes.
And if the company is at the projected 2018 level of 140 emails per day, that's 20 emails per hour during a 7-hour workday, which would give each reviewed three (3) minutes per email.
But we can tell you from experience that a reviewer can spot a False Positive in only a few seconds. We provide a "help" here. From the general training set, we built a database of words that are subject-matter related to the risk. We pass the email output through this database and it highlights those words for the reviewers.
Accordingly, for companies generating two million emails per month, it may take only one (1) reviewer a single day to decide which emails (from the day before) to escalate to a second reviewer or initiate an investigation.
But many companies generate more than two million emails per month, but that's no stopper. These models run on Graphics Processing Units (GPUs) and they're not only fast, they run in parallel. Processing capacity and speed are cost issues, but are no longer system limitations.
Thus, with Intraspexion, a risky email might rise to the surface, and be visible to reviewers, only a day or so after it was written.
But the number of reviewers will also depend on how many types of litigation risks a company wants to model.
So the answer to the "other costs" question is an answer often given by attorneys: it depends.
WHY OUR INNOVATION IS SUCH GOOD NEWS
Well, let's start by admitting that neural networks have been around for decades. (For that history, click on "decades.") To make another long story short, there were winters (disappointments) and springs (hope and the hype that goes with it).
But in 2012, Deep Learning (the "street name" for multi-layer neural networks) matured and started producing extraordinary results.
The results were so strong that Andrew Ng--whose resume includes teaching computer science at Stanford, co-founding Coursera, leading Deep Learning teams at Google and Baidu, and more--has said that Deep Learning “is the new electricity,” and that, as such, Deep Learning will be as broadly impactful today as electricity was during the Industrial Revolution.
(Prof. Ng was quoted in an October 2016 Fortune cover story, "Why Deep Learning Is Suddenly Changing Your Life," which you can access by clicking here.)
Thus, with Intraspexion, we’re working with today’s new electricity, and we're the first to use it for the legal profession, which is why we have patents.
Intraspexion is a litigation early warning system for risk, in order to avoid litigation. And who doesn't want less litigation?
Your company is like a ship riding on an ocean of data. We don't want your ship to hit a litigation mine and sink to the bottom. But once a potential lawsuit is identified and our system provides an early warning, then there’s a realistic hope of avoiding that mine.
And that's not hype. For the legal profession, our New Sonar is a new hope.