The battle between man and (AI-powered) machine has been a long, ongoing one. Think Garry Kasparov vs. IBM’s Deep Blue, Jeopardy contestants vs. Watson, Dave vs. the HAL 9000. And now a new skirmish has recently concluded, after a host of British solicitors faced off against an artificial intelligence program with access to a law library.
Last week, over a hundred of London’s finest lawyers took part in a week-long challenge against predictive technology to see who could best forecast the outcome of legal problems. The winner? The machines.
Robot Lawyers Win—This Time
The competition was organized by the UK legaltech company CaseCrunch, whose mission is to “solve law” through the use of data science and legal prediction systems. The challenge saw 112 lawyers pit their predictive skills against CaseCrunch, including, some attorneys from the UK’s most prestigious firms. According to the Global Legal Post, participants included:
Magic Circle partners, barristers, and in-house counsel with participating law firms including Bird & Bird, Kennedys, Weightmans, Allen & Overy, Berwin Leighton Paisner, DLA Piper, DAC Beachcroft, DLA Piper, and more. Teams were entered from large firms including Pinsent Masons and Eversheds Sutherlands.
The attorneys were given scenarios taken from real cases decided by the Financial Ombudsman Service, a UK authority that deals with consumer complaints involving banks, insurers, and the like. CaseCrunch explains:
They were presented with factual scenarios of PPI mis-selling claims, and asked to predict “yes or no” as to whether the Financial Ombudsman would uphold the claim. The same factual scenarios were given to CaseCrunch: whoever had the highest accuracy, won. 775 predictions were submitted by the participants.
A legal judge and a technology judge verified the fairness of the competition.The lawyers were "unsupervised," meaning they could use any legal research tools available to help predict how a case would be resolved.
Attorneys predicted the correct outcome 62.3 percent of the time. CaseCruncher Alpha, the company's AI contestant, had a validation accuracy of 86.6 percent.
What made technology come out so far ahead here? “The main reason for the large winning margin,” CaseCruncher explains, “seems to be that the network had a better grasp of the importance of non-legal factors than lawyers.” That is, lawyers know the law. Computers, or at least this computer program, was able to master the law and other factors, too. This time, at least.
The End of Flesh-Based Lawyering?
The company cautions that “evaluating these results is tricky,” however.
These results do not mean that machines are generally better at predicting outcomes than human lawyers. These results show that if the question is defined precisely (such as - was this complaint about PPI mis-selling upheld or rejected by the FOS?), machines are able to compete with and sometimes outperform human lawyers.
So, will AI-powered “robot lawyers” replace attorneys in the meatspace? No, of course not. Well, at least not all of them. There have been great failures when it comes to applying tech to law, and even the most sophisticated legal technology only eliminates a small part of the total work real lawyers can accomplish. But for some tasks, the machines seem to be winning.
And that’s a good thing, as attorneys can now leverage new technology to improve services and accomplish more—benefiting both lawyers, clients, and possibly even robots.
This post was authored by Casey C. Sullivan, who leads education and awareness efforts at Logikcull. You can reach him at casey.sullivan@logikcull.com or on Twitter at @caseycsull.
