Resources

A new study attempts to measure how knowledge gleaned from Wikipedia may play out in one specific realm: the courts.

A team of researchers led by Neil Thompson, a research scientist at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), recently came up with a friendly experiment: creating new legal Wikipedia articles to examine how they affect the legal decisions of judges. They set off by developing over 150 new Wikipedia articles on Irish Supreme Court decisions, written by law students. Half of these were randomly chosen to be uploaded online, where they could be used by judges, clerks, lawyers, and so on — the “treatment” group. The other half were kept offline, and this second group of cases provided the counterfactual basis of what would happen to a case absent a Wikipedia article about it (the “control”). They then looked at two measures: whether the cases were more likely to be cited as precedents by subsequent judicial decisions, and whether the argumentation in court judgments echoed the linguistic content of the new Wikipedia pages.

It turned out the published articles tipped the scales: Getting a public Wikipedia article increased a case’s citations by more than 20 percent. The increase was statistically significant, and the effect was particularly strong for cases that supported the argument the citing judge was making in their decision (but not the converse). Unsurprisingly, the increase was bigger for citations by lower courts — the High Court — and mostly absent for citations by appellate courts — the Supreme Court and Court of Appeal. The researchers suspect this is showing that Wikipedia is used more by judges or clerks who have a heavier workload, for whom the convenience of Wikipedia offers a greater attraction.
“To our knowledge, this is the first randomized field experiment that investigates the influence of legal sources on judicial behavior. And because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation,” says Thompson, the lead author of the study. “The fact that we wrote up all these cases, but the only ones that ended up on Wikipedia were those that won the proverbial ‘coin flip,’ allows us to show that Wikipedia is influencing both what judges cite and how they write up their decisions.”

“Our results also highlight an important public policy issue,” Thompson adds. “With a source that is as widely used as Wikipedia, we want to make sure we are building institutions to ensure that the information is of the highest quality. The finding that judges or their staffs are using Wikipedia is a much bigger worry if the information they find there isn’t reliable.”

The paper describing the study has been published in ” The Cambridge Handbook of Experimental Jurisprudence.”

On May 31 last year, 25-year-old Safarain Herring was shot in the head and dropped off at St. Bernard Hospital in Chicago by a man named Michael Williams. He died two days later. Chicago police eventually arrested the 64-year-old Williams and charged him with murder (Williams maintains that Herring was hit in a drive-by shooting). A key piece of evidence in the case is video surveillance footage showing Williams’ car stopped on the 6300 block of South Stony Island Avenue at 11:46 p.m. – the time and location where police say they know Herring was shot. How did they know that’s where the shooting happened? Police said ShotSpotter, a surveillance system that uses hidden microphone sensors to detect the sound and location of gunshots, generated an alert for that time and place. Except that’s not entirely true, according to recent court filings.

That night, 19 ShotSpotter sensors detected a percussive sound at 11:46 p.m. and determined the location to be 5700 South Lake Shore Drive – a mile away from the site where prosecutors say Williams committed the murder, according to a motion filed by Williams’ public defender. The company’s algorithms initially classified the sound as a firework. That weekend had seen widespread protests in Chicago in response to George Floyd’s murder, and some of those protesting lit fireworks. But after the 11:46 p.m. alert came in, a ShotSpotter analyst manually overrode the algorithms and “reclassified” the sound as a gunshot. Then, months later and after “post-processing,” another ShotSpotter analyst changed the alert’s coordinates to a location on South Stony Island Drive near where Williams’ car was seen on camera. “Through this human-involved method, the ShotSpotter output in this case was dramatically transformed from data that did not support criminal charges of any kind to data that now forms the centerpiece of the prosecution’s murder case against Mr. Williams,” the public defender wrote in the motion.

The document is what’s known as a Frye motion – a request for a judge to examine and rule on whether a particular forensic method is scientifically valid enough to be entered as evidence. Rather than defend ShotSpotter’s technology and its employees’ actions in a Frye hearing, the prosecutors withdrew all ShotSpotter evidence against Williams. The case isn’t an anomaly, and the pattern it represents could have huge ramifications for ShotSpotter in Chicago, where the technology generates an average of 21,000 alerts each year. The technology is also currently in use in more than 100 cities. Motherboard’s review of court documents from the Williams case and other trials in Chicago and New York State, including testimony from ShotSpotter’s favored expert witness, suggests that the company’s analysts frequently modify alerts at the request of police departments – some of which appear to be grasping for evidence that supports their narrative of events.

The Department of Justice has been dismissing child pornography cases in order to not reveal information about the software programs used as the basis for the charges. An array of cases suggest serious problems with the tech tools used by federal authorities. But the private entities who developed these tools won’t submit them for independent inspection or hand over hardly any information about how they work, their error rates, or other critical information. As a result, potentially innocent people are being smeared as pedophiles and prosecuted as child porn collectors, while potentially guilty people are going free so these companies can protect “trade secrets.” The situation suggests some of the many problems that can arise around public-private partnerships in catching criminals and the secretive digital surveillance software that it entails (software that’s being employed for far more than catching child predators).

With the child pornography cases, “the defendants are hardly the most sympathetic,” notes Tim Cushing at Techdirt. Yet that’s all the more reason why the government’s antics here are disturbing. Either the feds initially brought bad cases against people whom they just didn’t think would fight back, or they’re willing to let bad behavior go rather than face some public scrutiny. An extensive investigation by ProPublica “found more than a dozen cases since 2011 that were dismissed either because of challenges to the software’s findings, or the refusal by the government or the maker to share the computer programs with defense attorneys, or both,” writes Jack Gillum. Many more cases raised issues with the software as a defense. “Defense attorneys have long complained that the government’s secrecy claims may hamstring suspects seeking to prove that the software wrongly identified them,” notes Gillum. “But the growing success of their counterattack is also raising concerns that, by questioning the software used by investigators, some who trade in child pornography can avoid punishment.”

“The UK Parliament caused a bit of a stir this week with the news that it would play host to its first non-human witness,” reports The Verge. “A press release from one of Parliament’s select committees (groups of MPs who investigate an issue and report back to their peers) said it had invited Pepper the robot to ‘answer questions’ on the impact of AI on the labor market.” From the report:

“Pepper is part of an international research project developing the world’s first culturally aware robots aimed at assisting with care for older people,” said the release from the Education Committee. “The Committee will hear about her work [and] what role increased automation and robotics might play in the workplace and classroom of the future.” It is, of course, a stunt.

As a number of AI and robotics researchers pointed out on Twitter, Pepper the robot is incapable of giving such evidence. It can certainly deliver a speech the same way Alexa can read out the news, but it can’t formulate ideas itself. As one researcher told MIT Technology Review, “Modern robots are not intelligent and so can’t testify in any meaningful way.” Parliament knows this. In an email to The Verge, a media officer for the Education Committee confirmed that Pepper would be providing preprogrammed answers written by robotics researchers from Middlesex University, who are also testifying on the same panel. “It will be clear on the day that Pepper’s responses are not spontaneous,” said the spokesperson. “Having Pepper appear before the Committee and the chance to question the witnesses will provide an opportunity for members to explore both the potential and limitations of such technology and the capabilities of robots.”

MP Robert Halfon, the committee’s chair, told education news site TES that inviting Pepper was “not about someone bringing an electronic toy robot and doing a demonstration” but showing the “potential of robotics and artificial intelligence.” He added: “If we’ve got the march of the robots, we perhaps need the march of the robots to our select committee to give evidence.”