Law Profs Aim to Predict SCOTUS Outcomes Via Transcript

Two law professors are using their new website to predict Supreme Court case outcomes based on oral arguments.

Oral arguments contain a lot of “unmined” information that provides fodder for analysis, Tonja Jacobi, who launched SCOTUSOA with Matthew Sag, told Bloomberg Law.

That information includes how much each side said relative to the other, and whether this relates to who wins and loses, Jacobi said. Do people who ultimately lose talk more? she asked.

Jacobi is a professor at Northwestern University’s Pritzker School of Law in Chicago. She teaches a class on contemporary Supreme Court cases that examines cases pending before the court.

Data Dump

Jacobi and Sag spent a lot of time “cleaning up data” from every oral argument transcript since 1955, Jacobi said.

Transcripts don’t come “neatly pre-packaged” so the two eliminated transcript errors like spelling mistakes, then combined the transcript data from each case with other information about the case like how each justice voted and how the lower court ruled, Sag said in an email.

Sag is a professor at Loyola University Chicago whose research focuses on the intersection of law and technology.

Transcript data includes information like how many words a particular judge said to a particular advocate, and the number of interruptions or questions asked in each case, Sag said.

This was then all combined with information about the justices, like year of appointment, whether appointed by Democrat or Republican, gender, and different measures of ideology he said.

Their posts so far have looked at overall trends like politeness in the courtroom, measured by the tendency of a justice to use terms like “I’m sorry” and “excuse me,” Jacobi said.

One finding was that judicial politeness at oral argument has increased since 2005.

They’ve also developed a variety of metrics of judicial activity—words used, duration of speech, interruptions made, whom they spoke to most, and comments made—and of what Jacobi called judicial advocacy—how often and when, for example, the justices present their comments and conclusions to the attorneys, rather than asking them questions.

They quantify and use these metrics to predict case outcomes, Jacobi said.

Jacobi and Sag plan to match these statistics to impressionistic accounts of the cases, including recordings of the arguments, to forecast how the justices will rule.

New Justice Hazing?

The new term will likely usher in a new justice and the site will definitely include “some analysis” of that person’s activity during argument, Jacobi said.

They plan to look at the justice’s use of polite questions, and how many questions and comments the new justice poses, she said.

But the earlier in a justice’s tenure, the less reliable the analysis is, Jacobi cautioned.

Their analysis of Justice Neil M. Gorsuch’s behavior contains caveats to that extent, she said.

Linguistic habits, for example, change over time as a justice settles in to the role, Jacobi said.

Justices tend to become more active in questioning and women justices reduce their use of polite language, she said. Jacobi believes this is because they are interrupted more frequently when they preface a comment or question with a phrase such as “excuse me.”

Jacobi admits there might be some aspects of arguments they miss out on—like facial expressions and reactions—by not being present, but “we’re not short on ideas” for trends to analyze, she said.

To contact the reporter on this story: Melissa Heelan Stanzione in Washington at mstanzione@bloomberglaw.com

To contact the editor responsible for this story: Jessie Kokrda Kamens at jkamens@bloomberglaw.com