Trial analytics

Using math to improve trial outcomes

2024 January

You might be familiar with the best-selling book Moneyball: The Art of Winning an Unfair Game (Michael Lewis 2004), and the subsequent Oscar-nominated movie adaptation. Moneyball tells the story of how 20 years ago, the Oakland Athletics, at the time a low-budget baseball team with a little-known team roster, pioneered a mathematical approach to team management and game strategy, defying conventional wisdom and propelling the team to a record-setting winning streak. Since then, the field of sports analytics has become a billion-dollar industry. Most professional sports teams now employ mathematicians and data analysts to guide player hiring, game strategy, field decisions, and even ticket sales. Success in sports is no longer dominated by the skill of individual players and long gone are the days of coaches and managers making important decisions based on hunches, intuition or “gut feeling.”

The similarities between professional- sports games and courtroom trials are many. On the playing field and in the courtroom, opposing teams attempt to prevail in a contest of strategy and skill. Both take place within a framework of rules and procedures that teams attempt to use to their benefit. Yet, while sports teams have adopted rigorous scientific strategies, litigators still typically approach many important aspects of the trial based on intuition and educated guesswork. A science-based paradigm shift has not taken hold in the courtroom.

There is, perhaps, good reason for this. For example, in the case of jury selection, the so-called scientific jury selection (SJS) method has not proven to be any more effective at determining juror favorability than the traditional intuitive approach. However, SJS addresses only one part of jury selection, namely using psychology to predict what potential jurors will do when it comes time to reach a verdict.

Even if such predictions could be made with certainty, there is still a path that must be navigated, from questionnaire analysis to voir dire to peremptory challenges to potential Batson challenges, to obtain the most favorable possible jury. Identifying potentially favorable and unfavorable jurors is only one step in the process. Trial analytics encompasses a broad range of methods and procedures that can help litigators navigate various courtroom procedures. In the case of jury selection, trial analytics addresses not just what jurors might do, but rather, what lawyers should do to optimize trial outcomes.

Trial analytics identifies and targets those aspects of litigation that are candidates for rigorous mathematical analysis. While this article focuses on jury selection, I have no doubt that trial analytics will be found applicable to other areas of litigation as well. Armed with both trial analytics and traditional experience-based approaches, lawyers will be well-positioned for optimal trial outcomes. This article will provide some example applications of trial analytics.

Questionnaire evaluation using cluster analysis

Juror questionnaires provide preliminary evaluations regarding the favorability of potential jurors. Questionnaires can identify areas of potential bias and provide avenues of follow-up during voir dire. Typically, when provided with completed juror questionnaires, the attorney examines them individually, going through the stack one by one, and rates the likely favorability of each juror based on an overall impression of their questionnaire responses.

Ratings are commonly made on a scale of 1 to 10 with higher numbers indicating more favorable potential jurors. The attorney may also make summary notes about the juror and identify areas for follow-up. The stack of questionnaires is sometimes divided among several evaluators to speed up the process.

The initial questionnaire-based ratings may be adjusted up or down when evaluators have a chance to compare notes, after voir dire questioning, and after conducting external research. However, not every potential juror in the venire will be flagged for further investigation and most will retain their initial questionnaire-based ratings. At the same time, many lawyers believe, and studies show, that if a juror provides conflicting or ambiguous information during voir dire, the initial written questionnaire response is the one that most accurately reflects the juror’s attitudes and beliefs. Accurate and reliable questionnaire evaluation is therefore critical to successful jury selection.

There are several pitfalls to this traditional questionnaire-evaluation method. Evaluators may be biased either consciously or subconsciously, leading to invalid rating assignments. Different evaluators may evaluate responses differently. Even a single evaluator may perceive similar responses differently at different times. Questionnaire evaluation can be time-consuming and resource intensive. Often, completed questionnaires are provided to the parties only a day or two before the start of voir dire, resulting in a rushed or incomplete evaluation process. Overall, the traditional one-by-one approach represents an inefficient use of the legal team’s valuable time and, unless extreme care is taken, risks invalid, inconsistent, or incomplete results.

Cluster analysis

Luckily, several efficient and robust questionnaire-evaluation methods have been developed for the social sciences and some of these can be applied directly to jury-questionnaire evaluation.  One of these methods, called cluster analysis, first uses a computer-based algorithm to group or cluster jurors according to the similarity of their questionnaire responses. Each group is then considered as a whole by the evaluator and given a rating. Each group member then receives this same rating. Jurors with unusual or outlying responses can be automatically flagged for further investigation. This method ensures that similar jurors receive similar ratings. The groups can be formed by computer algorithm in a matter of seconds, drastically reducing the amount of time a lawyer would otherwise spend reading, evaluating, and rating individual questionnaires. The lawyer need only evaluate the representative characteristics of a small number of groups, significantly reducing workload. Of course, the lawyer is free to scan each group for outliers and adjust ratings appropriately.

Both the traditional one-by-one approach and the cluster analysis method result in a partitioning of the venire into groups according to juror favorability. However, the traditional approach is more time-consuming and error prone. The cluster analysis approach allows most of the busy work to be done up front automatically and requires specialized legal knowledge only at the end of the process when assigning ratings to the resulting groups. This method does, however, require questionnaire responses to be in electronic format. If the questionnaires are provided in paper format, they will have to be trans-coded into electronic format, for example using an Excel spreadsheet. This can be done relatively quickly by staff and requires little legal training or knowledge.

Peremptory challenges and game theory

The exercise of peremptory challenges is typically done by first identifying a small number of the least favorable prospective jurors. It is decided beforehand that if such a juror appears in the jury box, he or she will be challenged. More experienced litigators may think ahead a few moves and factor in the favorability of the replacement juror, what the likelihood is that the opposing party will strike this replacement, who the second replacement will be, and so on. Litigators may also factor in how many challenges remain, whether they should be conserving challenges for use against unfavorable jurors later in the order, and what the likelihood is that the opposing party may themselves strike one of these least favorable jurors. These experienced litigators are, perhaps unwittingly, using a rudimentary form of mathematical analysis called game theory.

Game theory is the mathematical study of competition and cooperation among players who participate in a game (in the most general sense) and who share a common outcome. Game theory is designed to optimize decisions made by each player during the game. Game theory is commonly used in economics, international relations, military planning, and a host of other disciplines. In jury selection, the players are the opposing litigants, the common outcome is the seated jury, and the decisions are whether and when to exercise peremptory challenges.

There is a long record of academic research on the use of game theory in jury selection. Simulations have shown that game theory can provide an advantage equivalent to having two to four more peremptory challenges, depending on the sophistication of the opposing party. If a litigant is using game theory, but their opponent is not, then the litigant can achieve a significant advantage in the ultimate value of the seated jury.

Game theory calculations are based on a so-called game tree, which maps every possible decision that could be made during the jury selection process. A decision point in the tree is called a node. At each node, either the plaintiff or defendant can choose to exercise a peremptory challenge or to pass. For a twelve-person jury, each node therefore presents thirteen possible choices. Given a set of juror ratings, the value of each choice can be calculated, based on the values of subsequent nodes further down the game tree. The best choice (challenge a juror or pass) is the one that maximizes the jury value for the choosing party.

Clearly, jury selection game trees are large, comprising tens or hundreds of millions of nodes. Like a game of chess, even the best players can only think a few moves ahead. After that, the possibilities become too complex. Often, what seems like a good decision leads to unforeseen negative consequences further down the road. Game theory allows for deeper analysis of the consequences of each choice. Often, counterintuitive choices end up being the best.

Unfortunately, game theory has not been seen as practical for courtroom use. Game theory calculations are computer intensive and could not, until recently, be done in real time during fast-paced courtroom jury selection. This obstacle has now been overcome due to improvements in laptop computing power and with the availability of specialized jury selection software. With these advances, game theory can be used effectively in the courtroom when selecting jurors for peremptory challenges.

Batson challenges and probability theory

The party raising a Batson (or similar) challenge must first make a prima facie argument that the opposing party has used peremptory challenges in an impermissible, prejudicial manner. The party must (a) identify a cognizable group and (b) show a pattern of discrimination against this group. The latter is generally considered to be the more difficult task. Noting a disproportionate number of challenges against a cognizable group may be sufficient to establish a prima facie claim of discrimination. However, this begs the question: What number of strikes against a group is disproportionate?

A simple coin toss illustrates the situation: If the coin is fair, then we expect that about 50% of the time we would get heads and 50% tails. Suppose we find that after 10 tosses, we get 10 heads and zero tails. Most people would consider this a disproportionate number, and we could make a prima facie argument that the coin is biased in favor of heads. What if the result is eight heads out of 10 tosses? What about seven heads? What number of heads or tails should we consider disproportionate?

Statisticians provide an answer to this question in terms of a quantity called the p-value. The p-value tells us the probability of an event occurring if the process is unbiased. Given an observation, for example the number of heads in a set of coin tosses, or the number of peremptory strikes against a cognizable group in a venire, the p-value tells us whether some form of bias is likely at play. If the p-value is too small, say less than 0.1, then we can make a mathematical argument that bias is present. Surely, a prima facie argument can, and arguably should, be based on math.

Calculation of p-values is not difficult and someone familiar with the basics of probability theory should be able to calculate the p-value for any given jury selection situation. As an example, suppose a venire of 18 potential jurors contains a cognizable group of six members. The opposing party strikes four of the six group members. The p-value for this situation is p = 0.057. Since this is smaller than 0.1, bias is likely at play.

On the other hand, the p-value for striking only three members of the cognizable group is p = 0.29, which is large enough that the strikes against the group could be the result of random chance, and not purposeful bias. In my consulting business, I would recommend a Batson challenge in the former case, but not in the latter. Specialized computer software is now available that can calculate p-values in real time during the jury selection process. The software can alert the lawyer when the opposing party is likely biased in their use of peremptory challenges.

Conclusion

The skill, experience and intuition of trial lawyers is central to prevailing in the courtroom. However, there are aspects of litigation where a mathematical approach can provide significant benefits. Incorporating math into legal processes may seem foreign to many litigators who are not generally trained in high-level mathematics. On the other hand, it is common practice in high-stakes cases to hire consultants for a range of activities including juror evaluation, legal research, conducting focus groups, mock trials, preparing presentation graphics, questionnaire preparation, witness preparation, and more. Why not employ consultants to help implement rigorous science-based methods that have proven advantages?

The above examples of mathematical procedures applied to jury selection can reduce the lawyer’s workload and at the same time are likely to produce superior trial outcomes. Much like the discovery of sports analytics 20 years ago, I believe that trial litigation is ready for a trial analytics paradigm shift.