Algorithmic Accountability in Judicial Decisions

Algorithmic Accountability in Judicial Decisions: How Courts Are Reviewing AI‑Assisted Sentencing and Risk Assessments

Algorithmic fairness and due process challenges in modern courtrooms

Meta Summary: This playbook examines how courts worldwide are reviewing algorithmic risk assessments and AI‑assisted sentencing tools. Structured for legal practitioners, policymakers, court administrators, and advocacy organisations, it covers foundational concepts, landmark case law, regulatory frameworks, human rights implications, and practical strategies for ensuring algorithmic accountability in judicial decision‑making.

Table of Contents

• Chapter 1: Foundations – Algorithmic Risk Assessment in Criminal Justice
• Chapter 2: Landmark Case Law – From State v. Loomis to Thomas v. Montgomery
• Chapter 3: The EU AI Act and Global Regulatory Responses
• Chapter 4: Human Rights and Algorithmic Accountability
• Chapter 5: Strategies for Ensuring Fairness and Due Process
• Related Topics
• FAQ
• References

Chapter 1: Foundations – Algorithmic Risk Assessment in Criminal Justice

⬅ Back to Table of Contents

Introduction – The Rise of Algorithmic Tools in Sentencing

Algorithmic risk assessment tools have become increasingly common in criminal justice systems across the United States and other jurisdictions. These instruments, which include proprietary systems like COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), STRONG‑R, and various pretrial risk assessment algorithms, are designed to predict the likelihood that an individual will reoffend (recidivism) or fail to appear for court proceedings. They aim to assist judges, parole boards, and probation officers in making more consistent and evidence‑based decisions at various stages of the criminal process.

Risk assessment tools are typically used at three critical junctures. First, during pretrial release decisions, algorithms analyse factors such as criminal history, employment status, and community ties to recommend whether a defendant should be detained or released pending trial. Second, during sentencing, presentence investigation reports frequently include recidivism risk scores that may influence the length or type of sentence imposed. Third, during parole and probation determinations, risk assessments help guide decisions about supervision levels, treatment placements, and early release eligibility.

Calls for algorithmic decision‑making in the criminal justice system gained momentum with a 2007 resolution from the Conference of Chief Justices, which encouraged state courts to adopt evidence‑based practices to reduce recidivism and promote public safety. Shortly thereafter, the American Bar Association also urged states to incorporate risk assessment tools into their sentencing and corrections frameworks.

Key Concepts – How Risk Assessment Tools Work

• Risk Assessment Algorithm (RAI): A computational model that processes input data about an individual—often including criminal history, age, employment status, family background, substance abuse history, and responses to standardised questionnaires—to produce a numerical score or risk category (e.g., low, medium, or high risk of recidivism).
• Recidivism Prediction: The algorithmic estimate of the probability that a defendant will commit another crime if released. This prediction serves as the primary output of many sentencing and pretrial risk assessment tools.
• Actuarial Justice Model: A framework that uses statistical and algorithmic methods to manage offender populations, based on the idea that risk can be calculated and managed through data‑driven classification systems, rather than through individualised judicial discretion.
• Predictive Validity: The statistical measure of how accurately a risk assessment tool predicts future behaviour. High predictive validity indicates that scores consistently correlate with actual recidivism outcomes across the population on which the tool is validated.
• Algorithmic Bias: Systematic and unfair disparities in algorithmic outcomes across demographic groups, often reflecting historical patterns of discrimination embedded in training data or model design. Studies have demonstrated that some risk assessment tools produce higher false‑positive rates for Black defendants compared to white defendants with similar recidivism profiles.
• Algorithmic Opacity (Black Box Problem): The inability of defendants, defence counsel, judges, or even independent auditors to examine how a proprietary risk assessment tool processes inputs and arrives at its outputs. This lack of transparency can violate fundamental due process rights to challenge the evidence used against an accused person.

The ProPublica COMPAS Investigation – Documenting Algorithmic Bias

In May 2016, ProPublica published a landmark investigative study examining the performance of the COMPAS risk assessment tool in Broward County, Florida. The analysis examined risk scores assigned to more than 7,000 people in Florida’s Broward County in 2013 and 2014. The investigation revealed significant racial disparities: Black defendants were nearly twice as likely as white defendants to be incorrectly classified as medium or high risk for reoffending, while white defendants were more likely to be incorrectly labelled as low risk despite having similar recidivism profiles.

The study demonstrated that COMPAS labelled Black defendants as high risk at roughly twice the rate of white defendants, despite equivalent reoffending rates. This bias was rooted in the algorithm’s reliance on historical data that reflected systemic inequalities in policing, arrest, and conviction patterns, thereby perpetuating discrimination through automated decision‑making processes. The defendants had to answer a questionnaire asking about employment, education, family background, friends and crime in their neighbourhood. These variables, when combined, led to racial bias according to scholars.

A Superior Court Judge commenting on COMPAS observed: “A guy who has molested a small child every day for a year could still come out as a low risk because he probably has a job. Meanwhile, a drunk guy will look high risk because he’s homeless.” The ProPublica investigation sparked an international debate about algorithmic bias and trust in AI‑driven judicial tools, directly influencing the development of risk‑based regulatory frameworks such as the EU AI Act and legislative proposals in multiple jurisdictions.

Read ProPublica’s investigation (ProPublica, 2016)

Chapter 2: Landmark Case Law – From State v. Loomis to Thomas v. Montgomery

⬅ Back to Table of Contents

State v. Loomis (Wisconsin Supreme Court, 2016) – The COMPAS Precedent

State v. Eric L. Loomis remains the most influential judicial decision addressing the use of algorithmic risk assessments in criminal sentencing. Eric Loomis was charged with five criminal counts related to a drive‑by shooting in La Crosse, Wisconsin. He pleaded guilty to two lesser charges and, during the sentencing phase, a Wisconsin Department of Corrections officer produced a Presentence Investigation Report (PSI) that included a COMPAS risk assessment. The trial court referred to the COMPAS assessment in its sentencing determination and, based in part on this assessment, sentenced Loomis to six years of imprisonment and five years of extended supervision.

Loomis appealed, arguing that the trial court’s consideration of the COMPAS risk assessment at sentencing violated his right to due process for three principal reasons. First, the proprietary nature of the COMPAS algorithm prevented him from examining how the risk score was calculated, thereby denying him the opportunity to challenge the accuracy or validity of the evidence against him. Second, the COMPAS assessment relied on gendered variables (such as “criminal associates” and “substance abuse”) that may have introduced statistical biases. Third, the use of risk assessment tools for sentencing purposes exceeded the tool’s original design intent, which was to allocate correctional programming resources rather than to determine punishment severity.

The Wisconsin Supreme Court unanimously affirmed the lower court’s decision. The Court held that the use of the COMPAS risk assessment at sentencing did not violate the defendant’s due process rights, even though the methodology used to produce the assessment was disclosed neither to the court nor to the defendant. The Court acknowledged that the COMPAS assessment was based largely on criminal history, along with variables such as criminal associates and substance abuse. The Court reasoned that risk assessment tools “continue to change and evolve” and that future research might address the concerns raised.

In an attempt to provide procedural safeguards, the Court crafted a series of “written advisement” requirements to accompany PSIs that include risk assessments. These advisements were intended to inform judges that COMPAS assessments are not designed to predict individual recidivism with certainty, that they may rely on gendered variables, and that proprietary algorithms cannot be examined by the defendant. However, legal scholars have critiqued this remedy as insufficient, arguing that the advisements are “unlikely to create meaningful judicial skepticism because they are silent on the strength of the criticisms of these assessments.”

Loomis subsequently petitioned the United States Supreme Court for review. The Acting Solicitor General was invited to file a brief expressing the views of the United States, but the Supreme Court ultimately declined to hear the case. Consequently, State v. Loomis remains the controlling precedent in Wisconsin and has been cited by courts in other jurisdictions as persuasive authority for the limited use of proprietary algorithmic risk assessments at sentencing.

Read State v. Loomis (FindLaw)

Harvard Law Review comment on State v. Loomis

Loomis v. Wisconsin (SCOTUSblog case file)

Thomas v. Montgomery (Sixth Circuit, 2025) – Algorithmic Risk Assessments and Parole

Thomas v. Montgomery represents a significant recent development in algorithmic accountability litigation, addressing the use of proprietary risk‑assessment tools in parole determinations. Two Tennessee inmates, Carvin L. Thomas and Terrell Lawrence, challenged the State’s reliance on the STRONG‑R risk‑assessment tool, alleging that it yielded inaccurate and opaque scores, thereby depriving them of liberty without due process of law.

The plaintiffs argued that the STRONG‑R assessments violated their procedural due process rights under the Fourteenth Amendment. They contended that the tool’s proprietary nature prevented meaningful challenge to its validity and that the alleged inaccuracies in the risk scores could affect parole outcomes. The district court dismissed the complaint under Federal Rule of Civil Procedure 12(b)(6), holding that Tennessee’s statutory scheme does not create a constitutionally protected liberty interest in parole.

The U.S. Court of Appeals for the Sixth Circuit affirmed the dismissal. The court held that even after the 2021 Reentry Success Act and despite the pervasive role of STRONG‑R, Tennessee’s statutory scheme does not create a constitutionally protected liberty interest in parole because the governing statutes use discretionary language (“may be paroled” and “shall deny”) rather than mandatory grant language. The court distinguished Montana’s statute, which included “shall release” wording and was found to create a liberty interest in Board of Pardons v. Allen (1987).

A critical holding of Thomas v. Montgomery is that alleged inaccuracies and secrecy surrounding a risk assessment tool cannot trigger due‑process protections in the absence of a cognizable liberty interest. This decision creates a significant procedural barrier for prisoners seeking to challenge algorithmic risk assessments in parole contexts. The court relied on the Supreme Court’s framework from Greenholtz v. Inmates of Neb. Penal & Corr. Complex (1979), which held that prisoners have no inherent federal constitutional right to parole but that a state can create a liberty interest through mandatory statutory language.

This case illustrates how the structure of state parole systems can insulate algorithmic risk assessments from constitutional scrutiny. Where statutes confer broad discretion on parole boards, as in Tennessee, even demonstrably flawed or biased algorithmic tools may evade judicial review because no protected liberty interest attaches to the parole decision. The decision has been noted by scholars for underscoring the need for legislative reform to ensure algorithmic accountability in discretionary criminal justice settings.

Read Thomas v. Montgomery analysis (Casemine)

International Cases – Canada, Russia, and Emerging Jurisprudence

Beyond the United States, courts in other jurisdictions are beginning to grapple with questions of algorithmic accountability in judicial decision‑making. These cases highlight both the global reach of AI‑assisted justice and the diverse legal frameworks through which accountability is pursued.

R. v. Chand (Ontario Court of Justice, 2025). This Canadian case addressed not algorithmic risk assessments themselves, but the use of generative AI in legal submissions. Justice Joseph F. Kenkel identified serious issues with defence submissions, including fictitious citations, case law that did not support the points cited, and unrelated civil cases. The court ordered that generative AI must not be used for legal research for submissions, issued a categorical prohibition on AI‑assisted legal research in that matter, and required numbered paragraphs, hyperlinked citations, and manual verification of all references. This case illustrates the emerging phenomenon of “AI hallucinations” in legal practice and the judicial response to unreliable AI‑generated content.

Ko v. Li (Ontario Superior Court, 2025). In a further Canadian development, Justice Fred Myers referred a lawyer to the Attorney General of Ontario for criminal contempt of court proceedings, marking the first Canadian instance in which misuse of generative AI moved from a professional negligence issue to a criminal obstruction of justice. This case signals that courts are increasingly willing to impose serious sanctions when AI is used to mislead proceedings or fabricate evidence.

Russian Federation – Eisk City Court (Krasnodar, 2025). In September 2025, a regional federal court in the southern Russian province of Krasnodar declined to change a sentence challenged because the sentencing decision contained fragments written by a large language model. The defence attorney submitted a linguistic analysis confirming that five fragments of the sentencing decision showed obvious signs of AI generation, including grammatical and stylistic errors, missing logical conclusions, and vocabulary not common in legal documents. The defence argued that the use of AI eliminates the personal responsibility of judges for their decisions, violating the Federal Law on the Status of Judges in the Russian Federation. The appellate court nonetheless left the sentence unchanged, ruling that the writing style selected by the court for drafting the sentence did not contradict criminal procedural legislation. The Russian legal community continues to debate the issue, and the federal bar association has noted that judges’ use of generative AI may cause their writing and analytical skills to atrophy.

These international cases demonstrate that algorithmic accountability in judicial decision‑making is a global concern, but the legal responses vary significantly across jurisdictions. Canadian courts have taken a cautiously restrictive approach to generative AI in legal submissions. Russian courts, in contrast, have permitted AI‑generated sections of judicial decisions despite clear violations of judicial responsibility norms. No international consensus has yet emerged on the appropriate standards for judicial use of AI.

Read R. v. Chand analysis (Law360 Canada, 2025)

Read Russian Federation AI ruling (Library of Congress Global Legal Monitor, 2025)

Chapter 3: The EU AI Act and Global Regulatory Responses

⬅ Back to Table of Contents

The EU AI Act – High‑Risk Classification for Judicial AI Systems

The European Union’s Artificial Intelligence Act (AIA) represents the world’s first comprehensive horizontal regulation of artificial intelligence. Adopted by the European Parliament on 13 March 2024 and entering into force on 1 August 2024, the AIA adopts a risk‑based approach that imposes increasingly stringent obligations on AI systems based on the level of risk they pose to health, safety, and fundamental rights.

Under the AIA, AI systems used in judicial and democratic processes are explicitly classified as “high‑risk” AI systems. Article 6 of the AI Act, together with Annex III, lists high‑risk AI systems as including those “assisting judicial authorities with researching and interpreting facts and the law and applying the law to a set of facts.” This classification means that any AI system used to support judicial decision‑making—including recidivism risk assessments, pretrial risk tools, and sentencing recommendation algorithms—must comply with the AIA’s extensive requirements for high‑risk systems.

These requirements include the following. A robust risk management system under Article 9 must be established. High standards for data quality and governance, under Article 10, are required to ensure training and operational data are relevant, representative, free of errors, and complete. Thorough technical documentation under Article 11 and effective record‑keeping under Article 12 must be maintained throughout the system’s lifecycle. Transparency and adequate information to deployers under Article 13, including clear instructions for human oversight, are mandatory. Human oversight measures under Article 14 must be implemented to ensure that final decision‑making remains a human‑driven activity. Accuracy, robustness, and cybersecurity under Article 15 must be guaranteed.

Article 10 (data governance) and Article 14 (human oversight) are considered particularly important for mitigating bias in AI systems like COMPAS. Legal scholars have explicitly connected the COMPAS case to these articles, and the European Commission’s Impact Assessment accompanying the AIA referenced the COMPAS case as an example where claims of discrimination have already led to pressure from public opinion. The Impact Assessment states that discriminatory AI systems used in judiciary or law enforcement may “lead to broader societal consequences, reinforcing existing or creating new forms of structural discrimination and exclusion.”

A notable limitation is found in Article 6(3) of the AIA, which allows providers of some AI systems that appear to be high‑risk to unilaterally decide that their system does not pose a significant risk to fundamental rights, thereby exempting themselves from high‑risk requirements. This self‑classification provision has been criticised as creating a potential loophole that could undermine the AIA’s protective objectives.

Read analysis of high‑risk classification rules (Mishcon de Reya)

Read EU Digital Compliance Tracker on AI Act Chapter III

Read The COMPAS case’s impact on the EU AI Act (Lov & Data, 2025)

The Council of Europe AI Convention – A Legally Binding International Treaty

On 17 May 2024, the Council of Europe adopted the Framework Convention on Artificial Intelligence and Human Rights, Democracy, and the Rule of Law. Opened for signature on 5 September 2024 in Vilnius, Lithuania, this Convention is the first‑ever international legally binding treaty specifically addressing the intersection of AI systems with fundamental rights. To date, the Framework Convention has been signed by 17 countries, including member and observer states of the Council of Europe, as well as the European Union.

The Convention covers the use of AI systems by public authorities—including private actors acting on their behalf—and by private actors. It seeks to ensure that activities within the lifecycle of AI systems are fully consistent with human rights, democracy, and the rule of law, while being conducive to technological innovation. The Convention obliges contracting parties under Article 4 to ensure the protection of human dignity at all stages of the AI system lifecycle.

Key provisions of the Convention include the following. It mandates that signatories provide remedies for breaches of human rights related to the treaty’s obligations and principles. It ensures a body is in place for lodging complaints regarding AI‑related human rights violations. It covers the use of AI systems by public authorities, including private actors acting on their behalf, and by private actors. It introduces an emerging human rights principle: the right not to be subject to an automated decision entailing important consequences—effectively requiring human‑in‑the‑loop oversight for significant AI‑assisted decisions.

The Convention utilises broad and uncontentious principles, such as upholding human rights and the rule of law, to create an inclusive system of regulation with a broad definition of covered AI activities. This framework is designed to be adaptable across diverse legal systems while maintaining core protective standards for individuals affected by AI‑assisted decisions.

Read the Council of Europe AI Framework Convention (CoE official site)

Read analysis of the AI Convention (Law360, 2024)

US Legislative Proposals – Algorithmic Accountability Act of 2025

In the United States, legislative attention has increasingly focused on algorithmic accountability, though specific regulation of judicial AI tools remains fragmented. The Algorithmic Accountability Act of 2025 has been introduced in multiple forms in the 119th Congress. A Senate version sponsored by Senator John Curtis and Senator Mark Kelly would amend Section 230 of the Communications Act to hold social media companies accountable for harms caused by their proprietary algorithms. A separate House bill (H.R. 5511) would direct the Federal Trade Commission to require impact assessments of certain algorithms and augmented critical decision processes, including in housing, employment, credit, education, and other critical areas.

Another version of the Act, introduced by Congresswoman Yvette Clarke, addresses automated decision systems in housing, employment, credit, education, and other sectors. A bipartisan bill introduced by Representatives McClain Delaney and Kennedy would hold social media companies accountable for harms caused by content pushed through their algorithms, requiring companies to responsibly design, train, test, deploy, and maintain their algorithmic systems in ways that prevent foreseeable bodily injury or death.

However, none of these proposals directly addresses the use of AI in judicial decision‑making. Criminal justice risk assessment tools remain largely governed by state law, judicial precedent, and voluntary guidelines. Scholars have called for federal legislation requiring transparency, validation, and ongoing auditing of risk assessment tools used in federal sentencing and pretrial release decisions. The absence of comprehensive federal regulation means that algorithmic accountability in US courts continues to develop primarily through case‑by‑case litigation and state‑level initiatives.

Read H.R. 5511 – Algorithmic Accountability Act of 2025 (Congress.gov)

Read Senator Kelly press release (2025)

Read Congresswoman Clarke press release (2025)

Chapter 4: Human Rights and Algorithmic Accountability

⬅ Back to Table of Contents

UN Guiding Principles on Business and Human Rights – Application to AI Systems

The United Nations Guiding Principles on Business and Human Rights (UNGPs), endorsed by the UN Human Rights Council in 2011, have become a global reference framework for governments and businesses. Under the UNGPs, States have a duty to protect individuals and communities from human rights abuses by third parties, including businesses. At the same time, business enterprises themselves bear an independent responsibility to respect human rights, both those developing AI and those using AI in their activities.

Applying the UNGPs to AI systems presents unique challenges. Unlike traditional industries, digital technologies scale almost instantly and are continuously updated, making static human rights due diligence processes potentially inadequate. In response, UN experts have called for the procurement and deployment of artificial intelligence systems to be aligned with the UN Guiding Principles on Business and Human Rights. The UN Working Group on Business and Human Rights has emphasised that businesses developing, procuring, or deploying AI have a responsibility to respect human rights, including by implementing human rights due diligence which, in the AI context, must be early, ongoing, and context‑specific.

For judicial AI tools, the UNGPs imply that developers of risk assessment algorithms must conduct human rights impact assessments before deploying tools in court settings. These assessments should examine potential disparate impacts on protected groups, risks of reinforcing systemic discrimination, and adequacy of remedies for individuals harmed by algorithmic errors. States procuring AI tools for judicial use must ensure that their procurement processes include human rights criteria and that deployers (courts, probation departments, parole boards) are trained on the human rights implications of algorithmic decision‑making.

The UN Human Rights Council adopted a resolution during its 59th Session (2025) on “New and Emerging Digital Technologies and Human Rights,” the first HRC resolution to address the overall human rights implications of new and emerging technologies, including artificial intelligence and big data. This resolution reinforces that existing human rights standards—including non‑discrimination, due process, and access to remedy—apply directly to AI‑enabled decision‑making and can anchor a people‑centred justice approach. The discussion emphasised that AI can assist judges but must never replace human judgment, accountability, or due process.

Read OHCHR Call for Inputs on AI and UNGPs (2025)

Read Harvard analysis on technology and human rights (2026)

UNESCO and Victorian Law Reform Commission – Principles for AI in Courts

International and national bodies have developed principles to guide the safe and ethical use of AI in judicial settings. In December 2025, UNESCO issued Guidelines for the use of AI systems in courts and tribunals, built around fifteen universal principles—from transparency, accountability, and human oversight to human rights protection. The guidelines aim to provide a global ethical and operational framework to ensure that AI serves justice rather than undermining it.

The Victorian Law Reform Commission (Australia) has proposed principles to guide the safe use of AI in Victoria’s courts and VCAT, drawing on principles relating to AI, justice and human rights. These principles affirm that the use of AI should not undermine applicable human rights, including the right to equality before the law and the right to a fair hearing before an impartial decision maker.

Existing human rights standards—legality, non‑discrimination, due process, and access to remedy—apply directly to AI‑enabled decision‑making. Courts and tribunals implementing AI tools must ensure that human oversight is maintained, that decisions remain reviewable, and that affected individuals have meaningful remedies. The guidance emphasises that AI can assist judges but must never replace human judgment, accountability, or due process.

Read Victorian Law Reform Commission AI in Courts project

Read UNESCO AI in Courts Guidelines (2025)

Empirical Research on Judicial Use of Risk Assessment Tools

Understanding how judges actually use risk assessment tools is essential for designing effective accountability mechanisms. A 2025 study by Dennis D. Hirsch, Jared Ott, Angie Westover‑Munoz, Christopher B. Yaluma, and Leslie Schneider, published in the Federal Sentencing Reporter, surveyed Ohio Courts of Common Pleas judges and staff and interviewed judges and other key stakeholders to learn how they view and use algorithmic risk assessment tools.

The study found that federal and state criminal justice systems use algorithmic risk assessment tools extensively, but much existing scholarship focuses on normative and technical analyses rather than on‑the‑ground implementation. Judges’ attitudes toward, and implementation of, algorithmic risk assessment tools profoundly affect how these tools impact defendants, incarceration rates, and the broader criminal justice system. The study describes how Ohio Common Pleas Courts implement algorithmic risk assessment tools and how judges view and utilise the tools and the risk scores they generate. It then compares Ohio practice to best practices identified in the literature and recommends how courts can better align their use of algorithmic risk assessment tools with core criminal justice values.

Key findings indicate wide variation in how judges interpret risk scores, inconsistent training on algorithmic tools, and a tendency for judges to defer to risk assessments without independent scrutiny. The study recommends standardised training, clear judicial guidelines on the weight to be given to risk scores, and mandatory periodic auditing of risk assessment tools for accuracy and bias.

Read Aligning Algorithmic Risk Assessments with Criminal Justice Values (Duke University Press, 2025)

Chapter 5: Strategies for Ensuring Fairness and Due Process

⬅ Back to Table of Contents

Transparency, Validation, and Adversarial Testing

For algorithmic accountability to be meaningful in judicial contexts, courts and legislators must adopt transparency, validation, and adversarial testing requirements. Transparency requires that the data inputs, model version, processing steps, and human oversight be documented and accessible to affected parties. This includes disclosure of the variables used in risk calculations, the statistical methods employed, and any known limitations or biases in the tool.

Validation requires that risk assessment tools be independently tested for predictive accuracy, bias across demographic groups, and calibration (the alignment between predicted risk scores and actual recidivism rates). Validation studies should be conducted on populations representative of the jurisdictions in which the tool is deployed, using current data. Validation results should be made publicly available, including subgroup analyses for race, gender, age, and other protected characteristics.

Adversarial testing requires that defendants and their counsel have an opportunity to challenge the risk assessment evidence against them. This includes access to the underlying data used to generate each defendant’s risk score, disclosure of the algorithm’s inputs and outputs, and the ability to retain independent experts to audit the tool’s performance as applied to the individual case. Where proprietary trade secrets prevent full disclosure, courts should consider protective orders that allow confidential access to algorithmic materials while preserving the defendant’s ability to mount a meaningful challenge.

Best practices increasingly require that risk assessment tools used in high‑stakes judicial decisions undergo regular independent auditing, with audit results made publicly available. Audits should examine both group fairness (whether error rates differ across demographic groups) and individual fairness (whether similarly situated individuals receive similar risk scores). Where bias or inaccuracies are detected, remedial action—including retraining of models, adjustment of scoring thresholds, or suspension of the tool—must be required.

Judicial Training and Guidelines

Judges cannot meaningfully scrutinise algorithmic risk assessments without adequate training on how these tools work, their statistical properties, and their limitations. Many judges lack formal training in statistics or machine learning and may be unaware of the potential for algorithmic bias or overreliance on risk scores. The Ohio study identified inconsistent training across courts and a tendency for judges to defer to risk assessments without independent scrutiny.

Recommended measures include the following. Mandatory continuing judicial education on algorithmic risk assessment tools should be required, covering statistical concepts such as predictive validity, calibration, false positive and false negative rates, and bias detection. Written judicial guidelines on the weight to be given to risk assessments should be developed, clarifying that risk scores are advisory rather than determinative. Model jury instructions for cases involving algorithmic evidence should be created, ensuring that fact‑finders understand the limitations of risk predictions. A requirement that judges articulate their reasoning when deviating from or relying upon risk scores should be implemented, creating a record for appellate review. Bench books and reference materials on algorithmic fairness should be provided to all judges, with practical guidance on identifying potential bias and evaluating expert testimony on AI systems.

The Loomis court’s “written advisement” approach has been criticised as insufficient because it does not provide judges with the tools to critically evaluate risk assessments. More robust approaches include requiring that any risk assessment tool used in sentencing meet minimum validation standards before judicial reliance, and that judges receive specific, actionable warnings about known biases and limitations of each tool they encounter.

Legislative Reforms and Due Process Protections

Meaningful algorithmic accountability ultimately requires legislative action to establish clear due process protections for individuals affected by algorithmic risk assessments. The patchwork of judicial precedent and state‑level regulation has proven inadequate to address systemic concerns about bias, opacity, and lack of meaningful challenge.

Proposed legislative reforms include the following. Require that any risk assessment tool used in criminal proceedings be validated for the specific population and jurisdiction in which it is used, with validation studies made publicly available. Mandate disclosure to defendants and their counsel of all variables, weights, and processing steps used to generate each individual’s risk score. Establish a right to independent algorithmic auditing at state expense for indigent defendants. Prohibit the use of proprietary trade secrets as a basis for withholding algorithmic information necessary to mount a due process challenge. Create a statutory cause of action for individuals harmed by demonstrably inaccurate or biased algorithmic risk assessments, with remedies including resentencing, release, and damages where appropriate. Require regular independent audits of all risk assessment tools used in judicial settings, with audit results submitted to the legislature and made publicly available.

The Thomas v. Montgomery decision illustrates a critical gap in current protections: where no liberty interest attaches to parole, even demonstrably flawed algorithmic assessments evade judicial scrutiny. Legislative reform to create protected liberty interests in parole and probation decisions, with attendant due process requirements, would directly address this gap.

The Role of Civil Society and Legal Advocacy

Civil society organisations and legal advocacy groups play a vital role in advancing algorithmic accountability. These actors bring test cases challenging the use of biased or opaque risk assessment tools, conduct independent research documenting algorithmic bias, advocate for legislative reform, and provide technical assistance to defence counsel and public defenders.

The ProPublica investigation into COMPAS, for example, catalysed a global movement for algorithmic accountability by providing empirical evidence of racial disparities. Similar investigative and research efforts are needed to evaluate the performance of risk assessment tools across different jurisdictions and contexts. Organisations such as the American Civil Liberties Union (ACLU), the Electronic Privacy Information Center (EPIC), and the Algorithmic Justice League have been at the forefront of efforts to expose algorithmic harms and advocate for reform.

Legal advocacy strategies include the following. Filing constitutional challenges to the use of risk assessment tools on due process, equal protection, and right‑to‑counsel grounds. Submitting amicus curiae briefs in appellate cases raising algorithmic accountability issues. Engaging in legislative advocacy for transparency, validation, and auditing requirements. Providing training and resources for public defenders and criminal defence practitioners on challenging algorithmic evidence. Public education campaigns to raise awareness of algorithmic bias in criminal justice settings. International advocacy to align national practices with human rights standards articulated in the Council of Europe AI Convention and UNGPs.

The growing body of case law discussed in this playbook demonstrates that courts are increasingly receptive to due process arguments concerning algorithmic evidence, though outcomes remain inconsistent. Continued litigation, combined with legislative advocacy and public pressure, offers the most promising path toward meaningful algorithmic accountability in judicial decision‑making.

Related Topics

⬅ Back to Table of Contents

The following topics expand this playbook into related areas of AI, law, and justice.

• AI and Evidentiary Integrity: How AI‑generated evidence (deepfakes, synthetic media) is challenging traditional rules of evidence in criminal and civil litigation.
• Algorithmic Bias in Employment and Housing: Legal challenges to AI hiring tools, credit scoring algorithms, and tenant screening systems under anti‑discrimination laws.
• Facial Recognition and Fourth Amendment: Constitutional challenges to warrantless use of facial recognition technology by law enforcement, and emerging state regulations.
• Data Privacy in Criminal Justice Settings: Intersection of biometric data collection, surveillance technologies, and privacy rights under GDPR, CCPA, and Fourth Amendment jurisprudence.
• AI in Immigration Adjudication: Use of algorithmic risk assessments in immigration detention decisions, asylum credibility determinations, and visa processing.
• International Human Rights Law and AI: Application of international human rights treaties to state use of AI, including right to fair trial and prohibition of discrimination.
• Automated Decision‑Making and Administrative Law: Judicial review of algorithmic government decisions in contexts such as welfare benefits, tax assessments, and regulatory enforcement.
• Legal Ethics and Generative AI: Professional responsibility implications of lawyers using large language models for legal research, drafting, and evidence preparation.

FAQ

⬅ Back to Table of Contents

What is COMPAS and why is it controversial?

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a proprietary risk assessment tool developed by Northpointe (now Equivant). It predicts the likelihood of recidivism and is used by courts in several US states for sentencing and pretrial release decisions. The controversy stems from a 2016 ProPublica investigation that found significant racial bias: Black defendants were twice as likely as white defendants to be incorrectly labelled as high risk. Additionally, because COMPAS is a trade secret, defendants cannot examine how their risk score was calculated, raising due process concerns.

Can defendants challenge risk assessment evidence against them?

This depends on the jurisdiction. Under State v. Loomis, the Wisconsin Supreme Court held that judges may consider COMPAS assessments if they provide certain written advisements, but defendants cannot access the proprietary algorithm. In other jurisdictions, defendants have successfully challenged risk assessments on due process grounds, particularly where the algorithm’s methodology is disclosed and challenges can be mounted. The EU AI Act requires transparency and human oversight for high‑risk judicial AI systems, but the US currently lacks comprehensive federal regulation.

What is the EU AI Act’s position on judicial AI?

The EU AI Act classifies AI systems used for judicial and democratic processes as “high‑risk” under Article 6(2) and Annex III. Such systems must comply with mandatory requirements including risk management (Article 9), data governance (Article 10), technical documentation (Article 11), transparency (Article 13), human oversight (Article 14), and accuracy, robustness, and cybersecurity (Article 15). Final decision‑making must remain a human‑driven activity.

Are there international legal standards for AI in justice systems?

Yes. The Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy, and the Rule of Law (opened for signature September 2024) is the first legally binding international treaty on AI. It requires signatories to ensure AI systems respect human rights, the rule of law, and democratic norms, and provides for remedies and complaint mechanisms. The UN Guiding Principles on Business and Human Rights also apply to developers and users of judicial AI tools.

How can jurisdictions improve algorithmic accountability in sentencing?

Recommended measures include mandatory independent validation of risk assessment tools for each jurisdiction, disclosure of variables and weighting to defendants and counsel, legislated right to independent algorithmic auditing, prohibition of trade secret exemptions for due process purposes, regular public audits, judicial training on statistical and algorithmic concepts, and clear statutory guidelines on the weight to be given to risk scores. Legislative creation of protected liberty interests in parole and probation decisions, with attendant due process requirements, is also essential.

References

⬅ Back to Table of Contents

Verified source links (embedded):

• State v. Loomis, 2016 WI 68, 881 N.W.2d 749 (Wis. 2016) – FindLaw
• State v. Loomis – Harvard Law Review comment (Vol. 130, March 2017)
• Loomis v. Wisconsin – SCOTUSblog case file
• ProPublica study on COMPAS bias (2016)
• Thomas v. Montgomery (6th Cir. 2025) – Casemine analysis
• R. v. Chand, 2025 ONCJ 282 – Law360 Canada (2025)
• Russian Federation AI ruling – Library of Congress Global Legal Monitor (2025)
• EU AI Act Chapter III – High‑risk AI systems (Snellman Digital Compliance Tracker)
• Classification Rules for High‑Risk AI Systems (Mishcon de Reya)
• The COMPAS case’s impact on the EU AI Act – Lov & Data (2025)
• Council of Europe Framework Convention on AI and Human Rights
• H.R. 5511 – Algorithmic Accountability Act of 2025 (Congress.gov)
• Algorithm Accountability Act – Senator Kelly press release (2025)
• Algorithmic Accountability Act – Congresswoman Clarke press release (2025)
• OHCHR Call for Inputs – AI and UN Guiding Principles on Business and Human Rights (2025)
• Has Technology Outpaced Human Rights Frameworks? – Harvard Kennedy School (2026)
• Victorian Law Reform Commission – Use of AI in Courts and Tribunals
• UNESCO Guidelines for AI in Courts (2025)
• Aligning Algorithmic Risk Assessments with Criminal Justice Values – Federal Sentencing Reporter (2025)