Like the ostrich that buries its head in the sand, Mr. Holder is wrong about data-driven sentencing

Attorney General Eric Holder, addressing criminal defense lawyers, expressed a concern about the use of empirical data to sentence people. One assumes he has similar concerns about the use of empirical data to to decide how offenders should be supervised when out on the streets on supervised release.

The Wall Street Journal wrote the following on August 1, 2014 detailing Holder’s remarks:

Attorney General Eric Holder warned Friday that a new generation of data-driven criminal justice programs could adversely affect poor and minority groups, saying such efforts need to be studied further before they are used to sentence suspects.

In a speech in Philadelphia to a gathering of the National Association of Criminal Defense Lawyers, Mr. Holder cautioned that while such data tools hold promise, they also pose potential dangers.

“By basing sentencing decisions on static factors and immutable characteristics—like the defendant’s education level, socioeconomic background, or neighborhood—they may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society,” Mr. Holder told the defense lawyers. Criminal sentences, he said, “should not be based on unchangeable factors that a person cannot control, or on the possibility of a future crime that has not taken place.”

Although I think Holder’s point of view is is naive and manifestly wrong, we should at least thank the Attorney General for flagging a very important issue. With that acknowledged, let’s look a little deeper.

As compared with Holder’s concern, there is a movement at the federal level to seriously examine the huge data bases that exist in order to make predictive judgments about how offenders are likely to do in the future and to factor that data driven analysis into the judicial process. Indeed, a former policy analyst for the federal courts wrote three years ago that:

Evidence-based sentencing is based upon social science. Criminological meta-analysis has identified fifteen key variables that are significantly related to recidivism: 1) criminal companions, (2) antisocial personality, (3) adult criminal history, (4) race, (5) pre-adult antisocial behavior, (6) family rearing practices, (7) social achievement, (8) interpersonal conflict, (9) current age, (10) substance abuse, (11) intellectual functioning, (12) family criminality, (13) gender, (14) socio-economic status of origin, and (15) personal distress. If those variables can be used in sentencing, it may be possible to safeguard public safety while reducing the financial and social costs associated with mass incarceration.

J.C. Oleson, Risk Assessment at Sentencing, ASU Law Journal (June 20, 2011). See also Administrative Office of the United States Courts Office of Probation and Pretrial Services, An Overview of the Federal Post Conviction Risk Assessment (September 2011).

Notice from the foregoing that “immutable characteristics” or “static factors” like race, gender and age are included. If race, gender or age are predictive as validated by good empirical analysis, and we truly care about public safety while at the same time depopulating our prisons, why wouldn’t a rationale sentencing system freely use race, gender or age as predictor of future criminality? The same can be said of factors like antisocial behavior, family criminality and other factors that have in the past made us queasy because they were thought to focus too much on the poor.

In my opinion, the use of empirical methods to assess risk at sentencing and upon supervised release is the most important aspect of the present movement toward criminal justice reform. We can and should aspire to depopulate our prisons. We can and should aspire to treat certain offenders less harshly. We can and should do a better job of helping offenders on supervised release. But if we believe that public safety is or should be a central goal of our criminal justice system we ought not to ignore the truth–certain characteristics that we have shied away from in the past because we worried too much about vague notions of “equality” or “fairness” tell us a lot about future danger. The Attorney General was wrong to put his head in the sand like the proverbial ostrich when he suggested those factors be ignored. The rest of us ought not to make the same mistake.


39 responses

  1. You’re just trying to be provocative judge. You don’t really think that race should be a factor considered in sentencing decisions, despite what the “data” may show about the race of people who commit certain crimes. You don’t really have the faith in “data” that you pretend to have in your piece above because you are old enough to remember Mark Twain’s maxim, “Figures don’t lie, but liars do figure.”

  2. Wall Street Journal left out the full quote, “Criminal sentences must be based on the facts, the law, the actual crimes committed, the circumstances surrounding each individual case and the defendant’s history of criminal conduct,” Holder said. “They should not be based on unchangeable factors that a person cannot control, or on the possibility of a future crime that has not taken place.”

  3. Why is race predictive? Is it immutable race as such, or is race a proxy for a bag of changeable factors that are inordinately associated with or experienced by one race or another? Sort of comes down to who’s defining the categories, how far they want to unpeeled the onion, etc.

  4. How can age be immutable? Gender used to female and male and now it is female, male, changeable and undecided. Some of us think we know what race is but others folks say there is no such thing as race and we have about 300 definitions of ethnicity.

    I am not willing to predict future criminal behavior based evidence based hogwash. It is hard to do research in the social sciences because there are too many variables, none can be controlled and the data quality is very low (some of it is garbage). I agree that risk assesment has improved but not enough to be used for sentencing. I think it is good enough to be used for community based correction but we will still have the occasional “nasty surprise”.

  5. So when sentencing, you want to deny equal protection under the law based on race, which totally violates the 14th Amendment, never mind statutes like the 1964 Civil Rights Act? It’s not like we don’t have 14% of the drug offenders in this country being black and 60% of those locked up for it being black already!

  6. Dear Judge

    I want to thank you for writing this blog and for being brave against your critics. It’s so refreshing for me to see your honest evaluation of the federal judiciary and our judicial system in general.

    I have represented poor individuals for the past 25 years and all I’ve heard from every past Attorney General is “less plea bargaining” and demand “higher sentences”, the stuff that politicians love. I remember the days when all we had were mandatory minimums and no safety valve. On more then one concession as a young lawyer, I would choke and be unable to speak on behalf of my clients at the time of their sentence because I knew that nothing I said would make a difference.

    Now we have our first black Attorney General advocating for the poor and less fortunate and all I hear from judges is negative comments like yours. I find this very frustrating. As you very well know, it takes a lot of guts to go against the current. And I am thankful that we have a brave Attorney General who decided to fight for the less fortunate. I respect your opinion but I have to say I find your criticism of Mr. Holder a bit too harsh.


    Andy Nogueras

  7. I understand your point and agree that’s the appropriate way. Unfortunately, we live in the times of the do nothing Congress. So what’s the solution?

  8. Judge:
    I respectfully disagree. As conservative as I am, I am nonetheless troubled by the consideration of admittedly immutable characteristics which are heritable (race, gender, age) in sentencing decisions. For instance, hair color and short sightedness are highly heritable but can be changed. That is not presently true of one’s race, gender or age. To consider these characteristics as predictive of future criminality–a dubious business in the very best of circumstances–is to run the risk fetishizing statistics at the expense of justice. For instance, while there is some suggestion that, from a genetic standpoint, cruelty (and/or potential criminality) is linked to a lack of empathy, not even that is true all of the time (go here:,-empathy-and-the-evolution-of-morality.aspx). As that writer states, in theory a brain scan, genotyping and a case history could “diagnose” many or most cruel people before they actually commit crimes. But we are likely a very long way from the world depicted in the movie “Minority Report.” Including immutable characteristics in sentencing decisions will not get us there any faster.

  9. Couldn’t agree more. Would love to sit in on a sentencing hearing wherein Mr. Holder was the judge, the case involved a particularly heinous crime, and the defendant’s record both as a juvenile and as an adult was a yard long; who came from nothing and had no reason in his or her background to exercise restraint in anything; and who, by all available information, was just downright dangerous. I would love to hear his explanation of the mitigating factors. How would he have sentenced Richard Speck, or Timothy McVeigh?

  10. John,

    With respect, I am not just trying to be provocative. Two brief responses.

    First, empirically based risk predictions commonly and justifiably use race among other immutable characteristis as as part of what scientists call “actuarial” assessments of risk. Everyone who is shocked (including Holder) that immutable characteristics are included should read John Monahan, A JURISPRUDENCE OF RISK ASSESSMENT: FORECASTING HARM AMONG PRISONERS, PREDATORS, AND PATIENTS, 92 Virginia Law Review 391 (2006) to understand how such predictions are properly used. As Professor Monahan suggests, these types of risk assessments are only part of what should be considered at sentencing but they should be considered. That is particularly so if our sentencing goals move from looking backward (what does the offender deserve for his or past crime) to moving forward (does the offender require incapacitation to protect the public from future crimes). If you want to reduce prison populations to only those who are truly dangerous if and when released, it is not reasonable to close one’s eyes to what acturial science tells us.


    “It is important to be clear on what a ‘risk factor’ is and what it is not. To call A risk factor for B means two things and only two things. It means that (1) A statistically correlates with B, and (2) A comes before B in time. A simple risk factor, in other words, is ‘a correlate that precedes the outcome,’ and nothing more. In particular, to call A a risk factor for B is not in any sense to imply that A ’caused’ B.” Id. at 413 n.85 (PDF p. 23.)

    Thus, I am not saying that race “causes” crime.

    All the best.


  11. Please see my response to John. (If race is predictive, and it is, why do we care why that is so? Remember at sentencing we are not charged with determining causation.)

  12. Do you currently consider race a factor when sentencing? Based on what you have written it appears you do.

  13. Anony.,

    No, not explicitly. The Guidelines prohibit it.

    However, taking into account what scientists call acturial data–things like large data sets predicated upon age, gender, race, personality (such as “anger”) as one part of the calcuclus–would be a new but potentially valid factor IF the goals of sentencing were changed to concentrate on the allocation of offenders to prison based upon the offense of conviction (past behavior) plus a prediction of future criminality. For example, even though a crime is “non-violent” should we automatically conclude that the offender is at a low risk of future violent behavior if the acturial data (plus what we know of the offender himself) might show otherwise.

    In short, and to emphasize, there is date available to the sentencing judge which is presently not used. Mr. Holder does not want that data used at all. I suggest that that if one of the goals of sentencing is to reduce prison populations by reducing the number of prisoners who are not likely to offend in a violent manner upon release, that acturial data is relevant to sentencing in the same sense that such data is relevent to the actuary who is underwriting an insurance policy. This data because of its very general and aggregated nature should never be the driver, but it is a factor.

    Finally, I can understand why race as a factor in these acturial data sets might be ignored. We would reject race not because it is not predictive about future violence but because the mere mention of race as a factor offends our customs and history. But if we do that, then we ought to acknowledge explicitly that we are ignoring data that may be highly predictive of future violence.

    All the best.


    PS In response to an earlier commentator, I cited Professor Monahan’s work. I should clarify that “Race should not be considered a risk factor in determining whether someone is likely to commit a violent crime in the future, Professor John Monahan said at a talk sponsored by the Center for the Study of Race and Law March 15.” (See here.) “Age, gender, personality, mental disorders, personality disorders, substance abuse disorders, prior violence, a pathological family environment, and victimization are all factors that can predict future violent behavior, Monahan said.”

  14. Pingback: Please read Scott Greenfield’s post “Numbers Don’t Lie (But People Do)” « Hercules and the umpire.

  15. Judge,

    This post is the best possible evidence that Holder is right. Here we have a Federal judge expressing the view that if race is statistically predictive of recidivism, then it should be explicitly considered in sentencing — the logical result of which being that a black defendant should receive a harsher sentence than a similarly situated white defendant.

    But why is race predictive of criminal convictions? Many would argue that it is due to the discriminatory nature of enforcement — a constant police presence in low-income and minority communities, stopping young black men on the street on a regular basis, often precipitating the very charges that the men are then convicted of (disorderly conduct, ABPO, resisting arrest). All this while in suburban neighborhoods and on college campuses, minor assaults and controlled substance offenses pass largely unremarked upon. If my brother and I, and our social circle, had been policed when we were growing up the way that urban communities of color are policed, hardly anyone I know would have a clean record.

    Holder’s point is that the pursuit of justice requires fundamental changes to the system, and that the injustices perpetuated by the system as it exists would only be exacerbated by “evidence-based” increases in sentencing for the people who the system already disproportionately cracks down on, when the evidence it’s based on is the fact that the system disproportionately cracks down on them.

  16. Regardless of any statistic correlation between race and future-risk, we must understand that correlation does not mean causation. What the federal judge is basically saying is that if black (or any other race) defendants in the past have participated in violent/risky behavior after release from prison, then black defendants in the future will do the same.

    This is absolutely ridiculous, as people are individuals – and one black person does not act the same as another black person simply because they are black (or white, hispanic, etc.). To enhance an individual’s prison sentence based on the actions of other people of the defendant’s race/ethnicity is absolutely wrong, whether a statistical correlation is seen or not.

    Putting aside the long discussion of how the criminal justice system is discriminatory from the police through the courts, it is quite disturbing that a Federal Judge, who is supposed to use the constitution to protect the rights of American citizens and ensure that all are treated equally, regardless of race, seems eerily comfortable to not treat defendants equally, BECAUSE OF RACE, during sentencing, based on the actions of other members of the defendant’s race and not the actions of the individual being sentenced.

  17. Pingback: Numbers Don’t Lie (But People Do) | Simple Justice

  18. Judge, I think that the missing link in your argument (and therefore the reason why I think you are on the wrong track) is the link between “a person with characteristics x y and z is more likely to re-offend upon release” and “therefore we should incarcerate that person for a longer period.”

    I mean, I do understand the link in a certain set of cases: i.e., horrible violent crimes, and a person who is adjudged very likely to commit the same horribly violent crimes if released, therefore let’s incarcerate him until he is too old to hurt anybody.

    But if you think about a different set of defendants and offenses – say, drug offenses without active use of weapons, or social security fraud – the link is much harder to justify. If we incarcerate the person longer, have we really kept the public safer? Not much in any event – and In the drug example, probably somebody else will just step in to do the job while this defendant is incarcerated – thus no positive effect on public safety.) But then the defendant eventually does get out, after more years which make it even harder to re-enter into lawful productivity. Have we really kept crime down in the long run? And (since there are things in this life that matter other than the level of crime) have we made the world better?

    To me it seems pretty obvious that the answer in cases other than horrific violent crime is “if your social science data tell you that this fellow has a greater percentage likelihood of reoffending, this is not a reason to imprison him longer. It is instead a reason to try to figure out what we can do to reduce his likelihood of reoffending, once he is released.”

  19. Mr. Kopf,

    How do you believe that outliers should be considered in a system of evidence based sentencing factors? In other words, take the factors that you have presented and plug them into a statistical software package such as STATA or R. Then, let the software run through the regression analysis using the mathematical equation modeling determined by the researcher (an entirely unique decisional process on its own). Apply the regression line on a plot and see what independent variables correlate with the dependent variable, that is recidivism. However, remember that the data going in represents individual unique persons, not just arbitrary figures. So, what about that person who shows up on the plot and has all of the variables that correlate with recidivism, but has not committed any further crime (lacks the dependent variable)? Or, to take the counter example, the individual whose data was entered and lacks the correlating variables, but did commit further crime (has the dependent variable).

    This is how regression analysis and scientific correlation studies work. While the regression line may be skewed by the outlier data (although there are various scientifically accepted ways to ‘deal’ with these outliers) the final ‘algorithm’ based off the analysis does not consider that a certain percentage of people with some, most, or all of the correlating factors will not actual commit future crime. Using this evidence-based sentencing is akin to treating everyone as though they were some variable being held constant at its modal value. If male=1 and female=0, then this value is 0.5 when being held constant to consider the causal effects that other variables have. This person with a sex of 0.5 does not exist. This is one of the many difficulties of applied social science using mathematical modeling. That is to say people are unique as opposed to being merely the average of their sum.

    In closing, do you believe that sacrificing those outliers for the greater good of empirically based sentencing is a concept agreeable with fundamental notions of justice? Can our system of justice tolerate the maxim: if in the course of sentencing 100 criminals based on an actuarial analysis of their characteristics, some of those criminals may not commit future crime, that is the price we pay for order and safety.

  20. Sean,

    I am not sure I understand your question. Wouldn’t the best data set for any actuarial analysis be based upon offenders who actually committed subsequent violent crimes and include both those who did and did not have the factors one posits is correlated with recividism?

    All the best.


  21. Sam,

    I tend to agree with you that this data is best used for offenders who have demonstrated a capacity for violence or who engage in criminal conduct that we know has a high liklihood violence being present more than occassionally. All the best.


  22. What you are referring to is called “selecting on the dependent variable” and is not considered an acceptable practice in the sciences. This is something that a first semester graduate student in the sciences would learn about in their research methods course, much like a first semester law student might learn about ‘but for’ causation in their criminal law course. Researchers are not supposed to select cases that fit their theory, rather they are supposed to select cases representative of the population from which they are estimating the parameters.

    A post by Dan Kahan of Yale University explains it better than I could:

    “Selecting on the dependent variable refers to the practice of restricting one’s set of observations to cases in which some phenomenon of interest has been observed and excluding from the set cases in which the phenomenon was not observed. Necessarily, any inferences one draws about the causes of such phenomenon will then be invalid because in ignoring cases in which the phenomenon didn’t occur one has omitted from one’s sample instances in which the putative cause might have been present but didn’t generate the phenomenon of interest – an outcome that would falsify the conclusion.”

    Falsifiability is one of the foundations of modern sciences, both the natural sciences and the social sciences. When it comes to the courts and science, C.J. Rehnquist may have said it best when he wrote “I defer to no one in my confidence in federal judges; but I am at a loss to know what is meant when it is said that the scientific status of a theory depends on its “falsifiability,” and I suspect some of them will be, too” (Daubert v. Merrell Dow Pharmaceuticals, Inc., 1993). Essentially, Rehnquist is admitting that he simply doesn’t understand what this term means, which is understandable as scientific research methods are taught in PhD programs, not JD programs. And, therein lies one of the fundamental problems that is inherent in my original post and your response. While you believe that “evidence-based sentencing” is useful, you, and others in the judiciary lack the training necessary to understand the nuances of how the research is conducted and both its benefits and pitfalls. Just like a social scientist lacks the training to adjudicate a complicated criminal case or patent dispute.

    Evidence-based sentencing should not be blindly accepted because it represents “empirical science” as that is such a loaded term and researchers in the social sciences spend years learning the philosophy of science and appropriate research methods. Additionally, statistical analysis is void and potentially dangerous if it lacks an underlying theory to explain the correlation. As someone who has commented numerous times on the book “The Behavior of Federal Judges,” I am sure you understand the relationship between theory and empirical data. A dataset composed of just those persons who have committed additional crimes may be useful from a pure data standpoint, but any conclusions derived from this dataset are not scientific and this type of data mining is not social science.

    So, in answer to your post, any research that would be based off your supposed dataset would fail to be accepted by the scientific community as it would be guilty of selecting on the dependent variable. Therefore, the evidence-based research that would be considered acceptable by the scientific community would inevitably contain the type of outliers that I discussed in my original post. And that brings me back to my original question on how these outliers should be reflected in sentencing.

    Maybe we truly are entering an era where legal proceedings are conducted before a judge and sentencing proceedings are conducted before a panel of social scientists.I for one hope this is not true.

  23. Your honor,

    If the goal of the criminal justice system is public safety then why should we wait until someone commits a crime to use evidence-based factors for punishment? Why not just use similar evidence to imprison those people most likely to commit crimes in the first place before they have a chance to violate public safety?

  24. “If race is predictive, and it is, why do we care why that is so? Remember at sentencing we are not charged with determining causation.”

    I haven’t time for a fuller response to the “see my response to John” bit, but this cries out for an answer.

    If race is a proxy for a bag of changeable factors associated with one race or another (say poverty, fatherless homes, etc.; I am not saying that is so, I am posing examples of possible “causative factors”), then sentencing based on race will result in more of those in a disfavored race being incarcerated more often and for longer periods. This will have the effect of increasing the percentage of those of the disfavored race who live in poverty, grow up in fatherless homes, etc. At some point, you have created the association you are measuring. Therefore, it very much matters whether you are using race as a proxy for a bag of changeable factors and, in particular, what those factors are. Either use the factors or don’t.

    It takes little knowledge of history to see where using race as a proxy for other attributes gets us.

  25. Kerry, when it comes to race, I think you make an important point. One could use the factors that make up the variable of race rather than using race alone. The difficulty, of course, with doing so is disaggregating all the relevant factors that make up “race” for predictive purposes. Thank you for writing.

    All the best.


    PS I may have a post tomorrow that further addresses sentencing based upon forward predictions of violence.

  26. Pingback: More about AG Holder as an ostrich « Hercules and the umpire.

  27. Pingback: News Roundup « North Carolina Criminal Law

  28. Pingback: May the Odds Be (Never) in Minorities’ Favor? Breaking Down the Risk-Based Sentencing Divide | Freelance HoneyBadger

  29. Pingback: May the Odds Be (Never) in Minorities' Favor? Breaking Down the Risk-Based Sentencing Divide | NEWS | Phones | Nigeria Science | Technology |Computers

  30. Pingback: Venturuso - Hispanic Venture Capital News

%d bloggers like this: