MFJ Overview

1

What does Measures for Justice do?

Measures for Justice (MFJ) is a nonprofit organization that measures the performance of the criminal justice system on a county-by-county basis. To do this, we have developed a series of performance Measures that track patterns in how cases move through the system from arrest to post-conviction.

All our Measures and analyses present data at the county level and are available for free to the public on a web-based Data Portal. The Portal is searchable and can be configured to break down performance data across multiple factors including race/ethnicity; sex; indigent status; age; offense type; offense severity; and attorney type. The Portal also allows for county-to-county comparison within and across states.

2

Who are your funders?

Measures for Justice receives funding from the Department of Justice, Google.org, and the Arnold, Donner, Ford, MacArthur, Open Society, and Pershing Square Foundations, among others. For more information, please see Partners on our supporters page.

3

Do you advocate specific criminal justice reforms?

Measures for Justice is not an advocacy organization. We collect and display data on local criminal justice because we support transparency and dialogue about practices and patterns that are relevant to how the justice system is functioning.

4

Do the Measures address any topics in particular?

Yes. Some of the foremost experts in the field of criminal justice research and measurement have settled on three areas that are most important when thinking through how to assess system performance: Fair Process, Public Safety, Fiscal Responsibility. Fair Process Measures get at whether due process is being afforded to defendants, which is a central tenant of the Constitution. Public Safety Measures address how the public is being protected from crime, which is one of the central purposes of any justice system. Fiscal Responsibility Measures address how taxpayer money is being spent, which is especially relevant to the pursuit of an efficient justice system.

5

How did you choose your initial six states?

We chose states that had statewide court databases, that were geographically diverse, and that each had a large number of counties.

The Measures

1

What are the Measures?

The Measures are a set of performance indicators designed to assess how the criminal justice system is providing basic legal services in three areas: Fair Process; Public Safety; and Fiscal Responsibility.

2

Who came up with your Measures and how?

MFJ developed an initial set of Measures in 2011 with the help of criminal justice practitioners, researchers, and experts in the development of performance measurement systems in other fields. In 2013, MFJ piloted these Measures using Wisconsin’s administrative court data.

Since then, we have reworked the Measures with two additional advisory groups of researchers and academics, as well as with many practitioners and other stakeholders. For more on our advisory councils, please see our Team page.

For more information on the criteria used to select the final list of Measures, please see our Methodology.

3

How do you get your data?

Our first step is to contact state-level stakeholders to locate and evaluate centralized data resources. This systematic evaluation examines whether statewide datasets exist, what information is captured in these datasets, and whether that information can be extracted in bulk or must be compiled on a case-by-case basis. MFJ’s first and primary source for data typically comes in the form of a bulk data extraction from statewide court data repositories.

When no statewide datasets are available or to supplement statewide court data, the MFJ team collects data at the county level from each criminal justice agency (court, largest arresting agency, prosecutor, public defender, and sheriff/jail).

4

You acknowledge that the data you receive are not always in the best shape. Does this cast doubt on the reliability of your findings?

No. We acknowledge that administrative data have limitations, but we are not suggesting that the data we collect are inappropriate for use. Most of the datasets we acquire are generally reliable. When we find issues in the data, they usually affect only parts of cases and almost never invalidate the whole dataset. To deal with these issues, we implement correctives such as coding problematic values as missing and excluding unreliable data elements. We have developed a careful and lengthy data quality control process through internal and external audits to ensure data are accurate and reliable. For a more detailed explanation of our process, please see Section Four in our Methodology.

5

Have these Measures been used previously?

Yes, other organizations have similar measures in use. For instance, the Institute for State and Local Governance uses several measures as part of MacArthur’s Safety and Justice Challenge that overlap with MFJ’s. What is unique to MFJ are the scope and breadth of our measurement: We collect data on the entire criminal justice system across the United States and facilitate county-by-county comparison via our Data Portal.

6

Why do some states have more Measures than others?

Our Measures need to be populated with the appropriate data, which are not always available for every state. Some states do not collect the data we need; others collect data but record them in ways MFJ cannot use (e.g., the data are missing key elements). We will continue to work with states to encourage more widespread and comprehensive data collection in the hopes of eventually populating all of our Measures.

7

What is your data management process?

MFJ researchers’ first step is to define a “case” to mean all the charges processed within a given jurisdiction that were associated with the same defendant and filed on the same date. This involves standardizing our data files across states by using a common codebook that identifies the driving characteristics of a case (e.g., case disposition, most serious charge, most severe sentence, etc.). That done, we remove personal identifiers from the dataset, create a unique case number that is fully de-identified to protect the privacy of defendants, and turn over the datasets and calculations to the Technology Team. The Technology Team, in turn, develops algorithms to render the calculations and datasets into the results that appear on the Portal.

For a more detailed explanation of our process, please see Section Three in our Methodology.

8

How do you ensure your data are accurate?

Addressing data quality and uniformity has been a critical MFJ goal from the outset. We have developed a careful and lengthy process to prepare our data in house and then run them through internal and external audits to make sure they are correct.

This process involves:

  • Scrutinizing the data to identify elements that may be unreliable.
  • Exploring possible variations between legal and technical language used across and within states.
  • Using text-mining techniques to identify the multiple ways in which the same concept may be recorded using different terms across jurisdictions.
  • Addressing any anomalies, which often reveal that some jurisdictions may not be recording certain events and that, as a result, such events can’t be accurately measured.
  • Approaching relevant stakeholders to inquire about anomalies and to confirm facts.

Per these reviews, MFJ does not use any data elements identified as problematic. For a more detailed explanation of our process, please see Section Four in our Methodology.

9

Do you provide population/demographic information? Is there contextual data?

Yes. We provide contextual Measures for each county including crime rates, demographics, public health indicators, the availability of certain resources, and voting records. We also provide legal context for each state as it relates to criminal justice statutes such as eligibility criteria for indigent defense, pretrial release assessments, use of sentencing guidelines, among other things.

10

How do you compare counties that are so different?

MFJ understands that every county is different, and that counties across the country have different cultures and practices that stem from those cultures. This is why we do not compare county characteristics or ideologies. Instead, we compare how similar cases are handled in different counties.

We have also made it a priority to provide information that encourages responsible comparison. To do this, we standardize all our data by looking for shared characteristics among cases, which ensures we are, in fact, comparing apples to apples. In addition, we continually seek input, local context, and information at multiple stages of our work that we pass on to users. Every Measure is accompanied by:

  • Footnotes. The footnotes provide technical information about individual state or county practices that may be germane to the Measure.
  • Legal context. We include relevant statutes (and links) to explain specific differences in state policies that affect criminal justice (e.g., state A uses sentencing guidelines while state B doesn’t, etc.).
  • County Context. County context includes information on demographics, population, crime, criminal justice resources, poverty, commute times, voting records, drug hospitalizations, and citizenship. This allows users to compare counties that are similar in size, demographics, etc.

    11

    How do you handle different collection and recording practices across and within states?

    We generally gather information about each state’s case management system to familiarize ourselves with local practices. When possible, we meet with local and state officials to discuss their data systems and specific guidelines for data collection and storage. We examine how each county names its data elements and groups its data, and use text-mining techniques to identify the multiple ways in which the same concept may be recorded using different terms across jurisdictions. As needed, we walk our data sources through how we got from local data sets to our finalized data sets to ensure accuracy and to fix any errors that might have occurred along the way.

    12

    Are counties given the chance to offer their feedback on your Measures?

    Yes. We meet state and local criminal justice stakeholders in person or we conduct webinars to explain our work and solicit input.

    13

    I am not sure what some of the Measures are trying to get at. How can I tell?

    We understand that not everyone who uses the Portal will be a criminal justice practitioner or criminologist and may need a little more context for why a Measure is important and how it plays into the criminal justice process. To help explain the bigger picture, each Measure includes a short and a long definition. The long definitions provide some additional background on the practices being discussed, and can be accessed by clicking the “information” button next to the Measure title. We want to get a conversation started about what the data show and why; that conversation begins with you.

    14

    Your goal is to cover the criminal justice system from arrest to post-conviction, but I don’t see any police data here. Why?

    We started with court data because they are the most widely available. But we are actively collecting law enforcement data and will have them available in the Portal when they are ready.

    15

    Why are some data missing?

    We are missing data for some Measures for several reasons:

    1. We have yet to collect all the data we need.
    2. We are still in the process of working with some of the data we have.
    3. We cannot use the data as provided, so we have excluded the Measure until better data are available.

    The Portal presents users with warnings when missing or unavailable data could be problematic, and we always report the percentage of missing values.

    16

    Will you update the data in the Portal?

    Yes. We will populate measures for which we don’t currently have data, add new states, and more years of data in the Portal once the data have been collected, processed, and audited.

    17

    Why do you use five years’ worth of data?

    We pool five years of data whenever possible because this allows us to include more counties in the Portal. If we used annual data, some small counties would not have enough data to present. When we are able to collect more years of data in the future, we will be better able to present trends over time.

    The Data Portal

    1

    Why did you decide to build a Data Portal?

    Our ambition was to design a way of displaying data that is comprehensive and detailed, but user friendly. We want to give users the opportunity to sift through an enormous amount of data and walk away with information that is actionable.

    2

    Who has access to the Data Portal?

    Everyone. We believe that transparency is critical for positive and productive dialogue among criminal justice stakeholders, citizens, and individuals who encounter the system every day. To this end, we think it’s important to make our data available without restriction.

    3

    What are people supposed to do with the data?

    MFJ hopes data on the Portal will:

    • give policymakers a new tool to inform their policy decisions, such as where to focus their resources and attention;
    • stimulate dialogue about practices and patterns that are relevant to how the justice system is functioning;
    • help the public understand and engage with the criminal justice system;
    • give the media and academics reliable data to support stories, reports, and research on criminal justice systems across the U.S.

    4

    Can I download the raw data?

    Our data sharing agreements prohibit the release of raw data and the identifying information they contain. But you can download and print out the results of any county data explorations you pursue on our platform.

    5

    Can you run me a customized report?

    At the moment, you can produce individualized reports quickly and with ease at the county level. We are currently not focused on customized reporting, but please contact us with your interest.

    6

    I think I see a problem in the Data Portal. What should I do?

    Please contact us with any issues, questions, or comments via our Contact page.

    7

    How do I use the Data Portal?

    Please see the wizard by clicking the '?' at the top of the Exploration page. Still not sure? Please contact us via our Contact page.

    8

    Do you make sure your data are secure?

    Yes. We follow industry standards’ best practices to make sure the data is secure.