Our Process

MFJ begins by collecting data from U.S. statewide court datasets. Often, these populate the majority of our Measures. If feasible, we also approach local agencies (prosecutors’ offices, sheriffs’ offices, public defenders’ offices, etc.) to acquire supplemental data in an effort to ensure we capture the entire criminal justice system. We do this in writing, but also, if possible, in person, via pairs of MFJ researchers who travel our states county by county.

MFJ understands that every county is different. Our process is iterative and responsive to stakeholder feedback. We continually seek input, local context, and information at multiple stages of our work to ensure our data are reliable and take into account county and state differences.

MFJ also recognizes that criminal justice data are not collected, recorded, or systematized uniformly across local jurisdictions. Thus, addressing data quality and uniformity has been a critical MFJ goal from the outset. We have developed a process to clean and code our data in-house and then run them through three audits--two internal and one external--to make sure they are correct. This process involves:

  1. Scrutinizing the data to identify data elements that may be unreliable given clear inconsistencies in data collection across counties.
  2. Exploring possible inconsistencies between legal and technical language used across and within states.
  3. Using text-mining techniques to identify the multiple ways in which the same concept may be recorded using different terms across jurisdictions.
  4. Addressing any anomalies, which often reveal that some jurisdictions may not be recording certain events and that, as a result, such events can’t be accurately measured.

Per these reviews, MFJ does not use any data elements identified as problematic.

Our complete methodology is available for download.

Data Portal

All our case-level data are aggregated to the county level so that the Measures can be calculated and centralized in an online repository—a public Data Portal—that is free and available to anyone. The Portal has been designed so that it’s easy to use but also comprehensive. We provide multiple views to satisfy users of all skill and interest levels (and no personal information is published for any parties involved in a case).

On the Portal, the Measures can be visualized as maps, bar graphs, and tables presenting county-level data. These views are available for download. Users can also compare counties within and across states.

Every Measure will be accompanied by:

  1. A short and long definition. The long definitions provide some background on the practices being discussed, suggest companion Measures, and note how the Measure was calculated and which cases were excluded.
  2. Footnotes. The footnotes provide information about individual county practices that are germane to the Measure. We acquire this information from counties or via research.
  3. Statutory baselines. We include relevant statutes (and links) to explain specific differences in state policies that affect criminal justice (e.g., state A uses sentencing guidelines while state B doesn’t, etc.). This kind of information helps to ensure our data are not misconstrued.
  4. Context. County context includes information on demographics, population, crime, criminal justice resources, poverty, commute times, voting records, drug hospitalizations, and citizenship.
  5. Opportunities to provide feedback. Users will be encouraged to send MFJ feedback on every page of our Data Portal. We will update our data and Portal as necessary based on user feedback.

User Notes

When viewing the measures users will be asked to keep the following in mind:

  • A STARTING POINT

    Our Measures are meant to be a starting point for a conversation about the criminal justice system that addresses what’s working well and what needs further attention. The aim is to create transparency.

  • ADULT CRIMINAL CASES

    Our system measures only the performance of counties on the processing of adult criminal cases. Therefore, we do not measure how juvenile, family, civil, and other cases may fare. Nonetheless, our Measures can be filtered by the age group of the defendant, including those under 18 (juvenile defendants who were waived to adult court).

  • FILTERS

    Our Measures can be filtered by defendant characteristics (race/ethnicity, indigent status, sex, and age) and by case characteristics (offense type, offense severity, court type, attorney type, and drug type--only for drug-specific Measures). We encourage users to explore the Measures using these filters. Some filters calculate disparities between two groups. However, we don't test the statistical significance of such disparities.

  • DATA QUALITY

    Measures for Justice (MFJ) works with data extracted from administrative case management systems. These data were originally collected by the sources for the purpose of tracking the processing of individual cases and not necessarily for the purpose of measurement. Nevertheless, they are suitable for measurement provided they are handled correctly. Often, these data are reliable. Just as often, they can be entered incorrectly or not at all, may be subject to errors at any stage of the recording and collection process, and may not be standardized across counties. MFJ has taken steps to account and adjust for these problems but cannot correct entirely for errors in data entry. For these reasons, and because jurisdictions use a variety of calculation methods, we encourage examining overall patterns instead of exact percentages.

  • CAUSATION

    MFJ’s research is descriptive and does not, by definition, tell us why things happen. As such, we do not test hypotheses about the reasons for the patterns the data reveal. When our Measures show differences between states, counties, or groups (e.g., in medians, percentages, or rates), we make no claim about the reasons for these differences, and do not test their statistical significance using inferential statistics tests.

  • CASE DEFINITION

    Criminal justice agencies use different methods to record cases. Some jurisdictions file all charges against a defendant under the same docket number and sometimes they do so even when the charges stemmed from different incidents. Others file each charge under separate docket numbers even when the charges are for the same incident. To standardize the definition of case across jurisdictions, we count all charges associated with the same defendant that were filed (or referred for prosecution, in the case of declinations) on the same date as a single case. We assume that when a prosecutor files multiple charges together, even when they originated from different incidents, they intend to resolve these charges simultaneously. Since the focus of our Measures is case processing, not case clearance, we believe this approach is currently the best way to standardize case definition across jurisdictions.

  • CASE SERIOUSNESS

    Because cases often involve multiple charges of differing severities, we define cases based on the most serious charge, according to the state's offense severity classification, that was present at each stage of the case processing, respectively referral, filing, and conviction.

  • CONTEXT

    Each Measure sheds light on a corner of a local criminal justice system, but to evaluate the health of that system in a more comprehensive way, all available Measures should be assessed together and interpreted with county context in mind.

  • COUNTY

    We measure criminal justice performance at the county level because it is usually at this level that charging, disposition, and sentencing decisions are made.

  • MORE DATA

    MFJ continues to seek out more data—especially law enforcement data—as part of our effort to measure all corners of the criminal justice system.

  • TIMELINE

    If you’ve given us data and don’t see them represented in the Portal yet, it’s because we are still working on them to ensure accuracy. Thank you for your participation and patience.

Methodology

methodology.pdf

Our complete methodology for creating the Measures in PDF format.