Introduction

The development of a “system map” is a first step in developing a common understanding of how the criminal justice process “works” in a jurisdiction. When completed, the system map reflects the flow of cases through the justice process, key decision points, decision makers, and the volume of cases flowing through the system during a particular period of time. And while it can surface system strengths and challenges, it is the first step to—rather than a substitute for —a more in-depth examination of the policies and practices that underlie the workings of the system.

Following the completion of the system map, a process to “dig deeper” into these decision points should be undertaken. This more in-depth analysis includes an examination of the following as they relate to each decision point:

• written policies;

• the application of those policies to practice, as well as other operational practices that are not formally articulated in policy;

• the various types of data and information collected at each decision point;

• the ways in which this data and information inform decisions;

• the ways in which information is stored and shared;

• the extent to which evidence-based information and research is available and used to make decisions;

• gaps and barriers that impede the use of evidence-based knowledge to inform these decisions; and

• other factors related to using data, information, and evidence in the most efficacious ways.

Purpose

Most matters of public policy are guided in part by an array of formal policies contained in federal or state law, county code, case law, individual agency policy, or memoranda of understanding among multiple organizations. An assessment of current policy begins with the identification of these operating guidelines. Formal policy only begins to define the manner in which decisions are made and processes are carried out; in most communities, written policy guides only a small portion of activity. When formal policy leaves activities undefined, informal practices emerge to fill the gaps. Informal practices are sometimes developed with clear purpose and great care; in other instances, they simply evolve over time. Developing an understanding of informal practice is as critical as understanding current policy.

A policy and practice analysis serves to objectively assess the extent to which

• clear, written policies dictate key decisions and practices;

• policies are consistent with evidence-based knowledge;

• policies are consistently carried out, consistent with one another, and support their stated purpose(s), agency missions, and jurisdiction vision;

• gaps in policy result in discretionary practices, and the degree to which that discretion results in practice that is evidence-based, consistent, and in support of vision and mission; and

• the collection, storage, sharing, and application of data and information support effective decisions.

Participants

A policy and practice analysis can be conducted by staff internal to the agency(ies) under examination, by staff from colleague agencies within the jurisdiction, and/or by external assessors. The latter approach offers the opportunity to bring to the jurisdiction the perspective of objective subject matter experts. While this might be the ideal approach, it may not always be possible.^[1]

Instructions

Using Outside Experts to Conduct an Analysis

1. Discuss the purposes of the policy and practice analysis. Make sure the team understands that the analysis will be conducted by individuals with expertise in specific areas who will assess the extent to which current policies and practices regarding key decision points are supported by research. Against that backdrop, engage the team in a discussion about their priorities for the analysis (i.e., What would you like to understand about these decision points? What would you like to have more information about? What (if any) change targets can you envision regarding the decision points? What information can be derived from the analysis to better inform the team about those potential targets?).

2. Discuss with the policy team the process and timing of the analysis: who will conduct it, the expertise of the individuals involved, the anticipated duration, schedule of interviews and observations, and the method of recording and reporting the findings.

3. Agree on an approach to informing affected agency staff of the purposes and timing of the assessment.

4. Ask the outside experts to provide a report following the assessment that outlines key findings and specific recommendations for changes that the site can make to further align itself with the most current research. Ideally, a debrief session with the policy team will be conducted to enable the team to ask questions, deliberate over findings with the aide of subject matter experts, and develop strategies for action.

Process for Conducting a Policy and Practice Analysis through Self-Assessment

1. Form subcommittees (or work groups) to conduct assessments of discreet areas (e.g., agencies, decision points, a combination of these).

2. Across all subcommittees, develop a shared sense of purpose for this process—what information is to be collected and why, an agreed-upon set of “ground rules” for carrying out the process (e.g., ethical and professional behavior during and following the analytic process, confidentiality limits and expectations, etc.), the timeline, the final product subcommittees are to produce, and how and with whom these final products will be shared.

3. Identify a “chair” for each subcommittee. This person should be responsible for calling together their subcommittee, facilitating their meetings, keeping the subcommittee on task, and sharing information with the policy team to ensure good communication, coordination, and information flow.

4. Subcommittees should become familiar with the relevant research in their area(s) as well as emerging practices that have yet to be demonstrated empirically (if they do not already have familiarity in these areas). This knowledge will be the “backdrop” against which the analysis will be conducted.

5. Be clear with subcommittees that part of their role is to educate the policy team members on their topic. The goal, in the end, is to have a common, core base of knowledge across all policy team members.

6. Create mixed-discipline subcommittees. For example, do not assign only those team members who are involved in law enforcement to the cite/detain decision point. Doing otherwise will provide a unique opportunity for cross-disciplinary learning and for an “outsider” to ask questions that those entrenched in current processes may not consider.

7. Early subcommittee meetings should be focused on developing a detailed work plan and timeline for conducting the analysis.

8. Subcommittees should continue to meet through the information collection and analysis process to ensure that the work is being carried out as planned, data is recorded in a useful manner, and difficulties with the information collection and analysis process are addressed early.

9. At the conclusion of the subcommittee’s work, it should be prepared to present its findings and recommendations to the entire team.^[2]

Tips

Consider the following tips for collecting information effectively:

• The quality of the information collected through this process will have tremendous bearing on the action planning decisions of the team. Seek quality information over quantity if time constraints are a factor. To that end, if it is not possible to thoroughly assess all decision points, agree with the policy team on the initial priorities for assessment.

• The information gathering process should be used as an opportunity to educate those outside of the formal team about the vision, mission, and goals of the policy team, and to learn their perspectives on the strengths and gaps in the jurisdiction relative to evidence-based decision making. Doing so is likely to both elicit cooperation and engender long-term support for the team’s work.

• If a team chooses to work together on all of the component sections rather than working in subcommittees, a work plan and timeline for each section should be developed. Dividing the work among individual group members and encouraging them to answer the questions independently—without at least processing them with the team—is not recommended. Doing so will provide only one person’s perspective, and is unlikely to give an accurate or comprehensive picture of policy and practice across the jurisdiction.

• Do not rely solely on the expertise of policy team members and/or subcommittee members to conduct the analysis. Instead, collect multiple data sources to ascertain current policies and practices. Ideally, interview decision makers regarding current policies and practices; review documents (i.e., written policies, procedure manuals, programmatic materials, forms, and data screens); conduct individual interviews or focus groups with supervisors and staff; and observe the activities being assessed. This approach will provide the broadest view and most reliable information. (For instance, in assessing how pre-trial release recommendations are made, review written statutes, bail schedules, and agency policies; interview the pre-trial administrator, chief prosecutor, and bail commissioner; interview pre-trial release officers; and observe pre-trial interviews, verifications, the development of reports, and bail review hearings).

• Expect variability in practice. It is not unusual to find that there is a lack of uniformity within a jurisdiction, with some practitioners or agencies doing things one way and others doing it completely differently.

• In those cases where numerous individuals are involved in carrying out a particular function, committees might consider convening a focus group to understand the variety of approaches in use. For example, it might be helpful to identify those prosecutors handling violations of parole/probation cases and to interview them together in a focus group format to understand their management of these cases.

Example:
EBDM Work Group Charter for Assessing Evidence-Based Policy, Practice, and Decision Making

What are we trying to achieve through the work group process?

The role of the work group is to educate and advise the EBDM policy team on strategies that will result in the greater use of evidence (research) to support decision making consistent with the team’s vision for the justice system.

What do we mean by “evidence”?

In the justice system, the term “evidence” is used in a variety of ways. ”Evidence” can refer to items collected at a crime scene, eyewitness accounts, or security camera footage. These types of evidence are referred to as legal evidence. For the purposes of the EBDM initiative, the term “evidence” is used to describe findings from empirically sound social science research. The Framework refers to the results of this research as evidence-based policy and practice.

What is “evidence-based decision making”?

Evidence-based decision making is the use of evidence (as defined above) to inform decisions throughout the criminal justice system process at the case level (e.g., applying practices in light of offenders’ risk level), at the agency level (e.g., providing agency direction and support that results in probation officers spending 20 minutes or more in their offender contacts, and focusing those contacts on risk reduction techniques), and at the system level (e.g., working collaboratively to collect and analyze data to determine if systemwide policy decisions are resulting in the desired outcomes, and making adjustments as needed).

Who should be on the work group?

The work group should be composed of individuals who represent all agencies that affect or are affected by the decision points the group is tasked with addressing. The selected individuals should represent diverse experiences and points of view in order to ensure the broadest perspective on the topics discussed. The work group serves as a standing committee that seeks input from others (“ad hoc members”) as needed.

What is the role of the chairs or co-chairs?

Chairs or co-chairs are tasked with leading and organizing the work group. To that end, it is their responsibility to make certain that they and their group members are clear about their tasks and the work they are undertaking and, based upon this understanding, develop a work process that will be successful in accomplishing specific tasks within the designated time frame. It will be up to each set of work group chairs to determine how and when the work group will meet. The work group chairs will also be responsible for reporting progress to, and sharing their products with, the EBDM policy team.

What is the work group’s time frame?

Work groups should be prepared to complete their work by April 2011. If completion is possible sooner, this is desirable, but not if it means sacrificing quality of work.

What tasks are the chairs/co-chairs and their work groups expected to accomplish?

1. Identify the agencies that/individuals who should be on the work group and secure their agreement to participate.

2. Ensure that all work group members have read the EBDM Framework and EBDM project overview documents, and are familiar with relevant portions of the Mesa County system maps.

3. Ensure that all work group members are aware of the EBDM policy team’s vision statement and have read the team’s charter.

4. Ensure that all work group members are clear about the work group’s tasks and purposes.

5. Create and manage a work plan (e.g., recruit members, establish a schedule of meetings that works for members for the next 2–3 months, clarify the tasks and work to be undertaken).

6. Review and ensure that the system map and system map notations are complete and accurate.

7. Identify all the decision points and decision options on the system map that are within the domain of the work groups. Ensure that relevant decision points are reflected by a diamond shape.

8. Educate yourselves! Identify and review the evidence-based practice literature that most applies to the decision points for which your work group is responsible.

9. Discuss the extent to which evidence-based information is available and used to inform each of these key decisions. For each decision point, ask yourself the following:

   1. What risk and harm reduction measures are we seeking to achieve (or contribute to in major ways) at this decision point?

   2. What guidance does the existing research literature have to offer at this decision point?

   3. What can be done to ensure that research informs decisions made at this point?

10. Identify gaps and barriers that impede the effective use of evidence-based knowledge to inform these decisions (e.g., offender risk/need information is collected but not used to inform a particular decision; risk/need information is not collected; or research suggesting that individuals identified as low risk through the use of an actuarial risk tool is not used to inform the intensity of the applied criminal justice intervention).

11. Identify further information needs that relate to #8 and that would inform the work group’s discussions (e.g., “We know that community corrections has risk/need information, but we need further information about how this information is used (i.e., consistently? accurately? appropriately?) and the extent to which it supports the outcomes we desire” or “We know that we have a variety of intervention options, but we do not know the criminogenic needs (risk factors) these interventions address or the risk levels of the individuals who are receiving these services”). Ask your work group’s data committee member to assist with getting answers to these questions.

12. Identify how best the information in #9 would be collected. Can the work group do this, or someone else within the Mesa County justice system or county? Is support needed from an outside expert?

13. Identify change targets and opportunities that will support the EBDM policy team’s efforts to achieve its vision.

14. Keep the executive committee informed of your progress and of any issues/problems that need the input and advice of the policy team.

[1] Jurisdictions might explore whether technical assistance is available for this purpose.

[2] See 3e: Prioritizing Your Team’s Targets for Change.