Module 9: Section 2: Evaluation Roadmap

An evaluation roadmap is a graphical summary outlining how you plan to evaluate your TJC initiative. Each step of the roadmap must be completed before the next step can occur. Here, the old adage applies: “If you don't know where you're going, you'll probably end up somewhere else.”

A theory of change model , often referred to as a logic model, is one of the most common tools used to map out your specific community's approach.
It will not only document your theory of change for the TJC initiative but articulate to your TJC partners the rationale for actions or strategies. It guides your self-evaluation by identifying how resources and activities or programs are linked to desired outcomes.

Your theory of change model will describe your TJC strategies and activities, and also help identify the data that should be collected by stating what measures will be used for its short- and long-term outcomes. A reentry council or criminal justice policy council, as well as each partnering agency, will typically develop its own theory of change model.

The flow of the theory of change model begins with identifying the target population and ends with long-term outcomes. We've listed the headings most often used for each column.

Figure 1. Theory of Change Column Headings

Figure 1. Theory of Change Column Headings

A Theory of Change Model:

  • Offers a written rationale or a theory for selecting various strategies, programs, or intervention activities to implement the TJC initiative.
  • Identifies the objective or change you hope will occur by implementing those various strategies, programs, or interventions to the population targeted.
  • States the strategies, programs, and activities to be implemented (Activities in figure 2).
  • Identifies the immediate, short-term outcome of the strategies, programs, or intervention activities (Outputs in figure 2).
  • Identifies the intermediate outcomes of the strategies, programs, or intervention activities (Outcomes in figure 2).
  • Identifies the final outcomes (long-term outcomes) of the strategies, programs, or intervention activities (Impact in figure 2).

Figure 2. Theory of Change Model

Figure 2. Theory of Change Model
(click for larger version)

Developing a theory of change model is a group activity; it requires input from multiple stakeholders central to your community's TJC initiative. Crafting a theory of change model begins by convening key stakeholders from the community and the jail to discuss and define the initiative's goals and objectives, and the interventions or activities your community views as essential to achieving these goals and objectives. As discussed in module 4, a data-driven understanding of local reentry issues should inform many of these early discussions.

A theory of change model, however, is only a first step. You will still need to define your key outcome measures, determine what data will be needed to measure these outcomes, decide how to collect and analyze the data, then do so, and report the results. See module 4 for more information on data collection and analysis.

Below, we have provided several examples of theory of change models, focusing on transitioning people back to the community, provided by the Criminal Justice Research and Evaluation Center at John Jay College.

Figure 3. Collaborative Initiative Program Plan
Criminal Justice Research and Evaluation Center, John Jay College.

Figure 3. Collaborative Initiative Program Plan
(click for larger version)

Figure 4. Reentry Committee Community Capacity Building—Process Logic Model
Criminal Justice Research and Evaluation Center, John Jay College

Figure 4. Reentry Committee Community Capacity Building—Process Logic Model
(click for larger version)

Figure 5. Mother and Child, Jail and Community Connections
Criminal Justice Research and Evaluation Center, John Jay College

Figure 5. Mother and Child, Jail and Community Connections
(click for larger version)

Performance Measures

The development of TJC performance measures is the next step after developing your theory of change model. Performance measurement refers to the “regular and systematic collection of quantitative information that will empirically demonstrate results (outcomes) of activities (e.g., modified policies, practices, new program activities). Performance measurement connects indicators (i.e., quantitative measures) with specific agency or jurisdictional objectives (i.e., expected outcomes).” 1

Performance measures determine the type of data you must collect to measure your short, intermediate, and long-term outcomes and are thus directly connected to your theory of change model. Reports of these data and outcomes should be readily available to assist decision makers in resource allocation and implementation strategy.

David Osborne and Ted Gaeble, in Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector 2 identify the power of performance measures:

  • What gets measured gets done.
  • If you don't measure results, you cannot tell success from failure.
  • If you can't see success, you can't reward it.
  • If you can't reward success, you're probably rewarding failure.
  • If you can't see success, you can't learn from it.
  • If you can demonstrate results, you can win public support.

Innovating agencies use performance measures to improve performance but they also recognize that for desired behaviors to replace old habits or behaviors, they must be reinforced. Below are several examples of actions or rewards to reinforce desirable organizational behaviors or actions:

  • Paying for performance: Use performance measures as a basis to determine and reward effectiveness.

    The New York City Department of Corrections, for example, employs performance-based contracts with its transitional service providers. Service providers bill the NYC DOC for individual clients, and the billing structure relies on a pay scale that ties increased fee amounts with continued client involvement. Performance-based contracts provide strong incentives for service providers to be aggressive in their efforts to maintain client engagement.

  • Managing for Performance: Use performance measures to improve outcomes through the following steps: identify problems, analyze them, locate the root cause, and develop and implement the solution.
  • Budgeting for Results: Use performance measures and information to allocate resources. As table 1 indicates, results-based budgeting is outcome driven, promoting ongoing evaluation and interagency collaboration.

Table 1. Traditional Budgeting vs. Budgeting for Results 3

Incremental or Traditional Budgeting Results-based Budgeting
Focus is on the allocation of “new monies” only Focus is on nearly all monies or the entire budget (excepting certain obligations, such as debt)
Concentration is on inputs (what you buy), that is, “objects of expenditure” Concentration is on outputs (what results are expected)
Narrow or marginal decisionmaking Comprehensive or enterprise-wide decisionmaking
Subjective based Objective based
Preserving the status quo Determining new, creative approaches to problems and needs
Agency or bureaucracy driven Outcome driven
Promotes restraints, restrictions, and red tape Encourages flexibility and ingenuity
Control orientation Planning and management orientation
Emphasizes compliance and preserving legality Emphasizes performance and innovation
Stresses audit trails and conformity Stresses program evaluation and improvement
Involves agency heads, elected officials, and advocacy groups Involves everyone wanting to participate, especially those wearing a “citizen's hat”
Encourages and perpetuates single-agency programs Encourages intra- and interagency cooperation among programs and activities

Begin by developing TJC performance measures for each of your theory of change model's short-term, intermediate, and long-term outcomes. This needs to be done before you develop your data collection plan to make sure your strategy for collecting data supports each performance measure used in your self-evaluation. There must be a clear and compelling link between your initiative's objectives, outcomes, performance measures, and indicators, and the data you plan to regularly collect and analyze.

Many people believe that rearrest, reconviction, and reincarceration rates are the only ways to measure successful transition strategies. Although long-term public safety is paramount, there are other important process and system outcomes to measure to enhance public safety and efficiency.

The TJC initiative has developed a menu of performance measures that reflect and support the initiative's broad goals to increase public safety, improve reintegration outcomes, and effect systems change. To help you with this process, we have identified the following system-level, public safety, and reintegration TJC initiative outcomes and performance indicators:

  • System-Level Outcomes
    • Screening, assessment, referrals, engagement, service utilization, and completion
  • Public Safety Indicators and Outcomes
    • Reduced reoffending, reduced jail stays, and increased time between jail stays
  • Reintegration Indicators and Outcomes
    • Reduced substance use, reduced homelessness and increased housing stability, increased employment and employment stability, and improved physical or mental health

The menu of TJC performance measures located at the end of this module offers several indicators (actual measures) for each above measure. The process measures, although difficult to track, should be a priority for each jurisdiction as they will allow progress to be monitored on an ongoing basis. Depending on the agreed upon definitions (e.g., indicators selected, specified time frame for measurement), the public safety and reintegration measures can take a long time to demonstrate progress and success. The process measures represent intermediate outcomes that should be monitored closely, keeping in mind that if the associated activities are targeted and implemented correctly, they should positively affect reintegration and public safety outcomes.


1 Rossman, Shelli B., and Laura Winterfield. 2009. Coaching Packet: Measuring the Impact of Reentry Efforts. Silver Spring, MD: The Center for Effective Public Policy, p. 8.

2 Osborne, David, and Ted Gaeble. 1992. Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector. Reading, MA: Addison Wesley.

3 Office of the Governor. 2005. The FY 2005-06 Executive Budget . Columbia, SC:
Author.

System-Level Performance Measures

This menu of system-level performance measures helps you identify those that are most important to your initiative. You may want to collect and review these data monthly to support internal monitoring but report on them quarterly to the broader stakeholder group to track outputs and short-term and intermediate outcomes.

System-Level Performance Measures

Goal Outcome Data Source Performance Measures

System-level change

Improve the frequency of risk, need screening, or assessment

Agency data

- Number or percentage of clients receiving screening in jail and community
- Number or percentage of clients receiving comprehensive risk needs assessments in jail and community

 

Increase transition planning for medium- and high-risk offenders

Agency data

- Number or percentage of medium- and high-risk clients receiving a transition plan
- Number or percentage of transition plans updated after release

 

Increase multiagency partnerships

Quarterly assessments, surveys, agency data

- Number or percentage of partnership agreements formed between the jail and the community

  1. Number of partner assets and needs

- Degree of continuity of practice between partner agencies
- Degree of integration of electronic and Information Management System
- Amount of reimbursement income from state and county entities
- Number of referrals to partnering agencies
- Degree of partner access to data systems, where relevant and appropriate
- Degree of efforts to establish a system data repository or database to which all partners contribute and have equal access
- Degree of trust, quality of communication, and partner-to-partner activities that are formalized through a criminal justice council or similar body
- Understanding of each partner's role and their agency's importance to the success of the system partnership
- Number and type of protocols and processes for referring clients
- Cost-benefit analysis of the TJC activities

Service engagement and use

Increase in participation in programs and services

Agency data, self-report

- Number or percentage of risk and needs-based jail programs
- Number or percentage of new risk and needs-based jail programs since TJC initiative began
- Number or percentage of target population referred to services (monthly), by service type
- Number or percentage using detox and treatment programs
- Number or percentage provided access to mental health counseling and services
- Number or percentage attending services or programs
- Number or percentage of days or sessions attended during the specified period
- Number or percentage of completing programs or services by program type
- Number or percentage not completing by reason for exit
- Number or percentage of days participated by completer or non-completers
Number or percentage of high-risk offenders targeted for services

Public Safety Performance Measures

Goal Outcome Data Source Performance Measures

Public safety

Reduce recidivism

Booking records, agency data

- Number or percentage of clients that remain crime-free for specified time (3,6, 9, or 12 months after release) as measured by new arrests, new convictions, and/or new incarcerations

 

Reduce
reoffending

Booking records, agency data

- Number or percentage of arrests and violations
- Number or percentage of arrests and violations for specified time (3,6,9,12 months)
- severity of new offenses/crimes

 

Reduce jail stays

Booking records, agency data

- Average length of stay by risk or need
- Number or percentage of repeat jail stays
- Number or percentage of two or more jail stays by age, sex, race, offense type & severity
- Time between repeat jail stays

 

Change in classification scores

Agency data

- Change in risk and need levels
- Integration of risk or needs information with systems of classification, examination, or sanction

Restorative justice measures

Increase in restitution collected

Agency data

- Hours of community service completed
- Number or percentage of clients in compliance with child support obligations

Reintegration Performance Measures

Goal Outcome Data Source Performance Measures

Community reintegration

Reduce drug and alcohol use

Urinalysis, self-report,
agency data

- Number or percentage testing positive for drugs or reporting use at screening or assessment
- Severity or frequency of substance use from screening and assessment results
- Number or percentage who have not used any substance for specified period (3, 6, or 12 months)
- Number or percentage of relapse episodes per client and number of days, weeks, or months between events
- Number or percentage enrolled in, or completing detoxification, residential, or out-patient substance abuse programs
- Number or percentage of positive drug tests or individuals who test positive
- Number or percentage applying for treatment upon release
- Number or percentage of treatment sessions completed
- Level of treatment enrollment (e.g., inpatient, outpatient).
- Number or percentage enrolled in aftercare and peer support groups to sustain sobriety and recovery

 

Improve behavioral healthcare

Agency data

- Number or percentage of mental health assessments received
- Number or percentage of clients with improved mental health functioning based on some standardized scale
- Number or percentage of psychiatric hospitalizations
- Number or percentage applying for treatment upon release
- Number or percentage of treatment sessions completed
- Level of treatment enrollment
- Number or percentage of former inmates who continued in program at 30-day intervals

 

Reduce homelessness

Homeless database, agency data, self-report

- Number or percentage of homeless clients
- Number or percentage of shelter stays/nights on the street in specified period (3, 6, or 12 months) after intervention
- Number or percentage of clients with a fixed address (own apartment/home)
- Number or percentage and descriptions of post-housing and shelter institutional disciplinary issues
- Changes in patterns of jail and shelter usage in the first 90, 180, and 365 days

 

Increase access to safe housing placement and retention

Homeless database, agency data, self-report

- Number or percentage of attempts at housing out-placement
- Number or percentage of housing placements
- Number or percentage of placed retaining housing

 

Increase access to benefits

Agency data, self-report

- Number or percentage of benefit-usage of clients
- Number or percentage of identification documents already possessed by clients at intake
- Number or percentage of clients enrolled in public benefits (e.g., SSI, Medicaid & food stamps)
- Number or percentage of clients maintaining enrollment in all eligible and appropriate benefits

 

Increase educational obtainment

Agency data, self-report

- Number or percentage of clients who participated in and completed vocational training
- Number or percentage attaining education (e.g., adult basic education, completed GED, pursue higher education)

 

Increase Employment

Agency data, self-report

- Number or percentage clients placed in jobs
- Number or percentage of clients employed
- Number or percentage of days employed
- Wages and benefits earned and taxes paid
- Number or percentage of days clients retain their jobs during specified period of time
- Degree of full-time employment (for those that need it)
- Degree of job stability over time (decrease in number of job changes)

 

Improve physical healthcare

Agency data, self-report

- Number or percentage enrolled in a health plan using a publicly funded health insurance program
- Number or percentage receiving a full physical followed by preventive health services and appropriate treatment for chronic conditions
- Number or percentage with hospital stays due to medical issues, drug use
- Number or percentage who use necessary health care services after release
- Number or percentage who attend appointments
- Number or percentage who complete treatment
- Number or percentage of contacts with primary care physicians
- Number or percentage of emergency room visits
- Number or percentage of medication adherence
- Number or percentage of testing for chronic and infectious diseases

 

Improve family and
community engagement

Agency data, self-report

- Number or percentage who acquire new skills – Job skills, trade/vocational skills, life skills (decisionmaking, social communication), English language
- Number or percentage who have strong positive social support networks
- Number or percentage with improved self-concept, self-awareness, and system awareness
- Number or percentage engaged prosocially (e.g., voting, community involvement, volunteer work)
- Number or percentage managing their finances (e.g., open bank account, paying bills on time, building credit)
- Number or percentage rebuilding prosocial family relationships and engagement
- Number or percentage initiating contact with prosocial family members
- Number or percentage paying child-support
-Number or percentage increasing child-support payments
- Number or percentage in compliance with child-support obligations.
- Number or percentage maintaining basic responsibilities (e.g., keeping appointments)

Data Collection

While the TJC initiative does not require a fully developed or collaborative management information system (MIS) in place for self-evaluation or inter-agency information-sharing, your agency should have the basic protocols and capacity to collect and access information relevant to operations and outcomes. An MIS is needed to measure performance and program success along desired outcomes through an external evaluation and a self-evaluation. This system will allow for each community to determine the who, what, where, and when of targeted service delivery. Refer back to Module 4: Data-Driven Understanding of Local Reentryfor Step 2: Data Collection.

Resources

1. Burke, Peggy. B. 2008. TPC Reentry Handbook: Implementing the NIC Transition from Prison to the Community Model. Chapter 6, “TPC Performance Measurement Framework."

2. Kent County, MI Sheriff’s Office. TJC mission-driven system-wide performance measures and outcomes.

3. Miami-Dade County, FL. 2008. Breaking the Cycle: Rehabilitation and Job Training in County Jails. Miami-Dade County Blue Ribbon Advisory Committee. A comprehensive report recommending initiatives to facilitate the reentry of inmates into the community and reduce recidivism.

4. Demographic characteristics the Montgomery County, Maryland, Correctional Facility uses to understand more fully the needs of its population.

5. University of Wisconsin Cooperative Extension. 2002. Enhancing Program Performance with Logic Models.

6. Urban Institute. TJC Performance Management Worksheet: Core. A detailed chart of TJC baseline measures of jail inmate population characteristics in Excel format and a memo providing guidance to assemble the initial TJC performance indicators.

7. Urban Institute. 2009. TJC Initiative Theory of Change Model. Brief explaining Theory of Change models.

8. W.K. Kellogg Foundation. Logic Model Development Guide.

Let's Review

Let's revisit what we have learned so far in the Self-Evaluation and Sustainability module. Please answer the following question.

Which of the following is not part of a theory of change model:

The purpose of the program and its overall objectives.

A list of activities used for other programs that are out of budget range.

The expected outcomes that you have determined your program will achieve.

Long-term impacts that describe the ultimate changes expected.

Summary

Now that you have completed this section, you understand that by developing a theory of change model and using performance measures to determine the success of your outcomes, you can demonstrate that your program is based upon a specific theory, has clear goals and objectives, and that if a comprehensive process is followed, you are likely to see measurable program successes and failures from start to finish.