Introduction
For a jurisdiction to be truly evidence-based, the professionals managing its justice system agencies[1]—and those working directly with the defendants and offenders in that system—should all share a common base of knowledge about key findings in the correctional literature. A knowledge survey can serve as a diagnostic tool to determine the level of current knowledge among these parties and potentially identify needs for knowledge enhancement. If it is determined that further knowledge development is needed, steps should be included in the jurisdiction’s strategic plan to enhance knowledge among key stakeholders and agency staff. This document provides a process for administering a knowledge survey and considering the implications of its results for the team’s future work.
Purpose
To assess the level of evidence-based knowledge among policy team members and of select groups of staff within their agencies.
To identify action steps to address gaps in knowledge as identified through the survey.
Participants
While the policy team will ultimately decide who will take the survey, it is recommended that knowledge survey participants include a sample of staff from all agencies represented on the team.
Instructions
Measuring the Impact of an Event
In Yamhill County, Oregon, the policy team used the EBP knowledge survey to measure changes in knowledge regarding EBP after a day-long awareness-building event for system stakeholders.
A greater percentage of respondents answered 17 out of 18 questions correctly following the event than prior to the event. These findings suggest that participation in this event had a positive impact on participants’ level of knowledge about various evidence-based practices.
During a policy team meeting, discuss the reason(s) why a knowledge survey is important to the team’s work and how it will aid in the development of the team’s strategic plan.
Determine what kind of knowledge (i.e., information that has been demonstrated to rise to the level of “evidence” and therefore be used to inform decision making) the team wants its members to know. From this feedback, either develop a knowledge survey or use the one in Appendix 1.
If the policy team decides to develop its own knowledge survey, it is recommended that the survey be reviewed by a respected researcher and pilot tested with practitioners who are knowledgeable about the research before it is administered.
Distribute the knowledge survey to policy team members.
Survey respondents should not be coached in advance as to the specifics of the survey or possible answers. Results will only reflect accurate indicators of existing agency knowledge if the survey is administered without foreknowledge of content.
Explain that not all agency staff will be surveyed. Instead, a selected group of policy and line level staff will be anonymously surveyed.
Secure a commitment from each policy team member to fully participate in the survey process.
Following the team meeting, discuss with each agency’s lead policymaker[2] which of their staff members should participate in the survey. It is recommended that only those who are in charge of policy decisions (i.e., directors/managers/supervisors) and those with direct contact with defendants, offenders, and victims be included in the survey.
Calculate the number of staff persons who will participate in the survey, based on the criteria agreed to in #3. This will be the total survey “pool” for each agency. If possible, the survey should be sent to all individuals in this survey pool.
The survey may be distributed in hard copy or through an automated online survey mechanism such as Survey Monkey.
Communicate with agency staff in advance about the purpose of the survey. Some recommended email scripts are provided in Appendix 2.
Once the survey results are compiled, review them during a policy team meeting.
Determine how confident you are that the survey results reflect the knowledge of the pool of staff in each agency. Keep this in mind as you discuss the results. (For instructions on how to do this, see Appendix 3.
Determine as a team the threshold for a satisfactory and unsatisfactory knowledge level.
Determine whether the survey results were satisfactory (as determined by the team). If not, for which agencies were the results unsatisfactory?
For those agencies with unsatisfactory knowledge survey scores, determine what action steps the team wants to take.
If one or more action items are identified to enhance knowledge, consider, as part of your action plan, re-administering the knowledge survey after these efforts are completed to determine whether the knowledge gaps have narrowed. In this case, use the same process to determine the survey sample for re-administration as you used for the initial survey. The same individuals do not need to be surveyed; however, the same protocol for identifying the survey group must be followed.
Tips:
Using an EBP Survey with Agency Leaders
In Charlottesville-Albemarle County, Virginia, the director of the local community corrections agency used the knowledge survey with her statewide agency heads to test their knowledge of EBP principles and practices, and gave out prizes for the highest scores.
Make sure that survey participants understand the purpose of the survey. (See the email template below.)
Allow, at a minimum, two weeks (10 business days) for participants to complete the survey.
Send an email to the survey participants at least two days before the beginning of the two-week survey period to let them know that they are being asked to participate in the survey. (Make sure the survey is up and running early, in case participants want to fill out the survey immediately!) It is preferable that this email be sent by the agency policymaker or by another person in a leadership position. See Appendix 2 for ideas about what this email should contain.
Send out one reminder during the two-week survey period.
If, after ten business days, all of the surveys are not returned, send out another reminder indicating that, although the deadline has passed, it is important that each person participate and, therefore, the deadline has been extended for three more business days.
Additional Resources/Readings
The knowledge survey contains information collected from numerous research studies. See NIC’s Evidence Based Decision Making Framework[3] for research studies that include some of the key findings.
Appendix 1:
EBP Survey with Answer Key
Evidence-Based Practices Survey | Answer | |
|---|---|---|
1 | True or false? Most offenders don’t handle stress well, so anxiety and stress reduction programs such as yoga and meditation are helpful in reducing the likelihood of rearrest. | F |
2 | Of the following, which is the most effective method to determine the likelihood that an individual will be rearrested?
| A |
3 | Which of the following interventions best reduces recidivism in the long term?
| D |
4 | To best achieve offender behavioral change, professionals should use social learning techniques that reduce recidivism. Which of the following is not a social learning technique?
| E |
5 | Which of the following statements is true?
| A |
6 | Which individual trait or circumstance, when present in a person’s life, does not likely contribute to whether that person commits a crime?
| A |
7 | True or false? Placing offenders with low self-esteem in programs that increase their confidence does not reduce the likelihood of rearrest. | T |
8 | True or false? It is generally true that treatment does not work in reducing rearrest rates. | F |
9 | True or false? Lack of employment is among the top four influences that can lead an individual to commit a subsequent crime. | F |
10 | True or false? Lack of education is among the top four influences that can lead an individual to commit a subsequent crime. | F |
11 | Which statement is most accurate when trying to maximize the effectiveness of programming for medium and high risk offenders?
| D |
12 | True or false? Failure of a sex offender to register has a direct correlation to convictions for new sex crimes. | F |
13 | Which one of the following programs reduces recidivism over the long term?
| F |
14 | True or false? It is better to invest in interventions for low risk offenders than for high risk offenders because their criminal tendencies are less hardened, or “fixed.” | F |
15 | True or false? Programs like “Scared Straight” and boot camps are particularly effective for youthful offenders between the ages of 16 and 25. | F |
16 | True or false? Jails and prisons are effective in changing future offender behavior after release if the conditions are severe enough that the offenders don’t want to return. | F |
17 | True or false? Giving offenders positive reinforcement and feedback when they exhibit prosocial behaviors supports positive changes in the future. | T |
18 | True or false? Of those probationers who are revoked from supervision and incarcerated, most are not revoked from supervision for new crime behavior. | T |
| 19 | True or false? Medium and high risk offenders benefit more from punishment than treatment. | F |
| 20 | Research suggests that the onset and use of drugs is different for female offenders than for male offenders. Which one of the following statements is not true about female offenders when compared to male offenders?
| C |
Appendix 2:
Email Templates for Conducting the Knowledge Survey
Email 1
Send this email a few days before the survey period. (Make sure that the survey is up and running before you send this email!)
Dear Colleague,
We would like to ask you to complete a short, online survey. It should take no more than 10 minutes of your time. The purpose of this survey is to determine the extent to which staff in our organization share common knowledge about key justice practices that are supported by research.
A number of policy level and line staff members have been selected to participate in this survey, including managers and those who work directly with defendants/offenders. Your answers will remain completely anonymous.
To take the survey, please click on the following link: xxxx.
The survey will be open for the next two weeks. If you have questions about how to take the survey, please contact Name and Email.
Thanks in advance for your cooperation.
Agency Leader Name
Email 2
Send this reminder email about halfway through the survey period.
Dear Colleague,
If you have not completed the Evidence-Based Practices survey, please do so as soon as possible.
To take the survey, please click on the following link: xxxx. Please complete this survey by: date.
Thanks,
Agency Leader Name
Email 3
Send this reminder the day after the survey period ends.
Dear Colleague,
Although the deadline has passed, we feel it is important that each person participate in the Evidence-Based Practices survey and therefore the deadline has been extended for three more business days.
If you have not already completed the survey, please do so by date by clicking here: xxxx.
Thanks,
Agency Leader Name
Appendix 3:
Determining the Minimum Number of Responses Desired for Accurate Inferences
In order to make more accurate inferences about the larger population, it is necessary that a certain percentage of the individuals in the target pool complete the survey. To determine the minimum number of responses necessary for confidence in the results, visit the following website: https://www.surveysystem.com/sscalc.htm#one. (This website provides a tool for determining survey sample sizes.)
Follow these instructions:
Only use the box entitled “Determine Sample Size.”
Click on the button marked “Confidence Level 95%.”
Enter the number 10 in the “Confidence Interval” box.
Enter the number of persons in the survey “pool” (as calculated in #4 above) in the “Population” box.
Click “Calculate.” The number displayed is the minimum number of respondents that are necessary to get results that reflect the target population.
Once this process is completed for each agency, make a list of the survey targets for the policy team using a chart similar to the one below. This will help document the survey pool, the number of surveys received, and level of confidence in the results.
Agency | Total # of Staff | Total # of Staff in (from #4 above) | Minimum # of | # of | Received More than the |
| Court | 12 | 10 | 9 | 4 | No |
| Prosecution | 53 | 46 | 31 | 37 | Yes |
| Police | 324 | 217 | 67 | 121 | Yes |
[1] Throughout this document, the term “agency” is used to represent independent entities such as a police department, court, organization of public defenders, community corrections agency, etc.
[2] For the purposes of this document, the “lead policymaker” is the person on the policy team representing his/her agency (e.g., chief of police, chief public defender, chief judge).
