Training Evaluation – Managing Bias

CNA is partnered with an independent scientific research team at Washington University in St. Louis. The evaluation team is led by Dr. Calvin Lai, Dr. Katie Lewis, and Grace Drake from Washington University in St. Louis. Dr. Lai is an associate professor of Psychological & Brain Sciences at Washington University in St. Louis. Dr. Lai’s research has focused on preventing racial discrimination in law enforcement. He has also advised law enforcement agencies on police training and hiring policies. Dr. Lewis is a postdoctoral research associate and Grace Drake is a graduate student in Dr. Lai’s lab. They are the main points of contact for the evaluation team. Dr. Lewis and Grace are happy to address any questions you may have and can be reached at lewis.k@wustl.edu and drake.g@wustl.edu.

What is the evaluation and what does this look like? The Managing Bias training is being evaluated in two ways and agencies will sign up for one of those two approaches. One approach is a Before-After design, and the other approach is a randomized control trial (RCT). The approach you will sign up for will be discussed in consultation with CNA. Larger departments will be eligible for the RCT design.

What is the Before-After design and what does this look like? The Before-After design involves taking surveys before and after training. Attendees of the training receive 4 surveys. The surveys should take about 5 to 10 minutes each to complete.

Survey 1: This can be completed the morning of the training via computer or phone, before the training begins.
Survey 2: This can be completed directly after the training has ended via a computer or phone.
Survey 3: Attendees receive an email and/or text message with the link to the third survey 3 weeks after the completion of the training.
Survey 4: Attendees receive an email and/or text message with the link to the fourth and final survey 6 weeks after the completion of the training.

What is a randomized controlled trial (RCT) and what does this look like? The RCT is considered the gold standard in scientific research because it allows us to determine whether the training causes positive outcomes while controlling for everything else (e.g., changes in crime rates). At its core, the RCT involves splitting the individuals receiving training into two groups: an intervention group will get the training earlier, and a waitlist group will get the training later. The intervention group and the waitlist group will take surveys at the same time. We will also work with departments to get relevant administrative data to examine how the training changes performance on the job. For a visual representation of the RCT design, please see Figure 1 below.

Figure 1. Randomized Control Trial (RCT) Infographic

How are participants being compensated for their time? To compensate participants for their time, they will receive Amazon gift cards from Dr. Katie Lewis or Grace Drake which will be delivered directly via email or phone. The compensation is generous and is designed to incentivize participants to complete all surveys.

Who can access my survey responses? Only the evaluation team (Dr. Lewis, Dr. Lai, and Grace Drake) will have access to any identifiable information provided during the process of the evaluation. The evaluation team will follow strict procedures that ensure all identifiable data is password-protected and secure. All data shared with us will be confidential and protected by a Federal Privacy Certificate (42 U.S.C. §3789g). The Federal Privacy Certificate prevents anyone outside our research team (i.e., the media, other agencies, other organizations, or subpoenas) from accessing any identifying information we collect in this study.