Predicting 911 Dispatcher Success with Multifaceted Testing with CritiCall 3D

Welcome to Testing Testing one two three, a podcast brought to you by TestGenius.

My name is Jenny Arnez, and I'm one of your hosts today. Also with me is Mike Callen. Mike is the president of TestGenius. Say hello, Mike.

Good day. I was gonna say good morning, but it may not apply.

I can tell. I could tell you were going to do that. And we're excited because today we have Doctor. Dan Biddle with us. Dan is the CEO of Biddle Consulting Group and TestGenius.

And Dan, welcome. We're sure glad you're here.

Thank you. This should be fun.

Yeah, I think so too. Absolutely. Dan, as we get started, would you share a little bit about your background and your education?

Yeah, so, oh gosh, that was a little while ago now that I did the education part. I do have a PhD in IO psychology from a light university. I got that several years back and likely like to say you get your PhD and then you want to put dust on it. So I put some dust on the on the PhD. But I've been the CEO for for Biddle Consulting Group for several years now.

And our companies range between thirty and fifty five people or so. And we do a lot of testing and a lot of validation, a lot of statistical work. I've spent about twenty plus years as a testifying expert in state and federal court cases dealing with selection, testing, validation, statistics, psychometrics, things in that realm. And for about a period of ten years, I was a classified consultant with the United States Department of Labor, where I got to see a lot of the behind the scenes tests that were on the eve of litigation that could have been potentially brought forward in class action lawsuits. And I got to lean into the government's perspective about whether these tests would pass a scrutiny as being valid underneath the Uniform Guidelines and Employees Selection Procedure.

So I did that for about ten years, and that enabled me to review dozens of assessments by several different employers and test publishers, and help the government weigh in on which ones were sufficiently valid in litigious situations. So did that for about ten years and and enjoyed it. It's a it's a very technical world, but I feel like that's my gift set and disposition. There's other parts of being a CEO that I love, but I I do like the technical side of things. So, yeah, that's a little bit of my background.

Wonderful. Thank you.

It's hard to believe, Dan, to sit down and talk about something like this now, that it was a little over twenty six years ago that we were first putting CritiCall together. And it's kind of

shocking. That is incredible. I still remember the early days with several of the engineers and founders of CritiCall and getting the first groups together. There were just a few dozen agencies, nine eleven P Steps that help us frame CritiCall, and got together with the job analysis, and the subject matter experts, everything. And we laid a solid foundation that we've been building on for a quarter of a century now.

So, yeah.

Yeah. And it's it's really it's really grown a lot recently. When when we first launched the product, you know, as you know, it was an an operational test. It was a hard skills test. The idea behind it was let's help agencies to determine which of their job applicants have the core skills and abilities that are necessary for success on the job, and particularly also multitasking and success in in being able to get through the training, the academy, and then moving on successfully into the job.

CritiCall three d, which has been launched in the last couple years here, is a real quantum leap forward. Yes. We've added behavioral and situational to that. Can you talk for a couple minutes about what's the importance of having a multifaceted test versus a single facet? Yeah, you know, if all you have is a hammer, everything looks like a nail.

And that's what typically happens in a lot

of selection and high stakes testing situations is that employers think, Well, I've heard of this test, let me just go use it. Or let's use just an interview or just a cognitive ability test or just this type of test. And holistic testing, or three sixty testing, or high definition testing, or in our case, what we call three d testing, is really the only and best solution to move forward in a holistic complete testing program. You really want to do measure different aspects of applicant competencies, and rather than just measuring a single fraction of their abilities, and then hoping that sticks on the job, In order to maximize the predictive efficiency or the effectiveness of a testing program, you really can't bite on the same side of the apple. You want to have an assessment program that's going to get the soft skills, interpersonal skills, cognitive ability, and forming that type of a battery is the best way to move forward to...

It's fair for the applicant as well as the employer, because you're looking for more of

a holistic match that way. When you talk about... Thank you. When you talk about biting the same side of the apple, I know there's there's something technical that you mean when you're talking about that. Why don't you kind of unpack that a little bit so that everybody understands what you mean?

Because that's a really important aspect of how critical three d works and how well it works.

Yes, well, if you just were to step back from a 50,000 foot view and look down on what types of assessments are the best for predicting strong performers across the highest number of occupations, that would be cognitive ability testing. That's the most commonly administered test in America. And that's for good reason, oftentimes because academic skills and cognitive ability skills make up the usually the strongest predictor in a set of hiring practices. But oftentimes people will leave out the softer skills, more drive, or motivation, or introversion versus extroversion. There's all types of complementary skills and abilities that should be added to a selection program.

Because, listen, if all you do is just have a cognitive ability test, and you're just getting smart capable people onto the job, well, about smart capable people who can't read a room, or don't have interpersonal skills, or can't verbally communicate, or don't understand context. It's the whole challenge between IQ and EQ, emotional intelligence versus intellectual capabilities. So it really does serve both the applicant and the employer best to have a holistic or more complete assessment package when hiring. That's just what we've

done with CritiCall three. That's great. When you say cognitive ability, what's kind of a... You know, people aren't... Probably everyone's heard that term before, but what's just kind of a layman's definition of cognitive ability, and what does that mean to them?

Yeah, it's going to be tough getting to a layman's definition, because from an industrial psychologist standpoint, there's really different, two different categories of cognitive ability. The first would be fluid. And what we mean by fluid is someone's just raw, innate, natural thinking skills, whether or not they're bright, are they a quick study? Can they read things? Do they have a photographic memory?

Do they have to read stuff six times or one time? Just your raw academic cognitive faculties, and that's called fluid IQ. And then there's what's called crystallized IQ, which is how we have learned or acquired certain skills, like mathematics, or languages, or physics would be another example. Those are instances where we have learned to acquire cognitive skills or knowledges that help us perform on the on the job. So it's really those two things, the fluid IQ and the crystallized IQ that make up the bigger set of what's called cognitive abilities.

Is that the fluid, is it more like mental ability, and then the crystallized would be things that you've learned and are able to access. And

That's a great way to summarize it, Mike. That's Is that Okay. What it is.

Yeah. And I'm guessing. Yeah. That's good. Got it.

Yeah. Okay. Good. That's great. That's great.

So, Dan, you had briefly mentioned that an assessment like CritiCall three d, when it does test for the operational, behavioral, and situational, it's good for the employer, but also good for the employee. Can you elaborate on why that type of testing is beneficial to the applicant?

You know, that is a great question. I'm glad that you asked that. So let's use another type of testing situation as an example. So if you show up to a firefighter physical ability test, and you see it being demoed, and then you hop in and take the test yourself, and let's say one of the components of the test is taking 165 pound dummy and dragging it 50 feet to simulate what you might have to do in an emergency firegrounds situation. If you go take the test and try to drag that dummy for 50 feet, and you have a really hard time doing it, well, and then you self select out and think my gosh, firefighters really have to do that, you're doing yourself a disservice to try to push past or maybe sneak barely by that physical ability test to get the job because you're going to end up getting a job that you really aren't well suited for.

So the best thing that you can do in a handshake situation where an applicant is meeting an employer standards of put forward as a test is you want a dual direction, a bidirectional situation there where the employer is looking for skills that are needed for the job. And the employee is looking or the applicant is going to be looking for a job that really fits their true skill set, the ones that they have and the ones that they don't. So it makes no sense to hop on a job like in that firefighter situation, if you really are having a hard time dragging that dummy 50 feet. And to get on the job where you're going to be doing that all the time, or things like that all of the time, it would be better to do a job that you're highly suited for. And that's why selection batteries that truly represent the most multifaceted nature of a job, present the fairest and the most robust selection tool that should be used for employment screening.

Oh, that makes sense. So I I wanna ask a couple of real basic questions. You literally wrote the book on adverse impact adverse impact, and test validation. Can you just define those terms for folks like, well, people like me who aren't professionals in this field?

Sure. Adverse impact is an important concept. It's really existed since the early 70s. And adverse impact is just when you have a practice procedure or test that has disproportionate passing rates between two groups. So on that firefighter physical ability test example that I shared with you, firefighter physical ability tests typically will exhibit adverse impact against women.

When you compare the female passing rate to the male passing rate, on most physical ability tests for police or fire jobs, they're going to exhibit a disproportionate passing rate that's statistically detectable. So not just a minuscule amount, but something is going to be a large enough passing rate difference, it's going to be detectable with the statistical tests and the different types of tests that the courts have deemed valid for determining when adverse impact happens or not. And whenever an employer has adverse impact, they're under a federal obligation, a federal legal obligation, to validate that practice procedure or test to show that it's significantly job related for the position in question and consistent with business necessity. They have to, in other words, show that, Yeah, I've got a test that's got adverse impact, but if I don't have applicants with these skills, they're not going to do the job, and the public's going to suffer, or the employer's going to suffer. So, you have to show that it's necessary for the business, and it's going to be locked into the job in a way that fairly represents what those knowledge, skills and abilities are on the job.

So that's kind of an overview of those two concepts.

It's really helpful. Can you talk a little bit about essential functions? How does that apply? Remember you telling me some stories in the past where some organizations who were focused on trying to screen certain people groups out of a process would test on non essential functions and fail people so that they could basically end up with the outcomes that they wanted. How does essential function, what are essential functions and how does it kind of come into play here?

So essential functions is in the realm of HR is typically a term of art that applies to the Americans with Disabilities Act. So that if you deem a function essential, first of all, it has to be critical and necessary for the job. Then it has to be something that you also can't readily assign to someone else without changing the fundamental nature of the job. And oftentimes it has to be something that represents a good percentage of the job, and not something that you could just take and farm out to someone else. So that's, generally speaking, what an essential function is under the Americans with Disability Act, but it can also mean just a broader term of what's a necessary or critical part of the job.

The uniform guidelines typically want, which is the government standard for validation, they typically want employers' tests and assessments to measure skills and abilities that are going to be linked or necessary for performing essential functions on the job. You don't want to just test for things that are only merely important, but may not be necessary or critical for the job. So in a validation sense, that's what we would mean by an essential function issue. Better have a test that's connected to an ability, and the ability's got to be tied to something that's an essential function of the job.

That's great. For anybody listening, if you're curious about the uniform guidelines, which is the main document that the federal government and attorneys and labor boards, all sorts of folks, people like Dan Biddle, will turn to when they're adjudicating a situation or they're looking at hiring situation, selection planning, those kinds of things. You can go to uniformguidelines.com. It's a website where we host copy of the uniform guidelines as well as the questions and answers that are associated with that. It's a nice resource that's available for you.

Yeah. We'll also put that link to that in the show notes as well. So recently, you conducted a pretty detailed, involved study of CritiCall three d. I'm wondering if you could talk about that today.

You know, it was probably the most... The highest summit of any validity project we've ever done as a company. And I don't mean that to discourage any of the work that I've done in the past or any of the other trades people that we've had in our company, but it... This was the summit of literally decades of work in this industry where, in a nutshell, we were able to infuse together three different types of assessment processes for CritiCall for screening in nine eleven dispatcher applicants, consisting of first the operational test, which is our critical assessment that's been tried and true. It's been adopted by over 1,000 agencies.

It's over twenty years old. And that really represents the most fundamental, let's say cognitive or academic types of skills that people have to be to be a nine eleven dispatcher. But we also added on top of that the second component, which is a situational test that's put together by a partner called Logiserve, which measures more of a contextual situational abilities that doesn't overlap tremendously with CritiCall, which is nice because anytime you're going to be building a multi pronged assessment like this, you don't want to again have it bite on the same side of the apple, like we talked about. You want the different types of the tests that are put together. And then the third one is just a good classic personality test.

We call it the behavioral assessment. We were able to take, after years of study on each of those three different tests, the best of the best of the best of all three of those components, and put them together in CritiCall three d, and show some really stratospheric validity coefficients that show that this thing is just the strongest predictive measure you could ever hope to put together for screening dispatchers. It's really phenomenal.

So how long... I have a little bit of insight into this, but how long did it take you once all the data was collected in order to be able to analyze the data, to figure out what it was that you had, and to create a report. Do you recall? Yeah, well, short answer is it took about six months longer than you wanted it to take, Mike.

So that was fun. You know, without giving a long version of it, I do need to say that the operational test, the critical test, had been through several statistical studies, and I had to go back and revisit all those original studies. Logiserv, same thing. A couple different validation studies and the personality test had been beat up for multiple years. So, and what we did is we statistically aggregated those three different tests into the shortest, most compressed, yet most effective assessment battery one could even imagine for this job.

And in fact, when we did it, we got an operational, what's called a validity coefficient of 0.634, which is really unheard of in the field. And when you take that number and you square it, fancy talk here, you get something called a coefficient of determination, which simply means that we're able to explain about 40% of what predicts job performance in the 09:11 call taker job with that battery of those three tests put together. Normally tests, or even the good tests, are only going to predict between 510% of job performance. We're predicting 40. It's almost got prophetic powers showing who's going to be in and who should be out.

And again, that's the fairest thing that you can do for both an applicant and an employer in that handshake situation, is you want the employee to fit the job if they're suited for it, if they've got the capabilities to excel and do well, and then you want the employer to match having the good people come in and the people who are less capable to be screened out. So that was the result of a culmination, a summit of about 10 plus of research on those three different batteries. We put them together, we shorten the tests to make the most powerful predictive set that I think you could ever possibly do for an industry. We've done it once before for nurses, and ended up with something similar. But the only way you can get there is by taking divergent different tests that are measuring different types of skill sets, different abilities, like the situational personality, and the operational and putting it together with each of them being strong and robust just by themselves, but when you add them together, you end up with something that's just in the ozone, as far as just power to predict who are going to be the most capable performers as dispatchers.

To put a little bit of comparison in terms of what 40%... Predicting 40% of job performance is really... It's really incredible. One of the things that we tend to think is really highly predictive is just your your standard non structured interview. A supervisor sits down with an applicant and talks to them.

But I know there have been studies on how effective that kind of test is, because an interview is another kind of test. Can you talk a little bit about that and some of the other types of tests that are out there? How predictive they are as compared to this?

Yes. So interviews come in two very different flavors. The first should be unstructured interviews, and the next would be a structured interview. And the psychological testing research really beats up the unstructured interviews, because Jenny might want to sit down and interview someone. Well, she's going to ask five questions and I sit down and interview someone.

I'm going to want to ask 12 questions and they're different questions. And then next Thursday, I would interview something different. I might ask 17 questions and they're going

to be

different. Unstructured interviews are really a crapshoot. People need to know this in the HR realm that it's not a way to do what we're trying to do here. Rich, remember, the pinnacle of testing is a handshake between the employer and the applicant. You're trying to fit find a mutual fit that is a consideration of both of the interests of those two groups, the applicant and the employer.

So a structured interview can do well. And a structured interview is where you do a job analysis, you extract the knowledge, skills and abilities that are going to be important for the first day of the job. And you develop competency based questions that are going to truly not be fakable, where an applicant just can't shine in the interview by making stuff up. But they're going to be competency based, generally speaking. And then they're going to be scored using some type of a rubric, some type of a scoring matrix so that if Jenny gives an interview, well, she's going be using the same scoring system that I'm going to be using.

And structured interviews could generally do do well on the job. But the unstructured interviews are not typically the highest rank when you look at your psychologists do this thing called a meta analysis, where they look at a whole bunch of different types of testing practices to see which ones, you know, summit to the top as being the most predictive. And structured interviews typically do well, but unstructured interviews are oftentimes just a crapshoot, you know, and an interview, an unstructured interview, you might be, you know, predicting three to 5% of what makes a person a good job performer. A structured interview can do better, but nothing like the three part battery that we've done for CritiCall. That's pretty amazing.

If you compare, you know, that's... Let's say it's four. Pick the pick the middle number.

If it's four, then 4%, then it's it's 10 times more predictive power than actually sitting down with a human being and and and talking to them. That's quite amazing.

Yeah. Good questions.

And we'll be right back after a word from our sponsor.

Ready to revolutionize your HR strategies? Head over to testgenius.com to discover our latest tools and solutions designed to streamline your hiring processes.

So, Dan, I know that you have put some information together to share for us. And if you might want to sort of step through that, we'll we'll try to verbally describe what's going on for people who are just listening to the podcast. Of course, we will provide these notes or these documents in the show notes afterwards. But why don't you talk to us a little bit about what you've prepared to talk about today? And as we're going through, maybe we can ask you questions along the way.

That sounds great. The first thing, by way of example, to consider is, let's say we have a test, like an unstructured interview, that has very little to no correlation to job performance. And sadly, that's what a lot of hiring practices are doing. They're not getting a lot of validity. They're having some robust measures screen in the applicants who can do great on the job and screen out the ones that can't.

So in this situation, we just have a graphical representation of a zero correlation test. So you see here, it's just a random scatter plot. There's no correlation, no association, there's no clustering between the higher test scores, resulting in higher job performance, the lower test scores, resulting in lower job performance. So it's just a random hiring process. And this is sadly what you might get with an unstructured interview.

If we go up the food chain a little bit and look at a stronger test, the one that has a great validity coefficient here, which is an example of 0.5 or correlation of 50. Now we see a clustering of the higher job performers. The people who score higher at the test are also turning out to be the higher performers on the job. You've got the middle test performers and the middle job performers. And then on the far left side, you've got some test performers who are scoring low on the exam, who are also not doing well on the job.

They're the lower job performers. So that's what correlations look like in general, when it comes to looking at test validity. And then in our situation with CritiCall, because we were able to take our CritiCall three d, you're able to take the three different somewhat divergent tests, the operational, which focuses on skills, the behavioral, which focuses on personality and the situational, which screens how well a person does with their interactive role playing assessments. And because those three tests didn't have a high overlap, in other words, they weren't too correlated with each other, when you put... But each one of them was in fact individually strongly correlated with job performance, the effect of putting them all together is a whopping validity coefficient of 0.634, which explains about 40% of job performance.

So you're really, you know, and some people will say, well gosh, for the 40%, maybe I want to go after 50% or 60%. Well, the way that tests work is that because they're unreliable, because every time you give a test, a person's going to exhibit a different score, That's called test reliability or consistency. And because tests don't typically have a strong correlation with the job because they're paper and pencil, or they're only loose representations of what the job skills really need, The average test is going to come up with a correlation of let's say 0.2 to 0.3 is normally what we see in selection science. And if you take a correlation of 0.2 and square it, that's only 4% of job performance, leaves 96% of what makes a person perform well on the job completely untested. And just to let you know by way of an example, there's actually been four courts that have ruled if you can get a test that's so good that you get just a point three correlation, that's strong enough to strictly rank order on a test.

And that's because, my gosh, you take the point three, you square it, that's 9%. Even that test that's predicting 9% of what makes a good job performer perform well, and another one that doesn't perform too well, just 9%. That's still leaving 91% of what makes people tick on the job left to chance. But CritiCall was able to do a really strong job of when you put all these three tests together, and do some fancy statistical work, you're looking at about a point six three four validity coefficient, what's called an operational validity coefficient. So when you put that into a model, we can see what happens.

So if you look over on this example, a person who scores, let's say 35 on critical three d, if you say the highest score is, let's say, going to be in 70 in this example, and a person scores all the way down to 35. You can follow that line up and see where crosses over what's called the linear prediction line, and they're predicted to perform a four out of eight on job performance. Then if you go over the continuum and say, well, what about a person who scores 60? Well, they're predicted to perform almost about six on an eight point scale of job performance. And that's what you can do with a strongly valid test, is you know we can start making predictions based upon how people score on the exam.

So we're no longer throwing assessment chance to chance and guesswork. Three sheets to the wind, what's going to... Oh, that person's a great hire, but I don't know how we got them. That person that you know, had to flunk out of the academy, or they flunked out after three months on the job. Why did it happen?

We're taking all that guesswork out of the selection process, where as much as we can, because we have a highly predictive instrument. So when we look at how this sizes up overall, there's been some studies that this is a big one that just came out about five years ago, where they did what's called a meta analysis, which is where they add up and aggregate several different types of statistical studies to find out what types of tests do the best or strongest to predict job performance. And this study came in and said, if you have a structured interview, you're probably looking at a point four two validity estimate, and if you swear that, you get about 18%. So this study is saying, if you do structured interviews, you're likely to get some decent predictive validity on your test. But if you use an unstructured one, you're not going to do that well, because that's going to be lower down in the rank.

And then job knowledge tests, you're going to have about a point four validity coefficient, or a bio data, or a work sample test. So we can start seeing how different types of tests in a hiring process are going to serve the employer's needs better. So what you're trying to do as an employer from a big picture standpoint, is you're trying to push away the guesswork and field in tests that are going to really have a predictive value or predictive power in forecasting your better performers from your not so good performers. And so from a big picture, that's what we're trying to do in assessment. What happened with CritiCall three d was really, really incredible because if you line up all these different types of tests, you can see here that the CritiCall three d, because it took parts of a work sample test, parts of a personality test, and part of a situational judgment test, and then parts of an emotional stability test, are really tapping into those four different things.

We're able to take those four different types of assessments, and because they're not very overlapping, we're able to stack up the predictive power into a combined assessment that has stratospheric power in predicting and screening in the people who will do well, and screening out the people who won't. So in a nutshell, that's

what we've got with CritiCall. It just stacks up. It's way higher than your average tests will be doing across the industry. That's amazing. Is there any way of knowing, you know, most employers will use different steps in the recruitment.

They'll use, maybe they'll use the application. You just have minimum qualifications you have to agree to. Then they'll do testing. Maybe they'll do a background check. Maybe they'll do a polygraph.

Maybe they'll sit down and do an interview or a structured interview. I'm guessing that you can't just add those up and determine how much because you don't know how much each of those different steps overlap.

Is that correct? Well, you almost can add them up if they're totally disassociated. Okay. If you're not biting on the same side of the apple, if they're not overlapping statistically, you almost can accumulate the benefit of giving divergent types of tests. But in actuality, testing practices, what we refer to in the industry as PPTs, or practices, procedures, and tests.

Like a practice might be, Hey, we only recruit people from these regions, or from this school, or whatever. A procedure might be some type of a basic qualification screen, then a test might be something like CritiCall. If you have different types of PPTs that are totally non overlapping, then you will be adding an incremental value in your testing process. What we've done here is we know that these three tests do have some correlational overlap because there are assessments that get at similar, but not totally overlapping domains. But because they are disassociated, and because each one has itself independently a strong correlation with job performance, we're able to see that additive effect of what happens when we stack all three tests together.

And again, gosh, I've done thousands of correlations for hundreds of different tests and people in my field, we jump up and down. Anytime you get over about a 0.25 of validity coefficient, they're like, Oh my gosh, it'll run. That's great. Let's go use that for assessment. Knowing that you're only predicting 5% of what makes people tick on the job.

But if you get a 0.3 or the low 0.3, you're like, my gosh, that's so profound. Now I can even rank order on this test. At least that's what court courts have said. And sometimes you're going to get lucky and get into the 40s. But in this situation, we have three different tests that themselves have predictive benefit and value to the job.

But when you add them all up together, because they're non overlapping to some extent, you have this additive effect where the culmination of putting these three things together is just unheard of in the industry. I've only seen it once before. And that's when we did the test for the nursing industry. Right. And it was the same kind of thing.

Had to take three different divergent tests and sandwich them together in a combined assessment that had this level of power as an overall predictor on the job. But look, this doesn't just come about by chance. You're looking at twenty six years of research, and literally hundreds and hundreds of job experts who participated in a criterion related validity coefficient. And I don't know, maybe at least six different studies that have gone on between these three different tests So it's really the culmination of a quarter century research into this narrow industry using these different three types of assessment. And what we're ending up with is something I can say with a very clear conscience, yes, you've got the best battery possible for that employer applicant match.

That handshake that we're trying to line up, that I can look an applicant in the face and say, You know, if you score low on critical three d, there's a really good chance that you're not going to be a good job performer. But you can look at the people who scoring in the top 10, 20%, and say, you know what? Move these people forward in your hiring process, because the chances are they're going to shine, and they're going to do really, really well on the job. And that's what you want for a predictive battery like this.

I know So Okay.

Go ahead.

Jenny.

I was gonna say, just imagine that I am a captain in a dispatch agency, and I don't have a background in HR, probably... Or might not have a whole lot of stats background either. Get really practical with me. Why do... Why does it matter to me in in directing my agency that we have a 40 of job performance?

Explain. What does that mean in our bottom line and in our day to day operations?

Let's put... Let's let's look at it this way. The number one thing in any business decision is the removal of uncertainty. We'll do that if we're gonna put police officers in the car. They're going code three.

Is the engine reliable? Is the cooling system adequate? What are the chances are the engine's gonna blow up when he's chasing a guy 95 miles an hour down the freeway? So if we're going use rigorous testing on the mechanical parts of emergency services, we need to use rigorous tools on the selection side of the process as well. So I want to...

If I was a hiring manager or a captain in a dispatch center, I would want to be able to look applicants in the face, because these bigger agencies, as you know Jenny, they're screening hundreds of people a year in some instances. I want to be able to look at the applicants that I'm sending a failure letter to. Do not pass. Do not collect go. Do not collect $200 I want to be able to sincerely say, Well, that's because you didn't pass our critical assessment battery.

And that's because you know, you have skill sets that would be better served elsewhere in different lines of employment. And it doesn't mean they're not capable or not right or whatever. It just means that they don't happen to fit the profile we're looking for in these jobs. The good thing is that you can look at the top 10, 20, 30% of the people who do score high in this job, and say, you know what? We're going to move these people forward.

We're not going to lose $40.50, $60,000 on training them, probably, because they've got the skill sets needed to do the job. So, we can therefore spend public money with good conscience, saying we have removed uncertainty, and we've only hired the people who statistically speaking, gonna have the highest chances of being our solid successful performers on the job.

That's awesome. You know, with the Classic CritiCall program, one of the things we talk about there is is the issue of defensibility and utility. So in defensibility, it's, you know, is the test valid and consistent with business necessity like you explained earlier? And on the other side, it's kinda like two sides of the same coin. The other side is the the utility, which is, you know, how well does the...

What what... How well does the test do a job of selecting the people that it's supposed to be selecting? Those are going to be successful. Part of that is where are your passfail points? So, you know, we put a lot of tools into the classic CritiCall that helps people to select cutoff scores.

You know, you have a test that's valid, but if you if you have your your pass point set too low, then you're gonna have a lot of people who should otherwise be screened out passing. If you have it set too high, then you're gonna have a lot of people who potentially might have performed well on the job, never getting an opportunity to get the job. But you mentioned here when you have this high of a validity coefficient, you can actually go through and strictly rank order. You talk a little bit about using CritiCall three d, you know, and you have a pool of results. What do you do with that pool of results when you're, you know, let's say you have 100 people who are testing, and they're going to have a variety of scores.

How do you then interpret who you should move on to whatever the next step is?

Yeah, there's a lot of factors that are going to come into play in that question, Mike, because the first one is, tell me about your recruiting program. Do you just throw up a sign on the public street that says, hey, anybody come in and apply? Or are there schools that are funneling into this situation? Are you doing targeted recruitment at street fairs, at state fairs, things like that? So who's applying?

If you just attract the general population that's not going to be screened down a little bit, Sometimes you will have a lot higher failure rate on CritiCall. But again, the neat thing about CritiCall three d is you can let it absorb and make that decision for you. If you're highly selective, and you're hiring for people who have a career aspiration to be dispatchers or have some training or they've been the best thing would be finding people who were doing a job shadowing type of work and that they're targeting this career, you're going to have a lower failure failure rate, generally speaking for those people who look into those jobs, because sometimes interest and aptitude are correlated. People are interested in a job or generally sometimes usually good at the job. So anytime you look at the utility of critical, it's going to be moving that magic bar about how far am I willing to screen this up.

I'm going to say, Hey, I'm willing to screen out 90% of the lower performers, and only screen in the top 10%. Rank ordering is a pretty hardcore decision making process. If you have a 100 people, you're going to... Willing to take Bob and Mary because they're the top two performers who are tied for the top rank. And then the next guy in line is the next guy in line only because they scored 0.3% lower than the top two people who were Do tied for first you want to make those types of distinctions?

We generally prefer banding, which is a statistical process where you can group similarly qualified applicants together. That takes it also considers the statistical unreliability of the tests that are in your program, because tests are not perfectly reliable. They always have a degree of error variance to them. So, you know, if you go take a 100 item test, the chances are you might score 82 the first time and 85 the next time, or you might go down and you're going to be some wobble around what's called your true score. But this really depends on the employer situation.

Are they in a situation where they have to screen in lots of people? Then they have to move their cutoff down. Do they have 10 positions to fill and only 20 people that have taken the exam? Well, that's going to be another consideration.

I see. Adverse action. Or one position to fill.

Yeah.

And you've got ten ten good applicants to look That's kind of the catbird seat right

there. Yes, exactly. So you could be picky in some situations, but I would say the employer's need is going to really drive where you're going to be sliding that cutoff score. But generally speaking, we would advise that clients have their job experts take CritiCall to establish a minimum effectiveness cutoff score for their agency, or you can use top down banding to group applicants in a way that collectively makes sense. Like these are my A players, these are my B players, these are people who are just barely above the cutoff score.

Break ordering is probably not the best way, depending upon the number of applicants you have coming in, how many openings you have, things like that. I'm really a proponent of statistical banding on a test, but you never want to band below where the minimum cutoff has been. And they're establishing the minimum cutoff score is generally an employer by employer situation where you want to get your job experts involved in that. But the neat thing is that CritiCall has got such strong validity to it and such great validity evidence that employers are really left to their own discretion for using it effectively in their hiring process. Yes, that they can go ahead and rank order if they want, use banding, use cutoffs.

We do advise considering levels of adverse impact for liability exposure and where they're going to be setting their cutoffs for. Sometimes you can completely miss your potential liability of adverse impact if your cutoff score doesn't have adverse impact. So there's a lot of things to be considered, but the neat thing is that we've got such strong validity with CritiCall that the employer can use a high continuum of discretion on where they want to

select that cutoff score. That's great. I'd like to mention for anybody who is listening that we do have tools that are available in the CritiCall three d program that will allow you to very quickly and easily establish that minimum qualifying cut off score for the operational test, which is the critical one test. One of the components critical three d. And it would basically look at basically, it would say you have a pool of applicants and some of those applicants don't possess the minimum skills and abilities to be able to perform well on the job.

So you could sort of screen those people out and then you go through and you consider the behavioral and situational and operational together for those people who do have the minimum qualifying skill level and decide kind of where your threshold is. Did I say that?

That's great, Mike. Yeah. Yeah. I mean, we always believe that cut off score should not be arbitrary. I disdain the average arbitrary 70% cutoff because that takes no consideration with how difficult the test is or where the job standards need to be.

Yeah. A lot of people would think, well, that's a c minus in in school, but it's really not because the 70% on some tests might be the highest score range. Another test is the lowest. And that's irrespective of where the job fit needs to be. So look, we want to do this right.

We typically advise that job experts take this test, see how they score, and then they can get it in what's called the adjusted Anghoff process that we use for establishing minimum competency cutoff scores. But the other thing that they could do is just do top down banding. But then again, you want to know how far can I go down in these bands before I I met you hit a line that the job experts would have drawn that now we're starting to hire people that are unqualified? Don't wanna do that.

One time somebody at a conference said to me, do you know what a 70% cutoff is? And I said, no. And they said, it's a lawsuit waiting to happen.

Oh gosh. Variable could be. Absolutely. Yeah. We like putting selection science into it, but the neat thing is that we've front loaded as much of that as possible into CritiCall.

And yes, we advise that people set a minimum and not an arbitrary cutoff for their agency, But the top down banding can also be a top way... A good way to go, because it's gonna give you generally a players and b players. You just want your job experts to tell you, well, maybe the c players are too low. So... But it's at least gonna give you a couple groups that you can readily hire from.

And I think to a certain extent, by by having a test that works this way and reporting that works this way, you can clearly say, I'm gonna sort of dip the label... Ladle in there, and I'm gonna grab those performers off the top, and I'm gonna go ahead and interview those people first. And then, you know, if you look at bands like you've described, you might come in for, you know, with the ladle for a second dip into there, and you might get the second band. But as you continue down through that, kind of at least you have some intuition around what it is that you're getting yourself into. You may find that you have to do a little bit more remediation, but you have a still a good picture about, you know, basically what kind of performer they are and basically what kind of skills and abilities they're walking into the job with.

Great point, Mike.

Yep. That's the neat thing about CritiCall is we can have your own job experts look at it and establish the cutting score level. Might be in the middle, it might be a low cutoff score. We might just be looking for base rate people who can just come in do the very basic minimum of the job. Or they might have a hiring crunch.

My gosh, we need to put as many people forward on the job as possible. And in other situations, they can be more selective. Hey, we've only got five openings, but we've got 100 people that took the test. So let's just take the the the top five people.

Yeah. Excellent. Well, is exciting. Jenny, is there any anything that you have? Any wrap up questions that you have to come to mind?

Well, I I I know that we've covered this already, but I think it would be really good to just do a a... Another review of the benefits of using CritiCall three d in in hiring. I believe you have a slide for that, Dan. I think that would be really a good way to wrap up our time together.

Oh, that sounds good. So, neat thing that we've done with CritiCall three d is really to take the best of the best of the best from all the different available tests. But we partnered in with the other two types of assessments, the behavioral and the situational. We home... CritiCall three...

The base test, the operational test is a homegrown test. Well, we grew it with dozens of support from different agencies and everything. That's like the core fundamental test. But the other ones, we were able to take a couple of great partners that specialize in those areas, and bolt on their assessments onto ours after careful study. So when we put them into a seamless testing experience, in the old days, you would have to give the critical tests and then close down that testing session, open up another testing session, and it would go over by an API bridge or some kind of connector, but say, Now you got to take this test.

And now you got to take this test. And by the time you're done, the applicants just beat up like, oh my gosh, what happens next? But now it's a seamless testing experience, which is great. We have that goal of wanting to fit applicants into career roles for which they are equipped. We talked about the mutual handshake that you're doing there.

It's the best fit for the employer and the best fit for the applicant. We want it to be fair to coworkers by screening in capable team members who are going to come along beside them. Look, we've all been in situations before where we're tied up with coworkers that are not toeing the line, and that creates a very interesting situation. Bad team dynamics happen with that. Critical is going to be a robust measure that's going to select and really qualify people to bring that good teamwork element that we're looking for.

Employers can be secure with having capable qualified workers. That's probably the lead thing that we're trying to do here is present a robust and highly predictive set of tests that's going to screen in the best workers possible. And of course, in doing that, the public's going be served. I love reading industry reports in the nine eleven field and seeing CritiCall referred to as a name brand. People know about this, this product.

We have 1,500 clients around the nation. And honestly, we didn't go after that for money, guys. Our commissioning, our burden and passion as a company is producing qualified, well rounded, and fair assessment batteries. We do that with police and fire, with the sister company. We've done it here at BCG with all kinds of different test batteries.

Our front mission as an organization is to produce fair and robust assessments. And I think we've done that with CritiCall. So we deserve that first place to be upon us. After twenty six years of piling into this industry, we have summoned it with what we think is the strongest testing battery that this industry has ever seen. It's not a boast.

I think it's kind of almost an objective look at what's happened with our company over the last twenty five years. We have poured in hundreds of pages of tech reports into this industry. We finally wrapped it up with what our our latest technical report, Mike, I think is all of 10 pages because it's referencing all of the past technical reports. Right. Right.

Which is great. Of course, it shortens training time. You guys, the biggest pain point... I reviewed hundreds of pages of industry reports going into building this last critical test film. You want to know what the number one pain point I saw on the nine one one industry is unqualified turnover from less qualified nine eleven call takers.

It's such a pain point. And I can't tell you how many times I saw the report that saying did not pass training, did not pass training, did not pass training, or didn't pass probation, didn't pass probation. There are some reports that were as high as 70%. I'm talking about statewide reports that say we're losing 70% of our people because of skills based incompetency. And that's nothing...

That's no way to serve the public. We want go out and prepare for that. Of course, we want to shorten training time and decrease qualification related turnover. What we've tried to go after very surgically, and I would say very effectively, is minimizing the type of turnover that happens because the applicants don't possess the skill sets to do well. Not just to perform at a baseline, we want to screen in the people who are going to be the most competent, most well rounded performers.

And the way that we got there is by taking the best of the best of the best of three different divergent tests, sandwiching them together, investigating them for hundreds of hours of statistical analyses, and extracting what is the shortest, most compressed, most powerful test battery this industry has ever seen. And I don't say that lightly. I've been able to look at hundreds of different tests in my lifetime. And even if we weren't involved in doing this, I think that would be the same credos that I would give this assessment. It really is because it reflects the industry and lots of different agents that poured into this.

We've been able to produce for the industry an incredible gem here, an ace of spades on the selection front. And I'm really happy that that we got there. It didn't have to turn out this well. I would have been happy with a 0.4 or 0.45 operational validity. Getting in a 0.6 realm, because we're able to take these three different tests that are themselves individually strong performers and push them together, is just unheard of.

It's a great battery, and we're really excited to to push it out to the industry. Well, thank

you, Yeah. Thank you. So thank you for tuning in, for listening, for watching us on YouTube. Make sure you check out those show notes. We'll have the the slide deck on there.

We'll also have the executive summary and the validation report and other links that you'll find useful and helpful. And make sure to reach out should you have any questions about TestGenius or CritiCall three d. Thanks so much.

Thanks for tuning in to Testing Testing one hundred twenty three brought to you by TestGenius and Biddle Consulting Group. Visit our website at testgenius.com for more information.