Voiceover: Welcome to Testing,
Testing 1-2-3 a podcast

brought to you by TestGenius.

Jenny Arnez: Welcome everybody.

Welcome to Testing, Testing 1-2-3.

It's a podcast produced by TestGenius
and Biddle Consulting Group.

And I'm Jenny Arnez.

I'm your host and I'm joined
today with Mike Callen.

He's our co-host, but also
the president of TestGenius.

Mike, you want to say hi?

Mike Callen: Good morning.

Jenny Arnez: Good morning.

Yes.

Or good afternoon.

Good evening.

Depending on when you're
depending on wherever

Mike Callen: you are on this round globe.

Jenny Arnez: That's right.

And we're sure glad you joined us today.

So for today's episode, we're going
to do things a little bit differently.

We don't have an additional
guest joining us today.

What we're going to do is look
at a specific question that

a client recently asked Mike.

It is all about some best practices
that they could incorporate

in their hiring practices.

And it led to an interesting conversation
about, validity and adverse impact.

And so today what I'm going to do is
present Mike with a scenario very similar

to what our client offered to him earlier.

And just have a conversation around that.

Does that sound good, Mike?

Mike Callen: Yeah, absolutely.

Jenny Arnez: Alright, so
let me present the scenario.

Let's just say Acme County has reached
out, and they want to hire, or they

are hiring for Dispatcher I positions.

Now here's some background.

Each department in the county
determines their own requirements

for the Dispatcher I position.

All departments are using a
pre employment skills test.

And after a phone interview, selected
candidates take the pre employment test.

It's only the highest scoring
candidates that are selected

for an in person interview.

Acme County has reached out to us and
they're concerned that their hiring

practices, including the pre employment
skills tests, are resulting in adverse

impact towards protected groups.

Mike, how would you
help a client like this?

Mike Callen: Yeah, there's
a lot to unpack there.

I, if maybe if we step down
through, Each of those bars on this

image right here, we can talk a
little about that about each one.

So you have a county that has different
departments, all of whom have a job title

called Dispatcher I and then they're
determining their own requirements

for these, each of these positions.

Presumably they're slightly different
from location to location and so one

of the things that you always want to
be concerned with if you're an employer

is the optics in terms of how do
things look and feel to the applicant.

And in the case of a county or a private
organization, often the applicants

are constituents or customers.

And you want to be a good citizen in
your environment and you want to be

doing things that feel good and fair and
righteous in terms of their eyes so that

you don't, sully your own reputation.

And if the Dispatcher I position
is different in each of those

organizations, it might be better
to title them slightly differently.

For instance, it could be Dispatcher
One Fire, Dispatcher One Police,

Dispatcher One EMS, or however
they're dividing things up.

And that way you get a little bit of
disparity from one title to the next.

And I think the reason that I mentioned
that is that you have job applicants

who are going to talk to one another.

And if you apply for a position over here
and you're tested or you're minimally

tested and you apply for another
one over here and you're maximally

tested, they really wear you out.

That's going to feel weird if
they're all titled the same thing.

The same job title within different
departments could vary and it

could be justifiable to have
different recruitment processes.

It probably is justifiable
to have different recruitment

processes for each position.

But again, in terms of optics, it
might be something to differentiate

between the Dispatcher I position
at each of those locations.

So that's definitely one
thing to look out for.

Another thing to look out for,

and this comes up very often in
these kinds of conversations,

is within the recruitment.

Let's say your recruitment process is
different in each of those three or four

departments but your recruitment process
within that one particular recruitment at

one department, it's got to be the same.

You can't do different things
for different applicants

within the same recruitment.

So you want to be really
consistent in that regard.

Jenny Arnez: so this might be a
bit of a tangent, but you mentioned

about departments having perhaps
changing the job title where

they don't all say Dispatcher I.

I'm wondering how job analysis might
relate to that we've had previous

clients or previous guests who've talked
about job analysis, just wondering if

you might be able to touch on that.

Just briefly.

Yeah.

Mike Callen: Absolutely.

So a selection plan, which is any part
of the recruitment process from beginning

to what should be the job posting all
the way through onboarding somebody

should be based upon a job analysis.

This is best HR practices and a job
analysis is a really deep dive into,

what the what are the tasks and
duties that are performed on the job?

What would be the minimum qualifications?

What would be the
desirable qualifications?

What would be the work schedule, the work
environment, what personal characteristics

are required in order to perform the job.

So it's really

all of the technical aspects that have
to do with any particular position

would be fleshed out in a job analysis
and job analyses are really time

consuming, challenging tasks to perform.

And while everybody's supposed to
have current job analyses, they very

often don't, and it becomes one of
those eating the elephant situations.

How do I do, job analyses when I have 500
different positions within my county or

my organization or company or whatever?

And so the answer for that is, you
basically just do it one bite at a time.

It's best in my recommendation, it's
best to start with the positions that

are most often recruited for, or the
positions that are most often hired

and then reallocated
within the organization.

So you have these entry level positions
that people come into and then rather

than posting a job externally, they
posted internally and then they move

people around within their organization.

So those would be the ones to start with.

And then you keep going, doing the next
most high volume positions and until

you get all the way through the process.

And then it's painting
the Golden Gate Bridge.

Once you get done, they're really
old in three years or when the

position fundamentally changes.

And so you really need to
go back and do them again.

But you can use the existing job
analyses for the redo, which can

make the process a lot easier.

So job analysis is super important
in this particular practice and some

things that come from the job analysis
are the job description and the job

description would be a formal paragraph
or several paragraphs that talk about

what this person does in this position.

And then from the job description, would
be distilled, typically a job posting,

which would be the text that's put into
an ad or a posting online that talks

about the job title that's gonna be
opening up, some basic requirements,

minimal requirements, some desirable
qualifications, and then that little

summary of the job description.

So those, they're a very important
part of the selection process.

Jenny Arnez: Okay great.

And actually, I think what
I'll do is in our show notes,

I'll post a link to IO predict.

They're one of our partners
who actually does job analysis.

And Dr.

Clinton Kelly has been on our
show before and talked about that.

Because I think definitely
that would be very helpful.

Definitely relates to this scenario here.

As we consider this process that Acme
County has for hiring Dispatcher I,

where are the tests in this process?

What are the tests?

Mike Callen: Yeah, that's,
it's a really important point

that you're making here, Jenny.

When we're approached by clients or
prospects and they talk about, testing

and validation and adverse impact, the
focus is always on the literal test.

And the fact of the matter is that there's
many other steps of the recruitment.

There are also tests.

I was just talking to Dan Biddle, our
CEO, a few minutes ago, and he was

telling me a story about he did some
consulting for the Department of Justice.

And he was at a conference where there
were about 500 HR practitioners and

the head of the DOJ asked all of the
attendees to raise their hands if they

agreed that an interview was a test.

And it was half or less
that raised their hand.

And the fact of the matter
is an interview is a test.

There are a variety of things called PPTs.

Those are practices, procedures,
and tests that occur throughout the

recruitment process in which decision
making factors are considered in

this employment selection process.

An interview absolutely positively is one.

People sit down, they talk to another
human being or a team of people and

they're rated on their responses and
some people are moved forward and

some people aren't moved forward.

And that is part of the selection process
and therefore an interview is a test.

But there's other things that are
PPTs that also act like tests.

For instance, we talked earlier when you
brought up the job analysis, one of the

derivatives of the job analysis is the job
posting and the job posting talks about,

hey, here's what you perform on the job.

And if the job posting and its
minimum requirements are not valid,

that is, they're not matched exactly
to that particular position, then

anyone who's reading the job posting
may actually deselect themselves

from the process based upon the
information that they've been provided.

And if the information that's
provided is incorrect, then you

could have just created adverse
impact in the very first step of your

recruitment process by not making an
accurate job posting and especially

accurate minimum qualifications.

So that's a test too.

And every little intermediate step
along the way that either allows

certain people to continue or

doesn't allow people to continue or rank
orders people or weights their scores.

Any of those aspects of the recruitment
process that make those kinds of

decisions are also tests and should
be scrutinized in the same way that

people and we scrutinize our own tests.

Jenny Arnez: Okay.

So if I'm understanding you correctly,
what you're saying is like in our scenario

here, Acme County, they're concerned about

there being adverse impact and certainly
they need to look at making sure that

they validated the actual pre employment
test, but they actually need to look at

their whole process from job description,
job posting, interviewing process,

that phone interview that happens that
determines who moves on to the next step

Mike Callen: background checks.

Yeah, that's absolutely correct
what you're saying there.

And to that end, people will ask us for,
data in terms of pass rates for, certain

protected groups, EEO protected groups.

And we don't have that data
because we're not, getting that

EEO data from the applicants.

The employers who are going
through the recruitment, they're

getting that information.

They're collecting that information up
front and they have a piece of software.

Typically known as an applicant
tracking system- an ATS.

And the ATS will actually do analysis
at each of these steps to let them

know where they may potentially have
problems so they can dig into it and

they can see whether or not there's
some adjustments that need to be made.

Now every time

there's adverse impact against a protected
group, it doesn't necessarily mean that

you have to do something different.

There are instances where adverse
impact occurs and that adverse impact

is simply a part of the process,
the part of living in a real world.

And we can talk a little bit
later about that as well.

But you always want to stop and you want
to look and you want to ask ," Okay.

Is there adverse impact?

Why is that adverse impact occurring?

What could we do differently
to avoid that adverse impact?"

Those kinds of things are
questions that you ask.

And let me throw out an example because
it's a big thing to wrap your head around.

There's a lot of things we're
talking about right here.

But the classic example is in a
job that requires physical ability.

Let's say, for instance, there's a person
who has to load concrete bags onto a

pallet so that it can get shrink wrapped
and loaded into a truck and shipped away.

So there's a conveyor belt
that brings out these,

these bags.

And so what you might do in terms
of a physical ability test to hire

somebody into that position is
you might have a conveyor belt.

You might have this
person grabbing the bags.

You tell them how you want them stacked.

And you see how many bags
they can stack in 5 minutes.

Or can they stack the number of bags
that come out in a 5 minute period at

the same rate that would come out later

on the conveyor belt on the job itself.

Now, let's say, for instance, that this
particular test is using 60 pound bags.

And so HR calls over to the plant
and says, "Hey, we're getting ready

to do this physical ability test.

I want to make sure that.

We're still using 60
pound bags in the plant."

And the supervisor says, "Oh
no, I'm so glad you called.

Remember when the risk analysis
people said we're going to have

far fewer incidents of injury
if we use 40 pound bags instead.

So we switched over to 40 pound bags.

It's we just dodged a bullet because we
would have been testing at a level that is

much higher than is required on the job."

And that

a physical ability test is
going to almost always result

in adverse impact against women.

Generally speaking, men are stronger
than women, and so there will be

a disparity in between the passing
rates for men and the passing

rates for women for a physical job,
especially one that's really demanding.

And so, there's baked in adverse impact
that comes with that kind of position,

but by measuring using the wrong
size bag of concrete, you're actually

creating additional adverse impact
that otherwise would not have occurred.

And that is an instance that
is impossible to defend.

You cannot defend that.

You can defend the adverse impact that
occurs between natural differences in

physical strength between men and women,
but you cannot defend against the adverse

impact that occurs because you ask
them to do more than the job requires.

Jenny Arnez: Okay.

Yeah, that makes sense.

So what you're saying then is that
there are times, provided that your

test is valid, that what you require,
what your test is requiring of the

candidate is actually a reflection of
what they'll be required to do on the job.

When that is in sync, there are times
where an adverse impact will occur.

Mike Callen: Yeah, absolutely.

There are going to be
times in which it occurs.

It doesn't mean that
you don't look for it.

You still look for it because
you want to evaluate the process.

If you can measure the same
thing in a different way, and it

doesn't have adverse impact, then
it's a better way to measure it.

You go that direction.

In our tests, we use something
called work sample tests.

So we're actually like the concrete bags.

We're actually simulating the job.

It's very hard to find a testing
process that's more valid or

equally valid to actually simulating
the job than simulating the job.

And as an employer, if I'm hiring
people to load pallets, to shrink

wrap that material and get it onto a
truck because I'm selling concrete, I

have the ability to make sure that the
pallet loader can load pallets, right?

That's just a fundamental thing.

So whether or not there's adverse
impact- it isn't important.

What matters is that the selection
process, the test in this case,

is it valid and consistent
with business necessity?

If it is, then you are justified
in having some adverse impact

that enters into the process.

Jenny Arnez: So you
said business necessity.

That sounds like that
might be an actual term.

Is there a set of key laws or guidelines
that govern pre employment testing?

Mike Callen: And
guidelines is the key word.

There's a huge document which can
be found at uniformguidelines.com

which is the Uniform Guidelines
on Employee Selection Procedures.

And it's a document that was put
together, I believe, in the 70's.

It's been revised a couple times and
now has a whole host of questions and

answers that comes with it as well.

But it was put together by
a really large committee

of practitioners who operate in this
space, and it covers things like

job analysis, job descriptions, test
validation, also sorts of different

kinds, and it really sets forth the
practices and principles that are the

best practices for doing these things.

Now, I don't know that it
was ever codified into law.

But the courts have relied
on it again and again.

And the DOJ, Department of Justice, the
OFCCP, the EEOC, all these different

governing bodies, including other state
bodies, have relied upon it as the

standards for doing things correctly.

And so it comes up very often in
legal trials and adjudications like

by labor boards and those kinds of
things that comes up very often.

So it's a very important document.

Another thing that's really
important is the ADA, the

Americans with Disabilities Act.

And we won't really get into
that here, because it gets into

a whole other area of essential
functions and those kinds of things.

But, yeah, in terms of, dealing with this
issue, the Uniform Guidelines is really

probably the main body of literature
by which everything should be measured

in terms of doing things correctly.

And that's what we do as well.

In fact, in our software and the
TestGenius software, CritiCall and OPAC,

we use a validation wizard, which
is based upon section 14c and

15c of the Uniform Guidelines.

Section 14c is the section that
deals with content validation.

It's the steps that you need to go
through in order to properly content

validate a test, and that's how you
determine whether or not it's valid and

consistent with the business necessity.

As well,

15c is the reporting section for the same.

So it talks about what are the things
that need to be outputted in a report

that will properly document that
this validation has taken place.

And so in our validation wizard, we're
running people through a process.

In this case, we would run
subject matter experts,

who are performing well on the job
for a year or longer or supervisors

of those people are trainers of those
people who are abundantly familiar

with the position, we're going to run
them through each of the tests that

employer would purport to measure.

And then, at the end of each test
will be a survey that will ask them

questions about the job relatedness
of that test and that the very

end of each test that would say,

"Jenny Arnez, your score on this test
was 96 but given your position and your

knowledge of this job what, in your
opinion, would be a minimum qualifying

score for somebody who's entering
into this position the first time?"

So you would say, "Hey, I've been doing
this for 20 years, and I'm an uber expert.

So 96, it's really good for me but because
I directly just felt the difficulty

of that test and I'm knowledgeable
about how well skilled enabled someone

should be coming into this position,
I would say that if somebody scored

83 or better on this test, then I
would consider them to be eligible

to move on from the applicant
to the candidate role."

And so the wizard will take in
all of the subject matter experts

opinions, including yours and the
suggestions for the cutoff scores.

And it will create a validation
report that says of this test

battery, let's say it had six tests
in there, for this test battery

you should only use these five tests
because one of them was determined

to be not valid for this position.

Maybe it was more difficult than the
position like the 60 pound bag of concrete

instead of the 40 pound bag of concrete.

And so for that reason,
it gets thrown out.

As well, here's four different
defensible job related cutoff

scores that you could use.

Your average of your suggested cutoff
scores, and then that minus one, two,

and three standard error of measurement,
which is a statistical step downward

that one might use to create different
significant layers below the high cutoff.

If you don't want to use, let's
say 86 was the first one, minus

one SEM might bring you down to 81.

So let's say we didn't have
enough people to interview.

We wanna step down to the
next appropriate layer.

We might use the cutoff minus one SEM,
which would be 81, and that gives us five

people to interview or maybe minus, minus
two SEM and it brings this down to 76.

And that might be 10 people to move on to
the next step, that, that sort of thing.

Jenny Arnez: Okay, so what you're
describing is the validation process,

and in particular, the validation
process if someone were to use

TestGenius as their pre employment test.

Mike Callen: Yeah, or if
they did it all on their own.

You can do this process manually,
and this process is outlined

in the Uniform Guidelines.

It's just that what we've done is
we've said, "We value this process so

much that we want to make it as easy
as possible for you, our clients.

And so we're creating a wizard that
literally just walks them through and

that formats the report and then creates
the report and sends it off to people.

Jenny Arnez: Okay.

So let's real briefly, just a high level

outline of the validation processes
using the test selection wizard, right?

Mike Callen: we didn't talk about that.

But yeah, you want to start off
with the test selection wizard.

Let's show the content validation slide
because that's helpful in understanding

the test selection wizard process.

That's okay.

Content validity is like building
a bridge across the river.

On one side, you have the test.

So go ahead and go to the next slide

and then on the other side,
go ahead and do the next one.

You have the job and it's
represented by these duties that

are performed and the bridge is
supported in the middle by this peer.

And that is the KSAs.

KSA stands for knowledge,
skills and abilities.

So if you look at this process
the test requires certain

knowledge, skills and abilities.

The job duties require certain
knowledge, skills and abilities.

And so what we're doing is we're
connecting the test on one side of

the river to the job on the other
side of the river, and the linkage is

the knowledge, skills and abilities.

If we look at the test selection
wizard, I don't think we have an image

for it, but basically what we're doing
on the test selection wizard is we're

listing duties that might be performed
on the job, and so a person is going

to go through, and they're going to
check off duties that are performed by

the position that's being recruited.

And as they go through, and they
check off those duties that have

linkages to KSA's- knowledge, skills
and abilities- which are linked to

specific tests within our system.

And so, you check off 10 duties, and
that might create a list of eight

or nine tests that we may suggest
that they use for selection purposes

for this particular position.

And so that's the test selection wizard.

The next step that one would go to
in order to determine whether or

not the test battery in this case,
a battery of individual tests,

is valid and consistent with business
necessity is to go through the next

part, which is the validation wizard.

And in that case,

we recommend 7 to 10 subject matter
experts, and again, those will be the

incumbents that I described earlier.

People who have been successfully
performing on the job one year

or longer or supervisors at that
position or trainers for that

position, but people that are
abundantly familiar with that job.

And you want to go through and you want
to cherry pick really high performers,

but you also want to get a
demographic mix that matches that

of your existing workforce and
particularly of your recruitment area.

But the idea here is that these people
are going to go through those tests.

They're going to take those tests with
that wizard turned on, and the wizard

is going to ask them survey questions,
cut off score suggestions, and it's

going to result in a validation report

that might be 30 or more pages long
that's going to have a table at the end.

It says, here's the test you validated
and each of these marked yes, are

ones that you should use on your
particular recruitment or could use

in your recruitment as well here's the
defensible job related cutoff scores.

So that's those two wizards, how
they work, step one, step two.

And then step three is
your recruitment itself.

So you take the validated test and
you actually employ it for use for

selection with your job applicants.

Jenny Arnez: Okay.

So let's say you have your valid test.

You have candidates taking it.

So do you just assume that the person
who scores the highest on the test,

that's your number one applicant?

What's your recommendation
there on how to use the scores?

Mike Callen: That's a
really great question.

We didn't talk about this, but one of
the one of the survey questions that's

asked at the end of each test when
you're running your subject matter

experts or your incumbent employees
through there is whether or not the

test is performance differentiating.

I'm going to give you an example.

Let's say that you're hiring
somebody who has to type.

Dispatcher is a good one.

They're doing short bursts of texts,
but they need to be able to type

relatively quickly and they need
to be able to type accurately.

So typing with speed and accuracy is
an important attribute for that job.

The fastest speed that they may need to
type might be 35 words a minute, right?

So the question becomes, does that
mean that somebody who types 70

words per minute is twice as good as
somebody who types 35 words per minute?

And in this case, the answer is no.

You really only need to do what you
need to do and going faster than you

need to go isn't necessarily a benefit.

And so that would be a good example
of a test, the results of which are

not performance differentiating.

And so what we would do is we
would look at everybody from 35

words per minute and above as
being equally suitable for the job.

So there wouldn't be any
rank ordering at all.

And that is really

just one of the main reasons why
setting the cutoffs is so important.

If you have a defensible job
related cutoff score then what it

does is, let's say you just chose
the person who could type 70 words

per minute and you hired them.

You've ignored all these other
attributes of a human being that may

or may not make them a good employee.

Sure, they can type, but that
is not the entirety of any job.

And so by choosing too high, you may
have ignored applicants who were equally

qualified in terms of keyboarding,
but had so many other things that

they could apply to this work area.

And so that's a really important
aspect to keep in mind.

Now there are some that people will
go through and say, "Yes, this is

performance differentiating because,
people who do scope score higher on

whatever knowledge, skill or ability it
is, they may be the best and everybody

who's less than them is less good."

But again, that's just one of the many

reasons why you make a selection decision.

And so again, cutoff scores
are really important.

Jenny Arnez: And the validation process
helps you to set those cutoff scores.

Mike Callen: It helps you
to set those cutoff scores.

And it also will report back
to you whether or not something

is performance differentiating.

Which lets you know that once you get
to that cutoff score, everybody from

there and above should be considered
equally skilled or able for that

particular performance dimension.

And we'll be right back after
a word from our sponsor.

Ready to revolutionize your HR strategies?

Head over to TestGenius.

com to discover our latest
tools and solutions designed to

streamline your hiring processes.

So I'm going to flip back real
quickly add our scenario back on here.

Just so we can summarize where we've
what we've touched on Acme County.

They're concerned about adverse impact.

So the first step really is they
need to remember that the testing

process or testing happens throughout
the whole selection process,

throughout their whole selection plan.

And so they have to pull back the
lens, not just focus on the pre

employment skills test, but also
to look at their job posting, their

interview practices, and so on.

All the other tests as well.

Jenny Arnez: All the other tests.

And then focusing in on what our
specialty is here at test, with TestGenius

is the pre employment skills test.

And then, so then they need to validate
that using their subject matter experts,

using the tools that, that they have
within TestGenius, assuming that they're

a TestGenius client, using the test
selection wizard, the validation wizard,

which then helps them set accurate

- is that the term, accurate cutoff scores?

Mike Callen: Defensible and job
related is the important aspect of it

and let me in interrupt one second.

This is our recommended best practice.

Not every client goes through
the validation process for every

administration of TestGenius.

We know that.

It's our strong recommendation that
they do it because of two things.

We think about the recruitment process.

Think of about it as
a coin with two sides.

Okay, on one side, you have defensibility.

So in the instance where somebody
questions the hiring practice, is

that hiring practice defensible?

Or if you go to your applicant tracking
system and you find that your physical

ability test had adverse impact.

Is that defensible in
that particular case?

So being able to answer some sort
of challenge to the recruitment

process is really important.

But when you flip that coin over
and you look at the other side

from the utility side, or sorry,
the defensibility side, utility.

And utility talks about how does this
test select people who are going to go on

to be really good employees on the job?

And the beautiful thing about this,
and this is the beauty that comes

with the brilliance of the Uniform
Guidelines, is because people who were

really experienced in the practices
that can both have and result in a

safe recruitment and hire people who do
really well on the job are contributing

to this framework that we follow here.

And so that's why it's really important
to go through and do the required work

or to do the recommended work in order
to be able to end up with a testing

process that's valid and consistent with
business necessity and cut off scores

that are defensible and job related.

So that's as far as we can say about that.

If somebody decides that they're not
going to go through the validation, that's

their choice, but they've taken on that
burden themselves in case the hiring

practice is challenged, for instance.

If it's challenged, then you know
they don't have a leg to stand on in

terms of the defensibility, if they
go retroactively and do the validation

process and the validation report shows
that the test isn't valid and consistent

with business necessity or that they
use cut off scores that were too high.

So people are taking that risk
upon themselves by skipping

that part of the process.

Jenny Arnez: And how often does the
validation process need to happen?

Do we do this every year?

What is that?

How do you determine that?

Mike Callen: So the validation
process is good to do, generally

speaking, at about a three year clip.

And that would be the same
pace that you would want to try

to review your job analysis.

Now, the difference, or sorry, the the
caveat for both of those scenarios would

be unless the job fundamentally changes.

There's been a just a ton of
technology that's come into the

dispatch world, for instance, which
is to follow our scenario here.

People can text videos and images.

Let's say somebody gives you a license
plate number and you type that into

the computer aided dispatch system
and they also send you a photograph

of a scene and the photograph has the
license plate on there and you can

clearly see that the person reported
the wrong license plate number.

So you have to be able to discern
the difference between the two.

You have to be able to decide which of
those has the right data and then you have

to be able to update it into the system.

Because if you're talking about
putting out a BOLO looking for somebody

with this license plate and you
use the wrong license plate number,

you know that's really CritiCall.

So that would be a good example of a
fundamental change that happened at a

dispatch center that might require you
to go back and look at the versions

of these tests that you're using.

Maybe before you were using text only
versions, now you want to use versions

that have photographs, images and
questions about those images in there.

Jenny Arnez: I think we've done a
good job or you've done a great job

on helping Acme County here take a
look at their hiring practices and

some action steps that they can take.

And anything else that you want
to add to today's conversation?

Mike Callen: We talked about content
validity and content validity is

when you're building that bridge
from the test to the KSAs, knowledge,

skills, and abilities over to

the job on the other side, and there's
another kind of validation, which we

don't do with our wizard, but can be done.

And it's very much valuable.

And that's called criterion validity.

And criterion validity is when you have
some sort of aspect of job performance

on the axis over on the left hand side,
and then you plot that against test

scores along the bottom access there.

So each of these data points
would be some sort of measured

metric of job performance.

Maybe it's supervisor ratings or could
be feedback from surveys or, like

tardiness or, all sorts of different
things, but each of those data points

would be the intersection between how
that particular aspect of job performance

was and what the test score was.

So down in the bottom left, you can
see there's two points down there.

So that's people who had very rare,
relatively low job performance, and they

also scored relatively low on the test.

And then over at the top
you have one blue dot.

That's the hot, the highest scorer
who also does very well on the job.

And that black line is a trend line
that shows, how did those work together.

So in this case, it's what you
would hope for from a test.

The lower scorers have
lower job performance.

The higher scorers have
higher job performance.

So this kind of study would be taken
with existing incumbent employees.

And that would be one way to do it.

Or you could test it against
job applicants and then wait A

year and measure how well they
were doing on the job and then

go back and correlate it later.

That's a hard thing to do because
people don't like to do all that

work and wait a year for the results.

So it's usually done called a concurrent
study that's done with incumbent

employees, in which case, you can get the
results at that particular point in time.

But what you end up knowing is that people
who score higher on the job are going to

have a greater degree of job performance.

And so you can be very confident
finding what your cutoff score is

and then interviewing those that
are north of the cutoff score.

And you can feel very confident that those
people are going to be good performers

on the job, at least in terms of this
particular aspect of job performance.

Now with the content validated
skill and ability testing, you

don't necessarily find that.

Again, to go back to the person who
types 35 words a minute versus the

person who types 70 words a minute,
that 70 scorer isn't not going to

be twice the performer that the 35
per minute scorer is going to be.

In many cases, what we're looking for
is, do you reach or exceed a threshold?

And that's really all
that we want to know.

And that's where you're, content
validation approach comes into play.

But in many instances, you can go on
and do a criterion validation study

that can be very helpful as well.

Personality tests are
always criterion validated.

So basically you're looking at what
are the personal aspects of a human

being that would contribute to high
performers on the job and what are the

personal aspects of, human beings that
contribute to lower performers on the job.

And so when you're testing for
personality, you're going to look

for those aspects of personality that
match with the higher performers.

And so criterion validation
is the only way to go.

Jenny Arnez: So our tests in
TestGenius for HR and then also for

our dispatcher test, CritiCall we
have the content validation tools but

they are also criterion validated.

Yes.

Yeah.

Can you speak to that a little bit?

Explain.

Mike Callen: Yeah it's always nice to be
able to do a criterion validation study.

And one of the beautiful things on
the dispatch side is that dispatchers,

call takers, telecommunicators,
their jobs are very similar.

There's a universality that exists
between little tiny communities

and large federal organizations.

We we re able to do a criterion validation
study with several different agencies

who are contributing data to it and then
come up with a chart like this one right

here for a variety of different tests.

And then back to the Uniform Guidelines

once again, the Uniform Guidelines
in Section 7b talks about how you

can transport criterion validity.

And so what that means is that for
criterion validation, they will allow

you to transport a validation study from
one organization to your organization

as long as you can establish that
the the job performance dimensions

or domains that were measured in that
original study are very similar to those

that exist within your organization.

And so it's a very simple survey.

You get in the room with 12
people, you go through those job

performance areas, and then they
rate it on a scale of one to five.

One means it doesn't match my environment.

Five means it's exactly like
my environment, and then

all the numbers in between.

And so if you get

12 people to agree that, it's 52 and a
half or better, roughly three, a rating

of three or better than you're going
to be able to transport that validation

study over to your own environment.

And then you can just attach your
transport ability sheets to that

validation study, and you can
literally claim it as your own.

It's a golden ticket kind of thing.

And I'm really glad that you asked
about this because this hits on a

point that we didn't talk about it all.

One of the main reasons that why we have
a validation wizard for content validity

is that you cannot transport content
validity from one organization to another.

You can't do it.

Employers are responsible for establishing
the validity of specific tests within

specific job titles at their organization.

And that's the difference.

That's why we can't do a universal
criterion validation study, because

even if you go with administrative
assistant, what does that mean?

If you go to a hundred different
organizations likely going

to be 100 different jobs with
100 different requirements.

And so you can't really do
that for that particular role.

But that's okay.

Because what we've done is we've
given you the tools that allow you to

do it in a matter of a couple hours.

And now you have a validation report that
says, for Acme County for the position of

Dispatcher I, we've given you to here's
our validation report, and it goes page

after page, and it gives them, really a
blessing to go through and use that test.

You go to another county, and they
want to go ahead and start testing,

they might ask, hey, ACME, send me
over the list of tests you're using.

ACME might share those tests with
them, but it doesn't mean that they're

valid and consistent with business
necessity, nor that the cutoff scores

are job related and defensible.

So that's why you go through the process
because, go back to the coin, it's

defensible on one side and has a high
degree of utility on the other side.

Jenny Arnez: Excellent.

That was very helpful.

Actually I have the
transportability sheet.

Do we want to show that?

Sure.

Mike Callen: Absolutely.

It's a pretty Excel spreadsheet.

It's always fun to show an
Excel file with colors in it.

So yeah, basically, we have
this for the CritiCall test.

As well we have it for the
CritiCall personality test.

And so on the left, you have the
work behaviors and orders are too

small for me to be able to see.

But basically, it's going
through the different dimensions.

That were measured in the original
validation study, and at the top, you

have SME1, SME2, SME as subject matter
expert through SME12, and then on the

far right is the rating scale, where
1 is that it doesn't match, and 5 is

that it does, and so basically, with
the help of a leader, you go through

and you just number the people and you
say, number one, five, number two, five,

number three, four, number four, five.

And you just put the numbers in there.

And then when you get done, it does
a calculation down at the bottom.

And I don't know if you have that here,
if this is just an image, but it will

just validate that all 12 have filled it
in and that the numbers are high enough.

Yeah, there it is down there.

So it's all red now down in the
bottom where the orange box is.

When you fill this out, it'll turn to
green, and then if it's valid, then

that little red box at the very bottom
will turn to green, and that's it.

You print that out, attach it to the
validation report again, and you can now

claim that validation report as your own.

On the content validation side,
you get your own validation report.

So every validation that you do using
the wizard is a custom validation

for content validity for a specific
position at a specific location

for use by those people alone.

Jenny Arnez: Oh, that's really cool.

And we will for our clients, if you're
listening to this and would like to have a

copy of that, just let reach out to us and
we'll make sure that we send that to you.

Mike Callen: Absolutely.

Jenny Arnez: Yeah, so Mike, we've
talked about content validity.

We talked about criterion validity.

I've also heard you mentioned
face validity, right?

Mike Callen: Face validity.

I'd like to use a physics example
here which some people hate physics,

but you have that something called
centripetal centrifugal force.

You're in the car and the car turns
left and the and you slam over into

the right hand side of the car, and
that's called centrifugal force.

And it's really what's
known as a pseudo force.

It's not a real force.

It's just that the car turned
from underneath you and your

momentum kept going forward.

And so that right hand door ran
into you, but it feels like a force.

And face validity is similar to that.

Face validity talks about it.

How does the test feel to the applicant?

Does it feel like a good experience?

And that's another reason why work
sample tests are very important.

If you use work sample tests, you're
presenting information to people, asking

them to do things that are similar to
things that would be done on the job.

But we're doing it in a way where we
don't require in prior job knowledge.

So we're asking them generic things,
not things that they would need to know

about this particular organization.

Like, how do you use Excel
in Acme County, for instance?

That's something that would be trained
and we call it WT and LBO- would train

or learned in a brief orientation.

And so you don't want to
test on those aspects.

But we test things in a generic format
that we can expect anybody with experience

with that particular Excel, for instance,
in this case, to be able to know.

And then we translate
that to how well they did.

And then then it's validated
for the particular job.

So that's how we do that for that part.

And actually we talked
about face validity.

So if a test feels like a job and I fail
the test, I get a feeling that I probably

wouldn't have done a good job on that job,
and I don't feel really bad about that.

Our personality test for dispatchers was
created with dispatch types of items, and

so it feels very much like it's asking you
questions that if you were a dispatcher.

A lot of standardized personality tests
have questions like, do you enjoy long,

quiet strolls on a beach by yourself?

Do you enjoy being in
a loud movie theater?

And so it, those things
are not very face valid.

They don't make you feel like
you just asked me something that

has any applicability to the job.

And so face validity really is not
important in terms of a test being valid,

but it goes back to the optics thing
that we talked about at the begin with

to begin with, we now in the world realm
of social media, everybody's got a voice.

And if I go and I test for a job
at Acme company, and it's asking me

about whether I prefer hot dogs or
hamburgers, I'm going to talk about that.

It's oh, when you take that test,
it's going to be stupid, now all

of a sudden you're branded as,
the people with the stupid test.

And so those are the reasons why you
want to know about face validity.

And that's another thing that you
can really capture, not directly

with our validation wizard.

But you've got a group of people
and it's asking them questions like,

is the, here's the knowledge, skill
or ability that's being measured.

Does it feel similar enough to the way
that the knowledge, skill or ability

is in use on this particular job?

So you do end up getting, that
litmus test as to whether or not

there was a face validity or not.

And that can be very helpful
just in terms of your reputation.

Jenny Arnez: Oh, for sure.

I've taken our CritiCall test and
just the stress that I felt and

then just not having the opportunity
to finish one job before the next

call came in was really stressful.

Yeah.

Mike Callen: And it's
really differentiating.

It's the thing that says, I started
taking that test in 1999 before

it was released and I'm not a
multitasker either, and I hate it.

So when I'm talking to people in the
dispatch community, it's all about,

Hey, look, I could never do your job
and I've never done their job, but

I've done the test and I hate the
test, and I don't score well on it and

so it's very effective in terms of it.

In fact, you know what happens with the
CritiCall test is people like me who are

not multitaskers will typically get up
and walk away from the test rather than

getting hired and get up and getting
up and walking away from the job after.

Somebody spent 60 or
$70, 000 to train them.

So it's a very useful attribute of
the testing process to make people

like you and me uncomfortable and let
the real multitaskers get the job.

Jenny Arnez: And we have
other tests too, right?

We have a situational judgment tests
that actually also is a face valid too.

And.

I found the same experience there.

And I think one of the tests had
to do with technical support or

customer service or something.

So all that's to say that TestGenius, and
I'm not trying to make a sales statement

here, it's just something I'm really
proud of about our software, is that our

tests are face valid You can have the
opportunity to make it content valid.

And then also we have tests that have
been criterion validated as well.

Mike Callen: Yes.

Yeah.

And those video situational tests that
you referred to are phenomenal tests.

They're really great.

And they really hit on
performance to mention attributes

that the other tests don't

don't hit on.

Not only does it ask you how you would
respond in certain situations, but it

asks you how will that person feel about
your response, given your response.

And so there's that it's not an EQ test.

It's not a, um, an EQ test per
se that is emotional quotient

which is a different aspect.

But what it does is it requires that
somebody has a degree of EQ to be able

to empathize to the person that they
would be talking to so that they can

understand how that they were feeling.

And in many jobs, it's a really important
aspect, particularly if it's front facing

or, interacting with members of the
public is just a really important thing.

And that, and that brings up did we
bring up the image with the the test?

The person the testing, the,

Jenny Arnez: oh, we didn't, no.

Mike Callen: Yeah let's show
that for a second, because that's

a, that's an important aspect.

We can close on that.

So basically, when you're
testing, that would be your job

applicant over to the left there.

And so as you add tests to your
test battery or things that you're

talking about in an interview, you're
going to get little tiny slivers

of job performance from each one.

And so by having a multifaceted
testing strategy, you actually

continue on through this battery,
and it might not all be TestGenius.

It might be personality testing.

It might be the Logi-Serve
situational judgment testing.

It might be the interview.

It might be all sorts of other
aspects, training and experience.

But each one of these things is
going to contribute to the picture

of how job ready this person is.

And as you continue on going through
this recruitment process, you're going

to learn more and more about them.

And if you pause for just a second
right here, what you want to really

remember about this, is that you
want to get as many different pieces

of the pie as possible, but it's
equally important to make sure that

you're measuring it in the right way.

For instance, audio comprehension.

You could call something audio
comprehension, but it's not, if it's

not valid and consistent with business
necessity, it's not going to give you

the results that you would expect.

So that's just to something,
something to consider and you

can go ahead and fill it out.

You're never going to, you're never
going to know everything that you

need to know about a person, but
you get to a point where, you've

measured all that you can measure.

And what it does is it
helps to mitigate the risk.

I know 80 percent of what I can know
about this person from having spent time

with them, talking to them reading their,
job application, their experiences,

testing them, looking at those results.

So my, my risk is very low in terms
of what I don't know about them.

And, so you can make a hiring decision
with a greater degree of confidence when

you have a testing strategy that fits
very well with the particular situation.

Jenny Arnez: Yeah, I love this.

This and you've already mentioned
this, but this list, it looks like

a dispatcher position, but it could
easily be administrative role where you

might have Microsoft Excel on there or

Mike Callen: right.

We're all sorts of

things, telephone answering they're
just, spoken ability, English

abilities, foreign language ability,
we can measure all of these things.

But it goes back to your
test selection wizard.

You're going to go through and you're
going to choose the duties that are

appropriate for that job, which are linked
to tests that are appropriate for that

job that you then go on and validate.

And then you administer and you have
a really good, holistic recruitment

strategy if you follow this process here.

Jenny Arnez: Yeah.

And that's really something
to feel good about.

You're creating a process that hires
the candidate who's most likely to

be successful on the job, your job,
which leads to their fulfillment and

you as an employer having a good fit.

And it's fair and defensible.

That's something to be proud of.

Mike Callen: Yeah, it's fair
and defensible and going to

end up with the right people.

One of the things that I don't
think people put enough emphasis

on, it's not just about the person.

It's not just about the person that you
hire or the person that you don't hire.

It's also about your existing employees.

They're the people who are carrying the
burden when you had the job openings.

And so you want to reward these people by
bringing somebody into the organization.

That is going to hit the ground running
that has the requisite skills and

abilities necessary that can be trained
up quickly, and then it's going to

fit with that organization and fit is
something that maybe, maybe only a human

being in an interview can determine.

So you want to make sure whoever
you're interviewing is somebody who

has the requisite skills and abilities
already, because if you find that fit.

And you fall in love with that
person and they can't do the job.

You have done a huge disservice
to your existing employees.

And they will get so depressed at
having gone through that process, waited

for somebody, got them all trained up
only to find they can't do the job.

And so the order is very important.

In fact, if you go back to that
first Acme slide that you have there.

A lot of people will do this after
a phone interview and that third

brown box selected candidates
take the preemployment test.

Our recommendation is, ask
people on the application.

Everything that you can ask.

Are you willing to work nights?

Are you willing to work?

Evenings, holidays.

Whatever is appropriate for that job.

Ask them on the application.

Then take the people who are qualified
applicants and then give them testing.

Don't ever even talk to them because
you risk falling in love with the human

being who is not able to do the job.

And in fact, you're using human resources.

And I'm saying that term generically
human, the resources of human beings.

You're using them to talk to
people, who might not even

take the pre employment test.

So why not save that valuable human
time with only the most highly qualified

applicant or candidate pool so that
the things that, that software cannot

discern, that a human being can discern
get sorted out in the interview.

And that will result in, the, a really
great hire with really great benefits.

Jenny Arnez: Yeah, that's, I'm so glad
you brought it back to this, Mike.

Thank you for that.

Mike Callen: Many

people do this.

They'll come up to us as shows or
on, on the phone and mention it.

And, we have to be careful
because, it might be, maybe

that's their civil service laws.

They have to do it that way,
so we were a little soft handed

in terms of telling people.

You know what it is that they can do
or should do, but the recommendation

is definitely leave the humans out
as long as possible and only employ

the human beings time when you
know you're working with a really

highly qualified list of candidates.

Applicants apply, candidates are
the people that you move forward.

And that's a differentiation, but
that's where you're, where you humans

should jump in and start measuring the
things that only humans can measure.

Jenny Arnez: I think we're
reaching an ending point today.

And Mike, I always learn so much when
I talk to you, I've worked with you

and known you for a long time and now
you've added another layer on for me.

So I appreciate it.

Mike Callen: I appreciate
all your questions.

Your questions are great.

And they really, cause you to start
thinking about the process and

think about other aspects of it.

So hopefully the discussions that
we've had today will be ones that

will result in, positive step forwards
for people who listen to the podcast.

And yeah, it's great.

Really.

Jenny Arnez: I hope so.

Make sure friends to
check out the show notes.

We'll have a links to the Uniform
Guidelines in there and other information

that we hope that you'll find useful.

So thanks again for joining us today.

Voiceover: Thanks for tuning
in to Testing Testing 1 2 3.

Brought to you by TestGenius
and Biddle Consulting Group.

Visit our website at testgenius.com
for more information.