AI & DEI Survey Findings With Henry Tsai of Greenhouse

William Tincup: [00:00:00] Ladies and gentlemen, this is William Tincup, and you’re listening to Recruiting Daily Podcast. But today we have Henry on from Greenhouse and our topic today is AI and DEI survey findings. They’ve done a survey as they do great surveys and great reports. And Henry and I are going to talk about the findings of this particular report.

So Henry, would you do us a favor and introduce yourself and what you do at Greenhouse and Greenhouse? [00:01:00]

Henry Tsai: Yeah, absolutely. Thanks. So my name is Henry Tsai. I run the product team at Greenhouse. We are just trying to help make companies better at hiring. We have this amazing software that really is about the the benefit of structured hiring and removing bias from that process. And we hope that it really does add a net benefit to everybody.

William Tincup: So do you, not report up, but the, is John who you report up into?

Henry Tsai: That’s right. I report into John Strauss, co founder and president. God bless you. Just

William Tincup: God. No, I’m just kidding. John and I talk on a monthly basis and I absolutely love him.

I’m actually jealous that you get to report up into him. Tell us tell us a little bit about this survey. What were y’all trying to figure out? What was the genesis of the idea of why you put this particular survey together and we’ll get into the findings.

Henry Tsai: Yeah, I think generative AI is a real hot topic especially in our industry.

And I think we wanted to just get a sense and a pulse of [00:02:00] how people were thinking about it, how our customers were thinking about it. Just to give an understanding of how fast it’s moving, how much it’s changing and how people really think about what it could do for for their customers and ultimately for candidates.

William Tincup: So what did we learn? What, what did, if we start with kind of the high level learns, what did we learn with this particular survey?

Henry Tsai: I think we learned that people or customers, HR leaders are using it today. 80% of them are using it. 50% of them are not yet monitoring for bias. And I think the…

HR professionals believe that it’s going to increase over time the use of bias and also the use of AI. And I think they have concerns about it. So it’s it’s an interesting perspective that they think it’s going to add a lot of value. They also are cautious about it and concerned about the implications of what it might do to

William Tincup: The [00:03:00] responsibility of I guess the algorithms to make sure that it’s not being biased is is that kind of a, kind of something a greenhouse, but also their clients? Are the, is it something that y’all work together on? Is it something, as a product owner, is it something that you’re constantly monitoring with the efficiency of an algorithm, but also how it degrades et cetera?

Again. When we created an algorithm, we put in our own biases, whether or not we like it or not, we do, because humans put in these, that’s how it starts. But over time, it gets smarter. Now it can get smarter and become more biased.

And just be faster and more biased which, that’s probably not our desired outcome, but what’s the responsibility of kind of the vendor and what’s the responsibility of the practitioner and kind of the communication layer between the two?

Henry Tsai: Yeah, that’s a great question. I think that’s what laws like the New York one are trying to [00:04:00] figure out.

I think the premise is that. We have, we as a vendor or as we like to call as a partner to our customers have a, I think have a responsibility to really show what the algorithm is doing what its intent is and what the dataset that is being trained on so that it can be audited in a way that it can be improved upon, but also that it’s held accountable to the to the outcome it’s trying to derive.

Thank I think when we are outcome driven, so it’s just saying I want to hire as fast as possible, use data sets that may include bias without you even knowing it. And so I think it’s ultimately responsible. I think the law is trying to say it’s ultimately responsible on the customer.

And the vendor’s job is to make it transparent. And I can tell you not all of them know what’s going on inside the machine. Oh,

William Tincup: 100%. The black box is the black box for a reason. You mentioned audited, and I’m curious as to your take on the auditing process, either [00:05:00] internal or external, and the ethical communication layer that needs to be there to just your viewpoints on what needs to be, what is ethical AI?

And again, I think why I bring that up is because practitioners are using it. Great, Jack, but they don’t trust it. They don’t trust the outcomes as much to be less biased. So if I can have that then the ethics might actually help. Having done more discussions about ethical AI might actually help them trust it more.

They trust the data more.

Henry Tsai: Yeah, I think you’re spot on. I think the problem is we’re moving, the industry is moving so quickly, right? Both from an adoption standpoint, as well as the infrastructure of AI technology that’s powering it all. And I think the first part is that customers need to be asking vendors the question, how is it affecting [00:06:00] my outcome?

What is the algorithm doing? And educating themselves. to get up to speed, right? I think in our survey if you just go down the list of eight percent of people of respondents said that we use another software to monitor and evaluate ten percent said we randomly spot check, twenty percent said we manually review and fifty percent said we don’t monitor, right?

So if you look at

William Tincup: the numbers, none of those numbers are good. None of those numbers are good.

Henry Tsai: Terrifying. They see we see in the industry an increased use of it. And I would be remiss to say it’s not all equal, right? It’s not just there’s this, doom and gloom machine where all hiring decisions are automated.

We’re seeing it mainly used in assessments, right? Helping you write content. And so those things probably have a lower necessity. For bias, but you should be asking the question, regardless, we should be learning a methodology or a framework [00:07:00] on our, can we validate what’s actually happening with a technology that may be further ahead than we’re that we understand.

And so if we don’t, if we can’t even ask the question, we should have some pause. And I think that’s where it’s the responsibility of ethical. ethical vendors who are using AI in those areas to show where they’re using it and how they’re using it. And it’s for vendors to ask the question to at least prompt that,

William Tincup: I think that’s a, that’s actually probably a great webinar or ebook is just the prompts the prompts that practitioners should ask of vendors and partners of, okay, I’ll, Everyone’s saying they’re using AI.

Might be machine learning. It might be something else, but okay, fine. How are they using it? And understanding not just how are they using it, but how they can check and just validate for themselves so that they trust it. A lot of this is them trusting the data and HR and recruiting have a long standing [00:08:00] relationship with distrust and data.

So if this is not new. But the thing is with AI, they might not, they might understand it in their personal life, but then they might not understand it as much as it relates to the work that they do. And so I’m worried, just naturally about that drift of them being more distrustful of data over time, because they don’t know what questions to ask.

Henry Tsai: Yeah, I think that, that’s a great point. And not to shift the conversation as it’s your podcast. No, you’re good. You’re good. But candidates use it, right? Candidates are using it to write cover letters to help them produce better resumes. And John and I were joking about this the other day about maybe the questions that we didn’t ask on interviews are now, how do you write prompts for generative AI?

If you think about like, When people used to write on their resumes, I have Microsoft experience in Word and Excel. But you get to can you write, Visual Basic? That is the real value. Can you do can you write a macro? And it’s [00:09:00] almost that the skill set now is not about the content that you’re creating, but how you do it.

And give this kind of goes back in if I can connect it back to what I was trying to say earlier about the vendor and the customer. It’s really about how you do it. There’s going to be with how the world has digitized. I think the proliferation of or the immensity of content being created now is so much greater.

And so you think about like application rates through time per job, there’s just so much more that’s happening because we have so much access to technology. And I think vendors are using this as how can we help our customers suss through all of it? And generative AI is amazing at its ability to summarize and to break apart the, the key concepts in there, but without knowing.

Both what the candidate is trying to accomplish and what you as a customer are trying to accomplish. It can just be a tool that’s not being used properly. And I think that’s when, unfortunately, bias and[00:10:00] just bad outcomes will come. So do

William Tincup: you think you, we see a world in the next couple of years where there’s a person on the vendor side, not on the vendor side, on the practitioner side?

I would say like in recruiting operations, but maybe it’s someone different. That’s their job? Like their job is to actually go through all the tools that they use and constantly make sure that it’s doing what they want it to do, et cetera. Do you see this, do you see this evolve to a person just being in charge of this?

Cause my natural inclination is okay, everything makes sense. We trained them. They’ve got other jobs, like a global head of talent acquisition. They’ve got a full time job just putting all the pieces together. So who, once we trained them. Who’s actually going to go do it and test it?

Henry Tsai: I think you’re on to something. You think through time where recruiters first started with, pieces of paper and they moved to spreadsheets and they moved to, ATSs. And [00:11:00] through that, as technology evolves in every sector, this happens, you get specialists, Salesforce, there are Salesforce admins and developers.

And so I think in a world where AI or generative AI becomes the key tool set, there is going to be, has to be some expert who has both the subject matter expertise of hiring and finding candidates. To ensure that the world is being, moved forward in a responsible and ethical manner.

I think every industry does that. If I think of risk and insurance, that was the world of actuaries and actuaries still exist today. And in fact, probably are more important to that industry just than the software that’s being that’s being used to maintain it all. So yeah, I think that probably, I’m not great at predicting the future of where jobs may go, but I think it’s a good bet.

I think it’s a good bet that expertise probably looks, has to live somewhere in the hiring space and in that technology stack.

William Tincup: That’s right. Yeah. Cause, cause if someone doesn’t own it, if we say [00:12:00] everyone owns it you find sometimes you talk to a practitioner and they’ll say everyone owns diversity. And of course my retort is if everyone owns it, no one owns it. That’s right. So my fear is again, I’ve I’m loaded with fears and anxieties, but my fear is that we teach, we train, we get them up to a certain speed.

They understand. And then. Who does it? And so that’s just something for us to solve. And it’s algebra. It’s just something for us to solve for in the future. As you mentioned, biases, something I’m playing with lately is just unpacking biases. So of course we start off with an algorithm to let’s do something gender.

Okay. Like the bit with Amazon that years ago, I’m not that far along but basically they started with an algorithm and it wanted to hire white middle aged males. And I thought, first of all, don’t pull the plug on that, actually learn from it. That’s not a bad thing. It’s a good thing to learn that it’s not doing what you want it to do [00:13:00] now.

How do you get it back on the tracks? So like gender and race and all the different things that you can think about. It’s the things that. Are coming next that I’m wondering what your take is like the biases that we don’t know, like we, there’s, we, there’s, we know of certain biases and y’all have a list, right?

There’s all these different types of hiring bias. We’re going to learn more as we go about other types of biases. And so it’s going to, there’s going to be a calibration or recalibration for every client, and maybe it’s just at greenhouse proper, and then it just goes out to everybody, and it’s something that’s centralized, but it’s just like, how do you, how are you anticipating where you’re looking?

Learning about biases and learning about, again this is Don Rumsfeld, knowns and unknowns. I don’t know if you ever saw that bit.

Henry Tsai: I love that bit.

William Tincup: It’s really a Marx brothers bit. It’s who’s on [00:14:00] first. And I don’t think he meant it that way, but like the guy, the reporter raises his hand and is Hey what do we know?

And then he goes into the bit of there’s knowns and known unknowns and unknown unknowns. And it’s just a great comedic bit, but yeah, the unknowns is really the thing that I want to scratch out here is how are you? Personally, and even your team thinking about, okay, there’s a whole series of things that we know now, how are we tracking for the things that we don’t know as it relates to hiring bias?

Henry Tsai: Yeah. I look at it and I think I alluded to earlier, there are certain elements of the hiring process. That have an over or are weighted more heavily in where bias would have a terrible impact. And those are the areas where you have to tread more cautiously and use more data or a better training set.

Or de risk it by not having the machine write [00:15:00] the rules. So we think of what is the purpose of all this AI? It’s time savings. It is better decision making. And when you use it in a method of I’ll just use an example, right? Recently there was this MIT student. I don’t know if you heard this.

And she used this app and said, make my headshot more professional. And it turned her into a Caucasian blue eyed herself. Welcome to the Aryan

William Tincup: nation.

Henry Tsai: Great. Yeah. It was very disturbing for her. And she’s a computer science major. Yeah. I get what’s happening, but the company that provided that, that service or the application was like it’s almost like the broken clock mentality.

Yeah. Broken clock, accurate twice. And that’s what happen. And I was the actual problem behind that is you don’t have a rule in place. Or you’re letting AI write the rule where anything can go. And you’re seeing this with how people are jailbreaking making bombs using chat GPT.

You phrase it in a different way, and [00:16:00] this goes back to prompt writing. If you phrase it in a way that tricks the machine or the safeguards of the machine, and you allow it to write those rules, that’s where you get into trouble. And fundamentally software, the digitization has always been about how do you automate things so that a person can make better decisions.

Have more time to really get context and build more understanding about the nuances of the human interaction. And I don’t think that generative AI changes that, but I think we need to be more aware of what places does it make the most sense. And so if you say I want it to filter out X person or X attribute of a person, and there’s bias in that rule because you’re trying to get to an outcome of hiring people faster.

That’s up to the human to decide that’s not a rule that I need to put in place, right? And allow the machine to write that rule because that just says, I need my time to hire to go to one hour. Yeah. It’s going to do what it’s going to do what you told it to. It’s going to do what it told you to and you’re not going to know why it did it.

And so I think that’s the key when we’re talking about that [00:17:00] person who needs to exist or that role that needs to exist is someone needs to be asking that question and someone needs to have the expertise both in why you wouldn’t want to do that and what candidates. You’re not going to be able to get if you put those restrictions in versus the ones that you do want that will actually help you get the best candidates and the best candidate experience.

William Tincup: Oh, that’s fantastic. Now, two other things. One is when I, whenever we go through findings of a report, I always ask the same question is like, when you went into this, you probably had some assumptions maybe thought. Things would play out a certain way. Is there anything in the findings that either shocked you or your look at it is okay, now that’s really good.

That’s a good number. That’s a great finding, et cetera. Is there anything in the report that you can think of?

Henry Tsai: I think what I was surprised by is it’s The fear and the excitement all in one, where you’re like, I’m scared because I don’t know what’s happening. But also I want to use this I like it.

I think that [00:18:00] was the, it’s, I think it, what it says is this isn’t, it isn’t all hype. There is real value here. And I don’t, I think people need to the message that I would take away from it is I don’t think you can shy away from it. You can’t put your arm or raise your nose to it that this is not going to happen.

And I think being able to show that customers want this. HR professionals want this and I think that comes back to the supply of data that’s coming into them is going to be insurmountable to find the diamonds in the roughs or to get there. And so you have to embrace it, but if you embrace it with the hat of responsibility and ethics, I think you will get the outcome that you want to get.

And I, that’s the, that’s the big takeaway that I saw out of this.

William Tincup: Yeah it’s really interesting to me, especially the juxtaposition between the two. It’s, they’re using, so there’s an excitement, yet they’re terrified. It’s if you’ve ever been bungee [00:19:00] jumping or if you’ve ever been skydiving, I’ve done both.

There’s a moment in which you’re like, why did I talk myself into this? And then once you do it, you’re like, That was fantastic. I’ll do that again. So I can see the terrifying and excitement running together. I think something that, that you also mentioned along the way is like people are using it.

Candidates are using it. Recruiters are using it. Hiring managers are using it. Everyone’s using it. Now, are they using it properly? That’s the terrifying part. I think that that the report is teasing out. Last thing is where do we find the report? Where’s that located?

Is it on the website?

Henry Tsai: It is on the website. I encourage everyone to look it up and look at the data. And draw their own conclusions, on who you choose as your vendor is going to be really important going forward. Who is going to be transparent with what they’re doing who is going to partner with you on getting the unbiased results.

I think it’s going to be really important. I think that’s [00:20:00] what’s going to differentiate us. It’s… It is very exciting, I will say, and it reminds me that change is hard. There are industries that will be, I think, inequitably affected especially in the content space. But ultimately, I think, with every technology that has changed lives, There’s always the expansion and the opportunity expansion.

I think that’s what that person needs to sit in there will probably doesn’t exist today and will exist tomorrow and will be extremely important. Yeah.

William Tincup: When the internet came along, it it destroyed the classified. Business. So the classifieds, if you ever read a paper they had a section in the paper that there’s classifieds.

Yeah. There’s no longer there because it’s on the internet. There’s just a better utility of that. There’s a better mechanism of that, but you’re right. The change, getting up that change, if you were a classified ad writer, that was a massive amount of change, but. Again, X number of years later, you can’t imagine [00:21:00] looking at a paper and looking at a paper, a section of paper and trying to find something that’s no one would want to do that because it’s just a

Henry Tsai: better way.

Yeah. No one would want to do that. And yet Craigslist.

William Tincup: I think Craigslist is actually, I think it’s a, there’s a darker side to Craigslist that’s what the reason it exists. Do you see this as a kind of longitudinal study that’s something that you’ll want to do every year to see how the audience is changing?

Henry Tsai: I think so. I think you’d want to see how it shifts.

I think there’s a lot to add to it as well. Clearly I don’t run this part of the business, so I don’t want to speak on behalf. But as a product person, as a person who would like to know the results, I would love to know the results. I think they’re really interesting. And we don’t really get to see this that often where you get an insight into people’s minds.

Like we do it with our customer advisory board. So you get a smaller sample size, but having this wide. Of a viewpoint and from from a [00:22:00] brand like Greenhouse, I think there’s a lot to learn from it.

William Tincup: You mentioned a really important part for the audience and that’s, the vendors that are the more transparent, the more, the less black box, the more they can explain what it’s doing, what questions, et cetera.

So I also think the vendors that do a better job training and teaching and certifying and getting the acumen and literacy. Of practitioners up to a certain level, I think those vendors are also going to win, cause again, it goes hand in hand and you can, transparency shows people that, Hey, we’re not hiding anything.

In fact, we’re going to teach you how to ask questions and to make sure that it’s on track doing what we want it to do and what you want it to do. So I think those vendors that just double down on training. Teaching and building classes and building content for folks. I think they also are going to win.

Henry, you’ve been fantastic. This has been awesome. Thank you so [00:23:00] much for coming on the podcast. Absolutely. Thanks for having me. Absolutely. And thanks everyone for listening. Until next time.