The Red Dress Effect: Are women in red sexier?

[Regina] (0:00 - 0:15)
We need to be transparent with our readers and tell it like it is and say, hey, this effect was found in a bunch of young college-age dudes who might be failing psychology class and are tired and probably hungover.

[Kristin] (0:20 - 0:44)
Welcome to Normal Curves. This is a podcast for anyone who wants to learn about scientific studies and the statistics behind them. It's like a journal club, except we pick topics that are fun, relevant, and sometimes a little spicy.

We evaluate the evidence, and we also give you the tools that you need to evaluate scientific studies on your own. I'm Kristin Sainani. I'm a professor at Stanford University.

[Regina] (0:44 - 0:50)
And I'm Regina Nuzzo. I'm a professor at Gallaudet University and part-time lecturer at Stanford.

[Kristin] (0:50 - 0:55)
We are not medical doctors, we are PhDs, so nothing in this podcast should be construed as medical advice.

[Regina] (0:55 - 1:53)
Also, this podcast is separate from our day jobs at Stanford and Gallaudet University. Kristin, today we're going to dive into psychology and really hit the sex and beauty angle of our podcast.

[Kristin]
Ooh, psychology. I'm intrigued.

[Regina]
Fun. We're going to talk about what's called the red dress effect, and that is a claim that wearing red can make a woman more physically and sexually attractive to men.

We'll look at the experimental evidence on this, including one of the original studies, the controversies that followed that work, and what the latest studies say about who this effect might work for and who it might not. I thought we would kick this off by seeing how this red dress effect is sometimes portrayed in the media. And I have a good one right here.

It is from your very own beauty magazine, Allure Magazine, and it was published in 2013. So when did you write for them again?

[Kristin] (1:54 - 2:01)
First of all, not my magazine, but I did write for them from 2005 to 2015.

[Regina] (2:01 - 2:15)
Hmm. Well, first of all, I always will think of it as your magazine. So that was during your era then.

Yes. But I'm guessing you were not covering the sexy red fashion beat. I did not cover fashion, no.

[Kristin] (2:15 - 2:21)
I covered health, things like wrinkles, smoking, weight loss, skin care.

[Regina] (2:22 - 3:18)
Which I guess are as important as fashion, I'll give you that, and more science probably behind that, too. All right. This article, which is not your article, it's by Grace Clark, and the headline is, How to Be Sexy, Seven Weird But True Rules of Attraction.

I'm going to read you the part that talks about red, all right? Wear red, yes, it's that simple. A bit of crimson has the power to put men in a lusty state of mind.

One study even found that they were 56% more likely to ask out a woman in a red top versus a white one. That's because men are hardwired to respond to the hue. The color was used as early as 10,000 BC in lip color and blush to mimic the red flush we get when we're turned on.

Throw on a red dress and let primal instinct take over.

[Kristin] (3:18 - 3:36)
Wow, that writing is very engaging, very evocative, nice summary of the red dress effect. I thought so. Of course, I'm always skeptical, and so my mind immediately goes to, you were talking about a study, this 56% more likely to ask a woman out, and I'm thinking, was that an odds ratio?

[Regina] (3:37 - 3:56)
I love how your brain goes, because it's not asking, exactly what shade of red can I use, you know, to drive men wild? What is the odd ratio study design? But you, Kristin, are right to be skeptical, because these were pretty strong statements.

Red will put men in a lusty frame of mind. That's very causal language.

[Kristin] (3:57 - 4:13)
Very causal language. To be fair, what, this author had maybe 100 words to describe an entire body of work in a study. You can't always get all the details in, in this kind of writing for the lay public, but there's no reason you can't go and pull the study.

[Regina] (4:14 - 4:18)
Exactly. Because, as you and I know, it's not always the same as what we read in the headlines.

[Kristin] (4:18 - 4:22)
There's always more detail, so it's fun to go pull the studies. That's what we do in this podcast.

[Regina] (4:22 - 4:41)
That is exactly, and that's what we're going to be doing today. All right, before we get to the science, I feel like we need to address really the burning question that's going on here. And that is, Kristin, do you have a red dress?

Do you wear it? And, most importantly, when you put it on, do you notice that it drives men wild with lust?

[Kristin] (4:41 - 4:46)
You're going to be so disappointed, Regina, because I don't think I own a red dress at all.

[Regina] (4:46 - 4:49)
I've seen you wear red. Your Stanford red gear.

[Kristin] (4:49 - 5:03)
Oh, well, the red cardinal, the Stanford pullover that we both have. That every time I see you, we are twins because it's our favorite shirt. I don't think that's what they're talking about when they're talking about wearing red to attract men.

[Regina] (5:03 - 5:04)
I beg to differ.

[Kristin] (5:05 - 5:06)
The Stanford sexy pullover?

[Regina] (5:06 - 5:10)
The sexy Stanford. So, it is giving nerd vibes. Sexy nerd vibes in there.

[Kristin] (5:10 - 5:12)
I'm skeptical, Regina. Yeah.

[Regina] (5:13 - 5:15)
I think that's the best kind of sexy there is.

[Kristin] (5:15 - 5:20)
But it might be the right shade of red, because it is a nice bright red. It's cardinal red. I think cardinal red.

[Regina] (5:20 - 5:39)
I don't know if red is my favorite color. I don't think it's great on me.

[Kristin]
It looks good on you. With your color tone, yeah.

[Regina]
Oh, okay. Thank you.

Because I confess that for my online dating profile right now, I do have as the main picture me in a red sweater, which I had to borrow from a friend, but I'm trying to get some of this red dress effect.

[Kristin] (5:39 - 5:44)
Is that evidence-based dating? Did you actually put the red sweater on because of the studies? I did, actually.

[Regina] (5:45 - 5:46)
I did, yes. I know.

[Kristin] (5:46 - 5:53)
Well, it must work, because you have gotten tons of dates. It's gotten a lot of attention. So, N of one, yes.

It's effective.

[Regina] (5:53 - 5:56)
We don't know it's causal, though.

[Kristin] (5:56 - 6:00)
You should do...

[Regina]
A controlled experiment?

[Kristin]
Yes. Two dating sites, two different colored sweaters.

[Regina] (6:00 - 6:01)
Yeah. Green sweater here.

[Kristin] (6:01 - 6:01)
Yeah.

[Regina] (6:01 - 6:04)
Red sweater here.

[Kristin] (6:04 - 6:11)
Yeah, we're not at all nerdy, Regina.

[Regina]
Yeah, right.

[Kristin]
And, Regina, you actually talked about this red dress effect on the Today Show.

This was a few years back.

[Regina] (6:12 - 6:28)
Can't believe you remember that. Yes. It was a few years back, many years back.

That was back in the day when I was writing for the LA Times, and I had a column called The Mating Game, and we described it as the science behind mating, dating, and sex.

[Kristin] (6:28 - 6:39)
I'm so impressed that you wrote this column, Regina. I just want to clarify, again, for listeners, this was about the science of sex. This was not about your personal dating life, not like sex in the city.

[Regina] (6:39 - 7:04)
This was not the science of Regina's sex life. That would be a very boring column, I got to say. So, this was December when I was writing this particular article, and I wanted something that fit with a holiday theme.

So, I wrote a piece giving people psychology, you know, tips and tricks to hack their way into finding a mate at a holiday party. That was the thing.

[Kristin] (7:04 - 7:09)
We're now finding mates at, like, the company Christmas party, seriously? Christmas party.

[Regina] (7:09 - 7:18)
I feel like you have to grab whatever opportunity presents itself, and if you're going to be there already, why not? Just like if you're on a plane, right?

[Kristin] (7:18 - 7:19)
Sure.

[Regina] (7:19 - 7:29)
An airport, you know, the subway, there's nothing wrong with that.

[Kristin]
Have you gotten dates on a plane or the subway, Regina?

[Regina]
Both, actually.

[Kristin] (7:29 - 7:30)
I haven't heard either of these stories. Do tell.

[Regina] (7:30 - 7:39)
I have so many stories. The subway. The subway fellow is actually very smart and interesting.

He was a mathematician, Ph.D. mathematician and journalist in Bolivia.

[Kristin] (7:39 - 7:40)
Impressive.

[Regina] (7:40 - 7:46)
I know, I know. I feel like produce aisle is also what I hear to be very popular place to meet men.

[Kristin] (7:46 - 7:51)
Back in my dating days, many, many years ago, I did get a few dates at Whole Foods.

[Regina] (7:52 - 7:54)
I think I remember that one.

[Kristin] (7:54 - 7:56)
Crunchy granola types? Yeah.

[Regina] (7:56 - 8:04)
I met a personal chef at Whole Foods, actually.

[Kristin]
To date or to go for a date?

[Regina]
I did not hire him.

We went out on one date.

[Kristin] (8:04 - 8:05)
Okay, only one.

[Regina] (8:05 - 8:50)
One date. Yeah, that was not fitted to be. I feel like you might forget now, as a married woman, what it's like to be single and need to grab these opportunities, wherever they are.

[Kristin]
Christmas party. Sure. I'm with you.

[Regina]
So I went to the literature, the evolutionary psychology, scientific literature, a lot of interesting things, and found all these weird quirks of human attraction.

[Kristin]
Oh, nice.

[Regina]
Yes.

And wrote a whole article. It was a little like this Allure article, except mine was 2008.

[Kristin]
So you scooped them?

I did scoop them, uh-huh. And one of the pieces of advice for women who wanted to attract men was wear red. Actually, since it was a holiday theme, I said that they should wear a sexy Santa outfit.

[Kristin] (8:52 - 8:58)
I'm guessing that the empirical study that was done did not involve Santa outfits because that would be weird.

[Regina] (8:59 - 9:00)
That was just my fantasy. I have to say, yeah.

[Kristin] (9:01 - 9:09)
All right. So you were on the Today Show.

You're talking about this red dress effect that had some studies behind it. Were you actually talking about the study?

[Regina] (9:10 - 9:12)
Like the study design?

[Kristin] (9:12 - 9:12)
Yeah. Yeah.

[Regina] (9:13 - 10:01)
I was not talking about the study design on the Today Show. No. I was there to give advice to viewers.

One of the NBC producers had read the article and said, hey, come up to New York next weekend and give advice for people who are about to go to a New Year's Eve party and need this advice. So I was there in person. 30 Rock, you know, with the big Christmas tree, right?

[Kristin]
Where they tape the Today Show. All that.

[Regina]
Yeah.

So I was there for the Today Show. Stats professor. You got to picture it, right?

Stats professor, geeky stats professor, live national television, giving dating advice to viewers.

[Kristin]
The one and only time.

[Regina]
This was not a repeatable experiment.

I was talking about red monkey butts. That was�

[Kristin] (10:01 - 10:03)
Okay, wait.

Red monkey butts. Why?
Why the red monkey butts?

[Regina] (10:03 - 10:06)
Red monkey butts. Well, first of all, it's just fun to say. Isn't it?

[Kristin] (10:06 - 10:07)
On national TV, sure.

[Regina] (10:07 - 10:34)
Right. That was�I feel like I can die happy now. That might have been the, you know, the peak of my life right there.

But the reason is I knew that the hosts were going to want to know why red? Why not brown or green? So the reason is because we primates, apes, humans, whatever, we evolved a third gene that lets us see red.

Because mammals�I don't know if you know this�they are red, green, colorblind, most of them.

[Kristin] (10:34 - 10:36)
Oh, other mammals? They can't see red.

[Regina] (10:36 - 10:37)
Yeah.

[Kristin] (10:37 - 10:40)
I did not know that.

Interesting. So my dog Nibbles can't see red?

[Regina] (10:41 - 10:46)
Your adorable Corgi cannot see red. He can't see the red ball.

[Kristin] (10:46 - 10:46)
Oh, okay.

[Kristin] (10:47 - 10:48)
So we should stop buying the red balls.

[Regina] (10:48 - 10:57)
Yeah. Stopped buying the red balls. But also, along the same line, fun fact, the matadors, right, with the bulls and the red cape that they're waving around.

[Kristin] (10:57 - 11:01)
Oh, wait.

I thought the red cape was to make the bulls more aggressive. No, that's not correct?

[Regina] (11:02 - 11:06)
I know. I thought the red cape was to make the fans more aggressive, and they buy more beer, maybe.

[Kristin] (11:06 - 11:07)
Yeah, maybe. Okay. All right.

[Regina] (11:08 - 11:17)
So anyway, the question is, why? What was the advantage to us evolving this ability to see red, right? It has to be something.

[Kristin] (11:17 - 11:19)
There had to be an evolutionary advantage. Right.

[Regina] (11:19 - 11:22)
Right. So investigators have theories. Of course, they're just theories.

[Kristin] (11:22 - 11:23)
Right.

[Regina] (11:23 - 11:30)
One of them is that seeing red helps us find the ripe berries, like on the bushes and trees.

[Kristin] (11:30 - 11:43)
Right. As opposed to the green ones that might make us sick.

[Regina]
There you go.

[Kristin]
But that doesn't explain the connection with romance and sex.

[Regina]
Whipped cream.

They go on the berries.

[Kristin]
Maybe not back in caveman days.

[Regina] (11:43 - 11:51)
Okay. You're right. Very astute question.

Bringing it back. The other theory is that it's a social signal for primates.

[Kristin] (11:51 - 11:51)
Okay.

[Regina] (11:51 - 12:07)
And that's the red monkey butt, because many primates, when the females are ovulating, which means they're fertile, they're looking for mate, right? They're on the make, they're in heat. Their butts, they turn bright red.

[Kristin] (12:08 - 12:09)
Oh, really? I did not know that.

[Regina] (12:10 - 12:30)
Oh, it's striking. Like, it's a neon, a juicy neon sign that is out there saying, come hither. Yes.

And it's across multiple species, too.

[Kristin]
Oh, interesting.

[Regina]
Yeah.

So it's clearly a strong need. It's a strong signal that females have evolved to be able to say, yes, now, or not tonight, honey, I have a headache. Stay away.

[Kristin] (12:31 - 12:41)
Okay. And you're saying that this fertility signal, which a lot of primates have, it might be hardwired in the back of the human male brain?

[Regina] (12:41 - 12:48)
Exactly. I feel like it's not, you know, that much of a stretch. So red means ready in primate language.

[Kristin]
Oh, interesting.

[Regina] (12:48 - 12:53)
That's the idea.

[Kristin] (12:53 - 12:56)
But you can test this claim empirically. There have been studies to put this to the test.

Tell us about the studies, because we like to talk about studies.

[Regina] (12:56 - 13:08)
This is what we're talking about. Let's get to the studies here. So the article, the research article that I was covering when I was writing about this, it was 2008 Journal of Personality and Social Psychology.

[Kristin] (13:08 - 13:10)
My favorite light reading for the weekend.

[Regina] (13:11 - 13:21)
Light reading. It's actually kind of interesting. Investigators at the University of Rochester in New York, and they looked at this link between red and sex and romance very specifically.

[Kristin] (13:22 - 13:24)
Was this the first study to look at this?

[Regina] (13:24 - 13:28)
Kristin, are you asking if this was the seminal study?

[Kristin] (13:28 - 13:30)
Did you just make a bad pun?

[Regina] (13:31 - 13:49)
Yeah. All right. That was pretty bad.

That was a bad pun. Yes, it was. It was a seminal study.

It was one of the first to get a lot of media attention on this to really look at this. So they reported on four experiments. And I think let's just talk one of them to get an idea of what they were doing.

[Kristin] (13:49 - 13:57)
So these were experimental studies and not observational studies, not something where they just went up to men and said, do you get turned on by seeing women in red, right?

[Regina] (13:57 - 14:10)
Which, of course, would have a lot of biases.

[Kristin]
That would have a lot of bias.

[Regina]
Right.

Or asking the women, you know, hey, do you get followed around when you're wearing red?

[Kristin]
Right. Okay.

[Regina]
That would be different. No, these were actual experiments where they manipulated a variable.

[Kristin] (14:10 - 14:14)
Oh, excellent. Because that means we have a proper control group.

[Regina] (14:14 - 14:27)
Control group and randomization, all of that stuff. But, okay, I have to say the sample here, male college students who are getting extra credit for their psychology class.

[Kristin] (14:27 - 14:48)
Yeah. So not the most representative group. And, Regina, I think a lot of people don't realize this, but a lot of these psychology studies are actually done on Psych 101 students, right, because these psychology professors have these huge classes, it's kind of a ready-made pool of study subjects, and they're bribable with extra credits.

[Regina] (14:48 - 14:53)
They are bribable. You are right. It's not what we picture, is it?

[Kristin] (14:53 - 14:53)
No.

[Regina] (14:53 - 15:26)
When we read about results of a psychology study, you know, in the media, in the magazine or something like that. So, Kristin, I think maybe next time you and I write about psychology studies for whatever we're writing for, I think we just need to be honest. We need to be transparent with our readers and tell it like it is and say, hey, this effect was found in a bunch of young college-age dudes who might be failing psychology class and are tired and probably hungover.

[Kristin] (15:27 - 15:27)
I like the honesty, yeah.

Context.

[Regina] (15:27 - 15:28)
It's all about the context, yeah.

[Kristin] (15:28 - 15:40)
It's what we call a convenience sample, and it's not always generalizable to everyone else. Yeah, yeah. You personally, Regina, may not be that interested in what 19-year-old college men think.

[Regina] (15:40 - 16:03)
Let's say not at all interested in what 19-year-old boys think of me. All right. So, that sets the stage.

We're working with college-age straight men in psychology class, and here's what the investigators did, all right? Here's the design. They took a color photo of a woman, a white woman, brunette, who was, quote, moderately attractive.

[Kristin] (16:04 - 16:08)
Moderately attractive, according to the PI of the study? Like, who judges that?
That seems a little�

[Regina] (16:09 - 16:18)
Very good question. They actually know this. It was quantifiable because they got it from HotOrNot.com.

Do you remember that?

[Kristin] (16:18 - 16:21)
I don't remember. What was that?

[Regina] (16:21 - 16:29)
You didn't play with this back in our grad school days?

[Kristin]
Oh, this was� Back in the internet era? 2000, 2001.

[Regina]
Right.

[Kristin] (16:29 - 16:36)
So, this was the early internet era. We lived through this. The dot-com boom and bust in Silicon Valley, yeah, we lived through that.

[Regina] (16:36 - 16:37)
We are survivors.

[Kristin] (16:37 - 16:43)
Well, we had no stake in it because we were grad students, and not like we were investing money in the stock market.

[Regina] (16:44 - 16:51)
Well, I don't know. My office mate in grad school went on to found Zappos, actually.

[Kristin] (16:51 - 16:53)
Really? Yes. Oh, you didn�t tell me that story.

[Regina] (16:53 - 16:59)
Yeah. Alfred Lin.

Anyway, HotOrNot.com, I'm so sorry that you missed out.

[Kristin] (16:59 - 17:01)
I am sure that I did not miss out on too much.

[Regina] (17:02 - 17:13)
This was a very fun site. So, what you do is you upload a photo of yourself or a friend. Headshot, clothed, right.

[Kristin] (17:13 - 17:14)
Right. G-rated.

[Regina] (17:14 - 17:34)
G-rated. And then when you go to the site, you see this running stream of photos, and you rate them from 1 to 10, hot or not. And then the key thing is that you get to see the cumulative average of what everyone else has rated that person up until now.

[Kristin] (17:34 - 17:41)
Well, this does not sound very PC. I don't think I'm going to fly today. Isn't that a little, like, objectifying women�well, objectifying men, too, I guess.

[Regina] (17:41 - 17:55)
It wasn't just women. It was both, right? But I think that was the idea, because for people who wanted to know, well, how hot am I?

Like, Kristin, if I were to ask you, how hot am I? I hope that you wouldn't be honest.

[Kristin] (17:55 - 17:56)
Well, you'd get a 10, of course.

[Regina] (17:56 - 17:57)
10, right. Of course.

[Kristin] (17:57 - 17:59)
But you did not upload your own photo, right?

[Regina] (17:59 - 18:11)
I did not upload my photo. You were just looking at other people. I did not upload my photo or anyone that I know, so don't worry, your photo is not on there.

But I did rate them with my office mate, not the one who founded Zappos.

[Kristin] (18:11 - 18:13)
This is what stats grad students do for fun.

[Regina] (18:13 - 18:28)
They're stats grad students. And we went through and we rated them, yeah, procrastinating, of course. And here's my big confession.

We saw people who were, let's say, not traditionally attractive, and we would give them extra high ratings.

[Kristin] (18:28 - 18:34)
Oh, so you were being nice. Yeah.

[Regina]
Yeah.

[Kristin]
You were also skewing the data.

[Regina] (18:34 - 18:47)
We were skewing the data.

And we went the other way. So, if they were just like a little too hot, then we downrated them.

We called it our squashing function, which is a thing in stats. It's a real thing in stats.

[Kristin]
That is very nerdy.

[Regina] (18:47 - 19:05)
Yeah. We were being dorks in there, and we were avoiding our work in there. So, interesting side note. I looked this up.

Hot or not was started by a couple of Berkeley engineers.

[Kristin]
Male, I assume.

[Regina]
Male, yes, of course.

It's right across the bay from us, and it sold in 2008 for something like $20 million.

[Kristin] (19:06 - 19:06)
Wow.

[Regina] (19:07 - 19:07)
Mm-hmm.

[Kristin] (19:07 - 19:11)
Well, a lot of people made some money on weird things in the internet boom.

[Regina] (19:12 - 19:18)
They did. This actually inspired, hotornot.com inspired both YouTube and Facebook.

[Kristin] (19:18 - 19:19)
Really? Okay, I didn't know this.

[Regina] (19:19 - 19:28)
The creators of YouTube originally said that they wanted to create a video version of Hot or Not where you would post your video and people would rate you there.

[Kristin] (19:28 - 19:30)
Oh, that's even more slimy.

[Regina] (19:30 - 19:47)
It is slimy. It's a little like Tinder, actually. Yeah. And then I think they realized that, you know, cat videos were actually more popular than that.

And Facebook, before Facebook, Mark Zuckerberg was trying to create a Hot or Not for Harvard.

[Kristin] (19:47 - 19:54)
Oh, I seem to remember. Wasn't it taking the freshmen, the Facebook, and having people rate the freshmen?

[Regina] (19:54 - 19:54)
Yeah, yeah. Face smash. Yeah. Mm-hmm.

[Kristin] (19:54 - 19:59)
So, Regina, you're saying these sites evolved because male engineers were obsessed with looking at hot women.

[Regina] (20:00 - 20:03)
I think we can say a lot of society has been built.

[Kristin] (20:04 - 20:06)
All goes back to the primate brain.

[Regina] (20:06 - 20:07)
The primate brain.

[Kristin] (20:07 - 20:09)
Which brings us back to the research study.

[Regina] (20:09 - 20:10)
Good job, Kristin.

[Kristin] (20:10 - 20:19)
Back to the research study. So, the reason the researchers knew that she was, quote, moderately attractive is they got, this is her picture of HotorNot.com.

[Regina] (20:19 - 20:47)
I think it was like a six or something.

[Kristin]
So, they were able to quantify it.

[Regina]
Right, right.

That's pretty clever, right? So, in this color photo, she was wearing what the researchers described as, quote, a plain, but form-fitting shirt. Form-fitting.

They Photoshopped it to have two versions.

So, one where she's wearing a blue shirt, one where she's wearing a red shirt, and then they showed men one or the other. They randomized men to see one or the other, but not both.

[Kristin] (20:47 - 21:04)
Okay. So, this was not a within-person design. The men didn't see the woman in red and the woman in blue.

That would have been more powerful. Statistically, that's a more powerful design because the men would serve as their own control. But it would be a little hard to do here because then they would see that it was the same woman, so I guess it's a little tricky, you know, to do that.

[Regina] (21:04 - 21:05)
Yeah, yeah, yeah.

[Kristin] (21:05 - 21:18)
But other important point, this was not a live woman. You're saying this is a photo, which, okay, it's probably a good thing. They don't want to parade women around to be ogled by college-age men.

[Regina] (21:18 - 21:24)
Oh, yeah. But it does mean it's not really real life.

It's just a photo. It is not realistic. And more than that, they asked the men a bunch of unnatural questions.

[Kristin] (21:25 - 21:25)
Like what?

[Regina] (21:25 - 21:49)
Yeah, yeah. I'll give you two just to give an example. One was, on a scale of one to nine, how much would you like to engage in sexual behaviors with this woman?

Ooh. Sexy, right?

[Kristin]
Good pick-up line.

[Regina]
Good pick-up line. And then the other one, picture you're going out on a date with this woman. You have $100 in your pocket.

How much money would you be likely to spend on your date?

[Kristin] (21:49 - 21:55)
Oh, that's an interesting way of quantifying. Although, I have to say, $100 for a dinner date? That's kind of cheap.

[Regina] (21:56 - 21:56)
Right?

[Kristin] (21:57 - 22:04)
2008, so we have to adjust for inflation. These were men in college.

I guess for a college student, this is a lot of money. Right, yeah. Okay, fine.

[Regina] (22:05 - 22:13)
Right, yeah. So, I should mention these were standardized questions.

[Kristin]
Oh, good.

Like validated?

[Regina]
Yeah, yeah. Their scales, other people used.

So, good on them for that.

[Kristin] (22:13 - 22:21)
All right.

What did they find? Did red work? What did they find?

[Regina]
It did. All of the comparisons were statistically significant in favor of red.

[Kristin] (22:21 - 22:25)
Okay, statistically significant. But we always have to ask, how big were the effects?

[Regina] (22:26 - 22:47)
I love that you do that. You and I are statisticians. So, of course, we're going to say statistical significant, sure.

But how big? Size matters, right? Tell me about the effect size.

All right. So, on average, they were willing to spend a little over $30 on the women in blue and just under $60 for the women in red.

[Kristin] (22:47 - 22:59)
Oh, so that's actually a double. That's a big difference. Yeah, that's a big difference in there.

Kind of like changing from maybe a Burger King date to the local sushi joint. Which is a big deal for a college-aged, right, kid.

[Regina] (23:00 - 23:09)
For the other outcome, that one to nine, it was about a two-point boost that you got in sexual desirability for red versus blue, two out of nine.

[Kristin] (23:09 - 23:21)
Okay, that one it's harder for me to interpret. This is what we call a Likert scale. We use these a lot.

I find it a little harder to intuitively know, what does a two-point boost actually mean?

[Regina] (23:21 - 23:23)
Yeah, yeah. How can we compare that to that 30 versus 60?

[Kristin] (23:23 - 23:24)
Right, it's a little easier with money.

[Regina] (23:25 - 23:38)
And you're right, Kristin. These one-to-nine scales are really arbitrary. But luckily, the investigators here did also report all of these results in a way that at least lets us compare across these situations.

[Kristin] (23:39 - 23:41)
Oh, you're talking about standardized effect sizes.

[Regina] (23:42 - 23:44)
Standardized effect sizes. Here they used Cohen's d.

[Kristin] (23:45 - 24:22)
My students love Cohen's d because it simplifies the units.

[Regina]
It does.

[Kristin]
Regina, I think we need to unpack Cohen's d for listeners.

We need to take a short break first. I'm sure that everybody's going to be on the edge of their seats waiting to hear. This is a good teaser.

Come back for Cohen's d.

[Kristin]
Regina, I've mentioned before on this podcast our introductory statistics course, Demystifying Data, which is on Stanford Online. I want to give our listeners a little bit more information about that course.

[Regina] (24:22 - 24:32)
It's a self-paced course where we do a lot of really fun case studies. It's for stats novices, but also people who might have had a stats course in the past, but want a deeper understanding now.

[Kristin] (24:33 - 25:10)
You can get a Stanford professional certificate as well as CME credit. You can find a link to that course on our website, NormalCurves.com, and our listeners get a discount. The discount code is normalcurves10.

Welcome back to Normal Curves. We were discussing the results of a 2008 study that claimed to provide evidence for the red dress effect. This is the idea that wearing red makes women more physically and sexually attractive to straight men.

And we were about to talk about a statistical tool called Cohen's d.

[Regina] (25:10 - 25:36)
Cohen's d effect size. So this is what they call a standardized effect size because we can compare across different situations. The d stands for difference, just to explain what's behind it.

And Cohen is the guy who named it in the late 80s, I think it was, Jacob Cohen, psychologist, statistician. Cohen's d is something we teach in our course on Stanford Online, the demystifying data course.

[Kristin] (25:37 - 25:47)
Right. So for anyone who wants to know even more about Cohen's d, they should definitely look into that course. It's an intro to stats course. I also cover this topic in my medical statistics program on Stanford Online.

[Regina] (25:48 - 26:04)
Cohen's d comes up lots of places and really important to understand. Kristin, I actually love standardized effect sizes like Cohen's d, but they're a little weird if you've never seen them before. So how do you explain it to your students?

[Kristin] (26:04 - 26:13)
I think of it like money.

There are all these different currencies in the world, but no matter where I am in the world, I'm always going to translate everything back into dollars. That's a unit I understand. So it's kind of like a universal currency.

[Regina] (26:13 - 26:20)
Oh, that makes sense. So you can, you know, the meals here in this country or the souvenirs, whatever, you're bringing it back into something you understand. I like that.

[Kristin] (26:21 - 26:24)
So what were the Cohen's d values for that 2008 study?

[Regina] (26:24 - 26:42)
Okay, so we talked about two different scales there. So the two points on that sexual desirability scale, that was a Cohen's d value of 1.1. And for the how much you were willing to spend on the date, that was a d of 1.4. Wow.

[Kristin] (26:42 - 26:51)
Cohen's d is above 1. Yeah. That's big.

Because a Cohen's d of 1, that is a one standard deviation difference between the groups.

[Regina] (26:51 - 26:59)
And a standard deviation is actually a lot. It's actually a lot. Of course, we don't, you know, need to go into the technical details of standard deviation to understand it.

[Kristin] (26:59 - 27:00)
Or another podcast.

[Regina] (27:00 - 27:02)
Another podcast. That's going to be highly exciting.

[Kristin] (27:03 - 27:12)
Oh, absolutely. Standard deviation is very exciting. It's just a measure of the spread of the data.

In essence, it quantifies the background variability.

[Regina] (27:12 - 27:32)
Right. So to avoid needing to think in terms of these standard deviations, because it can be a little awkward these days, when I explain it to people, I usually bring it back to a common scale that Cohen himself had introduced in the book. Yeah, it's kind of interesting.

He used examples of heights of teenagers. So maybe I can just explain that.

[Kristin] (27:32 - 27:42)
Yeah, that sounds very intuitive and concrete. I like it.

We should say first, though, that a Cohen's d of zero, that's a good place to start. Zero means no difference between the groups we're comparing, right?

[Regina] (27:43 - 27:58)
Right, right. That's good. So let's start with a null, right, and then go from there.

So Cohen gave the example of a d value of 0.2. He said, hey, this is the same as the difference in average height between two groups of girls, 15-year-olds and 16-year-olds.

[Kristin] (27:59 - 28:12)
Okay, that's going to be very subtle, because there's a lot of variability in heights among 15- and 16-year-old girls. So sure, on average, the 15-year-olds are going to be a little bit shorter compared with the 16-year-olds, but it's going to be very hard to see.

[Regina] (28:12 - 28:28)
Yeah, yeah, and he said this would be hard to see, and he called it a small effect size. But of course, I am going to put small in air quotes, because Cohen himself in the book said, hey, don't get caught up on small, medium, large, because it's all relative.

[Kristin] (28:28 - 28:54)
Right. This is one of my pet peeves, actually, Regina. He gave some numbers for small, medium, and large that we'll talk about.

He meant them as these kind of rough rules of thumb, just as a guideline to get you an intuitive feel for it. But yet I see in all these papers that people will write it, well, a small effect size, that was 0.2, as if it's this black and white.

[Regina]
God-given.

[Kristin]
It gets really overused, which can lead to all sorts of problems. Yeah, don't get me started.

[Regina] (28:55 - 29:01)
I know. You know, I think the human brain likes to categorize things and put it into these rigid buckets.

[Kristin] (29:02 - 29:06)
Statistical categorization. Right. That's a topic for another podcast.

[Regina] (29:06 - 29:25)
Another? Let's put that on the list. Another thing that drives me crazy.

We're going to have some good ranting on that one. Okay, so that was 0.2, so let's get a little bigger. Cohen talked about 0.5, and he said, hey, that's the difference in average height between, again, two groups of girls, 14-year-old girls and 18-year-old girls.

[Kristin] (29:26 - 29:31)
So I might be able to notice that, right? 14-year-olds and 18-year-old girls, they look pretty different.

[Regina] (29:31 - 29:39)
Yep, exactly. 0.5 is when we start to be able to notice it a little bit more, you know, to the naked eye, and Cohen called that a medium effect.

[Kristin] (29:39 - 29:40)
Again, in quotes.

[Regina] (29:40 - 29:49)
In quotes again, and then going out even more. A Cohen's d of 0.8, he said, would be the same as the difference between 13-year-old girls and 18-year-old girls.

[Kristin] (29:50 - 29:57)
Which would be much more striking because 13-year-old girls are still going through their growth spurt, so they're going to be a lot shorter on average than 18-year-olds.

[Regina] (29:57 - 30:02)
Yeah, and he called effects 0.8 and bigger large effects.

[Kristin] (30:02 - 30:16)
Again, in quotes. Uh-huh, rough guage. But bringing it back to the study, they found that the difference between the red and blue groups, those were greater than a Cohen's d of one.

What would that translate to for our listeners in terms of heights, do you think?

[Regina] (30:17 - 30:22)
Right, so we have 1.1, 1.4. I would say maybe, what, like 11-year-olds and 18-year-olds, right?

[Kristin] (30:22 - 30:37)
Ooh, that's big. That's a big jump, yeah. Yeah, you would notice that.

So big, Regina, in fact, that I must admit I'm a little skeptical. I mean, is wearing red really going to change how men see you by that much?

[Regina] (30:37 - 30:37)
I know.

[Kristin] (30:37 - 30:50)
Like the difference between 11- and 18-year-olds and height? That seems like if it was that large. We would know it already.

We would all be running around wearing red. Head to toe. Or avoiding red, should we not want male attention, right?

[Regina] (30:52 - 31:05)
Depending on the context and what we want to do, yeah, yeah. You should be skeptical, Kristin, because I have been withholding information from you. Yeah, for example, I did not tell you the sample size of this study, did I?

[Kristin] (31:05 - 31:08)
I just noticed that. You did not. That was sneaky.

[Regina] (31:08 - 31:13)
I know. Okay, so can you guess the sample size? What would be a reasonable sample size here?

[Kristin] (31:13 - 31:20)
So I do see where you're going with this. I am guessing that this was a small sample size, maybe 50 men.

[Regina] (31:21 - 31:31)
23.

[Kristin]
23 per group?

[Regina]
23 total.

[Kristin]
23 total? What?

[Regina]
12 saw the woman in red.

11 saw the woman in blue.

[Kristin] (31:32 - 31:46)
Oh, well, this is just lazy. I mean, give the psych students a harder test so that more of them need extra credit. I mean, you've got 300 people in your class and you can only get 23 men?

Give me a break. Bribes are cheap. Give more extra credit, something.

Yeah, this is not, yeah.

[Regina] (31:46 - 32:07)
Yeah, they were lazy. And, of course, the problem with small samples is that it's very easy to get an inflated effect size. Like there might be some difference in reality, but the results in the sample will show, you know, a difference or an improvement or an effect that's bigger than that reality.

So inflated, an inflated effect size.

[Kristin] (32:08 - 32:20)
Regina, when you talk about inflated effect sizes, I'm picturing those giant helium balloons at the Macy's Day Thanksgiving Parade, Mickey Mouse, Snoopy, Kermit the Frog, inflated beyond their normal size, not reality.

[Regina] (32:20 - 32:28)
I love that. Big balloons floating around on parade getting attention. Yes, lots of attention.

Getting attention. That's kind of like these small sample studies, right? Right.

[Kristin] (32:28 - 33:13)
And the problem with these small samples is, say your study only involves 23 people. It's very sensitive then to random fluctuation. Let's say we just happen in our red group to get two men who love to spend money.

And they're going to spend $100 no matter what the woman looks like. And let's say in the blue group that day we just happen to get two men who are total cheapskates and no matter how beautiful the woman appears to them, they're going to spend $10, right? Just by random chance.

Just by random chance. So it has nothing to do with the red and the blue. But those are extreme values.

And when you average four extreme values in with only 19 other data points, it's totally going to skew the results. It's going to exaggerate the impact of red.

[Regina] (33:13 - 33:28)
Exactly. But bigger samples, of course, are less sensitive to these kinds of flukes. So if you average in a couple of cheapskates and a couple of, you know, hungry foodies, whatever, with 100 people, then they get washed out in the average.

[Kristin] (33:28 - 33:31)
It gets totally washed out. That's why we love big sample sizes.

[Regina] (33:32 - 33:55)
I like big samples and I cannot lie. That was a bad one, wasn't it? Here's something else that people don't always know.

For the same reason, right, with small samples, it's really easy to get effects that are much smaller than they are in reality or they go the other way, right? So the opposite of inflated, which is what? Deflated, shrunken, shriveled?

[Kristin] (33:56 - 33:58)
Kermit the Frog, the day after Thanksgiving.

[Regina] (33:58 - 34:01)
Oh, sad day after balloons. Right.

[Kristin] (34:01 - 34:08)
But this happens, of course, because with small samples, it's random and the effects can fluctuate wildly in either direction.

[Regina] (34:08 - 34:11)
Either direction. Like maybe the cheapskates are all in the red group.

[Kristin] (34:11 - 34:12)
Exactly.

[Regina] (34:12 - 34:26)
But here's the problem that you and I know, is that we see these shrunken, shriveled effects less often because it's only these big, exaggerated effects that get our attention and get published and get talked about.

[Kristin] (34:27 - 34:51)
Right. Publication bias, those inflated effects that support the hypothesis that we want to see, those are the ones that we rush to publish, that get a lot of attention. But let's say it goes in the other direction in a given study.

Oh, blue worked better. Must have done something wrong in that study. We're going to shove that one in the file drawer.

It doesn't get published. So we tend to only see those inflated effects that support the hypothesis.

[Regina] (34:51 - 35:22)
That are exciting, right. This is actually what's behind what's sometimes called the winner's curse, right, in scientific research. Interesting phenomenon.

People started writing about this maybe 15, 20 years ago. I remember reading this article in The New Yorker, and they were describing all these different lines of research where the initial study would be this big, exciting fact, and it was surprising, and then investigators would try to replicate that. And the effect sizes would be smaller, and they had no idea.

[Kristin] (35:22 - 35:22)
They would shrink.

[Regina] (35:22 - 35:30)
They were shrinking, and they had no idea what was going on. And the title of the piece was something like The Truth Wears Off.

[Kristin] (35:30 - 35:46)
Okay, The Truth Wears Off. I don't know if I would call it that. Isn't it more like The Lie Wears Off, or The Falsehood, or The Untruth Wears Off, and The Truth Comes Out?

Because they were getting closer and closer to the actual effect size.

[Regina] (35:46 - 35:54)
The actual truth, but I don't know if that's quite as catchy for a headline in The New Yorker.

[Kristin]
Not as catchy, but it's more accurate.

[Regina]
Yeah, it is.

[Kristin] (35:54 - 36:50)
But winner's curse, I actually remember this very well from a seminar at Stanford that I attended a long time ago that the speaker was talking about the winner's curse. And I have to say, I don't always pay that close attention at seminars I've sat through at Stanford. There's many that I don't remember.

[Regina]
I won't tell anyone.

[Kristin]
Right, sometimes they're boring. But this was actually a really well-done, interesting talk.

So in 1981, there was this small case-control study that came out in the New England Journal of Medicine. About a few hundred people with pancreatic cancer. That's pretty small for that study design.

But it made a big splash because it reported that heavy coffee drinkers had almost a three-fold risk increase in pancreatic cancer compared with non-coffee drinkers. Wow, three-fold. That's big.

And this is one of those hidden cancers, hard to detect, tends to be fatal, and we love our coffee. So this was kind of shocking and worrisome.

[Regina] (36:50 - 36:56)
It got a lot of press. Oh, I bet. So you said heavy coffee drinking?

How many cups is that?

[Kristin] (36:56 - 36:57)
Three cups per day was considered heavy.

[Regina] (36:58 - 36:58)
Wow.

[Kristin] (36:59 - 36:59)
Are you worried now?

[Regina] (36:59 - 37:00)
I am.

[Kristin] (37:00 - 37:20)
Of course, other researchers then went and followed up on this because it was so shocking. Lots and lots of studies were done on coffee and pancreatic cancer. And guess what?

Guess what? These studies didn't replicate the effect. And over time, they realized that it's not there.

There's no evidence that coffee causes pancreatic cancer.

[Regina] (37:20 - 37:29)
So maybe there were other studies on pancreatic cancer in coffee that were done before this New England Journal article. But we didn't see them.

[Kristin] (37:29 - 37:33)
I'm sure other people looked at it before. They didn't find anything, so they didn't publish it. But then this one inflated effect size.

[Regina] (37:33 - 37:44)
Just by chance pops up. That got published, got all the splash and the attention. Yeah, exactly.

[Kristin]
Yeah, surprising results get a lot of attention.

[Regina]
They do. We should do a podcast about coffee.

[Kristin] (37:44 - 37:55)
Well, not coffee and pancreatic cancer, but coffee has been linked to a lot of other beauty and health-related outcomes, so we will do a podcast on it sometime.

[Regina]
What about sex?

[Kristin]
That one I'm not sure.

[Regina] (37:56 - 38:28)
Coffee and sex.

[Kristin]
I'll look that up before the podcast.

[Regina]
All right, so back to this 2008 study that we've been talking about.

Red and attraction made a big splash like this, big effect, and now other researchers wanted to follow up and get some of that fame for themselves. And what do you think they found when they tried to replicate?

[Kristin]
I'm guessing they didn't replicate.

Winner's curse.

[Regina]
Yeah. Some didn't replicate.

Some found no effect at all. But some found a much smaller effect, right? Shrunken, but not�they didn't disappear completely, right?

[Kristin] (38:28 - 38:29)
But smaller effects, yeah.

[Regina] (38:30 - 38:35)
Sometimes they actually found an effect in the opposite direction. Oh, like where the blue did better?

[Kristin] (38:35 - 38:40)
That makes sense, because when things fluctuate wildly, then they fluctuate all over the place.

[Regina] (38:40 - 38:49)
All over the place. So, this is around the time of the so-called research reproducibility or replication crisis in science.

[Kristin] (38:49 - 38:50)
Well, that's still going on, actually.

[Regina] (38:50 - 39:04)
Uh-huh. Good point. This was not long after John Ioannidis' essay from 2005 that you and I love in PLOS Medicine titled Why Most Published Research Findings Are False.

That is a great title for an essay.

[Kristin] (39:04 - 39:08)
It's a great title. I distinctly remember. I remember like where I was when that essay came out.

[Regina] (39:08 - 39:09)
Yeah, yeah. Tell me.

[Kristin] (39:09 - 39:16)
Because he was saying the thing that everyone was thinking, but no one wanted to say. The emperor has no clothes.

[Regina] (39:17 - 39:35)
Yeah, yeah, yeah. So, in this paper, he argued that more than 50 percent of findings published in the scientific literature are false. Kind of like that pancreatic, you know, cancer in coffee study that you were talking about.

The people are publishing conclusions that are simply not true. They're spurious findings. This was a huge deal.

[Kristin] (39:36 - 40:03)
But that more than 50 percent actually felt right to me. That felt, yeah, like honest to me.

And I actually credit that paper with keeping me in academia.

[Regina]
Oh, how so? Really?

[Kristin]
I was debating getting out of academia because I felt like I was in this world where we were all pretending that everything was more rigorous than it actually is, and I was able to see how the sausage was made, right, as a grad student.

[Regina] (40:03 - 40:03)
You did.

[Kristin] (40:03 - 40:26)
And I was like, okay, it's not, yeah, it's not that rigorous. The fact that he was able to say this out loud, that actually a lot of this stuff is garbage, made me feel, ironically, like I could stay in academia because then I knew it was okay to talk about. And actually, I've spent a lot of my career talking about just that and pointing out the garbage.

[Regina] (40:26 - 40:44)
You have, and you've done this amazing job of calling out all this bad stuff, talking truth to power, right? And then holding up the good research and saying this is what it should be like. But at the time, this was a bit of a bombshell.

People did not universally love this.

[Kristin] (40:44 - 40:55)
Not everyone liked this paper. Yes, I mean, it was calling out, questioning people. I think there was also a little professional jealousy over the attention he got over this.

[Regina] (40:55 - 41:02)
I think so. He is not afraid to wade into controversy.

[Kristin] (41:02 - 41:10)
He is not afraid to be controversial, no.

He also, I should mention, published another influential paper that was on Winter's Curse. So he has a really interesting paper on that phenomenon as well.

[Regina] (41:10 - 41:13)
He's always right in the heart of things.

[Kristin] (41:13 - 41:14)
In the thick of things.

[Regina] (41:14 - 41:28)
Yes, not afraid to mix it up. He was calling out in this 2005 essay, bad research practices, which is interesting. He was saying that the problems that we're seeing are not just due to fraud, which people were saying at the time.

[Kristin] (41:28 - 41:36)
I think most of this is probably not due to fraud. He wasn't really talking about people making things up. It's just more of our inherent human biases.

[Regina] (41:36 - 41:55)
Exactly. Like sometimes people really want something to be true, and often they have enough flexibility in their study, right, in the design or the data analysis or what have you, that they can manage to get the results. Consciously or unconsciously, they can take advantage of that to get the results they want.

[Kristin] (41:55 - 42:22)
I don't think this is always intentional. I mean, I work with some great researchers, and sometimes they are absolutely convinced that their intervention works, even when it doesn't, right? They're not deliberately doing anything wrong.

They're just optimists, right? It's human nature. I serve as the independent statistician on these projects, which means I am also the Grinch, because I'm the one who has to come back and tell them, you know, no, the data don't actually show that.

[Regina] (42:23 - 42:41)
Yeah, yeah, you are the Grinch. But, you know, I blame the incentive system here because researchers, they need to get positive results. Because they need to publish, and especially if a group's research portfolio is around something, they've got all this professional identity and pride around a particular effect or drug.

[Kristin] (42:41 - 42:47)
Right, there's an incentive to find it if you think, you know, if you found it before, to keep finding it, right?

[Regina] (42:47 - 42:51)
Right, to keep finding it. This is why we really need more independent statisticians out there.

[Kristin] (42:51 - 42:52)
We need more statisticians, yes.

[Regina] (42:52 - 42:58)
We just need more statisticians in general because they already have the gas pedal. We are the brake.

[Kristin] (42:58 - 43:00)
They need the brake. Exactly.

[Regina] (43:00 - 43:01)
We're the party poopers. The one saying, hey, not so fast.

[Kristin] (43:02 - 43:20)
Don't call me a party pooper. Well, okay, but, you know, even though I am the Grinch, I have been working with the same groups for a very long time.

We have a good working relationship because they trust me at the end of the day that we are going to get the right answer in the long run.

[Regina] (43:20 - 43:21)
In the long run.

[Kristin] (43:21 - 43:25)
And they appreciate that, yeah. I feel like it's the marshmallow experiment, right?

[Regina] (43:25 - 43:26)
Oh, the marshmallow test. Delayed gratification.

[Kristin] (43:26 - 43:35)
So you can have the spurious finding and eat your marshmallow now. Or you can hold out, be rigorous.

[Regina] (43:35 - 43:37)
And get more marshmallows in the future.

[Kristin] (43:38 - 43:39)
And get more marshmallows in the long run.

[Regina] (43:39 - 43:48)
I love that analogy. That's what it is because science ultimately is self-correcting in the long run. But in the short term, it gets a little messy, right?

[Kristin] (43:48 - 44:03)
Yeah, yeah, exactly. So, Regina, though, how does the replication crisis fit in with this research on the red dress effect? So you mentioned there were a lot of follow-up studies.

Those were kind of all over the map. We're meeting some shrinking of effect sizes. Do we have a consensus yet?

[Regina] (44:03 - 44:48)
Okay, well, fast forward 10 years from the original study. So now we're 2018. And by this time, there have been a lot of studies, right?

People are excited. Nice, easy thing. The headlines, love it.

But there was one group of researchers in particular, Dominican University in Illinois, they were skeptics. They were public skeptics. They didn't believe that this red effect was so big or maybe not even a thing at all.

They wanted to look at the whole body of evidence that had been published, everything, right, from a skeptical point of view. So this is the interesting part. They invited their rivals, that original group from University of Rochester that we talked about, to come work with them in what's called an adversarial collaboration.

[Kristin] (44:50 - 44:58)
Okay, so we've got this original team from the University of Rochester and we have this group of skeptics from Dominican University and they're saying, hey, let's work together.

[Regina] (44:58 - 44:59)
Come on in. Let's have a big party.

[Kristin] (44:59 - 45:07)
Right. This is great, this adversarial collaboration, because you have to be open-minded and it challenges your thinking.

[Regina] (45:08 - 45:39)
Oh, it does. So this is in favor of Daniel Kahneman. The Nobel Prize-winning economist, psychologist, died recently.

He continued to be an active researcher across the years. And he talked about adversarial collaborations. He published on it.

He did it himself. I interviewed him about it when I wrote about it. He said, team up with your rivals.

The idea is that you team up with these rivals. You collaborate on the data collection, the study design, the data analysis. You agree to disagree as little as possible.

That was how he put it.

[Kristin] (45:40 - 45:41)
Keeps everyone honest.

[Regina] (45:41 - 45:41)
It does.

[Kristin] (45:41 - 45:47)
Because you know your rivals are looking over your shoulder. Regina, I think this is also called red teaming.
[Regina]
Oh, I do not.

[Kristin] (45:47 - 45:48)
Have you heard that term before?

[Regina] (45:48 - 45:54)
No, I haven't.

[Kristin]
It comes from the military. They would, you know, try to think like the enemy essentially.

[Regina] (45:54 - 45:55)
Oh, interesting.

[Kristin] (45:55 - 46:07)
And the red comes from the Soviet Union era. So red team was the Soviet Union. Blue team was the U.S.

[Regina]
It didn't have anything to do with sexy.

[Kristin]
Right, but doesn't that fit in with the theme of our podcast today?

[Regina] (46:07 - 46:21)
It does. It does. Sexy red teaming, I think, is what it is.

But you're right. So it keeps everyone honest because your rivals are looking. So you write the paper together, but then you have separate interpretations at the end for both groups.

[Kristin] (46:21 - 46:22)
Separate discussion sections.

[Regina] (46:23 - 46:24)
Yep.

[Kristin] (46:24 - 46:25)
Must make for some fun reading.

[Regina] (46:26 - 46:26)
Yeah.

[Kristin] (46:26 - 46:27)
A little controversy.

[Regina] (46:27 - 46:40)
Yeah. Kahneman said, well, he pointed out that the way it's typically done now is that someone comes out with a study and then you have the skeptics. Another team will disagree.

They write a letter to the editor. Right. And then you have rejoinder.

[Kristin] (46:40 - 46:53)
Right. We know how that goes. So you write a letter to the editor because you're skeptical about an article.

The authors always get the last word. They get to write a response and they're like, oh, no, no, it's all fine. We didn't do anything wrong.

And then it ends there and nothing comes of it.

[Regina] (46:54 - 46:57)
And nothing comes of it. And, of course, the media generally aren't seeing this.

[Kristin] (46:57 - 47:00)
They're not reading the letters to the editor section of scientific journals.

[Regina] (47:00 - 47:10)
They're not reading the letters to the editor, right. So it doesn't really go anywhere from that. So Kahneman said that he actually thought that this adversarial discussion, you know, sort of thing made for better reading.

[Kristin] (47:10 - 47:16)
Yeah. I think it does, yeah.

All right. So this original group from the University of Rochester, they actually agreed to do the adversarial collaboration with the skeptics?

[Regina] (47:17 - 47:21)
They did. So brave. Right.

And not for the faint of heart doing the adversarial collaboration.

[Kristin] (47:21 - 47:28)
I mean, essentially, they were willing to put their whole line of work out, air their dirty laundry, to be scrutinized. Wow.

[Regina] (47:28 - 47:39)
To be scrutinized. Especially because this group, the original authors, they were very involved in studying the psychological effects of color, and especially red.

This was very personal.

[Kristin] (47:39 - 47:39)
Yeah.

[Regina] (47:39 - 47:42)
This was brave. I admire that so much.

[Kristin] (47:42 - 48:02)
Well, this makes them so much more credible in my mind because it tells me that they are trying to do good science. They're not just looking for sexy headlines, right? They actually wanted to find the truth and do good science.

Regina, I am really curious to hear what happened in this adversarial collaboration, but let's take a short break first.

[Regina] (48:08 - 48:20)
Kristin, we've talked about your medical statistics program. It's just a fabulous program available on Stanford Online. Maybe you can tell listeners a little bit more about it.

It's a three-course sequence.

[Kristin] (48:20 - 48:33)
If you really want that deeper dive into statistics, I teach data analysis in R or SAS, probability, and statistical tests, including regression. You can get a Stanford professional certificate as well as CME credit.

[Regina] (48:33 - 48:37)
You can find a link to this program on our website, NormalCurves.com.

[Kristin] (48:45 - 49:01)
Welcome back to Normal Curves. We're discussing the results of a 2018 adversarial collaboration to synthesize the evidence on the claim that wearing red makes women more attractive to straight men. So, what did this collaboration involve, Regina?

[Regina] (49:01 - 49:12)
Right. So, University of Rochester, Dominican University, and what they did here with a meta-analysis, which is a great summary of evidence. Kristin, how about if you explain what a meta-analysis is?

[Kristin] (49:12 - 49:34)
This is where researchers take data from studies that are already completed, either published or unpublished. The study's already done, and they're just gathering some of the data, the summary data, for a specific claim, something like red makes women more attractive to men, and then they're pooling the data across all these studies to come up with a more reliable estimate of the effect.

[Regina] (49:35 - 49:50)
Great summary. I love that. So, this meta-analysis was published in a journal called Evolutionary Psychology.

They looked at 45 studies for a pooled sample of almost 3,000 men.

[Kristin] (49:50 - 49:51)
That's a lot bigger than the 23 we started out with. Great. What did they find?

[Regina] (49:52 - 50:05)
Actually, I'm going to draw out the suspense a little bit, because first I want to talk about the studies themselves, because the studies are interesting. And this meta-analysis included 17 unpublished studies, which is pretty unusual to have that much.

[Kristin] (50:05 - 50:15)
Wow, yes. It's pretty unusual to get that many unpublished studies in a meta-analysis. But it's important because it goes to what we were talking about earlier, this publication bias.

So, we don't want to just look at the published research.

[Regina] (50:16 - 50:22)
Right, right, because of all the studies where they didn't find anything interesting, just got tucked away, like you said, in a file drawer, that's going to bias the results.

[Kristin] (50:23 - 50:26)
But how did they find the unpublished studies? They're not published, so where do you go to find them?

[Regina] (50:27 - 50:35)
Yeah, they worked really hard. They advertised on Listserv, and they also just kind of asked other colleagues and looked in their own files. They were diligent.

[Kristin] (50:36 - 50:50)
Good, good. That is very diligent. And listeners might be wondering, like, why would you want to include unpublished studies?

Because they're not peer-reviewed. But again, it's to help counter this worry about publication bias. It's important to include them.

[Regina] (50:50 - 50:58)
And in fact, they found that the unpublished studies were much less likely to have significant findings. So, it does illustrate that publication bias.

[Kristin] (50:58 - 50:58)
Exactly what you'd expect, yeah.

[Regina] (50:59 - 51:04)
Yeah. Another interesting thing that they did with this meta-analysis is include studies that had been preregistered.

[Kristin] (51:05 - 51:07)
Oh, preregistration. I love this.

[Regina] (51:07 - 51:12)
Yeah, me too. Let's talk a little bit about preregistration, why we are such fangirls at this.

[Kristin] (51:13 - 51:31)
This is where you publish your protocol publicly ahead of time before you start the study. And everybody can see it, so it locks you in. It helps you to resist the temptation of doing any shenanigans, shall we say, later on with your data.

It helps you to not cherry-pick later.

[Regina] (51:31 - 51:33)
Cherry-pick. It also lets you prove that you didn't cherry-pick.

[Kristin] (51:34 - 51:34)
Yes.

[Regina] (51:34 - 51:38)
Because you set out to do something, and then you can show that you actually did it.

[Kristin] (51:38 - 51:53)
You can point people there and say, yeah, that's what I said I was going to do. Like insurance. Exactly.

The other thing I love is it reminds you about all the little details that you need to do in your protocol because you know people are going to be looking at it. So, you might remember to put in things like a sample size calculation.

[Regina] (51:54 - 52:16)
People often forget this when they do these studies. Right, right. So, if it's preregistered, then it's out there.

So, someone who's doing a meta-analysis like here, they can find that preregistered protocol, and even if the study itself, the results don't get published, then meta-analysis investigators can go back to those original authors and ask for the data.

[Kristin] (52:16 - 52:17)
Right, they know to look for the data.

[Regina] (52:18 - 52:41)
Right, right, right. So, it's not as hidden that way. Absolutely.

So, for this meta-analysis, another thing that they found was that nearly every study they included was really small. So, the results just fluctuated a lot, study to study, and more recent studies tended to have smaller effect. Oh, winner's curse at work.

Winner's curse. So, are you ready now for�

[Kristin] (52:41 - 52:42)
Oh, you're going to let us know the results now?

[Regina] (52:43 - 53:06)
Now there's enough suspense in here. All right. Finally, the overall pooled effect was Cohen's d of 0.26. So, it had a p-value of 0.0004. Had to make sure I got all those zeros in there. So, this is where the Cohen's d really comes in handy because we're able to combine then all of these effects across all these different studies.

[Kristin] (53:07 - 53:37)
We have all these different studies, and everybody might be measuring different things, a little bit different takes on attraction and different scales. So, how do we make them all into one? Well, we can put them all in Cohen's d.

That's a great use of Cohen's d in meta-analysis. Regina, I feel like everybody's going to get caught up in the p of 0.0004. Everybody gets really excited about teeny-tiny significant p-values. But it's really important to focus not just on the p-value, but that effect size, that Cohen's d of 0.26. How big is it?

[Regina] (53:37 - 53:51)
Right, yes. What does this mean in real life? And it's not super big.

[Kristin]
No, it is not.

[Regina]
So, going back to Cohen's example with the height, Cohen's d of 0.2, remember that was equivalent to the difference in height between 15-year-old and 16-year-old girls.

[Kristin] (53:52 - 53:56)
So, 0.26 might be like 14-and-a-half-year-old girls versus 16-year-olds. Definitely not big.

[Regina] (53:57 - 54:05)
Not big at all. And remember in that original study that we were talking about, the Cohen's d's were over 1, 1.1, 1.4. Right.

[Kristin] (54:05 - 54:06)
So, this is winner's curse in action.

[Regina] (54:07 - 54:07)
It is.

[Kristin] (54:07 - 54:23)
Because they started with these really big effect sizes, 1.1, 1.4, then there was a whole bunch of studies, and the effect did not go to zero. It did not work. Like the coffee and pancreatic cancer study, but it went down to 0.26, much smaller than those original effect sizes from the 2008 paper.

[Regina] (54:24 - 54:43)
Much smaller. I feel like it would be really hard to see this effect in everyday life, right? With an effect size of, you know, 1, 1.1, whatever, you'd be able to see it as soon as I put on a red sweater, but it's 0.26. You're getting like half an appetizer extra for dinner? It might be worth it, but pretty subtle in there.

[Kristin] (54:43 - 54:44)
Half of a plate of french fries? Yeah.

[Regina] (54:44 - 54:56)
Here's another catch. You're going to like this one. They separated out the preregistered studies and just looked at those.

The effect size was 0.1.

[Kristin]
Ooh.

[Regina] (54:56 - 55:10)
Ooh, I know.

[Kristin]
One-tenth of a standard deviation. That is really small.

I mean, it's almost so small that it's probably too small to care about. That's like three french fries extra? Not practically significant, is what we would say.

[Regina] (55:10 - 55:16)
If it's french fries, I still might go for it, but right. I mean, this, we don't even care about it. I thought it was interesting how they separated that out.

[Kristin] (55:16 - 55:23)
Yeah, and you might think those preregistered studies were more trustworthy, so you might be prone to believe the 0.1 a little more than the 0.26.

[Regina] (55:23 - 55:40)
Right, right. I love how they did that. It was interesting how the researchers discussed these results in the paper because, remember, we had skeptics and advocates, and they had separate discussion sessions, right? So they had different interpretations of the data and what this means.

[Kristin] (55:40 - 55:42)
Ooh, so I want to hear the different takes on this.

[Regina] (55:42 - 55:58)
Mm-hmm. So let's start with a skeptical author at Dominican University. They said, quote, The simplest conclusion from our results is that the true effect of incidental red on attraction is very small, potentially nonexistent.

[Kristin] (55:58 - 56:09)
All right, well, that's actually quite fair because it started at 0.26, but if you look at the preregistered studies, it's much, much smaller. That seems like a pretty fair interpretation of the results.

[Regina] (56:09 - 56:24)
I thought so, too. But let's hear from the advocate, those original authors from University of Rochester. They said, quote, For men rating women, there is clear evidence of a small, statistically reliable effect.

[Kristin] (56:25 - 56:30)
Oh, that's actually pretty fair, too, right? Mm-hmm. We are seeing an effect.

It doesn't disappear. It doesn't go to zero.

[Regina] (56:30 - 56:48)
It's reliable but small.

[Kristin]
Mm-hmm. Wow, I can actually see both sides of this one. I'm torn.

[Regina]
Yeah, I can see both sides of that. They went on to say, It's possible that there's a reliably strong effect under some conditions and a weak or nonexistent effect under others.

[Kristin] (56:49 - 57:05)
Oh, interesting. So they're saying the fact that these effect sizes keep fluctuating around, maybe it's not just due to the fact that we have small samples, random chance, but maybe red works in some situations and not in others, so depending on the study subject, the shade of red.

[Regina] (57:06 - 57:09)
The experimental design. Yeah, exactly, yes. Who the woman was.

[Kristin] (57:09 - 57:10)
Right, the women.

[Regina] (57:10 - 57:19)
All of that. So maybe it's not a question of is there an effect of red, but for whom and when.

[Kristin] (57:20 - 57:24)
Okay, well, that sounds like reasonable speculation, right? We call these moderators.

[Regina] (57:24 - 57:39)
Moderators, to use a little jargon in there. And then it gets even better. Okay, so these advocate authors, they admitted that not a single study in this meta-analysis would be considered exemplary because they all had small sample sizes.

[Kristin] (57:40 - 57:53)
Interesting because a lot of the studies that they included in the meta-analysis were their own. Their own. They are admitting that they did less than exemplary work.

I love the honesty. I love the self-awareness.

[Regina] (57:54 - 58:10)
I admire them so much for this, I have to say. It takes a lot of integrity to come forward and say this. They also recommended that to do better, future studies should carefully attend a sample size and publicly pre-register their protocol.

[Kristin] (58:10 - 58:19)
So not only did they admit they got things wrong, but they're saying, and here's what we should have done, and even better, here's what we're going to do in the future.

[Regina] (58:20 - 58:35)
I know. Love it. I know.

Okay, so now fast forward another five years. We're now in 2023, 15 years since the original study. And researchers have been going deeper, and they've been testing Red in very specific situations, very specific conditions, context.

[Kristin] (58:35 - 58:46)
Okay, thinking that maybe the Red works in some situations but not others, right? They speculated that in the meta-analysis that maybe, again, this is what we call moderators. Right, right, right.

So they're testing these moderators.

[Regina] (58:46 - 58:53)
Right, right, and this makes a lot of sense. So here's a summary of what has been found to date. Prepare yourself.

It's going to be a bit grim, Kristin.

[Kristin] (58:54 - 58:54)
Uh-oh, really?

[Regina] (58:55 - 59:14)
Why? Well, I personally found it to be a bit grim. All right.

One, it turns out that Red does not seem to increase attractiveness for middle-aged or older women, postmenopausal women, only young women.

[Kristin]
Oh, that's not fair. That's completely not fair.

[Regina]
I need this too, not just the 20-year-olds.

[Kristin] (59:14 - 59:16)
Yeah, arguably we need it more.

[Regina] (59:16 - 59:29)
I think we need it more. All right, second, Red works for women with feminine but not masculine facial characteristics. And only for women that are perceived as being at least moderately attractive.

[Kristin] (59:29 - 59:31)
Well, that's definitely not fair.

[Regina] (59:31 - 59:52)
They don't need any help. They don't need any help. Here's how one group summarized it.

It is not the case that Red uniformly increases men's attraction to women. Rather, Red only increases attraction in contexts that might facilitate sexual interactions. Contexts that might facilitate sexual interactions.

[Kristin] (59:52 - 59:58)
That is very academic language. Yeah, this jargon. I think that just means where sex is on the table.

[Regina] (59:59 - 1:00:41)
And on the countertop. The bearskin rug in front of the fireplace.

[Kristin]
Okay, maybe we should stick to the stilted academic language.

[Regina]
All right. So now 2023, these original investigators from the University of Rochester, same lead author, common thread throughout that. They have a new study.

Kristin, here is the thing. You ready for what I think is the most amazing part of this entire story? These investigators really changed how they did research.

[Kristin]
How so?

[Regina]
They changed, I know. First, they preregistered their study.

I knew you would like that one. They registered at a site called aspredicted.org.

[Kristin] (1:00:42 - 1:00:52)
Okay, well, I'm familiar with clinicaltrials.gov. A lot of the clinical trials in medicine get registered there, but I assume this is another one for other site domains other than medicine.

[Regina] (1:00:53 - 1:01:13)
Non-medical, right. That's why I mentioned it. I thought it was interesting.

It's from the University of Pennsylvania Wharton Credibility Lab. It's interesting. So, same thing.

Thousands of authors are publishing their protocols there and pledging that they will follow them, including our researchers. And here they explain how they did their sample size calculations.

[Kristin] (1:01:13 - 1:01:18)
Oh, sample size calculation. That warms my heart. In fact, they realized they needed more than 23.

[Regina] (1:01:19 - 1:01:40)
They did. So, they reported on three experiments in this. They did sample size calculations for each one separately, and they even exceeded their target sample size each time.

They got 116 men, 238 men, and 236 men for each of them. Big. Much bigger than 23.

[Kristin] (1:01:42 - 1:01:56)
Much bigger. I love this. I think this is amazing because not only were they willing to take a hard look at their own work, be open to criticism from others, which I have to say a lot of scientists are not.

[Regina]
Are not.

[Kristin] (1:01:57 - 1:02:06)
Oh, no, no. I didn't do anything wrong.

And not only that, but they said, okay, we are not only going to be open to criticism, but then we are going to change our behavior.

[Regina] (1:02:06 - 1:02:13)
This is how science is supposed to work. Science really is self-correcting this way. It evolves.

Methods evolve.

[Kristin] (1:02:13 - 1:02:31)
Right. A lot of people had poorer methods 15 years ago. There's a lot more awareness now about good methods, so people need to change their habits. I love the fact that they actually did change.

They admitted they made a mistake and they fixed it, and this makes them completely more credible in my mind.

[Regina] (1:02:31 - 1:02:42)
I know. It's usually not people established in their careers that are changing and evolving this way. Usually, the people established in their careers are a little defensive.

[Kristin] (1:02:42 - 1:02:44)
Oh, I've seen that a lot, yes. People get very defensive.

[Regina] (1:02:45 - 1:02:48)
They do. So, I'm a super fan girl now at this group.

[Kristin] (1:02:48 - 1:02:48)
Me, too.

[Regina] (1:02:48 - 1:02:51)
Yes. They evolved. They changed.

All right. They are credible.

[Kristin] (1:02:51 - 1:02:55)
All right. So, they did this study. They preregistered it.

Good sample size. Tell me more about the actual study.

[Regina] (1:02:55 - 1:03:10)
All right. Three studies. Let's just focus on one again.

Well, the experimental design changed a bit. Okay. Yeah.

They took gorgeous models, underwear models.

[Kristin]
Oh, like from Victoria's Secret ads? You can photoshop them.

[Regina] (1:03:10 - 1:03:20)
Yeah. Like I said, a popular women's clothing company. So, they took photos of them wearing underwear.

Wearing lingerie. Photoshopped their lingerie to be either red or green.

[Kristin]
Green?

[Regina] (1:03:20 - 1:03:30)
Green.

Green underwear. Oh. And now it's like St. Patrick's Day underwear. Not totally sexy. There were good color psychology reasons for this because red and green are opposite on the color wheel.

[Kristin] (1:03:30 - 1:03:31)
Oh, it was a good control.

[Regina] (1:03:32 - 1:03:32)
Right.

[Kristin] (1:03:32 - 1:03:32)
Right.

[Regina] (1:03:32 - 1:03:33)
Okay. Yeah.

[Kristin] (1:03:33 - 1:03:37)
But why underwear models? Why did you switch to underwear models at this point?

[Regina] (1:03:37 - 1:03:53)
Well, because as we talked about since 2008, they learned about all these moderators and they said, all right, let's get in there and focus where we think there is a real effect, right? Which is young, premenopausal, highly attractive women with feminine features.

[Kristin] (1:03:53 - 1:03:53)
That makes sense. Yeah.

[Regina] (1:03:53 - 1:03:55)
It does make sense. It's just a little sad.

[Kristin] (1:03:56 - 1:03:56)
So, what did they find?

[Regina] (1:03:58 - 1:04:12)
Right. The results. Okay.

The men found that these underwear models in red underwear, they found them to be more physically attractive, more sexually attractive, and they found them to be more sexually receptive than the women in green underwear.

[Kristin] (1:04:13 - 1:04:15)
Oh, they thought that they looked more open to sex.

[Regina] (1:04:15 - 1:04:25)
Is that what you're saying? More open to sex. So, the Cohen's D values were between 0.54 and 0.75. Oh, that's in that more noticeable range.

[Kristin] (1:04:26 - 1:04:39)
Uh-huh. Bigger than 0.26. Yep. Because now it's in contexts that might facilitate sexual interactions.

Young, attractive women in underwear. So, maybe the effect is actually bigger in this situation.

[Regina] (1:04:39 - 1:04:50)
Uh-huh. Uh-huh. Okay.

Right. And they also found that the more a guy thought that a woman was sexually receptive, the more he also found her to be physically and sexually attractive.

[Kristin] (1:04:51 - 1:04:52)
Oh, interesting.

[Regina] (1:04:52 - 1:04:52)
Uh-huh.

[Kristin] (1:04:52 - 1:04:54)
It makes sense that those variables would go together.

[Regina] (1:04:55 - 1:05:10)
Yes. Right. Right.

But, Kristin, once the research is statistically controlled for this sexual receptivity that we're talking about, this whole red underwear, red dress effect vanished. Pretty much vanished. Disappeared.

Gone.

[Kristin] (1:05:10 - 1:05:11)
What do you mean by that?

[Regina] (1:05:11 - 1:05:22)
Uh-huh. Well, according to the results of the study, it seems like the whole reason that women in red look more attractive and more appealing is because it looks like they are open and ready for sex.

[Kristin] (1:05:22 - 1:05:49)
Okay. Let's unpack what we mean by this. Like, what's the statistics under the hood here?

You're telling me that they did some kind of statistical adjustment.

Like a regression model. And once they accounted for how much these women seemed sexually receptive, there was no longer any independent effect of the red color on physical and sexual attractiveness. The entire effect was really driven by the men just thinking that sex was on the table.

That's why they rated the women as more attractive.

[Regina] (1:05:49 - 1:06:12)
I know. I know. It's sad, right?

So, I think what this means for me is I can't just wear a red dress or a red sweater and simply look, you know, more beautiful or more sexually attractive. No. I wear a red dress and it makes me look like I'm in heat.

And that's what makes me suddenly look attractive and appealing and sexy.

[Kristin] (1:06:13 - 1:06:19)
Basically, you're saying this all comes back to the red monkey butts. I mean, almost literally in this study, right?

[Regina] (1:06:20 - 1:06:24)
Neon sign. Red means ready right now at dinner.

[Kristin] (1:06:26 - 1:06:58)
All right, Regina. So, now we're at the point in the podcast where we are going to wrap it up and give the strength of the evidence for this claim. And the claim that we're looking at today is that wearing red makes women more physically and sexually attractive to men.

And how do we rate that claim with our trademark smooch rating scale? One to five smooches. One smooch means little to no evidence for the claim.

And five smooches means very strong evidence for the claim. So, Regina, kiss it or diss it?

[Regina] (1:06:59 - 1:07:00)
I'm going to kiss this one, actually. I'm going to give it three smooches.

[Kristin] (1:07:00 - 1:07:01)
Okay, very nice.

[Regina] (1:07:01 - 1:07:16)
Is it a big effect? Probably not. Is it going to help me drive men wild with lust?

Probably not. But is there something there? I think so.

So, I'm going to give it three. And I really love how they improved their methods.

[Kristin] (1:07:16 - 1:07:20)
Oh, I totally agree with you, Regina. I'm going to go wild on this one and go four smooches.

[Regina] (1:07:21 - 1:07:21)
No.

[Kristin] (1:07:21 - 1:07:43)
Which is going to probably be a rare occurrence on this podcast. I agree with you that I think there is something there.

It's small. It's only in certain cases. But it seems to be reliable.

And I must admit, I'm throwing in a bonus smooch, like a bonus point on the test here, because of how they turned around their methods. I am so impressed by that, that they get an extra smooch.

[Regina] (1:07:44 - 1:07:45)
I love it. You're generous.

[Kristin] (1:07:46 - 1:08:08)
I'm being generous today, yeah. I am encouraged by this Cinderella story, Regina. It renews my faith in research and science.

The other thing we do on this podcast, of course, Regina, is methodologic morals, because we're not just talking about the claim, we're talking about how to evaluate evidence. It's a little like Aesop's fables. Regina, do you have a methodologic moral for us today?

[Regina] (1:08:08 - 1:08:16)
I do. Here it is. The smaller the sample, the flashier the result, the less you should trust it.

[Kristin] (1:08:16 - 1:08:36)
Oh, I love it. That's that winner's curse. Yes. All right.

You want to hear mine?

[Regina]
Yeah, of course.

[Kristin]
Good scientists learn from their statistical mistakes and fix them.

[Regina]
Aw, the Cinderella story.

[Kristin]
The Cinderella story. Yes, this really has been super interesting, Regina.

I love the Cinderella story. It's like a fairy tale. Thanks so much.

[Regina]
Thank you.