There’s an explanation for why you and your best friend can never agree on whether a romantic interest is hot or homely.

According to a new study, it’s our life experiences—not a perfectly chiseled jaw or sultry bedroom eyes—that make a person’s face appealing to us. Sure, symmetrical features are generally more attractive than non-symmetrical ones, but an even face only partially accounts for someone’s overall “attractiveness,” researchers find. After that, our tastes can vary dramatically.

The two-part study, published this week in the journal Current Biology, asked 35,000 people to visit a website, where they were asked to rate faces for attractiveness. The resulting data showed that most individuals agreed only half of the time.

Was this because genetics and environment influence our perceptions of physical beauty? In the study's second half, researchers studied the facial preferences of 547 pairs of identical twins and 214 pairs of same-sex fraternal twins. Sure enough, fraternal twins—who grew up in the same households yet share 50 percent of the same genes—ended up disagreeing on standards of attractiveness. However, the identical twins were also “really, really different from each other in their facial aesthetic preferences,” one of the paper’s lead researchers, Jeremy Wilmer of Wellesley College, told Time.

In short, physical attraction is highly personal—even among relations who've had similar upbringings. Researchers chalked up the differences to our own distinct life experiences, which can vary widely thanks to co-workers, peers, past relationships, and media exposure.

If experience influences attraction, then what types of experiences make us prefer one face to another? The paper didn’t measure this, but researchers think it has to do with positive associations, as study lead author Laura Germine, of Massachusetts General Hospital, told Time. If you’ve had good experiences with people who have certain facial characteristics, you’ll most likely find them attractive. As time passes, others who look like them will seem good-looking to you as well. 

[h/t Time]