This month yielded yet another published study — which received positive media attention — based on the National Longitudinal Lesbian Family Study. The NLLFS is about to enter its third decade of following the same 78 respondents, who were “planned” and born to lesbian mothers employing artificial reproductive technology; in nearly all the families studied, the children were being raised by their biological mother and her partner. While any sociologist worth his or her degree can appreciate the laborious task of keeping track of and reinterviewing the same group of people over many years, this particular data-collection effort probably ought to be retired. And yet it continues to appear in peer-reviewed journal articles in the health and social sciences. What exactly is the NLLFS and why do I say it should be retired?
First, let me say that my skepticism about the NLLFS — shared by plenty of social scientists who wouldn’t go on record with their misgivings in the present chilling scholarly climate — is not at all due to the researchers who oversee the project, the authors who make use of it, or even the politically motivated funders who underwrite it. We all have our perspectives and interests, and the academy is big enough for all of us. (I hope.)
No, my misgivings are due to the great likelihood that the data sources — the respondents themselves — have been increasingly compromised, placing the very validity and reliability of the data in question. How so?
The NLLFS employs a convenience sample, recruited entirely from announcements posted “at lesbian events, in women’s bookstores, and in lesbian newspapers” in Boston, San Francisco, and Washington, D.C. As the late family sociologist Steven Nock warned, the level of sample bias such an approach introduces is significant. The lesbian parents whose children are being studied are whiter (94 percent), more educated (67 percent college graduates), of higher socioeconomic status (82 percent held professional or managerial positions), and more politically motivated than lesbians who do not frequent such “events” or bookstores, or who live in cities like San Antonio or Kansas City, or in smaller towns across the country. (Aren’t they important, too?) Anything that is correlated with educational attainment, for example — better health, more deliberative parenting, greater access to social capital and educational opportunities for children — will be biased in analyses. Any claims about a population (in this case, American lesbian parents) based on a subgroup that does not represent the whole will be distorted, since its sample is far less diverse (given what we know about it) than a representative sample would be. Indeed, there’s nothing “national” about the NLLFS.
I have no objection at all to the collection of snowball-sample data, only to its popular use as a source of information about all children of lesbian parents. If the NLLFS were simply used to understand the world of lesbian parents and their children among the elites in those three cities, then that would be just fine. But it’s not. In this case, the practical result and conventional use of its findings — and that is key — is to generalize to the population of lesbian parents across America. While researchers themselves commonly note this limitation, it is entirely lost in the translation and transmission of findings by the media to the public.
So for nearly 20 years the scholarly community has been treated to studies of this unique small cluster of kids, and the general public has been left with the impression that they represent the whole population of children of American lesbian mothers. They do not. The NLLFS’s sample doesn’t even share the population characteristics of same-sex-couple households that its sponsor — the Williams Institute of UCLA — describes in its own publication. In social reality — the one described in the U.S. Census — such households are less well-off economically, less white, and with less-educated parents than the households in the NLLFS. Everything about the NLLFS suggests a sample of highly privileged children being raised by parents with significant resources living in communities that offer notable social support.
In other words, it’s not the Census (or even the NFSS, the New Family Structures Study, with which I am affiliated) that needs to explain the diversity of lesbian motherhood that really exists in America. They’ve already done so. It’s the NLLFS that ought to explain its utter lack of diversity. It is from this select group of kids, not from the Williams Institute’s population-based estimates, that the media get their information about how the children of lesbian households are faring.
And yet all this is not actually why I think it’s time for the NLLFS to shutter its operation. No, the reason is that its sample — 78 kids growing up in activist households — is no longer a source for valid, reliable information. Why?
Studying child outcomes of gay and lesbian households is a politicized thing, as I discovered this past summer. Hence it is also a very public thing. Is it not thus reasonable to wonder whether the 78 respondents tracked for the past two and a half decades are truly unaware of — and hence unaffected by — the media attention regularly paid to the study in which they still participate.
The “Hawthorne effect” refers to the tendency of study participants to work harder or perform better because they know they are being studied. While it is typically applied to experimental research studies of worker productivity, the same could be true here. It’s a cousin to “social desirability bias,” which is closer to what I’m suggesting. In this case, I’m concerned that the kids feel pressure to give better-than-accurate portrayals of their household and personal life. When the adolescent children of lesbian parents are being intermittently interviewed for a study whose results have proven quite politically important — and almost always covered favorably by the mainstream media — it’s prudent for scholars to be skeptical about whether respondents are still offering valid and reliable responses years after they were first contacted. Some kids will always offer valid information, but given the fishbowl these 78 have lived in, I’m concerned that social desirability bias will affect disproportionate numbers of them, especially in contrast to far larger survey projects.
To be sure, some information — like high-school graduation or household changes — can seem innocuous to respondents. Other information — about things like emotional health, substance use, and reflections on their childhood — may be more problematic. Being one of these 78 kids is a little like being the child of someone elected to public office. It’s a big responsibility, and people — including the media — are watching you. Could it be that these children’s parents have never sought to influence their responses, or reminded them of the ramifications of their answers, or shown them the media attention accorded them, or simply introduced them to the NLLFS website? The risk of compromised data sources here, I assert, is elevated.
Nevertheless, peer-reviewed journals continue to give the NLLFS data nearly carte blanche, so far as I can tell. And so the media narrative about the study’s children that flows from the NLLFS publicity office and the Williams Institute at UCLA remains one of nonstop good news.
And yet my misgivings are not about the good news. I believe the analysts and authors are not making it up. I just don’t believe the 78 kids in the NLLFS are capable of reporting unbiased information any more, not after a childhood and adolescence spent entirely in a fishbowl. Even the NLLFS’s principal investigators suggested that 25 years of data collection may be enough. I would concur, and — since the study commenced in 1986 — we eclipsed that mark in 2011. Perhaps it’s time to commit significant funds — and a panoply of research perspectives — to a very large (and hence expensive), longitudinal, population-based data-collection effort that would make fans of the NLLFS and fans of the NFSS alike content with its methodology.
I’m all for more information. But if the data are to be valid and reliable, the study needs to be as free of source bias as is humanly possible. I won’t hold my breath, though, because in the case of lesbian parenting, a nationally representative sample is not what many of my scholarly, rational, and allegedly dispassionate colleagues in the social sciences appear to want.
— Mark Regnerus is an associate professor of sociology at the University of Texas at Austin and a research associate with the university’s Population Research Center. He is also the principal investigator of the New Family Structures Study.