Social Science Research 52 (2015) 615–626

Contents lists available at ScienceDirect

Social Science Research

journal homepage: www.elsevier.com/locate/ssresearch

Measurement, methods, and divergent patterns: Reassessing

the effects of same-sex parents q
Simon Cheng a,1, Brian Powell b,1
a
344 Mansfield Rd., Department of Sociology, University of Connecticut, Storrs, CT 06269, United States
b
744 Ballantine Hall, 1020 E. Kirkwood Ave., Department of Sociology, Indiana University, Bloomington, IN 47405-7103, United States

a r t i c l e i n f o a b s t r a c t

Article history: Scholars have noted that survey analysis of small subsamples—for example, same-sex par-
Received 8 October 2013 ent families—is sensitive to researchers’ analytical decisions, and even small differences in
Revised 24 March 2015 coding can profoundly shape empirical patterns. As an illustration, we reassess the findings
Accepted 8 April 2015
of a recent article by Regnerus regarding the implications of being raised by gay and lesbian
Available online 23 April 2015
parents. Taking a close look at the New Family Structures Study (NFSS), we demonstrate
the potential for misclassifying a non-negligible number of respondents as having been
Keywords:
raised by parents who had a same-sex romantic relationship. We assess the implications
Children
Family structure
of these possible misclassifications, along with other methodological considerations, by
Methodology reanalyzing the NFSS in seven steps. The reanalysis offers evidence that the empirical pat-
Same-sex parenting terns showcased in the original Regnerus article are fragile—so fragile that they appear lar-
Sexuality gely a function of these possible misclassifications and other methodological choices. Our
replication and reanalysis of Regnerus’s study offer a cautionary illustration of the impor-
tance of double checking and critically assessing the implications of measurement and
other methodological decisions in our and others’ research.
Ó 2015 Elsevier Inc. All rights reserved.

1. Introduction

Research communities in the social sciences have long been aware that methodological decisions can potentially affect
the inferences of survey research (Firebaugh, 2008). This threat to the validity of research inferences is particularly challeng-
ing for studies that focus on a very small group of interest, such as some racial minority groups, atypical families, and same-
sex couples (Cheng and Powell, 2005, 2011). In such research, even a tiny percentage of measurement errors for the small
subsamples could powerfully distort patterns from the surveys, and other methodological choices can similarly affect empir-
ical results. When research findings from these analyses are used as policy guidelines, the threat goes even beyond scientific
communities. It therefore is incumbent for scholars to critically assess the implications of these decisions in their own work
as well as that of others.
In this paper, we use a recent article by Regnerus (2012a) in Social Science Research as an example to illustrate these
points. In ‘‘How different are the adult children of parents who have same-sex relationships? Findings from the New
Family Structures Study,’’ Regnerus (2012a) introduces the New Family Structures Study (NFSS) and, with these data,

q
An earlier version of this paper was presented at the 2013 annual meetings of the American Sociological Association. We appreciate the suggestions of
Kathryn Lively and Michael Yacavone.
1
Authors are listed in alphabetical order; the authors’ contributions are equal.
E-mail addresses: [email protected] (S. Cheng), [email protected] (B. Powell)

http://dx.doi.org/10.1016/j.ssresearch.2015.04.005
0049-089X/Ó 2015 Elsevier Inc. All rights reserved.
616 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626

compares the outcome profiles of 236 adult children whose parents reportedly had a same-sex romantic relationship with
the profiles of those who grew up in other family types, including ‘‘intact biological families,’’ stepfamilies, and single-parent
families. Examining 40 social, emotional, and relational outcomes, Regnerus concludes that adult children of same-sex par-
ents generally fare less well than those from intact two-biological-parent families.
It is an understatement to describe this article as eliciting a great deal of interest. This is one of the most visible and con-
troversial articles to appear in this journal—or, more broadly, social science journals—in recent history. It has been vigorously
defended and critiqued in this journal (Amato, 2012; Barrett, 2012; Eggebeen, 2012; Gates et al., 2012; Schumm, 2012), other
academic journals and forums (Perrin et al., 2013), the courts (Brief of Amicus Curiae American Sociological Association,
2013; Brief of Amici Curiae Social Science Professors, 2014), and the public sphere (Davidson, 2012; Gallagher, 2012;
Luscombe, 2012). Defenders often point to what they see as the high quality of the data, which, they argue, ‘‘deserve[s]
to be considered the gold standard in this field’’ (Sprigg, 2012). Osborne (2012), identified as ‘‘key collaborator’’ on the
NFSS website, praises the study for being ‘‘one of the most comprehensive and rigorous studies that has been conducted
in this field to date’’ (p. 779). In contrast, critics call into question, among other things, the study design, the quality of
the data, review process, and even the motives of the author and funders of this project (Cohen, 2013; Perrin et al., 2013;
Sherkat, 2012). Both sides of the debate often characterize the other side as non-scientific and overly political.
We take a different approach in evaluating the NFSS and the findings reported by Regnerus. We agree with Smith (2012)
who, in challenging critics of Regnerus, contends that ‘‘science already has its own ways to deal with controversial research
results. Studies should be replicated. Data sets should be made public and reanalyzed. . . Eventually the truth comes out. By
those means, Regnerus might be shown to have been wrong or perhaps be vindicated. That is how science is supposed to
work.’’ To his credit, Regnerus has made his data publicly available and, in fact, notes that a goal of his original article is
to ‘‘serve[s] as a call’’ (2012a, p. 766) to analyze NFSS. We have accepted this invitation to reanalyze these data. In this article,
we report on the results of this reanalysis.2
The fact that Regnerus’s findings are so markedly different from those reported by previous studies suggests that scholars
and policymakers should more carefully scrutinize his analysis before reflexively accepting—or rejecting—its conclusion. To
explain his different findings, Regnerus suggests that, ‘‘[t]he answer lies in part with the small or nonprobability samples so
often relied upon in nearly all previous studies—they have very likely underestimated the number and magnitude of real
differences between the children of lesbian mothers (and to a lesser extent, gay fathers) and those raised in other types
of households’’ (2012a, p. 756).
We are hesitant to accept this explanation without further examination of the data because, as others have noted in their
reanalyses of other national surveys (Bearman and Parigi, 2004; Fischer, 2009), findings from empirical analyses often are
also affected by other factors, including the conceptualization and operationalization of key concepts and other methodolog-
ical decisions made by the researcher in the research process (Firebaugh, 2008). These considerations are directly relevant to
the comparison of same-sex parent families and other family forms because analyses of small-population groups using large
survey data are particularly sensitive to different analytical decisions (Black et al., 2007; Cheng and Powell, 2005, 2011;
Gates and Steinberger, 2009; O’Connell et al., 2010). In the case of Regnerus’s study, the NFSS data are new, the measures
of family types and respondents with same-sex parents are somewhat novel and potentially problematic, and the analytical
decisions made by Regnerus arguably are not entirely consistent with the general practices in the field. In revisiting the
Regnerus article and reanalyzing the NFSS, we ask one fundamental question: To what extent are the patterns reported by
Regnerus attributable to the conceptualization and operationalization of family types—in particular, gay/lesbian/bisexual fami-
lies—and other analytical decisions?
Our empirical reexamination of Regnerus’s analysis is designed to answer this question. More broadly, it underscores the
importance of, in the words of Firebaugh (2008), ‘‘build[ing] reality checks into your research’’ (p. 64)—in particular, ‘‘inter-
nal reality checks’’ (p. 65), checks on ‘‘dubious values and incomplete data’’ (p. 65), and checks on ‘‘consistency in concep-
tualization and measurement’’ (p. 69)—and the serious implications of not attending to these concerns (Bearman and Parigi,
2004; Cheng and Powell, 2011; Fischer, 2009). In addition, it highlights the general challenges that social scientists continue
to face in our examination of same-sex parent households and other emerging family forms using nationally representative
datasets (Cheng and Powell, 2005).
Below, we first discuss the NFSS and Regnerus’s measures of family types using the data, and then highlight the difficul-
ties in using the NFSS to accurately distinguish between family types, using adoptive households and intact biological fam-
ilies as illustrations. We then discuss the challenges in accurately identifying same-sex families. We follow this discussion
with a closer look at the NFSS survey and demonstrate the potential for misclassifying a non-negligible number of respon-
dents as having been raised by parents who had a same-sex romantic relationship. Finally, we assess the cumulative impli-
cations of these possible classification errors and other methodological considerations from various stages of the research
process by reanalyzing the NFSS in seven steps.3
These reanalyses provide a ‘‘reality check’’ regarding the conclusions from the original Regnerus study. The patterns from
these reanalyses offer evidence of the fragility of these conclusions—so fragile, in fact, that they are due primarily to the
methodological choices made by Regnerus. Or to put it another way, when equally plausible and, in our view, preferred

2
In the spirit of full disclosure: one of the authors declined an invitation by Regnerus to participate as a paid consultant on the NFSS sampling strategy and
measurement.
3
For an insightful, complementary reanalysis that focuses primarily on same-sex couple households, see Rosenfeld (2012).
 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626 617

methodological decisions are used, a different conclusion emerges: adult children who lived with same-sex parents show
comparable outcome profiles to those from other family types, including intact biological families. That this revised conclu-
sion is consistent with those reported in most previous studies and inconsistent with Regnerus’s findings illustrates how the
accumulation of research decisions throughout the research endeavor—and, in particular, measurement decisions that over-
look inconsistent information within the data—may lead to questionable conclusions, even with a population-based large
sample.

2. The New Family Structures Study and measures of family types

As described in its website, the New Family Structure Study ‘‘is a comparative project which seeks to understand how
young adults (ages 18–39) raised by (our emphasis) same-sex parents fare on a variety of social, emotional, and relational
outcomes when compared with young adults raised in homes with their married biological parents, those raised with a step-
parent, and those raised in homes with two adoptive parents’’ (Regnerus, 2012b). The data collection was in two stages. In
the first stage, a screener survey was used to identify family types that respondents were raised in, while in the second stage,
a detailed survey was used to gauge, among other items, respondents’ experiences in young adulthood. Regnerus differen-
tiates this study from others in four regards: (1) it uses a national population-based sample instead of snowball or conve-
nience samples; (2) it uses a larger sample than do most other studies of same-sex families; (3) it focuses on the current
experiences and ‘‘lives of young adults between the ages of 18 and 39, but not about children or adolescents’’ (Regnerus,
2012a, p. 755); and (4) it includes a wide array of items intended to gauge ‘‘subsequent life outcomes for adult children’’
(Regnerus, 2012c, p. 1367).
In his first article using the NFSS data, Regnerus (2012a) notes that his study offers ‘‘statistical comparisons of them [re-
spondents’ adult outcomes] among eight different family structures/experiences of origin’’ (p. 755), including adoptive fam-
ilies, single-parent families, and stepfamilies. He focuses, though, on the distinction between ‘‘intact biological families’’ (IBF)
and ‘‘lesbian mother’’ (LM) and ‘‘gay father’’ (GF) families. In multivariate analyses of 40 outcomes,4 he finds significant LM–
IBF differences for 24 outcomes and GF–IBF differences for 19 outcomes. Differences in some of these outcomes are merely dif-
ferences, not necessarily disadvantages: for example, whether the young adult identifies as entirely heterosexual and whether
the young adult is in a same-sex romantic relationship. Recognizing that some of these outcomes may be more consequential
than others, he emphasizes outcomes ‘‘that are obviously suboptimal’’ (p. 764), pointing to ‘‘education, depression, employment
status, or marijuana use’’ (p. 764) as examples. Regnerus indicates that the goal of his article is not to identify the reasons behind
the patterns he reports here. That said, he does express concern that families with two same-sex parents still exhibit ‘‘a dimin-
ished context of kin altruism (like adoption, step-parenting, or nonmarital childbirth), which have [sic] typically proven to be a
risk setting, on average, for raising children when compared with married, biological parenting’’ (p. 765). In later writings, he,
along with some fellow social scientists, refers to the ‘‘benefit from the unique parenting contributions of both men and
women’’ (Brief of Amici Curiae Social Science Professors, 2014, p. 4).
The data collection efforts for this project are certainly impressive, especially with its large nationally representative sam-
ple,5,6 multiple outcomes covered, and attempts to identify different family structures. While some portray the NFSS and, in
turn, the Regnerus analysis, as a gold standard in family research, we contend that a critical hallmark of any study is its ability
to accurately measure its key variables of interest—in this case, the different family types.
In revisiting the NFSS, we were struck by the difficulties in unequivocally categorizing respondents by family type—or to
put it another way, the challenges in developing valid measures of family type. For example, Regnerus relies on a screener
survey to identify 101 adult respondents who were ‘‘adopted by one or two strangers at birth or before age 2.’’ The restriction
to respondents who were adopted at such an early date presumably is to ensure that any documented patterns for this group
can be attributed to having been raised in an adoptive family since early childhood. Regnerus also notes the presence of ‘‘‘cal-
endar’ data from each respondent about their relationship to people who lived with them in their household (for more than
4 months) from birth to age 18, as well as who has lived with them from age 18—after they have left home—to the present’’
(2012a, p. 757). Regnerus acknowledges that these data are ‘‘only sparingly used’’ in his analysis, but affirms that ‘‘. . . such
rich data enables [sic] researchers to document who else has lived with the respondent for virtually their entire life up to the
present’’ (2012a, p. 757).
These data are rich, and Regnerus deserves credit for collecting these complicated data. When we compare the responses
from the calendar data to those from the screener survey, however, we notice that 9 of the 101 respondents report that they

4
Controls include respondent’s age, gender, race/ethnicity, mother’s education, family income while growing up, experience of being bullied in childhood,
and state’s legislative gay friendliness.
5
Others have questioned Regnerus’s reliance on internet surveys collected by Knowledge Networks, now GfK (Sherkat, 2012); however, the quality of data
collected from internet surveys completed by a nationally representative, probability-based survey web panel is comparable to that of other data collection
efforts that also rely on random digit dialing (Chang and Krosnick, 2009). Despite some limitations to internet surveys of this type—or, for that matter, surveys
in general—it bears pointing out these surveys have been productively used in sociological scholarship on family and relationships (Doan et al., 2014; Rosenfeld
and Thomas, 2012; Seltzer et al., 2012).
6
Response rates from Knowledge Networks/GfK are similar to the industry norms. That said, the ‘‘65% within survey response rate’’ reported by Regnerus
(2012a, p. 756), although technically accurate, might mislead readers into believing that the overall response rate for NFSS is very high. As reported by others
who have analyzed data collected by Knowledge Networks/GfK, the cumulative response rate—which takes into account not only within survey response rate
but also recruitment rate and demographic profile completion rate—typically is less than 15%, a rate that still is consistent with those from comparable data
collection efforts (Rosenfeld and Thomas, 2012; Seltzer et al., 2012).
618 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626

actually had only lived with their adoptive parents for a very short period of time: 4 for less than a year, 2 for one year, 1 for
two years, and 2 for three or four years. In one case, for example, the respondent reports that she lived with her adoptive
father for one year only (when she was 1) while also living with her biological mother from birth until she was 16 years
old. In another case, the respondent reports having lived with his adoptive mother for three years and then his biological
mother for the next 15 years. Technically speaking, these 9 cases may be consistent with Regnerus’s description that the
respondents were ‘‘adopted by one or two strangers at birth or before age 2’’; however, if we are to take the responses seri-
ously (an issue we return to shortly), then the inconsistencies between the screener survey and the calendar data should give
us concern over whether these cases are accurately classified as ‘‘adoptive families.’’7 These concerns, however, are not lim-
ited to adoptive families, but also extend to the descriptions and classifications of stepfamilies, single-parent families, and, per-
haps most importantly as we discuss shortly, ‘‘lesbian mother’’ and ‘‘gay father’’ families in the Regnerus analysis.
Even in the absence of inconsistencies in responses, there is the potential to miscode or misclassify family types. For
example, Regnerus’s operationalization of ‘‘intact biological families’’ is puzzling. Of the 2988 respondents included in
Regnerus’s analysis, 1195 answer ‘‘yes’’ to the screening question ‘‘Did you live together with BOTH your biological mother
AND biological father the entire time from when you were born until age 18 (or until you left home to be on your own)?’’
(original emphases). Yet, Regnerus chooses to include only 919 of these respondents in the category of ‘‘intact biological fam-
ily’’ (IBF). Excluded from this category are the 116 respondents whose parents were not married at the time of the interview,
which he places in the category ‘‘divorced later or had joint custody.’’
To the extent that the NFSS is intended to, among other things, examine the outcomes of ‘‘young adults (ages 18–39) who
were raised (emphasis ours) in different types of family arrangements’’ (Regnerus, 2012a, p. 752), the distinction between
these two groups—which, in the absence of other information, appear to be virtually identical in structure during the respon-
dents’ childhood—cannot be reconciled with the goals of the project.8,9 Where these two groups may differ is in the quality or
functionality of the marriage—characteristics that may affect the well-being of children. For example, parents who divorced
later may have had an unhappy marriage but nevertheless stayed married until their children had left their home. If so, the deci-
sion to exclude this group from the category IBF could overstate the positive consequences of being raised in this family type,
especially compared to being raised in other family types.

3. Challenges in identifying parents who had a same-sex romantic relationship

The preceding discussion underscores the challenges in identifying children who were raised in adoptive families and
intact biological families. Our primary concern, however, is in regards to Regnerus’s classification of same-sex parent fam-
ilies. Regnerus (2012a) identifies children raised by same-sex parents on the basis of responses to the question, ‘‘From when
you were born until age 18 (or until you left home to be on your own), did either of your parents ever have a romantic rela-
tionship with someone of the same sex?’’ If respondents responded affirmatively to this question, they were then asked, ‘‘Did
you ever live with your mother/father while s/he was in a romantic relationship with another woman/man?’’
Even if we are to accept Regnerus’s position that these items accurately measure ‘‘LM (child of a lesbian mother), and GF
(child of a gay father)’’ (2012a, p. 758)—a position that we challenge below—it is telling that these questions apparently were
not asked of all respondents in the NFSS. Notably exempt from answering these questions, as indicated in the screener survey
and the subsequent survey,10 were respondents who reported living ‘‘together with BOTH your biological mother AND biolog-
ical father the entire time from when you were born until age 18 (or until you left home to be on your own).’’ In other words,
Regnerus’s analysis is based on the assumption that parents in intact biological families never have ‘‘romantic relationships with
someone of the same sex’’ while the parents are married—an assumption that is highly difficult to defend.11 To the extent that
he equates lesbian and gay families with the parental relationship history (during the respondents’ childhood), Regnerus under-
estimates the number of children from LM and GF households and, in turn, overestimates the number of children of IBF
households.

7
We take a conservative approach in identifying these questionable cases of ‘‘adoptive families.’’ Not included among these 9 cases, for example, is a
respondent who claims to have always (from birth until leaving home) lived with her biological mother, adoptive mother, adoptive father, grandmother,
grandfather, and foster parents.
8
Respondents who fall into these two categories apparently were not asked to complete the calendar data. Instead, the calendar data for these two groups
are imputed solely on the responses to the screener survey: that is, as described in the NFSS survey, these respondents are ‘‘automatically assign[ed] ‘always’ to
‘biological mother’ and ‘always’ to ‘biological father.’’’ In other words, in contrast to the respondents from other family types, there is no mechanism to check for
consistency, or reliability, in responses for these two groups or to identify other adults who may have lived in the household.
9
Perhaps the only way to justify this distinction is to point to the problematic nature of the screening question. Although the question asks whether
respondent lived ‘‘together with BOTH your biological mother AND biological father,’’ it is possible that a small number of respondents might have interpreted
this to mean being raised by each parent but in a joint custody arrangement. Surprisingly and unfortunately, the NFSS does not ask when respondents’ parents
were divorced. In the absence of this potentially confirming or disconfirming information, the most intuitive coding, especially given the phrasing of the
question, is to include the 119 respondents in the IBF category.
10
It is possible that Regnerus was reluctant to ask this question in the screener survey because of concern that doing so would result in a lower response rate
from the IBF sample. That said, as in the case of other potentially controversial items, it could have been asked in the subsequent survey or, following protocol
regarding potentially controversial items, could have been included in the end.
11
Nor does the screener survey or the latter survey ask whether parents in intact biological families ever had a romantic relationship with an opposite-sex
partner other than one’s spouse.
 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626 619

After examining the data, however, we see a potentially larger threat to validity at hand: that Regnerus also may have over-
estimated the number of children from LM and GF households and, in turn, misidentified respondents from other family types
as coming from LM and GF households. In replying to critics, Regnerus (2012c) defends his reliance on respondents’ reports of
parental relationships, but acknowledges that these measures do not necessarily correspond with respondents’ assessments
of parental sexual orientation or parental assessments of their own behavior. What he does not acknowledge, however, is the
possibility of inaccuracy even in respondents’ reports of parental relationships. Like all studies that rely on surveys, Regnerus’s
study assumes that respondents interpret survey questions in the same way that the researcher intended them to be inter-
preted. Yet, there is ample evidence to question this assumption. In our own work, for example, we have documented the chal-
lenges that some individuals face and mistakes they make in understanding and interpreting questions regarding racial
identification (Cheng and Powell, 2011), and, more directly relevant to the question of same-sex households, words such
as heterosexual, bisexual, civil unions, and ‘‘two women (two men) living together as a couple’’ (Powell et al., 2010, 2015).
Similarly, Savin-Williams and Joyner (2014) attribute the large number of ‘‘dubious’’ cases of gay, lesbian, and bisexual ado-
lescents in the Add Health data set to confusion over the meaning of ‘‘romantic attraction’’—as well as to mischievous joke-
sters who are not truthful or careful in their responses regarding their sexuality.
The potential for misconstrued interpretations—and, therefore, threats to validity—is among the reasons that so many
scholars conduct cognitive interviews or pretesting before fielding their surveys. It is unclear, though, whether Regnerus fol-
lowed this standard, or at least preferred, practice. Even if he did, it would be difficult to detect misinterpretations—or even
careless or cavalier responses—made by a very small percentage of respondents. Fortunately, for most studies, these small
errors likely have little impact on key patterns coming from surveys. Unfortunately, for studies focusing on a very small
group of interest—in this case, people who were raised in same-sex parent households—even a tiny percentage of error could
powerfully distort patterns from the surveys.
Researchers using nationally representative datasets to study same-sex parent families have routinely checked for poten-
tial coding errors or inconsistent cases in their data. For example, in their analysis of the National Longitudinal Survey of
Adolescent Health (Add Health) (Harris and Udry, 2010), Wainright et al. (2004) identified 44 cases of adolescents in
same-sex families—a small sample size that Regnerus critiques for minimizing the likelihood of finding significant differ-
ences between same-sex and other families. To their credit, though, they also minimized the number of misclassified cases
of adolescents in same-sex families by checking for consistency in parental reports of their sex and family relationship. Doing
otherwise would have markedly overestimated the number of same-sex parental households: as reported in the Add Health
codebooks, a cross-check of parental responses identified 339 ‘‘male mother figures’’ and 45 ‘‘female father’’ figures.
Similarly, programmers of the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K) performed extensive tri-
angulation checks to eliminate the inconsistent coding of same-sex parent families and other family types: for example, fam-
ilies with a ‘‘male mother’’ or a ‘‘biological mother over age 80’’ (Potter, 2012; Tourangeau et al., 2006). Several years ago, we
too contemplated using the National Education Longitudinal Study of 1988 (NELS) to study children from gay/lesbian house-
holds. But in looking more closely at the parental and adolescent responses of the 69 cases that initially appeared to fit this
description, we concluded that the high rate of discrepancies in responses (over 80%) presented an insurmountable problem
of reliability and, in turn, of validity that effectively precluded any meaningful examination of same-sex families (Cheng and
Powell, 2005).
In the most publicized example of the misidentification of same-sex households, the U.S. Bureau of the Census retracted
its 2010 estimates of same-sex couple households. The initial estimates were derived from two questions from the Census:
relationship to householder and the sex of each person. The Census Bureau subsequently discovered that the number of
same-sex couple households estimated from these two questions could be greatly inflated if a very small fraction of
opposite-sex couples checked the wrong box for the sex of one’s partner or spouse (O’Connell, 2011). In correcting the
errors,12 the Census Bureau lowered its original estimate of 901,997 same-sex couple households to 646,464—a reduction of
28%. The drop in the number of same-sex married couples was even more dramatic, from 349, 377 to 131,729—a reduction
of 62%.13

4. Revisiting the Regnerus categorization of LM and GF households

In light of these documented cases of errors that appreciably inflated the number of same-sex couple households, it is
untenable to automatically assume that NFSS is immune from challenges to validity. Still, Regnerus does not check for, or
apparently even consider the possibility of, inconsistent, uncertain, and unreliable cases in his data—even though some other
items in the NFSS offer some limited means to assess this possibility. For example, Regnerus (2012c) acknowledges that,
according to the aforementioned calendar data, over half of the respondents never lived with a parent’s same-sex partner,
but fails to mention that many respondents—approximately one-third—also never lived with their same-sex parents or lived
with them very briefly. As seen in Table 1, of the 236 respondents classified as being raised by a gay father (GF) or lesbian
mother (LM), 24 (15 GF, 9 LM) report they had never lived with the parent from birth to age 18, 34 (18 GF, 16 LM) report
they had lived with the parent for a year, and 18 (9 GF, 9 LM) report they had lived with the parent for only two to four years.

12
See O’Connell (2011) for discussion of the techniques used to correct for these discrepancies.
13
The error rate in overestimating same-sex couple households and married same-sex couple households for the 2000 Census is even higher: 40% and 83%,
respectively.
620 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626

Table 1
Numbers of years respondents reported living with a same-sex parent or same-sex parent’s partner, NFSS.

Gay father (GF) Lesbian mother (LM)

N % N %
A. Number of years with a same-sex parent
Never 15 20.5 9 5.5
1 year 18 24.7 16 9.8
2–4 years 9 12.3 9 5.5
More than 4 years 31 42.5 129 79.1
Total 73 100.0 163 100.0

Father’s boyfriend Mother’s girlfriend

N % N %
B. Number of years with same-sex parent’s partner
Never 56 76.7 82 50.3
1 year 12 16.4 29 17.8
2–4 years 3 4.1 32 19.6
More than 4 years 2 2.7 20 12.3
Total 73 100.0 163 100.0

Note—Analyses are restricted to the 236 LM/GF respondents identified in Regnerus (2012a).

While one should not discount the potential influence of non-residential parents and one should be cautious in identifying
the exact number of years that a child needs to live with a parent to be considered raised by that parent, it is difficult to
reconcile these patterns with Regnerus’s assertion that the 236 respondents ‘‘were raised (emphasis ours) by parents that
had a same-sex relationship.’’ (2012a, p. 755).
Upon closer inspection of the calendar data and other responses, we discovered additional inconsistencies that call into
question the coding of a sizeable number of the 236 LM and GF respondents. To identify the inconsistencies, each coauthor
examined each case independently. A summary of our reanalysis, which displays only those cases in which the coauthors’
ratings correspond, is provided in Table 2.
As a standard procedure of data analysis, we begin by first detecting 9 cases with highly unlikely or potentially unreliable
and, in turn, invalid responses to other questions in the survey. The most blatant example of highly suspicious responses is
the case of a 25 year-old man who reports that his father had a romantic relationship with another man, but also reports that
he (the respondent) was 7-feet 8-inches tall, weighed 88 pounds, was married 8 times and had 8 children. Other examples
include a respondent who claims to have been arrested at age 1 and another who spent an implausibly short amount of time
(less than 10 minutes) to complete the survey.14 These cases are akin to the aforementioned jokesters in the Add Health data
set (Savin-Williams and Joyner, 2014) and also are consistent with ongoing concerns regarding truthfulness and satisficing in
internet surveys (Baker et al., 2014).
After identifying these 9 cases, we compare responses in the screener survey with calendar responses in the following
survey and locate an additional 53 respondents who report that they lived with their lesbian mother or gay father for a year
or less.15 We then find 20 other respondents whose answers in the calendar data and screener survey appear inconsistent or
improbable.16 Among these are:

1. Four respondents who report that they lived with their biological parent, that parent’s opposite-sex partner (i.e., respon-
dent’s stepparent), and that parent’s same-sex partner in the same year.17
2. Eight respondents who report that they lived with mother’s girlfriend or father’s boyfriend while the mother/father was
absent in the family. That the biological parent also lived with an opposite-sex (step)parent, never lived with both the
respondent and alleged same-sex partner at the same time, and/or was absent in the family suggests a good possibility
that the ‘‘boyfriend’’ or ‘‘girlfriend’’ figure is potentially a close, non-romantic friend rather than the parent’s partner.
3. Eight respondents who show various degrees of inconsistencies in their responses to calendar data and in other survey
questions. For example, one respondent reports having always lived alone but also claims to have always lived with
mother, father, and two grandparents. Another two respondents report that during their childhood, their mother had a
same-sex relationship but also report that they always lived with mother but never lived with the mother while she
had a same-sex relationship.

14
According to the codebook, the median time to complete the main survey was 34 min.
15
Of these 9 cases, a few also report that they lived in a LM or GF household for a year or less, thereby explaining the seeming discrepancy between the 53
cases reported here and the 58 cases reported in Table 1.
16
It is possible, of course, that the respondents were inaccurate when completing the calendar data. This possibility, however, would place Regnerus in the
paradoxical and indefensible position of defending the accuracy of one section of the NFSS by discounting the accuracy of another section.
17
There is a possibility that within the same year the parent switched partners; however, this possibility is slight given other problematic responses among
these four cases (e.g., regarding income, household size, length of time to complete the survey). Of note, in each case the respondent reports living with the
parent’s same-sex partner in only one year.
 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626 621

Table 2
Potential unreliable, inconsistent, and uncertain cases, NFSS.

N % Cum N Cum %
Unreliable and inconsistent cases
Unreliable responses 9 3.8 9 3.8
Lesbian mother (LM) in household for a year or less 23 9.8 32 13.6
Gay father (GF) in the household for a year or less 30 12.7 62 26.3
Inconsistent responses in screener survey and calendar data 20 8.5 82 34.7
Uncertain cases 6 2.5 88 37.3
Same-sex parent minor roles 15 6.4 103 43.6
Total 103 43.6

Note—Analyses are restricted to the 236 LM/GF respondents identified in Regnerus (2012a).

Taken together, these 82 cases account for over one-third (34.7%) of the 236 respondents categorized by Regnerus as LM or
GF. Although this figure might include isolated cases in which inconsistencies merely reflect very complex family situations,
this slim possibility does not match up with available information in the NFSS. In fact, this figure may be a conservative esti-
mate of the rate of misclassification of respondents. In 6 cases, for example, the responses are sufficiently problematic or
inconsistent that without additional information that was unfortunately not provided in the survey it is difficult to conclude
with confidence that they actually had lived in a LM or GF household.18 An additional 15 of the remaining respondents report
having lived in a LM or GF household for only 2–4 years—such a short period of time that it is uncertain whether these respon-
dents are most accurately defined as having been ‘‘raised’’ by parents that had a same-sex relationship. If one includes these
cases, the rate increases to 43.6%—a figure that still is consistent with the error rates detected, and then adjusted for, in the
Census and other national surveys mentioned earlier in this paper.

5. Implications of methodological decisions and alternative coding

We readily acknowledge that some of our coding decisions are open to different interpretations and can be debated. What
cannot be debated, however, is that there are uncertainties and potential errors in Regnerus’s operationalization of LM and
GF respondents and that there appears to be little attempt on Regnerus’s part to uncover these possible classification errors
or, more importantly, to assess the implications of the inclusion of cases that at minimum are contestable. As scholars have
suggested elsewhere, in the analysis of small-subsample groups, even a small number of misidentified cases may alter the
conclusions researchers draw from their data. Below we evaluate the extent to which the patterns reported by Regnerus are
contingent on his coding and other methodological decisions.
We begin by replicating Regnerus’s analysis (2012a), and then assess the implications of using alternatives to Regnerus’s
analytical decisions—alternatives that are common practices in social scientific research. Next, we consider how the poten-
tially miscoded cases affect outcome differences between children raised by same-sex parents and children raised by intact
biological families. Finally, we repeat this analysis but restrict the sample to children from same-sex couple households—i.e.,
households in which the same-sex partner ever lived with the child. These results are summarized in Table 3, in which we
identify LM–IBF and GF–IBF differences that are significant at the .05 level at each step of our reanalysis. This table is
restricted to the 32 of the 40 outcome variables that were significant at any point in Regnerus’s multivariate analysis or
in ours.

5.1. Correcting analytical considerations

We begin with a baseline model that replicates Regnerus’s (2012a) original analysis. As others have noted, Regnerus did
not report the regression coefficients or standard errors in the article (Perrin et al., 2013). Nevertheless, we were able to
replicate the reported mean scores of the outcome variables for IBF, LM, and GF. Of the total 40 outcome variables (as noted
above, only 32 are shown in Table 3), LM is significantly different from IBF in 24, and GF is significantly different from IBF in
19 (Table 3, first row). Since our attempts to replicate are successful, this baseline model allows us to assess the implications
of alternative analytical and measurement considerations and corrections.
In the second step, we adjust for four coding decisions that either are errors or have a plausible alternative:

1. In two binary outcomes, refusals to respond to the question were coded as ‘‘0’’ when they should be coded as missing.
2. For the question about voting in the last presidential election, respondents who were not old enough to vote at the time of
the election were included in the analysis when they should have been coded as missing.
3. Several outcomes measures have identifiable units (e.g., household income in thousand dollars), but were coded as cat-
egorical and analyzed using OLS. We recode these variables by their identifiable units.

18
Among these are respondents who lived with both parents and then lived with their father and stepmother and with their mother and stepfather.
 622
Table 3
Multivariate analyses of the outcome differences between children raised by same-sex parent families and those living with both biological parents until age 18.
Currently Family Currently Currently Currently Voted in last Thought Recently Identifies as Is in a Had affair while Has Ever Ever Educational Family- Family- Closeness CES-D Attachment Level of Current Current Frequency Frequency Frequency Frequency Frequency N of female N of female N of male N of male Total
cohabiting received on public employed unemployed presidential recently or entirely same-sex married/cohabiting ever touched forced to attainment of-origin of-origin to depression scale household relationship relationship of of smoking of of having pled guilty sex partners sex sex sex

S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626

welfare assistance full-time election about currently heterosexual romantic had sexually have sex safety/ negative biological index (depend) income quality is in trouble marijuana watching been to non- (among partners partners partners
growing suicide in relationship an by parent/ against security impact mother index use TV arrested minor women) (among (among (among
up therapy STI adult will offense men) women) men)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

1 Replicate LM                        x 24
Regnerus GF                    19
(2012a)
2 Adjust coding LM                          25
for outcome GF                 16
measure
3 Expand IBF LM                     20
category GF             12
4 Add LM           10
supplementary GF             12
controls
5 Impute missing LM            11
values in GF           10
controls
6 Add potential LM       6
coding error GF    3
controls
7 Family type by Same-     4
the calendar sex
data couple

Note—LM signifies lesbian mother; GF signifies gay father; same-sex couple signifies LM or GF households in which the respondent lived with both the parent and the parent’s same-sex partner for at least one
year. This table does not include 8 of the 40 outcomes in which neither Regnerus (2012a) nor we found significant LM or GF effects. ‘‘’’ indicates that the difference from IBF (intact biological families) is
statistically significant at the .05 level.
 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626 623

4. The coding of number of sexual partners was not consistent with other comparable items in the study, which Regnerus
(2012a) coded as count variables. We correct for this inconsistency by recoding the number of sexual partners as count
variables and using count models.

Coding decisions such as these underscore the importance of taking the time to double check all analyses and consider
alternative coding of dependent variables. In this case, however, these adjustments have minimal effect on the outcomes. As
shown in Table 3 (second row), these corrections actually increase the number of significant differences between LM and IBF
from 24 to 25. They also, however, decrease the number of significant differences between GF and IBF from 19 to 16.
In the third step, we expand the category IBF to include all respondents who reported living together with both their bio-
logical mother and biological father from birth to age 18. As discussed earlier, Regnerus identified 116 respondents who fit
this description but coded them as a separate category because their parents were no longer married at the time of survey.
This coding decision is incompatible with Regnerus’s goal of analyzing the influence of the family arrangements in which
youths were raised and further complicates any causal claims regarding outcomes in adulthood. In fact, this coding decision
appears to have artificially increased the LM–IBF and GF–IBF differences. As shown in Table 3 (third row), when the IBF cat-
egory is expanded, the number of significant differences decreases (to 20 between LM and IBF and 12 between GF and IBF).
In the fourth step, we reconsider Regnerus’s inclusion of controls in multivariate models. Regnerus commendably added
controls (e.g., gender, age, level of mother’s education) to make sure that any ostensible effects of family structure on ‘‘sub-
sequent life outcomes for adult children’’ (2012c, p. 1367) are not a function of sociodemographic background of the respon-
dent or respondent’s family. We too add these controls but make two adjustments—one involving recoding and one
involving the inclusion of other controls. We recode the measure of race/ethnicity so that instead of making a mere binary
distinction between white and non-white respondents, we can distinguish among respondents who identify as white, black,
Hispanic, multiracial, or other. As Regnerus’s collaborator Osborne (2012, p. 780) recommended, ‘‘to be consistent with the
rest of the analysis,’’ the variable ‘‘family received welfare growing up’’—which certainly is reflective of family socioeconomic
status—should be counted as a control variable. Following this recommendation, we add this variable—along with biological
mother’s age and biological father’s age at birth of child, region, and residential area (metropolitan = 1)—as additional con-
trols.19 As shown in Table 3 (fourth row), doing so reduces the number of LM–IBF significant differences to 10, but does not
change the number of GF–IBF significant differences.20 This change suggests that Regnerus’s analysis underestimates the effects
of respondents’ sociodemographic background, at least in regards to LM–IBF differences.
In the fifth step, we use multiple imputation techniques for missing values in control variables. There are different
approaches to handling missing cases, but Regnerus’s decision to delete missing cases contradicts his goal of maximizing
the sample sizes of respondents who report being raised by same-sex parents.21 It also is at odds with the assessment of some
applied statisticians that multiple imputation offers a more efficient use of existing data, produces more unbiased estimates in
multivariate analyses, and thus is a preferred solution to the missing data problem (Acock, 2005; Allison, 2002; von Hippel,
2007). When the missing data are completely at random (MCAR) or at random (MAR), the use of multiple imputation increases
the likelihood of significant patterns. This applies to the LM–IBF comparison, as the number of significant differences increases
from 10 to 11 (Table 3, fifth row). When the MCAR or MAR conditions are not satisfied, however, listwise deletion could lead to
biased estimates. In Table 3, we see that the use of multiple imputation actually further reduces the number of significant
GF–IBF differences from 12 to 10, which suggests that the GF–IBF differences in the deleted cases are smaller than the
differences in the other cases. Compared to Regnerus’s original findings, the difference that these simple methodological
modifications from these five steps—corrections that certainly are not atypical practices in the discipline—make is remarkable.

5.2. Controlling for misclassified and uncertain cases

As noted in our earlier discussion (and presented in Tables 1 and 2), we have strong reason to question Regnerus’s clas-
sification of over one-third of the 236 respondents identified as living with LM or GF parents. Eighty-two (i.e., 34.7%) pro-
vided responses that appear unreliable, indicated that they had lived with their LM or GF for a very short period of time
(i.e., one year or less), or offered other information that seriously undermines Regnerus’s classification scheme. We also have
some doubts about the classification of another 21 respondents (8.9% of the 236 cases) either because of insufficient infor-
mation in the data set or because they reported living with their LM or GF parents for only 2–4 years. As a corrective, in the
sixth step we keep Regnerus’s measures of the 163 LM and 73 GF respondents in the model, but we also add two

19
Osborne (2012) also suggests that two other outcomes—whether one was ‘‘ever touched sexually by a parent/adult’’ or ‘‘ever forced to have sex against
will’’—could be considered candidates for additional control variables. Although the models reported in Table 3 do not include these controls, following her
suggestion would further reduce the number of significant differences across family structure.
20
Although Regnerus’s use of ‘‘family received welfare while growing up’’ as a dependent variable is contrary to his stated purpose is to explore ‘‘the lives of
young-adult children of gay lesbian parents’’ and ‘‘their experiences and accomplishments as adults’’ (2012a, p. 755), we retain this childhood experience as a
dependent variable in Table 3 and include all other control variables in the multivariate models predicting this item.
21
In Table 1 of his original study, Regnerus (2012a) reports 27 missing cases in experience being bullied as a youth, 8% of missing cases in mother’s education,
and 22% of missing data in family income. The missing cases in mother’s education and family income are kept in the analyses using dummy variables. This
approach tends to result in biased estimates in multivariate analyses. To replicate the results in Regnerus (2012a), however, we also use a series of dummy
variables (including dummy variables for missing cases) for mother’s education and family income. We also found one respondent with missing value in
gender, which is not reported in Regnerus (2012a).
624 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626

dichotomous variables for the 82 and 21 cases to control for the potential confounding effects of these misclassified or uncer-
tain cases, respectively.
As shown in Table 3 (row 6), the number of significant LM–IBF differences is reduced to only 6 of the 40 outcome mea-
sures, while the number of significant GF–IBF differences is cut to only 3.22,23 Additional sensitivity analyses suggest that, in
the case of LM respondents, the significant effects for 3 of the 6 outcomes—i.e., family security, frequency of being arrested, and
frequency of pleading guilty—are so fragile that they disappear simply by deleting 1 or 2 cases from the analysis.24 With 163
respondents in the LM category, this is not an issue of statistical power.
These results suggest that 3 of the 6 significant coefficients are highly sensitive to 1 or 2 influential cases. The only three
outcomes in which a significant LM–IBF difference remains are: (1) family received welfare assistance growing up, (2) self-
identification as entirely heterosexual, and (3) had affair while married/cohabitating (#2, #9, and #11 in Table 3). Of these,
receiving welfare assistance in childhood is an outcome that, as we noted earlier and others have articulated elsewhere
(Osborne, 2012), more appropriately should be considered a control variable and certainly is not an indicator of the respon-
dents’ experiences as adult. Similarly, whether a respondent self-identifies as homosexual or heterosexual should carry no
advantageous or disadvantageous implications as an outcome measure (i.e., neither outcome should be seen as, in Regnerus’s
term, ‘‘suboptimal’’); moreover, this pattern already has been confirmed in other studies (Stacey and Biblarz, 2001). If these
two variables are excluded from the list, only 1 coefficient for LM respondents is statistically significant and could conceiv-
ably be seen as a possible disadvantage to adult children from LM households.25

5.3. Assessing the consequences of living in a two-parent LF or GM household

The above analyses focus on the experiences of respondents who report living in a household in which at least one parent
had a same-sex romantic relationship. Of the 236 respondents identified by Regnerus (2012a) as living in a LM or GF house-
hold, we identify only 51 that can plausibly be coded as being raised for at least a year in a same-sex couple household.26,27 The
other respondents are better characterized as living in other family types. In Table 3, we further examine whether and how the
outcome profiles of the 51 adult children respondents from same-sex-two-parent households differ from the profiles of those
from IBF households. Here we find only four significant differences, although the differences either are not indicative of any LM/
GF disadvantage (i.e., sexual self-identification and having a same-sex romantic relationship) or do not gauge adult experiences
(i.e., receiving public assistance in childhood and sense of safety and security while growing up). These patterns also are highly
fragile and based in part on a couple of influential cases or outliers. Admittedly, even with a large overall sample, a subsample of
51 cases still limits the statistical power of the analysis. Still, the results are either inconclusive or suggestive that adult children
raised by same-sex two-parent families show a comparable adult profile to their peers raised by two-biological-parent families.

6. Conclusion

The standard advice in survey research textbooks—and presumably in most courses on research methods—is that
researchers should double-check their concepts, variables, and statistical analyses, and be initially skeptical of the results,
even if they correspond with the researchers’ expectations. Confidence increases if the patterns are sufficiently robust that
they hold up with the use of different coding and control variables and additional analysis of potential outliers and influen-
tial cases. These ‘‘reality checks’’—as recommended by Firebaugh (2008)— can both build up trust in the substantive conclu-
sions and increase the credibility of our research community as a whole.
In this paper, we document the empirical implications of not following this recommendation by using Regnerus’s recent
article on adult children of same-sex parents as a case in point. Our reanalysis of the NFSS and the Regnerus study

22
Recognizing that there can be disagreement over which cases should be classified as misclassified or uncertain, we also considered alternative
classifications that were more expansive or more restrictive. Models using these alternative classifications also resulted in a notable decrease in the number of
significant LM–IBF and GF–IBF differences.
23
Importantly, the controls for misclassified or uncertain cases are significant for 13 outcomes. For example, respondents in the misclassified category are
significantly more likely to report they thought recently about suicide and were not close to their mother. They also indicated a higher frequency of drinking to
get drunk. In addition, and perhaps even more importantly, they also rated higher on the CES-D depression scale, family-of-origin negative impact scale, and
lower on the current relationship quality index. That this group significantly differs from others in the LM and GF sample, as well as others from the overall
sample, offers further support for our contention that respondents in this group were misclassified by Regnerus.
24
Residual analysis showed significant proportions of cases as potential outliers (i.e., standardized residuals greater than 2.5 standard deviations) in five of
the six outcomes (approximately 3–4%). These large numbers of potential outliers may signal the abnormality of the data, the failure of the statistical models to
capture the important characteristics of the data, or both. Because the patterns of outliers are likely to change with different model specifications, our
sensitivity analyses are not restricted to outliers.
25
As seen in Table 3, there are statistically significant GF–IBF differences for only three outcomes: whether the family received welfare at some point in the
respondent’s childhood, the number of male sexual partners among female respondents, and the number of female sexual partners among male respondents.
As noted earlier, the first—whether the family received welfare at some point in the respondent’s childhood—is more appropriately considered a control
variable than as an adult outcome. The patterns regarding the other two outcomes suggest that respondents from GF households are more sexually active (i.e.,
more opposite-sex partners) than those from IBF households. The extent to which these differences imply a disadvantage or advantage is unclear.
26
In the supplementary analyses, we also differentiated between respondents from LM-couple and GF-couple families. Given the small number of cases,
however, the analysis in Table 3 is based on 51 respondents from either type of household.
27
For a detailed analysis of NFSS that focuses on same-sex couple parents, see Rosenfeld (2012).
 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626 625

demonstrates how the accumulation of contestable research decisions—from the initial conceptualization and measurement
in the questionnaires to inattention to inconsistences in survey responses to coding, modeling and treatment of missing
cases—can result in a notably ambitious study that still yields disputable patterns. The methodological problems we describe
not merely are those of a given research question or one particularly flawed article, but pose a risk more generally to infer-
ence from social surveys.
Our primary concern regarding the NFSS in this paper, however, is in the measurement—or what we believe to be the
mismeasurement—of same-sex families. Although the number of households headed by same-sex parents have rapidly
increased over the past few decades, their proportion in the population remains very small (O’Connell, 2011; Rosenfeld,
2010; Stacey and Biblarz, 2001). Scholars have noted that the analysis of same-sex parent families is sensitive to researchers’
analytical decisions, and even small coding errors can seriously compromise empirical conclusions from the research (Cheng
and Powell, 2005; Gates and Steinberger, 2009). Our replication and reanalysis of Regnerus’s study offer a cautionary illus-
tration of this point.
Regnerus’s analysis of the NFSS generated strong reactions—some laudatory, some scathing—from various stakeholders in
debates regarding family structure and same-sex marriage, despite Regnerus’s assertion in the article that ‘‘the study is
intended to neither undermine nor affirm any legal rights’’ regarding same-sex marriage (2012a, p. 766). What the analysis
did not generate, however, was much empirical analysis. In fact, in a subsequent amicus brief advocating for the ‘‘govern-
ment to continue to recognize marriage as a man-woman union’’ (Brief of Amici Curiae Social Science Professors, 2014, p.
21), Regnerus and his coauthors note that ‘‘despite the attention and scrutiny, the study remains in print and subsequent
analyses of the (now publicly-accessible) data have revealed no analytic errors’’ (Brief of Amici Curiae Social Science
Professors, 2014, p. 19).
Our study is an exception. Taking seriously both Smith’s recommendation to reanalyze the NFSS and Firebaugh’s rule to
‘‘build reality checks’’ in social science research, we revisit Regnerus’s analysis, identify serious problems in his decisions
regarding measurement and models, and offer evidence that the empirical patterns showcased in his article are largely a
function of these decisions. In reanalyzing the data, we find:

1. A non-negligible number of respondents were miscounted as having been raised in LM or GF households. The sources of
these potential errors—which we estimate to exceed one-third of Regnerus’s subsample of LM and GF—were the inclusion
of individuals whose highly implausible responses to other questions call all of their responses into doubt, individuals
who reported living in these households for a very short period of time, and individuals whose responses in the calendar
data were incompatible with the original categorization of being raised in a LM of GF household.
2. A number of other methodological and modeling decisions made by Regnerus—decisions that have plausible alternatives
that at minimum should be checked to assess the robustness of the patterns—appear to artificially inflate the differences
between LM/GF and IBF households.
3. Once corrections to these potential coding errors and alternatives to these methodological choices are made, the putative
disadvantage in the outcome profile of respondents from same-sex parent families (both single-parent LM and GF house-
holds and two-parent LM and GF households) decreases dramatically—with some of the remaining differences not ‘‘sub-
optimal’’ (e.g., whether or not the respondent identifies as entirely heterosexual and the number of other-sex partners) or
a function of one or two influential cases.

We do not claim that the coding we followed or other methodological choices we made are the only reasonable ones, but
we do contend that for a pattern to be believable—especially those that are antithetical with the patterns found in nearly
every other study on the same topic—it should hold up to empirical scrutiny and should withstand the use of different coding
and alternative specifications.28 Regnerus’s analysis does not meet this core requirement. In turn, it does not provide suffi-
ciently credible counterevidence to the longstanding body of scholarship that confirms minimal differences in the consequences
of living with same-sex or opposite-sex parents.

References

Acock, Alan C., 2005. Working with missing values. J. Marriage Fam. 67, 1012–1028.
Allison, Paul D., 2002. Missing Data. Sage, Thousand Oaks, CA.
Amato, Paul R., 2012. The well-being of children with gay and lesbian parents. Soc. Sci. Res. 41, 771–774.
Baker, Reg, Miller, Chuck, Kachhi, Dinaz, Lange, Keith, Wilding-Brown, Lisa, Tucker, Jacob, 2014. Validating respondents’ identity in online samples: the
impact of efforts to eliminate fraudulent respondents. In: Callegaro, Mario, Baker, Reg, Bethlehem, Jelke, Goritz, Anja S., Krosnick, Jon A., Lavral, Paul J.
(Eds.), Online Panel Research: A Data Quality Perspective. John Wiley and Sons, New York, NY, pp. 451–452.
Barrett, Don, 2012. Presentation, politics, and editing: the Marks/Regnerus articles. Soc. Sci. Res. 41, 1354–1356.
Bearman, Peter, Parigi, Paolo, 2004. Cloning headless frogs and other important matters: conversation topics and network structure. Soc. Forces 83, 535–
557.
Black, Dan, Gates, Gary, Sanders, Seth, Taylor, Lowell, 2007. The measurement of same-sex unmarried partner couples in the 2000 U.S. Census. CCPR
Working Paper Series CCPR-023-07.
Brief of Amici Curiae Social Science Professors in Support of Defendants. Robicheaux, et al., v. James D. Caldwell, et al. (May 12, 2014) (No. 13–5090).

28
We similarly believe that scholars must maintain equally high standards when assessing studies that reach different conclusions than Regnerus’s.
Reflecting this belief, we have been in the position of (successfully) recommending rejection of such studies that have relied on other data sets for the some of
the same concerns that we have outlined in this paper.
626 S. Cheng, B. Powell / Social Science Research 52 (2015) 615–626

Brief of Amicus Curiae American Sociological Association in Support of Respondent Kristin M. Perry and Respond Edith Schlain Windsor. Dennis
Hollingsworth et al. v Kristin M Perry, et al. and United State v. Edith Schlain Windsor (February 28, 2013) (Nos. 12–144, 12–307).
Chang, Linchiat, Krosnick, Jon A., 2009. National surveys via RDD telephone interviewing versus the internet: comparing sample representativeness and
response quality. Public Opin. Quart. 73, 661–674.
Cheng, Simon, Powell, Brian, 2005. Small samples, big challenges: studying atypical family forms. J. Marriage Fam. 67, 926–935.
Cheng, Simon, Powell, Brian, 2011. Misclassification by whom? A comment on Campbell and Troyer. Am. Sociol. Rev. 76, 347–355.
Cohen, Philip N., 2013. Regnerus Affair Timeline, with Maze. Family Inequality (accessed 15.09.13) <http://familyinequality.wordpress.com/page/3/>.
Davidson, Amy, 2012. A Faulty ‘Gay Parenting’ Study. New Yorker Online (accessed 15.09.13) <http://www.newyorker.com/online/blogs/closeread/2012/
06/a-faulty-gay-parenting-study.html>.
Doan, Long, Loehr, Annalise, Miller, Lisa R., 2014. Formal rights and informal privileges for same-sex couples: evidence from a national survey experiment.
Am. Sociol. Rev. 79, 1172–1195.
Eggebeen, David J., 2012. What can we learn from studies of children raised by gay or lesbian parents. Soc. Sci. Res. 41, 775–778.
Firebaugh, Glenn, 2008. Seven Rules for Social Research. Princeton University Press, Princeton, NJ.
Fischer, Claude S., 2009. The 2004 GSS finding of shrunken social networks: an artifact? Am. Sociol. Rev. 74, 657–669.
Gallagher, Maggie, 2012. Do children with gay parents do just as well? The new social-science debate. National Review Online (accessed 15.09.13) <http://
www.nationalreview.com/corner/302324/do-children-gay-parents-do-just-well-new-social-science-debate-maggie-gallagher>.
Gates, Gary J., Steinberger, Michael D., 2009. Same-sex unmarried partner couples in the American Community Survey: the role of misreporting, miscoding
and misallocation. In: Presented at the Annual Meeting of the Population Association of America. Detroit, MI.
Gates, Gary J. et al, 2012. Letter to the editors and advisory editors of social science research. Soc. Sci. Res. 41, 1350–1351.
Harris, Kathleen Mullan, Udry, J. Richard, 2010. National Longitudinal Study of Adolescent Health (Add Health), 1994–2008: Parental Questionnaire Code
Book. Carolina Population Center, Chapel Hill, NC.
Luscombe, Belinda, 2012. Do Children of Same-Sex Parents Really Fare Worse? Time Magazine Online (accessed 15.09.13) <http://healthland.time.com/
2012/06/11/do-children-of-same-sex-parents-really-fare-worse/#ixzz2ezF1avaO>.
O’Connell, Martin, 2011. Statistics on same-sex couple households from 2010 Census. In: Quarterly Meeting: the Council of Professional Association on
Federal Statistics. Washington, DC.
O’Connell, Martin, Lofquist, Daphne, Simmons, Tavia, Lugaila, Terry, 2010. New estimates of same-sex couple households from the American community
survey. In: Annual meeting of the Population Association of America. Dallas, Texas.
Osborne, Cynthia, 2012. Further comments on the papers by Marks and Regnerus. Soc. Sci. Res. 41, 779–783.
Perrin, Andrew J., Cohen, Philip N., Caren, Neal, 2013. Are children of parents who had same-sex relationships disadvantaged? A scientific evaluation of the
no-differences hypothesis. J. Gay Lesbian Ment. Health 17, 326–336.
Potter, Daniel, 2012. Same-sex parent families and children’s academic achievement. J Marriage Fam. 74, 556–571.
Powell, Brian, Bolzendahl, Catherine, Geist, Claudia, Steelman, Lala Carr, 2010. Counted Out: Same-Sex Relations and Americans’ Definitions of Family.
Russell Sage Foundation, New York, NY.
Powell, Brian, Quadlin, Natasha Yurk, Pizmony-Levy, Oren, 2015. Public opinion, the courts, and same-sex marriage: four lessons learned. Soc. Curr. 2, 3–12.
Regnerus, Mark, 2012a. How different are the adult children of parents who have same-sex relationships? Findings from the new family structures study.
Soc. Sci. Res. 41, 752–770.
Regnerus, Mark, 2012b. The New Family Structures Study. The Population Research Center. The University of Texas at Austin.
Regnerus, Mark, 2012c. Parental same-sex relationships, family instability, and subsequent life outcomes for adult children: answering critics of the new
family structures study with additional analyses. Soc. Sci. Res. 41, 1367–1377.
Rosenfeld, Michael J., 2010. Nontraditional families and childhood progress through school. Demography 47, 755–775.
Rosenfeld, Michael J., 2012. Parental Influence on Children’s Outcomes: Revisiting the Data from the New Family Structure Study. Unpublished Manuscript.
Rosenfeld, Michael J., Thomas, Reuben J., 2012. Searching for a mate: the rise of the internet as a social intermediary. Am. Sociol. Rev. 77, 523–547.
Savin-Williams, Ritch C., Joyner, Kara, 2014. The dubious assessment of gay, lesbian, and bisexual adolescents of Add Health. Arch. Sex. Behav. 43, 413–422.
Schumm, Walter R., 2012. Methodological decisions and the evaluation of possible effects of different family structures on children: the new family
structures survey (NFSS). Soc. Sci. Res. 41, 1357–1366.
Seltzer, Judith A., Lau, Charles Q., Bianchi, Suzanne M., 2012. Doubling up when times are tough: a study of obligations to share a home in response to
economic hardship. Soc. Sci. Res. 41, 1307–1319.
Sherkat, Darren E., 2012. The editorial process and politicized scholarship: Monday morning editorial quarterbacking and a call for scientific vigilance. Soc.
Sci. Res. 41, 1346–1349.
Smith, Christian, 2012. An Academic Auto-da-fé. Chronicle of Higher Education. Opinions and Idea Section (accessed 15.09.13) <http://chronicle.com/
article/An-Academic-Auto-da-F-/133107/>.
Sprigg, Peter, 2012. New Study on Homosexual Parents Tops All Previous Research. Family Research Council (accessed 15.09.13) >http://www.frc.org/
issuebrief/new-study-on-homosexual-parents-tops-all-previous-research>.
Stacey, Judith, Biblarz, Timothy J., 2001. (How) Does the sexual orientation of parents matter? Am. Sociol. Rev. 66, 159–183.
Tourangeau, Karen, Nord, Christina, Le, Thanh, Pollack, Judith M., Atkins-Burnett, Sally, 2006. Early Childhood Longitudinal Study, Kindergarten Class of
1998–99(ECLS-K): Combined User’s Manual for the ECLS-K Fifth-Grade Data Files and Electronic Codebooks (NCES 2006-032). National Center for
Education Statistics, Washington, DC.
von Hippel, Paul T., 2007. Regression with missing Ys: an improved strategy for analyzing multiply imputed data. Sociol. Methodol. 37, 83–117.
Wainright, Jennifer L., Russell, Stephen T., Patterson, Charlotte J., 2004. Psychosocial adjustment, school outcomes, and romantic relationships of adolescents
with same-sex parents. Child Dev. 75, 1886–1898.