Building Status in an Online Community - Gwern.net
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
ORGANIZATION SCIENCE
Articles in Advance, pp. 1–22
http://pubsonline.informs.org/journal/orsc ISSN 1047-7039 (print), ISSN 1526-5455 (online)
Building Status in an Online Community
Inna Smirnova,a,* Markus Reitzig,b Olav Sorensonc
a
School of Information, University of Michigan, Ann Arbor, Michigan 48109; b Department of Accounting, Innovation, and Strategy, University
of Vienna, 1090 Vienna, Austria; c Anderson School of Management, University of California, Los Angeles, Los Angeles, California 90095
*Corresponding author
Contact: [email protected], https://orcid.org/0000-0003-2275-1166 (IS); [email protected],
https://orcid.org/0000-0002-8562-3754 (MR); [email protected], https://orcid.org/0000-0002-0599-6738 (OS)
Received: August 11, 2020 Abstract. We argue that the actions for which actors receive recognition vary as they
Revised: May 7, 2021; September 23, 2021 move up the hierarchy. When actors first enter a community, the community rewards
Accepted: October 27, 2021 them for their easier-to-evaluate contributions to the community. Eventually, however, as
Published Online in Articles in Advance: these actors rise in status, further increases in stature come increasingly from engaging in
February 11, 2022 actions that are more difficult to evaluate or even impossible to judge. These dynamics pro-
https://doi.org/10.1287/orsc.2021.1559 duce a positive feedback loop, in which those who have already been accorded some stat-
ure garner even greater status through quality-ambiguous actions. We present evidence
Copyright: © The Author(s) 2022 from Stack Overflow, an online community, and from two online experiments consistent
with these expected patterns.
Open Access Statement: This work is licensed under a Creative Commons Attribution 4.0 International Li-
cense. You are free to copy, distribute, transmit and adapt this work, but you must attribute this
work as “Organization Science. Copyright © 2022 The Author(s). https://doi.org/10.1287/orsc.2021.
1559, used under a Creative Commons Attribution License: https://creativecommons.org/licenses/
by/4.0/.”
Funding: All authors would like to acknowledge funding from the Austrian Science Fund [Grant P
25768-G16].
Supplemental Material: The online appendix is available at https://doi.org/10.1287/orsc.2021.1559.
Keywords: status attainment • action ambiguity • online communities • stack overflow • experiment
Introduction Lynn et al. 2009), a process that Correll et al. (2017)
Those held in high esteem enjoy numerous advan- have labelled as socially endogenous inference. Scien-
tages. High-status individuals attract more attention tists, for example, assess contributions not only by
(Simcoe and Waguespack 2011, Bowers and Prato reading articles and attending seminars but also by
2018, Reschke et al. 2018), receive outsized credit for paying attention to who else has cited researchers,
their contributions (Kim and King 2014, Waguespack giving deference to them. In evaluating the quality of
and Salomon 2015), and can more readily access a va- a wine, consumers incorporate both their own opin-
riety of resources (Merton 1968, Bol et al. 2018). High- ions about whether the wine tasted good and their be-
status firms can negotiate better terms from buyers liefs about what others thought (Roberts et al. 2011).
and suppliers (Benjamin and Podolny 1999, Hsu 2004, But this explanation for status attainment also poses
Nanda et al. 2020), receive favorable treatment from a puzzle. Status hierarchies often appear steepest and
authorities (McDonnell and King 2018), and can hire the benefits of status most pronounced in settings in
more able employees without offering higher salaries which consumers cannot even determine ex post—
(Bidwell et al. 2015, Tan and Rider 2017). after they have consumed the goods—whether what
How do firms and individuals come to hold high they received had been of high quality (Sauder et al.
status? The most common claim has been that com- 2012, Sorenson 2014, Ertug et al. 2016). Consider man-
munities award status to those who have provided agement consulting or investment banking. Even after
the most value to them, both through commitment to receiving a recommendation, clients have little basis
the community and through high-quality contribu- for assessing whether BCG or Goldman Sachs provid-
tions (e.g., Ridgeway 1981, Podolny and Phillips 1996, ed better advice than they might have received from
Sauder et al. 2012, Hahl and Zuckerman 2014). Differ- some less-celebrated firm. Given that these settings of-
ences in the value of these contributions nevertheless fer little in the way of actual, verifiable information on
become amplified because, in assessing quality, audi- past performance, it would seem that socially endoge-
ences rely not just on their own prior experiences but nous inference has nothing to amplify. How then do
also on the judgments of others (e.g., Gould 2002, status differences arise in these settings?
1
Smirnova, Reitzig, and Sorenson: Building Status in an Online Community 2 Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) One possibility is that initial differences in quality such as artistic expression (McCall 1975, Sgourev and or perceived quality emerge entirely by chance (Gould Althuizen 2014). These dynamics produce a positive 2002, Lynn et al. 2009). In venture capital, for example, feedback loop, in which those who have already re- Nanda et al. (2020) demonstrate that early perfor- ceived some recognition become further distanced mance differences emerge from investing in the right from the rest. place at the right time, something that appears almost We explore this question empirically using data entirely random. These early successes nevertheless from Stack Overflow (SO), an online community for allow these investors to become central players in the seeking and providing coding advice. Online commu- community, as entrepreneurs and other investors in- nities have become increasingly important settings terpret these random successes as signals of quality. for the exchange of information (Hwang et al. 2015, Another possibility, on which we elaborate here, is Botelho 2018)—book reviews on Goodreads, travel that the actions for which actors receive status shift as advice on TripAdvisor, and product reviews on they move up the hierarchy. Both individual and or- Amazon, to name a few. These communities, more- ganizational actors engage in a range of activities that over, usually incorporate an evaluation system— vary in the ease with which others can assess their val- upvoting, likes, useful votes—as a means of motivating ue. Some actions are objectively better or worse. people to provide reviews and of allowing users to sort Others involve a mix of objective elements and those through the information (Constant et al. 1996, Lakhani open to debate. At the extreme, ambiguous actions and von Hippel 2003, Wasko and Faraj 2005). These sys- elude any objective evaluation. A management con- tems create status hierarchies, helping to determine sultant, for example, could provide benchmarking in- who becomes most influential to a wide variety of pur- formation or he or she might proscribe a particular chasing and consumption activities (Bianchi et al. 2012). strategy. With a little research, clients could verify the Understanding the dynamics of status attainment on former. But, for the latter, they have little hope of de- these systems represents an important question in its termining whether another course of action would own right. have been better. But SO also offers some notable advantages for un- People pay attention to different types of actions derstanding the origins of status more broadly: We and evaluate those actions differently depending on can observe community members from the day that the status of the actor performing them. When actors they enter the community, before they have been ac- first enter a community, we argue that community corded any status. By contrast, interactions in person members attend primarily to easier-to-assess actions, almost always occur under the shadow of preexisting awarding status to those who exhibit commitment to status. Even when actors first enter communities, they the community and competence and quality on rela- usually arrive with signals of status from their affilia- tively objective criteria (Ridgeway 1981, Hahl and tions, their ascriptive characteristics, or their strategic Zuckerman 2014). However, for actors who have al- choices (e.g., Ridgeway 1991, Stuart et al. 1999, Phil- ready attained some status, people increasingly pay lips et al. 2013, Askin and Bothner 2016). attention to their harder-to-assess actions, where val- Community members engage in three main activi- ue judgments also become more subjective. Because ties on SO: asking questions, answering them, and members of the community perceive these middle- to commenting on questions and answers. We find that high-status actors as being competent and producing when individuals first enter, asking questions most high-quality outputs, they interpret these quality- strongly predicts their initial movement up the status ambiguous actions as being valuable. These harder-to- hierarchy. As they gain stature, however, further assess actions therefore contribute increasingly to the movement up the ladder depends primarily on an- attainment of further status as actors move up the swering questions and on commenting. Much of the hierarchy. rise from the top 10% to the elite of the elite, the top A statistician beginning his or her career might first 5% and higher, appears to depend on commenting. To gain status by providing accurate answers to objective the extent that these activities range from the value of questions; but as he or she gained standing, audiences questions being easier to evaluate to that of answers would increasingly pay attention to and accord fur- and comments being harder to evaluate, these results ther status to him or her for weighing in on matters are consistent with our expectations. open to debate, such as the right approach to research Although our use of individual-level fixed effects in or the significance of open problems. Early on, judges our analysis of the SO data allows us to reject many al- and audiences similarly accord status to artists and ternative interpretations for these patterns, we cannot musicians in terms of their technical abilities (McCall rule out within-person increases in objective quality 1975). For those performers who have already reached over time, learning, as an alternative explanation. Al- a moderate level of status, however, the receipt of ad- though the observational data do not allow for an ditional status depends on more subjective criteria, easy resolution to this potential confound, we also ran
Smirnova, Reitzig, and Sorenson: Building Status in an Online Community
Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) 3
two online experiments in which we exogenously as- 2001, Bendersky and Shah 2012). Although these as-
signed the status of a contributor to each action (a sessments seem unsurprising in settings where people
question, an answer, or a comment). Those experi- can easily assess the value added by community
ments produced qualitatively consistent results: the members, status orderings curiously emerge even in
status of the questioner did not influence the status settings—such as management consulting and invest-
gains associated with questions, but higher status did ment banking—where it would seem that people have
lead to more positive perceptions of answers and little or no objective basis for evaluating quality (e.g.,
comments. Status also appeared somewhat more im- Podolny 1993, Ertug et al. 2016). In fact, these settings
portant to the evaluation of comments than it did to appear to produce some of the steepest and most resil-
that of answers. We discuss the implications of our re- ient status hierarchies (Sorenson 2014).
sults both for online communities, such as SO, and for We can reconcile this apparent puzzle and more
the emergence and consequences of status hierarchies broadly understand status attainment processes by
in offline communities. recognizing that actors engage in a variety of actions,
some of which allow for relatively easy and objective
Status Attainment valuation, others of which do not. Some actions are
Actors do not claim status. People bestow status on in- easy to evaluate as objectively useful or not. Others
mix elements that are easy to evaluate with others
dividuals and on organizations. They have been
that are open to interpretation. Yet others, ambiguous
thought to do so on the basis of the value that
actions, may elude any objective evaluation. Academ-
they perceive that a particular actor has provided to
ics, for example, inform each other on a range of is-
the community (e.g., Ridgeway 1981, Podolny and
sues, from the factual to the speculative. Investment
Phillips 1996, Hahl and Zuckerman 2014). But actors
bankers similarly advise their clients on many
can influence these conferrals of status through their
decisions, from the quickly verified pricing of public
actions. These perceptions of value presumably come
securities to the harder-to-assess identification and
from a combination of the effort or commitment that
valuation of private firms to acquire.
the actor has demonstrated to the community and the
In updating their beliefs about the competence or
competence or quality of their actions. Because status
quality of actors, we expect that people will attend to
stems in part from these quality perceptions, it simul- different types of actions, or to different dimensions
taneously serves as a signal of quality (Berger et al. of those actions, depending on the status of the actors
1972, Podolny 1993, Podolny and Phillips 1996, Cao performing them.
and Smith 2021).
When actors first enter relationships, groups, and Easy-to-Assess Actions
communities, they begin those interactions without Easy-to-assess actions, by definition, do not require
status. Being without status does not mean being low much time, effort, or expertise to evaluate. This fact
status. Low status would imply that others believed also means that they should generally involve evalua-
the actor incompetent or of poor quality. Being with- tion on objective criteria.
out status instead means that alters simply do not Even complex and seemingly ambiguous actions of-
have any beliefs about what value the actor might ten have such easy-to-evaluate components. In many
provide. settings, for example, simply expending time or effort
People attend to a wide variety of queues as they at- on a community may serve as one of the easiest actions
tempt to situate people in the status hierarchy. Many to evaluate. Time and effort signal commitment to the
of these signals provide only diffuse information. community. Numerous experiments have therefore
They may, for example, infer the status of an individu- found that groups and communities bestow status on
al based on the average status of others with the same those who expend effort on their behalf, particularly
ascriptive characteristics, such as gender (e.g., Ridge- when that effort appears altruistic (e.g., Ridgeway
way 1991). Or they might observe which other actors 1981, Willer 2009, Hahl and Zuckerman 2014).
in the community interact and affiliate with the en- Beyond simply the time involved, many actions
trant, updating their beliefs based on the status of have other easy-to-evaluate components. Clients of
those alters (e.g., Podolny 1993, Stuart et al. 1999, Jen- management consultants and investment bankers, for
sen 2006). Symbolic actions might also provide signals example, can assess the accuracy of the factual infor-
of status (e.g., Askin and Bothner 2016). mation and calculations reported in a proposal or pre-
But community members also begin to form direct sentation. They can also easily evaluate their quality
judgments of commitment, competence, and quality on more superficial features, such as the absence of
and to accord status to entrants to the community on misspellings and grammatical errors.
the basis of the actions of those newcomers (Berger When actors enter a community, community mem-
et al. 1972, Ridgeway 1981, Henrich and Gil-White bers first attend to these easy-to-evaluate actions andSmirnova, Reitzig, and Sorenson: Building Status in an Online Community
4 Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s)
components of actions when assessing and conferring As actors climb the status hierarchy, those interact-
status on the actors. They reward those who spend ing with them then have higher expectations about
time in and on the community. They also hold in their commitment, competence, and the quality of all
higher regard those who perform well on easy-to- of their actions. These expectations rise regardless of
evaluate objective criteria, such as being accurate or whether alters have observed the actors themselves or
technically able. whether they have simply inferred the status of those
Even if the perceived quality of these easy-to- actors based on affiliations or patterns of deference.
evaluate elements does not depend on the status of As the expectations of community members rise, it
the actor producing them, the extent to which they becomes increasingly difficult for actors to exceed
contribute to conferrals of status will. Before people these expectations on easy-to-evaluate actions. They
have strong priors about an actor, observations of ef- have already demonstrated commitment. Their accu-
fort and high quality on these easy-to-evaluate dimen- racy cannot exceed 100%. Easy-to-evaluate actions
sions will lead alters to update their beliefs about the therefore eventually become self-limiting in terms of
competence or quality of the actor to regard them as their further contributions to standing in the commu-
higher status. nity. At some point, the expectations of others for
Given this line of reasoning, we expect the following: commitment and competence, based on the actor’s
Hypothesis 1. At low levels of status, actions that are eas- status, match the actor’s observed easy-to-evaluate
ier to assess contribute to increases in status. performances.
We therefore expect the following:
Even though these easy-to-evaluate elements often
represent but some of the actions or some of the com- Hypothesis 2. As status increases, actions that are easier
ponents of the actions in which actors engage, the to assess contribute less and less to further increases in
evaluation of them influences beliefs about the general status.
quality of the actor for at least two reasons. On the
one hand, much as people use “test” features when Difficult-to-Assess Actions
assigning category membership (Hannan et al. 2007), As actors rise in the status hierarchy, community
people may infer that quality on one type of action members increasingly pay attention to more-difficult-
should correlate positively with quality on other sorts to-assess actions or components of actions. Similar to
of actions. If the analysis in a research paper appears Phillips and Zuckerman’s (2001) argument that actors
solid in technical terms, the reader might place more often require sufficient status to enter the consider-
faith even in the paper’s review of the literature. If a ation set, community members will only exert the ef-
lawyer’s brief gets all of its facts right, then readers fort necessary to assess the quality of these actions if
might give greater credence to any leaps of legal they believe the actor performing them of sufficient
argumentation. ability or quality to justify their time. People therefore
On the other hand, such spillovers in beliefs also allocate more attention to the ideas and outputs of
stem from automatic psychological processes. Peo- higher-status actors (Merton 1968, Simcoe and
ple encode their perceptions of quality as moods or Waguespack 2011). Consistent with the idea that this
emotions (e.g., Swinyard 1993, Danner et al. 2016). stems from expectations regarding the quality of their
But once encoded as a feeling, people can no longer outputs, Cao and Smith (2021) demonstrate that peo-
connect that feeling to a specific component of ple only differentially attend to those of higher status
the product or the service or the producer. The pos- when they believe status serves as a meaningful signal
itive effect created by these perceptions therefore of quality.
creates a general mood that leads alters both to re- At the extreme, the hardest-to-assess actions, am-
call their past experiences with actors more posi- biguous actions or components of actions, defy any
tively and to overestimate the probability of having objective evaluation. Ambiguity does not imply that
good experiences with them in the future (Bower people vary in their preferences, in what they would
1981, Johnson and Tversky 1983, Wright and Bower regard as high value. Almost everyone would agree
1992). that high-quality management consulting should im-
The reverse also holds true. The negative effect as- prove the performance of the firm receiving the ad-
sociated with undesirable experiences can create a vice. Similarly, most would concede that a highly
pall over the actors responsible and everything that competent investment banker should accurately pre-
they do (Johnson and Tversky 1983, Wright and Bow- dict the price that investors would pay for a company
er 1992). When flights have been delayed, for exam- in an initial public offering or acquisition (Podolny
ple, passengers perceive the plane as less clean, the 1993).
seats as less comfortable, and the food as lower quali- The ambiguity rather resides in the near impossibil-
ty (Anderson et al. 2009). ity of assessing whether these objectives have beenSmirnova, Reitzig, and Sorenson: Building Status in an Online Community Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) 5 met. Consider, for example, career advice. Although people tend to pay attention to the information that the person receiving any such advice might perceive would support their existing opinions and to ignore it as useful at the time, any objective evaluation of its that which would contradict them (Wason 1960, Klay- quality can only be made far in the future, after the re- man and Ha 1987). For firms and individuals held in cipient has had the opportunity to act on it. Even high regard then, audiences may selectively attend to then, evaluation would prove difficult. To assess its information, even noise, that affirms their opinions. quality, two types of counterfactuals are needed. First, But this effect probably also stems, in part, from so- what would have happened to the individual in the cially endogenous inference. When faced with uncer- absence of the advice, if they had followed a different tainty about how to evaluate an action, people rely on path? Second, what career advice might another per- the choices and opinions of others—assuming that son have given at the time? Without solid evidence of those individuals have information or insight that both counterfactuals, any evaluation of the quality of they do not—as a means of resolving their uncertainty the advice becomes largely subjective. (Ridgeway and Erickson 2000, Lynn et al. 2009, Cor- Such fundamental ambiguity in evaluation exists for rell et al. 2017). Scientists, for example, assess contri- actions in many settings. Consider a management con- butions not only by reading articles and attending sulting firm giving strategy advice. What recommen- seminars but also by paying attention to who else has dations would another consulting firm have given? cited these researchers. In evaluating the quality of a For an investment bank underwriting a public offer- song, listeners incorporate both their own opinions ing, would another bank have proposed a more accu- and their beliefs about what others thought (Salganik rate initial price? With sufficient time and information, et al. 2006). some of these actions might be open to objective evalu- However, whereas the existing literature on socially ation. Someone could, for example, examine the aver- endogenous inference has generally treated the pro- age level of underpricing across many public offerings cess as a property of the setting (e.g., Podolny 1993, or the average performance of client firms many years Lynn et al. 2009), we would argue that it only occurs down the road. But, in any individual instance and at for certain types of actions and, more crucially, that it the time the actions have been performed, the quality can only begin after actors have been accorded some of these actions remains ambiguous. stature on the basis of easy-to-evaluate actions. The In the face of this ambiguity, we argue that the per- extent to which it operates therefore varies across ac- ceived quality of these actions will depend on the sta- tors within settings and also over time for any given tus of the actor performing them. People will find it actor near impossible to judge the ambiguous actions of Although the perceived quality of these ambiguous those without status, those about whom they have no actions stems from the status of the actors performing priors of competence or quality. People may even them, we believe that they will nevertheless contrib- treat ambiguous actions from low-status actors as con- ute to further gains in the perceived competence or firming evidence of incompetence or of low-quality quality of these actors. If people understood that their performance (Riecken 1958). favorable perceptions of these difficult-to-evaluate ac- But as status rises, the fact that the actor has status tivities reflected the status of the actors, then that un- positively influences the audience’s interpretation of derstanding might inoculate them from using these the ambiguous action (Merton 1968, Correll et al. biased opinions to update their beliefs. However, 2017). Sgourev and Althuizen (2014), for example, viv- whether due to confirmation biases or socially endog- idly recount how the same style inconsistency that enous inference, we suspect that people are either not critics denigrated early in Picasso’s career (before he aware of these biases or underestimate the extent to had status) became seen as evidence of his genius af- which they operate. Status then increases the per- ter he had attained prominence. ceived quality of difficult-to-evaluate actions, which Importantly, in contrast to easy-to-evaluate dimen- leads to further increases in status, creating a virtuous sions on which expectations of quality based on status cycle of positive feedback. will eventually match observed quality, in the absence This line of reasoning leads us to propose the of objective evaluation, perceptions rule uncon- following: strained. Advice from a Goldman Sachs or a Nobel Prize winner almost automatically becomes seen as Hypothesis 3. As status increases, difficult-to-evaluate ac- important and insightful, as high quality. When a tions contribute more and more to further increases in high-status actor weighs in on some topic, audiences status. perceive those opinions as further evidence of the individual’s brilliance. Stack Overflow In part, this effect probably stems from confirma- We investigate the dynamics of status formation on tion bias. When presented with conflicting evidence, Stack Overflow. SO provides a forum in which people
Smirnova, Reitzig, and Sorenson: Building Status in an Online Community 6 Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) can find solutions to their programming problems, best answer given, submitted by “Mark Byers” (again, can help solve others’ problems, and can discuss a see the area just below the box). If one scrolled down range of topics related to computer programming and the screen, one would see the other answers, se- software engineering. quenced in terms of the number of “useful” votes that SO provides an amazing resource. It is the most ac- they had received. The smallest box, box 3, mean- tive online exchange for programming-related infor- while, surrounds one of the comments, offered by mation, with almost 20 times as many questions and “Ankur-m” nearly two years after the question had answers as the next most active exchange. Since its in- originally been posed. ception in 2008, users have posted more than 20 mil- As in many online communities, status plays an im- lion questions and have received more than 30 million portant role here. The platform does not moderate answers to those questions.1 Most questions receive participation and questions, answers, and comments answers in a matter of minutes (Mamykina et al. vary tremendously in their quality. As in other com- 2011). Every month, 50 million unique visitors search munities, the solution to this problem involves a type the site for programming-related information. of crowdsourced quality evaluation. Members of the SO also offers an excellent setting for examining the community can upvote (or downvote) questions, eval- dynamics of status formation. First and foremost, uating them as clear and useful (or not). In Figure 1, most members of the community interact only for example, one can see that the question received 75 through the platform and the platform documents more upvotes than downvotes (see the number be- nearly all of their activity. We therefore have a com- tween triangles to the left of box 1). Members can also plete archival record of the actions that contribute to evaluate answers and comments as useful (or not). status attainment. Second, the fact that few of these The top answer to this question also happened to individuals have prior experience with each other out- have received 75 more upvotes than downvotes (see side the platform means that members join the com- the number between triangles to the left of box 2). munity without any preexisting status. The platform uses community members’ reactions Anyone can join SO. Joining allows a user not just to questions and answers to award points and badges to read the existing discussions but also to contribute to members who provide content. These points and content. Members undoubtedly participate for a varie- badges both provide rewards for contributing and sig- ty of reasons. Some may derive satisfaction from nals to those consuming the content. SO displays contributing to the public good; others may enjoy the them prominently. Consider Figure 1 again. Look at social exchange or the recognition garnered from their the line just below the user names for the people ask- contributions; yet others may see providing advice as ing and answering questions. The first number reports a form of generalized reciprocity for the benefits that the points that the individual has received; the num- they themselves have received (Constant et al. 1996, bers to the left of the colored dots detail the number of Lakhani and von Hippel 2003, Penoyer et al. 2018, badges that the user has received. “Amit Patil,” for ex- Chen et al. 2019). ample, has 708 points and has earned 3 gold badges, We downloaded our data from the archive (https:// 11 silver badges, and 22 bronze badges.2 archive.org/details/stackexchange) that Stack Over- Users can also find out more about any particular view released to the public on March 13, 2017. Our member, in their profile, by clicking on the person’s full data set includes information on all activities on username. Figure 2 provides an example of a profile: the SO platform from its inception, on July 31, 2008, to “nc3b” has been a member for more than nine years our download date, of March 13, 2017. Because of the (though one can also see that the user has not been ac- large size of the archive, our analyses focus on a 1% tive since 2013), asking 19 questions and posting more simple random sample of members (30,418 accounts). than 176 answers. Immediately below the avatar on Each member who had registered on SO before March the left-hand side of the screen, you can see the points 13, 2017, had an equal probability of being included in (10,879) and the number of badges that nc3b has been the sample, regardless of their status, their year of reg- awarded. Based on the tag information reported in istration, and their level of activity. We did, however, the middle of the page, this member appears to have exclude elected moderators from this sample as they expertise primarily in the C programming language. often appear as outliers in their degree of activity on But we have no information about the individual the platform. beyond his or her activity on SO. The line— Figure 1 depicts an example of what one would see “Apparently, this user prefers to keep an air of mys- as a set of actions related to a particular question. Box tery about them.”—represents generic text that SO 1 surrounds the initial question. It includes an expla- displays for all members who have not provided self- nation of the problem and a snippet of code reporting descriptions in their profiles. what the person, “Amit Patil” (see the shaded area Although this example does not include any identi- just below the box), had tried. Box 2 highlights the fying information, some users do provide personal
Smirnova, Reitzig, and Sorenson: Building Status in an Online Community Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) 7 Figure 1. (Color online) Example of a Stack Overflow Question Page Source. https://stackoverflow.com/questions/3361768/copy-data-from-one-column-to-other-column-which-is-in-a-different-table/ 13454906#13454906 (accessed April 30, 2019). information. SO does not maintain official statistics on top scorers). For most individuals, therefore, SO users the proportion of members who have completed their would have little if any information on which to base profiles. We therefore examined two subsets of 100 prior beliefs about their status (cf. Bianchi et al. 2012). members—one selected at random from our sample High-status members also do not appear to differ and a second set of the 100 users who had accumulat- from the average community member on this outside ed the most points—and hand coded their profiles. information. Although nothing requires SO members to choose user names that identify them, 46 of 100 individuals in the random sample and 41 of the top 100 scorers chose Measures user names that resembled a combination of a fore- Dependent Variables. Status has usually been mea- name and a surname. Of those potentially using real sured in one of two ways. The first involves selecting names, fewer than half provided any additional iden- some award, such as the Nobel Prize or an endowed tifying information, such as an employer, in their pro- chair (e.g., Merton 1968, Reschke et al. 2018). This ap- file (19 of 46 of the random sample and 20 of the 41 proach has the advantage of having a high degree of
Smirnova, Reitzig, and Sorenson: Building Status in an Online Community 8 Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) Figure 2. (Color online) Example of a Member Profile Page Source. https://stackoverflow.com/users/226266/nc3b (accessed April 30, 2019). face validity. Few would argue that the Nobel Prize award or penalize another user in six main ways: (1) does not confer prestige on its recipient. But these Upvoting (downvoting) someone else’s question adds prizes and positions also reflect status. People win 5 points to (subtracts 2 points from) that person’s Nobel Prizes and receive chairs because they are al- score. (2) Upvoting another member’s answer adds 10 ready held in high regard. Studies based on this ap- points to (subtracts 2 points from) that member’s proach therefore compare the pinnacle of the prestige score. (3) When the question asker selects an answer hierarchy to the merely elite (Reschke et al. 2018). as the best one offered, the person providing that an- A second approach collects information on patterns swer receives an additional 15 points. (4) A member of deference (e.g., Podolny 1993). Highly cited scien- can also offer a “bounty” on a question. If they choose tists, for example, have higher status on average than to award the bounty to a particular answer, the person those receiving less attention. Our own measures fol- awarding the bounty effectively transfers those points low the logic of this second approach. from their own score to the person receiving the boun- We examine two outcomes. Our first measure ty. (5) If a user proposes an edit to a question, answer, builds off of the score that SO uses to summarize how or comment and the original poster accepts that edit, other members have evaluated the person’s contribu- the person proposing the edit receives 2 points. (6) If a tions. The second measure captures attention, some- person’s post receives six flags identifying it as spam thing strongly correlated with status (Merton 1968, or as being offensive, the person loses 100 points. Simcoe and Waguespack 2011). These reactions account for nearly all points awarded SO scores its members based on how other to SO members.3 members respond to their contributions. Much as Returning to the example in Figure 1, the asker here publishing a paper does not ensure that anyone cites received 375 points for this question based on the re- it, posting a question, an answer, or a comment does actions of other users ( 5 × 75). The person who pro- not guarantee the poster any points. During the peri- vided the answer meanwhile received 765 points for od covered by our data, community members could the combination of the upvotes plus being accepted as
Smirnova, Reitzig, and Sorenson: Building Status in an Online Community
Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) 9
the best answer ( 10 × 75 + 15). But these numbers we scraped the website for profile views every other
represent outliers. The modal question and answer re- week from November 30, 2016, to March 18, 2017 for
ceive no reactions—no upvotes or downvotes—and all SO members, generating eight observations for
therefore do not result in any points being rewarded each member (242,872 user-period records).
to the posters. Members also vary a great deal in their visibility.
The archival data from SO only provided this The average individual had received 24.7 profile
score at the time of downloading, but SO reports the views by March 2017. But the cumulative number of
algorithm that it uses to calculate these scores and profile views to that point ranged from just 1 to more
our data include nearly all of the relevant informa- than 9,000.
tion for this calculation. We therefore used the al- Our dependent variables correlate with each other
gorithm together with the activity information to at 0.75. But each measure has its strengths and weak-
create time-varying imputed evaluation scores. We nesses. The primary weakness of the evaluation score
computed this variable at the user-month level for is that SO has defined the weights for how particular
the period from SO’s inception up until March 2017, reactions contribute to status. Visibility, meanwhile,
giving us a total of 1,420,359 distinct user-period ob- has the advantage of not assuming any weights but
servations. Our manually reconstructed scores for has the disadvantage that members may view profiles
March 2017 correlate to those available from SO at a for reasons not connected to status, introducing noise
level of 0.98.4 into that measure. To the extent that both measures re-
Figure 3 depicts the distribution of these (logged) veal similar patterns, however, it should increase our
evaluation scores in our data in a violin plot. The confidence that the results reflect actual status attain-
width of the violin at each point depicts the propor- ment processes.
tion of the mass of the distribution at that point; the
dot and boxplot down the center represent the mean Independent Variables. Our theory argues that differ-
and interquartile range, respectively. One can clearly ent types of activities contribute to status formation at
see that a large proportion of members register but different points in the process. SO members engage in
then never receive any attention for their activity on three main activities: posting questions, answering
the platform. The average individual received 160.6 them, and commenting.5 Questions seem easiest to
points over our observation window. But this score evaluate. They demonstrate engagement with the
has a very long tail, with one person being awarded community. Users can understand most aspects of
more than 116,000 points. their value simply from reading them.
Our second dependent variable stems from the fact Consider an example. One new user posted a ques-
that with status comes attention. This measure, which tion, “Combining two vectors element-by-element,”
we label as visibility, counts the cumulative profile with the following text: “I have 2 vectors [examples]. I
views—the page depicted in Figure 2—that a member would like to combine them so that the resulting vec-
has received up to a given point of time. SO, again, tor is [example]. I can easily do this with a loop but it
only provides cross-sectional information on this mea- is very slow so can anyone provide a fast way to do
sure. Because we could not reconstruct it retroactively, this?” The title clearly and succinctly describes the is-
sue. Readers can readily assess whether they would
value a resolution to it. To date, it has received eight
Figure 3. (Color online) Distribution of the Logged upvotes, adding 40 points to the evaluation score of
Evaluation Scores the asker.
Evaluating answers, by comparison, requires more
effort. Simply providing an answer demonstrates
commitment to the community. Members devote time
to writing them. Answers to questions often run to
multiple paragraphs and include lines and lines of
code.
In our hand-coded sample, however, upvotes and
downvotes came not from length but from whether
the solution worked. Determining that demands more
effort or expertise. Either the evaluator must have
sufficient experience that they had tried the solution
before or the person must attempt to implement the
advice. Many problems also have multiple solutions.
Determining the best approach might require a great
deal of expertise and may depend on the situation.Smirnova, Reitzig, and Sorenson: Building Status in an Online Community
10 Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s)
Comments, meanwhile, often seem at least as diffi- unfold over time, we want to account for any matura-
cult as answers to evaluate in terms of their value.6 tion effects, such as learning at the level of the individ-
Understanding the value of these comments often re- ual. A user tenure variable, therefore, captures the
quires reading and understanding the entire thread, logged number of days since the user joined the plat-
not just the original question but also the proposed form. All of the models also include a count of the
solutions. Consider some examples. As a response to logged number of “favorite” tags that a user has giv-
a proposed answer, the original asker commented: “I en, for the logged number of times that a user answers
don’t want R to store 50,000 zeros. Rather, I want his or her own question, and for the logged number of
some type of sparse storage within each loop.” Anoth- times a user accepts his or her own answer as being
er user responded to this comment with another com- the best one, even though these represent rather rare
ment, “plenty of results here on sparse matrices,” events and even though they have no mechanical rela-
with links to two additional SO threads. In another tionship to either of our dependent variables. We add
thread, as a comment on a question, one user re- one to all of these counts prior to logging to avoid the
sponded “just count?? the order of the variables is the generation of missing values.
same as the order of the columns.” Effectively, the Our models also include a variety of measures of
user offered a solution to the question without posting activity and objective quality at the question-answer
an official answer. level. The models include the ratio of questions and
To the extent that these contributions range from answers posted by the individual that include snip-
questions being easier to evaluate to answers and pets of code. The models also control for the number
comments being more difficult to evaluate, we there- of bounty points received.
fore expect that the SO community will pay more We also included variables to capture the propor-
attention to questions for posters at lower levels of tion of the users’ questions and answers that had been
status but that they will increasingly attend to answers in popular categories. Because these categories have
and comments as posters rise in the status ranks. more people posting and reading questions, answers,
Our independent variables measure each of these and comments, activity in these domains may attract
activities: more votes and attention. Table 1 reports descriptive
Questioning activity counts the (logged) cumulative statistics for the variables used in our analyses (sum-
number of questions (plus one) an individual has mary statistics for the control variables appear in Ta-
posted on the platform up to a given point of time.7 In
ble A1 in the online appendix).
our full sample, SO members post three to four ques-
tions, on average; but the range runs from 0 to 752.
Estimation Strategy. Our theory argues that the ac-
Answering activity counts the (logged) cumulative
tions that the community values and for which it
number of answers (plus one) an individual has posted
awards status vary as a function of the actor’s current
on the platform up to a given point of time. In our sam-
status. One obvious approach to exploring this idea
ple, the median member posted five answers; but the
would involve regressing the evaluation score in one pe-
range in answering activity runs from 0 to 1,966 posts.
riod on a set of interaction effects between the various
Commenting activity, meanwhile, counts the (logged)
cumulative number of comments (plus one) that a user
Figure 4. (Color online) Logged Average Number of Posts
has submitted.8 The average member in our sample
on Stack Overflow per Quarter per User by Status Level for
posted about 13 comments, but some have posted
All Users Who Reached the 95th Percentile of the Evaluation
more than 5,000. Score Distribution
Figure 4 depicts the natural logs of the quarterly
distributions of activities by status level for all users
who eventually reached the 95th percentile of the
evaluation score distribution. It therefore provides
some sense of how activity on the platform evolves
with status, within user. As users rise through the
ranks, they become more active on the system. But
even those at the lowest levels post comments, and
even those at the highest levels still pose questions.
The most notable shift in behavior appears to be that
users flatten out in their rate of asking questions after
reaching the 75th percentile of the evaluation score
distribution.
We also included control variables to adjust for a
number of user attributes. Because these processesSmirnova, Reitzig, and Sorenson: Building Status in an Online Community
Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) 11
Table 1. Descriptive Statistics for Key Variables various quantile ranges, with the lowest status level
appearing at the top of each grouping and with status
Variables N Mean Std. dev. Min Max
increasing as one moves down.
Evaluation scores 1,420,359 160.6 1,203.25 1 116,114 Consider first the effects of asking questions. At the
Visibility 242,872 24.74 145.05 1 9,037 lowest levels of status, nearly all gains in status ap-
Questioning activity 1,420,359 3.6 12.95 0 752
pear associated with asking questions (consistent with
Answering activity 1,420,359 5.03 34.08 0 1,966
Commenting activity 1,420,359 12.85 75.57 0 5,027 Hypothesis 1). The coefficient implies that a one-unit
increase in questioning activity predicts a 0.57% in-
Note. Std. dev., standard deviation.
crease in a person’s evaluation score (p < 0.001). Ask-
actions and the evaluation score in the prior period. But ing questions continues to predict gains in status all
the way up to the 95th percentile of the evaluation
that approach has the disadvantage of imposing a func-
score distribution. The apparent value of asking ques-
tional form on how attention to types of action change
tions in terms of additional status gains, however, de-
with status.
clines rapidly as users move up the distribution of
To allow the relationship between the reactions to
evaluation scores. Based on only questioning activity,
actions and status to vary flexibly, we used a modified
an individual entering the platform could move into
version of quantile regression. We began by specify-
the top half of the evaluation score distribution by
ing five quantile intervals—0%–50%, 50%–75%,
posting 21 questions with average reactions. Moving
75%–90%, 90%–95%, and above 95%—for each of our
from the 50th percentile to the 75th percentile would
dependent variables. For the evaluation score, the cut
require another 35 average-reaction questions. At the
points fall at 8, 37, 187, and 506 points; for visibility,
very highest levels, in the top 5%, posting questions
the boundaries between the quantile intervals come at
actually has a negative association with the evaluation
4, 10, 35, and 79 page-views. Although our definition
score. At that level, questions disappoint. Consistent
of these boundaries stem from the distributions of with Hypothesis 2, then, the value of easy-to-evaluate
these variables at the end of our period, the inclusion activities for status gains exhibits diminishing margin-
of period fixed effects should account for the fact that al returns.
the underlying distribution evolves over time.9 Answering activity, by contrast, does little for status
We then estimated coefficients for our independent at the very lowest levels (we cannot even reject the
variables within each of these quantile intervals using null hypothesis that it has no effect). At middle and
a series of models with user-level fixed effects.10 These high levels of status, however, answering questions
fixed effects should capture time-invariant unob- begins to predict increases in status, consistent with
served differences across community members, such Hypothesis 3.
as gender, native language, and formal education (as Commenting activity, perhaps the most difficult to
well as ancillary information available on profiles). evaluate of the three actions, also has no significant ef-
Our estimates therefore reflect the changing reactions fect on status gains for those in the bottom half of the
of the SO community to various types of actions with- status distribution. At moderate to high levels of sta-
in a specific individual within a particular range of tus, posting comments has a more pronounced associ-
the status distribution. ation with status attainment (β3 0.16 and 0.20 at the
SO user-period provides the unit of analysis in 75th and 90th percentiles, p < 0.001). Consistent with
these regressions. Standard errors have been clustered Hypothesis 3, the relationship between commenting
at the user level.11 We estimate our regression models and status gains becomes ever stronger as individuals
according to: climb the status hierarchy.
Evaluation scorei,t or Visibilityi,t α + β1 Figure 6 again depicts the main results, this time us-
ing visibility as the measure of member status (for the
× questioning activityi,t + β2 × answering activityi,t
corresponding table, see Table 3). The results largely
+ β3 × commenting activityi,t + γ1 × user controlsi,t mirror those in Figure 5. At low levels of status, the
+ time dummiest + εi,t , (1) community-wide interest correlates primarily with
questioning activity, consistent with Hypothesis 1.
where i refers to the individual user and t to the peri-
Over this range of status, a one-unit increase in the
od (either a month or two-week interval).
number of questions predicts a roughly 0.25% in-
crease in profile views (p < 0.001). At these modest
Results levels of status, however, providing answers and
Figure 5 plots the coefficient estimates for the relation- comments adds little to visibility within the SO
ship between actions and evaluation scores. (Table 2 community.
reports the results in table form). The plots group the As status increases (75%–90% and 90%–95% quan-
coefficients for a particular type of activity across the tile intervals), however, more difficult-to-evaluateSmirnova, Reitzig, and Sorenson: Building Status in an Online Community 12 Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) Figure 5. (Color online) Within-Quantile Coefficient Estimates for the Relationship Between Various User Actions on Stack Overflow and Their Evaluation Scores actions become the more powerful predictors of fur- be elite may have changed over time. Although this ther increases in community-wide attention. Asking smaller subsample produces noisier estimates (see Ta- questions becomes less important and does not even ble A2 in the online appendix), the point estimates fol- differ significantly from zero once users pass the 90th low a similar pattern to that found in the full sample. percentile threshold (p > 0.2). Answering questions also adds little to further increases in visibility at the highest levels of the status distribution (β2 0.04 at Experiments the 95th percentile, p 0.027). Only posting comments The individual-level fixed effects allow us to rule out a continues to correspond to increasing attention at the wide range of alternative interpretations. For example, highest levels. Moving from the 90th percentile to the if members revealed their gender or nationality 95th percentile of visibility would require millions of through their user names or on their profiles and those questions generating average reactions, hundreds of characteristics led to differences in status, the fixed ef- average-reaction answers, or roughly five average- fects would absorb those effects. But one important po- reaction comments. tential confound remains. Members might get better at We estimated a number of additional models to as- these activities over time, meaning that the quality of sess the extent to which our results might reflect some their answers and comments might rise in tandem sample selection or estimation choice. We first restrict- with their score and their visibility. Solving this simul- ed the analysis to those who eventually achieved high taneity problem would either require exogenous varia- status (the 95th percentile). In other words, this re- tion in status or accurate measures of the objective gression estimates what accounted for the status gains components of question, answer, and comment quali- of the elite users as they moved from having no status ty. Because neither of these solutions seemed feasible to being in the top status category. Figure 7 depicts in the archival data, we developed a pair of online ex- the coefficient estimates for the same models within periments as a second test of our predictions. this subset of users (Table 4 reports the results in table We had a panel of Python experts create realistic form). As one can see, the patterns appear the same threads of questions, answers, and comments. We as- even among this set of elite users. sembled these threads using the same formatting as We next restricted the analysis to those who joined an SO thread, so that they would appear almost as the platform during its first full year of operations screen shots from the SO website (see Figures A3 and (July 31, 2008 to July 31, 2009). This subset addresses A4 in the online appendix). However, as opposed to two potential issues. First, it accounts for the fact that an actual thread, the experiment allowed us to assign the platform and the nature of contributions to it randomly the status of the users associated with the might have evolved over time. Second, it addresses question, answers, and comments in each thread. We the possibility that the definition of what it means to ran two online experiments.12 The first tested our
Smirnova, Reitzig, and Sorenson: Building Status in an Online Community
Organization Science, Articles in Advance, pp. 1–22, © 2022 The Author(s) 13
Table 2. Within-Quantile OLS User Fixed-Effects Regressions for the Relationship Between Various User Actions on Stack
Overflow and Their Evaluation Scores
Variables (1) (2) (3) (4) (5)
0%–50% 50%–75% 75%–90% 90%–95% Above 95%
log(Questioning activity + 1) 0.57*** 0.36*** 0.16*** 0.05* −0.06*
(0.024) (0.015) (0.017) (0.022) (0.024)
log(Answering activity + 1) −0.00 0.18*** 0.21*** 0.17*** 0.24***
(0.018) (0.009) (0.008) (0.019) (0.032)
log(Commenting activity + 1) −0.00 0.08*** 0.16*** 0.20*** 0.31***
(0.020) (0.014) (0.012) (0.020) (0.049)
log(User tenure + 1) 0.07*** 0.25*** 0.43*** 0.29*** 0.59***
(0.011) (0.016) (0.040) (0.080) (0.122)
log(Answering activity to own questions + 1) −0.03** −0.01 0.02*** 0.01* 0.02***
(0.011) (0.004) (0.004) (0.004) (0.005)
log(Number of “favorite” votes given + 1) 0.00 0.01** 0.01*** 0.01*** 0.01***
(0.003) (0.003) (0.002) (0.002) (0.003)
log(Number of “favorite” votes given to self + 1) −0.00 −0.00 −0.00 0.00** 0.00*
(0.003) (0.002) (0.001) (0.001) (0.002)
log(Accepting own answers as best + 1) 0.00 0.00* −0.00 −0.01** −0.01**
(0.005) (0.002) (0.002) (0.003) (0.003)
Ratio of posed questions with snippets of code 0.08*** −0.02* −0.04*** −0.01 0.00
(0.012) (0.009) (0.010) (0.014) (0.018)
Ratio of given answers with snippets of code −0.00 0.01** 0.01 0.00 0.02
(0.006) (0.004) (0.005) (0.011) (0.032)
Ratio of posed popular questions 0.00 −0.01† −0.01 −0.02* 0.01
(0.009) (0.007) (0.009) (0.011) (0.014)
Ratio of given answers to popular questions −0.00 −0.01 −0.01** 0.00 −0.06*
(0.006) (0.004) (0.005) (0.009) (0.030)
log(Number of bounty points received + 1) 0.01*** 0.00** 0.00*** 0.00† 0.00**
(0.000) (0.001) (0.000) (0.000) (0.000)
Time dummies (month) YES YES YES YES YES
N 689,765 373,569 214,686 71,306 71,033
Notes. OLS user fixed-effects panel regressions where robust standard errors clustered at the user level are reported in parentheses. All
independent variables are normalized. OLS, ordinary least squares.
†p < 0.10; *p < 0.05; **p < 0.01; ***p < 0.001 (two-tailed tests).
arguments for questions and the second our predic- question asker to one of three levels: low status (a score
tions related to answers and comments. of 6 points), medium status (158 points), or high status
(1,714 points). We selected these values based on the
Experiment 1: Methods distribution of the SO scores in our field data, with low
Participants. We recruited 90 English-speaking partic- status being a below-median value, medium status fall-
ipants, who had prior knowledge of Python, through ing in the 50th-to-75th percentile interval, and high sta-
Amazon’s Mechanical Turk (MTurk) online plat- tus being in the top 5% of the SO score distribution.
form.13 MTurk provides a diverse participant pool for Each participant read six fictitious threads (presented
academic research, one demographically similar to the in random order), so each of the 18 experimental condi-
general population (Buhrmester et al. 2011, Chandler tions of our 6 (six question threads) × 3 (status of the
and Shapiro 2016). To ensure that our participants question asker: low, medium, or high) design appears 30
had the relevant expertise to evaluate the questions, times in our data. Two threads involved simple
answers, and comments, we screened potential partic- beginner-level Python questions; two touched on more
ipants for their prior knowledge of the Python pro- intermediate issues; and two concerned advanced topics.
gramming language. We embedded this screening Out of each pair, one question included a snippet of
question in a set of questions about their experience code and the other did not. Participants then evaluated
with several programming languages to reduce the each of the six questions by giving them a “downvote,”
likelihood that participants might not answer truthful- a “no vote,” or an “upvote,” mirroring the SO setting.
ly (see Figure A1 in the online appendix).14
Experiment 2: Methods
Design. We used a between-subjects design with Participants. For the second experiment, we recruited
three different conditions per thread, in which we ex- a second set of English-speaking participants with pri-
perimentally manipulated the status level of the or knowledge of Python on the MTurk platform (270You can also read
