This is a huge brain dump of my notes and thoughts around digital communities and general group social dynamics.

Here's a quick tour of the sections:


The problems I'm particularly interested in


The ideal vision for the project's outcome


Why such a project is needed


Guidelines to keep in mind

Useful Concepts

Helpful tools, frameworks, and ideas that others have come up with for understanding digital communities

Threat Modeling / How Communities Fail

A catalog of where digital community in its current state falls short

Existing Approaches

An overview of what other organizations and companies have tried or are trying to solve some of the same problems

Algorithmic Approaches

A survey of the computational techniques attempting to solve some of these problems

Possible Features

A potpourri of features and design elements which could be useful


A smorgasbord of other useful but not directly relevant social and behavioral theories which have helped shape my thinking around these problems



Building some sustainable form of Ta-Nehisi Coates' "Horde":

So how did Coates foster a comment section in which—wonder of wonders—intelligent adults thoughtfully share ideas and knowledge, and where trolling, rudeness and bad faith aren’t tolerated?

Coates frequently highlighted particular insights, pulling them into blog posts of their own, and, as he’d promised from the start, he increasingly turned to the commenters for information about areas he didn’t know well—Fannie Mae and Freddie Mac, for instance. Blog posts sometimes consisted entirely of a request for input in the comments.

The rules of the space were evolving, too. Trolling was verboten, and commenters were discouraged from responding to people who were obviously looking to provoke; he didn’t want the discussion devolving into an un-resolvable argument. (The blog’s homegrown etiquette was later codified in what became known as the “dinner party rules.”) As the blog grew in popularity, Coates’ reminders about bad behavior became more frequent; repeat offenders were banned, and sometimes, he had to resort to closing comments on a post entirely. (worth reading the whole thing)

Or something along the lines of the community Annemarie Dooling mentions in Forcing Commenters to Use Real Names Won’t Root Out the Trolls:

I engage almost daily with a group of science enthusiasts on who look forward to a weekly open thread on climate change, an often distraught subject on forums. Though I cannot verify the names of these community members, their understanding of climate issues is apparent through the numerous emails, shared links, and friendships they have formed on the site. As a group they alert moderators when there is an attack, they suggest editorial topics, and share links with other friends in the field. They consider it a personal mission to stop conversations on climate change from derailing into political debate, and so, spend hours and days, in some cases, returning to the same discussion thread, flagging and responding, and keeping the conversation moving.

When I’m managing communities, particularly ones around controversial news like my former work at The Huffington Post or current work with, I’m eager to hear from commenters motivated by a deep interest in the subject matter. These readers feel a personal ownership to the work they are reading, draw in others, and move them to action. They teach new members how to integrate and share the rules. They email about typos, and become acquainted with columnists. The most involved community members defend writers from casual, careless attacks. They spend their time crafting perfect responses, and often save these responses off-site, feeling immense pride at their contributions.

And they do all of this from behind pseudonyms they have created to fit their new social family.


Seeing Like a State provides some cautionary tales - large-scale social engineering projects tend to fail because they need to structure environments and systems in such a way that they can be managed on some level of abstraction, but often this loses out on important contextual nuances and details which are the bedrock of many of the desirable emergent properties of such systems and environments.

Also, an acknowledgement of this being a social problem just as much as (or more so than) it is a technological one:

So perhaps the solution isn't just about making the best, newest tool possible. It's not about a better algorithm, filter or team of moderators.

Maybe the way to encourage intelligent, engaging and important conversation is as simple as creating a world where we actually value the things that make intelligent, engaging and important conversation. You know, such as education, manners and an appreciation for empathy. Things we used to value that seem to be in increasingly short supply. - ISO civility in online comments, Joshua Topolsky.


These are not new problems:

Most of what you hear on CB radio is either tedious (truck drivers warning one another about speed traps) or banal (schoolgirls exchanging notes on homework), but at its occasional--and illegal--worst it sinks a pipeline to the depths of the American unconscious. Your ears are assaulted by the sound of racism at its most rampant, and by masturbation fantasies that are the aural equivalent of rape. The sleep of reason, to quote Goya's phrase, brings forth monsters, and the anonymity of CB encourages the monsters to emerge. Not often, of course; but when they do, CB radio becomes the dark underside of a TV talk show. - Fifteen Years of the Salto Mortale by Kenneth Tynan, February 20, 1978

The following paper surveys how discussion networks for individuals have shrunk from 1985 to 2004. From the abstract:

Discussion networks are smaller in 2004 than in 1985. The number of people saying there is no one with whom they discuss important matters nearly tripled. The mean network size decreases by about a third (one confidant), from 2.94 in 1985 to 2.08 in 2004. The modal respondent now reports having no confidant; the modal respondent in 1985 had three confidants. Both kin and non-kin confidants were lost in the past two decades, but the greater decrease of non-kin ties leads to more confidant networks centered on spouses and parents, with fewer contacts through voluntary associations and neighborhoods. Most people have densely interconnected confidants similar to them. Some changes reflect the changing demographics of the U.S. population. Educational heterogeneity of social ties has decreased, racial heterogeneity has increased. The data may overestimate the number of social isolates, but these shrinking networks reflect an important social change in America. - Social Isolation in America: Changes in Core Discussion Networks over Two Decades, Miller McPherson, Lynn Smith-Lovin, Matthew E. Brashears

Comments influence perception of content:

We asked 1,183 participants to carefully read a news post on a fictitious blog, explaining the potential risks and benefits of a new technology product called nanosilver. These infinitesimal silver particles, tinier than 100-billionths of a meter in any dimension, have several potential benefits (like antibacterial properties) and risks (like water contamination), the online article reported.

Then we had participants read comments on the post, supposedly from other readers, and respond to questions regarding the content of the article itself.

Half of our sample was exposed to civil reader comments and the other half to rude ones — though the actual content, length and intensity of the comments, which varied from being supportive of the new technology to being wary of the risks, were consistent across both groups. The only difference was that the rude ones contained epithets or curse words, as in: “If you don’t see the benefits of using nanotechnology in these kinds of products, you’re an idiot” and “You’re stupid if you’re not thinking of the risks for the fish and other plants and animals in water tainted with silver.”

The results were both surprising and disturbing. Uncivil comments not only polarized readers, but they often changed a participant’s interpretation of the news story itself.

In the civil group, those who initially did or did not support the technology — whom we identified with preliminary survey questions — continued to feel the same way after reading the comments. Those exposed to rude comments, however, ended up with a much more polarized understanding of the risks connected with the technology.

Simply including an ad hominem attack in a reader comment was enough to make study participants think the downside of the reported technology was greater than they’d previously thought. - This Story Stinks, Dominique Brossard and Dietram A. Scheufele

The “Nasty Effect:” Online Incivility and Risk Perceptions of Emerging Technologies Ashley A. Anderson, Dominique Brossard, Dietram A. Scheufele, Michael A. Xenos and Peter Ladwig a more recent study, Ionnis Kareklas, Darrel D. Muehling, and TJ Weber, all of Washington State University, found that the comments on a public-service announcement about vaccination affected readers' attitudes as strongly as the P.S.A. itself did. When commenters were identified by their level of expertise with the subject (i.e. as doctors), their comments were more influential than the P.S.A.s.

Online readers may put a lot of stock in comments because they view commenters "as kind of similar to themselves," said Mr. Weber -- "they're reading the same thing, commenting on the same thing." And, he added, many readers, especially those who are less Internet-savvy, assume commenters "know something about the subject, because otherwise they wouldn't be commenting on it." The mere act of commenting, then, can confer an unearned aura of credibility. - What Your Online Comments Say About You, Anna North

Create less toxic and more inclusive spaces:

According to a 2005 report by the Pew Research Center, which has been tracking the online lives of Americans for more than a decade, women and men have been logging on in equal numbers since 2000, but the vilest communications are still disproportionately lobbed at women. We are more likely to report being stalked and harassed on the Internet--of the 3,787 people who reported harassing incidents from 2000 to 2012 to the volunteer organizationWorking to Halt Online Abuse, 72.5 percent were female. Sometimes, the abuse can get physical: A Pew survey reported that five percent of women who used the Internet said "something happened online" that led them into "physical danger." And it starts young: Teenage girls are significantly more likely to be cyberbullied than boys. Just appearing as a woman online, it seems, can be enough to inspire abuse. In 2006, researchers from the University of Maryland set up a bunch of fake online accounts and then dispatched them into chat rooms. Accounts with feminine usernames incurred an average of 100 sexually explicit or threatening messages a day. Masculine names received 3.7.

...But no matter how hard we attempt to ignore it, this type of gendered harassment--and the sheer volume of it--has severe implications for women's status on the Internet. Threats of rape, death, and stalking can overpower our emotional bandwidth, take up our time, and cost us money through legal fees, online protection services, and missed wages. I've spent countless hours over the past four years logging the online activity of one particularly committed cyberstalker, just in case. And as the Internet becomes increasingly central to the human experience, the ability of women to live and work freely online will be shaped, and too often limited, by the technology companies that host these threats, the constellation of local and federal law enforcement officers who investigate them, and the popular commentators who dismiss them--all arenas that remain dominated by men, many of whom have little personal understanding of what women face online every day. - Why Women Aren't Welcome on the Internet, Amanda Hess

The public sphere:

[Habermas'] public sphere was a space within which people of varying backgrounds could come together to discuss the issues, problems, and culture of the commonweal. It was a space for reason and public criticality. But, significantly, it was also a place in which bourgeois and aristocrats came together as if they did not have social class differences and therefore different personal interests in the public problems under debate. Their ability to come together as if they did not have class or social interests was premised on the exclusion of the vast majority of society: women, workers, peasants, conservative nobles, slaves, etc. The pre-September 1993 Usenet can be seen as such a public sphere, before the baptism of the lower classes. Sixteen years hence, the 'as if' problem still remains: how do we organize ourselves civilly if we let just anybody join in? - Attacked from Within, anaesthetica


Support pseudonymity

The people who most heavily rely on pseudonyms in online spaces are those who are most marginalized by systems of power. "Real names" policies aren't empowering; they're an authoritarian assertion of power over vulnerable people.

... Likewise, the issue of reputation must be turned on its head when thinking about marginalized people. Folks point to the issue of people using pseudonyms to obscure their identity and, in theory, "protect" their reputation. The assumption baked into this is that the observer is qualified to actually assess someone's reputation. All too often, and especially with marginalized people, the observer takes someone out of context and judges them inappropriately based on what they get online. Let me explain this in a concrete example that many of you have heard before. Years ago, I received a phone call from an Ivy League college admissions officer who wanted to accept a young black man from South Central in LA into their college; the student had written an application about how he wanted to leave behind the gang-ridden community he came from, but the admissions officers had found his MySpace which was filled with gang insignia. The question that was asked of me was "Why would he lie to us when we can tell the truth online?" Knowing that community, I was fairly certain that he was being honest with the college; he was also doing what it took to keep himself alive in his community. If he had used a pseudonym, the college wouldn't have been able to get data out of context about him and inappropriately judge him. But they didn't. They thought that their frame mattered most. I really hope that he got into that school.- danah boyd

Quotes from this survey.

Anonymity on the internet allows people to set aside some aspects of their identity in order to safely express others. - Deviant Behavior in Online Multimedia Communities and Strategies for Managing it, John Suler

From Annemarie Dooling in Forcing Commenters to Use Real Names Won’t Root Out the Trolls (emphasis mine):

The thought process behind non-anonymity is simple, in that anyone who has their identity attached to their comments will be more careful about what they say in a digital forum because it can be traced back to their family and career. But to believe that a system of name verification would deter uncivil discourse, we’d have to believe that all off-color comments are the results of malicious intent, that is, comments specifically for the purpose of aggravation, to cause harm or instill fear. Purposefully hurtful comments would be embarrassing or harmful to attach to your name, the opinions you want to hide from your family and job. But, the truth is that many vitriolic comments come from readers who are proud to associate these views with their identity.

...To enable real names for all commenters, you leave your civil readers open to cyber-bullying on multiple outlets...

...When I’m managing communities, particularly ones around controversial news like my former work at The Huffington Post or current work with, I’m eager to hear from commenters motivated by a deep interest in the subject matter. These readers feel a personal ownership to the work they are reading, draw in others, and move them to action. They teach new members how to integrate and share the rules. They email about typos, and become acquainted with columnists. The most involved community members defend writers from casual, careless attacks. They spend their time crafting perfect responses, and often save these responses off-site, feeling immense pride at their contributions.

And they do all of this from behind pseudonyms they have created to fit their new social family.

Knowledge Verification

From Annemarie Dooling in Forcing Commenters to Use Real Names Won’t Root Out the Trolls:

Our fear of anonymity is an extension of our fear of the unknown. Without a recognizable name, these commenters could be anyone! But if we could associate an internet history, even a brief one, to that handle, the human connection becomes instantly apparent. A great example of this is Disqus and its universal login, which creates a history of comments (and flags) across all sites.

Likewise, /r/science, at reddit, stumbling over the spread of misinformation, decided to handle knowledge verification on their own. While anyone can comment, the group elevates scientists and researchers who can create a more thoughtful discussion from behind a pseudonym. To be eligible, a member must have a degree in one of the fields related to subreddit discussion, including astronomy, biology and the social sciences, and message the moderators with “a photo of your diploma or course registration, a business card, a verifiable email address,” or other proof of your relationship to the subject matter. All proof of verification is deleted immediately and never made public.

From Judith S. Donath in We Need Online Alter Egos Now More Than Ever:

Insisting that people use their real names online to prevent trolling and ensure civility ignores the fact that using real names online is quite different than using them in person. In the physical world, space and time separate facets of our lives, providing everyday privacy. Even though you use your real name in conversations you have in person with your podiatrist or pastor, those conversations and opinions are not accessible to your co-workers and neighbors. Online, however, the product review you generously provided for an underarm deodorant or for books about coping with binge eating or bed-wetting, will, if written under your real name, be part of your online portrait, what your neighbors, kids and random strangers see about you. Online, words persist forever, in vast searchable databases. Anything you say or do using your real name is permanently attached to it.

...A persistent pseudonym establishes a local identity: you always use it on a certain site or sites, and you build up a history and reputation under that name. You might use one pseudonym to write all sorts of product and service reviews, another in a support group for a personal health issue, and use your real name in discussions on professional forums and to comment on news stories.

The key to making pseudonymous participation productive is to inspire people to care about the impression they are making on others. In physical environments, the body anchors identity; online, one’s history of contributions and interactions functions as one’s “body”, but it can be difficult to see. We can fix this by designing visualizations – data portraits – that make identities based on words and data vivid and easily perceived. Data portraits encapsulate each person’s history and reputation within a community, and thus encourage people to take responsibility for their words, inhibiting bad behavior. At the same time, they can be pseudonymous, giving people the freedom to discuss things they would be reticent to do under their real name.

...For anyone in a vulnerable position – people seeking a job, people whose beliefs are at odds with their neighbors or co-workers – the ability to participate in such discussions depends, effectively, on being able to do so pseudonymously.

...Insistence on real names also prevents people from seeking support for personal problems.

...Online, using pseudonyms is actually more like our ordinary face-to-face experience – and it is essential for managing the impression we make. Face to face, we develop relationships in separate contexts — and the things we talk about, the jokes we make, the secrets we reveal – vary tremendously. ... You present yourself differently to your neighbor, lawyer, teacher, children, grandmother — you use different words and talk about different things. This is not a lack of integrity, but a feature of being an adaptable person in multiple social contexts, understanding the varied mores of the different situations. Pseudonyms allow us to maintain such separate contexts online. Furthermore, discussions among people using pseudonyms may be more interesting. Uncoupled from real world identity, people are quicker to talk about personal subjects, creating a rapid sense of intimacy and closeness. Forced to use real names, many people, aware of the privacy issues raised by this policy, choose to say little, their contributions politely, innocuously stilted.

...A pseudonym without history is just anonymity with a temporary name. If I sign a comment as “Alyce” or “Dude69”, and this name does not connect with anything else I have done, the pseudonym is meaningless (other than the cue it provides about my taste and aspirations).

A basic – and useful – way to incorporate history is to have pseudonymous profiles that show all of a user’s contributions.

Also, a fantastic point which illustrates the importance of negotiating this problem: we are bringing the digital problem of identity into the physical world (when it should be the other way around - bringing the desirable properties/limitations of physical world identity into the digital world):

Soon, however, the physical world will lose its local privacy. The coming ubiquity of cameras everywhere combined with face recognition means that the taken-for-granted ability to facet our lives in the physical world may be coming to an end. A glimpse of your face, a quick search and I’ll have access to your words – and your appearance in other cameras, from other times and places. And faces are harder to disguise than names. (Perhaps we will face this future wearing masks, the equivalent of the pseudonym for the physical world).


Some of our earlier data showed us that offensive players can be rehabilitated and that they manage to modify their communication so that they don't cause negative interactions. To be able to do that though, they need to know when they're producing negative interactions, and the communication ban system lets them know this. Our data shows this is working exactly as we hoped - many players banned eventually reach a ban free communication style, and the percentage of players being reported for communication bans is dropping over time. ... 60% of players who receive bans go on to modify their behavior and don't receive further bans. Dota 2 Communication Reports

The Principle of Charity & Assume Good Faith

In philosophy and rhetoric, the principle of charity requires interpreting a speaker's statements to be rational and, in the case of any argument, considering its best, strongest possible interpretation. In its narrowest sense, the goal of this methodological principle is to avoid attributing irrationality, logical fallacies or falsehoods to the others' statements, when a coherent, rational interpretation of the statements is available. According to Simon Blackburn "it constrains the interpreter to maximize the truth or rationality in the subject's sayings." - Wikipedia

Assuming good faith is a fundamental principle on Wikipedia. It is the assumption that editors' edits and comments are made in good faith. Most people try to help the project, not hurt it. If this were untrue, a project like Wikipedia would be doomed from the beginning. This guideline does not require that editors continue to assume good faith in the presence of obvious evidence to the contrary (vandalism). Assuming good faith does not prohibit discussion and criticism. Rather, editors should not attribute the actions being criticized to malice unless there is specific evidence of malice. - Wikipedia

General Guidelines

Prescribed by anaesthetica in Attacked from Within:

More from further on in the essay:

Focus on discussion, not on the comment or the user

If the point is to encourage constructive discourse, focusing on the comment as the object of study does not make sense, since the unit of discourse is the relationship between comments, just as focusing on the user does not make sense because discourse is equally about the relationship between users.

anaesthetica, in Attacked from Within, suggests looking at dyads of comments as the unit of analysis:

We can take a two comment dyad as an example and apply an AND logic to the pair of comments' worth (as judged by both passive and user moderation):

They make a good comparison to PageRank:

Just as hyperlinks between web pages express a relationship of value, as Sergey Brin and Larry Page realized by 1998, so too do replies and moderation create a network of interlinked users. ... A modified PageRank algorithm could take into account the positive and negative links between users, establishing overall assessments of users useful for distinguishing malicious users from normal users and for dispensing selective incentives to users producing valued contributions.

Keep barriers low, allow for reputation-building

Keep barriers as low as possible but have a system of reputation-building, where a user's visibility is affected by their participation and reputation in the site, such that users reputation corresponds with a users' investment in the community; someone who is more deeply invested will have greater visibility. But the possibility should be open that a particularly insightful new user can still obtain that visibility if the community (through some mechanism) feels it's appropriate.

Useful Concepts

Projections and Transference

While many people are convinced that how they read an email is the only way it can be read, the truth is, how we read a text, or view a work of art, often says more about ourselves than it does about the message or the messenger.

All of our communications, online and in real-time, are filled with projections. We perceive the world through our expectations, needs, desires, fantasies, and feelings, and we project those onto other people. For example, if we expect people to be critical of us, we perceive other people's communication as being critical - it sounds critical to us even though it may not be. We do the same thing online; in fact we are more likely to project when we are online precisely because we don't have the visual or auditory cues to guide us in our interpretations. How we hear an email or post is how we hear it in our own heads, which may or may not reflect the tone or attitude of the sender.

We usually can't know from an email or post alone whether someone is shouting, using a criticizing tone, or speaking kindly. Unless the tone is clearly and carefully communicated by the messenger, and/or we are very skilled at understanding text and human communication, we most likely hear the voice we hear, or create in our head and react to that. This is one of the reasons why controversial or potentially conflictual issues are best dealt with by using great care and explicit expressions of our tone, meaning, and intent. - Conflict in Cyberspace: How to Resolve Conflict Online, Kali Munro, M.Ed.

Behavioral Profiles

There are a lot of behavior taxonomies out there; this is a good starting point (these are taken straight from the page cited below):

Core participants

There are usually as small group of people who quickly adapt to online interaction and provide a large proportion of an online group's activity. Some speculate that 10% of the membership make up 90% of the community activity. These individuals visit frequently and post often. They are important members. On the flip side, be careful that they do not dominate and make it hard for less active folks to participate


Readers or Lurkers are the unseen forces that DO affect a community. Community owners estimate that there are approximately 10 to 100 readers per active poster. They represent a combination of people new to the community, those not yet comfortable in posting, people who will only read and never post, and people who come in and then drift away without engaging. This group represents a huge pool of potential active members.


People who post frequently influence the pace of an online interaction space and can, unknowingly and unintentionally, dominate that space making it harder for others to participate. Most often, dominators don't know they are dominating. Facilitators can gently ask via email for the member to give others a little more time to respond, while also acknowledging their important contribution, for the line between core member and dominator is pretty fuzzy.

Linkers, weavers and pollinators

The bumblebees and butterflies! This group of people is very important in larger communities where there may be a large selection of conferences and topics from which to choose. These members tend to participate across a range of interests, and in doing so, are in the best position to let others know of interesting happenings across the community. They keep spaces from getting dull or stale. On the other hand, they can disrupt slower, deeper conversations with their "flitting" in and out.


Flamers live, as they say, to flame. Flaming is defined as sending hostile, unprovoked messages. What is actually considered a flame varies by community, but often there are people who enjoy challenging other members just for the "fun of it." Name-calling, innuendo and such are the tools of flamers. The interesting dynamic of flaming is that to an extent, it draws community interest as a form of entertainment. At the other end, it drives people away if it goes over the line of community norms.

Actors and Characters

Some people very successfully develop online personas with "bigger than" life personalities and characteristics. They may be the online version of the "Class Clown, " the humorist or one-line master, or just have a unique way of communicating that stands out. These are strong attractors of community attention, especially in social communities. They can help lighten the atmosphere for a community, helping balance tense situations and introduce ways for people to reveal more about themselves in a potentially less threatening manner. When they push too hard against community norms, they can be perceived as negative influences for two main reasons: interrupting "serious" threads or conversations, and for not knowing when to quit based on group norms (usually unspoken norms).

Energy Creatures

Perhaps the most famous archetype in online communities, the Energy Creature is an individual who so irritates a community that they form up around him or her to try and counteract the "creature's" energy. They community may try shunning the energy creature, but often get pulled into the vortex and become energy creatures themselves. At their worst, energy creatures can destroy a conversation or community. At their best, they are often caricatured mirrors of the community, helping us recognize our own potentially negative patterns.


Defenders sometimes defend an individual (sometimes to the point of being perceived as a slavish defender) or groups. They can be hypersensitive to the smallest slight or suggestion of attack, perhaps because of previous experiences. They may also have highly developed intuitive skills, which can be very productive for a community and serve as an "early warning" signal of a changing community dynamic.


It only takes one line, repeated, inserted, and insinuated, over time, to recognize a needler. They have a point to make and it appears again, and again, and again. Often in the form of a cynical "I told you so," Needlers know they are right and won't let you forget it. Their point may be insightful or irrelevant, but the value of the point is quickly lost on an audience who gets fatigued from the repetition. This is different from a spammer because the point is often "on point." But it can loose its power and context, regardless of the quality. In some cases, this may be from a visionary who is ahead of her/his time, who needles with the best interest of the group in mind.


Sometimes called "clueless newbies," newbies are members new to a community. They might also be new to online interaction. When new folks jump into an online interaction without checking it out, observing the interaction or learning the community norms, they can be perceived as rude and clueless. In some communities, newbies are treated to a baptism of fire by old hands as a way of either being accepted or rejected from the group. Newbies are also the source of new blood, ideas, interest and "pollination," thus the new-bee appellation. Newbies deserve our attention and should be supported with information to help them become part of the group.


Also known as the PC (politically correct) Police. PollyAnnas also operate across a range of "acceptable" behavior, from being a source of appreciation of community members, to the being "nice" at the expense of being honest or "real." They see the bright side to most anything, so they can be a positive influence. However, Pollyannas drive some people so nuts they will leave a thread just to escape. PollyAnnas avoid conflict and withdraw before clarity is reached because they are averse to conflict.

"Black and White" Folks

These are the people who present immutable positions. They appear to be initially unwilling to see points of view beyond their own. They push instead of probe. They are usually willing to take the blame for their style (ownership) but shy away from the responsibility of the impact of their style. They engage only on their own terms, but may refuse to engage others who utilize the same tactics.

"Shades of Grey" Folks

Sometimes characterized as wishy-washy, with no clear convictions, and as members who shrink away from the tough issues. Often they won't fully engage or justify their positions. On the other side, they often can help neutralize a polarized situation and offer new, combined viewpoints for a community. They tend to carry new information into a group that has polarized on issues and can be a breath of fresh air.

Untouchable Elders

We tend to thrust this archetype on others -- the expert, the guru -- and sometimes unconsciously create a different set of rules or norms for the elder. Most often, the elder does not seek this recognition. Elders may not held accountable to the same community norms or scrutiny of the other members. Elders can dominate new members by a few words, regardless of the value of the words of others around them. Their wisdom is gold to a community, but their influence can inadvertently muzzle the rest of the group who might feel uncomfortable posting in such company.

Community Member Roles and Types, Nancy White

Path Dependence

Path dependent processes are ones in which slight differences in starting conditions can lead to drastically different outcomes down the line.

As an example, consider a jar of a red ball and a blue ball. You pick out a ball at a random, then you the original ball plus another ball of the same color back into the jar. Of course, this increases the likelihood that you'll draw another ball of that color. And if you do, the process makes it even more likely to happen again, and so on. So the kinds of balls in the jar after several trials will be quite different depending on if you picked a red or a blue ball first.

Path Dependence in Voting Systems

Ranking systems should consider the possibility of herd behavior-driven path dependence:

For 5 months, every comment submitted by a user randomly received an "up" vote (positive); a "down" vote (negative); or as a control, no vote at all. The team then observed how users rated those comments. The users generated more than 100,000 comments that were viewed more than 10 million times and rated more than 300,000 times by other users.

At least when it comes to comments on news sites, the crowd is more herdlike than wise. Comments that received fake positive votes from the researchers were 32% more likely to receive more positive votes compared with a control,, the team reports online today in Science. And those comments were no more likely than the control to be down-voted by the next viewer to see them. By the end of the study, positively manipulated comments got an overall boost of about 25%. However, the same did not hold true for negative manipulation. The ratings of comments that got a fake down vote were usually negated by an up vote by the next user to see them. Source, Social Influence Bias: A Randomized Experiment. Lev Muchnik, Sinan Aral, Sean J. Taylor.

All the coefficients show that top-level comments, early comments, and comments with higher starting scores were more likely to receive moderation and to get higher final scores, even when controlling for the potential confounds. - Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space, Cliff Lampe & Paul Resnick

From a post introducing Reddit's new comment ranking system illustrating this phenomenon:

reddit is heavily biased toward comments posted early. When a mediocre joke gets posted in the first hour a story is up, it will become the top comment if it's even slightly funny. (I know this particularly well because I have dozens of reddit fake identities and with them have posted hundreds of mediocre jokes.) The reason for this bias is that once a comment gets a few early upvotes, it's moved to the top. The higher something is listed, the more likely it is to be read (and voted on), and the more votes the comment gets. It's a feedback loop that cements the comment's position, and a comment posted an hour later has little chance of overtaking it -- even if people reading it are upvoting it at a much higher rate.

There are periodically big threads with subjects like "what's your favorite joke/best advice/worst secret/weirdest habit?" A few weeks ago, I took one of these stories when it was 8 hours old and did a count of the top (root) comments. Of the top 10 comments, ALL were posted either "7 hours ago" or "8 hours ago" -- that is, in the first hour or two the story had been up. The "top" list was simply a list of the best jokes from within the first hour. Not a single joke from the last six hours (when most of the comments had been posted) made it near the top. And they never got a chance to; the story fell off the frontpage and they all stopped accumulating votes. They may have been getting upvotes from everyone who saw them, but that didn't let them catch up to the older comments at the top. One effect of this bias, which you may have seen, is posts saying "sorry to hijack your top comment, but I need to post some important information that refutes the main article." They know that no matter how good their comment is, it won't make it to the top. - reddit's new comment sorting system, Randall Munroe

The Online Disinhibition Effect

Proposed by John Suler in this essay, which is a version of the paper Suler, J. (2004). CyberPsychology and Behavior, 7, 321-326

Suler distinguishes benign disinhibition - "[People] show unusual acts of kindness and generosity." - from toxic disinhibition - "Out spills rude language and harsh criticisms, anger, hatred, even threats".

Suler categorizes a few different causes for the effect:

Is the disinhibited self the "true" self?

The concept of disinhibition may mistakenly lead us into thinking that what is disinhibited is more real or true than the part of us that inhibits. ... This is a simplistic interpretation of the much more dynamic psychoanalytic model which states that the inhibitory processes of repression and defense mechanisms are components of personality no less real or important than others. Psychoanalytic clinicians believe that understanding defenses is crucial to the success of the therapy because it reveals hidden thoughts, feelings, and needs. Why does a person repress something? Why is it being inhibited? Bypassing defenses to get to the "true" self may also bypass the opportunity to discover aspects of the inhibiting self that are just as true. When these defenses and elements of the inhibited self are worked through, remnants of them sometimes remain to serve an important function. Sometimes they evolve into productive aspects of one's personality independent of the problematic emotions that were originally defended.

The same is true online. Some people in some online situations become disinhibited and reveal aspects of themselves. However, at the same time, they may not be not grappling with the underlying causes of that inhibition, and therefore are missing an opportunity to discover something important about themselves - something very true about themselves, but often unconscious.

...Sometimes we act, think, or feel one way, and sometimes the opposite. We have ambivalent, sometimes opposing emotions. ... Different communication environments convey different facets of these polarities in self. ... Each media allows for a particular expression of self that differs - sometimes greatly, sometimes subtly - from another media. ... If a man suppresses his aggression in life but expresses it online, both behaviors reflect important aspects of his personality that surface under different conditions. ... Different communication modalities enable different expressions of oneself. They allow us to see the different perspectives of that complex thing we call "identity."

Social Inhibition

Social inhibition generally means the restriction of behavior due to social contexts. The Wikipedia page has a lot more information.

Although the disinhibition effect may be at play in anonymous or pseudonymous situations, social inhibition effects may be present as well (especially in pseudonymous environments where there is still persistent identity and reputation).

The Fluff Principle

Proposed by Paul Graham in What I've Learned from Hacker News.

The most dangerous thing for the frontpage is stuff that's too easy to upvote. If someone proves a new theorem, it takes some work by the reader to decide whether or not to upvote it. An amusing cartoon takes less. A rant with a rallying cry as the title takes zero, because people vote it up without even reading it.

Hence what I call the Fluff Principle: on a user-voted news site, the links that are easiest to judge will take over unless you take specific measures to prevent it.

Scalability of Communities

The downside of going for size and scale above all else is that the dense, interconnected pattern that drives group conversation and collaboration isn't supportable at any large scale. Less is different -- small groups of people can engage in kinds of interaction that large groups can't. And so we blew past that interesting scale of small groups. Larger than a dozen, smaller than a few hundred, where people can actually have these conversational forms that can't be supported when you're talking about tens of thousands or millions of users, at least in a single group. - A Group Is Its Own Worst Enemy, Clay Shirky

Shared Awareness

Trolls attract replies by exploiting the general absence of shared awareness on most mediated communications platforms. Shared awareness is a special state of meta-knowledge among collaborators that allows them to act cooperatively on the knowledge.

If I know something and you know something but I don't know that you know it and you don't know that I know it, we can't build on that knowledge even though we both know the same thing.

Shared awareness is not reached until: I know something, you know something, I know that you know, you know that I know, I know that you know that I know, and you know that I know that you know. (Whew!)

Once we all know that we all know the thing, we can then act on it. Without that shared awareness, our ability to put the information to use is sharply curtailed.

In a conventional online comment system, several participants may each know something, but they do not necessarily have shared awareness about it (everyone knows that everyone knows).

When a troll posts a comment, for example, other participants may decide independently that the comment is inappropriate and/or wrong and/or offensive, but they have no way of knowing whether anyone else feels the same way.

A conscientious forum-goer could just ignore the troll and hope for the best, but silence is affirmation, and a comment left to stand unchallenged starts to look like it represents the prevailing view of the forum participants., and a comment left to stand unchallenged starts to look like it represents the prevailing view of the forum participants.

Hence, the conscientious forum-goer feels no choice but to reply to the troll, if only to assure others that at least someone else disagrees with it.

Now the fish is hooked, and a skillful troll can keep that fish on the line for a long time.

The Strength of Weak Ties

Although we're more likely to share information from our close friends, we still share stuff from our weak ties--and the links from those weak ties are the most novel links on the network. Those links from our weak ties, that is, are most likely to point to information that you would not have shared if you hadn't seen it on Facebook. The links from your close ties, meanwhile, more likely contain information you would have seen elsewhere if a friend hadn't posted it. These weak ties "are indispensible" to your network, Bakshy says. "They have access to different websites that you're not necessarily visiting." - The End of the Echo Chamber, Farhad Manjoo

Economic sociologist Mark Granovetter was one of the first to popularize the use of social networks in understanding the spread of information. In his seminal 1973 paper, The Strength of Weak Ties, Granovetter found that surprisingly, people are more likely to acquire jobs that they learned about through individuals they interact with infrequently rather than their close personal contacts.

What do these structures have to do with information access? Since people in these clusters all know each other, any information that is available to one individual spreads quickly to others within the cluster. These tight-knit social circles tend to be small relative to people's entire social network, and when it comes to information about future job opportunities, it can be hard to find new leads.

Granovetter used the relationship between interaction frequency and social structure to explain why information about jobs is instead found through weak ties that we interact with infrequently. Weak ties help spread novel information by bridging the gap between clusters of strong tie contacts. The strength of weak ties informs much of the popular understanding of information spread in social networks.

...We found that information shared by a person's weak ties is unlikely to be shared at a later point in time independently of those friends. Therefore, seeing content from a weak tie leads to a nearly tenfold increase in the likelihood that a person will share a link. In contrast, seeing information shared by a strong tie in News Feed makes people just six times as likely to share. In short, weak ties have the greatest potential to expose their friends to information that they would not have otherwise discovered. ...It turns out that the mathematics of information spread on Facebook is quite similar to our hypothetical example: the majority of people's contacts are weak tie friends, and if we carry out this same computation using the empirical distribution of tie strengths and their corresponding probabilities, we find that weak ties generate the majority of information spread. - Rethinking Information Diversity in Networks, Eytan Bakshy

Negativity Effect

...we discover that community feedback does not appear to drive the behavior of users in a direction that is beneficial to the community, as predicted by the operant conditioning framework. Instead, we find that community feedback is likely to perpetuate undesired behavior. In particular, punished authors actually write worse in sub sequent posts, while rewarded authors do not improve significantly. ...Surprisingly, we find that negative feedback actually leads to more (and more frequent) future contributions than positive feedback does.

...the detrimental impact of punishments is much more noticeable than the beneficial impact of rewards. This asymmetry echoes the negativity effect studied extensively in social psychology literature: negative events have a greater impact on individuals than positive events of the same intensity.

How Community Feedback Shapes User Behavior, Justin Cheng, Cristian Danaescu-Niculescu-Mizil, Jure Leskovec

Other findings from the same paper:

As users receive feedback, both their posting and voting behavior is affected. When comparing the fraction of up-votes received by a user with the fraction of up-votes given by a user, we find a strong linear correlation. This suggests that user behavior is largely “tit-for-tat”. If a user is negatively/positively evaluated, she in turn will negatively/positively evaluate others. However, we also note an interesting deviation from the general trend. In particular, very negatively evaluated people actually respond in a positive direction: the proportion of up-votes they give is higher than the proportion of up-votes they receive. On the other hand, users receiving many up-votes appear to be more “critical”, as they evaluate others more negatively. For example, people receiving a fraction of up-votes of 75% tend to give up-votes only 67% of the time.

...The more balanced the network, the stronger the separation of the network into coalitions — nodes inside the coalition up-vote each other, and down-vote the rest of the network.

Community vs Society

The following quotes are from Attacked from Within, anaesthetica.

German sociologist Ferdinand Tonnies first investigated the difference between 'community' and 'society' (respectively, Gemeinschaft and Gesellschaft). Small groups can exist in a sense of organic community, not requiring formal rules because a sense of common mores or norms unite them. Personal relationships can be cultivated and are quite strong, and there is little need for external enforcement. ... Larger groups find community hard to sustain. Individual interest rules behavior rather than common mores. Society, as opposed to community, is based on explicit rules that require enforcement. Society possesses greater flexibility and potentially more capability, but individuals are subject to greater anomie and anti-social behavior. Internal factional conflicts occur more frequently, despite the greater modularity of individuals' function in society.

The internet is still dealing with the problem of community collapse. Each site that attempts to build community and grow in size inevitably reaches this tipping point in which socialization into community is no longer possible. ... Society scales easily because users are interchangeable, community scales with difficulty because relationships and identity are not interchangeable.

...Society, especially civil society, depends on shared culture, mores, norms. At the smaller scale, community can enforce its own mores, but as greater and greater scale comes, community collapses into society, and the mores that sustained the older users are incapable of being effectively transmitted to the newly inducted masses.

Maintaining the common institution of culture can be conceived of as a collective action problem. Mancur Olson gave the definitive treatment of the subject in The Logic of Collective Action. According to Olson, small groups are qualitatively different from large groups, when considered in terms of their respective abilities to achieve collective goods. Small groups are small enough that an individual's actions are noticeable by other members. In large groups, the effects of any given user's bad behavior are not necessarily discoverable by all members and there is little incentive for individual members to enforce the group's rules. This is why communities can only function when small, but collapse into societies when their growth outstrips the institutional capacity for individual behavior to be noticed (and punished).

The threshold between community and society may be somewhere around Dunbar's number. Although, maybe it could be larger online: "Face-to-face relationships obviously have different requirements for their maintenance than do online relationships."

What makes a good community?

Discussion Taxonomy

Threat Modeling / How Communities Fail

It's trolls all the way down

It's trolls all the way down


Astroturfing is the practice where some organization or company coordinates a simulated social movement or response in order to make it look "grassroots" (hence the name).

For his film (Astro)Turf Wars, Taki Oldham secretly recorded a training session organised by a rightwing libertarian group called American Majority. The trainer, Austin James, was instructing Tea Party members on how to "manipulate the medium". This is what he told them:

"Here's what I do. I get on Amazon; I type in "Liberal Books". I go through and I say "one star, one star, one star". The flipside is you go to a conservative/ libertarian whatever, go to their products and give them five stars. … This is where your kids get information: Rotten Tomatoes, Flixster. These are places where you can rate movies. So when you type in "Movies on Healthcare", I don't want Michael Moore's to come up, so I always give it bad ratings. I spend about 30 minutes a day, just click, click, click, click. … If there's a place to comment, a place to rate, a place to share information, you have to do it. That's how you control the online dialogue and give our ideas a fighting chance." - Reclaim the Cyber-Commons, George Monbiot.

See also: Robot Wars, George Monbiot.

Sybil Attacks & Sockpuppet Accounts

Sockpuppet accounts are accounts created by users with existing accounts to conduct some kind of activity under the guise of being someone else.

Sybil attacks are where users create multiple accounts or identities in order to coordinate some kind of manipulative or harmful action or sustain some kind of existing antagonistic behavior (e.g. one account gets banned, create another, and so on).

Some attacks are detailed in Bazaar: Strengthening user reputations in online marketplaces, Ansley Post, Vijit Shah, Alan Mislove:

So for example, a Sybil attack can be where a user or a small group of users create many, many more accounts to vote up their own content or vote down someone else's.


The term "troll" has really lost its nuance; it is now used to refer to almost any abusive or aggressive internet behavior.

Brendan Koerner has a description which I think captures the original usage of the term:

The word troll is used too often these days as a slur. It’s trotted out to describe virtually any Internet commenter who spouts ugly sentiments—cyberbullies, race baiters, sexist pigs. There is no subtlety to the manner in which these jerks sling insults, nor any purpose to their horrid conduct aside from making themselves feel good by making others feel bad. They’re just twisted individuals who deserve nothing but scorn.

But genuine trolls would resent being lumped together with such clumsy degenerates. They take pride in their incendiary craft, which ideally involves a certain degree of subterfuge. These trolls cloak their mischievous aims behind a veneer of earnestness, one that fools the gullible into thinking they’ve encountered someone worthy of engaging in debate. But this kind of troll has no interest in meaningful dialog and ideally prefers to remain silent while their unsuspecting victims go bonkers. One of the most infamous examples, a 1996 Usenet post lambasting college fraternities, sparked approximately 3,500 replies before a single additional word from the original author. But when skillful trolls do choose to converse with their critics, they poison the discussion with more over-the-top provocations masquerading as sincere statements. Their reward is the palpable rage they elicit from people too thin-skinned or naive to heed one of this millennium’s most important maxims: “Don’t feed the trolls.”

Though dedicated trolls are too odious to be lovable, they can’t be dismissed as easily as the punks who litter YouTube comments with vile tirades. The most accomplished trolls force online communities to ponder the limits of free speech in a medium that was supposed to obviate censorship. Should a commenter be banned just because they hold strong views that are diametrically opposed to those of the majority? Or because they’re obviously out to create mischief, albeit without employing vulgarities or ad hominem attacks? In the course of satisfying their bizarre and selfish need to foment chaos, trolls also provide the Internet with a valuable reminder that its ideals and its realities often diverge. - Hey Troll, Who You Calling a Troll?

Another good definition:

The term "troll" comes originally from the fishing method of dangling a shiny lure out the back of a boat while putting along just quickly enough to grab the interest of a fish. The fish that can't resist the lure is snared on the hook and eaten. ... sophisticated trolls can keep a debate going for days by dancing on the fine line between feigned reasonableness and deliberate obtusity.

...Pointless, unending debates between trolls and earnest opponents do two things:

...The three necessary conditions for a troll disrupting a forum are:

Shared Awareness: A Better Way to Manage Comment Trolls, Ryan McGreal


Since all three conditions are necessary for disruption to take place, we have three points of attack in trying to prevent it: for disruption to take place, we have three points of attack in trying to prevent it:

...For inappropriate comments, there needs to be a mechanism for members of the community to flag them as inappropriate without actually posting replies; and that flag must be without actually posting replies; and that flag must be visible to everyone so that an individual looking at the comment can see how many other people have already flagged it.

Shared Awareness: A Better Way to Manage Comment Trolls, Ryan McGreal

Trolls don't give up when they're no longer seen, just like spammers don't give up when a new filter is implemented, and terrorists don't give up even if you stop televising their propaganda videos and attacks on the eleven o'clock news. There is no end of history for trolling--if you are not prepared to live in a world of endless trolling, your moderation system and community will fail and you will log off a broken and bitter neckbeard. - A Non-Affirming Silence, anaesthetica

Another good definition from Paul Graham:

There are two senses of the word "troll." In the original sense it meant someone, usually an outsider, who deliberately stirred up fights in a forum by saying controversial things. ...This sort of trolling was in the nature of a practical joke, like letting a bat loose in a room full of people.

The definition then spread to people who behaved like assholes in forums, whether intentionally or not. Now when people talk about trolls they usually mean this broader sense of the word. Though in a sense this is historically inaccurate, it is in other ways more accurate, because when someone is being an asshole it's usually uncertain even in their own mind how much is deliberate. That is arguably one of the defining qualities of an asshole.

...There's a sort of Gresham's Law of trolls: trolls are willing to use a forum with a lot of thoughtful people in it, but thoughtful people aren't willing to use a forum with a lot of trolls in it. Which means that once trolling takes hold, it tends to become the dominant culture. - Trolls, Paul Graham

Also worth reading Confessions of a Former Internet Troll, Emmett Rensin

Witch Hunts

How One Stupid Tweet Blew Up Justine Sacco’s Life

Monica Lewinsky's The Price of Shame on the online culture of humiliation

Eternal September

The phenomenon (or just the fear of the phenomenon) of a increase in the rate of new users arriving, which decreases the quality of content and discussion the site and generally erodes is culture, until there is an exodus of the older members.


This may be local to Reddit, but vote brigading is where an organized group of users, typically aligned along some ideological axis, go to a post or comment and downvote it en masse.


The release of personally identifying information (PII) to the public, against someone's will.

Gaming the System

Misbehaving users will find a unique way to abuse almost any unique feature you offer them. If you build it, some will exploit it. - Deviant Behavior in Online Multimedia Communities and Strategies for Managing it, John Suler

Low-Investment Material (LIM) vs Greater-Investment Material (GIM)

A model proposed by u/LinuxFreeOrDie. I've adapted it here.

Say we have a community of size \(n = 1000\). The first group, \(U_G\), has 400 users who only like GIM. The second group, \(U_{LG}\), has 300 users who like LIM and GIM equally. The final group, \(U_L\), also has 300 users, but they like LIM only.

Say that the average investment time for LIM is \(I_L = 10sec\) and for GIM it is \(I_G = 100sec\). This time includes the actions of submitting a vote in addition to viewing the linked content.

Say that the percent of users from each group active on the site at any given time is \(u = 0.05\).

So the active users of \(U_G\) is 20 and, limited by \(I_G\), can contribute only 0.2 votes per second.

The active users of \(U_L\) have only 15 users active but with the \(I_L\) rate can contribute 1.5 votes per second.

The middle group, group \(U_{LG}\), is indifferent about whether content is LIM or GIM, but given the behavior of the first two groups, they are likely to see more LIM content (since that has the potential to accrue many more votes than the GIM content). This exposure to more LIM content causes them to vote more on LIM content, causing a positive feedback loop.

If you change the model such that \(U_G\) users start actively downvoting LIM content, spending half of their time doing so. Since these users are not actually viewing the content, their average investment time for LIM will be lower, say \(I_L/2 = 5sec\). So they will only be able to contribute 0.1 votes per second for the GIM content, and can contribute 4 downvotes per second for the LIM content.

Toxic Behavior requires less time investment

A related phenomenon to LIM vs GIM:

The problem with sites like Wikipedia and Digg is that there are always registered users with less of a life than you. Persistence, not quality, counts for more than anything else. - Attacked from Within, anaesthetica

Power user oligarchies

Small groups can be effectively governed when one or a few members are granted greater capabilities to preserve the culture of the community--we call these moderators. While they play a crucial role in most online communities, their ability to police ever-larger numbers of participants is limited. Unfortunately, as the pool of moderators grows and as moderator status becomes increasingly institutionalized, the iron law of bureaucracy sets in. - Attacked from Within, anaesthetica

Iron Law of Bureaucracy

Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representative who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions. - Jerry Pournelle

Moderation/voting as approval of thought, not quality

Alternately, moderation systems like Slashdot's Karma and Digg's approval-style voting put moderation of content (as opposed to moderation of users) into the hands of the userbase as a whole. While viewing Slashdot comments with an appropriately high threshold is effective in displaying only high quality comments, a vast amount of material that is high-quality yet counter to Slashdot's groupthink remains below the threshold. ... Moderation abuse is hardly countered by offering these same users incentives (more moderation power) to moderate moderations. - Attacked from Within, anaesthetica

Failures of trust/reputation metrics

...regarding the negative consequences of trust metrics: they focus on the individual rather than on their contributions (comments, stories), the outcome being neither community nor society but class conflict and stifling monoculture. As Paul Graham notes in his assessment of lessons learned from administering Hacker News: "It's bad behavior you want to keep out more than bad people. User behavior turns out to be surprisingly malleable. If people are expected to behave well, they tend to; and vice versa." - Attacked from Within, anaesthetica

Existing Approaches


Essentially, "If you start a thread, you become its moderator."

....Still no indication given as to how "Featured Discussions" are chosen, whether algorithmically or editorially. There are a few clues in Kinja's FAQ:

...Kinja makes a "dismiss" button available to the creator of the thread, which allows them to cast away malicious, or simply annoying, comments. What will remain when the jetsam has been excised is a clean, on-topic thread that reads like a conversation. To maximize the value of this feature, Kinja's divergent branches are modular: Each discussion thread that breaks off the main one can be independently linked to or embedded. An algorithm will evaluate the threads and highlight the best ones. - More on Gawker, anaesthetica

Stack Exchange

Meta Forums

Stack Exchange (and Discourse) have meta forums where users can discuss the platform, on the platform itself. It is an explicit place for communities to figure out their own norms and rules and identity and purpose and bring up grievances about the community or platform.


Our method of dealing with disruptive or destructive community members is simple:their accounts are placed in timed suspension. Initial suspension periods range from 1 to 7 days, and increase exponentially with each subsequent suspension. We prefer the term "timed suspension" to "ban" to emphasize that we do want users to come back to their accounts, want users to come back to their accounts, if they can learn to refrain from engaging in those disruptive or problematic behaviors. they can learn to refrain from engaging in those disruptive or problematic behaviors. It's not so much a punishment as a time for the user to cool down and reflect on the nature of their participation in our community. (Well, at least in theory.)

...A hellbanned user is invisible to all other users, but crucially, not himself. From their perspective, they are participating normally in the community but user is invisible to all other users, but crucially, not himself. From their perspective, they are participating normally in the community but nobody ever responds to them.. They can no longer disrupt the community because they are effectively a ghost. It's a clever way of enforcing the "don't feed the troll" rule in the community. When nothing they post ever gets a response, a hellbanned user is likely to get bored or frustrated and leave.

...There is one additional form of hellbanning that I feel compelled to mention because it is particularly cruel - when hellbanned users can see only themselves and other hellbanned users.

...A slowbanned user has delays forcibly introduced into every page they visit. From their perspective, your site has just gotten terribly, horribly slow. And stays that way. user has delays forcibly introduced into every page they visit. From their perspective, your site has just gotten terribly, horribly slow. And stays that way.

...An errorbanned user has errors inserted at random into pages they visit. user has errors inserted at random into pages they visit.

Suspension, Ban, or Hellban?, Jeff Atwood

See the Stack Exchange FAQ for a lot more good stuff.


OStatus, GNU Social and other federated social web protocols

Trying to gather more info about this, surprisingly hard to find.

As far as I can tell, GNU Social (formerly StatusNet) is server software to run social network software that adheres to the FSF's standards.

It's motivated by the ideas around the "federated web" or "federated social web", which is: means that there are distinct entities that control parts of the system, but those parts are connected with agreed-upon rules to make a pleasing and usable whole. - What is the Federated Social Web?

So multiple sites/servers run GNU Social and they form an aggregate social network (as opposed to one organization, e.g. Twitter, controlling the entire network). They adhere to standards so for all intents and purposes, they are an integrated whole.

OStatus is a protocol for the federated social web.

It uses PubSubHubbub (PuSH):

In the PuSH system, a site can subscribe to updates for a feed from a hub server associated with that feed. Whenever a new post is created, the publishing site pings the hub, and the hub sends out just the new posts to all the subscribers.

OStatus is built on that simple base: each participating site produces Atom feeds of updates and uses PuSH subscriptions to send relevant updates to other sites.

The real beauty of it is that at this point we already have something useful, without anything site specific. In fact you can subscribe to someone's public Wordpress feed as an OStatus remote user, and they haven't had to do anything special at all! - is another social protocol which grew out of these.


League of Legends (Riot Games)

Some of the reforms Riot came up with were small but remarkably effective. Originally, for example, it was a default in the game that opposing teams could chat with each other during play, but this often spiraled into abusive taunting. So in one of its earliest experiments, Riot turned off that chat function but allowed players to turn it on if they wanted. The impact was immediate. A week before the change, players reported that more than 80 percent of chat between opponents was negative. But a week after switching the default, negative chat had decreased by more than 30 percent while positive chat increased nearly 35 percent. The takeaway? Creating a simple hurdle to abusive behavior makes it much less prevalent.

...The team also found that it's important to enforce the rules in ways that people understand. When Riot's team started its research, it noticed that the recidivism rate was disturbingly high; in fact, based on number of reports per day, some banned players were actually getting worse after their bans than they were before. At the time, players were informed of their suspension via emails that didn't explain why the punishment had been meted out. So Riot decided to try a new system that specifically cited the offense. This led to a very different result: Now when banned players returned to the game, their bad behavior dropped measurably.

...All of these tactics helped League of Legends redefine its community norms, the shared beliefs about how people are expected to behave.

...In another initiative by Riot's player- behavior team, League of Legends launched a disciplinary system called the Tribunal, in which a jury of fellow players votes on reported instances of bad behavior. Empowered to issue everything from email warnings to longer-term bans, users have cast tens of millions of votes about the behavior of fellow players. When Riot asked its staff to audit the verdicts, it found that the staff unanimously agreed with users in nearly 80 percent of cases. And this system is not just punishing players; it's rehabilitating them, elevating more than 280,000 censured gamers to good standing. Riot regularly receives apologies from players who have been through the Tribunal system, saying they hadn't understood how offensive their behavior was until it was pointed out to them. Others have actually asked to be placed in a Restricted Chat Mode, which limits the number of messages they can send in games--forcing a choice to communicate with their teammates instead of harassing others. - _Curbing Online Abuse Isn't Impossible. Here's Where We Start, Laura Hudson

Riot found that nastiness was a communitywide problem: "If we remove all toxic players from the game, do we solve the player behavior problem? We don't," said Jeffrey Lin, Riot's lead designer of social systems. Persistently bad-behaving players only produced 13 percent of the harassment on the site; the remainder of harassment was lodged by "players whose presence, most of the time, seemed to be generally inoffensive or even positive." The takeaway: "Banning the worst trolls wouldn't be enough to clean up League of Legends. ...Nothing less than community-wide reforms could succeed."

...Riot also found a way to reduce recidivism rates among its harassing users by clearly spelling out its justification for suspending users over offensive comments instead of just quietly banning them, which led to a rash of apologies from players, many of whom said they didn't think before they spoke or didn't fully understand the impact of their comments. (Riot says it's since lifted 280,000 gamers from offender status to upstanding members of the community.) - How One Video Game Company Is Leading the Charge Against Online Harassment, Amanda Hess

The LoL Tribunal

The Tribunal identifies players who have been consistently reported by the community over a large number of games and builds a Tribunal case for them. These cases are presented to random community members who use the Tribunal who then review the case files and render a judgment--pardon or punish. Player Support then uses this information to help assign the right penalties to the right players.

...Summoners must be Level 20 or above and not currently receiving a punishment by the Tribunal to participate.

...The Justice Review is a report card for contributing Tribunal members, providing details on judgment accuracy and how effective their voting has been.

...The Justice Rating is the relative skill level of Tribunal contributors based on how often and how accurately they vote. Players get bonus rating for getting streaks of correct cases.

Reform Cards

Toxic players are consistently toxic, normal (98%) players are not:

Toxic vs Normal players

Toxic vs Normal players

"if we remove all toxic players from the game, do we solve the player behavior problem?"

The answer: no, because normal players occasionally behave toxically.

This behavior can be viral, in a sense. one player's bad day (leading to toxic behavior) can trigger another player's bad day (leading to further toxic behavior), and so on.

In the Tribunal, "Tribunal reform cards" were added. This is additional information in Tribunal cases which show the decision, the agreement (e.g. majority or minority), and the resulting punishment. This info was sent to players banned by the tribunal.

Prior to reform cards, most players would get worse after receiving a vague ban:

% Change in Reports per Game after a punishment, without reform cards

% Change in Reports per Game after a punishment, without reform cards

This changed with reform cards - now players had decrease in reports afterwards:

% Change in Reports per Game after a punishment, with reform cards

% Change in Reports per Game after a punishment, with reform cards

However still many players complained that the tribunal system was broken (i.e. unjust), that the bans were unfair, etc. they would post these complaints in the LoL forums with the reform card link for that case.

The responses to these threads are often in support of the community decision. For example: "You deserve every ban you got with language like that." so the forum offered another avenue for community reinforcement of these values.

Forum responses to a complaining player

Forum responses to a complaining player

On the flip side, the reform cards have helped other players realize that they were being toxic, and they decided to change their ways:

Players' testimonials after reform cards

Players' testimonials after reform cards

Slides are from Jeffery Lin's GDC 2013 talk, The Science Behind Shaping Player Behavior in Online Games.

Civic IQ

The idea is to connect like-minded people with a series of subtle questions, rather than grouping them by party, and give them a space to develop proposals to specific policy problems...

...The crucial thing here is that you're not asked about your party affiliation right off the bat. Rather, users receive a "fingerprint" based on their answers, that organizes their views into a hexagonal matrix rather than a binary labeling system. It's entirely possible (in fact, it's likely) that a registered Republican and Democrat could overlap on certain nodes in the matrix, like "social progressivism" or "individualism." The idea is to find points of consensus, rather than divide users into camps. Users can up-vote specific issues to show their support, rather than ascribing to generalized political platforms.

...Perhaps the biggest difference between Civic IQ and other open government platforms is that the system is designed to produce full-fledged proposals, not generalized demands. Once a topic receives enough points, it moves into a brainstorm phase where users build a plan of action.

...It's the full proposal that is eventually voted upon, rather than a one-liner issue. So rather than asking Congress to, say, "make school lunches healthier," the Civic IQ system would deliver a detailed proposal for how much it would cost to make lunches healthier, and where such funding could come from. - Can Design Make Online Debate Less Toxic?, Kelsey Campbell-Dollaghan


Branch... [allows] people to form a new branch from any given comment (although it's not clear if sub-branches are discoverable from the main branch). Ideally this would keep most off-topic posts in their own separate conversation. Will probably start to fail when the scale of conversations ramps up.

...You can subscribe to conversations, and you can request to be invited to the conversation. This is probably going to mitigate some of the effects of scale, by preventing just anyone from joining into the conversation uninvited. - Branch and conversation on the internet, anaesthetica


Metafilter has a one-time, lifetime $5 sign up fee.

After you sign up, there is a waiting period before you can post.

There's also a waiting period between posts, no matter how long you have been a user:

...After you have posted a thread (a post in MetaFilter, a question in AskMetafilter, etc), you have to wait this long before posting a second thread in that same area of the site. There is no similar limit for comments.

...With a few exceptions, people who have been banned from MetaFilter may return under our Brand New Day policy. BND means that a user can sign up with a new account and mods will not "out" that user as being someone who held an account previously. This is a courtesy we extend to people who feel ready to change a problematic old persona, and it comes with certain expectations of good faith effort to change the old behavior. Users who open a new account and continue problematic behavior that caused their old account to be banned may lose the right to use that new account and/or set up new accounts.

...There is an upper limit of 240 favorites per day (that you can add), to keep people from abusing the favorites system.

...The Recent Comments tab in Metafilter and MetaTalk will show you which threads have recent comments in them. This will often include the most recently posted threads, but will also catch older threads that are still chugging along.

...Some people want to follow threads in a way similar to Recent Activity, but without commenting in them. Right now, you can do this by favoriting only threads you want to follow, and then using the My Favorites tab (see here) to track activity from those threads.

...Recent Activity, linked in the header bar, is a convenient way to keep up with conversations you have been part of, without needing to go check each one separately. This feature allows you to keep track of new comments in threads you have either commented in or posted. ... If you are tired of seeing updates from a given thread, you can remove it from Recent Activity by clicking "remove from activity" next to that thread on your Recent Activity page.

...After a certain amount of time, threads are automatically closed to new comments. The length of time varies on different parts of the site.

...On the footer of every MetaFilter page is: (C) 1999-2013 MetaFilter Network Inc. All posts are (C) their original authors. What this means is that people own their own content.


Hacker News

I think it's important that a site that kills submissions provide a way for users to see what got killed if they want to. That keeps editors honest, and just as importantly, makes users confident they'd know if the editors stopped being honest. - What I've Learned from Hacker News, Paul Graham

Hacker News also starts fading the text colour of comments with net negative scores. The benefit to this is that it sends a strong visual message about how the community feels about the comment. - Shared Awareness: A Better Way to Manage Comment Trolls, Ryan McGreal


Used in some subreddits, flair (a small text icon by your username) is used for many reasons, but sometimes to mark expertise, affiliation, or some other kind of legitimacy. On certain subreddits, flair has to be applied for and proof must be given to the moderators to verify.



Slashdot has a system where you can only vote on content/comments if your account is in the 90% oldest accounts.

New York Times

The NYT experimented with structured comments:

Structured input form

Structured input form

Resulting comments

Resulting comments

Video Games

Distributing rewards/recognition

Video games have complex participation systems which quantifying how much someone contributed to some accomplishment. For example, a group of players kill a powerful boss - how do you divide the experience or loot? For awhile, video games worked that whoever got the last hit got the rewards, even if another player did 99% of the work until that point. But now games incorporate systems which give rewards proportionally to the amount each player contributed to the effort.

Vote-kick systems

Multiplayer shooters often have "vote-kick" systems that allow the majority of players to kick an annoying player out of the game. While a good idea in principle, more often than not, the vote-kick system itself becomes a social annoyance rather than social boon.

Griefers often call votes to kick innocent people out of the game; or two players will get into a feud and will repeatedly call votes to kick the other player out. Often, no one is kicked because many people choose not to vote -- they don't want to be judges, they just want to play.

...Two specific changes we made to the typical vote-kick system:

If either of the feuding pair calls a vote against the other, the vote is handled differently than with non-feuding pairs -- instead of the possibility that neither feuder gets kicked, a vote between a feuding pair always results in one player getting kicked. This change meant that players learned to not to call a vote in a feud -- unless they're willing to leave the game session rather than continue playing with the other person.

...Furthermore, if two players got into a feud, the feud would end quickly (at the second vote), thus sparing the other players from having to tolerate the toxic environment of two players who are more intent on personal attacks than playing the game. Once bad blood happens, it's just best to separate the parties. - Fixing Online Gaming Idiocy: A Psychological Approach, Bill Fulton


Systems where users comment directly on parts of the articles. Used at Quartz, Medium ("Notes"), experimentally at NYT ("Quips").




Reddit has a simple karma system where having more karma confirms no advantages other than social status.

StackOverflow's karma system provides access to new functionality and privileges - ways of participating (commenting, editing existing content, etc), in addition to making expertise visible.


League of Legends has an "honor" system which is a system for explicitly pointing out positive player behavior, categorized as "Helpful", "Friendly", "Teamwork", or "Honorable Opponent".

Reddit Gold

Reddit Gold has a premium way of recognizing someone's contribution via Reddit Gold, which is a sort of badge people pay for (to support Reddit's servers) and then confer onto others. Because of the financial cost involved, Reddit Gold communicates a clearer signal of approval/quality compared to an upvote.


"Jury duty" moderation

Moderation is like jury duty: You never know when you'll be selected, and when you get it, you only do it for a little bit.

The options available for moderating a given comment:

The criteria for a moderator:

How moderation works:

When a moderator is given access, they are given a number of points of influence to play with. Each comment they moderate deducts a point. When they run out of points, they are done serving until next time it is their turn.

Moderation takes place by selecting an adjective from a drop down list that appears next to comments. Descriptive words like 'Flamebait' or 'Informative'. Bad words will reduce the comments score by a single point, good words increase a comments score by a single point. All comments are scored on an absolute scale from -1 to 5. Logged in users start at 1 (although this can vary from 0 to 2 based on their overall contribution to discussions) and anonymous users start at 0.

Moderators can not participate in the same discussion as both a moderator and a poster. This is to prevent abuses, and while it is one of the more controversial aspects of the system, I'm sticking to it. There are enough lurkers that moderate, that if you want to post, feel free.

Moderation points expire after 3 days if they are left unused. You then go back into the pool and might someday be given access again.

Concentrate more on promoting than on demoting. The real goal here is to find the juicy good stuff and let others read it. Do not promote personal agendas. Do not let your opinions factor in. Try to be impartial about this. Simply disagreeing with a comment is not a valid reason to mark it down. Likewise, agreeing with a comment is not a valid reason to mark it up. The goal here is to share ideas. To sift through the haystack and find needles. And to keep the children who like to spam Slashdot in check.

Metamoderation (M2)

"Regular" moderation is "M1".

...For every moderator out there pushing an agenda, there are usually several good ones making sure that everyone is getting a fair say. To counter unfair moderation, though, we've come up with a system of meta-moderation. ... Metamoderation is a second layer of moderation. It seeks to increase fairness by letting logged-in users "rate the rating" of randomly selected comment posts. ... In order to metamoderate, your account must be among the oldest 92.5% accounts on the system.

For a more detailed analysis of the M1/M2 system's effectiveness, see Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space, Cliff Lampe & Paul Resnick.

The design implications from that study:

...four design goals for distributed moderation systems. First, comments should be moderated quickly. Second, they should be moderated accurately according to the community norms. Third, each individual moderator should have limited impact on any particular comment. Fourth, the burden on moderators should be minimized, to encourage their continued participation.

...In the Slashdot system, two to five people (depending on a comment’s initial score) must provide positive moderations before a comment reaches a score of +4. This limits the impact of any individual moderator.

...At Slashdot, moderators choose which comments to attend to, and only provide feedback on comments that they think should be moved from their current score. This minimizes disruption to moderators’ usual reading patterns. Our analysis showed, however, that it leads to biases. Comments with lower current scores, comments not at top-level, and comments later in a thread received slower moderation and lower scores on average than they deserved. Alternative designs might cause treasures to be discovered more quickly and consistently, at the expense of a little more moderator effort. For example, there could be a special moderator’s view of a conversation. It would hide comments below certain thresh olds, as with the view presented to other readers. But comments the system had flagged as needing additional moderator attention would not be hidden. Recently posted comments and those with recent moderation would be flagged. Once a flagged comment had been presented to enough moderators, the system would infer from the lack of any explicit moderator action that the item was correctly classified and stop highlighting it for future moderators. All comments would reach their final score much faster, and the problems of uncorrected moderation errors and buried treasures would be reduced significantly.



Worth looking at Discourse's Universal Rules of Civilized Discourse as a set of community guidelines.

Exchange 2.0

Waidehi Gilbert-Gokhale of Soliya gave one of the most impressive presentations at the conference. Like a lot of other projects, Soliya aims to build peace through online discussion. Unlike a lot of other projects, Soliya can articulate why conversation alone is not enough. In Gilbert-Gokhale's words: "unmoderated chat polarizes." Here she is referencing a wide body of work that shows that bringing people with conflicting opinions together to talk can actually reinforce pre-existing divisive beliefs, not moderate them.

Soliya sees their online cross-cultural interactions as a new form of "exchange" program and even calls their new platform Exchange 2.0. Online interactions typically take place in a school setting, which gives teachers the chance to moderate and guide the discussion. - What I learned at Build Peace, the first conference for technology and conflict resolution, Jonathan Stray


Reddit allows any user to create a "subreddit", which is a self-contained area which interested users can join and have local discussions in. A link may be shared on a bigger subreddit, and also shared on a smaller one; the nature of conversations may vary vastly across the two, and some users will be attracted to one or the other. This allows users to escape the less appealing parts of Reddit, without ever leaving the site.

Misc. Examples


This subreddit is worth looking more closely at... It's a subreddit dedicated to changing peoples' minds. People present some view that they have, and invite others to show them another way of looking at things.

Other social networks

Federated/decentralized services

Other forums/bulletin boards


Algorithmic Approaches

Reddit's Ranking Algorithm

\[ rating = log(score) - \frac{hours}{12.5} \]

Where \(score\) is upvotes minus downvotes, and \(hours\) is the age of the post.



Structural features of discussions

From tldr: interfaces for large scale online discussion spaces, by Srikanth Narayan.

The application visualizes structures and patterns within ongoing conversations

A typical discussion on Reddit

A typical discussion on Reddit

A discussion where most messages are positively received

A discussion where most messages are positively received

A discussion about a controversial topic

A discussion about a controversial topic

A discussion about a controversial topic

A discussion about a controversial topic

A typical discussion on Reddit

A typical discussion on Reddit

A discussion with mostly one-line messages

A discussion with mostly one-line messages

A discussion with more verbose messages

A discussion with more verbose messages

Temporal distribution of comments in a discussion

Temporal distribution of comments in a discussion

Temporal distribution of comments in a discussion

Temporal distribution of comments in a discussion

Narayan, Srikanth and Cheshire, Coye - "Not too long to read: The tldr Interface for Exploring and Navigating Large-Scale Discussion Spaces". The 43rd Annual Hawaii International Conference on System Sciences - Persistent Conversations Track - Jan 2010

Reputation-Weighted Voting

from Fixing Hacker News: A mathematical approach, Gorgi Kosev.

The system is based on the mathematics described in the beta reputation system, which is a system for measuring trust in online e-commerce communities.

Here is a short description of the system:

Suppose we have a content item \(C\) submitted by the user \(U_c\). Now a voter \(V\) comes to vote for it and clicks on the +1 button.

The voter has his own submissions for which he has received a total amount of positive vote \(p_V\) and a total amount of negative vote \(n_V\). As a result, his voting influence \(i_V\) is modified: its not +1 but calculated according to the formula:

\[ i_V = f_W(p_V, n_V) \]

where \(f_W\) is the lower bound of Wilson score confidence interval. While a simple average such as:

\[ i_V = \frac{p_V}{p_V + n_V} \]

might work when the number of positive and negative votes is large enough, its not good enough when the number of votes is low. The Wilson score confidence interval gives us a better, flexible balance between desired certainty in the result and the result itself.

This vote in turn is received by the content item \(C\). Because its a positive vote, the amount of positive vote \(p_C\) is changed for this content item

\[ p_C \leftarrow p_C + i_V \]

and as a result, it has a new rating

\[ r_c = f_W(p_c, n_c) \]

but the positive points \(p_U\) of the creator of the content item are also changed:

\[ p_U \leftarrow p_U + i_V \]

and as a result the voting influence \(i_U\) of submitter is also changed:

\[ i_U = f_W(p_U, n_U) \]

or in other words, he has "earned" a bigger influence in the voting system by submitting a well-rated content item.

This means that new members have no voting influence. As they submit content and receive votes their influence may rise if the existing users with high influence in the system consider their content to be good.

This is where the reference users \(R\) come in. Their influence is fixed to always be 1

\[ i_R = 1 \]

Because of this, influence propagates through the system from them to other users who submit content which is deemed high-quality by the reference users. Those users in turn also change influence by voting for others and so forth.

Its also possible to scale down votes as they age. The two possible strategies are to scale all \(p_X\) and \(n_X\) values daily, for all content items and all users by multiplying them with a certain aging factor \(k_a\)

\[ p_X \leftarrow k_a p_X \]

\[ n_X \leftarrow k_a n_X \]

or to simply keep all positive and negative votes \(V_p\) and \(V_n\) in the database and recalculate \(p_X\) and \(n_X\) according to the age of the votes \(a_V\), for example:

\[ p_X = \sum_{\forall V_p} { i_V k_a^{a_V} } \]

\[ n_X = \sum_{\forall V_n} { i_V k_a^{a_V} } \]

One of the convenient aspects of this system is that its easy to test-drive. It doesn't require more user action than simple democratic voting. It only requires an administrator to specify some reference users at the start which seed and then propagate influence throughout the system.

Reddit's Comment Ranking Algorithm

As detailed in reddit's new comment sorting system, Randall Munroe:

There's a solution [to path dependence problems in voting], and it's the new 'Best' ranking. When a few people have voted on a comment, you get a rough idea of its quality. The more people who vote on it, the better an idea you get of where it 'should' ultimately end up. With this algorithm, you quantify exactly how much you can tell about a comment from a limited number of votes.

If everyone got a chance to see a comment and vote on it, it would get some proportion of upvotes to downvotes. This algorithm treats the vote count as a statistical sampling of a hypothetical full vote by everyone, much as in an opinion poll. It uses this to calculate the 95% confidence score for the comment. That is, it gives the comment a provisional ranking that it is 95% sure it will get to. The more votes, the closer the 95% confidence score gets to the actual score.

If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom. (Picky readers might observe that some comments probably get a higher rate of votes, up or down, than others, which this system doesn't explicitly model. However, any bias which that introduces is tiny in comparison to the time bias which the system removes, and comments which get fewer overall votes will stay a bit lower anyway due to lower confidence.)

A clarification from Reddit's comment ranking algorithm, possiblywrong

There frequently seems to be confusion about just what information is conveyed by a confidence interval. For example, Evan describes the Wilson score lower bound as the answer to the question, "Given the ratings I have, there is a 95% chance that the "real" fraction of positive ratings is at least what?" This is misleading, since no claim can be made about the probability that the "real" fraction of positive ratings lies within a particular computed interval. (Wikipedia actually does a great job of explaining this.)

The Wilson score confidence interval endpoints formula, which is the approach Randall Munroe describes above, is:

\[ \frac{\hat p + \frac{z^2}{2n} \pm z \sqrt{\frac{\hat p (1-\hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}} \]

where \(\hat p\) is the fraction of observed votes that are positive and \(z\) is the appropriate quantile of the standard normal distribution.

According to possiblywrong, the implementation that Reddit was using (at the time of their article, June 5, 2011), is incorrect, and recommended this implementation:

from math import sqrt

def confidence(ups, downs):
    if ups == 0:
        return -downs
    n = ups + downs
    z = 1.64485 # 1.0 = 85%, 1.6 = 95%
    phat = float(ups) / n
    return (phat+z*z/(2*n)-z*sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)

This implementation corrects some errors in the original, but also ensures that downvotes, when there are zero upvotes, push items down much faster. This may or may not be a desirable property!

Reddit's current implementation is available here.

The original post to which both of the above are referring to is How Not to Sort by Average Rating, Evan Miller.

Say we have positive ratings \(p\) and negative ratings \(n\).

He dismisses the following calculations of score \(s\) :

\[ s = p - n \]

This one neglects the ratio of positive to negative ratings.

For example, if \(p_1 = 600, n_1 = 400, p_2 = 5500, n_2 = 4500\), then \(s_1 = 200, s_2 = 1000\). This approach places item 2 above item 1 since its score is higher, but item 1 only has a 55% positive rating, whereas item 2 has 60% positive.

\[ s = \frac{p}{p+n} \]

This averaging approach falls apart if the number of ratings can vary wildly.

For example, if \(p_1 = 1, n_1 = 0, p_2 = 100, n_1 = 1\), then \(s_1 = 1, s_2 = 0.99\). Item 1 is put first, but clearly this is not what is the desired effect.

Evan suggests using the lower bound of the Wilson score confidence interval for a Bernoulli parameter. Which asks, "Given the ratings I have, there is a 95% chance that the "real" fraction of positive ratings is at least what?" This is detailed above.

Hacker News's Ranking Algorithm

Detailed in How Hacker News ranking algorithm works, Amir Salihefendic

def calculate_score(votes, item_hour_age, gravity=1.8):
      return (votes - 1) / pow((item_hour_age+2), gravity)

It's pretty straightforward - there is a gravity parameter which determines how strong an effect time has on pulling a post's score down.

Hacker News ranking effect over time

Hacker News ranking effect over time

Though this implementation may be dated now.

Approaches against Sybil Attacks


There are two categories of social network-based Sybil defense schemes. The first category, called Sybil detection schemes, operate by detecting identities that are likely to be Sybils. In contrast, the second category, called Sybil tolerance schemes, do not attempt to label identities as Sybil or non-Sybil. Instead, they try to bound the leverage an attacker can gain by using multiple Sybil identities. - Exploring the design space of social network-based Sybil defenses, Bimal Viswanath, Mainack Mondal, Allen Clement, Peter Druschel, Krishna P. Gummadi, Alan Mislove, Ansley Post

The general approach of Sybil detection schemes:

Social network-based Sybil detection schemes rely on the assumption that although the attacker can create an arbitrary number of Sybil identities in the social network, he or she cannot establish an arbitrarily number of social connections to non-Sybil identities in the network. Intuitively, this assumption is rooted in the observation that establishing new social links with honest users’ identities takes some effort, because honest users are unlikely to accept a friend invitation from an identity they do not recognize. - Exploring the design space of social network-based Sybil defenses, Bimal Viswanath, Mainack Mondal, Allen Clement, Peter Druschel, Krishna P. Gummadi, Alan Mislove, Ansley Post

The general approach of Sybil tolerance schemes:

they assume that users perform pairwise transactions (e.g., sending a message, purchasing an item, casting a vote). They achieve a defense against Sybils by assigning credits to the network links, and then allowing actions only if paths with sufficient credit exist between the source and destination of an action. In Ostra, a message can only sent if a path with at least one credit exists between the source and destination; in Bazaar, a item can only be purchased if a path with the item’s price in credits exists between the buyer and seller; in SumUp, a user can only vote if a path with at least one credit exists between the voter and vote collector. - Exploring the design space of social network-based Sybil defenses, Bimal Viswanath, Mainack Mondal, Allen Clement, Peter Druschel, Krishna P. Gummadi, Alan Mislove, Ansley Post

Other papers to look at:


Bazaar is meant for online marketplaces but may have some relevance.

Bazaar creates and maintains a risk network in order to predict whether potential transactions are likely to be fraudulent. The risk network consists of weighted links between pairs of users who have successfully conducted transactions in the past. When a transaction is about to be completed, Bazaar calculates the max-flow between the buyer and seller; if it is lower than the amount of the transaction, the transaction is flagged as potentially fraudulent.

...Bazaar provides a number of useful security properties: First, malicious users in Bazaar cannot conduct more fraud together than they could separately, and as a result,there is noi ncentive for malicious users to collude. Second, malicious users cannot gain any advantage from conducting Sybil attacks, and thus, there is no incentive to create multiple identities. Third, Bazaar explicitly allows users to create as many identities as they wish; this is sometimes a desired feature in online marketplaces, where sellers may own multiple businesses or wish to maintain separate identities for different types of goods. Fourth, Bazaar provides a strict guarantee that each user can only defraud others by up to the amount of valid transactions the user has participated in, regardless of the number of identities the user possesses, thereby bounding the potential damage.

... We view a successful transaction as linking two identities in an undirected fashion, where the weight of the link is the aggregate monetary value of all successful transactions—successfully rewarded shared risk— between the two identities. For example, if identities \(A\) and \(B\) participated in two successful transactions for $5 and $10, there would be an \(A \leftrightarrow B\) link with weight $15. Note that link weights must always be non-negative.

... Bazaar ... allow[s] identities who are not directly connected to engage in a transaction as long as there is a set of paths of sufficient weight connecting them.

...In Bazaar, reputations are a function of both the user who is being asked about as well as the user who is asking...Malicious users who conspire to inflate their reputations do not necessarily increase their reputations from the perspective of non-malicious users.

...New users, by definition, have no transaction history and therefore have a max-flow of 0 to all other users. To allow new users to participate without having all of their transactions flagged as potentially fraudulent, Bazaar uses two techniques. First, Bazaar allows users to create virtual links to their real-world friends (in the same manner as malicious users can create links in the risk network between their identities by conducting fictitious transactions). ...Second, if the new user does not have any real-world friends in the marketplace, Bazaar allows him to optionally provide the marketplace operator with an amount of money to hold in escrow. In return, the marketplace operator creates links between the new user’s identity and other, random identities with a total value of the amount in escrow. This approach does not open up a new vector for attack, as (a) the most the new user could defraud is the amount of escrowed money, and (b) if the user does commit such a fraud, he would lose his escrowed money.

The paper has more details: Bazaar: Strengthening user reputations in online marketplaces, Ansley Post, Vijit Shah, Alan Mislove

Reddit's Story Ranking Algorithm

As detailed in How Reddit ranking algorithms work , by Amir Salihefendic

Reddit's hot ranking uses a logarithm function to weigh the first votes higher than the rest, e.g. the first 10 upvotes have the same effect as the next 100, which have the same effect as the next 1000, and so on

The effect of Reddit's logarithmic weighting of votes

The effect of Reddit's logarithmic weighting of votes

Voting on Reddit without the logarithm scale

Voting on Reddit without the logarithm scale

The story ranking algorithm similarly weighs stories by submission time - newer stories are ranked higher than older.

from datetime import datetime, timedelta
epoch = datetime(1970, 1, 1, tzinfo =

cpdef double epoch_seconds(date):
    """Returns the number of seconds from the epoch to date. Should
       match the number returned by the equivalent function in
    td = date - epoch
    return td.days * 86400 + td.seconds + (float(td.microseconds) / 1000000)

cpdef long score(long ups, long downs):
    return ups - downs

cpdef double hot(long ups, long downs, date):
    return _hot(ups, downs, epoch_seconds(date))

cpdef double _hot(long ups, long downs, double date):
    """The hot formula. Should match the equivalent function in postgres."""
    s = score(ups, downs)
    order = log10(max(abs(s), 1))
    if s > 0:
        sign = 1
    elif s < 0:
        sign = -1
        sign = 0
    seconds = date - 1134028003
    return round(sign * order + seconds / 45000, 7)

Reddit's current implementation can be viewed here.

New York Times "Picks" detection

From The Editor’s Eye: Curation and Comment Relevance on the New York Times, Nicholas Diakopoulos.

Two metrics, article relevance and conversational relevance, calculated like so:

Article relevance was computed by taking the cosine similarity score or dot product of the respective normalized feature vectors for a comment and the article to which it is attached. The higher the cosine similarity score, the more similar the comment is to the article. The notion of conversational relevance measures how similar a comment is to other comments on the same article. Only those articles with 10 or more comments were considered in order to ensure that there was enough of a discussion to produce robust feature vectors . To measure conversational relevance, for each article’s comments a centroid feature vector was created representing the text of all of the comments on the article that were posted before a given comment. This represents the terms used across the thread up to that point in time. Then, for each comment in the thread its cosine similarity to this centroid representation was calculated in order to measure the comment’s conversational relevance.

These metrics appear to be indicators of quality:

The article relevance of the comment is positively associated with a higher chance of it being selected by an editor. ...A comment with a 0.05 article relevance score has approximately a 2% chance of being selected whereas a comment with a 0.40 article relevance score has closer to an 8% chance.

...those comments that editors do select tend to be much more on-topic and relevant to the other comments in the thread. As with article relevance, there is a very strong correlation between the conversational relevance and the rate of editor’s selections.

Modeling the Detection of Textual Cyberbulling

Modeling the Detection of Textual Cyberbulling, Karthik Dinakar, Roi Reichart, Henry Lieberman

On a corpus of 4500 YouTube comments, the researchers applied various binary and multiclass classifiers to automatically detect textual cyberbulling. The best results were with individual topic-sensitive classifiers.

There was cleaning of stopwords, stemming, and removal of unimportant sequences of characters like "@someuser", "lolllll", "hahahahaha", and so on.

The comments were then annontated (labeled) according to the nature of bullying: against sexual minorities/sexist attacks against women, attacks about race and culture, and attacks on intelligence and mental capacity.

For features, they used:

The four approaches used were - Repeated Incremental Pruning to Produce Error Reduction ("JRip"), a two-step process which incrementally learns rules (growing and pruning them) and then optimizing them - J48, a decision tree based classifier - support vector machines - Naive Bayes classifier

They found that "binary classifiers trained for individual labels fare much better than multiclass classifiers trained for all the labels" and that "JRip gives the best performance in terms of accuracy, whereas SMO [SVM] is the most reliable as measured by the kappa statistic."

Scoring based on distributions

Reddit's comment ranking algorithm revisited by possiblywrong is an interesting approach.

Instead of giving each comment a fixed score based on its upvotes \(u\) and downvotes \(d\), we use those values to generate a beta distribution \((\alpha, \beta) = (u+1, d+1)\) and then randomly draw some value \(X\) from that beta distribution each time the comments need to be sorted.

That is, \(X\) is a random variable drawn From the corresponding beta distribution.

So for a post \(k\), it would be:

\[ X_k \sim Beta(u_k+1, d_k+1) \]

So to start, a comment with \(u=0, d=0\) (hence notated \((u, d)\)) will have a uniform distribution (equally likely to be good, bad, or anywhere in between).

As votes accumulate, we have a better idea of the distribution.

The randomness might be too incoherent when it comes too sorting.

possiblywrong suggests using pairwise comparisons like so:

\[ (u_i, d_i) < (u_j, d_j) \Leftrightarrow P(X_i > X_j) < \frac{1}{2} \]

That is, rank comment \(j\) higher than comment \(i\) if it is less than likely that \(i\) should be ranked higher than \(j\).

There are some problems with this approach that possiblywrong brings up.

For instance, if comments have an equal number of upvotes and downvotes, \((u, u)\), then you get situations where the comment's randomized score is equally likely to be higher or lower than the other. So this approach may not yield a total order. But you could arbitrarily order these comments in this scenario.

But possiblywrong has not been able to prove that this even yields a partial order.

And finally, this approach may not be very computationally efficient.

It's an interesting approach but it has to be refined/explored further.

Possible Features

Multiple people ganging up on you to report you in the same game has no effect on whether or not you are banned. We are looking at patterns of behavior over time only. Dota 2 communication reports

These are suggestions for Twitter, from

Suggestions from Clay Shirky's A Group Is Its Own Worst Enemy: have to design a way for there to be members in good standing. Have to design some way in which good works get recognized. The minimal way is, posts appear with identity. You can do more sophisticated things like having formal karma or "member since." ... having some kind of additional accretion so you can tell how much involvement members have with the system.

...when you join that group, your user name is appended with the user name of the person who is your sponsor. You can't get in without your name being linked to someone else. You can see immediately the reputational effects going on there, just from linking two handles.

So in that system, you become a member in good standing when your sponsor link goes away and you're there on your own report. If, on the other hand, you defect, not only are you booted, but your sponsor is booted. need barriers to participation.This is one of the things that killed Usenet. You have to have some cost to either join or participate, if not at the lowest level, then at higher levels. There needs to be some kind of segmentation of capabilities.

...Now, this pulls against the cardinal virtue of ease of use. But ease of use is wrong. Ease of use is the wrong way to look at the situation, because you've got the Necker cube flipped in the wrong direction. The user of social software is the group, not the individual.

...finally, you have to find a way to spare the group from scale. Scale alone kills conversations, because conversations require dense two-way conversations. In conversational contexts, Metcalfe's law is a drag. The fact that the amount of two-way connections you have to support goes up with the square of the users means that the density of conversation falls off very fast as the system scales even a little bit. You have to have some way to let users hang onto the less is more pattern, in order to keep associated with one another.

Sometimes you can do soft forking. Live Journal does the best soft forking of any software I've ever seen, where the concepts of "you" and "your group" are pretty much intertwingled. The average size of a Live Journal group is about a dozen people. And the median size is around five.

But each user is a little bit connected to other such clusters, through their friends, and so while the clusters are real, they're not completely bounded -- there's a soft overlap which means that though most users participate in small groups, most of the half-million LiveJournal users are connected to one another through some short chain.

IRC channels and mailing lists are self-moderating with scale, because as the signal to noise ratio gets worse, people start to drop off, until it gets better, so people join, and so it gets worse. You get these sort of oscillating patterns. But it's self-correcting.

And then my favorite pattern is from MetaFilter, which is: When we start seeing effects of scale, we shut off the new user page. "Someone mentions us in the press and how great we are? Bye!" That's a way of raising the bar, that's creating a threshold of participation. And anyone who bookmarks that page and says "You know, I really want to be in there; maybe I'll go back later," that's the kind of user MeFi wants to have.

From Paul Graham's What I've Learned from Hacker News:

There are two main kinds of badness in comments: meanness and stupidity.

...The most dangerous form of stupid comment is not the long but mistaken argument, but the dumb joke. Long but mistaken arguments are actually quite rare. There is a strong correlation between comment quality and length; if you wanted to compare the quality of comments on community sites, average length would be a good predictor.

...if someone posts a stupid comment on a thread, that sets the tone for the region around it. People reply to dumb jokes with dumb jokes.

Maybe the solution is to add a delay before people can respond to a comment, and make the length of the delay inversely proportional to some prediction of its quality. Then dumb threads would grow slower.

Conceal, don't delete

Some sites completely delete moderated comments. Depending on the severity of the case, moderated comments. Shouldn't always be deleted, but rather, made hidden or less visible (Hacker News greys out downvoted comments). That way it can be voted up again in case the moderation was in error (as it may be with automoderation systems).

Threading or Flat Discussions?

There is some interesting discussion here. I am pro-threading.

I like this idea for making it visually obvious where new replies are located within a thread hierarchy:

the age of the reply affects its color (fades from white to grey) so that you can easily see where fresh new replies are popping up

Front-end anonymity, back-end reputation

See the following from Attacked from Within, by anaesthetica:

...all comments would appear without any indicators of identit comments would appear without any indicators of identity. ... such a system would allow for user identity, but only in private. A user could have an account, but there would be no public acknowledgment of their identity linked to their posts. ...In such a system, there would be a lesser incentive to trolling. Without particular individuals to follow, persistently baiting and harassing individual users would more difficult. The attention whoring and unwarranted self-importance of trolls would be more difficult to sustain in forced anonymity. Of course, this would not prevent more generic trolling (starting flamewars on partisan politics, operating systems, religion), but it would mitigate some of the more abusive forms.

Reputation and other rewards for contribution should also be concealed

Unlike sites that use social status to indicate constructive users, and thereby focus on individual vs. individual comparisons (giving new targets for griefing, trolling, and anti-social behavior), the incentives provided to users ought to be private in keeping with the forced anonymity. - Attacked from Within, by anaesthetica

Incentives should be about producing constructive contributions, not increased control

Incentives for constructive behavior generally revolve around granting users more influence: more moderation power, more prominent comments, more access to control, more influence over the front page. Most of these are fine, but the focus is off: instead of rewarding good behavior with unique opportunities for more constructive contributions, they reward good behavior with opportunities for control, influence, and negation. - Attacked from Within, by anaesthetica

Moderation systems for signaling quality, not agreement/consensus

...moderation systems ought to be geared toward identifying quality contributions, rather than signaling agreement. Current moderation systems are based on the premise that better comments will end up with better scores. This approach is wrongheaded and flawed. As anyone familiar with Digg's wretched comments can attest, clicking 'thumbs up' on a snarky, flamebaiting, or erroneous one-liner signals almost nothing about the actual quality of the comment. Approval voting systems, wherein comment worth is represented by a raw number score, create an "I agree with this post" dynamic to moderation. There is precious little difference between numerical score-based moderation and the Me too!!! posts that began flooding into Usenet in September 1993.

Slashdot is the only major forum with a comment moderation system that takes a step in the right direction. While all of its moderation options are either +1 or -1, they all include some kind of descriptor allowing the moderator to assert why the post deserves a higher (or lower) score: insightful, informative, interesting, funny, offtopic, troll, flamebait, etc. Yet they're still wedded to a score-based moderation system. A set of moderation options that reflected quality rather than "I agree with this post" would be a further step in the right direction. No numerical score ought to be visible. The moderation options would be the descriptions of the comments we'd like to see--informative, informative links, engages parent directly, witty--and of the comments we'd like to see less of--one-liner, personal attack, flamebait, troll, abusive links, spam, offtopic. Options to express agreement could be provided too, in order to prevent the descriptive moderation options from standing in as proxies for agreement (moderators rating comments they disagreed with highly in terms of quality might be given extra weight, assuming they're moderating in good faith). Score-based moderation systems foster groupthink and the promotion of content-less one-liners to the detriment of actual conversation. Moderation centered around what makes a good post provides an institutional foundation for altering the dynamics of users' moderation behavior.

To further emphasize the quality-not-agreement aspect of moderation, scarcity ought to be applied. Slashdot's system of dispensing a few moderation pellets to its users on occasion works on the basis of scarcity, but suffers from being arbitrary and temporally contingent. A moderation system that operates on scarcity could value a user's moderations at a certain weight over a period of time--the more they moderate, the more they dilute their influence. The stock of moderation weight (ranging from pro-ana to rpresser) assigned to each user could vary according to criteria such as length of membership and quality of their contributions. Unlike Digg, where persistent users can set up scripts to digg hundreds of stories a day, thereby rewarding hideously pathetic levels of persistence, a system in which individual influence is scarce reduces the returns to becoming a 'power user.' - Attacked from Within, by anaesthetica

Dichotomous systems of society and community

Kuro5hin provides a good, but limited, model for the emergence of community from society. On k5, interaction in the queue and on the front page constitutes 'society,' whereas interaction in the diary ghetto constitutes 'community.' K5 was the first major forum (as far as I can discern) to provide this kind of separation between social content (for the discussion of the whole userbase) and communal content (diaries with personal content that can be followed on a per user basis). However, mashing all users together into a unified diary ghetto ended in tears, as different subgroups of users came into cultural conflict, and the griefers drove off the beautiful souls to Hulver's diary-only site.

For community to emerge from anonymous society, communal interaction and social interaction ought to be separated as they are on k5. But, whereas on k5 all communal content is placed together, leading to conflict between different communities and daily content overload (back when there were more users), on our hypothetical forum each user's community would be a unique set of neighbor users determined by their prior constructive social interaction. Because each user would have a different set of users constituting their community, the strategy of setting up dupe accounts for the purpose of harassing specific users would be rendered ineffective. Within the community section, it might be possible (even positive) to allow users' to have and display identity markers (username, icon, signature, homepage, etc.), while still maintaining forced anonymity in the society section of the forum. Thus identity would emerge alongside community, allowing affective bonds between users to develop. - Attacked from Within, by anaesthetica

Instead of voting on content, score content based on discussion

Don't let users vote directly on content, as it is with Reddit, rather, score content based on the quality of discussion it is generating.

Even if gaming [this] system could promote a story, it would not capture the top of the front page without being able to sustain users' interest enough to post thoughtful comments in response to the story and to one another. - Attacked from Within, by anaesthetica


These are a lot of random ideas and concepts which I've found useful in thinking about community, but aren't totally directly related.

Individual Rationality => Group Irrationality

Condorcet paradox

Each person has a preference ordering, e.g. one may prefer \(A > B > C > D\), another may prefer \(C > B > D > A\), etc.

We can aggregate those preferences - say in a vote - to see overall what option is most preferred.

In this scenario, the Condorcet winner is the option (or candidate) which beats out every other option head-to-head.

For instance, say we have the following outcome:

# Voters 3 5 7 6
best A A B C
worst D D A A

Here \(C\) is the Condorcet winner since it beats out every other candidate head to head:

However, consider the following scenario (preference is from top to bottom, top being better):

Voter 1 Voter 2 Voter 3

Note that \(X \geq_M Y\) means that the majority prefers \(X\) over \(Y\).

So we have what is called a Condorcet cycle: \(A \geq_M B \geq_M C \geq_M A\), and this is Condorcet's Paradox.

Each participant (each individual) has a rational set of preferences that conforms to transitive preferences, but collectively, their set of preferences is irrational.

The Banality of Evil

A phrase from Hannah Arendt's book on Adolf Eichmann, a Nazi bureaucrat complicit in the Holocaust, but who appeared to be "ordinary", that is, not psycopathic in anyway (who knows if that was actually the case). He was "doing his job".

A good example of how individual rational action - rational as defined by a particular social context - can contribute to a devastating outcome.

The Tragedy of the Commons

The classic example

Hardin, G. (1968). The tragedy of the commons. Science, 162, 1243-1248.

Tyranny of Small Decisions

The Tragedy of the Commons is an example of the tyranny of small decisions, from a 1966 essay published by Alfred E. Kahn. Many small, individual, rational decisions can, in aggregate, lead to non-optimal or undesired outcomes. These effects may be not be realized until it is too late to reverse or choose an alternative course.

Kahn, Alfred E. (1966) "The tyranny of small decisions: market failures, imperfections, and the limits of economics" Kvklos, 19:23-47.

Social Traps

...where men or organizations or whole societies get themselves started in some direction or some set of relationships that later prove to be unpleasant or lethal and that they see no easy way to back out of or to avoid. (pp641)

Converse to a social trap, is a social fence, a "countertrap":

The consideration of individual advantage prevents us from doing something that might nevertheless be of great benefit to the group as a whole. (pp641)

A social fence can also be described as situations where individual action (or inaction) is in the interest of "immediate personal goals or self-interest, even rather weak self-interest" but cause "long-range societal effects which are to almost no one's self-interest" (pp641).

There are also nested traps: essentially, traps that are intertwined with each other (pp650-651).

Social traps & fences can be described in terms of (Skinner) reinforcement: (pp642-643)

Skinner reinforcement: situation or stimulus \(S\), where a subject emits some behavior \(B\), followed by some reinforcement or result \(R\). \(R+\) is a positive reinforcement (reward), makes the initial behavior \(B\) more probable when \(S\) occurs. \(R-\) is negative reinforcement (punishment), making \(B\) less probably when \(S\) occurs.

Platt makes a distinction between short-term \(R\) (\(RS\)) and long-term \(R\) (\(RL\))

In these terms, a social trap can be described as:

\[ S \to B \to RS+ \to RL- \]

That is, a stimulus elicits a behavior and there is a short-term reward (encouraging that behavior) but ultimately causes a long-term punishment.

A fence can be described as:

\[ S \to B \to RS- \to RL+ \]

That is, a stimulus elicits a behavior and there is a short term punishment (discouraging that behavior), but there would be a long-term reward.

"Reversal of reinforcers" can also be applied to traps, where it isn't a comparison of short/long-term, but rather that personal reward/punishment (\(RP+\) or \(RP-\)) equates to group punishment/reward (\(RG-\) or \(RG+\)). e.g. something that rewards the individual is ultimately to the disadvantage of the group.

Three major types of traps/countertraps: (pp642)

One Person Traps (pp643-644)

Subgroups of this type:

Missing Hero Traps (pp644-645)

When group profit (\(RG+\)) is blocked by \(RP-\) for any personal action.

Example: "the mattress problem" - a mattress falls into the middle of the road, everyone tries to drive around it, no one bothers getting out and moving it.

Commons Traps (pp645-646)

Where \(RG-\) follows because of an excessive number of \(RP+\) practitioners

Tragedy of the Commons is this type of social trap.

This type of group trap cannot be solved by one or two heroes

Examples: prisoner's dilemma, Shubik's dollar game (a group of people bid on the dollar, the winner gets the dollar, but the winner and the second-highest bidder must both pay their bids).

Ways out of social traps (pp648-650)

Platt, J. (1973). Social Traps. American Psychologist, 28(8), 641-651. doi:10.1037/h0035723.

Abilene Paradox

A paradox where a group of people collectively decide on a course of action that is counter to the preferences of any of the individuals in the group. This results from a breakdown in group communication where each member mistakenly believes their own preferences are counter to the group's and therefore do not raise objections.

Schelling's Model of Segregation

Think of a neighborhood as a checkerboard:

1 2 3
4 X 5
6 7 8

An agent X evaluates the neighbors in the surrounding squares, to see which neighbors are the same race (or income) as they are. Here, the value will be some number out of 7 (because there are 7 neighbors in this checkerboard), but in general it will be some percentage. The agent will have a threshold value, under which the agent decides to move (for example, if their threshold is 3/7, and only 2/7 of their neighbors are the "same" as they are, then they will move).

If you run the model, the result actually ends up being that, on the macro-level, the % similarity of a given agent and their neighbors is much higher than their threshold value. (for example, their threshold could be 30%, but at equilibrium, the system ends up with 70% of neighbors being similar to a given agent).

So even with very tolerant agents, the macro effect can be high levels of segregation.

However, if agents have very high thresholds, the system never reaches equilibrium because that threshold is too high to satisfy (that is, people will always be moving).

Therefore: micromotives do not need to be equal to macrobehavior.

Peer Effects

General causal mechanisms for homogeneity

Homophily (Sorting)

People who are already similar cluster together (i.e. people tend to be friends with people who are intrinsically similar).

Peer Influence

People who know each other influence each others' behavior and this causes the clustering of behaviors (as their behaviors converge).

Granovettor's Model

Say there are \(N\) individuals. Each individual has a threshold \(T_j\) for a person \(j\). Person \(j\) joins in a movement if \(T_j\) others join. A threshold of 0 means that person joins at the onset.

If no one has a threshold of 0, a movement never takes place.

Collective action is more likely if:

The Standing Ovation Model

This is an extension of Granovettor's Model.

Standing ovation is a peer effect as well as an information effect (that is, if someone sitting nearby appears to know a lot about theater, and they immediately give a standing ovation, you're likely to follow - the fact that they were driven to give a standing ovation communicates some information about the quality of the play that you, as a philistine, might not know).

In this model, we have the following values:


  1. The higher \(Q\) is, the more people will stand.
  2. The lower \(T\) is, the more people will stand.
  3. The lower \(X\) is, the more people will stand.
  4. If \(Q < T\), but there is more variation in \(E\), such that for some people \(Q > T\), then more will stand.


There 1000 people. \(Q = 50, T = 60\).

\(E\) might be very big if:

A more advanced standing ovation model:

This model applies to many different scenarios, including issues discussed by the news.

Sorting or Peer Effect?

Is it sorting or peer effect? That is, did people take that action to be around people like them (sorting), or did they just end up becoming like the people around them (peer effect)?

Example: Suppose there are two groups: AABBAA and BBABBA

With sorting, the Bs in the first group would move to the second group, and the As in the second group would move to the first group, resulting in AAAAAA and BBBBBB

With peer effect, the Bs in the first group would become As, and the As in the second group would become Bs, resulting in AAAAAA and BBBBBB

Thus two different causes lead to the same outcome, making it hard to distinguish what the original cause was.

You can show something is due to sorting if you can present evidence that there is actual movement between groups. If you can present evidence that the people changed, then you can show its peer effects. In either case you need dynamic data.

Heuristics and Biases

There are tons of these, any many could probably be framed in a way relevant to the concerns here. Here is a selection which I think is particular relevant to online discussions.

Fundamental Attribution Error

Attributing someone else's mistakes to internal causes (e.g. flaws of their thinking, personality, etc), as opposed to external causes, and doing the opposite for yourself (attributing your mistakes to external causes, as opposed to internal ones).

Self-Serving Bias

The Fundamental Attribution Error is a self-serving bias, which is any distortion of thought which has the purpose of preserving one's self-esteem. This could also surface as taking more credit for successes and less for failures.

Introspection Illusion

Closely related to Fundamental Attribution Error is introspection illusion, where individuals feel they have an accurate insight into their own motivations and mental states, whereas everyone else does not (that is, assuming that everyone else has a faulty level of insight into their own motivation sand behavior).

Just-World Fallacy

A bias where an individual holds the belief that the world is just, such that if something bad happens to someone, it must have been deserved.

This is basically what motivates victim-blaming.

Dunning-Kruger Effect

A failure in thinking where incompetent people fail to recognize their own incompetence because they are incompetent - that is, they don't have the knowledge or understanding to appropriately evaluate their competence.

Group Attribution Error

A bias where the characteristics of an individual are mistakenly taken as exemplary for an entire group. Like most biases, this happens even if there is evidence to the contrary. A similar bias is the outgroup homogeneity bias, where individuals believe their own groups have greater diversity than others.

Ingroup Bias

A tendency to favor those one perceives to be in their own group.

Group Behavior


When people are in groups, they are likely to make riskier decisions than they would individually, because the shared risk makes them perceive the individual risk as less. There's also the diffusion of responsibility which can help avoid concerns of blame.


Groupthink is a psychological phenomenon that occurs within groups of people. It is the mode of thinking that happens when the desire for harmony in a decision-making group overrides a realistic appraisal of alternatives. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative ideas or viewpoints. (Wikipedia)

Normative conflict model: "strongly identified members are attentive to group-related problems" and may perceive the status quo to be harmful to the collective, in which case they will express dissenting opinions. These strongly identified members can afford the social costs associated with dissent. (pp546)

Packer, D. (2009). Avoiding groupthink: Whereas weakly identified members remain silent, strongly identified members dissent about collective problems. Psychological Science, 20(5), 546-548. doi:10.1111/j.1467-9280.2009.02333.x.

Janis has documented eight symptoms of groupthink:

Janis, Irving L. (1972). Victims of Groupthink. New York: Houghton Mifflin.

Pluralistic Ignorance

Example: students "anticipate negative social consequences for failing to participate in drinking rituals"...but "students were privately less comfortably with alcohol use than they (falsely) perceived other students to be" (pp1010)

"Pluralistic ignorance describes situations where a majority of group members privately reject a norm, but assume (incorrectly) that most others accept it" and thus in public they act like they accept it (pp1010)

The illusion of sincerity:

...most people will enforce the norm, as opposed to merely complying with it, due to social pressure from others in the group, "for whom mere compliance is not enough". That is, mere compliance is interpreted as being an imposter, or a "poser"; you have to have the right motivations as well. (pp1012-1014)

Compliance may be motivated by anxiety, i.e. "the illusion of transparency" - "a tendency to overestimate the ability of others to monitor our internal states", in addition to a tendency to overestimate how harshly others will judge a mishap.

The norms become self-enforcing in that individuals who are posturing will become part of the force of intimidation towards outsiders, in order to appear as if they are true in their convictions (i.e., not posturing), and therefore incite insecurities (anxieties) in the outsider that lead to the compliance behavior in the first place.

In a classic study, Asch (1951) showed that participants would conform to a consensus judgment they knew to be false rather than risk social isolation as a deviant. When participants were assured anonymity, the false compliance disappeared. (pp1011-1012)

Centola, D., Willer, R., & Macy, M. (2005). The emperor's dilemma: A computational model of self-enforcing norms. American Journal of Sociology, 110(4), 1009-1040.

False Consensus

Opposite of pluralistic ignorance is false consensus - it is the incorrect belief that others are similar, when in reality they are not.

There is a tendency for people to assume that their own opinions, beliefs, preferences, values and habits are 'normal' and that others also think the same way that they do. ... This false consensus is significant because it increases self-esteem. The need to be "normal" and fit in with other people is underlined by a desire to conform and be liked by others in a social environment. (Wikipedia)

Social Identity Theory

Having a particular social identity means being at one with a certain group, being like others in the group, and seeing things from the group's perspective. (pp226)

Social identity formation involves self-categorization & social comparison:


Accentuation of perceived similarities between self and in-group and accentuation of perceived differences between self and out-group (pp225).

Social comparison

Enhancing one's self-esteem by evaluating the in-group and out-group on dimensions that lead the in-group to be perceived positively and the out-group negatively (pp225).

Uniformity of perception (within a group) can "be categorized along cognitive, attitudinal, and behavioral lines". (pp226)

Individuals who identify with a group feel a strong attachment to the group as a whole, independent of individual attachments within the group.

In-group identification leads to greater commitment to the group & less desire to leave the group, even when the group's status is relatively low.

Groupthink is much more likely under conditions of high social identification.

Social identification is one of the prime bases for participation in social movements.

People are tied organically to their groups through social identities; they are tied mechanically through their role identities within groups (pp228)

Example: Teacher & student, both part of the "school" group, and both separate roles each with meanings/expectations. At the same time, teacher & student constitute their own social groups (an in/out group pair), focusing more on membership rather than on performance.

Example 2: Husband & wife, both roles with meanings/expectations, but only occasionally constitute an in/out group pair) always and simultaneously occupies a role and belongs to a group, so that role identities and social identities are always and simultaneously relevant to, and influential on, perceptions, affect, and behavior. (pp228)

Group belongingness may be a function not only of self-categorization, but also of assuming a high-status role in the group (pp228)


The central cognitive process in social identity theory is depersonalization, or seeing the self as an embodiment of the in-group prototype (a cognitive representation of the social category containing the meanings and norms that the person associates with the social category) rather than as a unique individual. (pp231)

Depersonalization can also give rise to group cohesiveness, cooperation and altruism, and collective action (pp232)


Another "central cognitive process" in social identity theory. "Seeing the self in terms of the role as embodied by the identity standard...[i]n this process, the person behaves so as to maintain consistently with the identity standard." (pp232)

Burke, P. J. & Stets, J. E. (2000). Identity theory and social identity theory. Social Psychology Quarterly, 63(3), 224-237.

Dunbar's Number

The theoretical upper limit on human relationships. The number is generally thought to be between 100 and 250 (usually 150 is used). This is not the number of that a person can know or be acquainted with, but the number they are actively in contact with. Wikipedia has more info.

Changing Minds

Inoculation Theory

Resistance to an idea (that is contrary to an individual's current beliefs) develops if that individual is exposed to a weakened form of that idea intended to produce attitudinal change, thereby increasing resistance in future encounters with the idea. (McGuire, 1961)

Jacobs Spillovers

Jane Jacobs, the great urbanist and economist, put these ideas to intelligent use in her observation of what made cities such evident crucibles of economic productivity. It was proximities, she said, and networks of proximity, that allowed people to exchange knowledge and creative activities. These "Jacobs spillovers" are a major subject of research in economic science today.

Something similar seems to help explain why cities are such resource-efficient places too. Just as cities promote "knowledge spillovers," they seem to promote "resource spillovers" too, where the waste of one process (say, heat from an energy plant) can become the input of another (say, to heat a building). But these efficiencies can only happen if there is a pattern of proximity, and the possibility of inter-connection. (Frontiers of Design Science: The Network City. Metropolis. Michael Mehaffy and Nikos Salingaros. December 19, 2011.)

Prisoner's Dilemma

Prisoner's dilemma may be a familiar scenario: there are two players, each can cooperate (\(C\)) or defect (\(D\)).

C 4,4 0,6
D 6,0 2,2

Collectively, they're better off if they both cooperate. Collectively, they're both worst off if they both defect.

But there are incentives for each player to defect.

So why cooperate?

Let's say there's a cost of cooperation \(c\), and a benefit to other(s) \(b\). Assume that \(b > c\).

Socially, you'd want to cooperate, because the other's benefit is larger than your cost. But individually, you wouldn't want to cooperate, because your cost is positive

Ways to Cooperation

Repetition (direct reciprocity)

Cooperate if the game may be played many times.

Say \(p\) is the probability we meet again. The payoff to deviate is 0, the payoff to cooperate is \(-c + pb\) (your cost plus the potential payoff of meeting again later). So if \(-c + pb > 0\), you should cooperate.

In other words, as long as \(p > c/b\), you should cooperate.

Reputation (indirect reciprocity)

Perhaps you tell others about how they behaved in the game, affecting their reputation.

Say \(q\) is the probability of someone knowing your reputation. The payoff to deviate is 0, the payoff to cooperate is \(-c - qb\). You should cooperate is \(-c + qb > 0\). In other words, if \(q > c/b\), you should cooperate.

Network Reciprocity

Say we have cells in a network - is it in their interest to cooperate with one another?

Say we have a regular graph (everyone has the same number of neighbors, \(k\)).

If \(k < b/c\) then we're likely to get cooperation.

b=5, c=2, k=2

\(b=5, c=2, k=2\)

In the first example, the benefit of cooperating is 5, cost of cooperating is 2, each node has 2 neighbors.

If you're playing with two cooperators (green), your payoff is 6 (the benefit is 5 for each, so 10 total, minus 2 each for the cost of cooperation, so 10 - 4 = 6).

The person on the edge (the green circle with a 1) sees that the defector to their left has a payoff of 5 (they're not cooperating with anyone, so they have no cost, and the edge person is cooperating with them, thus they have a payoff of 5), and the cooperator to their right has a payoff of 6, so cooperation looks more appealing to them.

b=5, c=3, k=2

\(b=5, c=3, k=2\)

Suppose now the cost of cooperating is 3.

Defectors don't change b/c they aren't cooperating with anyone (thus no change in cost)

Now the edge person sees that it looks better to defect, so they will now change to a defector.

b=5, c=2, k=5

\(b=5, c=2, k=5\)

Suppose now there are 4 neighbors.

Looking at the central person (with the number 7), they see their defecting neighbor to their left has a payoff of 15, and the other 3 neighbors have payoffs of only 12. Thus they will likely defect.

The more connected you get, the greater your incentive to defect.

Someone surrounded by cooperators has the payoff \(k(b-c)\). The boundary defector has a payoff \((k-1)b\). The boundary defector will cooperate if \(k(b-c) > (k-1)b\), which is true when \(b > kc\) or \(b/c > k\).

That is, the benefit of cooperating divided by the cost of cooperating must be greater than the number of neighbors. so as \(k\) increases, \(b\) must increase (or \(c\) must decrease, or both) to keep cooperation the better choice.

From this, we see that, denser ties are better for reputation, whereas denser ties are worse for network reciprocity.

Group Selection

If you think of the game on the level of groups, within the group defectors may do better, but when looking at interactions between groups, groups that have more cooperators fare better.

Kin Selection

Different members have different amounts of relatedness. Say we have a relatedness value \(r\). You should cooperate is \(rb > c\).


Make defective behavior illegal.


Encourage cooperative behavior with rewards or punishments.

Collective Action Problem

This can be thought of a "\(n\)-person" prisoner's dilemma. That is, there is individual benefit to defect, but collectively things are better off if everyone cooperates.

Let \(x_j\) be the cost of the action of person \(j\) (\(x_j = 1\) if they cooperate). The payoff to \(j\) is:

\[ -x_j + \beta \sum_{i=1}^N x_i \]

Where \(\beta \in (0,1)\).


Say there are 10 people, and \(\beta = 0.6\). If 9 people cooperate, and you (the 10th person), choose not to cooperate, your payoff is \(0 + 0.6(9) = 5.4\).

If you cooperate, the payoff for yourself is \(-1 + 0.6(10) = 5\).

Thus it is in your interest to not cooperate.

This is an example of a free rider problem. You benefit if everyone else cooperates and if you don't cooperate.

Common Pool Resource Problem

There is some shared resource that the population uses, which has the possibility of reproducing itself (for example, cows, which may reproduce and make more cows). This is a formulation of the Tragedy of the Commons.

Let \(x_j\) be the amount of the resource consumed by person \(j\). \(X\) is the total amount consumed. The amount available in the next period is \(C_{t+1} = (C_t - X)^2\). So the more people consume now, the less of the resource will be available to reproduce/replenish the next period.


Say \(C_1 = 25, X_1 = 20\). So the next period would be \(C_2 = (25-20)^2 = 25\). So things are in equilibrium.

But if \(X_1\) increases to 21, \(C_2 = (25-21)^2 = 16\), and suddenly consumption (\(X\)) is greater than the available supply.


Satisficing is the behavior where a user will click on a link that might be what they're looking for. If it's wrong, they just hit back and continue looking. This behavior (to the user) is more efficient than reading the whole page to find an exact match. More generally, satisficing (a portmanteau of "satisfy" and "suffice") is sacrificing optimal decision-making for something that's "good enough" (which can end up being close to optimal)

For instance, selecting something that meets the criteria, but might not excel or be the best amongst the qualifying options.

Maybe this behaviour surfaces in other ways as well.

The idea gained popularity due to Herbert A. Simon in 1956:

Simon, H. A. (1956). "Rational choice and the structure of the environment". Psychological Review, Vol. 63 No. 2, 129-138.

The Construction of Networks

Preferential Attachment

Preferential attachment is a process where a node arrives at a network and then must "decide" who to connect to. In the example of Twitter, this would be a new user deciding who to follow.

If Twitter recommends people to follow based on popularity or, if you have already followed some people, based on, say, the average path length from you, then the probability that you (the new node) follow some user (an existing node) is proportional to that users' degree (e.g. how many people they are connected to).

This is a path dependent process; that is, the particular starting conditions and sequence of events have a lot of influence over the current state of things ("extreme sensitivity to initial conditions"). The people which are more recommended end up being more connected and thus are more recommended, and so on.

So you end up with consolidation in the network, which can be visualized as a long-tail distribution (a few super connected nodes, and many, many more more modestly connected nodes).

Random Clique Network

In this kind of network, each node is part of a clique (a cluster of interconnected nodes) and may be connected to some random nodes, which also belong to their own cliques. This structure means that everyone is connected by a fairly short path - e.g. six degrees of separation.

  1. Lauche, K., Postma, C., Stappers, P. J. (2012). Social Theory as a Thinking Tool for Empathic Design. DesignIssues, 28 (1), 30-49.