Facebook’s AI-Generated Spam Problem Is Worse Than You Realize
The platform has been overrun with bizarro images of babies, soldiers, and muscle-bound Jesus
The picture, posted July 4 on the Facebook page “Love Shares 3.0” for 71,000 followers, appears to be an aerial view of St. Peter’s Basilica in the Vatican, with people gathered in the square. Overhead, dangling from a gigantic black helicopter, is what can only be a massive Bible, but the lettering on the book is garbled, and the cross on the cover has an extra arm. It looks as if the aircraft is about to drop this tome on the crowd below, flattening them. The caption is pure gibberish: “Close your eyes 70% and see magic / Today’s my graduation / May 2024 is Your Best Year.” Additional hashtags identify the image as “art” and “painting.”
It is, quite clearly, AI-generated — though nobody in the comments mentions this. “Praise the lord,” writes one user. Many others reply with a simple “amen.” The image has close to 6,000 likes and “heart” engagements.
In the early 2010s, Facebook reshaped digital life as we know it. But in the past few years, a confluence of trends has left it uniquely vulnerable to click-farming pages that churn out AI-created junk. At a critical moment when online creators are weighing the benefits of integrating controversial AI tech into their personal brands, a shadow army of spam “creators” have already leveraged it to invade a platform mostly abandoned by such internet celebrities, eating away at whatever social value it has left. Worst of all, the very structure of Facebook appears to have encouraged this rot.
One factor, of course, is the rise of text-prompt image generators like Midjourney, Stable Diffusion, and DALL-E, which make it profoundly easy to create “original” works at a terrific rate. Facebook, meanwhile, was investing billions in CEO Mark Zuckerberg‘s “metaverse” boondoggle, and still has no real user base to show for it: Gen Z famously disdains the social network as uncool, preferring TikTok and the also Meta-owned Instagram, leaving Facebook with an aging demographic.
If the AI-reliant spam pages are any indication, many of those still scrolling Facebook can’t tell or don’t care when an image is fake, and have a particular fondness for certain comforting signifiers: Bibles, babies, American flags, soldiers, animals, luxurious homes, landscapes, and Jesus Christ. Then there are the images meant to evoke pity: patients in hospital beds, crying or endangered children, amputees, the unhoused, the starving. Sometimes the imagined figure is shown holding a sign asking for birthday wishes, or explaining that they’re a veteran. And in certain cases, there’s no earthly explanation for what you’re looking at — like this military truck that seems to be transporting giant carrots, but is also made out of them:
Editor’s picks
Theodore, a 19-year-old who lives in Paris, has become the top curator of what he calls “Insane Facebook AI slop,” and shares his favorite examples on a dedicated Twitter account with more than 100,000 followers. On that site, it’s customary to mock the content, as well as the gullible Boomers assumed to be eating it up. There’s even an AI slop bingo card meme that followers can use to keep track of some common tropes, including the generic captions most often appended to the images, like: “Why don’t pictures like this ever trend” and “You will never regret liking this photo.” (Scarlett Johansson‘s name also shows up a lot, for some reason.)
“I don’t use Facebook a lot myself but saw plenty of screenshots of those types of posts on Twitter, and the insane amount of likes these had made the phenomenon funnier,” Theodore tells Rolling Stone. “So I thought I might as well document those.” At this point, he almost exclusively shares content he receives in DM submissions.
Theodore believes the success of AI slop is mostly due to the elderly and naive. “Facebook is full of old people and tech-illiterate people in general,” he says, which is “also why the kind of posts that get the most likes are the ones that will target old people, using soldiers, American flags, Jesus imagery, etc.” Since some themes are so dominant, he tries to maintain variety, lately branching out into illustrations of “third-world children doing all sorts of impossible crafts.”
Meta has taken a rather gentle approach to AI-derived images and videos, opting to add an “AI info” label “when we detect industry standard AI image indicators or when people disclose that they’re uploading AI-generated content,” it said in a statement this summer. This label did not appear on any of the AI images shared by more than a dozen engagement-farming pages reviewed by Rolling Stone. As for more stringent moderation, Meta’s Oversight Board has argued that they “unnecessarily risk restricting freedom of expression when we remove manipulated media that does not otherwise violate our Community Standards.”
That is to say, because the typical AI slop isn’t strictly misinformation (nor is it hate speech, graphic nudity, and so on), there’s no glaring reason to delete it — unless Meta were to deem it spam. The company’s policy states that it does not allow spam, which it defines as “content that is designed to be shared in deceptive and annoying ways or attempts to mislead users to drive engagement.” The AI slop pages are certainly deceptive, repetitive, and gaming the system for engagement; there’s also no telling how many are monetized and profiting from an abundance of reactions and comments.
Meta tells Rolling Stone that while “eradicating spam is a nearly impossible task,” the company takes significant measures against it, since this type of content “detracts from people’s ability to engage authentically in online communities.” In the first quarter of 2024, it reported, Meta “removed 436 million pieces of spam content from Facebook,” with 98.2 percent of this “actioned before it could be reported.”
The question remains whether Meta would deem the AI-generated images to be spam in violation of their policy, or just take the engagement as a sign to feed those users more of the same. The pseudonymous data researcher and software developer Conspirador Norteño in June published a Substack investigation that suggested the latter: that Facebook’s recommendation algorithm is favoring the AI slop over authentic content.
After logging in with a dummy account he had only ever used to shop for music gear, Conspirador simply scrolled through more than a thousand posts on his feed over half an hour and collected information on both the content and the pages that shared them. He found that less than 5 percent of the material corresponded to his past browsing of music equipment, 12.7 percent was sponsored ads, and the rest was “mostly a mix of AI-generated images and plagiarized photographs posted by content aggregator accounts with large numbers of followers.” AI-generated content accounted for 22 percent, or almost a quarter, of all the posts he viewed.
“I think a lot of people still operate with the expectation that images that look like photos are real photos, and don’t notice that they’re really looking at an AI image unless there are really obvious problems,” Conspirador says. He noted the prevalence of pages such as “Log Cabin Living,” “Mountain Cabins,” and “Barndominium Gallery,” which tend to feature images of lavishly appointed country mansions set in lush nature scenes — pictures that often don’t immediately betray signs of being fake. These reliably pick up likes, comments, and subscribers. “Unusual-looking houses and pretty outdoor scenes tend to do well, even when an account posts dozens or hundreds that are basically the same,” Conspirador says. A typical “Log Cabin Living” post features a preposterously large house with an uncanny pool that seems to blur the distinction between artificial and natural bodies of water:
People gazing with admiration at fantasy houses that don’t exist is one thing, and a social media algorithm amplifying this stuff is another. But Conspirador was particularly surprised by “Facebook’s failure to intervene in cases where the account [was] obviously hijacked” from an ordinary user before it began spewing AI spam. This was the situation with “Barndominium Gallery,” a page which he found had previously belonged to a hair salon in Oklahoma. In January, the owner of the salon posted that her page had been hacked and asked friends to report it to Facebook, yet eight months later, no action has been taken, while the page has accrued hundreds of thousands of followers with its AI rubbish. Worse, whoever stole the page has begun promoting links to a fake home-construction business that charges the customer $79.99 for floor plans and cost-of-build estimates on its AI-generated architecture.
Carissa Véliz, an associate professor of philosophy at the University of Oxford’s Institute for Ethics in AI, views the slop takeover as the logical result of how Facebook works. “This phenomenon seems to be a secondary effect of social media’s focus on engagement,” she says. “Instead of focusing on fostering wholesome relationships, or in being trustworthy sources of information, social media has focused on gaining people’s attention. And it turns out that what attracts most of our attention is akin to junk food.”
“Generative AI was designed to engage people,” Véliz says. “It’s good at producing images that people find captivating.” Facebook, she points out, needs that content that keeps people glued to the app, and may tolerate clickbait because they’re concerned that “users were no longer sharing as many personal stories as before.” What’s more, Véliz proposes, companies like Meta are incentivized to create Frankenstein’s monsters beyond their control, as the “inadequate current regulation doesn’t demand of social media companies anything that can be too burdensome.” Therefore, if you have a system running amok, but fixing it would be too difficult, “it magically absolves you from its effects.”
In the meantime, spectators like Theodore and his audience will have plenty of absurdities to laugh at, from fish with legs to Incredible Hulk porn to phony black-and-white “historical” photos. He’s also started pulling material from YouTube and X/Twitter, which are hardly immune from the same trend. But perhaps not all is lost just yet. Theodore doesn’t usually look at the comments on Facebook AI slop, since there’s no need to read “amen” a hundred times in a row. When he does, however, he’s treated to the occasional surprise: “Someone calling out the post for being made by an AI.”