We’re excited to announce beta access to customizable attributes for Perspective API. Customizable attributes enable users to provide their community’s guidelines in natural language and receive a score that reflects whether any given comment was consistent with them. This capability, built with the latest Gemini models, empowers any user, from mods to other community members, to highlight the comments they care about, which they can then label or analyze according to their own needs. We hope our customizable attributes will complement Perspective API’s existing pre-defined attributes to enable an approach that combines broad coverage with fine-grained management.
Enabling these new attributes could, for example, allow mods for a community focused on local information to more easily separate out concerns of local residents from those of tourists, or an advice community experiencing rapid growth to maintain the engaging and supportive quality of its posts, without overwhelming the members working to manage the influx of new posts.
Because we still have a lot to learn, we’re inviting developers and researchers to join us in a limited beta program to experiment with these customizable attributes. This collaborative approach will allow us to provide technical support, gather valuable feedback, and iterate on our research in real-world scenarios.
The challenges for online communitiesMany platforms, publishers, and researchers deploy Perspective API every day, often as a small component of a comprehensive effort to enable flourishing online conversations. But, though users are able to set their own thresholds for our pre-defined attributes, they fall short for many communities that have unique norms or content. To better understand these needs, we worked with ReD Associates to interview dozens of people who manage and participate in online communities across 13 such communities on Reddit, Discord, Slack, Bluesky, and Mastodon (publication forthcoming). A clear theme emerged: communities considered themselves most “healthy” and “thriving” when members and community managers, called “mods,” have a shared understanding of the community’s unique norms.
Crucially, we saw that these differing community norms also lead to different kinds of challenges. For example, some communities, like local groups, exist to share resources. In one such Reddit group, we saw that during a hurricane, it became critical for them to quickly filter crucial updates from an overload of irrelevant content. Other communities are places to discuss common interests. For example, in a firearm enthusiast community, mods noted the need to carefully weigh legitimate hobby discussion with discussion of firearms with illegal intents to harm others. Beyond hobbyist topics, other communities exist to facilitate authentic human connection on sensitive subjects, as in the case of a mental health forum that bans AI-generated responses mimicking genuine human support.
Overall, mods felt hampered by the sheer volume of posts and routine tasks needed to keep their communities safe and relevant for their members. “It’s a lot of drudgery,” one moderator admitted, leaving less time to use their expertise and experience to improve their community and foster meaningful interactions. This work can also carry a heavy emotional burden, leading to burnout. As one mod of a mental health community said: “it’s really heavy work. You hear about suicidal plans, graphic details of abuse, you see pictures…and there is so much of it. People burn out.”
Across communities, mods struggled to manage conversations using current approaches like keyword and phrase-matching which missed nuanced gray areas or violations of their unique community guidelines. For example, in a city subreddit with a “locals only” rule, a question about the best pizza places in the city center actually violates the rules against tourist-related content! In another community, a post containing violent language like “stabbing” is, in fact, acceptable in the context of friendly banter about a video game. When envisioning their ideal future, mods described a workflow with “less manual labor [and] the ability to modify AI to your community” to help them make decisions with their own context in mind.
While many platforms, like YouTube and Discord, are already developing their own native tooling to improve these workflows, we saw an opportunity to offer greater flexibility and choice across all communities and platforms.
What’s nextAhead of this beta release, we’ve tested our implementation with fictional guidelines based on real-life examples from online communities. The model tended to perform better on guidelines around harmful or illegal content, where we saw F1 scores as high as 0.95. Meanwhile, the model scored more modestly on guidelines that identified topics or categories, ranging from 0.62 to 0.70. The model struggled most with highly contextual guidelines, for which the model — or even a person — might require more information than what’s available solely in the comment.
Looking ahead, we aim to deepen our collaboration with partners to improve how our models perform in real-world contexts. We will also continue to publish updates on our progress, particularly related to our explorations around evaluating open-ended LLM tasks.
Customizable attributes are a step to empower communities and those who moderate them. As one mod put it, “We don’t want algorithms running the show. At the same time, I think most of us, who’ve been doing it for a little while, are like, we could use a few algorithms that help with the manual tools.” We agree and are glad to place these tools in the hands of the communities themselves, without sacrificing what’s working already.
If you’d like to explore the possibilities for LLMs to support online conversations, please join our beta program to experiment with these customizable attributes today.
By Lucas Dos Santos, Tech Lead; Emily Saltz, Sr. UX Researcher; Tin Acosta, Sr. Product Manager
We’d like to acknowledge and thank all of the community mods who participated in our ethnographic research and codesign workshops, including Manley, AngelRose, paiuxfluens, mary-anns-hammocks, WizardMama, yellowmix, Lexicaleigh, and many others.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.3