Moderation

On-device content safety for an app that shows what people post, written as a local Swift package and shipped as a Granite agent package. One call writes Packages/ModerationKit: a keyword filter, an image-safety gate on a Core ML classifier the app provides, a policy the app persists, a Moderator that judges each piece of content into allow, blur or hide with the reason, a SwiftUI censor overlay and a keyword list editor, with their tests; then links it into every app target of the project in the folder.

It is one of the Rao Craft Agents: independent packages that, tagged in prompts along the life of a project, build an app of Loom's quality. moderation generalizes Loom's ModerationKit, its NSFW Shield and its keyword filter.

How an app finds it

ModerationAgent is a GraniteAgentPackage, and it carries a record in its binary:

@used @section("__DATA_CONST,__granite")
let moderationRecord: GraniteRoster.Record = { GraniteRoster.record(ModerationAgent.self) }

In Craft, install it from the Agent Store or with craft agents add https://github.com/Rao-Craft-Agents/ModerationAgent.git, then say $moderation in a request:

$moderation keep spoilers and crypto spam out of the recipe feed, and blur food photos the model flags

The model writes the call; it shows on Craft's agent map as moderation-1, and the files it names are built and tested with the turn.

What a call takes and answers

{}
{"keywords": ["spoiler", "store-bought", "crypto*"], "threshold": 0.7}
{"modelURL": "https://models.example.com/food-photo-safety.mlmodel", "label": "unsafe"}
{"features": ["keywords", "editor"], "keywords": ["spoiler"]}
{"name": "RecipeSafety", "into": "Packages/RecipeSafety", "link": false}

The answer is a ModerationResult: every file written (path, paths, absolute, the project or manifest it edited among them), the package's name and folder, its parts, where it was linked, the files an earlier call wrote that this one left out (removed), and a sentence for anything it changed beside the package, like a deployment target raised.

What it writes

Moderator is an actor: judging never runs on the main thread, a verdict is kept by the content's id while the content and the policy stay the same, and an image that cannot be screened never hides content. No model ships in the package.

How the package is made

No model is asked: the same call makes the same files every time.

Testing

swift test                       # the record, the schema, the guide, every set of parts, both linkers, a call through the roster
MODERATION_BUILD=1 swift test    # the packages built and tested, for macOS and the iOS simulator, linked into an app and a package

The gated test builds and tests the whole package and three smaller ones, builds it for the iOS simulator, builds an app project it is linked into for macOS and iOS, and runs a tool in a package it is linked into that scores images with a real (tiny, made-for-the-test) Core ML classifier: downloaded, compiled, kept, and loaded again from the cache.

MIT licensed.