Back to projects

BeeBOT

Used by real people

The Discord bot for the BeemodTools community. It catches scams with a fine-tuned DistilBERT model and triages BeePEE crash reports with an AI agent.

BeeBOT runs the BeemodTools Discord server and has two main jobs: catching scams and triaging BeePEE's crash reports. It's about 12k lines of Python, running on a live server.

Scam detection

Every message goes through a layered pipeline:

  • a DistilBERT classifier I fine-tuned (97.9% accuracy, 0.979 F1)
  • GPT-4o for the uncertain cases
  • perceptual image hashing plus OCR to catch scam screenshots
  • a cross-channel spam detector that auto-mutes anyone posting the same thing in several channels

It's smart about who it scans. Mods skip it, active long-time members skip image scans, and watchlisted users always get checked. The model keeps improving: mods confirm or reject flagged messages with buttons, and half of GPT's verdicts are automatically added to the training data.

Crash reports

BeePEE sends crash reports to an HTTP endpoint on the bot, which is rate-limited and filters out junk reports. A fast model first checks whether the bug was already reported, and merges it into the existing thread if so.

Otherwise an agent using OpenAI tool calling reads through the user's .bpee package file by file to find the likely cause. It then opens a forum thread tagged with the app version, and mods can close it with /resolve.

The rest

It also handles the usual moderation (timed mutes and bans that survive restarts, purge, softban, logging). It runs on SQLite locally and Postgres in production on Railway.