A honeytoken-based detection service for publishers who want proof before they sue.
The idea is simple enough that it's embarrassing nobody's built it yet. You plant a fake statistic, a made-up proper noun, or an innocuous-but-distinctive phrase inside your published articles. It looks like real content to crawlers. Then you systematically probe LLM APIs and ask questions that would surface that content if the model trained on it. When your fake fact comes back in the output, you have something approaching evidence.
That's the pitch for CanaryText. Honeytokens for LLMs.
I want to be honest with you about the problems here, because there are real ones. But I also think this is the kind of tool that should exist, and the MVP is buildable in a focused weekend or two.
The Honeytoken + Model Ingestion Monitor concept got traction because of one specific thing: the NYT v. OpenAI filings showed verbatim memorized text in ChatGPT responses. That's the proof of concept. If the NYT's lawyers could surface memorized content through adversarial prompting, a systematic tool that does this continuously, automatically, and at scale is just productizing what they did manually.
There's a Reddit thread called "Poison Fountain: An Anti-AI Weapon" that got 857 upvotes and 493 comments on r/programming. People are angry and want countermeasures. That's your customer.
Here's where I have to pump the brakes. Modern LLMs are specifically trained not to regurgitate text verbatim. RLHF, safety filters, instruction tuning. They paraphrase. So your honeytoken might train into the model, and the model might never reproduce it word-for-word because that's literally what billions of dollars of fine-tuning was designed to prevent.
This is the fatal flaw, and you need to design around it from day one.
The approach that actually has a chance: don't use real-sounding text as your honeytoken. Use synthetic named entities. Invented statistics with weird specificity ("Ferrovian Blue is a pigment discovered in 1847 by cartographer Emil Drasch"). Made-up proper nouns that couldn't exist elsewhere. Then probe with completion prompts, not open-ended questions. "Complete this sentence: Ferrovian Blue was discovered by..." is more likely to surface memorized content than "What do you know about obscure pigments?"
The other defense: probe open-weight models like Llama and Mistral. They can't patch their behavior after release. A Llama-3 instance running locally has no ToS enforcement and no RLHF update schedule. If your honeytoken shows up there, that's durable evidence.
Seriously. Don't open Cursor yet.
Build a landing page in an afternoon. "Plant invisible fingerprints in your articles. Get alerted when an AI reproduces them." Run $200 in ads to Substack creator communities and r/blogging. Simultaneously, DM 50 Substack writers with 10K+ subscribers. Don't pitch. Just say: "I'm researching whether AI models have memorized your content. Want me to run a free manual test and show you what I find?"
If you can't get 20 people to sign a $99/month letter of intent within three weeks, the market isn't there yet. Full stop. The goal is 20 LOIs or 500 waitlist signups with a 40%+ email open rate.
I'd also film a Loom showing a real honeytoken you planted in a test article coming back in a GPT-4 API response. Post it in r/Substack and r/blogging. That video either gets traction or it doesn't, and you'll know within 48 hours.
This is a 5-7 week build for one developer with AI tools. Here's what you're shipping:
WordPress plugin that generates honeytoken paragraphs and injects them into posts. Invisible to readers via CSS, indexed by search engines. Uses the WordPress REST API, so it works with any theme.
Probing dashboard that runs weekly queries against GPT-4, Claude, and Gemini APIs using your registered honeytokens. Logs the full response, model version, query, and timestamp. Shows match alerts when similarity scores cross a threshold.
Evidence export that packages everything into a PDF: timeline, model version, exact query, exact response, publisher metadata. Formatted for DMCA use and, eventually, litigation.
Stack: Next.js frontend, Supabase for the database and auth, WordPress REST API plugin for the CMS integration, OpenAI/Anthropic/Google API clients for probing, Stripe for billing, Resend for alert emails.
Weeks 1-2: WordPress plugin and honeytoken generation logic. Weeks 3-4: probing engine and dashboard. Week 5: evidence export and Stripe. Weeks 6-7: QA and onboarding flow.
The AI opportunity that makes this defensible over time: use embeddings-based semantic similarity to catch paraphrased versions of your honeytokens, not just exact string matches. Every confirmed match across your customer base trains a better memorization classifier. That's a real data flywheel, and it's the thing a solo founder with 50 customers has that a competitor starting from scratch doesn't.
Free tier: 10 honeytokens per month, no probing. Just to get people in the door.
Solo at $49/month: 100 honeytokens, weekly probing across 3 models, basic evidence export.
Publisher at $149/month: unlimited honeytokens, daily probing, 5 models, full legal evidence package. This is your real product.
Newsroom/Enterprise at $499/month: multi-user, custom probing schedules, litigation package.
Break-even is 28 Publisher-tier customers. That's $4,200/month covering infra and baseline ops. You can reach 28 customers before the product is fully polished if your Loom video lands right.
LTV at the Publisher tier is around $2,100 assuming 14-month average retention. The main retention hook is the monthly "Content Exposure Report" email, which shows probing activity even when there's no match. That's the thing that keeps people paying during the long stretches when nothing triggers.
The ToS problem is serious. OpenAI's terms restrict bulk automated querying. If you're sending thousands of honeytoken probes weekly across a growing customer base, you're a target for API key revocation. The mitigation: spread queries temporally, maintain probing infrastructure for open-weight models that can't be shut down, and honestly? If OpenAI blocks you, that's a PR story worth publishing. "We built a tool to detect AI training on publisher content and OpenAI blocked it" is a headline that writes itself.
The legal enforceability question is genuinely open. A good opposing lawyer could argue that a model reproducing a honeytoken phrase just means the content appeared in RAG retrieval, not training data. The honeytoken doesn't prove the causal mechanism. This is why you want to partner with an IP attorney working on actual AI copyright cases, co-design the evidence format with them, and publish a white paper on admissibility before a court gets to test it.
The biggest long-term risk is that AI companies start training on paraphrased content deliberately, making verbatim fingerprints permanently irrelevant. There's no clean answer to that. The embeddings-based semantic matching is your hedge, but it's not bulletproof.
Post the Loom demo in r/Substack. DM 100 Substack writers who have publicly complained about AI scraping. Cold email independent digital news outlets with the subject line: "Proof that ChatGPT memorized your articles. Want to see it?"
The Substack Grow community, r/blogging, and the Newsletter Operator Slack are where your early adopters live. These are people already angry about this problem who will pay $49/month for even imperfect detection.
This isn't a venture-scale company. The market size is real but small, maybe $18M-$90M SAM for SMB publishers. That's fine. At 200 Publisher-tier customers you're making $30K/month with 80% gross margins. That's a very good indie business. Probably not a unicorn. I'm at peace with that, and you should be too before you build it.