The case for a vendor-neutral trust dashboard that GitHub and Cursor will never build themselves.
There's a thread on r/programming from a few months ago, 2,038 upvotes, 748 comments, about whether AI coding is "solved." If you scroll past the takes, you find engineers saying something more specific and more anxious: they've stopped using AI assistants on critical paths because they have no way to tell what broke what.
That's not a vibes problem. That's a metrics problem.
Right now, most teams using AI coding assistants are flying blind. GitHub will tell you your Copilot acceptance rate. Cursor won't tell you much at all. Nobody will tell you that one tool modified your test files 3x more than the other, or that PRs from a particular assistant have a statistically higher CI failure rate in the 48 hours after merge. That data exists in your repos. It's just not being collected.
This is the gap that AI Assistant Watchdog / Trust Dashboard is trying to fill.
Here's the awkward truth about AI coding assistant vendors: they are structurally incentivized to show you flattering metrics. GitHub isn't going to surface a dashboard showing that Copilot degrades test coverage over time. Cursor isn't going to publish its regression introduction rate. These are businesses selling you a product, and the metrics they expose are the ones that make the product look good.
So the moment an engineering manager needs to compare tools, or justify spend to a VP, or figure out why their CI pipeline has been misbehaving since they onboarded three new AI assistants, they have nowhere to go. They're stuck building spreadsheets from PR comments and gut instinct.
The idea here is simple to describe, hard to build well: a GitHub App that auto-labels PRs as AI-assisted using commit metadata, then computes per-assistant metrics across your repos. PR acceptance rate, CI failure rate post-merge, test-file modification rate, and a computed weekly trust score. Threshold alerts when behavior crosses a configurable limit. The whole thing installs in 15 minutes, no IDE plugins, no SDK changes.
Teams aren't using one AI assistant anymore. The pattern right now is experimentation: some devs on Copilot, others using Cursor, a few running Claude Code locally. Engineering managers are trying to figure out which tool to standardize on, or whether to standardize at all. That decision is currently made on vibes, blog posts, and whoever argues loudest in Slack.
LLM observability has become a real category. LangSmith and Langfuse are solving this for AI app developers. Nobody has built the equivalent for AI coding assistants at the team level. LinearB and DX have been selling developer productivity metrics to engineering managers for years, but they don't have AI-specific trust scoring, and they don't do regression attribution per tool.
The gap is real. Whether the timing is right is a harder question.
About 50,000 engineering orgs globally have between 50 and 200 developers, according to GitHub's own data. If 30% of them are actively trialing multiple AI assistants right now (a conservative estimate given the current moment), and willingness to pay for observability tooling sits at $150-300/month, the realistic SAM lands somewhere between $225M and $450M. That's enough to build a meaningful business if you can capture even 1% of it.
The pricing structure makes sense: free tier for solo devs to drive installs and word-of-mouth, $99/month for teams up to 20 developers, $299/month for organizations up to 100. At 18 paying customers at $99/month average, you're covering infrastructure and keeping the lights on. That's not an outlandish number.
Unit economics look good on paper. LTV around $2,400 at 85% annual retention, 88% gross margins, CAC payback under two months at organic channels. Those numbers are theoretical until you have real customers, but the structure is there.
No directly comparable YC-funded company exists in this specific niche. That's either an opportunity or a warning sign, depending on your temperament. I lean toward opportunity here, because the reason no one has built this isn't that it's impossible. It's that the AI coding assistant market has only recently matured enough to make cross-tool comparison a real enterprise need.
The adjacent players are telling. LangSmith and Langfuse are focused on LLM observability for application developers, not coding workflows. LinearB is deep on DORA metrics but has no AI-specific layer. GitHub's built-in Copilot metrics give you acceptance rate and nothing else. The incumbents are either too broad or too narrow, and none of them have incentive to build the vendor-neutral comparison layer.
That vendor-neutrality is the actual moat, not just a feature. The data flywheel argument is legitimate: once you have 200+ teams instrumented, the anonymized cross-org benchmark data ("assistants in fintech introduce regressions 3x more than in greenfield projects") becomes something no single-vendor tool can replicate. GitHub can never publish that. Cursor can never publish that. A neutral third party can.
I want to be honest about where this idea gets uncomfortable, because the analysis on this one is unusually self-aware.
Attribution accuracy is the product's core value and its weakest point simultaneously. If you can't reliably determine that a specific regression was introduced by Cursor rather than a human developer who used Cursor as a reference, the trust score is noise. Commit metadata fingerprinting breaks down quickly with squash merges, shared accounts, or custom commit signatures. One viral tweet showing a trust score that incorrectly blamed an assistant for a human-introduced regression could permanently damage the product's credibility. There's no easy fix here. The suggested mitigation (only flag PRs where 70%+ of changed lines match AI commit signatures AND CI fails within 48 hours post-merge, with confidence intervals surfaced to users) is honest and probably the right call. But it also means your trust scores will sometimes be frustratingly non-committal.
The market consolidation risk deserves more weight than it typically gets in these analyses. If Cursor or GitHub Copilot wins the coding assistant market in the next 12-18 months, the structural value of cross-tool comparison evaporates. You'd be building a single-vendor observability tool competing against that vendor's own free native analytics. LinearB survived by going deep on DORA metrics before consolidation happened to them. This product might not have the same runway.
There's also a privacy problem that isn't discussed enough. To attribute regressions to specific AI assistants, you need to read PR diffs and commit contents from customer repos. Enterprise security teams will block the GitHub App installation the moment they understand this, even with good data handling practices. SOC2 Type II certification is probably required before any enterprise sale is possible. That's a 6-12 month detour that makes bootstrap viability much harder than the timeline suggests.
And the GitHub API rate limiting issue is genuinely architectural. At 200 organizations with 10+ active repos each, you'll hit the 5,000 requests/hour limit during business hours when PRs are most active. Solvable, but it needs to be solved from day one, not as an afterthought when your first large customer complains about stale metrics.
The tech stack recommendation is sensible: Next.js, Supabase, GitHub Apps API, Inngest for async job processing, Trigger.dev for CI webhook processing, Resend for email. The 5-6 week solo dev timeline is aggressive but plausible if you've worked with GitHub Apps before and aren't starting from scratch on auth.
Weeks 1-2: GitHub App plus PR labeling. This is the foundation and it needs to be right. The zero-config experience matters enormously for activation.
Weeks 3-4: Metrics computation and the dashboard. The dashboard design will make or break the product. Engineering managers need to be able to forward a screenshot to their VP and have it be immediately legible. Invest in this.
Weeks 5-6: Alerts and billing. The Slack alert integration isn't just a feature. It's a retention mechanism. Once a trust score alert appears in a team's incident response Slack channel, removing it creates a reporting gap that someone will notice.
The suggested validation approach is the right one: Notion doc describing the trust score methodology, three Figma mockups, posted in r/ExperiencedDevs with a direct question about willingness to pay, combined with DMs to 20 engineering managers on LinkedIn who've posted about AI coding tool evaluations in the last 60 days.
Success metric: 5 engineering managers book a 30-minute discovery call and at least 2 say they'd pay $99+ per month before the call ends. That's a reasonable bar. If you can't hit that with a Figma mockup, you shouldn't write the code.
The aha moment the product is targeting is specific and believable: a user sees their first week of data showing one assistant has a 3x higher test-modification rate than another, and forwards the dashboard to their VP. That's the loop. That's what turns a free install into a paying customer and a paying customer into a referral.
The opportunity is real. The timing is plausible. The fatal flaws are genuinely fatal if you ignore them. The thing that makes or breaks this product isn't the dashboard or the metrics or even the pricing. It's whether you can solve the attribution problem well enough that a single bad data point doesn't destroy trust in the trust scores. That's a hard technical problem dressed up as a business problem, and the path to solving it runs through a very large dataset of real customer repos.
If you're a developer who's been frustrated by this exact problem in your own team, this might be worth a month of evenings to validate. If you're looking for a low-risk SAAS idea to build on weekends, the hidden complexity here will surprise you.