Three production lessons from shipping Strummy

Strummy is a CRM I run for guitar teachers — student rosters, lesson scheduling, song repertoires, practice tracking. It has ~25 daily active users. That's a small number. It's also the number where the gap between “side project” and “production system” becomes legible. Three lessons from the last few months that wouldn't have surfaced at zero users.

Mocked Supabase tests don't test RLS

Strummy is multi-tenant: each teacher has students, lessons, and songs that no other teacher should see. Postgres row-level security enforces that. The Supabase client just attaches the teacher's auth token; the database does the work.

I had a healthy integration test suite. I also had a __mocks__/@supabase/supabase-js.ts file that aliased the entire client to a hand-rolled stub. Every test that looked like it was exercising RLS was actually talking to my stub.

The fix wasn't to delete the mock — unit tests benefit from one. The fix was to add a separate test track that talks to a real local Supabase, with no mocking, and run it before any change to RLS or auth flow.

const config: Config = {
  testEnvironment: 'node',
  testTimeout: 30_000,
  // No moduleNameMapper alias for @supabase/supabase-js.
  // Crucially, this config DOES NOT mock the client — every other test
  // setup in this repo aliases it to a stub, which would defeat the
  // purpose of testing RLS policies.
  testMatch: ['<rootDir>/**/*.rls.test.{ts,tsx}'],
  maxWorkers: 1,
};

Two configs (jest.config.ts for unit + integration with mocks; jest.config.rls.ts for RLS-real). Two npm scripts. Tests opt into one or the other by filename suffix. The RLS suite runs against a service-role-keyed local Supabase, seeds two teachers with overlapping student names, and asserts that each teacher's queries return only their own rows.

The lesson is older than this project. It's still the one I see violated most often: a mock that's faithful enough to make tests pass is faithful enough to hide the bug you needed the test for. Decide which boundary the test is supposed to cross, and don't quietly stub past it.

A schema rename will quietly break your dashboard

Early in Strummy I had a user_roles table joining users to a role enum. Later I collapsed it into a role column on profiles — simpler, one less join, fewer RLS policies to maintain. The migration ran clean. The app worked. Tests passed.

Two weeks later the admin dashboard started throwing 500s. The “Active Students” metric on the homepage was querying the dropped user_roles table. The query had been written months earlier, lived in a single API route, and was never touched by the migration PR.

The migration was reviewed. The dashboard wasn't.

The fix was twofold. First, the obvious patch — replace the dead query with a profiles query that returns the right shape. Second, the real fix — define what “active student” actually means as a first-class field, not a query everyone re-derives.

alter table profiles
  add column student_status text
  check (student_status in ('active', 'paused', 'churned'));

-- daily cron flips status to 'paused' after 28 days without a logged lesson

Now Active Students is a single column query maintained by a daily cron job. No joins, no business-logic-in-SELECT, no chance that the next schema rename leaves it stale. When the underlying tables change, the cron breaks loudly in one place, not silently in five dashboard tiles.

The lesson: derived metrics that show up in admin UIs deserve to be persisted, not re-computed in every read path. The performance argument is real but secondary; the real win is centralizing the definition. There's exactly one place that knows what “active” means.

Three CI providers building the same code is two too many

Strummy ships through Vercel. I had Vercel's git integration enabled. I also had a GitHub Actions CI workflow that ran tests, linted, and on main triggered a Vercel deploy. I also had an automated post-merge workflow that bumped the version and pushed a commit, which itself was a push to main, which triggered Vercel again.

A single PR merge produced three to four Vercel builds for the same commit range. None of them caught anything different. They burned build minutes, race-conditioned each other on Vercel's “current deployment” pointer, and meant the URL strummy.app could be serving any of three commits depending on which build won.

 // vercel.json
 {
+  "git": {
+    "deploymentEnabled": false
+  }
 }

 # .github/workflows/ci-cd.yml — removed
-deploy-preview-main:
-  needs: test
-  if: github.event_name == 'push' && github.ref == 'refs/heads/main'
-  ...
-deploy-production:
-  needs: test
-  if: github.event_name == 'push' && github.ref == 'refs/heads/production'
-  ...

The new shape: PR previews still deploy automatically (the only deploy that matters during code review). Production deploys are a manual workflow_dispatch from the Actions UI — one click, one build, one canonical commit on strummy.app. Build minutes dropped to a quarter of where they were. The URL means what it says.

The lesson is dull but worth saying: automation that fights other automation costs more than you think. Each tool was reasonable in isolation. The bill came from the overlap. Audit your deployment graph the same way you audit your dependency graph — the second build is a liability, not a backup.

What they have in common

Three lessons, three different parts of the stack — testing, schema, CI. The connective tissue is that none of them surface at zero users, on a side project, or against a mocked test environment. They surface when there's a real client running real queries, when migrations have to be safe in production, when build minutes show up on a bill.

That's the underrated thing about running something small in production: 25 users is enough to make every shortcut visible. Not enough to justify infrastructure spend, but enough to make sloppy testing, sloppy schema, and sloppy CI cost you something concrete every week. Side projects let you avoid those bills. Production won't.

Strummy is the project on my CV that other engineers read first. Not because it's the most ambitious — it isn't — but because the questions it raises are the ones that recur on every real system you'll ever maintain.