What built-in security scans miss in AI-built apps

Short answer: a passing scan means nothing obvious is exposed. It doesn't mean your database rules are right, and it can't tell you whether a stranger can sign up, get your email, log in and pay. Scans that run inside your builder also usually look at your project files when you publish, not at the live site your customers use. Use the scan, then check the four things below yourself.

1. Rules that are switched on, but wrong

Many scans check that row-level security (RLS) is enabled on each Supabase table. Enabled isn't the same as correct. A policy like this is "on" but lets anyone read everything:

create policy "read" on profiles for select using (true);

A policy that only lets people see their own rows looks like this:

create policy "own rows" on profiles for select using (auth.uid() = user_id);

Also check:

Update policies need with check, or a user can change a row so it belongs to someone else.
Views skip the rules of the tables underneath unless created with security_invoker = true.
Storage buckets set to public can be listed or read by anyone who has the address.
Firebase: rules like allow read, write: if true;, or "test mode" rules that only stop working on a date, leave the whole database open.

Your Supabase anon (publishable) key is meant to be public, so finding it in your code isn't the problem. What matters is what that key can read. To see for yourself, on your own app only, ask for a table the way a stranger would:

curl "https://<your-project>.supabase.co/rest/v1/profiles?select=*&limit=1" \
  -H "apikey: <your anon key>"

If that returns real rows without logging in, strangers can read that table.

2. Payments that don't unlock

No scan can see your Stripe webhook settings. A missing or misconfigured webhook doesn't expose anything, so nothing gets flagged, but customers pay and get nothing. How to check and fix it.

3. Email that never arrives

If sign-up needs a confirmation email, your email setup decides whether anyone can join. Supabase's built-in email sender is meant for testing: it has a low hourly limit and, by default, only delivers to people on your project's team. Real customers need your own email provider (custom SMTP), with your domain's SPF and DKIM records set so messages don't land in spam.

4. Whether the live app works at all

A scan run when you publish looks at your code, not at the app on your domain with its real settings, real keys and real email. Logins break after a domain change (fix), environment variables differ between preview and production, and a small change can break sign-up weeks after launch. The only way to know is to go through it like a new customer: private window, new email, sign up, confirm, log in, pay.

A quick checklist

Run your builder's scan and fix what it finds.
Read each table's policies, not just whether RLS is on. Try the curl test above.
Check your Stripe webhook's event deliveries show 200 responses.
Set up custom SMTP and send a test sign-up to an outside email address.
Sign up, log in and pay in a private window on your real domain, and do it again after every big change.

Check your live app in under a minute

AppFirstAid's free check looks at your live site from the outside: leaked keys, files left in public, safety settings, and whether your sign-up, login and checkout pages open. No signup. Journey tests that sign up, log in and try checkout like a real customer are coming soon (email us to hear when).

Run the free check

Settings move around as these tools update. If a menu name has changed, look for the same words nearby. Spotted something out of date? Tell us.