# Security of AI-built systems: what to review in operations

> Security of AI-built systems: a step-by-step review of access, keys and data in the app a team built with AI, so you can decide to keep, fix or shut it down.

Author: José Carlos Maciel, Founder and developer. Published 2026-11-16, updated 2026-11-16.
Source: https://boosting.tech/en/blog/ai-built-systems-what-to-review/

**Quick answer:** Reviewing the security of an AI-built system starts with three questions: where it runs and under whose account, who can get in without logging in or see someone else's data, and which keys and files are exposed in the browser. With those answers, the company can decide to keep, fix or shut the system down before expanding its use.

The operations manager finds out about the system in a routine meeting. The tracking spreadsheet the team used to rely on became an app in a few days, built by an analyst with an AI tool. It has a login screen and reports, and the team has been entering real data into it for weeks. When someone asks where it is hosted, who has the database password and whether the login screen actually protects the internal pages, the room goes quiet.

This article deals with the system that already exists and is in use, with a focus on who can get in and which data is visible. You will find a step-by-step review plan and criteria for deciding whether to keep, fix or shut it down. If your question is still where AI adds value in an enterprise system, start with the article on [AI in enterprise systems](/en/blog/ai-in-enterprise-systems/).

## What is an AI-built system?

An AI-built system is an application whose code was generated, entirely or largely, by an artificial intelligence tool from requests in plain language. The common name for this way of building is "vibe coding": a person describes what they want, accepts the code the tool returns and adjusts based on what shows up on screen, often without reading what was written underneath.

The UK's National Cyber Security Centre (NCSC) published guidance in June 2026 that treats vibe coding as a [spectrum of oversight](https://www.ncsc.gov.uk/blogs/the-vibe-coding-spectrum-approach-to-ai-assisted-software-development). For prototypes, demos or internal tools with no sensitive data, the NCSC considers it acceptable to let the AI work with little supervision. For login and permission logic, sensitive personal data and the handling of passwords and keys, it recommends reviewing what the AI produced, understanding the code, checking for vulnerabilities and confirming it does what you expect.

The trouble starts when a prototype crosses that line without anyone noticing. The app was born to test an idea, it worked, and the team started using it with customer, employee or contract data. The code is still the prototype's code, only now it holds real data.

## Why do these systems reach operations with access flaws?

These systems reach operations with access flaws because the tool delivers what shows on screen first, and access control is exactly the part that does not show. The login screen works, the report opens, and nobody tests what happens when someone skips the screen and asks the database directly.

Recent public reports show this pattern. In March 2025, researcher Matt Palmer analyzed 1,645 projects published by an AI app-building platform and found 170 of them, about 10%, with database tables lacking adequate access rules. The issue was registered as [CVE-2025-48757](https://mattpalmer.io/posts/2025/05/statement-on-CVE-2025-48757/). Through the exposed tables it was possible to read user records, third-party service keys, transactions and subscriptions, and in some cases write new data. The scan only looked at home pages, without logging in.

In February 2026, the security company Wiz reported a similar case at Moltbook, a social network for AI agents whose founder said publicly he had vibe-coded it. Browsing as a regular user, the researchers found a database access key in the code the browser downloads. Because the database had no row-level access rules configured, that key opened [read and write access to every production table](https://www.wiz.io/blog/exposed-moltbook-database-reveals-millions-of-api-keys), including 1.5 million API tokens, 35,000 email addresses and private messages. The flaw was fixed within hours of the report.

Neither case required a sophisticated attack. In both, a rule saying who can see what was missing. That matches what OWASP, the foundation behind the most widely used list of web application risks, published in the 2025 edition of its Top 10: [broken access control](https://owasp.org/Top10/2025/A01_2025-Broken_Access_Control/) remains in first place, and every application tested in the list's dataset had some form of it. In a system built in a hurry, the odds that this layer was left for later are higher.

There is also an organizational factor. A system built inside a business area usually runs on the personal account of whoever built it. When that person goes on vacation or leaves the company, nobody else can get into the hosting dashboard.

## What should you review now and what can wait?

Review now everything that decides who gets in and what is visible; code quality, performance and looks can wait for a second round. The table splits the two groups so the first meeting does not get lost in detail.

| Area | Review now | Can wait |
| --- | --- | --- |
| Hosting and accounts | Whose name it is under, who has admin access | Switching hosting providers |
| Login and session | Internal pages without login, open sign-up, sessions of people who left | Single sign-on with the company directory |
| Database | Queries without a logged-in user, access to other people's records | Data modeling, indexes, performance |
| Keys and passwords | What shows in the browser, what went through a chat or repository | A secrets vault with automatic rotation |
| Uploaded files | Permanent public links to documents and photos | A full retention policy |
| Code | Parts that handle permissions and personal data | Structure, automated tests, conventions |
| Operations | A backup that restores and a log of who changed what | Finer monitoring and alerts |

What goes into the second round does not change the answer to the urgent question: today, can someone outside see or change the data the team enters in this system? If the system handles personal data, the review of screens, reports and logs in the article on [Brazil's LGPD in internal systems](/en/blog/lgpd-in-internal-systems-what-to-review/) complements this one. The LGPD is Brazil's general data protection law, close in spirit to the GDPR.

## Which review plan should you follow for an AI-built system?

The plan below organizes the review in four stages, from inventory to decision. It is written to be worked through by the person who built the system together with someone from IT, without requiring the manager to master every technical detail.

### Review plan for an AI-built system

Go through the stages in order, with the person who built the system and someone from IT in the same room. Each item is a test or a check that someone can mark as done. The plan points to the most common gaps. It does not replace a penetration test or a legal review of the personal data involved.


**Stage 1: inventory**

1. The system has a named owner in the business area and a technical contact who can work on it
2. It is written down where the system runs (hosting, database, domain) and under which account
3. Hosting and database accounts use a company email, with more than one person holding admin access
4. There is a list of what gets entered in the system (personal, financial, customer, operational data)

**Stage 2: who can get in**

1. An internal page opened in a private window, without logging in, sends you back to the login screen
2. With a regular user, changing the record number in the address does not open another person's or unit's record
3. The database refuses queries made without a logged-in user (row-level access rules or server-side queries only)
4. New user sign-up is not open to any email address on the internet
5. Removing access for someone who left the company takes minutes and ends their open session

**Stage 3: keys and visible data**

1. No database password, API key or token appears in the code the browser downloads
2. Keys that have passed through an AI chat, a spreadsheet or a public repository have been rotated
3. Uploaded documents and photos sit in private storage, with no permanent public link
4. Spreadsheet and PDF exports follow the same permission filter as the screen

**Stage 4: before expanding use**

1. A backup exists, and a test restore has been done at least once
2. The AI tool or agent that generates the code has no direct access to the production database
3. The system logs sign-ins and record changes, with who did it and when
4. The decision to keep, fix or shut down is written down, with an owner and a deadline

Run the stage 2 tests for real, with the browser open. Copy the address of an internal page, open a private window and paste it. Sign in with a test user and change the number at the end of the address. Open the browser's developer tools and search the loaded files for words like "key", "secret" or "password". Each failure goes on a list with a screenshot and the profile used, and that list becomes the scope of the fix.

If any stage 2 test fails and the system holds customer or personal data, treat it as an ongoing incident. Restrict access first and investigate afterwards.

## How do you put the review into practice and decide to keep, fix or shut down?

The review fits in one short week, with a written decision at the end.

1. On day one, take inventory with the person who built the system. Write down hosting, database, domain, accounts and who has access. If the account sits on someone's personal email, add a second company administrator the same day.
2. On day two, run the stage 2 access tests with two different profiles. The private-window test and the record-number test take minutes and reveal most of the serious problems.
3. Next, look for keys and passwords in the code the browser downloads and in the history of conversations with the AI tool. Every key that showed up in the wrong place must be rotated at the source service, even if the code has already been fixed.
4. Check the backup by doing a test restore in a separate environment. Many companies only find out the backup is useless on the day they need it.
5. Ask someone who can read code to review only the parts that handle login, permissions and personal data. AI-generated code tends to be long, and reviewing everything at once stalls the process.
6. Bring the business area and IT together and decide based on what you found. The decision goes in writing, with an owner and a deadline.

Three scenarios cover most cases.

- Keeping it makes sense when the access tests passed, the accounts belong to the company, there is a backup that restores and the system holds low-impact data. Even then, name who looks after it and schedule the next review.
- Fixing it is the way to go when the system solves a real operational problem and the flaws are concentrated in identifiable spots, such as a database access rule, an exposed key or a public file link. The fix comes with a short deadline and a new test.
- Shutting it down, or suspending it until fixed, applies when sensitive data is exposed without login, when nobody can take over maintenance or when the code cannot separate who sees what without rewriting its core.

Shutting down has a cost, because the system was solving something. Plan where the data goes and how the team works in the meantime, or the old spreadsheet comes back with the same data and less control.

## What if the system already holds customer data or nobody knows who built it?

The plan changes in a few common situations.

When the system already holds customer data or sensitive personal data and an access test failed, the priority is containment. Restrict access, rotate the keys and preserve whatever access logs exist. Bring in the company's legal team and data protection officer to assess whether data was exposed and what the law requires from there. In Brazil that means the LGPD and guidance from the national data protection authority (ANPD); elsewhere, the local equivalent. The technical review helps answer what was visible and for how long, but the decision on notification belongs to the data controller.

When nobody knows who built the system, or the person has left, start with the domain and the invoice. The domain registration and the hosting bill usually point to the account. Without access to the dashboard, the company does not control the system, and the practical decision becomes moving the data somewhere under its control.

When the system connects to other company systems, such as the ERP, check which credential the app uses for that integration and what it allows. A key with overly broad permissions turns a small app into a doorway to larger data.

When several companies or units use the same system, isolation between them is a requirement. In the systems we build and maintain, such as the [access control system for an industrial complex](/en/case-studies/industrial-access-control/), each tenant company sees only its own records, and the history keeps who approved or denied each access. In an app generated in a hurry, that separation rarely exists in the data model and has to be tested on purpose.

## When is an internal review not enough?

An internal review closes the obvious gaps, but it does not replace an assessment by people who look for flaws every day. It does not cover penetration testing, a full dependency review, architecture design or the legal analysis of personal data. Nor does it solve a system that has grown so much that fixing it costs more than rebuilding it with a clear scope, the subject of the article on [custom software projects](/en/blog/custom-software-project/).

It is worth getting outside help when the system already supports a critical process, holds personal or financial data in volume, or when the review found flaws and nobody in the company feels confident fixing the code. In [support and modernization](/en/services/support-and-modernization/), this work usually starts with access control, keys and backups.

If your company already depends on a system put together in a hurry, boosting.tech, a custom software company, runs a fixed-scope security assessment to point out what to fix first. The result comes in management language, with a list of what to close and in what order. The first conversation is free. Write to jcm@boosting.tech or use the [contact](/en/contact/) page.

## Frequently asked questions about the security of AI-built systems

### Is AI-generated code less secure than code written by people?

It depends on who reviews it. The tool produces code that works on screen, and what tends to be missing are the invisible layers, such as database access rules, server-side validation and care with keys. Code written by people has flaws too. The practical difference is volume and speed: a whole app appears in days, and the review a team would do over weeks never happens.

### How do I know if the app's database is open?

The simplest test is to open an internal page in a private window, without logging in, and see whether it loads data. Then, with a regular user, change the record number in the address. A more complete test needs someone technical to query the database without a logged-in user, using the key that shows in the browser. If the query returns data, the database relies on the screen alone for protection.

### Can I ask the AI tool itself to review the security?

You can use it as support, but not as the only review. The tool that generated the code tends to repeat the same blind spots when reviewing it, and guidance such as the NCSC's asks for someone who understands the code to check login, permissions and sensitive data. A person with security experience, or at least a second tool with specific instructions, raises the odds of catching what slipped through.

### What should I do first if I find an exposed key?

Rotate the key at the source service, whether it is the database, the email provider or a paid API. Removing the key from the code is not enough, because it may already have been copied. Then check the service's logs for unusual use and move the new key to the server, out of the code the browser downloads. If the service holds personal data, let your data protection officer know.

### Does banning AI tools in the company solve the problem?

Rarely. A ban tends to push use into personal accounts, away from any control, and the systems keep appearing. Clear rules work better: prototypes are fine, real data only after review, accounts always in the company's name and a technical owner for every system that goes into use. That way the business area keeps solving problems quickly and IT knows what is running.

## References

- [Wiz: Hacking Moltbook, AI social network reveals 1.5M API keys (2026)](https://www.wiz.io/blog/exposed-moltbook-database-reveals-millions-of-api-keys)
- [Matt Palmer: Statement on CVE-2025-48757 (2025)](https://mattpalmer.io/posts/2025/05/statement-on-CVE-2025-48757/)
- [OWASP Top 10:2025, A01 Broken Access Control](https://owasp.org/Top10/2025/A01_2025-Broken_Access_Control/)
- [NCSC (UK): The 'vibe coding spectrum' approach to AI-assisted software development (2026)](https://www.ncsc.gov.uk/blogs/the-vibe-coding-spectrum-approach-to-ai-assisted-software-development)
