Security of AI-built systems: what to review in operations

Security of AI-built systems: a step-by-step review of access, keys and data in the app a team built with AI, so you can decide to keep, fix or shut it down.

01 Inventory Where it runs and under whose account Accounts and data 02 Access Who gets in without logging in Browser tests 03 Keys and data Secrets and files in plain sight Browser code 04 Decision Keep, fix or shut down The builder and IT in the same review, before expanding use AI-built systems

Quick answer

Reviewing the security of an AI-built system starts with three questions: where it runs and under whose account, who can get in without logging in or see someone else's data, and which keys and files are exposed in the browser. With those answers, the company can decide to keep, fix or shut the system down before expanding its use.

What you will find

  • What counts as an AI-built system and why it reaches operations without review
  • What recent public reports show about open databases and broken access control
  • What to review now and what can wait for a second round
  • A step-by-step review, from inventory to the decision to keep, fix or shut down
  • When an internal review is not enough and it pays to bring in people who do this every day
On this page
  1. What is an AI-built system?
  2. Why do these systems reach operations with access flaws?
  3. What should you review now and what can wait?
  4. Which review plan should you follow for an AI-built system?
  5. How do you put the review into practice and decide to keep, fix or shut down?
  6. What if the system already holds customer data or nobody knows who built it?
  7. When is an internal review not enough?
  8. Frequently asked questions

The operations manager finds out about the system in a routine meeting. The tracking spreadsheet the team used to rely on became an app in a few days, built by an analyst with an AI tool. It has a login screen and reports, and the team has been entering real data into it for weeks. When someone asks where it is hosted, who has the database password and whether the login screen actually protects the internal pages, the room goes quiet.

This article deals with the system that already exists and is in use, with a focus on who can get in and which data is visible. You will find a step-by-step review plan and criteria for deciding whether to keep, fix or shut it down. If your question is still where AI adds value in an enterprise system, start with the article on AI in enterprise systems.

What is an AI-built system?

An AI-built system is an application whose code was generated, entirely or largely, by an artificial intelligence tool from requests in plain language. The common name for this way of building is "vibe coding": a person describes what they want, accepts the code the tool returns and adjusts based on what shows up on screen, often without reading what was written underneath.

The UK's National Cyber Security Centre (NCSC) published guidance in June 2026 that treats vibe coding as a spectrum of oversight. For prototypes, demos or internal tools with no sensitive data, the NCSC considers it acceptable to let the AI work with little supervision. For login and permission logic, sensitive personal data and the handling of passwords and keys, it recommends reviewing what the AI produced, understanding the code, checking for vulnerabilities and confirming it does what you expect.

The trouble starts when a prototype crosses that line without anyone noticing. The app was born to test an idea, it worked, and the team started using it with customer, employee or contract data. The code is still the prototype's code, only now it holds real data.

Why do these systems reach operations with access flaws?

These systems reach operations with access flaws because the tool delivers what shows on screen first, and access control is exactly the part that does not show. The login screen works, the report opens, and nobody tests what happens when someone skips the screen and asks the database directly.

Recent public reports show this pattern. In March 2025, researcher Matt Palmer analyzed 1,645 projects published by an AI app-building platform and found 170 of them, about 10%, with database tables lacking adequate access rules. The issue was registered as CVE-2025-48757. Through the exposed tables it was possible to read user records, third-party service keys, transactions and subscriptions, and in some cases write new data. The scan only looked at home pages, without logging in.

In February 2026, the security company Wiz reported a similar case at Moltbook, a social network for AI agents whose founder said publicly he had vibe-coded it. Browsing as a regular user, the researchers found a database access key in the code the browser downloads. Because the database had no row-level access rules configured, that key opened read and write access to every production table, including 1.5 million API tokens, 35,000 email addresses and private messages. The flaw was fixed within hours of the report.

Neither case required a sophisticated attack. In both, a rule saying who can see what was missing. That matches what OWASP, the foundation behind the most widely used list of web application risks, published in the 2025 edition of its Top 10: broken access control remains in first place, and every application tested in the list's dataset had some form of it. In a system built in a hurry, the odds that this layer was left for later are higher.

There is also an organizational factor. A system built inside a business area usually runs on the personal account of whoever built it. When that person goes on vacation or leaves the company, nobody else can get into the hosting dashboard.

What should you review now and what can wait?

Review now everything that decides who gets in and what is visible; code quality, performance and looks can wait for a second round. The table splits the two groups so the first meeting does not get lost in detail.

AreaReview nowCan wait
Hosting and accountsWhose name it is under, who has admin accessSwitching hosting providers
Login and sessionInternal pages without login, open sign-up, sessions of people who leftSingle sign-on with the company directory
DatabaseQueries without a logged-in user, access to other people's recordsData modeling, indexes, performance
Keys and passwordsWhat shows in the browser, what went through a chat or repositoryA secrets vault with automatic rotation
Uploaded filesPermanent public links to documents and photosA full retention policy
CodeParts that handle permissions and personal dataStructure, automated tests, conventions
OperationsA backup that restores and a log of who changed whatFiner monitoring and alerts

What goes into the second round does not change the answer to the urgent question: today, can someone outside see or change the data the team enters in this system? If the system handles personal data, the review of screens, reports and logs in the article on Brazil's LGPD in internal systems complements this one. The LGPD is Brazil's general data protection law, close in spirit to the GDPR.

Which review plan should you follow for an AI-built system?

The plan below organizes the review in four stages, from inventory to decision. It is written to be worked through by the person who built the system together with someone from IT, without requiring the manager to master every technical detail.

Review plan for an AI-built system

Ready to use · 17 items

Go through the stages in order, with the person who built the system and someone from IT in the same room. Each item is a test or a check that someone can mark as done. The plan points to the most common gaps. It does not replace a penetration test or a legal review of the personal data involved.

Stage 1: inventory

Stage 2: who can get in

Stage 3: keys and visible data

Stage 4: before expanding use

Run the stage 2 tests for real, with the browser open. Copy the address of an internal page, open a private window and paste it. Sign in with a test user and change the number at the end of the address. Open the browser's developer tools and search the loaded files for words like "key", "secret" or "password". Each failure goes on a list with a screenshot and the profile used, and that list becomes the scope of the fix.

If any stage 2 test fails and the system holds customer or personal data, treat it as an ongoing incident. Restrict access first and investigate afterwards.

How do you put the review into practice and decide to keep, fix or shut down?

The review fits in one short week, with a written decision at the end.

  1. On day one, take inventory with the person who built the system. Write down hosting, database, domain, accounts and who has access. If the account sits on someone's personal email, add a second company administrator the same day.
  2. On day two, run the stage 2 access tests with two different profiles. The private-window test and the record-number test take minutes and reveal most of the serious problems.
  3. Next, look for keys and passwords in the code the browser downloads and in the history of conversations with the AI tool. Every key that showed up in the wrong place must be rotated at the source service, even if the code has already been fixed.
  4. Check the backup by doing a test restore in a separate environment. Many companies only find out the backup is useless on the day they need it.
  5. Ask someone who can read code to review only the parts that handle login, permissions and personal data. AI-generated code tends to be long, and reviewing everything at once stalls the process.
  6. Bring the business area and IT together and decide based on what you found. The decision goes in writing, with an owner and a deadline.

Three scenarios cover most cases.

  • Keeping it makes sense when the access tests passed, the accounts belong to the company, there is a backup that restores and the system holds low-impact data. Even then, name who looks after it and schedule the next review.
  • Fixing it is the way to go when the system solves a real operational problem and the flaws are concentrated in identifiable spots, such as a database access rule, an exposed key or a public file link. The fix comes with a short deadline and a new test.
  • Shutting it down, or suspending it until fixed, applies when sensitive data is exposed without login, when nobody can take over maintenance or when the code cannot separate who sees what without rewriting its core.

Shutting down has a cost, because the system was solving something. Plan where the data goes and how the team works in the meantime, or the old spreadsheet comes back with the same data and less control.

What if the system already holds customer data or nobody knows who built it?

The plan changes in a few common situations.

When the system already holds customer data or sensitive personal data and an access test failed, the priority is containment. Restrict access, rotate the keys and preserve whatever access logs exist. Bring in the company's legal team and data protection officer to assess whether data was exposed and what the law requires from there. In Brazil that means the LGPD and guidance from the national data protection authority (ANPD); elsewhere, the local equivalent. The technical review helps answer what was visible and for how long, but the decision on notification belongs to the data controller.

When nobody knows who built the system, or the person has left, start with the domain and the invoice. The domain registration and the hosting bill usually point to the account. Without access to the dashboard, the company does not control the system, and the practical decision becomes moving the data somewhere under its control.

When the system connects to other company systems, such as the ERP, check which credential the app uses for that integration and what it allows. A key with overly broad permissions turns a small app into a doorway to larger data.

When several companies or units use the same system, isolation between them is a requirement. In the systems we build and maintain, such as the access control system for an industrial complex, each tenant company sees only its own records, and the history keeps who approved or denied each access. In an app generated in a hurry, that separation rarely exists in the data model and has to be tested on purpose.

When is an internal review not enough?

An internal review closes the obvious gaps, but it does not replace an assessment by people who look for flaws every day. It does not cover penetration testing, a full dependency review, architecture design or the legal analysis of personal data. Nor does it solve a system that has grown so much that fixing it costs more than rebuilding it with a clear scope, the subject of the article on custom software projects.

It is worth getting outside help when the system already supports a critical process, holds personal or financial data in volume, or when the review found flaws and nobody in the company feels confident fixing the code. In support and modernization, this work usually starts with access control, keys and backups.

If your company already depends on a system put together in a hurry, boosting.tech, a custom software company, runs a fixed-scope security assessment to point out what to fix first. The result comes in management language, with a list of what to close and in what order. The first conversation is free. Write to jcm@boosting.tech or use the contact page.

Frequently asked questions about the security of AI-built systems

Is AI-generated code less secure than code written by people?

It depends on who reviews it. The tool produces code that works on screen, and what tends to be missing are the invisible layers, such as database access rules, server-side validation and care with keys. Code written by people has flaws too. The practical difference is volume and speed: a whole app appears in days, and the review a team would do over weeks never happens.

How do I know if the app's database is open?

The simplest test is to open an internal page in a private window, without logging in, and see whether it loads data. Then, with a regular user, change the record number in the address. A more complete test needs someone technical to query the database without a logged-in user, using the key that shows in the browser. If the query returns data, the database relies on the screen alone for protection.

Can I ask the AI tool itself to review the security?

You can use it as support, but not as the only review. The tool that generated the code tends to repeat the same blind spots when reviewing it, and guidance such as the NCSC's asks for someone who understands the code to check login, permissions and sensitive data. A person with security experience, or at least a second tool with specific instructions, raises the odds of catching what slipped through.

What should I do first if I find an exposed key?

Rotate the key at the source service, whether it is the database, the email provider or a paid API. Removing the key from the code is not enough, because it may already have been copied. Then check the service's logs for unusual use and move the new key to the server, out of the code the browser downloads. If the service holds personal data, let your data protection officer know.

Does banning AI tools in the company solve the problem?

Rarely. A ban tends to push use into personal accounts, away from any control, and the systems keep appearing. Clear rules work better: prototypes are fine, real data only after review, accounts always in the company's name and a technical owner for every system that goes into use. That way the business area keeps solving problems quickly and IT knows what is running.

System support and modernization

We take over existing systems and modernize them in parts, without stopping the operation.

Talk on WhatsApp See the service

Founder and developer, boosting.tech

Founded boosting.tech and has built custom software for more than 10 years. Writes from the systems the company builds and maintains, the oldest of them in continuous development since 2019.

Let us understand your process

The first conversation has no cost. You leave knowing where to start, even if you decide not to go ahead with us.

Example: today we track permits to work on paper and spreadsheets.