Post-Mortem: Phala Cloud API Authorization Vulnerability (June 1, 2026)

Sat, 06 Jun 2026 01:57:00 -0000

# Post-Mortem: Phala Cloud API Authorization Vulnerability (June 1, 2026) **Status:** Resolved & patched · **Severity:** P0 · **Window:** May 31 – June 1, 2026 (UTC) · This is a detailed follow-up to our [June 1 security notice](https://phala.com/posts/security-incident-notice-phala-cloud-api-vulnerability). On June 1, 2026, an attacker used a bug in a Phala Cloud API endpoint to change CVMs they didn't own. We've fixed the bug, undone every unauthorized change, banned the accounts involved, and emailed affected customers directly. This post explains what happened, what was and wasn't affected, what we did, and what we're changing so it doesn't happen again. It includes full technical detail, including the attacker's script. **In short** — A few of our CVM "change" endpoints let someone edit a CVM by its ID without checking that they owned it. An attacker used this to replace the **pre-launch script** on other people's CVMs with a script that copies their secrets to a server the attacker controls. Saving that change also restarts the CVM, so the script ran and the secrets on those CVMs were exposed. **Only Offchain KMS CVMs were affected. Onchain KMS CVMs and the TEE / attestation chain were not.** **If we emailed you that your CVM was affected:** treat that CVM as compromised and redeploy it, then rotate the secrets in its encrypted environment variables and any registry credentials it used. Full checklist is near the end. We've contacted every affected customer directly — if you didn't get an email from us, your CVMs were not affected. --- ## What happened Phala Cloud groups your resources into **workspaces**. Every CVM belongs to one workspace, and our API is supposed to check that the person making a change belongs to the workspace that owns the CVM. A few of the endpoints that change a CVM skipped this check. They found the CVM by its ID and made the change without checking who owned it. So anyone who knew a CVM's ID could change CVMs in other people's workspaces. The endpoint that mattered most was the one that sets a CVM's **pre-launch script** — a script that runs as root inside the secure VM, just before your containers start, with access to your decrypted secrets. This is a normal, useful feature for the owner, but the same access makes it a high-value target if someone can change it without permission. The attacker used the bug to replace this script with one that copies secrets to a server they control. Saving a new pre-launch script also restarts the CVM to apply it. So the script ran during that restart, and the secrets on the affected CVMs were exposed to the attacker. --- ## Who was affected | Area | Status | | --- | --- | | **Offchain KMS CVMs** | **Affected — the ones we emailed about** | | Onchain KMS CVMs | Not affected (these use a different path the bug couldn't reach) | | TEE hardware & attestation chain | Not affected (the bug was in the cloud API, not the secure hardware) | | Phala platform infrastructure (KMS, control plane) | Not affected (the attacker probed some platform CVMs, but none were modified; KMS keys and the attestation trust root are intact) | | Phala account logins & billing data | Not affected (the vulnerability could not reach account or billing systems) | The attacker first tested on their own CVMs, then focused on one customer's workspace (three CVMs), and finally ran two automated rounds — one aimed at 170 CVM IDs, the other at 165 — pushing the bad pre-launch script across the platform. The change was applied about 130 times across the two rounds; after removing duplicates, that's roughly **100 distinct CVMs**. Because saving the change restarts the CVM, the script ran on those CVMs, so we treat all of them as compromised. We identified the affected CVMs and emailed those customers directly with what to do. --- ## Timeline (all times UTC) - **May 31, 15:04** — Attacker starts scanning workspaces and apps, then spends about 7 hours building and testing their tools on their *own* CVMs. - **May 31, 22:26** — First confirmed unauthorized change to a customer CVM. - **May 31 → June 1** — Attacker moves through several customer CVMs, then runs two automated rounds pushing the bad pre-launch script to many CVM IDs, keeping a deliberately slow pace to avoid notice. - **June 1, 14:47** — A customer reports something odd. *(Thank you — this is how we found out.)* - **June 1, 14:54** — Our team responds; we open a dedicated incident channel at 15:31. - **June 1, 15:47** — Bug fixed. This is also the **last** request we saw from the attacker. - **June 1, 15:53** — All affected CVMs stopped and the unauthorized changes undone. - **June 1, 18:50** — Public notice published. The attack ran for about a day before we caught it, and a customer reported it before our own monitoring did. --- ## Why it happened Our code had two ways to find a CVM by its ID: - a **workspace-check** version that says "not found" if the CVM isn't in your workspace, and - a **no-check** version that returns any CVM by its ID. The affected endpoints used the **no-check** version. From the outside, both behave the same — the request succeeds and the right CVM is changed — so the missing check didn't show up in normal testing. The common name for this kind of bug is **Broken Access Control** (OWASP A01). Two things made it possible, and we want to be straight about both: 1. **The safe check was optional.** Each endpoint had to choose the safe version itself. The unsafe version was just as easy to pick, and nothing stopped code from shipping with the wrong one. 2. **The endpoint was written with help from an AI coding assistant**, which used the simpler no-check version. The generated code worked and passed our tests; the missing ownership check doesn't show up in normal functional testing, so neither our tests nor our review caught it. --- ## How the attacker found targets To change a CVM, the attacker first needed its ID (App ID). They gathered IDs from a public listing endpoint that returned apps running on the platform, and from places where app IDs are openly available (for example, public attestation data). One point is worth being clear about: a CVM or App ID is a private identifier, not an access credential. Knowing one was never meant to let anyone act on your CVM. We've also retired the deprecated public listing endpoint the attacker used to gather IDs in bulk, which removes an easy way to scan for targets. --- ## What the script could reach The script the attacker added was a broad "grab everything" tool. It ran in the background, so the CVM still started up normally and there was no obvious sign, and it sent whatever it found to a server the attacker controlled. On the CVM itself, it could read: - **Decrypted environment variables** and **KMS-derived app keys** — the most sensitive items, and the main exposure here - **Docker registry credentials** (for example, AWS ECR) - Your **deployment configuration** (the compose file and app settings) - Basic network and process details from the VM The script also tried to grab other things — TLS and SSH keys, wallet and keystore files, and stray `.env`/secrets files. Those normally live **inside your containers**, not on the CVM itself. The pre-launch script runs *before* your containers start, so at that point the containers aren't running and their files aren't mounted yet — the attacker tried for them but couldn't reach them through this path. The full script is in **Appendix B** if you want to see exactly what it looked for. --- ## What we did right away 1. **Fixed the affected endpoints** so they use the workspace check. The safe version already existed, so the fix was about one line per endpoint. 2. **Stopped every affected CVM and reverted the malicious pre-launch scripts** to their original state. Every CVM change on Phala Cloud is saved with full before/after history, which let us find and undo the changes quickly. (Reverting stops any further exfiltration, but it can't undo secrets that were already exposed — which is why affected CVMs still need a clean redeploy.) 3. **Found all the accounts the attacker controlled and banned them**, and preserved the incident logs. 4. **Emailed affected customers** with clear next steps. 5. **Published a public notice the same day.** --- ## What we're changing We're addressing this at four levels: preventing the bug, detecting an attack sooner, limiting the damage if something gets through, and responding faster. ### Prevention The root problem was that the safe ownership check was optional, so a single endpoint could skip it. We're removing that choice: - **Build-time checks (✅ done).** Our test suite now inspects every CVM endpoint and fails the build if one that operates on a specific CVM doesn't go through the workspace ownership check. This would have caught the exact bug behind this incident. - **Full endpoint review and isolation tests (✅ done).** We reviewed all of our API endpoints again to confirm each one requires the right login and permissions, and added automated tests that a user in one workspace cannot reach another workspace's CVM. New endpoints are covered automatically. - **Safe-by-default routing (🔧 in progress).** We're reorganizing the API so an endpoint's location sets its security level automatically. "Remember to add the check" stops being a manual step anyone can forget. - **Focused review of sensitive code (🔧 in progress).** We're grouping the most security-critical code so it gets focused human review. As more of our code is written with AI assistance, concentrating these critical paths for deeper review matters more, not less. - **A separate permission service (🗓️ planned).** Our product changes quickly, but the code that decides who is allowed to do what doesn't need to. By moving these permission checks into a separate service that changes rarely and is reviewed more strictly, we keep this critical logic stable and lower the chance that fast day-to-day development breaks it. ### Detection This is where we fell short. The attack ran for about a day, and a customer found it before we did. We keep a full audit log of every action, which let us reconstruct what happened, but we weren't monitoring it in real time. That changes: - **Alerts on cross-workspace attempts (🗓️ planned).** Acting on a CVM outside your own workspace is now blocked, and we're adding alerts on any such attempt. - **Automated review of API access patterns (🗓️ planned).** The attacker kept a slow pace on purpose (about 3–5 requests a minute) to stay under simple rate limits, so rate limits alone won't catch this. We plan to use an LLM-based system to scan API access patterns — for example, one account editing the pre-launch script across many CVMs, or scanning a lot of resources — and flag risky behavior for review. The aim is to catch this kind of activity much sooner than we did this time. ### Mitigation Even with the bug, a few things would have reduced the impact. We're adding them: - **Confirmation for sensitive changes (🗓️ planned).** Optional email confirmation or 2FA before high-impact Offchain KMS CVM changes, like editing the pre-launch script. A valid session alone wouldn't be enough. - **Outbound firewall rules (🗓️ mid-to-long term).** The script worked by sending data to an outside server. Adding network ACLs to limit where a CVM can send traffic would reduce this kind of leak even if a malicious script runs. This is a larger change, so we're tracking it as longer-term work. - **Faster recovery (🗓️ planned).** Better tooling to rotate secrets and redeploy affected CVMs, so cleanup takes minutes. ### Response Once we knew, we moved quickly (see "What we did right away"). Two things made that possible: - The safe mechanism already existed, so the fix was about one line per endpoint and shipped roughly an hour after the report. - Every CVM change is saved with full before/after history, so we could undo the attacker's changes cleanly and completely. We're also adding: - **A written incident playbook and clearer on-call (🗓️ planned)** so this response time is consistent. - **An audit log in your dashboard (🗓️ planned)** so you can see changes made to your own CVMs and check what happened yourself, without waiting on us. --- ## What you should do We've identified the affected CVMs and emailed those customers directly. **If you didn't get an email from us about this, your CVMs were not affected and there's nothing you need to do.** If we did notify you, please: - [ ] **Redeploy** the affected CVM from a clean start, and treat the old one as compromised. Redeploying generates fresh KMS-derived app keys; once the old CVM is destroyed, the exposed keys can no longer be used. - [ ] **Revoke and replace every secret in its encrypted environment variables** — API keys, tokens, database passwords, private keys. Assume the old values are already in the attacker's hands, so invalidate them; don't just issue new ones alongside. - [ ] **Follow the blast radius downstream.** For each exposed secret, check what it unlocks — external APIs, databases, wallets, signing keys — and rotate or lock those too. - [ ] **Rotate the registry credentials** (AWS ECR and any others) the CVM used. - [ ] **If you keep egress/network logs for the CVM** (many setups don't), check them for connections to the attacker's server or any other unexpected address (see Appendix A). If you don't have such logs, assume exposure and complete the steps above. If you'd like to confirm the impact yourself, you can check when the CVM last restarted (around May 31–June 1 UTC) and inspect its pre-launch script for the injected block shown in Appendix B. Not sure about anything, or want a hand working through it? Email us at [cloud@phala.com](mailto:cloud@phala.com) and we'll go through your setup with you. --- ## Our commitment We're sorry this happened, and we don't take your trust for granted. The part we're least happy about is the detection gap — that the attack ran for a day and a customer found it before we did. That's why detection and response are front and center in the plan above, alongside the prevention work, and we'll keep you posted as the rest of it lands. We know how much you rely on this, and we're committed to doing better. --- ## Appendix A — Indicators of compromise (IOCs) If you'd like to check your own logs: - **Attacker's server:** `http://199.91.221.65:9998` (data was sent to paths like `/x//...`) - **Endpoint that was abused:** `PATCH /api/v1/cvms/{id}/pre-launch-script` (also `/docker-compose` and `/envs`) - **Attacker source IPs:** `160.202.160.203`, `160.202.160.239`, `160.202.160.248`, `160.202.160.56`, `61.97.243.9`, `195.154.153.4`, `199.91.221.65` (the last address is also the attacker's server listed above) - **User agents seen:** `curl/7.81.0`, `python-requests/2.31.0`, `python-requests/2.33.1`, `node` If you see traffic from one of your CVMs going to the attacker's server, treat that CVM's secrets as exposed. --- *This is a customer-facing post-mortem. For questions or to schedule a review of your setup, contact the Phala Cloud team at [cloud@phala.com](mailto:cloud@phala.com).*

Maintenance: Prod10 TCP-R update maintenance

Wed, 29 Apr 2026 02:54:33 -0000

We're scheduling Intel TCB Recovery (TCB-R) firmware upgrades across our nodes. This upgrade is required to keep attestation quotes valid — after May 13, 2026, unupdated nodes will generate OutOfDate quotes. Please review the maintenance windows below and let us know if you have any concerns. Node: prod10 - Date: April 29, 2026 - Time: 03:00–05:00 UTC (11:00–13:00 CST) - Expected downtime: ~2 hour(s)

Maintenance: use1 maintainance

Thu, 23 Apr 2026 03:09:00 -0000

use1 maintainance

Maintenance: prod1 maintainance

Wed, 22 Apr 2026 03:09:34 -0000

prod1 maintainance

Maintenance: [Maintenance] Phala Network Node — Intel TEE & TCB Update

Sat, 28 Mar 2026 06:10:46 -0000

We're scheduling a maintenance window to apply Intel TEE firmware updates to one of our production nodes. Window: April 1, 2026 | 02:00–08:00 UTC What's changing: Intel TDX/SGX microcode upgrade + TCB refresh Why it matters: - Attestation: Verifiers enforce strict TCB level checks. An outdated TCB causes quotes to fail UpToDate status, which triggers consensus check failures and temporary node restrictions. - Security: Patches for recent Intel side-channel vulnerabilities. - Stability: Low-level fixes for TDX resource reclamation and lifecycle management. Impact: - The affected node will restart to load new microcode. Expect up to 3 hours of downtime for your running instances. - TEE environments reinitialize automatically on boot. After the window, verify your instance status and quote generation. Please adjust your workload scheduling accordingly. If you hit initialization issues or quote failures post-update, reach out through the standard support channels. — Phala Network Operations

Networking unavailable

Fri, 20 Mar 2026 12:44:00 -0000

We are currently experiencing networking issues affecting some nodes in US-West. Our team is actively investigating, and we will share updates as soon as possible. Thank you for your patience.

Maintenance: Urgent Maintenace Notice

Thu, 19 Mar 2026 19:00:00 -0000

We're investigating a hardware failure and outage affecting prod7, prod7-v3x, prod8-v3x, and working to resolve it. We working on the recoverying. We apologize for that and keep posting here if any update.

Maintenance: 2026-03-02 prod7 Urgent Hardware Maintenance

Sun, 01 Mar 2026 14:11:42 -0000

Dear Customers, We need to perform urgent maintenance to ensure the continued stability and reliability of our platform. This maintenance affects the following nodes only: 🔧 Affected nodes: prod7, prod7-v3x, prod8-v3x ⏱️ Expected duration: 30-60 minutes During the maintenance window: - Services deployed on the above nodes will be temporarily unavailable - Deployments on all other nodes are not affected and will continue to run normally - No action is required on your end If you have any questions or concerns, please don't hesitate to reach out. Thank you for your understanding.

Maintenance: Phala Cloud Maintenance Service on Prod5

Thu, 16 Oct 2025 02:00:56 -0000

Phala Cloud has initiated a scheduled maintenance on node Prod5.

Maintenance: Scheduled Service Interruption Notice for prod7

Wed, 02 Jul 2025 09:39:23 -0000

We have scheduled a maintenance window for our prod7 environment due to a data center migration by our infrastructure partner. Expected Impact: Service interruptions of up to 6 hours during the migration Exact maintenance date and time will be announced at least 2 weeks prior Recommended Action: Users may proactively migrate their CVMs to other production environments before the scheduled maintenance to avoid potential service disruption. Next Steps: We will provide the specific maintenance schedule and detailed instructions once confirmed by our data center partner. For questions or assistance with CVM migration, please contact our support team. We apologize for any inconvenience and appreciate your understanding.

Prod5 Hardware Failure and Reboot Automatically

Mon, 30 Jun 2025 09:44:00 -0000

We’re back! prod5 should be up and running. Thanks for bearing with us.

Maintenance on all nodes

Wed, 18 Jun 2025 14:05:00 -0000

A regular component update and maintenance on all nodes. The uptime will be affected because the health check endpoints have been updated.

Status updates | Phala Cloud