Incidents | Phala Cloud

gpt-oss-20b recovered

Tue, 16 Jun 2026 08:01:25 +0000

gpt-oss-20b recovered

gpt-oss-20b went down

Tue, 16 Jun 2026 07:55:36 +0000

gpt-oss-20b went down

gpt-oss-20b recovered

Tue, 16 Jun 2026 07:45:35 +0000

gpt-oss-20b recovered

gpt-oss-20b went down

Tue, 16 Jun 2026 07:40:21 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct recovered

Tue, 16 Jun 2026 07:09:31 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 16 Jun 2026 07:04:18 +0000

qwen-2.5-7b-instruct went down

gpt-oss-20b recovered

Tue, 16 Jun 2026 03:50:36 +0000

gpt-oss-20b recovered

gpt-oss-20b went down

Tue, 16 Jun 2026 03:45:21 +0000

gpt-oss-20b went down

deepseek-v3.2 recovered

Mon, 15 Jun 2026 20:41:13 +0000

deepseek-v3.2 recovered

deepseek-v3.2 went down

Mon, 15 Jun 2026 20:31:37 +0000

deepseek-v3.2 went down

Use2 KMS 0.5 recovered

Mon, 15 Jun 2026 19:20:05 +0000

Use2 KMS 0.5 recovered

Use2 KMS 0.5 went down

Mon, 15 Jun 2026 19:16:14 +0000

Use2 KMS 0.5 went down

gpt-oss-20b recovered

Mon, 15 Jun 2026 16:55:17 +0000

gpt-oss-20b recovered

gpt-oss-20b went down

Mon, 15 Jun 2026 16:46:00 +0000

gpt-oss-20b went down

gemma-3-27b-it recovered

Mon, 15 Jun 2026 14:46:05 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Mon, 15 Jun 2026 14:45:19 +0000

gpt-oss-20b recovered

gemma-3-27b-it went down

Mon, 15 Jun 2026 14:35:39 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Mon, 15 Jun 2026 14:35:31 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct recovered

Sun, 14 Jun 2026 01:30:41 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 14 Jun 2026 01:05:21 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Fri, 12 Jun 2026 20:50:54 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Fri, 12 Jun 2026 20:32:42 +0000

gemma-3-27b-it went down

qwen-2.5-7b-instruct recovered

Fri, 12 Jun 2026 01:15:35 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 12 Jun 2026 01:01:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 12 Jun 2026 00:30:39 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 12 Jun 2026 00:16:16 +0000

qwen-2.5-7b-instruct went down

Use1 KMS 0.5 recovered

Thu, 11 Jun 2026 21:26:48 +0000

Use1 KMS 0.5 recovered

Use1 KMS 0.5 went down

Thu, 11 Jun 2026 21:22:58 +0000

Use1 KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Thu, 11 Jun 2026 20:39:31 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 11 Jun 2026 20:29:42 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Thu, 11 Jun 2026 15:20:41 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Thu, 11 Jun 2026 15:11:21 +0000

gemma-3-27b-it went down

gemma-3-27b-it recovered

Thu, 11 Jun 2026 05:25:42 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Thu, 11 Jun 2026 05:20:26 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Thu, 11 Jun 2026 05:04:30 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it went down

Thu, 11 Jun 2026 04:55:55 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Thu, 11 Jun 2026 04:55:35 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Thu, 11 Jun 2026 04:54:28 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Thu, 11 Jun 2026 04:20:40 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Thu, 11 Jun 2026 04:20:21 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Thu, 11 Jun 2026 04:14:31 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it went down

Thu, 11 Jun 2026 04:10:58 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Thu, 11 Jun 2026 04:10:33 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Thu, 11 Jun 2026 04:04:25 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 11 Jun 2026 02:05:56 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 11 Jun 2026 01:51:22 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 11 Jun 2026 01:20:27 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 11 Jun 2026 01:06:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 11 Jun 2026 00:40:49 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 11 Jun 2026 00:21:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 13:04:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 12:49:47 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 12:29:17 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 12:21:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 11:44:52 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 11:36:26 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 09:29:23 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 09:21:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 08:04:23 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 07:56:13 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 07:19:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 07:11:15 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Wed, 10 Jun 2026 06:11:00 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Wed, 10 Jun 2026 06:05:37 +0000

gemma-3-27b-it went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 01:55:07 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 01:36:29 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 01:15:27 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 00:56:22 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 10 Jun 2026 00:30:21 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 10 Jun 2026 00:16:29 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 09 Jun 2026 16:04:50 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 09 Jun 2026 15:56:24 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 recovered

Tue, 09 Jun 2026 08:50:24 +0000

deepseek-v3.2 recovered

deepseek-v3.2 went down

Tue, 09 Jun 2026 07:23:10 +0000

deepseek-v3.2 went down

qwen-2.5-7b-instruct recovered

Tue, 09 Jun 2026 01:51:14 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 09 Jun 2026 01:41:19 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 09 Jun 2026 01:11:07 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 09 Jun 2026 01:01:22 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 09 Jun 2026 00:31:06 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 09 Jun 2026 00:16:19 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 recovered

Mon, 08 Jun 2026 15:56:19 +0000

deepseek-v3.2 recovered

deepseek-v3.2 went down

Mon, 08 Jun 2026 15:41:10 +0000

deepseek-v3.2 went down

qwen-2.5-7b-instruct recovered

Mon, 08 Jun 2026 11:29:24 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 08 Jun 2026 11:19:20 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 08 Jun 2026 07:49:17 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 08 Jun 2026 07:26:13 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 08 Jun 2026 01:10:48 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 08 Jun 2026 00:51:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 08 Jun 2026 00:26:56 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 08 Jun 2026 00:21:17 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 recovered

Sun, 07 Jun 2026 15:51:09 +0000

deepseek-v3.2 recovered

deepseek-v3.2 went down

Sun, 07 Jun 2026 15:41:08 +0000

deepseek-v3.2 went down

qwen-2.5-7b-instruct recovered

Sun, 07 Jun 2026 04:39:24 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 07 Jun 2026 04:31:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 07 Jun 2026 02:14:38 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 07 Jun 2026 01:41:26 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 07 Jun 2026 01:20:30 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 07 Jun 2026 01:01:22 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 07 Jun 2026 00:45:23 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 07 Jun 2026 00:21:25 +0000

qwen-2.5-7b-instruct went down

Post-Mortem: Phala Cloud API Authorization Vulnerability (June 1, 2026)

Sat, 06 Jun 2026 01:57:00 -0000

# Post-Mortem: Phala Cloud API Authorization Vulnerability (June 1, 2026) **Status:** Resolved & patched · **Severity:** P0 · **Window:** May 31 – June 1, 2026 (UTC) · This is a detailed follow-up to our [June 1 security notice](https://phala.com/posts/security-incident-notice-phala-cloud-api-vulnerability). On June 1, 2026, an attacker used a bug in a Phala Cloud API endpoint to change CVMs they didn't own. We've fixed the bug, undone every unauthorized change, banned the accounts involved, and emailed affected customers directly. This post explains what happened, what was and wasn't affected, what we did, and what we're changing so it doesn't happen again. It includes full technical detail, including the attacker's script. **In short** — A few of our CVM "change" endpoints let someone edit a CVM by its ID without checking that they owned it. An attacker used this to replace the **pre-launch script** on other people's CVMs with a script that copies their secrets to a server the attacker controls. Saving that change also restarts the CVM, so the script ran and the secrets on those CVMs were exposed. **Only Offchain KMS CVMs were affected. Onchain KMS CVMs and the TEE / attestation chain were not.** **If we emailed you that your CVM was affected:** treat that CVM as compromised and redeploy it, then rotate the secrets in its encrypted environment variables and any registry credentials it used. Full checklist is near the end. We've contacted every affected customer directly — if you didn't get an email from us, your CVMs were not affected. --- ## What happened Phala Cloud groups your resources into **workspaces**. Every CVM belongs to one workspace, and our API is supposed to check that the person making a change belongs to the workspace that owns the CVM. A few of the endpoints that change a CVM skipped this check. They found the CVM by its ID and made the change without checking who owned it. So anyone who knew a CVM's ID could change CVMs in other people's workspaces. The endpoint that mattered most was the one that sets a CVM's **pre-launch script** — a script that runs as root inside the secure VM, just before your containers start, with access to your decrypted secrets. This is a normal, useful feature for the owner, but the same access makes it a high-value target if someone can change it without permission. The attacker used the bug to replace this script with one that copies secrets to a server they control. Saving a new pre-launch script also restarts the CVM to apply it. So the script ran during that restart, and the secrets on the affected CVMs were exposed to the attacker. --- ## Who was affected | Area | Status | | --- | --- | | **Offchain KMS CVMs** | **Affected — the ones we emailed about** | | Onchain KMS CVMs | Not affected (these use a different path the bug couldn't reach) | | TEE hardware & attestation chain | Not affected (the bug was in the cloud API, not the secure hardware) | | Phala platform infrastructure (KMS, control plane) | Not affected (the attacker probed some platform CVMs, but none were modified; KMS keys and the attestation trust root are intact) | | Phala account logins & billing data | Not affected (the vulnerability could not reach account or billing systems) | The attacker first tested on their own CVMs, then focused on one customer's workspace (three CVMs), and finally ran two automated rounds — one aimed at 170 CVM IDs, the other at 165 — pushing the bad pre-launch script across the platform. The change was applied about 130 times across the two rounds; after removing duplicates, that's roughly **100 distinct CVMs**. Because saving the change restarts the CVM, the script ran on those CVMs, so we treat all of them as compromised. We identified the affected CVMs and emailed those customers directly with what to do. --- ## Timeline (all times UTC) - **May 31, 15:04** — Attacker starts scanning workspaces and apps, then spends about 7 hours building and testing their tools on their *own* CVMs. - **May 31, 22:26** — First confirmed unauthorized change to a customer CVM. - **May 31 → June 1** — Attacker moves through several customer CVMs, then runs two automated rounds pushing the bad pre-launch script to many CVM IDs, keeping a deliberately slow pace to avoid notice. - **June 1, 14:47** — A customer reports something odd. *(Thank you — this is how we found out.)* - **June 1, 14:54** — Our team responds; we open a dedicated incident channel at 15:31. - **June 1, 15:47** — Bug fixed. This is also the **last** request we saw from the attacker. - **June 1, 15:53** — All affected CVMs stopped and the unauthorized changes undone. - **June 1, 18:50** — Public notice published. The attack ran for about a day before we caught it, and a customer reported it before our own monitoring did. --- ## Why it happened Our code had two ways to find a CVM by its ID: - a **workspace-check** version that says "not found" if the CVM isn't in your workspace, and - a **no-check** version that returns any CVM by its ID. The affected endpoints used the **no-check** version. From the outside, both behave the same — the request succeeds and the right CVM is changed — so the missing check didn't show up in normal testing. The common name for this kind of bug is **Broken Access Control** (OWASP A01). Two things made it possible, and we want to be straight about both: 1. **The safe check was optional.** Each endpoint had to choose the safe version itself. The unsafe version was just as easy to pick, and nothing stopped code from shipping with the wrong one. 2. **The endpoint was written with help from an AI coding assistant**, which used the simpler no-check version. The generated code worked and passed our tests; the missing ownership check doesn't show up in normal functional testing, so neither our tests nor our review caught it. --- ## How the attacker found targets To change a CVM, the attacker first needed its ID (App ID). They gathered IDs from a public listing endpoint that returned apps running on the platform, and from places where app IDs are openly available (for example, public attestation data). One point is worth being clear about: a CVM or App ID is a private identifier, not an access credential. Knowing one was never meant to let anyone act on your CVM. We've also retired the deprecated public listing endpoint the attacker used to gather IDs in bulk, which removes an easy way to scan for targets. --- ## What the script could reach The script the attacker added was a broad "grab everything" tool. It ran in the background, so the CVM still started up normally and there was no obvious sign, and it sent whatever it found to a server the attacker controlled. On the CVM itself, it could read: - **Decrypted environment variables** and **KMS-derived app keys** — the most sensitive items, and the main exposure here - **Docker registry credentials** (for example, AWS ECR) - Your **deployment configuration** (the compose file and app settings) - Basic network and process details from the VM The script also tried to grab other things — TLS and SSH keys, wallet and keystore files, and stray `.env`/secrets files. Those normally live **inside your containers**, not on the CVM itself. The pre-launch script runs *before* your containers start, so at that point the containers aren't running and their files aren't mounted yet — the attacker tried for them but couldn't reach them through this path. The full script is in **Appendix B** if you want to see exactly what it looked for. --- ## What we did right away 1. **Fixed the affected endpoints** so they use the workspace check. The safe version already existed, so the fix was about one line per endpoint. 2. **Stopped every affected CVM and reverted the malicious pre-launch scripts** to their original state. Every CVM change on Phala Cloud is saved with full before/after history, which let us find and undo the changes quickly. (Reverting stops any further exfiltration, but it can't undo secrets that were already exposed — which is why affected CVMs still need a clean redeploy.) 3. **Found all the accounts the attacker controlled and banned them**, and preserved the incident logs. 4. **Emailed affected customers** with clear next steps. 5. **Published a public notice the same day.** --- ## What we're changing We're addressing this at four levels: preventing the bug, detecting an attack sooner, limiting the damage if something gets through, and responding faster. ### Prevention The root problem was that the safe ownership check was optional, so a single endpoint could skip it. We're removing that choice: - **Build-time checks (✅ done).** Our test suite now inspects every CVM endpoint and fails the build if one that operates on a specific CVM doesn't go through the workspace ownership check. This would have caught the exact bug behind this incident. - **Full endpoint review and isolation tests (✅ done).** We reviewed all of our API endpoints again to confirm each one requires the right login and permissions, and added automated tests that a user in one workspace cannot reach another workspace's CVM. New endpoints are covered automatically. - **Safe-by-default routing (🔧 in progress).** We're reorganizing the API so an endpoint's location sets its security level automatically. "Remember to add the check" stops being a manual step anyone can forget. - **Focused review of sensitive code (🔧 in progress).** We're grouping the most security-critical code so it gets focused human review. As more of our code is written with AI assistance, concentrating these critical paths for deeper review matters more, not less. - **A separate permission service (🗓️ planned).** Our product changes quickly, but the code that decides who is allowed to do what doesn't need to. By moving these permission checks into a separate service that changes rarely and is reviewed more strictly, we keep this critical logic stable and lower the chance that fast day-to-day development breaks it. ### Detection This is where we fell short. The attack ran for about a day, and a customer found it before we did. We keep a full audit log of every action, which let us reconstruct what happened, but we weren't monitoring it in real time. That changes: - **Alerts on cross-workspace attempts (🗓️ planned).** Acting on a CVM outside your own workspace is now blocked, and we're adding alerts on any such attempt. - **Automated review of API access patterns (🗓️ planned).** The attacker kept a slow pace on purpose (about 3–5 requests a minute) to stay under simple rate limits, so rate limits alone won't catch this. We plan to use an LLM-based system to scan API access patterns — for example, one account editing the pre-launch script across many CVMs, or scanning a lot of resources — and flag risky behavior for review. The aim is to catch this kind of activity much sooner than we did this time. ### Mitigation Even with the bug, a few things would have reduced the impact. We're adding them: - **Confirmation for sensitive changes (🗓️ planned).** Optional email confirmation or 2FA before high-impact Offchain KMS CVM changes, like editing the pre-launch script. A valid session alone wouldn't be enough. - **Outbound firewall rules (🗓️ mid-to-long term).** The script worked by sending data to an outside server. Adding network ACLs to limit where a CVM can send traffic would reduce this kind of leak even if a malicious script runs. This is a larger change, so we're tracking it as longer-term work. - **Faster recovery (🗓️ planned).** Better tooling to rotate secrets and redeploy affected CVMs, so cleanup takes minutes. ### Response Once we knew, we moved quickly (see "What we did right away"). Two things made that possible: - The safe mechanism already existed, so the fix was about one line per endpoint and shipped roughly an hour after the report. - Every CVM change is saved with full before/after history, so we could undo the attacker's changes cleanly and completely. We're also adding: - **A written incident playbook and clearer on-call (🗓️ planned)** so this response time is consistent. - **An audit log in your dashboard (🗓️ planned)** so you can see changes made to your own CVMs and check what happened yourself, without waiting on us. --- ## What you should do We've identified the affected CVMs and emailed those customers directly. **If you didn't get an email from us about this, your CVMs were not affected and there's nothing you need to do.** If we did notify you, please: - [ ] **Redeploy** the affected CVM from a clean start, and treat the old one as compromised. Redeploying generates fresh KMS-derived app keys; once the old CVM is destroyed, the exposed keys can no longer be used. - [ ] **Revoke and replace every secret in its encrypted environment variables** — API keys, tokens, database passwords, private keys. Assume the old values are already in the attacker's hands, so invalidate them; don't just issue new ones alongside. - [ ] **Follow the blast radius downstream.** For each exposed secret, check what it unlocks — external APIs, databases, wallets, signing keys — and rotate or lock those too. - [ ] **Rotate the registry credentials** (AWS ECR and any others) the CVM used. - [ ] **If you keep egress/network logs for the CVM** (many setups don't), check them for connections to the attacker's server or any other unexpected address (see Appendix A). If you don't have such logs, assume exposure and complete the steps above. If you'd like to confirm the impact yourself, you can check when the CVM last restarted (around May 31–June 1 UTC) and inspect its pre-launch script for the injected block shown in Appendix B. Not sure about anything, or want a hand working through it? Email us at [cloud@phala.com](mailto:cloud@phala.com) and we'll go through your setup with you. --- ## Our commitment We're sorry this happened, and we don't take your trust for granted. The part we're least happy about is the detection gap — that the attack ran for a day and a customer found it before we did. That's why detection and response are front and center in the plan above, alongside the prevention work, and we'll keep you posted as the rest of it lands. We know how much you rely on this, and we're committed to doing better. --- ## Appendix A — Indicators of compromise (IOCs) If you'd like to check your own logs: - **Attacker's server:** `http://199.91.221.65:9998` (data was sent to paths like `/x//...`) - **Endpoint that was abused:** `PATCH /api/v1/cvms/{id}/pre-launch-script` (also `/docker-compose` and `/envs`) - **Attacker source IPs:** `160.202.160.203`, `160.202.160.239`, `160.202.160.248`, `160.202.160.56`, `61.97.243.9`, `195.154.153.4`, `199.91.221.65` (the last address is also the attacker's server listed above) - **User agents seen:** `curl/7.81.0`, `python-requests/2.31.0`, `python-requests/2.33.1`, `node` If you see traffic from one of your CVMs going to the attacker's server, treat that CVM's secrets as exposed. --- *This is a customer-facing post-mortem. For questions or to schedule a review of your setup, contact the Phala Cloud team at [cloud@phala.com](mailto:cloud@phala.com).*

qwen-2.5-7b-instruct recovered

Sat, 06 Jun 2026 01:50:41 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sat, 06 Jun 2026 01:41:24 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sat, 06 Jun 2026 01:13:28 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sat, 06 Jun 2026 00:51:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sat, 06 Jun 2026 00:31:05 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sat, 06 Jun 2026 00:18:10 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 recovered

Fri, 05 Jun 2026 18:26:20 +0000

deepseek-v3.2 recovered

deepseek-v3.2 went down

Fri, 05 Jun 2026 18:17:11 +0000

deepseek-v3.2 went down

qwen-2.5-7b-instruct recovered

Fri, 05 Jun 2026 07:39:25 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 05 Jun 2026 07:26:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 05 Jun 2026 06:56:31 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 05 Jun 2026 06:46:31 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Fri, 05 Jun 2026 05:55:48 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Fri, 05 Jun 2026 05:45:56 +0000

gemma-3-27b-it went down

qwen-2.5-7b-instruct recovered

Fri, 05 Jun 2026 01:58:59 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 05 Jun 2026 01:41:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 05 Jun 2026 01:11:01 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 05 Jun 2026 01:01:23 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 04 Jun 2026 16:39:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 04 Jun 2026 16:11:18 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 recovered

Thu, 04 Jun 2026 15:16:18 +0000

deepseek-v3.2 recovered

deepseek-v3.2 went down

Thu, 04 Jun 2026 14:46:50 +0000

deepseek-v3.2 went down

qwen-2.5-7b-instruct recovered

Thu, 04 Jun 2026 08:05:29 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 04 Jun 2026 07:51:16 +0000

qwen-2.5-7b-instruct went down

gpt-oss-20b recovered

Thu, 04 Jun 2026 02:35:18 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Thu, 04 Jun 2026 02:34:30 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it recovered

Thu, 04 Jun 2026 02:30:43 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Thu, 04 Jun 2026 02:25:36 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Thu, 04 Jun 2026 02:25:22 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Thu, 04 Jun 2026 02:24:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 04 Jun 2026 01:54:25 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 04 Jun 2026 01:46:21 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 04 Jun 2026 01:25:28 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 04 Jun 2026 01:06:21 +0000

qwen-2.5-7b-instruct went down

Prod9 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:35:24 +0000

Prod9 Gateway 0.5 recovered

Prod10 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:30:22 +0000

Prod10 Gateway 0.5 recovered

Prod11 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:29:21 +0000

Prod11 Gateway 0.5 recovered

Prod7 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:27:59 +0000

Prod7 Gateway 0.5 recovered

Prod3 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:24:32 +0000

Prod3 Gateway 0.5 recovered

Prod2 Gateway 0.3 recovered

Wed, 03 Jun 2026 17:24:21 +0000

Prod2 Gateway 0.3 recovered

Use2 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:20:11 +0000

Use2 Gateway 0.5 recovered

Prod5 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:18:44 +0000

Prod5 Gateway 0.5 recovered

Prod12 Gateway 0.5 recovered

Wed, 03 Jun 2026 17:15:00 +0000

Prod12 Gateway 0.5 recovered

Prod6 Gateway 0.5 recovered

Wed, 03 Jun 2026 11:45:32 +0000

Prod6 Gateway 0.5 recovered

Prod6 Gateway 0.5 went down

Wed, 03 Jun 2026 11:34:26 +0000

Prod6 Gateway 0.5 went down

Prod12 Gateway 0.5 went down

Wed, 03 Jun 2026 11:33:42 +0000

Prod12 Gateway 0.5 went down

Prod5 Gateway 0.5 went down

Wed, 03 Jun 2026 11:33:40 +0000

Prod5 Gateway 0.5 went down

Use2 Gateway 0.5 went down

Wed, 03 Jun 2026 11:33:38 +0000

Use2 Gateway 0.5 went down

Prod3 Gateway 0.5 went down

Wed, 03 Jun 2026 11:33:27 +0000

Prod3 Gateway 0.5 went down

Prod2 Gateway 0.3 went down

Wed, 03 Jun 2026 11:33:15 +0000

Prod2 Gateway 0.3 went down

Prod7 Gateway 0.5 went down

Wed, 03 Jun 2026 11:32:49 +0000

Prod7 Gateway 0.5 went down

Prod10 Gateway 0.5 went down

Wed, 03 Jun 2026 11:32:36 +0000

Prod10 Gateway 0.5 went down

Prod11 Gateway 0.5 went down

Wed, 03 Jun 2026 11:32:33 +0000

Prod11 Gateway 0.5 went down

Prod9 Gateway 0.5 went down

Wed, 03 Jun 2026 11:32:13 +0000

Prod9 Gateway 0.5 went down

qwen-2.5-7b-instruct recovered

Tue, 02 Jun 2026 01:29:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 02 Jun 2026 00:21:17 +0000

qwen-2.5-7b-instruct went down

Prod7 Eth KMS 0.5 recovered

Sat, 30 May 2026 14:24:29 +0000

Prod7 Eth KMS 0.5 recovered

Prod9 Base KMS 0.5 recovered

Sat, 30 May 2026 14:24:06 +0000

Prod9 Base KMS 0.5 recovered

Prod7 Base KMS 0.5 recovered

Sat, 30 May 2026 14:24:03 +0000

Prod7 Base KMS 0.5 recovered

Prod5 Base KMS 0.5 recovered

Sat, 30 May 2026 14:23:59 +0000

Prod5 Base KMS 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Sat, 30 May 2026 14:23:58 +0000

Prod9 Eth KMS 0.5 recovered

Use1 Base KMS 0.5 recovered

Sat, 30 May 2026 14:23:38 +0000

Use1 Base KMS 0.5 recovered

Prod7 Eth KMS 0.5 went down

Sat, 30 May 2026 14:20:38 +0000

Prod7 Eth KMS 0.5 went down

Prod9 Eth KMS 0.5 went down

Sat, 30 May 2026 14:20:35 +0000

Prod9 Eth KMS 0.5 went down

Use1 Base KMS 0.5 went down

Sat, 30 May 2026 14:20:26 +0000

Use1 Base KMS 0.5 went down

Prod9 Base KMS 0.5 went down

Sat, 30 May 2026 14:20:18 +0000

Prod9 Base KMS 0.5 went down

Prod7 Base KMS 0.5 went down

Sat, 30 May 2026 14:20:18 +0000

Prod7 Base KMS 0.5 went down

Prod5 Base KMS 0.5 went down

Sat, 30 May 2026 14:19:49 +0000

Prod5 Base KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Sat, 30 May 2026 04:54:21 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sat, 30 May 2026 04:46:14 +0000

qwen-2.5-7b-instruct went down

Prod10 Gateway 0.5 recovered

Fri, 29 May 2026 06:34:25 +0000

Prod10 Gateway 0.5 recovered

Prod10 Gateway 0.5 went down

Fri, 29 May 2026 06:24:27 +0000

Prod10 Gateway 0.5 went down

Prod7 KMS 0.5 recovered

Fri, 29 May 2026 04:11:35 +0000

Prod7 KMS 0.5 recovered

Prod7 KMS 0.5 went down

Fri, 29 May 2026 03:57:40 +0000

Prod7 KMS 0.5 went down

Prod12 KMS 0.5 recovered

Thu, 28 May 2026 00:40:49 +0000

Prod12 KMS 0.5 recovered

Prod12 KMS 0.5 went down

Thu, 28 May 2026 00:36:55 +0000

Prod12 KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Wed, 27 May 2026 11:19:20 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 27 May 2026 10:56:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 27 May 2026 09:34:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 27 May 2026 08:46:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 27 May 2026 08:25:04 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 27 May 2026 08:11:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 27 May 2026 03:39:22 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 27 May 2026 00:21:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 26 May 2026 01:29:27 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 26 May 2026 00:21:18 +0000

qwen-2.5-7b-instruct went down

Prod5 Base Gateway 0.5 recovered

Mon, 25 May 2026 16:53:39 +0000

Prod5 Base Gateway 0.5 recovered

Prod5 Base Gateway 0.5 went down

Mon, 25 May 2026 16:46:48 +0000

Prod5 Base Gateway 0.5 went down

qwen-2.5-7b-instruct recovered

Sun, 24 May 2026 05:14:29 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 24 May 2026 05:01:17 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 recovered

Sat, 23 May 2026 16:01:08 +0000

deepseek-v3.2 recovered

deepseek-v3.2 went down

Sat, 23 May 2026 13:51:06 +0000

deepseek-v3.2 went down

qwen-2.5-7b-instruct recovered

Sat, 23 May 2026 01:29:23 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sat, 23 May 2026 00:16:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 16:54:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 16:46:25 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 16:19:20 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 16:01:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 12:24:25 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 12:16:24 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 05:24:29 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 05:11:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 04:45:24 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 04:26:27 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 04:16:35 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 03:31:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 03:14:54 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 03:01:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 02:50:52 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 02:16:19 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 01:55:04 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 01:30:56 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 22 May 2026 01:15:52 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it recovered

Fri, 22 May 2026 00:55:45 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Fri, 22 May 2026 00:55:18 +0000

gpt-oss-20b recovered

gemma-3-27b-it went down

Fri, 22 May 2026 00:40:58 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Fri, 22 May 2026 00:40:30 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Fri, 22 May 2026 00:21:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 21 May 2026 23:54:30 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 21 May 2026 23:41:25 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 21 May 2026 23:05:20 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 21 May 2026 22:56:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 21 May 2026 22:24:23 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 21 May 2026 22:11:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 21 May 2026 21:39:28 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 21 May 2026 21:26:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 21 May 2026 20:10:03 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 21 May 2026 20:01:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 21 May 2026 19:29:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 21 May 2026 19:11:15 +0000

qwen-2.5-7b-instruct went down

Use3 KMS 0.5 recovered

Thu, 21 May 2026 02:32:45 +0000

Use3 KMS 0.5 recovered

Use3 KMS 0.5 went down

Thu, 21 May 2026 02:29:03 +0000

Use3 KMS 0.5 went down

Prod5 KMS 0.5 recovered

Thu, 21 May 2026 01:49:40 +0000

Prod5 KMS 0.5 recovered

Prod5 KMS 0.5 went down

Thu, 21 May 2026 01:45:41 +0000

Prod5 KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Thu, 21 May 2026 01:20:17 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 21 May 2026 01:01:16 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Wed, 20 May 2026 18:25:37 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Wed, 20 May 2026 18:25:25 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Wed, 20 May 2026 18:19:23 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it went down

Wed, 20 May 2026 18:06:01 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Wed, 20 May 2026 18:05:45 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Wed, 20 May 2026 18:01:46 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 20 May 2026 11:10:12 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 20 May 2026 09:36:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 19 May 2026 11:04:24 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 19 May 2026 10:46:21 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 19 May 2026 01:21:10 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 19 May 2026 01:11:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 19 May 2026 01:06:09 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 19 May 2026 00:36:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 18 May 2026 17:04:17 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 18 May 2026 16:51:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 18 May 2026 03:09:17 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 18 May 2026 02:56:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 18 May 2026 02:24:35 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 18 May 2026 02:06:14 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 18 May 2026 01:44:57 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 18 May 2026 00:21:24 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 18 May 2026 00:14:33 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 18 May 2026 00:01:21 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 17 May 2026 23:30:03 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 17 May 2026 23:16:18 +0000

qwen-2.5-7b-instruct went down

Prod9 KMS 0.5 recovered

Sat, 16 May 2026 11:44:50 +0000

Prod9 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Sat, 16 May 2026 11:44:43 +0000

Prod3 KMS 0.5 recovered

Prod3 KMS 0.5 went down

Sat, 16 May 2026 11:41:08 +0000

Prod3 KMS 0.5 went down

Prod9 KMS 0.5 went down

Sat, 16 May 2026 11:40:56 +0000

Prod9 KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Sat, 16 May 2026 01:50:49 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sat, 16 May 2026 01:41:22 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Fri, 15 May 2026 15:24:19 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 15 May 2026 15:06:16 +0000

qwen-2.5-7b-instruct went down

Prod4 Gateway 0.3 recovered

Thu, 14 May 2026 05:22:42 +0000

Prod4 Gateway 0.3 recovered

Prod4 Gateway 0.3 went down

Thu, 14 May 2026 04:55:33 +0000

Prod4 Gateway 0.3 went down

gemma-3-27b-it recovered

Thu, 14 May 2026 04:06:01 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Thu, 14 May 2026 04:00:41 +0000

gemma-3-27b-it went down

qwen-2.5-7b-instruct recovered

Thu, 14 May 2026 01:25:10 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 14 May 2026 01:16:28 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 14 May 2026 00:45:12 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 14 May 2026 00:26:24 +0000

qwen-2.5-7b-instruct went down

Use3 VMM 0.5 recovered

Wed, 13 May 2026 18:40:06 +0000

Use3 VMM 0.5 recovered

Use3 VMM 0.5 went down

Wed, 13 May 2026 18:36:16 +0000

Use3 VMM 0.5 went down

Use3 KMS 0.5 recovered

Wed, 13 May 2026 18:01:03 +0000

Use3 KMS 0.5 recovered

Use3 KMS 0.5 went down

Wed, 13 May 2026 17:57:42 +0000

Use3 KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Wed, 13 May 2026 01:20:37 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 13 May 2026 01:11:16 +0000

qwen-2.5-7b-instruct went down

Prod7 Gateway 0.5 recovered

Tue, 12 May 2026 19:01:57 +0000

Prod7 Gateway 0.5 recovered

Prod7 Gateway 0.5 went down

Tue, 12 May 2026 15:03:05 +0000

Prod7 Gateway 0.5 went down

Prod7 KMS 0.5 recovered

Mon, 11 May 2026 23:52:41 +0000

Prod7 KMS 0.5 recovered

Prod7 KMS 0.5 went down

Mon, 11 May 2026 23:47:48 +0000

Prod7 KMS 0.5 went down

Prod7 KMS 0.5 recovered

Mon, 11 May 2026 23:24:48 +0000

Prod7 KMS 0.5 recovered

Prod7 KMS 0.5 went down

Mon, 11 May 2026 23:12:48 +0000

Prod7 KMS 0.5 went down

Prod7 KMS 0.5 recovered

Mon, 11 May 2026 21:27:39 +0000

Prod7 KMS 0.5 recovered

Prod7 KMS 0.5 went down

Mon, 11 May 2026 21:22:47 +0000

Prod7 KMS 0.5 went down

Prod7 KMS 0.5 recovered

Mon, 11 May 2026 19:16:40 +0000

Prod7 KMS 0.5 recovered

Prod7 KMS 0.5 went down

Mon, 11 May 2026 19:12:55 +0000

Prod7 KMS 0.5 went down

Prod7 VMM 0.5 recovered

Mon, 11 May 2026 19:04:30 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.5 went down

Mon, 11 May 2026 18:58:31 +0000

Prod7 VMM 0.5 went down

Prod7 KMS 0.5 recovered

Mon, 11 May 2026 18:56:49 +0000

Prod7 KMS 0.5 recovered

Prod7 KMS 0.5 went down

Mon, 11 May 2026 18:52:43 +0000

Prod7 KMS 0.5 went down

Prod10 KMS 0.5 recovered

Sun, 10 May 2026 15:26:16 +0000

Prod10 KMS 0.5 recovered

Phala Cloud API recovered

Sun, 10 May 2026 15:26:16 +0000

Phala Cloud API recovered

Prod2 KMS 0.3 recovered

Sun, 10 May 2026 15:26:12 +0000

Prod2 KMS 0.3 recovered

Use2 KMS 0.5 recovered

Sun, 10 May 2026 15:26:03 +0000

Use2 KMS 0.5 recovered

Prod9 KMS 0.5 recovered

Sun, 10 May 2026 15:25:55 +0000

Prod9 KMS 0.5 recovered

Prod12 KMS 0.5 recovered

Sun, 10 May 2026 15:25:49 +0000

Prod12 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Sun, 10 May 2026 15:25:46 +0000

Use3 KMS 0.5 recovered

Prod6 KMS 0.5 recovered

Sun, 10 May 2026 15:25:31 +0000

Prod6 KMS 0.5 recovered

Phala Cloud API went down

Sun, 10 May 2026 15:17:13 +0000

Phala Cloud API went down

Prod10 KMS 0.5 went down

Sun, 10 May 2026 15:17:11 +0000

Prod10 KMS 0.5 went down

Use2 KMS 0.5 went down

Sun, 10 May 2026 15:17:10 +0000

Use2 KMS 0.5 went down

Use3 KMS 0.5 went down

Sun, 10 May 2026 15:16:57 +0000

Use3 KMS 0.5 went down

Prod9 KMS 0.5 went down

Sun, 10 May 2026 15:16:52 +0000

Prod9 KMS 0.5 went down

Prod12 KMS 0.5 went down

Sun, 10 May 2026 15:16:49 +0000

Prod12 KMS 0.5 went down

Prod6 KMS 0.5 went down

Sun, 10 May 2026 15:16:39 +0000

Prod6 KMS 0.5 went down

Prod2 KMS 0.3 went down

Sun, 10 May 2026 15:16:27 +0000

Prod2 KMS 0.3 went down

Prod5 Gateway 0.5 recovered

Sun, 10 May 2026 15:15:42 +0000

Prod5 Gateway 0.5 recovered

Prod12 KMS 0.5 recovered

Sun, 10 May 2026 14:48:41 +0000

Prod12 KMS 0.5 recovered

Prod5 Gateway 0.5 went down

Sun, 10 May 2026 14:44:49 +0000

Prod5 Gateway 0.5 went down

Prod12 KMS 0.5 went down

Sun, 10 May 2026 14:37:48 +0000

Prod12 KMS 0.5 went down

Prod2 KMS 0.3 recovered

Sun, 10 May 2026 00:38:13 +0000

Prod2 KMS 0.3 recovered

Prod2 KMS 0.3 went down

Sun, 10 May 2026 00:33:29 +0000

Prod2 KMS 0.3 went down

Prod9 Base KMS 0.5 recovered

Fri, 08 May 2026 11:04:15 +0000

Prod9 Base KMS 0.5 recovered

Prod9 Base KMS 0.5 went down

Fri, 08 May 2026 09:50:58 +0000

Prod9 Base KMS 0.5 went down

Prod9 Base KMS 0.5 recovered

Fri, 08 May 2026 09:50:43 +0000

Prod9 Base KMS 0.5 recovered

Prod9 Base KMS 0.5 went down

Fri, 08 May 2026 09:36:00 +0000

Prod9 Base KMS 0.5 went down

Use1 Base KMS 0.5 recovered

Thu, 07 May 2026 01:41:49 +0000

Use1 Base KMS 0.5 recovered

Use1 Base KMS 0.5 went down

Thu, 07 May 2026 01:33:38 +0000

Use1 Base KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Wed, 06 May 2026 01:16:12 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 06 May 2026 00:56:23 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 05 May 2026 17:24:26 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 05 May 2026 17:10:32 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 04 May 2026 13:55:17 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 04 May 2026 13:41:24 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 04 May 2026 13:14:17 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 04 May 2026 13:01:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Mon, 04 May 2026 01:20:41 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Mon, 04 May 2026 00:16:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 03 May 2026 23:50:20 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 03 May 2026 23:36:23 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 03 May 2026 23:09:45 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 03 May 2026 22:56:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 03 May 2026 22:34:26 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 03 May 2026 22:11:18 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 03 May 2026 01:30:54 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 03 May 2026 01:11:15 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 03 May 2026 00:50:28 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 03 May 2026 00:25:21 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sat, 02 May 2026 00:40:28 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sat, 02 May 2026 00:26:25 +0000

qwen-2.5-7b-instruct went down

Prod6 Eth Gateway 0.5 recovered

Fri, 01 May 2026 07:14:47 +0000

Prod6 Eth Gateway 0.5 recovered

Prod6 Base Gateway 0.5 recovered

Fri, 01 May 2026 07:03:54 +0000

Prod6 Base Gateway 0.5 recovered

Prod6 Gateway 0.5 recovered

Fri, 01 May 2026 07:03:26 +0000

Prod6 Gateway 0.5 recovered

Prod6 KMS 0.5 recovered

Fri, 01 May 2026 07:03:19 +0000

Prod6 KMS 0.5 recovered

Prod6 Gateway 0.5.1 recovered

Fri, 01 May 2026 07:03:16 +0000

Prod6 Gateway 0.5.1 recovered

Prod6 VMM 0.5.1 recovered

Fri, 01 May 2026 07:03:07 +0000

Prod6 VMM 0.5.1 recovered

Prod6 VMM 0.3 recovered

Fri, 01 May 2026 07:03:06 +0000

Prod6 VMM 0.3 recovered

Prod6 Gateway 0.3 recovered

Fri, 01 May 2026 07:03:06 +0000

Prod6 Gateway 0.3 recovered

Prod6 Node recovered

Fri, 01 May 2026 07:02:37 +0000

Prod6 Node recovered

Prod6 VMM 0.5 recovered

Fri, 01 May 2026 07:02:20 +0000

Prod6 VMM 0.5 recovered

Prod6 Base Gateway 0.5 went down

Fri, 01 May 2026 06:06:54 +0000

Prod6 Base Gateway 0.5 went down

Prod6 Gateway 0.3 went down

Fri, 01 May 2026 06:06:42 +0000

Prod6 Gateway 0.3 went down

Prod6 Node went down

Fri, 01 May 2026 06:06:35 +0000

Prod6 Node went down

Prod6 Gateway 0.5 went down

Fri, 01 May 2026 06:06:20 +0000

Prod6 Gateway 0.5 went down

Prod6 VMM 0.5 went down

Fri, 01 May 2026 06:06:16 +0000

Prod6 VMM 0.5 went down

Prod6 Gateway 0.5.1 went down

Fri, 01 May 2026 06:06:15 +0000

Prod6 Gateway 0.5.1 went down

Prod6 KMS 0.5 went down

Fri, 01 May 2026 06:06:14 +0000

Prod6 KMS 0.5 went down

Prod6 VMM 0.3 went down

Fri, 01 May 2026 06:06:07 +0000

Prod6 VMM 0.3 went down

Prod6 VMM 0.5.1 went down

Fri, 01 May 2026 06:06:04 +0000

Prod6 VMM 0.5.1 went down

Prod6 Eth Gateway 0.5 went down

Fri, 01 May 2026 06:05:49 +0000

Prod6 Eth Gateway 0.5 went down

qwen-2.5-7b-instruct recovered

Fri, 01 May 2026 01:20:29 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 01 May 2026 01:10:42 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 30 Apr 2026 21:14:18 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 30 Apr 2026 21:01:17 +0000

qwen-2.5-7b-instruct went down

Proof t16z recovered

Thu, 30 Apr 2026 15:33:58 +0000

Proof t16z recovered

Proof t16z went down

Thu, 30 Apr 2026 15:16:40 +0000

Proof t16z went down

Proof t16z recovered

Thu, 30 Apr 2026 15:12:20 +0000

Proof t16z recovered

Proof t16z went down

Thu, 30 Apr 2026 15:08:31 +0000

Proof t16z went down

Prod10 Eth KMS 0.5 recovered

Thu, 30 Apr 2026 11:36:51 +0000

Prod10 Eth KMS 0.5 recovered

Prod6 Eth KMS 0.5 recovered

Thu, 30 Apr 2026 11:36:43 +0000

Prod6 Eth KMS 0.5 recovered

Prod6 Base KMS 0.5 recovered

Thu, 30 Apr 2026 11:36:35 +0000

Prod6 Base KMS 0.5 recovered

Prod6 KMS 0.5.1 recovered

Thu, 30 Apr 2026 11:36:30 +0000

Prod6 KMS 0.5.1 recovered

Prod10 Base KMS 0.5 recovered

Thu, 30 Apr 2026 11:36:25 +0000

Prod10 Base KMS 0.5 recovered

Prod10 Eth Gateway 0.5 recovered

Wed, 29 Apr 2026 19:46:13 +0000

Prod10 Eth Gateway 0.5 recovered

Prod10 Base Gateway 0.5 recovered

Wed, 29 Apr 2026 19:41:59 +0000

Prod10 Base Gateway 0.5 recovered

Prod10 Gateway 0.5 recovered

Wed, 29 Apr 2026 10:23:24 +0000

Prod10 Gateway 0.5 recovered

Prod10 KMS 0.5 recovered

Wed, 29 Apr 2026 10:23:17 +0000

Prod10 KMS 0.5 recovered

Prod10 VMM 0.5 recovered

Wed, 29 Apr 2026 10:14:04 +0000

Prod10 VMM 0.5 recovered

Prod10 Node recovered

Wed, 29 Apr 2026 10:13:13 +0000

Prod10 Node recovered

Prod10 Node went down

Wed, 29 Apr 2026 10:08:05 +0000

Prod10 Node went down

Prod10 Node recovered

Wed, 29 Apr 2026 09:57:12 +0000

Prod10 Node recovered

Prod10 Eth KMS 0.5 went down

Wed, 29 Apr 2026 03:27:16 +0000

Prod10 Eth KMS 0.5 went down

Prod10 Eth Gateway 0.5 went down

Wed, 29 Apr 2026 03:27:14 +0000

Prod10 Eth Gateway 0.5 went down

Prod10 VMM 0.5 went down

Wed, 29 Apr 2026 03:27:04 +0000

Prod10 VMM 0.5 went down

Prod10 Node went down

Wed, 29 Apr 2026 03:27:04 +0000

Prod10 Node went down

Prod10 Base Gateway 0.5 went down

Wed, 29 Apr 2026 03:26:52 +0000

Prod10 Base Gateway 0.5 went down

Prod10 Base KMS 0.5 went down

Wed, 29 Apr 2026 03:26:47 +0000

Prod10 Base KMS 0.5 went down

Prod10 Gateway 0.5 went down

Wed, 29 Apr 2026 03:26:17 +0000

Prod10 Gateway 0.5 went down

Prod10 KMS 0.5 went down

Wed, 29 Apr 2026 03:26:15 +0000

Prod10 KMS 0.5 went down

Prod10 TCP-R update maintenance

Wed, 29 Apr 2026 03:00:00 -0000

We're scheduling Intel TCB Recovery (TCB-R) firmware upgrades across our nodes. This upgrade is required to keep attestation quotes valid — after May 13, 2026, unupdated nodes will generate OutOfDate quotes. Please review the maintenance windows below and let us know if you have any concerns. Node: prod10 - Date: April 29, 2026 - Time: 03:00–05:00 UTC (11:00–13:00 CST) - Expected downtime: ~2 hour(s)

qwen-2.5-7b-instruct recovered

Wed, 29 Apr 2026 02:39:26 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 29 Apr 2026 02:31:16 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Wed, 29 Apr 2026 02:04:27 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Wed, 29 Apr 2026 01:46:17 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Tue, 28 Apr 2026 21:39:28 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 28 Apr 2026 21:31:18 +0000

qwen-2.5-7b-instruct went down

Prod6 Eth Gateway 0.5 recovered

Tue, 28 Apr 2026 08:24:59 +0000

Prod6 Eth Gateway 0.5 recovered

Prod6 Base Gateway 0.5 recovered

Tue, 28 Apr 2026 07:59:00 +0000

Prod6 Base Gateway 0.5 recovered

Prod6 Gateway 0.5.1 recovered

Tue, 28 Apr 2026 07:31:14 +0000

Prod6 Gateway 0.5.1 recovered

Prod6 Gateway 0.5 recovered

Tue, 28 Apr 2026 06:59:35 +0000

Prod6 Gateway 0.5 recovered

Prod6 KMS 0.5 recovered

Tue, 28 Apr 2026 06:58:35 +0000

Prod6 KMS 0.5 recovered

Prod6 VMM 0.5 recovered

Tue, 28 Apr 2026 06:58:20 +0000

Prod6 VMM 0.5 recovered

Prod6 VMM 0.5.1 recovered

Tue, 28 Apr 2026 06:58:08 +0000

Prod6 VMM 0.5.1 recovered

Prod6 VMM 0.3 recovered

Tue, 28 Apr 2026 06:58:07 +0000

Prod6 VMM 0.3 recovered

Prod6 VMM 0.5.1 went down

Tue, 28 Apr 2026 06:52:05 +0000

Prod6 VMM 0.5.1 went down

Prod6 VMM 0.3 went down

Tue, 28 Apr 2026 06:52:00 +0000

Prod6 VMM 0.3 went down

Prod6 Gateway 0.5 went down

Tue, 28 Apr 2026 06:50:54 +0000

Prod6 Gateway 0.5 went down

Prod6 KMS 0.5 went down

Tue, 28 Apr 2026 06:50:21 +0000

Prod6 KMS 0.5 went down

Prod6 VMM 0.5 went down

Tue, 28 Apr 2026 06:50:17 +0000

Prod6 VMM 0.5 went down

Prod6 KMS 0.5 recovered

Tue, 28 Apr 2026 06:46:37 +0000

Prod6 KMS 0.5 recovered

Prod6 Gateway 0.5 recovered

Tue, 28 Apr 2026 06:46:26 +0000

Prod6 Gateway 0.5 recovered

Prod6 VMM 0.3 recovered

Tue, 28 Apr 2026 06:43:13 +0000

Prod6 VMM 0.3 recovered

Prod6 VMM 0.5.1 recovered

Tue, 28 Apr 2026 06:43:12 +0000

Prod6 VMM 0.5.1 recovered

Prod6 Node recovered

Tue, 28 Apr 2026 06:42:44 +0000

Prod6 Node recovered

Prod6 Gateway 0.3 recovered

Tue, 28 Apr 2026 06:42:41 +0000

Prod6 Gateway 0.3 recovered

Prod6 VMM 0.5 recovered

Tue, 28 Apr 2026 06:42:09 +0000

Prod6 VMM 0.5 recovered

Prod6 VMM 0.3 went down

Tue, 28 Apr 2026 06:32:04 +0000

Prod6 VMM 0.3 went down

Prod6 Node went down

Tue, 28 Apr 2026 06:31:36 +0000

Prod6 Node went down

Prod6 Gateway 0.3 went down

Tue, 28 Apr 2026 06:31:35 +0000

Prod6 Gateway 0.3 went down

Prod6 VMM 0.5.1 went down

Tue, 28 Apr 2026 06:30:10 +0000

Prod6 VMM 0.5.1 went down

Prod6 Gateway 0.5 went down

Tue, 28 Apr 2026 06:29:37 +0000

Prod6 Gateway 0.5 went down

Prod6 KMS 0.5 went down

Tue, 28 Apr 2026 06:29:28 +0000

Prod6 KMS 0.5 went down

Prod6 VMM 0.5 went down

Tue, 28 Apr 2026 06:29:22 +0000

Prod6 VMM 0.5 went down

Prod6 Gateway 0.5 recovered

Tue, 28 Apr 2026 06:21:38 +0000

Prod6 Gateway 0.5 recovered

Prod6 KMS 0.5 recovered

Tue, 28 Apr 2026 06:20:26 +0000

Prod6 KMS 0.5 recovered

Prod6 Gateway 0.5 went down

Tue, 28 Apr 2026 06:18:26 +0000

Prod6 Gateway 0.5 went down

Prod6 KMS 0.5 went down

Tue, 28 Apr 2026 06:16:40 +0000

Prod6 KMS 0.5 went down

Prod6 Gateway 0.5 recovered

Tue, 28 Apr 2026 06:11:34 +0000

Prod6 Gateway 0.5 recovered

Prod6 KMS 0.5 recovered

Tue, 28 Apr 2026 06:10:36 +0000

Prod6 KMS 0.5 recovered

Prod6 Gateway 0.3 recovered

Tue, 28 Apr 2026 05:52:47 +0000

Prod6 Gateway 0.3 recovered

Prod6 VMM 0.3 recovered

Tue, 28 Apr 2026 05:52:23 +0000

Prod6 VMM 0.3 recovered

Prod6 VMM 0.5 recovered

Tue, 28 Apr 2026 05:52:14 +0000

Prod6 VMM 0.5 recovered

Prod6 Node recovered

Tue, 28 Apr 2026 05:52:10 +0000

Prod6 Node recovered

Prod6 VMM 0.5.1 recovered

Tue, 28 Apr 2026 05:52:04 +0000

Prod6 VMM 0.5.1 recovered

Prod6 VMM 0.5 went down

Tue, 28 Apr 2026 05:44:15 +0000

Prod6 VMM 0.5 went down

Prod6 VMM 0.5.1 went down

Tue, 28 Apr 2026 05:44:13 +0000

Prod6 VMM 0.5.1 went down

Prod6 Gateway 0.5.1 went down

Tue, 28 Apr 2026 05:44:06 +0000

Prod6 Gateway 0.5.1 went down

Prod6 VMM 0.3 went down

Tue, 28 Apr 2026 05:44:04 +0000

Prod6 VMM 0.3 went down

Prod6 Node went down

Tue, 28 Apr 2026 05:43:08 +0000

Prod6 Node went down

Prod6 Gateway 0.3 went down

Tue, 28 Apr 2026 05:42:38 +0000

Prod6 Gateway 0.3 went down

Prod6 KMS 0.5 went down

Tue, 28 Apr 2026 05:42:33 +0000

Prod6 KMS 0.5 went down

Prod6 KMS 0.5.1 went down

Tue, 28 Apr 2026 05:42:25 +0000

Prod6 KMS 0.5.1 went down

Prod6 Gateway 0.5.1 recovered

Tue, 28 Apr 2026 05:39:16 +0000

Prod6 Gateway 0.5.1 recovered

Prod6 VMM 0.3 recovered

Tue, 28 Apr 2026 05:38:07 +0000

Prod6 VMM 0.3 recovered

Prod6 Node recovered

Tue, 28 Apr 2026 05:37:49 +0000

Prod6 Node recovered

Prod6 Gateway 0.3 recovered

Tue, 28 Apr 2026 05:37:46 +0000

Prod6 Gateway 0.3 recovered

Prod6 Gateway 0.5.1 went down

Tue, 28 Apr 2026 05:35:04 +0000

Prod6 Gateway 0.5.1 went down

Prod6 VMM 0.3 went down

Tue, 28 Apr 2026 05:34:56 +0000

Prod6 VMM 0.3 went down

Prod6 Gateway 0.3 went down

Tue, 28 Apr 2026 05:33:38 +0000

Prod6 Gateway 0.3 went down

Prod6 Node went down

Tue, 28 Apr 2026 05:33:37 +0000

Prod6 Node went down

Prod6 Gateway 0.5 went down

Tue, 28 Apr 2026 05:33:34 +0000

Prod6 Gateway 0.5 went down

Prod6 Gateway 0.5.1 recovered

Tue, 28 Apr 2026 04:27:16 +0000

Prod6 Gateway 0.5.1 recovered

Prod6 KMS 0.5.1 recovered

Tue, 28 Apr 2026 04:26:27 +0000

Prod6 KMS 0.5.1 recovered

Prod6 Base KMS 0.5 went down

Tue, 28 Apr 2026 04:19:49 +0000

Prod6 Base KMS 0.5 went down

Prod6 Base KMS 0.5 recovered

Tue, 28 Apr 2026 04:04:50 +0000

Prod6 Base KMS 0.5 recovered

Prod6 Gateway 0.5 recovered

Tue, 28 Apr 2026 03:48:36 +0000

Prod6 Gateway 0.5 recovered

Prod6 KMS 0.5 recovered

Tue, 28 Apr 2026 03:47:24 +0000

Prod6 KMS 0.5 recovered

Prod6 VMM 0.5.1 recovered

Tue, 28 Apr 2026 03:34:09 +0000

Prod6 VMM 0.5.1 recovered

Prod6 VMM 0.3 recovered

Tue, 28 Apr 2026 03:34:06 +0000

Prod6 VMM 0.3 recovered

Prod6 Gateway 0.3 recovered

Tue, 28 Apr 2026 03:33:47 +0000

Prod6 Gateway 0.3 recovered

Prod6 Node recovered

Tue, 28 Apr 2026 03:33:37 +0000

Prod6 Node recovered

Prod6 VMM 0.5 recovered

Tue, 28 Apr 2026 03:33:29 +0000

Prod6 VMM 0.5 recovered

Prod6 TCP-R update maintenance

Tue, 28 Apr 2026 03:15:00 -0000

Prod6 TCP-R update maintenance

Prod6 Base Gateway 0.5 went down

Tue, 28 Apr 2026 03:11:14 +0000

Prod6 Base Gateway 0.5 went down

Prod6 Gateway 0.5.1 went down

Tue, 28 Apr 2026 03:11:11 +0000

Prod6 Gateway 0.5.1 went down

Prod6 Eth KMS 0.5 went down

Tue, 28 Apr 2026 03:11:06 +0000

Prod6 Eth KMS 0.5 went down

Prod6 VMM 0.5.1 went down

Tue, 28 Apr 2026 03:11:02 +0000

Prod6 VMM 0.5.1 went down

Prod6 VMM 0.3 went down

Tue, 28 Apr 2026 03:10:57 +0000

Prod6 VMM 0.3 went down

Prod6 Eth Gateway 0.5 went down

Tue, 28 Apr 2026 03:10:52 +0000

Prod6 Eth Gateway 0.5 went down

Prod6 Gateway 0.3 went down

Tue, 28 Apr 2026 03:10:44 +0000

Prod6 Gateway 0.3 went down

Prod6 Base KMS 0.5 went down

Tue, 28 Apr 2026 03:10:37 +0000

Prod6 Base KMS 0.5 went down

Prod6 Gateway 0.5 went down

Tue, 28 Apr 2026 03:10:34 +0000

Prod6 Gateway 0.5 went down

Prod6 Node went down

Tue, 28 Apr 2026 03:10:29 +0000

Prod6 Node went down

Prod6 KMS 0.5.1 went down

Tue, 28 Apr 2026 03:10:26 +0000

Prod6 KMS 0.5.1 went down

Prod6 VMM 0.5 went down

Tue, 28 Apr 2026 03:10:20 +0000

Prod6 VMM 0.5 went down

Prod6 KMS 0.5 went down

Tue, 28 Apr 2026 03:09:47 +0000

Prod6 KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Tue, 28 Apr 2026 01:35:26 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Tue, 28 Apr 2026 01:16:25 +0000

qwen-2.5-7b-instruct went down

Use2 Gateway 0.5 recovered

Mon, 27 Apr 2026 02:22:21 +0000

Use2 Gateway 0.5 recovered

Use2 KMS 0.5 recovered

Mon, 27 Apr 2026 02:22:09 +0000

Use2 KMS 0.5 recovered

Use2 VMM 0.5 recovered

Mon, 27 Apr 2026 02:11:52 +0000

Use2 VMM 0.5 recovered

Use2 Node recovered

Mon, 27 Apr 2026 01:51:29 +0000

Use2 Node recovered

Use2 Node went down

Mon, 27 Apr 2026 01:03:28 +0000

Use2 Node went down

Use2 Gateway 0.5 went down

Mon, 27 Apr 2026 01:03:19 +0000

Use2 Gateway 0.5 went down

Use2 KMS 0.5 went down

Mon, 27 Apr 2026 01:03:06 +0000

Use2 KMS 0.5 went down

Use2 VMM 0.5 went down

Mon, 27 Apr 2026 01:02:48 +0000

Use2 VMM 0.5 went down

qwen-2.5-7b-instruct recovered

Sun, 26 Apr 2026 01:49:25 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 26 Apr 2026 01:31:33 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Sun, 26 Apr 2026 00:35:08 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Sun, 26 Apr 2026 00:26:24 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 23 Apr 2026 22:44:54 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 23 Apr 2026 22:36:27 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 23 Apr 2026 21:04:16 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 23 Apr 2026 20:51:23 +0000

qwen-2.5-7b-instruct went down

qwen-2.5-7b-instruct recovered

Thu, 23 Apr 2026 19:45:16 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Thu, 23 Apr 2026 19:26:20 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Thu, 23 Apr 2026 07:30:44 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Thu, 23 Apr 2026 07:30:18 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Thu, 23 Apr 2026 07:29:16 +0000

qwen-2.5-7b-instruct recovered

Use1 Base Gateway 0.5 recovered

Thu, 23 Apr 2026 06:57:44 +0000

Use1 Base Gateway 0.5 recovered

Use1 VMM 0.5 recovered

Thu, 23 Apr 2026 04:24:34 +0000

Use1 VMM 0.5 recovered

Use1 Gateway 0.5 recovered

Thu, 23 Apr 2026 04:24:19 +0000

Use1 Gateway 0.5 recovered

Use1 Base KMS 0.5 recovered

Thu, 23 Apr 2026 04:24:01 +0000

Use1 Base KMS 0.5 recovered

Use1 KMS 0.5 recovered

Thu, 23 Apr 2026 04:23:42 +0000

Use1 KMS 0.5 recovered

Use1 Node recovered

Thu, 23 Apr 2026 04:23:26 +0000

Use1 Node recovered

Use1 Node went down

Thu, 23 Apr 2026 04:17:17 +0000

Use1 Node went down

Use1 VMM 0.5 went down

Thu, 23 Apr 2026 04:17:11 +0000

Use1 VMM 0.5 went down

Use1 VMM 0.5 recovered

Thu, 23 Apr 2026 04:08:46 +0000

Use1 VMM 0.5 recovered

Use1 Node recovered

Thu, 23 Apr 2026 04:07:26 +0000

Use1 Node recovered

Use1 Base KMS 0.5 went down

Thu, 23 Apr 2026 03:55:00 +0000

Use1 Base KMS 0.5 went down

Use1 KMS 0.5 went down

Thu, 23 Apr 2026 03:54:55 +0000

Use1 KMS 0.5 went down

Use1 Gateway 0.5 went down

Thu, 23 Apr 2026 03:54:46 +0000

Use1 Gateway 0.5 went down

Use1 VMM 0.5 went down

Thu, 23 Apr 2026 03:54:44 +0000

Use1 VMM 0.5 went down

Use1 Node went down

Thu, 23 Apr 2026 03:54:17 +0000

Use1 Node went down

Use1 Base KMS 0.5 recovered

Thu, 23 Apr 2026 03:46:15 +0000

Use1 Base KMS 0.5 recovered

Use1 Gateway 0.5 recovered

Thu, 23 Apr 2026 03:43:05 +0000

Use1 Gateway 0.5 recovered

Use1 KMS 0.5 recovered

Thu, 23 Apr 2026 03:41:52 +0000

Use1 KMS 0.5 recovered

Use1 VMM 0.5 recovered

Thu, 23 Apr 2026 03:34:40 +0000

Use1 VMM 0.5 recovered

Use1 Node recovered

Thu, 23 Apr 2026 03:34:34 +0000

Use1 Node recovered

gemma-3-27b-it went down

Thu, 23 Apr 2026 03:10:46 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Thu, 23 Apr 2026 03:10:30 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Thu, 23 Apr 2026 03:09:27 +0000

qwen-2.5-7b-instruct went down

use1 maintainance

Thu, 23 Apr 2026 03:09:00 -0000

use1 maintainance

Use1 VMM 0.5 went down

Thu, 23 Apr 2026 03:08:41 +0000

Use1 VMM 0.5 went down

Use1 Node went down

Thu, 23 Apr 2026 03:08:29 +0000

Use1 Node went down

Use1 Base Gateway 0.5 went down

Thu, 23 Apr 2026 03:08:13 +0000

Use1 Base Gateway 0.5 went down

Use1 Base KMS 0.5 went down

Thu, 23 Apr 2026 03:08:09 +0000

Use1 Base KMS 0.5 went down

Use1 Gateway 0.5 went down

Thu, 23 Apr 2026 03:08:02 +0000

Use1 Gateway 0.5 went down

Use1 KMS 0.5 went down

Thu, 23 Apr 2026 03:08:00 +0000

Use1 KMS 0.5 went down

Prod1 Gateway 0.3 recovered

Wed, 22 Apr 2026 20:52:30 +0000

Prod1 Gateway 0.3 recovered

Prod1 VMM 0.3 recovered

Wed, 22 Apr 2026 20:46:54 +0000

Prod1 VMM 0.3 recovered

Prod1 Node recovered

Wed, 22 Apr 2026 20:19:12 +0000

Prod1 Node recovered

Prod1 Node went down

Wed, 22 Apr 2026 19:07:10 +0000

Prod1 Node went down

Prod1 Gateway 0.3 went down

Wed, 22 Apr 2026 19:06:16 +0000

Prod1 Gateway 0.3 went down

prod1 maintainance

Wed, 22 Apr 2026 03:10:00 -0000

prod1 maintainance

Prod1 VMM 0.3 went down

Wed, 22 Apr 2026 03:03:30 +0000

Prod1 VMM 0.3 went down

gemma-3-27b-it recovered

Fri, 17 Apr 2026 11:00:58 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Fri, 17 Apr 2026 10:41:42 +0000

gemma-3-27b-it went down

deepseek-v3.2 recovered

Fri, 17 Apr 2026 10:17:02 +0000

deepseek-v3.2 recovered

gpt-oss-20b recovered

Fri, 17 Apr 2026 10:10:34 +0000

gpt-oss-20b recovered

deepseek-v3.2 went down

Fri, 17 Apr 2026 10:06:13 +0000

deepseek-v3.2 went down

gemma-3-27b-it recovered

Fri, 17 Apr 2026 10:00:54 +0000

gemma-3-27b-it recovered

qwen-2.5-7b-instruct recovered

Fri, 17 Apr 2026 09:54:19 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it went down

Fri, 17 Apr 2026 09:50:47 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Fri, 17 Apr 2026 09:50:29 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Fri, 17 Apr 2026 09:45:31 +0000

qwen-2.5-7b-instruct went down

gemma-3-27b-it recovered

Fri, 17 Apr 2026 07:10:54 +0000

gemma-3-27b-it recovered

qwen-2.5-7b-instruct recovered

Fri, 17 Apr 2026 07:04:18 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it went down

Fri, 17 Apr 2026 07:01:50 +0000

gemma-3-27b-it went down

qwen-2.5-7b-instruct went down

Fri, 17 Apr 2026 06:54:18 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 recovered

Fri, 17 Apr 2026 05:46:39 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 17 Apr 2026 05:45:46 +0000

gemma-3-27b-it recovered

qwen-2.5-7b-instruct recovered

Fri, 17 Apr 2026 05:44:38 +0000

qwen-2.5-7b-instruct recovered

qwen-2.5-7b-instruct went down

Fri, 17 Apr 2026 05:39:32 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 went down

Fri, 17 Apr 2026 05:37:13 +0000

deepseek-v3.2 went down

gemma-3-27b-it went down

Fri, 17 Apr 2026 05:35:53 +0000

gemma-3-27b-it went down

Prod9 Base KMS 0.5 recovered

Tue, 14 Apr 2026 00:45:56 +0000

Prod9 Base KMS 0.5 recovered

Prod5 Base KMS 0.5 recovered

Tue, 14 Apr 2026 00:45:41 +0000

Prod5 Base KMS 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Tue, 14 Apr 2026 00:45:25 +0000

Prod9 Eth KMS 0.5 recovered

Proof t16z recovered

Tue, 14 Apr 2026 00:45:19 +0000

Proof t16z recovered

Use2 KMS 0.5 recovered

Tue, 14 Apr 2026 00:45:05 +0000

Use2 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:49 +0000

Use3 KMS 0.5 recovered

Prod9 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:49 +0000

Prod9 KMS 0.5 recovered

Use1 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:49 +0000

Use1 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:49 +0000

Prod3 KMS 0.5 recovered

Prod7 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:46 +0000

Prod7 KMS 0.5 recovered

Prod6 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:46 +0000

Prod6 KMS 0.5 recovered

Prod11 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:40 +0000

Prod11 KMS 0.5 recovered

Prod5 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:36 +0000

Prod5 KMS 0.5 recovered

Prod10 KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:32 +0000

Prod10 KMS 0.5 recovered

Prod2 KMS 0.3 recovered

Tue, 14 Apr 2026 00:44:30 +0000

Prod2 KMS 0.3 recovered

Prod7 Eth KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:25 +0000

Prod7 Eth KMS 0.5 recovered

Phala Cloud API recovered

Tue, 14 Apr 2026 00:44:14 +0000

Phala Cloud API recovered

Prod6 Eth KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:03 +0000

Prod6 Eth KMS 0.5 recovered

Use1 Base KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:03 +0000

Use1 Base KMS 0.5 recovered

Prod10 Eth KMS 0.5 recovered

Tue, 14 Apr 2026 00:44:00 +0000

Prod10 Eth KMS 0.5 recovered

Prod7 Base KMS 0.5 recovered

Tue, 14 Apr 2026 00:43:58 +0000

Prod7 Base KMS 0.5 recovered

Prod6 Base KMS 0.5 recovered

Tue, 14 Apr 2026 00:43:57 +0000

Prod6 Base KMS 0.5 recovered

Prod10 Base KMS 0.5 recovered

Tue, 14 Apr 2026 00:43:56 +0000

Prod10 Base KMS 0.5 recovered

Prod6 KMS 0.5.1 recovered

Tue, 14 Apr 2026 00:43:53 +0000

Prod6 KMS 0.5.1 recovered

Prod6 KMS 0.5.1 went down

Tue, 14 Apr 2026 00:40:24 +0000

Prod6 KMS 0.5.1 went down

Phala Cloud API went down

Tue, 14 Apr 2026 00:40:10 +0000

Phala Cloud API went down

Prod10 KMS 0.5 went down

Tue, 14 Apr 2026 00:40:05 +0000

Prod10 KMS 0.5 went down

Use2 KMS 0.5 went down

Tue, 14 Apr 2026 00:39:58 +0000

Use2 KMS 0.5 went down

Prod5 Base KMS 0.5 went down

Tue, 14 Apr 2026 00:39:53 +0000

Prod5 Base KMS 0.5 went down

Prod9 KMS 0.5 went down

Tue, 14 Apr 2026 00:39:49 +0000

Prod9 KMS 0.5 went down

Prod9 Eth KMS 0.5 went down

Tue, 14 Apr 2026 00:39:47 +0000

Prod9 Eth KMS 0.5 went down

Prod7 Eth KMS 0.5 went down

Tue, 14 Apr 2026 00:39:46 +0000

Prod7 Eth KMS 0.5 went down

Use3 KMS 0.5 went down

Tue, 14 Apr 2026 00:39:44 +0000

Use3 KMS 0.5 went down

Prod11 KMS 0.5 went down

Tue, 14 Apr 2026 00:39:43 +0000

Prod11 KMS 0.5 went down

Prod5 KMS 0.5 went down

Tue, 14 Apr 2026 00:39:32 +0000

Prod5 KMS 0.5 went down

Prod6 KMS 0.5 went down

Tue, 14 Apr 2026 00:39:21 +0000

Prod6 KMS 0.5 went down

Prod6 Eth KMS 0.5 went down

Tue, 14 Apr 2026 00:39:21 +0000

Prod6 Eth KMS 0.5 went down

Use1 Base KMS 0.5 went down

Tue, 14 Apr 2026 00:39:21 +0000

Use1 Base KMS 0.5 went down

Prod9 Base KMS 0.5 went down

Tue, 14 Apr 2026 00:39:18 +0000

Prod9 Base KMS 0.5 went down

Prod10 Eth KMS 0.5 went down

Tue, 14 Apr 2026 00:39:15 +0000

Prod10 Eth KMS 0.5 went down

Prod7 Base KMS 0.5 went down

Tue, 14 Apr 2026 00:39:14 +0000

Prod7 Base KMS 0.5 went down

Prod2 KMS 0.3 went down

Tue, 14 Apr 2026 00:39:13 +0000

Prod2 KMS 0.3 went down

Prod6 Base KMS 0.5 went down

Tue, 14 Apr 2026 00:39:12 +0000

Prod6 Base KMS 0.5 went down

Prod10 Base KMS 0.5 went down

Tue, 14 Apr 2026 00:39:06 +0000

Prod10 Base KMS 0.5 went down

Use1 KMS 0.5 went down

Tue, 14 Apr 2026 00:38:45 +0000

Use1 KMS 0.5 went down

Prod3 KMS 0.5 went down

Tue, 14 Apr 2026 00:38:43 +0000

Prod3 KMS 0.5 went down

Prod7 KMS 0.5 went down

Tue, 14 Apr 2026 00:38:38 +0000

Prod7 KMS 0.5 went down

Proof t16z went down

Tue, 14 Apr 2026 00:38:21 +0000

Proof t16z went down

gpt-oss-20b recovered

Mon, 13 Apr 2026 20:05:19 +0000

gpt-oss-20b recovered

Use3 KMS 0.5 recovered

Mon, 13 Apr 2026 20:04:57 +0000

Use3 KMS 0.5 recovered

qwen-2.5-7b-instruct recovered

Mon, 13 Apr 2026 20:04:16 +0000

qwen-2.5-7b-instruct recovered

Use2 KMS 0.5 recovered

Mon, 13 Apr 2026 20:04:09 +0000

Use2 KMS 0.5 recovered

Use1 KMS 0.5 recovered

Mon, 13 Apr 2026 20:03:04 +0000

Use1 KMS 0.5 recovered

Use1 KMS 0.5 went down

Mon, 13 Apr 2026 19:59:51 +0000

Use1 KMS 0.5 went down

Use2 KMS 0.5 went down

Mon, 13 Apr 2026 19:59:13 +0000

Use2 KMS 0.5 went down

Use3 KMS 0.5 went down

Mon, 13 Apr 2026 19:55:37 +0000

Use3 KMS 0.5 went down

gpt-oss-20b went down

Mon, 13 Apr 2026 19:55:35 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Mon, 13 Apr 2026 19:54:42 +0000

qwen-2.5-7b-instruct went down

Use1 KMS 0.5 recovered

Mon, 13 Apr 2026 00:40:55 +0000

Use1 KMS 0.5 recovered

gpt-oss-20b recovered

Mon, 13 Apr 2026 00:40:27 +0000

gpt-oss-20b recovered

Use3 KMS 0.5 recovered

Mon, 13 Apr 2026 00:35:46 +0000

Use3 KMS 0.5 recovered

Use2 KMS 0.5 recovered

Mon, 13 Apr 2026 00:35:38 +0000

Use2 KMS 0.5 recovered

Use3 KMS 0.5 went down

Mon, 13 Apr 2026 00:31:45 +0000

Use3 KMS 0.5 went down

Use2 KMS 0.5 went down

Mon, 13 Apr 2026 00:31:25 +0000

Use2 KMS 0.5 went down

Use1 KMS 0.5 went down

Mon, 13 Apr 2026 00:31:17 +0000

Use1 KMS 0.5 went down

gpt-oss-20b went down

Mon, 13 Apr 2026 00:30:56 +0000

gpt-oss-20b went down

gemma-3-27b-it recovered

Mon, 13 Apr 2026 00:25:44 +0000

gemma-3-27b-it recovered

Use3 KMS 0.5 recovered

Mon, 13 Apr 2026 00:24:53 +0000

Use3 KMS 0.5 recovered

Use2 KMS 0.5 recovered

Mon, 13 Apr 2026 00:23:59 +0000

Use2 KMS 0.5 recovered

Use1 KMS 0.5 recovered

Mon, 13 Apr 2026 00:23:53 +0000

Use1 KMS 0.5 recovered

Use2 KMS 0.5 went down

Mon, 13 Apr 2026 00:16:21 +0000

Use2 KMS 0.5 went down

Use3 KMS 0.5 went down

Mon, 13 Apr 2026 00:16:21 +0000

Use3 KMS 0.5 went down

Use1 KMS 0.5 went down

Mon, 13 Apr 2026 00:16:14 +0000

Use1 KMS 0.5 went down

gemma-3-27b-it went down

Mon, 13 Apr 2026 00:16:02 +0000

gemma-3-27b-it went down

Use1 KMS 0.5 recovered

Sun, 12 Apr 2026 22:47:55 +0000

Use1 KMS 0.5 recovered

Use1 KMS 0.5 went down

Sun, 12 Apr 2026 22:39:07 +0000

Use1 KMS 0.5 went down

gpt-oss-20b recovered

Sun, 12 Apr 2026 12:10:20 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Sun, 12 Apr 2026 12:09:26 +0000

qwen-2.5-7b-instruct recovered

Use1 KMS 0.5 recovered

Sun, 12 Apr 2026 12:08:53 +0000

Use1 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Sun, 12 Apr 2026 12:08:51 +0000

Use3 KMS 0.5 recovered

gpt-oss-20b went down

Sun, 12 Apr 2026 12:00:48 +0000

gpt-oss-20b went down

Use3 KMS 0.5 went down

Sun, 12 Apr 2026 12:00:05 +0000

Use3 KMS 0.5 went down

qwen-2.5-7b-instruct went down

Sun, 12 Apr 2026 11:59:43 +0000

qwen-2.5-7b-instruct went down

Use1 KMS 0.5 went down

Sun, 12 Apr 2026 11:59:01 +0000

Use1 KMS 0.5 went down

gpt-oss-20b recovered

Sat, 11 Apr 2026 18:40:21 +0000

gpt-oss-20b recovered

Use1 KMS 0.5 recovered

Sat, 11 Apr 2026 18:38:59 +0000

Use1 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Sat, 11 Apr 2026 18:38:56 +0000

Use3 KMS 0.5 recovered

gpt-oss-20b went down

Sat, 11 Apr 2026 18:31:06 +0000

gpt-oss-20b went down

Use3 KMS 0.5 went down

Sat, 11 Apr 2026 18:30:08 +0000

Use3 KMS 0.5 went down

Use1 KMS 0.5 went down

Sat, 11 Apr 2026 18:23:49 +0000

Use1 KMS 0.5 went down

Use1 KMS 0.5 recovered

Sat, 11 Apr 2026 13:53:47 +0000

Use1 KMS 0.5 recovered

Use1 KMS 0.5 went down

Sat, 11 Apr 2026 13:49:02 +0000

Use1 KMS 0.5 went down

Use3 KMS 0.5 recovered

Sat, 11 Apr 2026 13:35:47 +0000

Use3 KMS 0.5 recovered

Use3 KMS 0.5 went down

Sat, 11 Apr 2026 13:31:03 +0000

Use3 KMS 0.5 went down

Use3 KMS 0.5 recovered

Wed, 01 Apr 2026 09:29:39 +0000

Use3 KMS 0.5 recovered

Use3 KMS 0.5 went down

Wed, 01 Apr 2026 09:25:58 +0000

Use3 KMS 0.5 went down

Use2 KMS 0.5 recovered

Wed, 01 Apr 2026 09:24:08 +0000

Use2 KMS 0.5 recovered

Prod4 Gateway 0.3 recovered

Wed, 01 Apr 2026 09:23:24 +0000

Prod4 Gateway 0.3 recovered

Prod4 VMM 0.3 recovered

Wed, 01 Apr 2026 09:23:04 +0000

Prod4 VMM 0.3 recovered

Use1 KMS 0.5 recovered

Wed, 01 Apr 2026 09:22:05 +0000

Use1 KMS 0.5 recovered

Use2 KMS 0.5 went down

Wed, 01 Apr 2026 09:20:11 +0000

Use2 KMS 0.5 went down

Use1 KMS 0.5 went down

Wed, 01 Apr 2026 09:17:51 +0000

Use1 KMS 0.5 went down

Prod10 KMS 0.5 recovered

Wed, 01 Apr 2026 09:16:15 +0000

Prod10 KMS 0.5 recovered

Prod10 KMS 0.5 went down

Wed, 01 Apr 2026 09:12:14 +0000

Prod10 KMS 0.5 went down

Prod6 KMS 0.5 recovered

Wed, 01 Apr 2026 08:50:26 +0000

Prod6 KMS 0.5 recovered

Prod5 KMS 0.5 recovered

Wed, 01 Apr 2026 08:47:32 +0000

Prod5 KMS 0.5 recovered

Prod6 KMS 0.5 went down

Wed, 01 Apr 2026 08:46:28 +0000

Prod6 KMS 0.5 went down

Prod5 KMS 0.5 went down

Wed, 01 Apr 2026 08:42:44 +0000

Prod5 KMS 0.5 went down

Prod4 Node recovered

Wed, 01 Apr 2026 07:32:09 +0000

Prod4 Node recovered

Prod4 Node went down

Wed, 01 Apr 2026 02:12:13 +0000

Prod4 Node went down

Prod4 Gateway 0.3 went down

Wed, 01 Apr 2026 02:11:21 +0000

Prod4 Gateway 0.3 went down

Prod4 VMM 0.3 went down

Wed, 01 Apr 2026 02:11:05 +0000

Prod4 VMM 0.3 went down

[Maintenance] Phala Network Node — Intel TEE & TCB Update

Wed, 01 Apr 2026 02:00:00 -0000

We're scheduling a maintenance window to apply Intel TEE firmware updates to one of our production nodes. Window: April 1, 2026 | 02:00–08:00 UTC What's changing: Intel TDX/SGX microcode upgrade + TCB refresh Why it matters: - Attestation: Verifiers enforce strict TCB level checks. An outdated TCB causes quotes to fail UpToDate status, which triggers consensus check failures and temporary node restrictions. - Security: Patches for recent Intel side-channel vulnerabilities. - Stability: Low-level fixes for TDX resource reclamation and lifecycle management. Impact: - The affected node will restart to load new microcode. Expect up to 3 hours of downtime for your running instances. - TEE environments reinitialize automatically on boot. After the window, verify your instance status and quote generation. Please adjust your workload scheduling accordingly. If you hit initialization issues or quote failures post-update, reach out through the standard support channels. — Phala Network Operations

Prod6 KMS 0.5 recovered

Mon, 30 Mar 2026 19:39:24 +0000

Prod6 KMS 0.5 recovered

Prod6 Base Gateway 0.5 recovered

Mon, 30 Mar 2026 19:35:52 +0000

Prod6 Base Gateway 0.5 recovered

Prod6 Gateway 0.3 recovered

Mon, 30 Mar 2026 19:35:42 +0000

Prod6 Gateway 0.3 recovered

Prod6 Base KMS 0.5 recovered

Mon, 30 Mar 2026 19:35:42 +0000

Prod6 Base KMS 0.5 recovered

Prod6 Node recovered

Mon, 30 Mar 2026 19:35:38 +0000

Prod6 Node recovered

Prod6 KMS 0.5.1 recovered

Mon, 30 Mar 2026 19:35:29 +0000

Prod6 KMS 0.5.1 recovered

Prod6 Gateway 0.5.1 recovered

Mon, 30 Mar 2026 19:35:18 +0000

Prod6 Gateway 0.5.1 recovered

Prod6 Eth KMS 0.5 recovered

Mon, 30 Mar 2026 19:35:17 +0000

Prod6 Eth KMS 0.5 recovered

Prod6 VMM 0.5.1 recovered

Mon, 30 Mar 2026 19:35:16 +0000

Prod6 VMM 0.5.1 recovered

Prod6 VMM 0.5 recovered

Mon, 30 Mar 2026 19:35:14 +0000

Prod6 VMM 0.5 recovered

Prod6 Gateway 0.5 recovered

Mon, 30 Mar 2026 19:34:23 +0000

Prod6 Gateway 0.5 recovered

Prod6 VMM 0.3 recovered

Mon, 30 Mar 2026 19:34:06 +0000

Prod6 VMM 0.3 recovered

Prod6 Eth Gateway 0.5 recovered

Mon, 30 Mar 2026 19:33:53 +0000

Prod6 Eth Gateway 0.5 recovered

Prod6 Gateway 0.5.1 went down

Mon, 30 Mar 2026 19:23:07 +0000

Prod6 Gateway 0.5.1 went down

Prod6 VMM 0.3 went down

Mon, 30 Mar 2026 19:23:03 +0000

Prod6 VMM 0.3 went down

Prod6 Eth KMS 0.5 went down

Mon, 30 Mar 2026 19:23:00 +0000

Prod6 Eth KMS 0.5 went down

Prod6 VMM 0.5.1 went down

Mon, 30 Mar 2026 19:22:58 +0000

Prod6 VMM 0.5.1 went down

Prod6 Base Gateway 0.5 went down

Mon, 30 Mar 2026 19:22:50 +0000

Prod6 Base Gateway 0.5 went down

Prod6 Eth Gateway 0.5 went down

Mon, 30 Mar 2026 19:22:45 +0000

Prod6 Eth Gateway 0.5 went down

Prod6 Gateway 0.3 went down

Mon, 30 Mar 2026 19:22:39 +0000

Prod6 Gateway 0.3 went down

Prod6 Base KMS 0.5 went down

Mon, 30 Mar 2026 19:22:34 +0000

Prod6 Base KMS 0.5 went down

Prod6 Node went down

Mon, 30 Mar 2026 19:22:27 +0000

Prod6 Node went down

Prod6 KMS 0.5 went down

Mon, 30 Mar 2026 19:22:23 +0000

Prod6 KMS 0.5 went down

Prod6 VMM 0.5 went down

Mon, 30 Mar 2026 19:22:22 +0000

Prod6 VMM 0.5 went down

Prod6 KMS 0.5.1 went down

Mon, 30 Mar 2026 19:22:22 +0000

Prod6 KMS 0.5.1 went down

Prod6 Gateway 0.5 went down

Mon, 30 Mar 2026 19:22:22 +0000

Prod6 Gateway 0.5 went down

gpt-oss-20b recovered

Mon, 30 Mar 2026 17:55:18 +0000

gpt-oss-20b recovered

gemma-3-27b-it recovered

Mon, 30 Mar 2026 17:50:44 +0000

gemma-3-27b-it recovered

qwen-2.5-7b-instruct recovered

Mon, 30 Mar 2026 17:44:22 +0000

qwen-2.5-7b-instruct recovered

gemma-3-27b-it went down

Mon, 30 Mar 2026 17:37:44 +0000

gemma-3-27b-it went down

deepseek-v3.2 recovered

Mon, 30 Mar 2026 17:36:22 +0000

deepseek-v3.2 recovered

gpt-oss-20b went down

Mon, 30 Mar 2026 17:31:40 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Mon, 30 Mar 2026 17:30:19 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 went down

Mon, 30 Mar 2026 17:27:13 +0000

deepseek-v3.2 went down

Prod7 Eth KMS 0.5 recovered

Mon, 30 Mar 2026 07:10:37 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Mon, 30 Mar 2026 07:10:33 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 Base Gateway 0.5 recovered

Mon, 30 Mar 2026 07:06:10 +0000

Prod7 Base Gateway 0.5 recovered

Prod7 Base KMS 0.5 recovered

Mon, 30 Mar 2026 07:05:07 +0000

Prod7 Base KMS 0.5 recovered

Prod7 Gateway 0.5 recovered

Mon, 30 Mar 2026 06:31:48 +0000

Prod7 Gateway 0.5 recovered

Prod7 KMS 0.5 recovered

Mon, 30 Mar 2026 06:30:52 +0000

Prod7 KMS 0.5 recovered

Prod7 VMM 0.5 recovered

Mon, 30 Mar 2026 05:51:43 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 p8 recovered

Mon, 30 Mar 2026 05:51:41 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 VMM 0.3 recovered

Mon, 30 Mar 2026 05:51:29 +0000

Prod7 VMM 0.3 recovered

Prod7 Gateway 0.3 p8 recovered

Mon, 30 Mar 2026 05:50:47 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Mon, 30 Mar 2026 05:50:39 +0000

Prod7 Gateway 0.3 recovered

Prod7 Node recovered

Mon, 30 Mar 2026 05:49:43 +0000

Prod7 Node recovered

Prod7 Gateway 0.3 p8 went down

Mon, 30 Mar 2026 05:45:55 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 VMM 0.3 went down

Mon, 30 Mar 2026 05:45:52 +0000

Prod7 VMM 0.3 went down

Prod7 VMM 0.5 went down

Mon, 30 Mar 2026 05:45:52 +0000

Prod7 VMM 0.5 went down

Prod7 VMM 0.3 p8 went down

Mon, 30 Mar 2026 05:45:43 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Node went down

Mon, 30 Mar 2026 05:45:43 +0000

Prod7 Node went down

Prod7 Gateway 0.3 went down

Mon, 30 Mar 2026 05:45:42 +0000

Prod7 Gateway 0.3 went down

Prod7 Eth KMS 0.5 went down

Mon, 30 Mar 2026 05:43:23 +0000

Prod7 Eth KMS 0.5 went down

Prod7 Base Gateway 0.5 went down

Mon, 30 Mar 2026 05:43:12 +0000

Prod7 Base Gateway 0.5 went down

Prod7 Gateway 0.5 went down

Mon, 30 Mar 2026 05:43:00 +0000

Prod7 Gateway 0.5 went down

Prod7 Base KMS 0.5 went down

Mon, 30 Mar 2026 05:42:59 +0000

Prod7 Base KMS 0.5 went down

Prod7 KMS 0.5 went down

Mon, 30 Mar 2026 05:42:44 +0000

Prod7 KMS 0.5 went down

Prod7 Eth Gateway 0.5 went down

Mon, 30 Mar 2026 05:42:34 +0000

Prod7 Eth Gateway 0.5 went down

Prod7 Base Gateway 0.5 recovered

Fri, 27 Mar 2026 19:39:22 +0000

Prod7 Base Gateway 0.5 recovered

Prod7 Gateway 0.5 recovered

Fri, 27 Mar 2026 19:38:51 +0000

Prod7 Gateway 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Fri, 27 Mar 2026 19:38:34 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 Base KMS 0.5 recovered

Fri, 27 Mar 2026 19:38:00 +0000

Prod7 Base KMS 0.5 recovered

Prod7 KMS 0.5 recovered

Fri, 27 Mar 2026 19:37:46 +0000

Prod7 KMS 0.5 recovered

Prod7 Eth KMS 0.5 recovered

Fri, 27 Mar 2026 19:37:27 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 VMM 0.3 p8 recovered

Fri, 27 Mar 2026 19:32:47 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 VMM 0.5 recovered

Fri, 27 Mar 2026 19:32:32 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 recovered

Fri, 27 Mar 2026 19:32:29 +0000

Prod7 VMM 0.3 recovered

Prod7 Gateway 0.3 p8 recovered

Fri, 27 Mar 2026 19:31:54 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Fri, 27 Mar 2026 19:31:43 +0000

Prod7 Gateway 0.3 recovered

Prod7 Gateway 0.3 p8 went down

Fri, 27 Mar 2026 19:28:13 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 VMM 0.3 p8 went down

Fri, 27 Mar 2026 19:28:06 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Gateway 0.3 went down

Fri, 27 Mar 2026 19:28:06 +0000

Prod7 Gateway 0.3 went down

Prod7 VMM 0.5 went down

Fri, 27 Mar 2026 19:28:03 +0000

Prod7 VMM 0.5 went down

Prod7 VMM 0.3 went down

Fri, 27 Mar 2026 19:28:02 +0000

Prod7 VMM 0.3 went down

Prod3 VMM 0.5 recovered

Fri, 27 Mar 2026 18:54:05 +0000

Prod3 VMM 0.5 recovered

Prod9 Eth Gateway 0.5 recovered

Fri, 27 Mar 2026 18:43:35 +0000

Prod9 Eth Gateway 0.5 recovered

Prod3 Gateway 0.5 recovered

Fri, 27 Mar 2026 18:43:31 +0000

Prod3 Gateway 0.5 recovered

Prod9 Gateway 0.5 recovered

Fri, 27 Mar 2026 18:43:14 +0000

Prod9 Gateway 0.5 recovered

Prod9 Base Gateway 0.5 recovered

Fri, 27 Mar 2026 18:43:11 +0000

Prod9 Base Gateway 0.5 recovered

Prod9 Base KMS 0.5 recovered

Fri, 27 Mar 2026 18:43:03 +0000

Prod9 Base KMS 0.5 recovered

Prod9 KMS 0.5 recovered

Fri, 27 Mar 2026 18:42:53 +0000

Prod9 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Fri, 27 Mar 2026 18:42:44 +0000

Prod3 KMS 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 27 Mar 2026 18:42:26 +0000

Prod9 Eth KMS 0.5 recovered

Prod3 Node recovered

Fri, 27 Mar 2026 18:20:45 +0000

Prod3 Node recovered

Prod3 Node went down

Fri, 27 Mar 2026 18:13:41 +0000

Prod3 Node went down

Prod9 VMM 0.5 recovered

Fri, 27 Mar 2026 18:10:14 +0000

Prod9 VMM 0.5 recovered

Prod9 VMM 0.3 p8 recovered

Fri, 27 Mar 2026 18:09:39 +0000

Prod9 VMM 0.3 p8 recovered

Prod9 Node recovered

Fri, 27 Mar 2026 18:08:58 +0000

Prod9 Node recovered

Prod9 Node went down

Fri, 27 Mar 2026 18:01:23 +0000

Prod9 Node went down

Prod9 VMM 0.5 went down

Fri, 27 Mar 2026 18:01:16 +0000

Prod9 VMM 0.5 went down

Prod9 VMM 0.3 p8 went down

Fri, 27 Mar 2026 18:01:16 +0000

Prod9 VMM 0.3 p8 went down

Prod9 VMM 0.5 recovered

Fri, 27 Mar 2026 17:56:19 +0000

Prod9 VMM 0.5 recovered

Prod7 Gateway 0.3 p8 recovered

Fri, 27 Mar 2026 17:55:54 +0000

Prod7 Gateway 0.3 p8 recovered

Prod9 VMM 0.3 p8 recovered

Fri, 27 Mar 2026 17:55:46 +0000

Prod9 VMM 0.3 p8 recovered

Prod7 VMM 0.3 p8 recovered

Fri, 27 Mar 2026 17:55:43 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Fri, 27 Mar 2026 17:55:42 +0000

Prod7 Gateway 0.3 recovered

Prod7 VMM 0.5 recovered

Fri, 27 Mar 2026 17:55:37 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 recovered

Fri, 27 Mar 2026 17:55:24 +0000

Prod7 VMM 0.3 recovered

Prod9 Node recovered

Fri, 27 Mar 2026 17:55:00 +0000

Prod9 Node recovered

Prod7 Node recovered

Fri, 27 Mar 2026 17:54:41 +0000

Prod7 Node recovered

Prod3 Node recovered

Fri, 27 Mar 2026 17:54:39 +0000

Prod3 Node recovered

Prod7 Base KMS 0.5 went down

Fri, 27 Mar 2026 17:50:03 +0000

Prod7 Base KMS 0.5 went down

Prod9 Base KMS 0.5 went down

Fri, 27 Mar 2026 17:50:03 +0000

Prod9 Base KMS 0.5 went down

Prod3 VMM 0.5 went down

Fri, 27 Mar 2026 17:50:03 +0000

Prod3 VMM 0.5 went down

Prod7 Gateway 0.5 went down

Fri, 27 Mar 2026 17:50:03 +0000

Prod7 Gateway 0.5 went down

Prod9 Node went down

Fri, 27 Mar 2026 17:50:03 +0000

Prod9 Node went down

Prod9 KMS 0.5 went down

Fri, 27 Mar 2026 17:49:58 +0000

Prod9 KMS 0.5 went down

Prod9 Eth Gateway 0.5 went down

Fri, 27 Mar 2026 17:49:58 +0000

Prod9 Eth Gateway 0.5 went down

Prod7 Gateway 0.3 p8 went down

Fri, 27 Mar 2026 17:49:57 +0000

Prod7 Gateway 0.3 p8 went down

Prod9 VMM 0.3 p8 went down

Fri, 27 Mar 2026 17:49:53 +0000

Prod9 VMM 0.3 p8 went down

Prod7 KMS 0.5 went down

Fri, 27 Mar 2026 17:49:52 +0000

Prod7 KMS 0.5 went down

Prod3 Gateway 0.5 went down

Fri, 27 Mar 2026 17:49:51 +0000

Prod3 Gateway 0.5 went down

Prod7 Eth Gateway 0.5 went down

Fri, 27 Mar 2026 17:49:51 +0000

Prod7 Eth Gateway 0.5 went down

Prod3 KMS 0.5 went down

Fri, 27 Mar 2026 17:49:46 +0000

Prod3 KMS 0.5 went down

Prod9 Gateway 0.5 went down

Fri, 27 Mar 2026 17:49:46 +0000

Prod9 Gateway 0.5 went down

Prod3 Node went down

Fri, 27 Mar 2026 17:49:45 +0000

Prod3 Node went down

Prod7 Node went down

Fri, 27 Mar 2026 17:49:44 +0000

Prod7 Node went down

Prod7 VMM 0.3 p8 went down

Fri, 27 Mar 2026 17:49:43 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Base Gateway 0.5 went down

Fri, 27 Mar 2026 17:49:43 +0000

Prod7 Base Gateway 0.5 went down

Prod9 Base Gateway 0.5 went down

Fri, 27 Mar 2026 17:49:42 +0000

Prod9 Base Gateway 0.5 went down

Prod9 VMM 0.5 went down

Fri, 27 Mar 2026 17:49:42 +0000

Prod9 VMM 0.5 went down

Prod7 Gateway 0.3 went down

Fri, 27 Mar 2026 17:49:36 +0000

Prod7 Gateway 0.3 went down

Prod7 VMM 0.3 went down

Fri, 27 Mar 2026 17:49:34 +0000

Prod7 VMM 0.3 went down

Prod9 Eth KMS 0.5 went down

Fri, 27 Mar 2026 17:49:33 +0000

Prod9 Eth KMS 0.5 went down

Prod7 VMM 0.5 went down

Fri, 27 Mar 2026 17:49:33 +0000

Prod7 VMM 0.5 went down

Prod7 Eth KMS 0.5 went down

Fri, 27 Mar 2026 17:49:33 +0000

Prod7 Eth KMS 0.5 went down

Prod5 KMS 0.5 recovered

Thu, 26 Mar 2026 06:13:35 +0000

Prod5 KMS 0.5 recovered

Prod5 Node recovered

Thu, 26 Mar 2026 06:13:25 +0000

Prod5 Node recovered

Prod5 Gateway 0.3 recovered

Thu, 26 Mar 2026 06:13:22 +0000

Prod5 Gateway 0.3 recovered

Prod5 VMM 0.3 recovered

Thu, 26 Mar 2026 06:13:11 +0000

Prod5 VMM 0.3 recovered

Prod5 VMM 0.5 recovered

Thu, 26 Mar 2026 06:13:08 +0000

Prod5 VMM 0.5 recovered

Prod5 Base Gateway 0.5 recovered

Thu, 26 Mar 2026 06:12:50 +0000

Prod5 Base Gateway 0.5 recovered

Prod5 Gateway 0.5 recovered

Thu, 26 Mar 2026 06:12:42 +0000

Prod5 Gateway 0.5 recovered

Prod5 Base KMS 0.5 recovered

Thu, 26 Mar 2026 06:12:38 +0000

Prod5 Base KMS 0.5 recovered

Prod5 Gateway 0.5 went down

Thu, 26 Mar 2026 06:03:41 +0000

Prod5 Gateway 0.5 went down

Prod5 Base KMS 0.5 went down

Thu, 26 Mar 2026 06:03:37 +0000

Prod5 Base KMS 0.5 went down

Prod5 KMS 0.5 went down

Thu, 26 Mar 2026 06:03:35 +0000

Prod5 KMS 0.5 went down

Prod5 Gateway 0.3 went down

Thu, 26 Mar 2026 06:03:22 +0000

Prod5 Gateway 0.3 went down

Prod5 Node went down

Thu, 26 Mar 2026 06:03:21 +0000

Prod5 Node went down

Prod5 VMM 0.3 went down

Thu, 26 Mar 2026 06:03:14 +0000

Prod5 VMM 0.3 went down

Prod5 VMM 0.5 went down

Thu, 26 Mar 2026 06:03:10 +0000

Prod5 VMM 0.5 went down

Prod5 Base Gateway 0.5 went down

Thu, 26 Mar 2026 06:02:58 +0000

Prod5 Base Gateway 0.5 went down

Prod5 Gateway 0.5 recovered

Thu, 26 Mar 2026 06:01:45 +0000

Prod5 Gateway 0.5 recovered

Prod5 Base KMS 0.5 recovered

Thu, 26 Mar 2026 06:01:41 +0000

Prod5 Base KMS 0.5 recovered

Prod5 KMS 0.5 recovered

Thu, 26 Mar 2026 06:01:35 +0000

Prod5 KMS 0.5 recovered

Prod5 Node recovered

Thu, 26 Mar 2026 06:01:24 +0000

Prod5 Node recovered

Prod5 Gateway 0.3 recovered

Thu, 26 Mar 2026 06:01:23 +0000

Prod5 Gateway 0.3 recovered

Prod5 VMM 0.3 recovered

Thu, 26 Mar 2026 06:01:12 +0000

Prod5 VMM 0.3 recovered

Prod5 VMM 0.5 recovered

Thu, 26 Mar 2026 06:01:10 +0000

Prod5 VMM 0.5 recovered

Prod5 Base Gateway 0.5 recovered

Thu, 26 Mar 2026 06:00:58 +0000

Prod5 Base Gateway 0.5 recovered

Prod5 VMM 0.5 went down

Thu, 26 Mar 2026 05:57:02 +0000

Prod5 VMM 0.5 went down

Prod5 Gateway 0.5 went down

Thu, 26 Mar 2026 05:56:46 +0000

Prod5 Gateway 0.5 went down

Prod5 Base Gateway 0.5 went down

Thu, 26 Mar 2026 05:56:46 +0000

Prod5 Base Gateway 0.5 went down

Prod5 Base KMS 0.5 went down

Thu, 26 Mar 2026 05:56:37 +0000

Prod5 Base KMS 0.5 went down

Prod5 KMS 0.5 went down

Thu, 26 Mar 2026 05:56:34 +0000

Prod5 KMS 0.5 went down

Prod5 VMM 0.3 went down

Thu, 26 Mar 2026 05:56:26 +0000

Prod5 VMM 0.3 went down

Prod5 Gateway 0.3 went down

Thu, 26 Mar 2026 05:56:26 +0000

Prod5 Gateway 0.3 went down

Prod5 Node went down

Thu, 26 Mar 2026 05:56:21 +0000

Prod5 Node went down

Prod11 Node recovered

Tue, 24 Mar 2026 21:27:20 +0000

Prod11 Node recovered

Prod11 Node went down

Tue, 24 Mar 2026 21:19:04 +0000

Prod11 Node went down

Prod11 Node recovered

Tue, 24 Mar 2026 21:15:12 +0000

Prod11 Node recovered

Prod11 Node went down

Tue, 24 Mar 2026 20:44:11 +0000

Prod11 Node went down

Prod11 Node recovered

Tue, 24 Mar 2026 20:33:15 +0000

Prod11 Node recovered

Prod11 Node went down

Tue, 24 Mar 2026 19:59:13 +0000

Prod11 Node went down

Prod6 KMS 0.5.1 recovered

Tue, 24 Mar 2026 05:30:29 +0000

Prod6 KMS 0.5.1 recovered

Prod6 KMS 0.5.1 went down

Tue, 24 Mar 2026 05:23:16 +0000

Prod6 KMS 0.5.1 went down

Prod6 KMS 0.5.1 recovered

Mon, 23 Mar 2026 09:34:49 +0000

Prod6 KMS 0.5.1 recovered

Prod7 Eth KMS 0.5 recovered

Sun, 22 Mar 2026 06:27:29 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 Base KMS 0.5 recovered

Sun, 22 Mar 2026 06:27:07 +0000

Prod7 Base KMS 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Sun, 22 Mar 2026 06:25:39 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 Base KMS 0.5 went down

Sun, 22 Mar 2026 06:16:03 +0000

Prod7 Base KMS 0.5 went down

Prod7 Eth KMS 0.5 went down

Sun, 22 Mar 2026 05:52:19 +0000

Prod7 Eth KMS 0.5 went down

Prod7 Eth Gateway 0.5 went down

Sun, 22 Mar 2026 05:51:30 +0000

Prod7 Eth Gateway 0.5 went down

Prod7 Eth KMS 0.5 recovered

Sun, 22 Mar 2026 05:51:27 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Sun, 22 Mar 2026 05:51:16 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 Base Gateway 0.5 recovered

Sun, 22 Mar 2026 05:19:14 +0000

Prod7 Base Gateway 0.5 recovered

Prod7 Base KMS 0.5 recovered

Sun, 22 Mar 2026 05:18:57 +0000

Prod7 Base KMS 0.5 recovered

Prod7 Gateway 0.5 recovered

Sun, 22 Mar 2026 05:18:57 +0000

Prod7 Gateway 0.5 recovered

Prod7 KMS 0.5 recovered

Sun, 22 Mar 2026 05:18:46 +0000

Prod7 KMS 0.5 recovered

Prod7 VMM 0.5 recovered

Sun, 22 Mar 2026 05:18:34 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 recovered

Sun, 22 Mar 2026 05:18:22 +0000

Prod7 VMM 0.3 recovered

Prod7 Gateway 0.3 p8 recovered

Sun, 22 Mar 2026 05:17:52 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 VMM 0.3 p8 recovered

Sun, 22 Mar 2026 05:17:43 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Sun, 22 Mar 2026 05:17:39 +0000

Prod7 Gateway 0.3 recovered

Prod7 Node recovered

Sun, 22 Mar 2026 05:16:48 +0000

Prod7 Node recovered

Prod7 VMM 0.3 p8 went down

Sun, 22 Mar 2026 05:02:41 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Node went down

Sun, 22 Mar 2026 05:02:39 +0000

Prod7 Node went down

Prod7 Gateway 0.3 went down

Sun, 22 Mar 2026 05:02:32 +0000

Prod7 Gateway 0.3 went down

Prod7 Eth Gateway 0.5 went down

Sun, 22 Mar 2026 05:02:32 +0000

Prod7 Eth Gateway 0.5 went down

Prod7 VMM 0.5 went down

Sun, 22 Mar 2026 05:02:29 +0000

Prod7 VMM 0.5 went down

Prod7 VMM 0.3 went down

Sun, 22 Mar 2026 05:02:20 +0000

Prod7 VMM 0.3 went down

Prod7 Base Gateway 0.5 went down

Sun, 22 Mar 2026 05:02:08 +0000

Prod7 Base Gateway 0.5 went down

Prod7 Base KMS 0.5 went down

Sun, 22 Mar 2026 05:01:59 +0000

Prod7 Base KMS 0.5 went down

Prod7 Gateway 0.3 p8 went down

Sun, 22 Mar 2026 05:01:51 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 KMS 0.5 went down

Sun, 22 Mar 2026 05:01:41 +0000

Prod7 KMS 0.5 went down

Prod7 Gateway 0.5 went down

Sun, 22 Mar 2026 04:56:49 +0000

Prod7 Gateway 0.5 went down

Prod7 Base KMS 0.5 recovered

Sun, 22 Mar 2026 01:15:06 +0000

Prod7 Base KMS 0.5 recovered

Prod6 KMS 0.5.1 went down

Sat, 21 Mar 2026 11:41:21 +0000

Prod6 KMS 0.5.1 went down

Prod7 Base KMS 0.5 went down

Sat, 21 Mar 2026 11:41:10 +0000

Prod7 Base KMS 0.5 went down

Prod7 Eth KMS 0.5 went down

Sat, 21 Mar 2026 11:40:29 +0000

Prod7 Eth KMS 0.5 went down

Prod7 Base Gateway 0.5 recovered

Sat, 21 Mar 2026 07:34:14 +0000

Prod7 Base Gateway 0.5 recovered

Prod7 Gateway 0.5 recovered

Sat, 21 Mar 2026 07:33:57 +0000

Prod7 Gateway 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Sat, 21 Mar 2026 07:33:33 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 Base KMS 0.5 recovered

Sat, 21 Mar 2026 07:33:02 +0000

Prod7 Base KMS 0.5 recovered

Prod7 VMM 0.3 p8 recovered

Sat, 21 Mar 2026 07:32:44 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 KMS 0.5 recovered

Sat, 21 Mar 2026 07:32:43 +0000

Prod7 KMS 0.5 recovered

Prod7 VMM 0.5 recovered

Sat, 21 Mar 2026 07:32:32 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 recovered

Sat, 21 Mar 2026 07:32:32 +0000

Prod7 VMM 0.3 recovered

Prod7 Eth KMS 0.5 recovered

Sat, 21 Mar 2026 07:32:26 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 Gateway 0.3 p8 recovered

Sat, 21 Mar 2026 07:31:52 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 Node recovered

Sat, 21 Mar 2026 07:31:45 +0000

Prod7 Node recovered

Prod7 Gateway 0.3 recovered

Sat, 21 Mar 2026 07:31:43 +0000

Prod7 Gateway 0.3 recovered

Prod7 Base KMS 0.5 went down

Sat, 21 Mar 2026 07:09:05 +0000

Prod7 Base KMS 0.5 went down

Prod7 Gateway 0.5 went down

Sat, 21 Mar 2026 07:08:59 +0000

Prod7 Gateway 0.5 went down

Prod7 Gateway 0.3 p8 went down

Sat, 21 Mar 2026 07:08:46 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 Node went down

Sat, 21 Mar 2026 07:08:43 +0000

Prod7 Node went down

Prod7 KMS 0.5 went down

Sat, 21 Mar 2026 07:08:43 +0000

Prod7 KMS 0.5 went down

Prod7 VMM 0.3 p8 went down

Sat, 21 Mar 2026 07:08:42 +0000

Prod7 VMM 0.3 p8 went down

Prod7 VMM 0.5 went down

Sat, 21 Mar 2026 07:08:41 +0000

Prod7 VMM 0.5 went down

Prod7 Gateway 0.3 went down

Sat, 21 Mar 2026 07:08:41 +0000

Prod7 Gateway 0.3 went down

Prod7 Eth Gateway 0.5 went down

Sat, 21 Mar 2026 07:08:40 +0000

Prod7 Eth Gateway 0.5 went down

Prod7 VMM 0.3 went down

Sat, 21 Mar 2026 07:08:36 +0000

Prod7 VMM 0.3 went down

Prod7 Eth KMS 0.5 went down

Sat, 21 Mar 2026 07:08:23 +0000

Prod7 Eth KMS 0.5 went down

Prod7 Base Gateway 0.5 went down

Sat, 21 Mar 2026 07:08:13 +0000

Prod7 Base Gateway 0.5 went down

Prod7 Base Gateway 0.5 recovered

Sat, 21 Mar 2026 06:09:15 +0000

Prod7 Base Gateway 0.5 recovered

Prod7 Base KMS 0.5 recovered

Sat, 21 Mar 2026 06:09:04 +0000

Prod7 Base KMS 0.5 recovered

Prod7 Gateway 0.5 recovered

Sat, 21 Mar 2026 06:08:51 +0000

Prod7 Gateway 0.5 recovered

Prod7 KMS 0.5 recovered

Sat, 21 Mar 2026 06:08:45 +0000

Prod7 KMS 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Sat, 21 Mar 2026 06:08:38 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 Eth KMS 0.5 recovered

Sat, 21 Mar 2026 06:08:27 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 VMM 0.5 recovered

Sat, 21 Mar 2026 06:04:28 +0000

Prod7 VMM 0.5 recovered

Prod7 Gateway 0.3 p8 recovered

Sat, 21 Mar 2026 06:03:55 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 VMM 0.3 p8 recovered

Sat, 21 Mar 2026 06:03:41 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Sat, 21 Mar 2026 06:03:33 +0000

Prod7 Gateway 0.3 recovered

Prod7 VMM 0.3 recovered

Sat, 21 Mar 2026 06:03:26 +0000

Prod7 VMM 0.3 recovered

Prod7 Node recovered

Sat, 21 Mar 2026 06:02:40 +0000

Prod7 Node recovered

Prod7 Base Gateway 0.5 went down

Sat, 21 Mar 2026 05:55:21 +0000

Prod7 Base Gateway 0.5 went down

Prod7 Base KMS 0.5 went down

Sat, 21 Mar 2026 05:55:07 +0000

Prod7 Base KMS 0.5 went down

Prod7 VMM 0.5 went down

Sat, 21 Mar 2026 05:55:02 +0000

Prod7 VMM 0.5 went down

Prod7 Eth KMS 0.5 went down

Sat, 21 Mar 2026 05:54:57 +0000

Prod7 Eth KMS 0.5 went down

Prod7 Gateway 0.5 went down

Sat, 21 Mar 2026 05:54:55 +0000

Prod7 Gateway 0.5 went down

Prod7 Gateway 0.3 p8 went down

Sat, 21 Mar 2026 05:54:55 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 KMS 0.5 went down

Sat, 21 Mar 2026 05:54:51 +0000

Prod7 KMS 0.5 went down

Prod7 VMM 0.3 went down

Sat, 21 Mar 2026 05:54:49 +0000

Prod7 VMM 0.3 went down

Prod7 VMM 0.3 p8 went down

Sat, 21 Mar 2026 05:54:49 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Gateway 0.3 went down

Sat, 21 Mar 2026 05:54:48 +0000

Prod7 Gateway 0.3 went down

Prod7 Node went down

Sat, 21 Mar 2026 05:54:48 +0000

Prod7 Node went down

Prod7 Eth Gateway 0.5 went down

Sat, 21 Mar 2026 05:54:48 +0000

Prod7 Eth Gateway 0.5 went down

Phala PCCS recovered

Sat, 21 Mar 2026 04:03:17 +0000

Phala PCCS recovered

Phala PCCS went down

Sat, 21 Mar 2026 03:52:28 +0000

Phala PCCS went down

deepseek-v3.2 recovered

Fri, 20 Mar 2026 19:51:13 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 19:50:43 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Fri, 20 Mar 2026 19:50:25 +0000

gpt-oss-20b recovered

deepseek-v3.2 went down

Fri, 20 Mar 2026 19:41:34 +0000

deepseek-v3.2 went down

gpt-oss-20b went down

Fri, 20 Mar 2026 19:40:44 +0000

gpt-oss-20b went down

gemma-3-27b-it went down

Fri, 20 Mar 2026 19:36:04 +0000

gemma-3-27b-it went down

Prod9 KMS 0.5 recovered

Fri, 20 Mar 2026 18:20:47 +0000

Prod9 KMS 0.5 recovered

qwen-2.5-7b-instruct recovered

Fri, 20 Mar 2026 18:19:22 +0000

qwen-2.5-7b-instruct recovered

Prod7 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 18:18:51 +0000

Prod7 VMM 0.3 p8 recovered

Prod9 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 18:18:36 +0000

Prod9 Eth Gateway 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 18:18:35 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 VMM 0.3 recovered

Fri, 20 Mar 2026 18:18:35 +0000

Prod7 VMM 0.3 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 18:18:33 +0000

Prod9 Eth KMS 0.5 recovered

Prod7 VMM 0.5 recovered

Fri, 20 Mar 2026 18:18:33 +0000

Prod7 VMM 0.5 recovered

Prod7 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 18:18:28 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 18:18:13 +0000

Prod7 Base Gateway 0.5 recovered

Prod9 Gateway 0.5 recovered

Fri, 20 Mar 2026 18:18:13 +0000

Prod9 Gateway 0.5 recovered

Use1 Base KMS 0.5 recovered

Fri, 20 Mar 2026 18:18:05 +0000

Use1 Base KMS 0.5 recovered

Prod9 Base KMS 0.5 recovered

Fri, 20 Mar 2026 18:18:05 +0000

Prod9 Base KMS 0.5 recovered

Prod9 Node recovered

Fri, 20 Mar 2026 18:18:01 +0000

Prod9 Node recovered

Use1 Gateway 0.5 recovered

Fri, 20 Mar 2026 18:18:01 +0000

Use1 Gateway 0.5 recovered

Prod10 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 18:18:01 +0000

Prod10 Base Gateway 0.5 recovered

Prod7 VMM 0.3 p8 went down

Fri, 20 Mar 2026 18:15:39 +0000

Prod7 VMM 0.3 p8 went down

Prod9 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 18:15:29 +0000

Prod9 Eth Gateway 0.5 went down

Prod7 VMM 0.5 went down

Fri, 20 Mar 2026 18:15:29 +0000

Prod7 VMM 0.5 went down

Prod7 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 18:15:28 +0000

Prod7 Eth Gateway 0.5 went down

Prod9 Eth KMS 0.5 went down

Fri, 20 Mar 2026 18:15:27 +0000

Prod9 Eth KMS 0.5 went down

Prod7 Eth KMS 0.5 went down

Fri, 20 Mar 2026 18:15:25 +0000

Prod7 Eth KMS 0.5 went down

Prod7 VMM 0.3 went down

Fri, 20 Mar 2026 18:15:23 +0000

Prod7 VMM 0.3 went down

Prod9 Gateway 0.5 went down

Fri, 20 Mar 2026 18:15:12 +0000

Prod9 Gateway 0.5 went down

Prod7 Base Gateway 0.5 went down

Fri, 20 Mar 2026 18:15:12 +0000

Prod7 Base Gateway 0.5 went down

Prod9 Base KMS 0.5 went down

Fri, 20 Mar 2026 18:14:53 +0000

Prod9 Base KMS 0.5 went down

Prod9 Node went down

Fri, 20 Mar 2026 18:14:51 +0000

Prod9 Node went down

Use1 Gateway 0.5 went down

Fri, 20 Mar 2026 18:14:51 +0000

Use1 Gateway 0.5 went down

Use1 Base KMS 0.5 went down

Fri, 20 Mar 2026 18:14:51 +0000

Use1 Base KMS 0.5 went down

Prod10 Base Gateway 0.5 went down

Fri, 20 Mar 2026 18:14:49 +0000

Prod10 Base Gateway 0.5 went down

Prod9 KMS 0.5 went down

Fri, 20 Mar 2026 18:14:20 +0000

Prod9 KMS 0.5 went down

deepseek-v3.2 recovered

Fri, 20 Mar 2026 18:11:27 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 18:10:39 +0000

gemma-3-27b-it recovered

qwen-2.5-7b-instruct went down

Fri, 20 Mar 2026 18:04:44 +0000

qwen-2.5-7b-instruct went down

deepseek-v3.2 went down

Fri, 20 Mar 2026 18:01:26 +0000

deepseek-v3.2 went down

gemma-3-27b-it went down

Fri, 20 Mar 2026 18:01:13 +0000

gemma-3-27b-it went down

Prod5 Gateway 0.5 recovered

Fri, 20 Mar 2026 17:58:50 +0000

Prod5 Gateway 0.5 recovered

Prod5 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 17:58:50 +0000

Prod5 Base Gateway 0.5 recovered

Prod3 KMS 0.5 recovered

Fri, 20 Mar 2026 17:58:48 +0000

Prod3 KMS 0.5 recovered

Prod3 KMS 0.5 went down

Fri, 20 Mar 2026 17:55:44 +0000

Prod3 KMS 0.5 went down

Prod5 Gateway 0.5 went down

Fri, 20 Mar 2026 17:55:43 +0000

Prod5 Gateway 0.5 went down

Prod5 Base Gateway 0.5 went down

Fri, 20 Mar 2026 17:55:41 +0000

Prod5 Base Gateway 0.5 went down

deepseek-v3.2 recovered

Fri, 20 Mar 2026 17:46:26 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 17:31:01 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Fri, 20 Mar 2026 17:30:27 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Fri, 20 Mar 2026 17:29:32 +0000

qwen-2.5-7b-instruct recovered

deepseek-v3.2 went down

Fri, 20 Mar 2026 17:11:20 +0000

deepseek-v3.2 went down

qwen-2.5-7b-instruct went down

Fri, 20 Mar 2026 17:04:52 +0000

qwen-2.5-7b-instruct went down

Prod6 Base KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:44 +0000

Prod6 Base KMS 0.5 recovered

Prod10 Base KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:43 +0000

Prod10 Base KMS 0.5 recovered

Prod5 Base KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:39 +0000

Prod5 Base KMS 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:35 +0000

Prod9 Eth KMS 0.5 recovered

Prod9 Base KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:35 +0000

Prod9 Base KMS 0.5 recovered

Prod6 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:12 +0000

Prod6 Eth KMS 0.5 recovered

Use1 Base KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:12 +0000

Use1 Base KMS 0.5 recovered

Prod10 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 17:03:11 +0000

Prod10 Eth KMS 0.5 recovered

gemma-3-27b-it went down

Fri, 20 Mar 2026 17:01:05 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Fri, 20 Mar 2026 17:00:49 +0000

gpt-oss-20b went down

Prod9 Eth KMS 0.5 went down

Fri, 20 Mar 2026 16:59:41 +0000

Prod9 Eth KMS 0.5 went down

Prod6 Eth KMS 0.5 went down

Fri, 20 Mar 2026 16:59:28 +0000

Prod6 Eth KMS 0.5 went down

Prod9 Base KMS 0.5 went down

Fri, 20 Mar 2026 16:59:26 +0000

Prod9 Base KMS 0.5 went down

Use1 Base KMS 0.5 went down

Fri, 20 Mar 2026 16:59:25 +0000

Use1 Base KMS 0.5 went down

Prod10 Eth KMS 0.5 went down

Fri, 20 Mar 2026 16:59:24 +0000

Prod10 Eth KMS 0.5 went down

Prod6 Base KMS 0.5 went down

Fri, 20 Mar 2026 16:59:08 +0000

Prod6 Base KMS 0.5 went down

Prod10 Base KMS 0.5 went down

Fri, 20 Mar 2026 16:59:03 +0000

Prod10 Base KMS 0.5 went down

Prod5 Base KMS 0.5 went down

Fri, 20 Mar 2026 16:58:49 +0000

Prod5 Base KMS 0.5 went down

gpt-oss-20b recovered

Fri, 20 Mar 2026 16:55:24 +0000

gpt-oss-20b recovered

deepseek-v3.2 recovered

Fri, 20 Mar 2026 16:41:28 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 16:40:42 +0000

gemma-3-27b-it recovered

qwen-2.5-7b-instruct recovered

Fri, 20 Mar 2026 16:39:28 +0000

qwen-2.5-7b-instruct recovered

Prod9 Base KMS 0.5 recovered

Fri, 20 Mar 2026 16:25:55 +0000

Prod9 Base KMS 0.5 recovered

Prod7 KMS 0.5 recovered

Fri, 20 Mar 2026 16:25:47 +0000

Prod7 KMS 0.5 recovered

Prod6 Base KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:48 +0000

Prod6 Base KMS 0.5 recovered

Prod10 Base KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:47 +0000

Prod10 Base KMS 0.5 recovered

Prod5 Base KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:45 +0000

Prod5 Base KMS 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 16:23:39 +0000

Prod7 Eth Gateway 0.5 recovered

Prod3 Gateway 0.5 recovered

Fri, 20 Mar 2026 16:23:39 +0000

Prod3 Gateway 0.5 recovered

Prod5 KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:38 +0000

Prod5 KMS 0.5 recovered

Prod7 VMM 0.5 recovered

Fri, 20 Mar 2026 16:23:34 +0000

Prod7 VMM 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:29 +0000

Prod9 Eth KMS 0.5 recovered

Prod4 Gateway 0.3 recovered

Fri, 20 Mar 2026 16:23:26 +0000

Prod4 Gateway 0.3 recovered

Prod7 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:25 +0000

Prod7 Eth KMS 0.5 recovered

Prod5 Node recovered

Fri, 20 Mar 2026 16:23:23 +0000

Prod5 Node recovered

Proof t16z recovered

Fri, 20 Mar 2026 16:23:23 +0000

Proof t16z recovered

Prod6 KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:21 +0000

Prod6 KMS 0.5 recovered

Prod7 VMM 0.3 recovered

Fri, 20 Mar 2026 16:23:20 +0000

Prod7 VMM 0.3 recovered

Prod5 Gateway 0.3 recovered

Fri, 20 Mar 2026 16:23:19 +0000

Prod5 Gateway 0.3 recovered

Phala PCCS recovered

Fri, 20 Mar 2026 16:23:18 +0000

Phala PCCS recovered

Prod9 VMM 0.5 recovered

Fri, 20 Mar 2026 16:23:16 +0000

Prod9 VMM 0.5 recovered

Prod10 KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:16 +0000

Prod10 KMS 0.5 recovered

Prod6 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:15 +0000

Prod6 Eth KMS 0.5 recovered

Prod5 VMM 0.3 recovered

Fri, 20 Mar 2026 16:23:14 +0000

Prod5 VMM 0.3 recovered

Phala Cloud API recovered

Fri, 20 Mar 2026 16:23:13 +0000

Phala Cloud API recovered

Prod9 Gateway 0.5 recovered

Fri, 20 Mar 2026 16:23:13 +0000

Prod9 Gateway 0.5 recovered

Prod4 Node recovered

Fri, 20 Mar 2026 16:23:13 +0000

Prod4 Node recovered

Use1 Base KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:12 +0000

Use1 Base KMS 0.5 recovered

Prod10 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:12 +0000

Prod10 Eth KMS 0.5 recovered

Prod7 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 16:23:12 +0000

Prod7 Base Gateway 0.5 recovered

Use2 KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:11 +0000

Use2 KMS 0.5 recovered

Prod5 VMM 0.5 recovered

Fri, 20 Mar 2026 16:23:08 +0000

Prod5 VMM 0.5 recovered

Prod3 VMM 0.5 recovered

Fri, 20 Mar 2026 16:23:08 +0000

Prod3 VMM 0.5 recovered

Prod9 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 16:23:08 +0000

Prod9 Base Gateway 0.5 recovered

Prod4 VMM 0.3 recovered

Fri, 20 Mar 2026 16:23:08 +0000

Prod4 VMM 0.3 recovered

Prod9 Node recovered

Fri, 20 Mar 2026 16:23:07 +0000

Prod9 Node recovered

Prod7 Base KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:06 +0000

Prod7 Base KMS 0.5 recovered

Prod9 KMS 0.5 recovered

Fri, 20 Mar 2026 16:23:01 +0000

Prod9 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Fri, 20 Mar 2026 16:22:56 +0000

Prod3 KMS 0.5 recovered

Prod7 Gateway 0.5 recovered

Fri, 20 Mar 2026 16:22:55 +0000

Prod7 Gateway 0.5 recovered

Prod7 Gateway 0.3 p8 recovered

Fri, 20 Mar 2026 16:22:55 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Fri, 20 Mar 2026 16:22:53 +0000

Prod7 Gateway 0.3 recovered

Prod3 Node recovered

Fri, 20 Mar 2026 16:22:52 +0000

Prod3 Node recovered

Prod5 Gateway 0.5 recovered

Fri, 20 Mar 2026 16:22:52 +0000

Prod5 Gateway 0.5 recovered

Prod7 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 16:22:52 +0000

Prod7 VMM 0.3 p8 recovered

Use1 KMS 0.5 recovered

Fri, 20 Mar 2026 16:22:52 +0000

Use1 KMS 0.5 recovered

Prod9 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 16:22:52 +0000

Prod9 VMM 0.3 p8 recovered

Prod5 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 16:22:50 +0000

Prod5 Base Gateway 0.5 recovered

Prod9 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 16:22:49 +0000

Prod9 Eth Gateway 0.5 recovered

Prod7 Node recovered

Fri, 20 Mar 2026 16:22:48 +0000

Prod7 Node recovered

Use3 KMS 0.5 recovered

Fri, 20 Mar 2026 16:22:48 +0000

Use3 KMS 0.5 recovered

Phala PCCS went down

Fri, 20 Mar 2026 15:45:36 +0000

Phala PCCS went down

deepseek-v3.2 went down

Fri, 20 Mar 2026 15:43:06 +0000

deepseek-v3.2 went down

gemma-3-27b-it went down

Fri, 20 Mar 2026 15:42:36 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Fri, 20 Mar 2026 15:42:19 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Fri, 20 Mar 2026 15:41:22 +0000

qwen-2.5-7b-instruct went down

Prod9 Gateway 0.5 went down

Fri, 20 Mar 2026 15:40:16 +0000

Prod9 Gateway 0.5 went down

Prod7 Base Gateway 0.5 went down

Fri, 20 Mar 2026 15:40:16 +0000

Prod7 Base Gateway 0.5 went down

Prod4 Node went down

Fri, 20 Mar 2026 15:40:15 +0000

Prod4 Node went down

Prod5 VMM 0.3 went down

Fri, 20 Mar 2026 15:40:14 +0000

Prod5 VMM 0.3 went down

Prod9 VMM 0.5 went down

Fri, 20 Mar 2026 15:40:14 +0000

Prod9 VMM 0.5 went down

Prod9 Base Gateway 0.5 went down

Fri, 20 Mar 2026 15:40:13 +0000

Prod9 Base Gateway 0.5 went down

Prod9 Node went down

Fri, 20 Mar 2026 15:40:05 +0000

Prod9 Node went down

Prod4 VMM 0.3 went down

Fri, 20 Mar 2026 15:40:02 +0000

Prod4 VMM 0.3 went down

Prod3 VMM 0.5 went down

Fri, 20 Mar 2026 15:39:59 +0000

Prod3 VMM 0.5 went down

Prod7 Base KMS 0.5 went down

Fri, 20 Mar 2026 15:39:59 +0000

Prod7 Base KMS 0.5 went down

Prod9 Base KMS 0.5 went down

Fri, 20 Mar 2026 15:39:59 +0000

Prod9 Base KMS 0.5 went down

Prod5 VMM 0.5 went down

Fri, 20 Mar 2026 15:39:58 +0000

Prod5 VMM 0.5 went down

Prod10 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:55 +0000

Prod10 KMS 0.5 went down

Phala Cloud API went down

Fri, 20 Mar 2026 15:39:54 +0000

Phala Cloud API went down

Use2 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:46 +0000

Use2 KMS 0.5 went down

Prod7 Gateway 0.5 went down

Fri, 20 Mar 2026 15:39:41 +0000

Prod7 Gateway 0.5 went down

Prod7 Gateway 0.3 p8 went down

Fri, 20 Mar 2026 15:39:41 +0000

Prod7 Gateway 0.3 p8 went down

Prod5 Base Gateway 0.5 went down

Fri, 20 Mar 2026 15:39:41 +0000

Prod5 Base Gateway 0.5 went down

Prod9 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:41 +0000

Prod9 KMS 0.5 went down

Prod3 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:41 +0000

Prod3 KMS 0.5 went down

Prod3 Node went down

Fri, 20 Mar 2026 15:39:38 +0000

Prod3 Node went down

Prod9 VMM 0.3 p8 went down

Fri, 20 Mar 2026 15:39:37 +0000

Prod9 VMM 0.3 p8 went down

Prod7 VMM 0.3 p8 went down

Fri, 20 Mar 2026 15:39:37 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Node went down

Fri, 20 Mar 2026 15:39:35 +0000

Prod7 Node went down

Prod5 Gateway 0.5 went down

Fri, 20 Mar 2026 15:39:33 +0000

Prod5 Gateway 0.5 went down

Prod7 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:33 +0000

Prod7 KMS 0.5 went down

Prod7 Gateway 0.3 went down

Fri, 20 Mar 2026 15:39:33 +0000

Prod7 Gateway 0.3 went down

Prod5 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:33 +0000

Prod5 KMS 0.5 went down

Prod7 VMM 0.5 went down

Fri, 20 Mar 2026 15:39:30 +0000

Prod7 VMM 0.5 went down

Prod6 Eth KMS 0.5 went down

Fri, 20 Mar 2026 15:39:29 +0000

Prod6 Eth KMS 0.5 went down

Prod9 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 15:39:27 +0000

Prod9 Eth Gateway 0.5 went down

Prod9 Eth KMS 0.5 went down

Fri, 20 Mar 2026 15:39:27 +0000

Prod9 Eth KMS 0.5 went down

Prod5 Base KMS 0.5 went down

Fri, 20 Mar 2026 15:39:26 +0000

Prod5 Base KMS 0.5 went down

Prod7 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 15:39:25 +0000

Prod7 Eth Gateway 0.5 went down

Prod7 Eth KMS 0.5 went down

Fri, 20 Mar 2026 15:39:25 +0000

Prod7 Eth KMS 0.5 went down

Use1 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:24 +0000

Use1 KMS 0.5 went down

Use1 Base KMS 0.5 went down

Fri, 20 Mar 2026 15:39:24 +0000

Use1 Base KMS 0.5 went down

Prod5 Node went down

Fri, 20 Mar 2026 15:39:24 +0000

Prod5 Node went down

Prod3 Gateway 0.5 went down

Fri, 20 Mar 2026 15:39:23 +0000

Prod3 Gateway 0.5 went down

Prod7 VMM 0.3 went down

Fri, 20 Mar 2026 15:39:23 +0000

Prod7 VMM 0.3 went down

Use3 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:23 +0000

Use3 KMS 0.5 went down

Prod4 Gateway 0.3 went down

Fri, 20 Mar 2026 15:39:23 +0000

Prod4 Gateway 0.3 went down

Prod10 Eth KMS 0.5 went down

Fri, 20 Mar 2026 15:39:23 +0000

Prod10 Eth KMS 0.5 went down

Prod5 Gateway 0.3 went down

Fri, 20 Mar 2026 15:39:19 +0000

Prod5 Gateway 0.3 went down

Proof t16z went down

Fri, 20 Mar 2026 15:39:06 +0000

Proof t16z went down

Prod6 KMS 0.5 went down

Fri, 20 Mar 2026 15:39:05 +0000

Prod6 KMS 0.5 went down

Prod10 Base KMS 0.5 went down

Fri, 20 Mar 2026 15:39:02 +0000

Prod10 Base KMS 0.5 went down

Prod6 Base KMS 0.5 went down

Fri, 20 Mar 2026 15:39:01 +0000

Prod6 Base KMS 0.5 went down

qwen-2.5-7b-instruct recovered

Fri, 20 Mar 2026 15:34:24 +0000

qwen-2.5-7b-instruct recovered

deepseek-v3.2 recovered

Fri, 20 Mar 2026 15:31:17 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 15:30:57 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Fri, 20 Mar 2026 15:30:30 +0000

gpt-oss-20b recovered

Prod9 KMS 0.5 recovered

Fri, 20 Mar 2026 15:22:57 +0000

Prod9 KMS 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 15:21:48 +0000

Prod9 Eth KMS 0.5 recovered

Use1 Base KMS 0.5 recovered

Fri, 20 Mar 2026 15:21:19 +0000

Use1 Base KMS 0.5 recovered

Prod10 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 15:21:16 +0000

Prod10 Eth KMS 0.5 recovered

Prod6 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 15:21:12 +0000

Prod6 Eth KMS 0.5 recovered

Prod9 Base KMS 0.5 recovered

Fri, 20 Mar 2026 15:21:09 +0000

Prod9 Base KMS 0.5 recovered

Prod6 Base KMS 0.5 recovered

Fri, 20 Mar 2026 15:20:49 +0000

Prod6 Base KMS 0.5 recovered

Prod10 Base KMS 0.5 recovered

Fri, 20 Mar 2026 15:20:49 +0000

Prod10 Base KMS 0.5 recovered

Prod5 Base KMS 0.5 recovered

Fri, 20 Mar 2026 15:20:45 +0000

Prod5 Base KMS 0.5 recovered

Prod9 KMS 0.5 went down

Fri, 20 Mar 2026 15:19:11 +0000

Prod9 KMS 0.5 went down

deepseek-v3.2 went down

Fri, 20 Mar 2026 15:01:26 +0000

deepseek-v3.2 went down

gemma-3-27b-it went down

Fri, 20 Mar 2026 15:01:10 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Fri, 20 Mar 2026 15:00:47 +0000

gpt-oss-20b went down

qwen-2.5-7b-instruct went down

Fri, 20 Mar 2026 14:59:44 +0000

qwen-2.5-7b-instruct went down

gpt-oss-20b recovered

Fri, 20 Mar 2026 14:55:23 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Fri, 20 Mar 2026 14:54:20 +0000

qwen-2.5-7b-instruct recovered

deepseek-v3.2 recovered

Fri, 20 Mar 2026 14:51:15 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 14:50:43 +0000

gemma-3-27b-it recovered

gemma-3-27b-it went down

Fri, 20 Mar 2026 14:31:00 +0000

gemma-3-27b-it went down

deepseek-v3.2 went down

Fri, 20 Mar 2026 14:21:37 +0000

deepseek-v3.2 went down

gpt-oss-20b went down

Fri, 20 Mar 2026 14:20:43 +0000

gpt-oss-20b went down

deepseek-v3.2 recovered

Fri, 20 Mar 2026 14:11:18 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 14:10:39 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Fri, 20 Mar 2026 14:10:27 +0000

gpt-oss-20b recovered

Prod9 KMS 0.5 recovered

Fri, 20 Mar 2026 13:57:57 +0000

Prod9 KMS 0.5 recovered

Proof t16z recovered

Fri, 20 Mar 2026 13:57:31 +0000

Proof t16z recovered

Prod6 KMS 0.5 recovered

Fri, 20 Mar 2026 13:57:25 +0000

Prod6 KMS 0.5 recovered

Phala Cloud API recovered

Fri, 20 Mar 2026 13:57:18 +0000

Phala Cloud API recovered

Prod10 KMS 0.5 recovered

Fri, 20 Mar 2026 13:57:17 +0000

Prod10 KMS 0.5 recovered

Use2 KMS 0.5 recovered

Fri, 20 Mar 2026 13:57:09 +0000

Use2 KMS 0.5 recovered

Prod7 KMS 0.5 recovered

Fri, 20 Mar 2026 13:56:45 +0000

Prod7 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Fri, 20 Mar 2026 13:56:44 +0000

Use3 KMS 0.5 recovered

Use1 KMS 0.5 recovered

Fri, 20 Mar 2026 13:56:41 +0000

Use1 KMS 0.5 recovered

Prod5 KMS 0.5 recovered

Fri, 20 Mar 2026 13:56:41 +0000

Prod5 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Fri, 20 Mar 2026 13:56:41 +0000

Prod3 KMS 0.5 recovered

Prod3 Node recovered

Fri, 20 Mar 2026 13:50:44 +0000

Prod3 Node recovered

Prod9 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 13:50:42 +0000

Prod9 VMM 0.3 p8 recovered

Prod9 VMM 0.5 recovered

Fri, 20 Mar 2026 13:49:39 +0000

Prod9 VMM 0.5 recovered

Prod7 Base KMS 0.5 recovered

Fri, 20 Mar 2026 13:49:08 +0000

Prod7 Base KMS 0.5 recovered

Prod7 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 13:49:08 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Node recovered

Fri, 20 Mar 2026 13:48:48 +0000

Prod7 Node recovered

Prod9 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 13:48:34 +0000

Prod9 Eth Gateway 0.5 recovered

Prod5 Node recovered

Fri, 20 Mar 2026 13:47:16 +0000

Prod5 Node recovered

Prod7 Gateway 0.3 p8 recovered

Fri, 20 Mar 2026 13:46:56 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 13:46:32 +0000

Prod7 Eth Gateway 0.5 recovered

Prod3 VMM 0.5 recovered

Fri, 20 Mar 2026 13:46:21 +0000

Prod3 VMM 0.5 recovered

Prod5 VMM 0.5 recovered

Fri, 20 Mar 2026 13:46:03 +0000

Prod5 VMM 0.5 recovered

Prod7 VMM 0.5 recovered

Fri, 20 Mar 2026 13:45:36 +0000

Prod7 VMM 0.5 recovered

Prod7 Gateway 0.5 recovered

Fri, 20 Mar 2026 13:44:57 +0000

Prod7 Gateway 0.5 recovered

Prod4 VMM 0.3 recovered

Fri, 20 Mar 2026 13:43:58 +0000

Prod4 VMM 0.3 recovered

Prod5 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 13:43:44 +0000

Prod5 Base Gateway 0.5 recovered

Prod7 Gateway 0.3 recovered

Fri, 20 Mar 2026 13:43:43 +0000

Prod7 Gateway 0.3 recovered

Prod9 Gateway 0.5 recovered

Fri, 20 Mar 2026 13:43:40 +0000

Prod9 Gateway 0.5 recovered

Prod7 VMM 0.3 recovered

Fri, 20 Mar 2026 13:43:36 +0000

Prod7 VMM 0.3 recovered

Prod4 Gateway 0.3 recovered

Fri, 20 Mar 2026 13:43:23 +0000

Prod4 Gateway 0.3 recovered

Prod7 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 13:43:16 +0000

Prod7 Base Gateway 0.5 recovered

Prod9 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 13:42:12 +0000

Prod9 Base Gateway 0.5 recovered

Prod3 Gateway 0.5 recovered

Fri, 20 Mar 2026 13:41:40 +0000

Prod3 Gateway 0.5 recovered

Prod5 Gateway 0.3 recovered

Fri, 20 Mar 2026 13:41:22 +0000

Prod5 Gateway 0.3 recovered

Prod9 Node recovered

Fri, 20 Mar 2026 13:41:21 +0000

Prod9 Node recovered

Prod4 Node recovered

Fri, 20 Mar 2026 13:41:16 +0000

Prod4 Node recovered

Prod5 VMM 0.3 recovered

Fri, 20 Mar 2026 13:41:12 +0000

Prod5 VMM 0.3 recovered

Prod7 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 13:40:57 +0000

Prod7 Eth KMS 0.5 recovered

Prod5 Gateway 0.5 recovered

Fri, 20 Mar 2026 13:40:50 +0000

Prod5 Gateway 0.5 recovered

Phala PCCS recovered

Fri, 20 Mar 2026 13:36:21 +0000

Phala PCCS recovered

Phala PCCS went down

Fri, 20 Mar 2026 12:53:43 +0000

Phala PCCS went down

Networking unavailable

Fri, 20 Mar 2026 12:44:00 -0000

We are currently experiencing networking issues affecting some nodes in US-West. Our team is actively investigating, and we will share updates as soon as possible. Thank you for your patience.

Prod10 KMS 0.5 went down

Fri, 20 Mar 2026 12:40:51 +0000

Prod10 KMS 0.5 went down

Proof t16z went down

Fri, 20 Mar 2026 12:40:39 +0000

Proof t16z went down

Use2 KMS 0.5 went down

Fri, 20 Mar 2026 12:40:21 +0000

Use2 KMS 0.5 went down

Use1 KMS 0.5 went down

Fri, 20 Mar 2026 12:40:19 +0000

Use1 KMS 0.5 went down

Use3 KMS 0.5 went down

Fri, 20 Mar 2026 12:39:56 +0000

Use3 KMS 0.5 went down

Use1 KMS 0.5 recovered

Fri, 20 Mar 2026 12:32:57 +0000

Use1 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Fri, 20 Mar 2026 12:31:43 +0000

Use3 KMS 0.5 recovered

Phala Cloud API went down

Fri, 20 Mar 2026 12:29:56 +0000

Phala Cloud API went down

Use1 KMS 0.5 went down

Fri, 20 Mar 2026 12:28:07 +0000

Use1 KMS 0.5 went down

Phala Cloud API recovered

Fri, 20 Mar 2026 12:25:22 +0000

Phala Cloud API recovered

Prod10 KMS 0.5 recovered

Fri, 20 Mar 2026 12:25:12 +0000

Prod10 KMS 0.5 recovered

Use1 KMS 0.5 recovered

Fri, 20 Mar 2026 12:22:08 +0000

Use1 KMS 0.5 recovered

Proof t16z recovered

Fri, 20 Mar 2026 12:19:33 +0000

Proof t16z recovered

Phala PCCS recovered

Fri, 20 Mar 2026 12:16:24 +0000

Phala PCCS recovered

Use2 KMS 0.5 recovered

Fri, 20 Mar 2026 12:14:24 +0000

Use2 KMS 0.5 recovered

deepseek-v3.2 went down

Fri, 20 Mar 2026 11:58:08 +0000

deepseek-v3.2 went down

Phala PCCS went down

Fri, 20 Mar 2026 11:51:57 +0000

Phala PCCS went down

deepseek-v3.2 recovered

Fri, 20 Mar 2026 11:51:14 +0000

deepseek-v3.2 recovered

Prod5 VMM 0.5 went down

Fri, 20 Mar 2026 11:26:23 +0000

Prod5 VMM 0.5 went down

Prod5 Gateway 0.5 went down

Fri, 20 Mar 2026 11:20:07 +0000

Prod5 Gateway 0.5 went down

Prod5 Gateway 0.3 went down

Fri, 20 Mar 2026 11:09:48 +0000

Prod5 Gateway 0.3 went down

Prod5 Node went down

Fri, 20 Mar 2026 11:09:38 +0000

Prod5 Node went down

Prod4 Gateway 0.3 went down

Fri, 20 Mar 2026 11:09:37 +0000

Prod4 Gateway 0.3 went down

Use2 KMS 0.5 went down

Fri, 20 Mar 2026 11:03:54 +0000

Use2 KMS 0.5 went down

Prod5 VMM 0.3 went down

Fri, 20 Mar 2026 11:02:33 +0000

Prod5 VMM 0.3 went down

Prod4 VMM 0.3 went down

Fri, 20 Mar 2026 10:56:25 +0000

Prod4 VMM 0.3 went down

Prod5 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:55:53 +0000

Prod5 Base Gateway 0.5 went down

Prod4 Node went down

Fri, 20 Mar 2026 10:53:54 +0000

Prod4 Node went down

Prod7 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:52:56 +0000

Prod7 Base Gateway 0.5 went down

deepseek-v3.2 went down

Fri, 20 Mar 2026 10:51:49 +0000

deepseek-v3.2 went down

Prod6 KMS 0.5 went down

Fri, 20 Mar 2026 10:51:26 +0000

Prod6 KMS 0.5 went down

gemma-3-27b-it went down

Fri, 20 Mar 2026 10:51:17 +0000

gemma-3-27b-it went down

gpt-oss-20b went down

Fri, 20 Mar 2026 10:50:49 +0000

gpt-oss-20b went down

Prod3 Gateway 0.5 went down

Fri, 20 Mar 2026 10:50:39 +0000

Prod3 Gateway 0.5 went down

Prod9 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 10:50:39 +0000

Prod9 Eth Gateway 0.5 went down

Use3 KMS 0.5 went down

Fri, 20 Mar 2026 10:50:38 +0000

Use3 KMS 0.5 went down

Prod7 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 10:50:26 +0000

Prod7 Eth Gateway 0.5 went down

Prod9 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:50:20 +0000

Prod9 Eth KMS 0.5 went down

Prod7 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:50:19 +0000

Prod7 Eth KMS 0.5 went down

Prod7 VMM 0.3 went down

Fri, 20 Mar 2026 10:50:18 +0000

Prod7 VMM 0.3 went down

Prod9 VMM 0.5 went down

Fri, 20 Mar 2026 10:50:14 +0000

Prod9 VMM 0.5 went down

Prod9 Gateway 0.5 went down

Fri, 20 Mar 2026 10:50:14 +0000

Prod9 Gateway 0.5 went down

Proof t16z went down

Fri, 20 Mar 2026 10:50:13 +0000

Proof t16z went down

Prod9 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:50:12 +0000

Prod9 Base Gateway 0.5 went down

Phala Cloud API went down

Fri, 20 Mar 2026 10:50:05 +0000

Phala Cloud API went down

Prod10 KMS 0.5 went down

Fri, 20 Mar 2026 10:50:04 +0000

Prod10 KMS 0.5 went down

Prod3 VMM 0.5 went down

Fri, 20 Mar 2026 10:50:00 +0000

Prod3 VMM 0.5 went down

Prod9 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:49:57 +0000

Prod9 Base KMS 0.5 went down

Prod7 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:49:57 +0000

Prod7 Base KMS 0.5 went down

Prod9 Node went down

Fri, 20 Mar 2026 10:49:56 +0000

Prod9 Node went down

Prod9 KMS 0.5 went down

Fri, 20 Mar 2026 10:49:49 +0000

Prod9 KMS 0.5 went down

Prod7 Gateway 0.5 went down

Fri, 20 Mar 2026 10:49:49 +0000

Prod7 Gateway 0.5 went down

Prod3 KMS 0.5 went down

Fri, 20 Mar 2026 10:49:46 +0000

Prod3 KMS 0.5 went down

Prod7 VMM 0.3 p8 went down

Fri, 20 Mar 2026 10:49:45 +0000

Prod7 VMM 0.3 p8 went down

Prod3 Node went down

Fri, 20 Mar 2026 10:49:43 +0000

Prod3 Node went down

Prod9 VMM 0.3 p8 went down

Fri, 20 Mar 2026 10:49:43 +0000

Prod9 VMM 0.3 p8 went down

Prod7 KMS 0.5 went down

Fri, 20 Mar 2026 10:49:42 +0000

Prod7 KMS 0.5 went down

Prod7 Node went down

Fri, 20 Mar 2026 10:49:41 +0000

Prod7 Node went down

Use1 KMS 0.5 went down

Fri, 20 Mar 2026 10:49:38 +0000

Use1 KMS 0.5 went down

Prod5 KMS 0.5 went down

Fri, 20 Mar 2026 10:49:34 +0000

Prod5 KMS 0.5 went down

Prod7 Gateway 0.3 went down

Fri, 20 Mar 2026 10:49:31 +0000

Prod7 Gateway 0.3 went down

Prod7 Gateway 0.3 p8 went down

Fri, 20 Mar 2026 10:49:31 +0000

Prod7 Gateway 0.3 p8 went down

Prod6 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:49:26 +0000

Prod6 Eth KMS 0.5 went down

Use1 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:49:24 +0000

Use1 Base KMS 0.5 went down

Prod10 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:49:23 +0000

Prod10 Eth KMS 0.5 went down

Prod5 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:49:14 +0000

Prod5 Base KMS 0.5 went down

Prod6 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:49:06 +0000

Prod6 Base KMS 0.5 went down

Prod7 VMM 0.5 went down

Fri, 20 Mar 2026 10:49:05 +0000

Prod7 VMM 0.5 went down

Prod10 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:49:02 +0000

Prod10 Base KMS 0.5 went down

Prod7 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 10:45:56 +0000

Prod7 VMM 0.3 p8 recovered

Prod6 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:40 +0000

Prod6 Base KMS 0.5 recovered

Prod10 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:40 +0000

Prod10 Base KMS 0.5 recovered

Prod7 KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:38 +0000

Prod7 KMS 0.5 recovered

Prod7 VMM 0.5 recovered

Fri, 20 Mar 2026 10:45:33 +0000

Prod7 VMM 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 10:45:32 +0000

Prod7 Eth Gateway 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:26 +0000

Prod9 Eth KMS 0.5 recovered

Proof t16z recovered

Fri, 20 Mar 2026 10:45:26 +0000

Proof t16z recovered

Prod4 Gateway 0.3 recovered

Fri, 20 Mar 2026 10:45:25 +0000

Prod4 Gateway 0.3 recovered

Prod7 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:24 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 VMM 0.3 recovered

Fri, 20 Mar 2026 10:45:21 +0000

Prod7 VMM 0.3 recovered

Prod9 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:45:19 +0000

Prod9 Gateway 0.5 recovered

Prod9 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 10:45:15 +0000

Prod9 Base Gateway 0.5 recovered

Prod7 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 10:45:14 +0000

Prod7 Base Gateway 0.5 recovered

Prod6 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:11 +0000

Prod6 Eth KMS 0.5 recovered

Prod10 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:08 +0000

Prod10 Eth KMS 0.5 recovered

Use2 KMS 0.5 recovered

Fri, 20 Mar 2026 10:45:08 +0000

Use2 KMS 0.5 recovered

Prod7 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:45:08 +0000

Prod7 Gateway 0.5 recovered

Prod9 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 10:44:56 +0000

Prod9 Eth Gateway 0.5 recovered

Prod7 Gateway 0.3 recovered

Fri, 20 Mar 2026 10:44:56 +0000

Prod7 Gateway 0.3 recovered

Prod9 KMS 0.5 recovered

Fri, 20 Mar 2026 10:44:55 +0000

Prod9 KMS 0.5 recovered

Prod5 KMS 0.5 recovered

Fri, 20 Mar 2026 10:44:53 +0000

Prod5 KMS 0.5 recovered

Prod7 Node recovered

Fri, 20 Mar 2026 10:44:52 +0000

Prod7 Node recovered

Prod3 Node recovered

Fri, 20 Mar 2026 10:44:51 +0000

Prod3 Node recovered

Use3 KMS 0.5 recovered

Fri, 20 Mar 2026 10:44:51 +0000

Use3 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Fri, 20 Mar 2026 10:44:51 +0000

Prod3 KMS 0.5 recovered

Use1 KMS 0.5 recovered

Fri, 20 Mar 2026 10:44:51 +0000

Use1 KMS 0.5 recovered

Prod3 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:44:51 +0000

Prod3 Gateway 0.5 recovered

qwen-2.5-7b-instruct went down

Fri, 20 Mar 2026 10:44:39 +0000

qwen-2.5-7b-instruct went down

Prod5 Node recovered

Fri, 20 Mar 2026 10:43:46 +0000

Prod5 Node recovered

Prod6 KMS 0.5 recovered

Fri, 20 Mar 2026 10:43:35 +0000

Prod6 KMS 0.5 recovered

Prod9 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 10:40:33 +0000

Prod9 Eth Gateway 0.5 went down

Prod7 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 10:40:30 +0000

Prod7 Eth Gateway 0.5 went down

Prod3 Gateway 0.5 went down

Fri, 20 Mar 2026 10:40:28 +0000

Prod3 Gateway 0.5 went down

Prod7 Gateway 0.3 went down

Fri, 20 Mar 2026 10:40:28 +0000

Prod7 Gateway 0.3 went down

Use3 KMS 0.5 went down

Fri, 20 Mar 2026 10:40:26 +0000

Use3 KMS 0.5 went down

Prod6 KMS 0.5 went down

Fri, 20 Mar 2026 10:40:24 +0000

Prod6 KMS 0.5 went down

Prod7 VMM 0.5 went down

Fri, 20 Mar 2026 10:40:23 +0000

Prod7 VMM 0.5 went down

Prod4 Gateway 0.3 went down

Fri, 20 Mar 2026 10:40:22 +0000

Prod4 Gateway 0.3 went down

Prod7 VMM 0.3 went down

Fri, 20 Mar 2026 10:40:19 +0000

Prod7 VMM 0.3 went down

Prod9 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:40:17 +0000

Prod9 Eth KMS 0.5 went down

Prod7 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:40:17 +0000

Prod7 Eth KMS 0.5 went down

Prod9 Gateway 0.5 went down

Fri, 20 Mar 2026 10:40:15 +0000

Prod9 Gateway 0.5 went down

Prod7 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:40:12 +0000

Prod7 Base Gateway 0.5 went down

Prod9 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:40:11 +0000

Prod9 Base Gateway 0.5 went down

Proof t16z went down

Fri, 20 Mar 2026 10:40:11 +0000

Proof t16z went down

Prod5 KMS 0.5 went down

Fri, 20 Mar 2026 10:39:59 +0000

Prod5 KMS 0.5 went down

Prod5 Node went down

Fri, 20 Mar 2026 10:39:57 +0000

Prod5 Node went down

Prod7 Gateway 0.5 went down

Fri, 20 Mar 2026 10:39:44 +0000

Prod7 Gateway 0.5 went down

Use1 KMS 0.5 went down

Fri, 20 Mar 2026 10:39:43 +0000

Use1 KMS 0.5 went down

Prod9 KMS 0.5 went down

Fri, 20 Mar 2026 10:39:42 +0000

Prod9 KMS 0.5 went down

Prod3 KMS 0.5 went down

Fri, 20 Mar 2026 10:39:39 +0000

Prod3 KMS 0.5 went down

Prod7 VMM 0.3 p8 went down

Fri, 20 Mar 2026 10:39:38 +0000

Prod7 VMM 0.3 p8 went down

Prod7 KMS 0.5 went down

Fri, 20 Mar 2026 10:39:37 +0000

Prod7 KMS 0.5 went down

Prod3 Node went down

Fri, 20 Mar 2026 10:39:37 +0000

Prod3 Node went down

Prod7 Node went down

Fri, 20 Mar 2026 10:39:34 +0000

Prod7 Node went down

Prod6 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:39:32 +0000

Prod6 Eth KMS 0.5 went down

Prod10 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:39:28 +0000

Prod10 Eth KMS 0.5 went down

Prod10 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:39:05 +0000

Prod10 Base KMS 0.5 went down

Prod6 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:39:05 +0000

Prod6 Base KMS 0.5 went down

Use2 VMM 0.5 recovered

Fri, 20 Mar 2026 10:39:01 +0000

Use2 VMM 0.5 recovered

Use2 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:38:51 +0000

Use2 Gateway 0.5 recovered

Use2 Gateway 0.5 went down

Fri, 20 Mar 2026 10:35:11 +0000

Use2 Gateway 0.5 went down

Use2 KMS 0.5 went down

Fri, 20 Mar 2026 10:35:03 +0000

Use2 KMS 0.5 went down

Use2 VMM 0.5 went down

Fri, 20 Mar 2026 10:34:51 +0000

Use2 VMM 0.5 went down

gpt-oss-20b recovered

Fri, 20 Mar 2026 10:30:24 +0000

gpt-oss-20b recovered

qwen-2.5-7b-instruct recovered

Fri, 20 Mar 2026 10:29:22 +0000

qwen-2.5-7b-instruct recovered

Prod6 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:27:12 +0000

Prod6 Eth KMS 0.5 recovered

Use1 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:27:07 +0000

Use1 Base KMS 0.5 recovered

Prod10 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:27:07 +0000

Prod10 Eth KMS 0.5 recovered

Prod9 Node recovered

Fri, 20 Mar 2026 10:27:06 +0000

Prod9 Node recovered

Prod7 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:48 +0000

Prod7 Gateway 0.5 recovered

Prod9 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:48 +0000

Prod9 KMS 0.5 recovered

Prod7 Gateway 0.3 p8 recovered

Fri, 20 Mar 2026 10:26:47 +0000

Prod7 Gateway 0.3 p8 recovered

Use1 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:47 +0000

Use1 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:45 +0000

Prod3 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:44 +0000

Use3 KMS 0.5 recovered

Prod5 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:43 +0000

Prod5 Base Gateway 0.5 recovered

Prod9 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 10:26:43 +0000

Prod9 VMM 0.3 p8 recovered

Prod3 Node recovered

Fri, 20 Mar 2026 10:26:42 +0000

Prod3 Node recovered

Prod6 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:42 +0000

Prod6 Base KMS 0.5 recovered

deepseek-v3.2 recovered

Fri, 20 Mar 2026 10:26:42 +0000

deepseek-v3.2 recovered

Prod7 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 10:26:42 +0000

Prod7 VMM 0.3 p8 recovered

Prod5 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:41 +0000

Prod5 Gateway 0.5 recovered

Prod7 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:41 +0000

Prod7 KMS 0.5 recovered

Prod10 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:40 +0000

Prod10 Base KMS 0.5 recovered

Prod7 Node recovered

Fri, 20 Mar 2026 10:26:40 +0000

Prod7 Node recovered

Prod7 Gateway 0.3 recovered

Fri, 20 Mar 2026 10:26:37 +0000

Prod7 Gateway 0.3 recovered

Prod7 VMM 0.5 recovered

Fri, 20 Mar 2026 10:26:37 +0000

Prod7 VMM 0.5 recovered

Prod5 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:37 +0000

Prod5 KMS 0.5 recovered

Prod5 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:34 +0000

Prod5 Base KMS 0.5 recovered

Prod9 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:32 +0000

Prod9 Eth Gateway 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:32 +0000

Prod7 Eth Gateway 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:32 +0000

Prod9 Eth KMS 0.5 recovered

Prod7 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:31 +0000

Prod7 Eth KMS 0.5 recovered

Prod3 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:30 +0000

Prod3 Gateway 0.5 recovered

Prod4 Gateway 0.3 recovered

Fri, 20 Mar 2026 10:26:30 +0000

Prod4 Gateway 0.3 recovered

Proof t16z recovered

Fri, 20 Mar 2026 10:26:30 +0000

Proof t16z recovered

Prod5 Node recovered

Fri, 20 Mar 2026 10:26:30 +0000

Prod5 Node recovered

Prod7 VMM 0.3 recovered

Fri, 20 Mar 2026 10:26:28 +0000

Prod7 VMM 0.3 recovered

Prod6 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:28 +0000

Prod6 KMS 0.5 recovered

Phala PCCS recovered

Fri, 20 Mar 2026 10:26:27 +0000

Phala PCCS recovered

Prod5 Gateway 0.3 recovered

Fri, 20 Mar 2026 10:26:24 +0000

Prod5 Gateway 0.3 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 10:26:22 +0000

gemma-3-27b-it recovered

Prod9 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:18 +0000

Prod9 Base KMS 0.5 recovered

Prod4 VMM 0.3 recovered

Fri, 20 Mar 2026 10:26:17 +0000

Prod4 VMM 0.3 recovered

Prod3 VMM 0.5 recovered

Fri, 20 Mar 2026 10:26:17 +0000

Prod3 VMM 0.5 recovered

Prod9 VMM 0.5 recovered

Fri, 20 Mar 2026 10:26:16 +0000

Prod9 VMM 0.5 recovered

Prod10 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:16 +0000

Prod10 KMS 0.5 recovered

Prod5 VMM 0.5 recovered

Fri, 20 Mar 2026 10:26:16 +0000

Prod5 VMM 0.5 recovered

Prod9 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:16 +0000

Prod9 Base Gateway 0.5 recovered

Prod9 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:16 +0000

Prod9 Gateway 0.5 recovered

Prod7 Base KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:16 +0000

Prod7 Base KMS 0.5 recovered

Phala Cloud API recovered

Fri, 20 Mar 2026 10:26:15 +0000

Phala Cloud API recovered

Prod4 Node recovered

Fri, 20 Mar 2026 10:26:15 +0000

Prod4 Node recovered

Prod7 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 10:26:14 +0000

Prod7 Base Gateway 0.5 recovered

Use2 KMS 0.5 recovered

Fri, 20 Mar 2026 10:26:14 +0000

Use2 KMS 0.5 recovered

Prod5 VMM 0.3 recovered

Fri, 20 Mar 2026 10:26:14 +0000

Prod5 VMM 0.3 recovered

Prod4 Gateway 0.3 went down

Fri, 20 Mar 2026 10:18:16 +0000

Prod4 Gateway 0.3 went down

Phala PCCS went down

Fri, 20 Mar 2026 10:17:40 +0000

Phala PCCS went down

deepseek-v3.2 went down

Fri, 20 Mar 2026 10:17:29 +0000

deepseek-v3.2 went down

gpt-oss-20b went down

Fri, 20 Mar 2026 10:17:21 +0000

gpt-oss-20b went down

gemma-3-27b-it went down

Fri, 20 Mar 2026 10:17:05 +0000

gemma-3-27b-it went down

Proof t16z went down

Fri, 20 Mar 2026 10:17:01 +0000

Proof t16z went down

qwen-2.5-7b-instruct went down

Fri, 20 Mar 2026 10:16:28 +0000

qwen-2.5-7b-instruct went down

Prod5 VMM 0.5 went down

Fri, 20 Mar 2026 10:16:28 +0000

Prod5 VMM 0.5 went down

Prod5 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:16:04 +0000

Prod5 Base Gateway 0.5 went down

Prod7 Node went down

Fri, 20 Mar 2026 10:15:41 +0000

Prod7 Node went down

Prod3 Node went down

Fri, 20 Mar 2026 10:15:41 +0000

Prod3 Node went down

Prod7 VMM 0.3 p8 went down

Fri, 20 Mar 2026 10:15:39 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Gateway 0.3 went down

Fri, 20 Mar 2026 10:15:33 +0000

Prod7 Gateway 0.3 went down

Prod9 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:15:33 +0000

Prod9 Base KMS 0.5 went down

Prod5 KMS 0.5 went down

Fri, 20 Mar 2026 10:15:31 +0000

Prod5 KMS 0.5 went down

Prod7 VMM 0.5 went down

Fri, 20 Mar 2026 10:15:30 +0000

Prod7 VMM 0.5 went down

Prod5 Node went down

Fri, 20 Mar 2026 10:15:27 +0000

Prod5 Node went down

Prod9 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:15:26 +0000

Prod9 Eth KMS 0.5 went down

Prod7 VMM 0.3 went down

Fri, 20 Mar 2026 10:15:24 +0000

Prod7 VMM 0.3 went down

Prod3 Gateway 0.5 went down

Fri, 20 Mar 2026 10:15:24 +0000

Prod3 Gateway 0.5 went down

Prod5 Gateway 0.5 went down

Fri, 20 Mar 2026 10:15:24 +0000

Prod5 Gateway 0.5 went down

Prod7 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:15:23 +0000

Prod7 Eth KMS 0.5 went down

Prod5 Gateway 0.3 went down

Fri, 20 Mar 2026 10:15:19 +0000

Prod5 Gateway 0.3 went down

Prod6 KMS 0.5 went down

Fri, 20 Mar 2026 10:15:15 +0000

Prod6 KMS 0.5 went down

Prod4 Node went down

Fri, 20 Mar 2026 10:15:15 +0000

Prod4 Node went down

Prod9 VMM 0.5 went down

Fri, 20 Mar 2026 10:15:10 +0000

Prod9 VMM 0.5 went down

Prod7 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:15:10 +0000

Prod7 Base Gateway 0.5 went down

Prod9 Base Gateway 0.5 went down

Fri, 20 Mar 2026 10:15:09 +0000

Prod9 Base Gateway 0.5 went down

Prod9 Gateway 0.5 went down

Fri, 20 Mar 2026 10:15:09 +0000

Prod9 Gateway 0.5 went down

Prod10 KMS 0.5 went down

Fri, 20 Mar 2026 10:15:03 +0000

Prod10 KMS 0.5 went down

Prod5 VMM 0.3 went down

Fri, 20 Mar 2026 10:14:56 +0000

Prod5 VMM 0.3 went down

Prod4 VMM 0.3 went down

Fri, 20 Mar 2026 10:14:54 +0000

Prod4 VMM 0.3 went down

Prod9 Node went down

Fri, 20 Mar 2026 10:14:51 +0000

Prod9 Node went down

Phala Cloud API went down

Fri, 20 Mar 2026 10:14:51 +0000

Phala Cloud API went down

Prod3 VMM 0.5 went down

Fri, 20 Mar 2026 10:14:51 +0000

Prod3 VMM 0.5 went down

Prod7 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:14:51 +0000

Prod7 Base KMS 0.5 went down

Prod7 Gateway 0.3 p8 went down

Fri, 20 Mar 2026 10:14:49 +0000

Prod7 Gateway 0.3 p8 went down

Use2 KMS 0.5 went down

Fri, 20 Mar 2026 10:14:47 +0000

Use2 KMS 0.5 went down

Prod7 Gateway 0.5 went down

Fri, 20 Mar 2026 10:14:47 +0000

Prod7 Gateway 0.5 went down

Prod9 KMS 0.5 went down

Fri, 20 Mar 2026 10:14:43 +0000

Prod9 KMS 0.5 went down

Prod9 VMM 0.3 p8 went down

Fri, 20 Mar 2026 10:14:43 +0000

Prod9 VMM 0.3 p8 went down

Prod7 KMS 0.5 went down

Fri, 20 Mar 2026 10:14:40 +0000

Prod7 KMS 0.5 went down

Prod9 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 10:14:40 +0000

Prod9 Eth Gateway 0.5 went down

Prod5 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:14:40 +0000

Prod5 Base KMS 0.5 went down

Prod3 KMS 0.5 went down

Fri, 20 Mar 2026 10:14:38 +0000

Prod3 KMS 0.5 went down

Prod7 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 10:14:36 +0000

Prod7 Eth Gateway 0.5 went down

Use1 KMS 0.5 went down

Fri, 20 Mar 2026 10:14:31 +0000

Use1 KMS 0.5 went down

Use3 KMS 0.5 went down

Fri, 20 Mar 2026 10:14:31 +0000

Use3 KMS 0.5 went down

Use1 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:14:16 +0000

Use1 Base KMS 0.5 went down

Prod6 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:14:08 +0000

Prod6 Base KMS 0.5 went down

Prod10 Base KMS 0.5 went down

Fri, 20 Mar 2026 10:14:03 +0000

Prod10 Base KMS 0.5 went down

Prod6 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:12:15 +0000

Prod6 Eth KMS 0.5 went down

Prod10 Eth KMS 0.5 went down

Fri, 20 Mar 2026 10:11:23 +0000

Prod10 Eth KMS 0.5 went down

Use2 Gateway 0.5 recovered

Fri, 20 Mar 2026 10:09:28 +0000

Use2 Gateway 0.5 recovered

Use2 Gateway 0.5 went down

Fri, 20 Mar 2026 10:06:09 +0000

Use2 Gateway 0.5 went down

deepseek-v3.2 recovered

Fri, 20 Mar 2026 09:46:27 +0000

deepseek-v3.2 recovered

gemma-3-27b-it recovered

Fri, 20 Mar 2026 09:45:37 +0000

gemma-3-27b-it recovered

gpt-oss-20b recovered

Fri, 20 Mar 2026 09:45:24 +0000

gpt-oss-20b recovered

Prod5 KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:49 +0000

Prod5 KMS 0.5 recovered

Prod6 Base KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:41 +0000

Prod6 Base KMS 0.5 recovered

Prod7 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:26 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 VMM 0.3 recovered

Fri, 20 Mar 2026 09:43:25 +0000

Prod7 VMM 0.3 recovered

Prod5 Node recovered

Fri, 20 Mar 2026 09:43:25 +0000

Prod5 Node recovered

Prod5 VMM 0.3 recovered

Fri, 20 Mar 2026 09:43:19 +0000

Prod5 VMM 0.3 recovered

Prod9 VMM 0.5 recovered

Fri, 20 Mar 2026 09:43:18 +0000

Prod9 VMM 0.5 recovered

Prod10 KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:18 +0000

Prod10 KMS 0.5 recovered

Prod9 Gateway 0.5 recovered

Fri, 20 Mar 2026 09:43:17 +0000

Prod9 Gateway 0.5 recovered

Phala Cloud API recovered

Fri, 20 Mar 2026 09:43:16 +0000

Phala Cloud API recovered

Prod4 Node recovered

Fri, 20 Mar 2026 09:43:15 +0000

Prod4 Node recovered

Prod9 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 09:43:15 +0000

Prod9 Base Gateway 0.5 recovered

Prod7 Base Gateway 0.5 recovered

Fri, 20 Mar 2026 09:43:15 +0000

Prod7 Base Gateway 0.5 recovered

Prod6 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:07 +0000

Prod6 Eth KMS 0.5 recovered

Use1 Base KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:07 +0000

Use1 Base KMS 0.5 recovered

Use2 KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:04 +0000

Use2 KMS 0.5 recovered

Prod3 VMM 0.5 recovered

Fri, 20 Mar 2026 09:43:02 +0000

Prod3 VMM 0.5 recovered

Prod9 Base KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:02 +0000

Prod9 Base KMS 0.5 recovered

Prod10 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:01 +0000

Prod10 Eth KMS 0.5 recovered

Prod7 Base KMS 0.5 recovered

Fri, 20 Mar 2026 09:43:01 +0000

Prod7 Base KMS 0.5 recovered

Prod9 Node recovered

Fri, 20 Mar 2026 09:43:00 +0000

Prod9 Node recovered

Prod5 VMM 0.5 recovered

Fri, 20 Mar 2026 09:42:59 +0000

Prod5 VMM 0.5 recovered

Prod7 Gateway 0.3 p8 recovered

Fri, 20 Mar 2026 09:42:55 +0000

Prod7 Gateway 0.3 p8 recovered

Prod4 Gateway 0.3 recovered

Fri, 20 Mar 2026 09:42:55 +0000

Prod4 Gateway 0.3 recovered

Prod3 Gateway 0.5 recovered

Fri, 20 Mar 2026 09:42:54 +0000

Prod3 Gateway 0.5 recovered

Prod7 VMM 0.5 recovered

Fri, 20 Mar 2026 09:42:54 +0000

Prod7 VMM 0.5 recovered

Prod7 Gateway 0.5 recovered

Fri, 20 Mar 2026 09:42:54 +0000

Prod7 Gateway 0.5 recovered

Prod9 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 09:42:53 +0000

Prod9 VMM 0.3 p8 recovered

Prod10 Base KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:53 +0000

Prod10 Base KMS 0.5 recovered

Prod5 Gateway 0.3 recovered

Fri, 20 Mar 2026 09:42:53 +0000

Prod5 Gateway 0.3 recovered

Prod9 KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:52 +0000

Prod9 KMS 0.5 recovered

Prod9 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 09:42:52 +0000

Prod9 Eth Gateway 0.5 recovered

Prod9 Eth KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:50 +0000

Prod9 Eth KMS 0.5 recovered

Prod5 Gateway 0.5 recovered

Fri, 20 Mar 2026 09:42:50 +0000

Prod5 Gateway 0.5 recovered

Prod7 KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:49 +0000

Prod7 KMS 0.5 recovered

Prod3 Node recovered

Fri, 20 Mar 2026 09:42:49 +0000

Prod3 Node recovered

Prod7 Gateway 0.3 recovered

Fri, 20 Mar 2026 09:42:48 +0000

Prod7 Gateway 0.3 recovered

Use1 KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:48 +0000

Use1 KMS 0.5 recovered

Prod3 KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:47 +0000

Prod3 KMS 0.5 recovered

Use3 KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:47 +0000

Use3 KMS 0.5 recovered

Prod7 Node recovered

Fri, 20 Mar 2026 09:42:46 +0000

Prod7 Node recovered

Prod7 Eth Gateway 0.5 recovered

Fri, 20 Mar 2026 09:42:46 +0000

Prod7 Eth Gateway 0.5 recovered

Prod5 Base KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:46 +0000

Prod5 Base KMS 0.5 recovered

Prod7 VMM 0.3 p8 recovered

Fri, 20 Mar 2026 09:42:46 +0000

Prod7 VMM 0.3 p8 recovered

Prod6 KMS 0.5 recovered

Fri, 20 Mar 2026 09:42:43 +0000

Prod6 KMS 0.5 recovered

Prod10 KMS 0.5 went down

Fri, 20 Mar 2026 09:39:44 +0000

Prod10 KMS 0.5 went down

Prod4 Node went down

Fri, 20 Mar 2026 09:38:40 +0000

Prod4 Node went down

Use2 KMS 0.5 went down

Fri, 20 Mar 2026 09:38:36 +0000

Use2 KMS 0.5 went down

Prod6 KMS 0.5 went down

Fri, 20 Mar 2026 09:38:14 +0000

Prod6 KMS 0.5 went down

Use1 KMS 0.5 went down

Fri, 20 Mar 2026 09:37:57 +0000

Use1 KMS 0.5 went down

Prod5 Gateway 0.5 went down

Fri, 20 Mar 2026 09:37:54 +0000

Prod5 Gateway 0.5 went down

Prod5 Gateway 0.3 went down

Fri, 20 Mar 2026 09:37:50 +0000

Prod5 Gateway 0.3 went down

Use3 KMS 0.5 went down

Fri, 20 Mar 2026 09:37:48 +0000

Use3 KMS 0.5 went down

deepseek-v3.2 went down

Fri, 20 Mar 2026 09:37:31 +0000

deepseek-v3.2 went down

gpt-oss-20b went down

Fri, 20 Mar 2026 09:37:22 +0000

gpt-oss-20b went down

Prod3 Node went down

Fri, 20 Mar 2026 09:37:13 +0000

Prod3 Node went down

Prod3 VMM 0.5 went down

Fri, 20 Mar 2026 09:37:02 +0000

Prod3 VMM 0.5 went down

Prod9 Node went down

Fri, 20 Mar 2026 09:37:01 +0000

Prod9 Node went down

Prod4 Gateway 0.3 went down

Fri, 20 Mar 2026 09:36:55 +0000

Prod4 Gateway 0.3 went down

Prod9 KMS 0.5 went down

Fri, 20 Mar 2026 09:36:55 +0000

Prod9 KMS 0.5 went down

Prod7 Gateway 0.3 p8 went down

Fri, 20 Mar 2026 09:36:54 +0000

Prod7 Gateway 0.3 p8 went down

Prod9 VMM 0.3 p8 went down

Fri, 20 Mar 2026 09:36:41 +0000

Prod9 VMM 0.3 p8 went down

Prod3 KMS 0.5 went down

Fri, 20 Mar 2026 09:36:40 +0000

Prod3 KMS 0.5 went down

Prod7 Node went down

Fri, 20 Mar 2026 09:36:37 +0000

Prod7 Node went down

Prod7 VMM 0.3 p8 went down

Fri, 20 Mar 2026 09:36:37 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Gateway 0.5 went down

Fri, 20 Mar 2026 09:36:36 +0000

Prod7 Gateway 0.5 went down

Prod7 VMM 0.5 went down

Fri, 20 Mar 2026 09:36:35 +0000

Prod7 VMM 0.5 went down

Prod7 Gateway 0.3 went down

Fri, 20 Mar 2026 09:36:33 +0000

Prod7 Gateway 0.3 went down

Prod7 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 09:36:33 +0000

Prod7 Eth Gateway 0.5 went down

Prod9 Eth Gateway 0.5 went down

Fri, 20 Mar 2026 09:36:33 +0000

Prod9 Eth Gateway 0.5 went down

Prod5 VMM 0.3 went down

Fri, 20 Mar 2026 09:36:33 +0000

Prod5 VMM 0.3 went down

Prod3 Gateway 0.5 went down

Fri, 20 Mar 2026 09:36:29 +0000

Prod3 Gateway 0.5 went down

Prod5 KMS 0.5 went down

Fri, 20 Mar 2026 09:36:26 +0000

Prod5 KMS 0.5 went down

Prod5 Base KMS 0.5 went down

Fri, 20 Mar 2026 09:36:24 +0000

Prod5 Base KMS 0.5 went down

Prod5 Node went down

Fri, 20 Mar 2026 09:36:21 +0000

Prod5 Node went down

Prod9 Eth KMS 0.5 went down

Fri, 20 Mar 2026 09:36:18 +0000

Prod9 Eth KMS 0.5 went down

Prod7 Eth KMS 0.5 went down

Fri, 20 Mar 2026 09:36:17 +0000

Prod7 Eth KMS 0.5 went down

Phala Cloud API went down

Fri, 20 Mar 2026 09:36:16 +0000

Phala Cloud API went down

Prod9 VMM 0.5 went down

Fri, 20 Mar 2026 09:36:15 +0000

Prod9 VMM 0.5 went down

Prod7 VMM 0.3 went down

Fri, 20 Mar 2026 09:36:15 +0000

Prod7 VMM 0.3 went down

gemma-3-27b-it went down

Fri, 20 Mar 2026 09:36:14 +0000

gemma-3-27b-it went down

Prod9 Gateway 0.5 went down

Fri, 20 Mar 2026 09:36:09 +0000

Prod9 Gateway 0.5 went down

Prod7 Base Gateway 0.5 went down

Fri, 20 Mar 2026 09:36:07 +0000

Prod7 Base Gateway 0.5 went down

Prod9 Base Gateway 0.5 went down

Fri, 20 Mar 2026 09:36:06 +0000

Prod9 Base Gateway 0.5 went down

Prod6 Base KMS 0.5 went down

Fri, 20 Mar 2026 09:36:06 +0000

Prod6 Base KMS 0.5 went down

Prod10 Base KMS 0.5 went down

Fri, 20 Mar 2026 09:36:03 +0000

Prod10 Base KMS 0.5 went down

Prod5 VMM 0.5 went down

Fri, 20 Mar 2026 09:36:02 +0000

Prod5 VMM 0.5 went down

Prod9 Base KMS 0.5 went down

Fri, 20 Mar 2026 09:35:59 +0000

Prod9 Base KMS 0.5 went down

Prod7 Base KMS 0.5 went down

Fri, 20 Mar 2026 09:35:58 +0000

Prod7 Base KMS 0.5 went down

Prod7 KMS 0.5 went down

Fri, 20 Mar 2026 09:33:39 +0000

Prod7 KMS 0.5 went down

Prod6 Eth KMS 0.5 went down

Fri, 20 Mar 2026 09:32:20 +0000

Prod6 Eth KMS 0.5 went down

Use1 Base KMS 0.5 went down

Fri, 20 Mar 2026 09:32:16 +0000

Use1 Base KMS 0.5 went down

Prod10 Eth KMS 0.5 went down

Fri, 20 Mar 2026 09:32:15 +0000

Prod10 Eth KMS 0.5 went down

Prod7 Eth KMS 0.5 recovered

Thu, 19 Mar 2026 19:48:28 +0000

Prod7 Eth KMS 0.5 recovered

Prod7 Base KMS 0.5 recovered

Thu, 19 Mar 2026 19:47:26 +0000

Prod7 Base KMS 0.5 recovered

Prod7 Gateway 0.5 recovered

Thu, 19 Mar 2026 19:28:55 +0000

Prod7 Gateway 0.5 recovered

Prod7 Eth Gateway 0.5 recovered

Thu, 19 Mar 2026 19:28:33 +0000

Prod7 Eth Gateway 0.5 recovered

Prod7 Base Gateway 0.5 recovered

Thu, 19 Mar 2026 19:28:13 +0000

Prod7 Base Gateway 0.5 recovered

Prod7 KMS 0.5 recovered

Thu, 19 Mar 2026 19:26:58 +0000

Prod7 KMS 0.5 recovered

Prod7 Gateway 0.3 p8 recovered

Thu, 19 Mar 2026 19:26:51 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 VMM 0.3 p8 recovered

Thu, 19 Mar 2026 19:26:43 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Node recovered

Thu, 19 Mar 2026 19:26:42 +0000

Prod7 Node recovered

Prod7 Gateway 0.3 recovered

Thu, 19 Mar 2026 19:26:36 +0000

Prod7 Gateway 0.3 recovered

Prod7 VMM 0.5 recovered

Thu, 19 Mar 2026 19:26:34 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 recovered

Thu, 19 Mar 2026 19:26:31 +0000

Prod7 VMM 0.3 recovered

Prod7 Gateway 0.3 p8 went down

Thu, 19 Mar 2026 19:03:16 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 VMM 0.3 p8 went down

Thu, 19 Mar 2026 19:03:16 +0000

Prod7 VMM 0.3 p8 went down

Urgent Maintenace Notice

Thu, 19 Mar 2026 19:00:00 -0000

We're investigating a hardware failure and outage affecting prod7, prod7-v3x, prod8-v3x, and working to resolve it. We working on the recoverying. We apologize for that and keep posting here if any update.

Prod7 Gateway 0.3 p8 recovered

Thu, 19 Mar 2026 18:59:56 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 VMM 0.3 p8 recovered

Thu, 19 Mar 2026 18:59:43 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Node went down

Thu, 19 Mar 2026 18:46:22 +0000

Prod7 Node went down

Prod7 VMM 0.3 p8 went down

Thu, 19 Mar 2026 18:46:16 +0000

Prod7 VMM 0.3 p8 went down

Prod7 VMM 0.3 went down

Thu, 19 Mar 2026 18:46:16 +0000

Prod7 VMM 0.3 went down

Prod7 VMM 0.5 went down

Thu, 19 Mar 2026 18:46:16 +0000

Prod7 VMM 0.5 went down

Prod7 Gateway 0.3 p8 went down

Thu, 19 Mar 2026 18:46:03 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 Gateway 0.3 went down

Thu, 19 Mar 2026 18:46:03 +0000

Prod7 Gateway 0.3 went down

Prod7 Gateway 0.3 p8 recovered

Thu, 19 Mar 2026 18:12:52 +0000

Prod7 Gateway 0.3 p8 recovered

Prod7 VMM 0.3 p8 recovered

Thu, 19 Mar 2026 18:12:51 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Thu, 19 Mar 2026 18:12:39 +0000

Prod7 Gateway 0.3 recovered

Prod7 VMM 0.5 recovered

Thu, 19 Mar 2026 18:12:36 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 recovered

Thu, 19 Mar 2026 18:12:26 +0000

Prod7 VMM 0.3 recovered

Prod7 Node recovered

Thu, 19 Mar 2026 18:11:40 +0000

Prod7 Node recovered

Prod7 Gateway 0.3 went down

Thu, 19 Mar 2026 18:06:59 +0000

Prod7 Gateway 0.3 went down

Prod7 VMM 0.3 p8 went down

Thu, 19 Mar 2026 18:05:42 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Node went down

Thu, 19 Mar 2026 18:05:37 +0000

Prod7 Node went down

Prod7 VMM 0.5 went down

Thu, 19 Mar 2026 18:05:32 +0000

Prod7 VMM 0.5 went down

Prod7 VMM 0.3 went down

Thu, 19 Mar 2026 18:05:18 +0000

Prod7 VMM 0.3 went down

Prod7 VMM 0.5 recovered

Thu, 19 Mar 2026 16:48:39 +0000

Prod7 VMM 0.5 recovered

Prod7 VMM 0.3 recovered

Thu, 19 Mar 2026 16:48:26 +0000

Prod7 VMM 0.3 recovered

Prod7 VMM 0.3 p8 recovered

Thu, 19 Mar 2026 16:47:45 +0000

Prod7 VMM 0.3 p8 recovered

Prod7 Gateway 0.3 recovered

Thu, 19 Mar 2026 16:47:38 +0000

Prod7 Gateway 0.3 recovered

Prod7 Node recovered

Thu, 19 Mar 2026 16:46:49 +0000

Prod7 Node recovered

Prod7 KMS 0.5 went down

Thu, 19 Mar 2026 16:40:44 +0000

Prod7 KMS 0.5 went down

Prod7 VMM 0.3 p8 went down

Thu, 19 Mar 2026 16:40:37 +0000

Prod7 VMM 0.3 p8 went down

Prod7 Node went down

Thu, 19 Mar 2026 16:40:37 +0000

Prod7 Node went down

Prod7 Eth Gateway 0.5 went down

Thu, 19 Mar 2026 16:40:28 +0000

Prod7 Eth Gateway 0.5 went down

Prod7 Gateway 0.3 went down

Thu, 19 Mar 2026 16:40:28 +0000

Prod7 Gateway 0.3 went down

Prod7 VMM 0.5 went down

Thu, 19 Mar 2026 16:40:24 +0000

Prod7 VMM 0.5 went down

Prod7 VMM 0.3 went down

Thu, 19 Mar 2026 16:40:19 +0000

Prod7 VMM 0.3 went down

Prod7 Gateway 0.5 went down

Thu, 19 Mar 2026 16:39:55 +0000

Prod7 Gateway 0.5 went down

Prod7 Eth KMS 0.5 went down

Thu, 19 Mar 2026 16:39:51 +0000

Prod7 Eth KMS 0.5 went down

Prod7 Base Gateway 0.5 went down

Thu, 19 Mar 2026 16:39:49 +0000

Prod7 Base Gateway 0.5 went down

Prod7 Gateway 0.3 p8 went down

Thu, 19 Mar 2026 16:39:20 +0000

Prod7 Gateway 0.3 p8 went down

Prod7 Base KMS 0.5 went down

Thu, 19 Mar 2026 16:39:19 +0000

Prod7 Base KMS 0.5 went down

2026-03-02 prod7 Urgent Hardware Maintenance

Mon, 02 Mar 2026 04:00:00 -0000

Dear Customers, We need to perform urgent maintenance to ensure the continued stability and reliability of our platform. This maintenance affects the following nodes only: 🔧 Affected nodes: prod7, prod7-v3x, prod8-v3x ⏱️ Expected duration: 30-60 minutes During the maintenance window: - Services deployed on the above nodes will be temporarily unavailable - Deployments on all other nodes are not affected and will continue to run normally - No action is required on your end If you have any questions or concerns, please don't hesitate to reach out. Thank you for your understanding.

Deepseek-v3.2 maintenance

Mon, 15 Dec 2025 11:30:07 -0000

Maintenance completed

Deepseek-v3.2 maintenance

Fri, 05 Dec 2025 00:00:44 -0000

Deepseek-v3.2 is currently being debugged and may experience instability. We apologize for any inconvenience.

Phala Cloud Maintenance Service on Prod5

Thu, 16 Oct 2025 02:00:56 -0000

Phala Cloud has initiated a scheduled maintenance on node Prod5.

2025-08-29 prod6 attestation generation failure

Sat, 30 Aug 2025 03:05:00 -0000

# 2025-08-29 prod6 attestation generation failure ## **1. Incident Summary** - **Incident Name**: prod6 attestation generation failure - **Date and Time**: 2025/08/29 23:43:22 - 2025/08/30 03:05:00 - **Duration**: 201 minutes - **Severity Level**: P0 - **Impact**: All CVMs in Prod6 are unable to get attestation. - **Participants:** Leechael, Kevin, Hang Yin --- ## **2. Timeline of Events** - **Pre-Incident Events**: - **2025/08/29 23:43:22** - qgsd emitted `[QPL] No certificate data for this platform.` in the log. - **Incident Detection**: - **2025/08/30** **00:54** - Customer reported attestation service failures - **Incident Response**: - **2025/08/30 01:04** - Engineering team acknowledged the issue and initiated investigation - **2025/08/30 01:36** - Isolated the impact to CVMs on prod6 - **2025/08/30** **01:44** - Escalated to additional engineering resources for deeper investigation - **Incident Resolution**: - **2025/08/30 02:43** - Identified QGS daemon (qgsd) failure and initiated service restart - **2025/08/30 03:00** - Service restoration confirmed, customer notified - **2025/08/30 03:05** - Full functionality verified across all affected systems --- ## **3. Root Cause Analysis (RCA)** - **Primary Cause**: - The primary cause was the unexpected termination of the Quote Generation Service (QGS) on the affected production server. The QGS, a critical Intel service component that hosts the TD Quoting Enclave, experienced a network-related failure that went undetected due to insufficient monitoring coverage. As the QGS must run on the same physical machine as the corresponding Trusted Domain for attestation verification, the service failure created a complete attestation outage for all CVMs on that host. - When we check the source code, it returns `NO certificate data for this platform` is because GQS sent malformed requests to PCCS and received 404 error. - [https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/a46ee8ab10569962c5cd7397b4babd4a47431976/QuoteGeneration/qcnl/linux/network_wrapper.cpp#L247](https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/a46ee8ab10569962c5cd7397b4babd4a47431976/QuoteGeneration/qcnl/linux/network_wrapper.cpp#L247) - **Contributing Factors**: - **Insufficient Monitoring and Alerting** The network failure affecting QGS was not detected promptly due to gaps in monitoring and alerting for service health and network connectivity. No automated alerts were triggered for the QGS process termination or its inability to reach the PCCS server. --- ## **4. Impact Analysis** - **Technical Impact & Business Impact** - All running CVMs in prod6 and guest-agent API. - No new CVMs can launch successfully while dealing with the incident. --- ## **5. Resolution Details** - **Steps Taken to Resolve the Incident**: - After restarting the qgsd service, everything is back online. --- ## **6. Lessons Learned** - **Platform Monitoring Gap**: The absence of dedicated monitoring on Cloud platform for the QGS service delayed detection and diagnosis of the failure

2025-08-29 prod6 attestation generation failure

Fri, 29 Aug 2025 23:43:00 -0000

prod6 attestation generation failure

Scheduled Service Interruption Notice for prod7

Thu, 31 Jul 2025 06:00:00 -0000

We have scheduled a maintenance window for our prod7 environment due to a data center migration by our infrastructure partner. Expected Impact: Service interruptions of up to 6 hours during the migration Exact maintenance date and time will be announced at least 2 weeks prior Recommended Action: Users may proactively migrate their CVMs to other production environments before the scheduled maintenance to avoid potential service disruption. Next Steps: We will provide the specific maintenance schedule and detailed instructions once confirmed by our data center partner. For questions or assistance with CVM migration, please contact our support team. We apologize for any inconvenience and appreciate your understanding.

Prod5 Hardware Failure and Reboot Automatically

Mon, 30 Jun 2025 09:44:00 -0000

We’re back! prod5 should be up and running. Thanks for bearing with us.

Maintenance on all nodes

Wed, 18 Jun 2025 14:05:00 -0000

A regular component update and maintenance on all nodes. The uptime will be affected because the health check endpoints have been updated.