Prod8 has experienced several issues recently. After investigation, we found that these were caused by a hardware fault specific to that server. To prevent further disruption, we’ll be moving all CVMs from the affected hardware to a new, stable server.

Prod8 is back. We keep monitoring.

2025-08-29 prod6 attestation generation failure

1. Incident Summary

• Incident Name: prod6 attestation generation failure
• Date and Time: 2025/08/29 23:43:22 - 2025/08/30 03:05:00
• Duration: 201 minutes
• Severity Level: P0
• Impact: All CVMs in Prod6 are unable to get attestation.
• Participants: Leechael, Kevin, Hang Yin

2. Timeline of Events

• Pre-Incident Events:
  • 2025/08/29 23:43:22 - qgsd emitted `[QPL] No certificate data ...