openSUSE:Post-mortem-20241122

• What/Problem: download.opensuse.org was returning 503, MirrorCache service was not operational
• When: 2024-11-22 07:30 to 08:00 UTC and 13:22 to 13:54 UTC
• Why: We identified a number of contributing issues:
  • On the MySQL server VM responsible for serving MirrorCache:
    • a background perl job allocated 10GB of RAM, causing OOMs in the mysql process and made the VM stuck.
    • the MySQL service segfaulted
    • the disk was temporarily full
    • few people know how to log in to this VM