AI Systems on LRZ Service Status

[Abgeschlossen] [Wartung] 2026-08-10 - AI Systems Maintenance from August 17th-18th, 2026

Tue, 18 Aug 2026 19:32:13

Mo., 17.08.2026 07:00 – Di., 18.08.2026 20:00
Betroffene Services: [AI Systems]

The AI Systems (including the BayernKI and MCML system segments) were under a scheduled maintenance August 17th and 18th, 2026. Normal user operation has resumed.

Web Service: DEGRADED (Jupyter & RStudio)
SSH Service: UP

[Störung] 2026-08-12 14:00 - Unscheduled Maintenance: CPU Partition and OOD (Jupyter/RStudio)

Mon, 17 Aug 2026 11:25:39

Mi., 12.08.2026 14:00 – So., 30.08.2026 14:00
Betroffene Services: [AI Systems]

The OOD web applications are operating in a degraded state as previously reported problems persist. Parts of the CPU partition are unvailable, affecting Jupyter Notebooks and RStudio.

Users using these services may experience a high queue time for their sessions.

All current-generation AI Systems resources, including the H100 and standard A100 GPU partitions, remain fully operational. SSH access and GPU batch jobs on these systems are not affected.

We are working to restore the affected resources as quickly as possible. Thank you for your understanding and patience.

Web Service: DEGRADED (Jupyter & RStudio)

[Behoben] [Störung] 2026-07-27 14:00 - Unscheduled Maintenance: CPU Partition and OOD (Jupyter/RStudio)

Thu, 30 Jul 2026 10:45:44

Fr., 24.07.2026 10:00 – Mi., 29.07.2026 14:00
Betroffene Services: [AI Systems]

The OOD web applications are currently unavailable, including Jupyter Notebooks and RStudio.

The CPU partition is currently unavailable due to maintenance. In addition, several legacy GPU systems (P100 and V100) are temporarily unavailable.

All current-generation AI Systems resources, including the H100 and standard A100 GPU partitions, remain fully operational. SSH access and GPU batch jobs on these systems are not affected.

We are working to restore the affected resources as quickly as possible. Thank you for your patience.

Web Service: DEGRADED (Jupyter & RStudio)
SSH Service: UP

[Behoben] [Störung] 2026-07-17 - AI Systems was undergoing unscheduled maintenance from July 9 15:00 to July 17 16:00.

Fri, 17 Jul 2026 16:11:44

Do., 09.07.2026 15:00 – Fr., 17.07.2026 16:00
Betroffene Services: [AI Systems]

The AI Systems are now operational again and were in unplanned maintenance in response to a security advisory. None of the jobs that were either running or queued during the maintenance had to be terminated and could be kept running.

We sincerely apologize for the disruption.

[Behoben] [Störung] 2026-06-30 - AI Systems was undergoing unscheduled maintenance from June 29 15:00 to June 30 12:00.

Tue, 30 Jun 2026 16:01:26

Mo., 29.06.2026 15:00 – Di., 30.06.2026 10:00
Betroffene Services: [AI Systems]

The AI Systems have entered an unplanned maintenance in response to an urgent advisory. We expect the system to be available again on Tuesday, June 30, at 10:00.

As part of the unscheduled maintenance, all running jobs were terminated and will need to be restarted once the AI Systems are back online. Jobs that were queued but had not yet started will remain in the queue and will be scheduled automatically after the AI Systems resume operation.

We sincerely apologize for the disruption and will keep you informed as new information becomes available. In the meantime, we kindly ask for your patience and understanding.

[Abgeschlossen] [Wartung] 2026-06-24 - AI Systems was under maintenance from June 22 to 24, 2026.

Thu, 25 Jun 2026 14:42:13

Mo., 22.06.2026 08:00 – Mi., 24.06.2026 18:00
Betroffene Services: [AI Systems]

The following changes were introduced:

Operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes.
Upgraded the GPFS parallel file system to version 5.3.2-4 for security and reliability improvements.
Upgraded the GPU software stack to NVIDIA Driver 580 and CUDA 13 for support of the latest AI software and CUDA features.

[Behoben] [Störung] 2026-05-26 18:00 - Offline due to Power Outage

Thu, 28 May 2026 14:04:36

Mo., 25.05.2026 04:00 – Di., 26.05.2026 18:00
Betroffene Services: [AI Systems]

Recovery of the AI Systems infrastructure is currently ongoing following the recent power outage.

Services and system components are being brought back online gradually and in a controlled manner. Please expect partial availability during the recovery phase. While many systems may already become reachable again, diagnostics, integrity checks, and validation procedures are still ongoing to ensure stable and reliable operation.

Users may therefore continue to experience temporary interruptions, reduced availability, or delayed service restoration for individual components while recovery work continues.

Further updates will follow as additional systems return to operation and diagnostics are completed.

previous title: “Offline due to Power Outage, 25 May 2026”

Due to a power grid outage, the AI Systems (including BayernKI and MCML) were unavailable between 25 May 04:00 CET and 26 May 2026 15:00 CET.

Action required: Jobs that were running at the time of the outage were abruptly terminated. Please review your job queue and history, and manually resubmit any affected jobs.

[Behoben] [Störung] Unscheduled Maintenance, 19 May 2026

Tue, 26 May 2026 08:21:40

Fr., 15.05.2026 09:30 – Di., 19.05.2026 11:30
Betroffene Services: [AI Systems]

The AI Systems login nodes were taken offline between 15.5.2026 09:30 CET and 19.5.2026 11:30 CET.

System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority.

Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation.

Another maintenance window will be scheduled to apply upstream patches as they become available.

[Behoben] [Störung] Unscheduled Maintenance, 7 May 2026

Tue, 26 May 2026 08:21:40

Do., 07.05.2026 23:00 – Fr., 08.05.2026 15:00
Betroffene Services: [AI Systems]

The AI Systems login nodes were taken offline between 7.5.2026 23:00 CET and 8.5.2026 15:00 CET.

Another maintenance window will be scheduled to apply upstream patches as they become available.

[Behoben] [Störung] Unscheduled Maintenance, 30 Apr 2026

Tue, 26 May 2026 08:16:08

Do., 30.04.2026 09:00 – Do., 30.04.2026 17:00
Betroffene Services: [AI Systems]

The AI Systems login nodes were taken offline between 30.4.2026 09:00 CET and 30.4.2026 17:00 CET.

Another maintenance window will be scheduled to apply upstream patches as they become available.

[Abgeschlossen] [Wartung] 2026-03-18 - AI Systems was under maintenance from March 16 to 18, 2026.

Thu, 25 Jun 2026 14:43:48

Mo., 16.03.2026 06:00 – Mi., 18.03.2026 18:00
Betroffene Services: [AI Systems]

The following changes were introduced:

Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes
Networking interface & switch configurations have been homogenized and switch firmware has been updated
Virtual infrastructure components have been moved to new backend

[Abgeschlossen] [Wartung] 2025-12-03 - AI Systems was under maintenance from December 1 to 3, 2025.

Thu, 25 Jun 2026 14:43:03

Mo., 01.12.2025 06:00 – Mi., 03.12.2025 17:00
Betroffene Services: [AI Systems]

The following changes were introduced:

The AI Systems have been added to status.lrz.de and any future maintenance and service disruption announcements will be made public primarily via this channel
Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes
The web-based frontend, Open OnDemand, has been updated to version 3.1.16
Data Science Storage configuration parameters have been updated