Restart NFS and SMB services, all access to M: and S: drives will be disrupted
Verifying - SMB and NFS services have been restarted. All file locks have been released. S: and M: drives are now available
Aug 04, 2026 - 17:15 NZST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 04, 2026 - 17:00 NZST
Scheduled - We will be clearing the current locked files on Wednesday July 29th at 1700hrs. All access to the M: and S: drives from Windows systems will be disrupted for a short time.
Aug 4, 2026 17:00-17:30 NZST
OnDemand and Globus Degraded Performance
Monitoring - OnDemand - a downed server was identified and rebooted and testing showed responsive performance.
Globus - has been restarted and maintained connectivity.

Should further issue reappear please do not hesitate to contact support.
Aug 04, 2026 - 13:23 NZST
Investigating - OnDemand continues to have severely (unusable) degraded performance.
- RStudio will open but takes several seconds to run command
- Jupyter does eventually open +10 minutes and run reasonably normally.

Globus has intermittent outages and that fault is being investigated

Locked Files - there are several cases of locked files, this has a known solution and will be resolved as soon as possible.
Aug 04, 2026 - 08:49 NZST

About This Site

Bioeconomy Science Institute - AgResearch eRI status

Uptime over the past 90 days. View historical uptime.
Identity Broker Service Operational
90 days ago
100.0 % uptime
Today
Managed Storage Service Operational
90 days ago
97.68 % uptime
Today
General Flexi HPC Platform Operational
90 days ago
100.0 % uptime
Today
Network connectivity Operational
90 days ago
100.0 % uptime
Today
Compute cluster Operational
90 days ago
97.84 % uptime
Today
Login nodes Operational
90 days ago
97.86 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Aug 5, 2026

No incidents reported today.

Aug 4, 2026

Tamaki Data Centre is overheating, all nodes potentially being shutdown
Resolved - The primary incident has been resolved and a separate status incident is being created to reflect the current outages.
Aug 4, 08:39 NZST
Update - Further investigation has shown that OnDemand is severely degraded/non operational. This is being investigated and will be resolved as soon as possible.

Access to and use of Globus also remains intermittent.
Aug 3, 14:04 NZST
Update - All services that were resumed last week seem to be working as expected with all resources now available. All remaining systems have now been restarted.

Please note that REANNZ are unable to test systems where we don’t have access. Please don’t hesitate to contact support with any issue discovered.
Aug 3, 10:03 NZST
Monitoring - All but one hugemem nodes are up, and multiple users have had jobs start and finish as expected. Coldfront should now be operational. DMF is up and looks ok.

OnDemand is available but testing has shown degraded performance. We are continuing to work with our vendors to health check tape drives media.

Please note that REANNZ are unable to test systems where we don’t have access. Please don’t hesitate to contact support with any issue discovered.

We will resume providing updates on Monday.
Jul 31, 16:56 NZST
Update - We are looking into gpu and hugemem nodes, which continue to be unavailable at present.

Service restoration of Coldfront is ongoing but progressing well.

HPE engineers are currently helping investigating the health of DMF and tape media services.

We plan to provide our next update around 17:00.
Jul 31, 15:10 NZST
Update - We have brought OnDemand, login and all compute nodes back online and are currently monitoring their health status. Jobs can be submitted. GPFS/SMB access via S: and M: drives is available. All jobs that were running either failed or have been cancelled.

Service restoration efforts to Nix, GQuery and Coldfront are ongoing.

We expect to provide our next update around 15:00.
Jul 31, 12:57 NZST
Update - Access to eRI via ssh and OnDemand is still closed as we test the compute nodes. When the GPFS nodes are mounted, files will be available via Windows drive access.

We are bringing compute nodes back online but they will not be accessible until we have completed health checks on them. Nix, GQuery, Peaks and related VMs will all be brought online as part of that process.

Service restoration efforts are ongoing. We do not yet have enough information to provide an accurate recovery estimate, but we will continue to share regular updates. Unless the situation materially changes, we expect to provide our next update around 13:00.
Jul 31, 09:58 NZST
Update - Recovery of login services is progressing well, however this work is ongoing. The compute nodes are not online yet, batch jobs are not running.

We expect to provide an update at approximately 10:00am tomorrow.
Jul 30, 17:06 NZST
Update - HPC storage services appear to be in good health, although final checks are ongoing.

In parallel, we are working to get login services up and running however, at this stage we do not expect to have compute nodes online today.

Freezer and tape-based storage is expected to remain offline for longer, while we perform drive and media consistency checks. We will provide further information on this as we know more.

We plan to provide another update at approximately 17:00 today.
Jul 30, 14:59 NZST
Update - We have powered on essential hardware to begin bringing systems online. We are actively working on ramping up services starting with HPC storage and validating the status of storage. We are bringing up the minimum number of virtual instances to manage the rest of the fleet once storage is online.

We are keeping a very close eye on the environmental metrics at the data centre to prevent any additional fluctuations that might impact our tape media.

We plan to provide another update at approximately 15:00 today.
Jul 30, 12:56 NZST
Update - Tamaki Data Centre (TDC) has confirmed that the cooling system is back online and they are performing environmental checks etc.
We need to do a slow and controlled resumption of business as usual and perform health checks on the systems as we go.
We plan to provide another update about the progress and status by 13:00 today.
Jul 30, 10:04 NZST
Update - The current status is that we have the two working chillers at the data centre - TDC and a team is working on restoring the full chiller capacity.
Jul 30, 10:02 NZST
Identified - Tamaki Data Centre has identified issues affecting two of the site's three chillers, resulting in reduced cooling capacity.
Teams are actively investigating the root cause and monitoring site temperatures and cooling performance. Our priority is the safe restoration of cooling services and maintaining the integrity of the platform.
Once temperatures have stabilised and cooling capacity has been validated, we will commence our recovery and startup plan. Further updates will be provided as the investigation progresses, this will likely be from tomorrow morning
Jul 29, 17:22 NZST
Update - We are continuing to investigate this issue.
Jul 29, 17:02 NZST
Update - Cooling at the Tamaki Data Centre has not yet been restored. All services and hardware remains down. We currently don't have any ETA.
Jul 29, 16:24 NZST
Update - Contractors are onsite at the TDC but have not yet confirmed the chiller issue
Jul 29, 15:21 NZST
Update - The data centre has chiller issues - Tamaki Data Centre (TDC) and all nodes are in danger of overheating. Consequently all nodes are being shutdown to protect the hardware
Jul 29, 14:59 NZST
Investigating - We are currently investigating this issue.
Jul 29, 14:57 NZST

Aug 3, 2026

Aug 2, 2026

No incidents reported.

Aug 1, 2026

No incidents reported.

Jul 31, 2026

Jul 30, 2026

Jul 29, 2026

Restart NFS and SMB services, all access to M: and S: drives will be disrupted
Completed - Maintenance cancelled due to data centre issues
Jul 29, 17:01 NZST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 29, 17:00 NZST
Update - This maintenance is cancelled due to the TDC outage. File locks should be released anyway once we have recovered
Jul 29, 15:57 NZST
Scheduled - We will be clearing the current locked files on Wednesday July 29th at 1700hrs. All access to the M: and S: drives from Windows systems will be disrupted for a short time.
Jul 29, 11:35 NZST
Hypervisor a01co02 maintenance 29 July 2026 from 9AM-3PM.
Completed - The scheduled maintenance has been completed.
Jul 29, 15:00 NZST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 29, 09:00 NZST
Scheduled - Description: Hypervisor a01co02 will undergo maintenance to apply OS, firmware and OpenStack configuration updates.

Why: Ensure server and OS stability, security and up to date configuration is applied.

Impact: node will be unusable for 4-6 hours

Running VMs will be live-migrated off this host and there should be no noticeable impact.
Jul 28, 16:15 NZST

Jul 28, 2026

No incidents reported.

Jul 27, 2026

No incidents reported.

Jul 26, 2026

No incidents reported.

Jul 25, 2026

No incidents reported.

Jul 24, 2026

No incidents reported.

Jul 23, 2026

No incidents reported.

Jul 22, 2026

Restart NFS and SMB services, all access to M: and S: drives will be disrupted
Completed - The scheduled maintenance has been completed.
Jul 22, 10:11 NZST
Verifying - The SMB and NFS services have been restarted, all file locks have been cleared. Access to the M: and S: drives is now available
Jul 20, 17:09 NZST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 20, 17:00 NZST
Scheduled - We will be clearing the current locked files on Monday July 6th at 1700hrs. All access to the M: and S: drives from Windows systems will be disrupted for a short time.
Jul 17, 16:53 NZST
← Incident History Powered by Atlassian Statuspage