Dear Hummingbird Users,
Please be advised that we are conducting a short maintenance cycle on Wednesday September 20, 2023 from 6:00am to 7:00am.
During this window, we will be rebooting the cluster login node. At that time, you will not be able to login to the cluster. In-flight jobs will continue, but pending jobs may be disrupted. If you submitted a job before the maintenance window and it was in a pending status, please check to make sure the job properly launched after the maintenance window is announced to be closed. You can still log into hb-feeder to access files and move data, but please log in to that machine directly (not through the login node). Any “screen” or “tmux” (or similar) sessions should be closed manually before the maintenance window begins. Sessions left open will be terminated automatically on reboot.
Please feel free to email hummingbird@ucsc.edu if you have any questions or concerns regarding this maintenance window.
SAFEGUARD AGAINST MISSING DATA
COPY versus MOVE
It has come to our attention that some users who are moving large chunks of data between locations on Hummingbird (e.g. from one directory to another) have unexpectedly seen the data disappear from their directories after the move has completed. This behavior seems to be associated only with moving TBs of data, at least more than just hundreds of gigabytes, but not smaller chunks of data.
The recommended solution is to COPY data (using the command rsync or cp) and NOT move the data (using the command mv). Once you have successfully copied the data, please check to see that the metadata is the same (e.g. the file size in bits), before you remove the old data from their original location.
If you observe that you have used MOVE and data are missing, we have no way to recover the lost data. This is a reminder that you should always have a copy of the data off Hummingbird for anything critical. We recognize that it’s not practical to make copies of everything on systems other than Hummingbird, but for final results and critical, non-intermediate data, this is good practice.
We will continue to investigate this problematic behavior, but your assistance by using COPY for moving large files or directories will help ensure that your data remain intact and available to you.
UC Santa Cruz Research Computing
