Downtime
Failing disk in caligula
caligula will be brought down sometime this evening for disk replacement. The subsequent downtime will take several minutes, and will bring down all websites hosted on caligula, as well as e-mail. However, no e-mail will be lost in the transition.
One of the disks in caligula’s RAID1 array is failing. Replacing it will actually only incur a few minutes downtime and with no data loss. However, the server’s performance will slow for a few hours after the downtime as the disk is re-added to the array.
More precise information will be added as it becomes available.
Update Jul 08, 2009 3:44 PM: Downtime is over. Failed disk has been replaced and the array is rebuilding.
Update Jul 09, 2009 11:22 AM: Array has finished rebuilding, and operations back to normal.
Network connectivity problems earlier this afternoon
Earlier this afternoon our datacenter was having internal network connectivity issues, preventing access to caligula for several hours. All issues have now been resolved and all services should be working as normal.
Errors trying to send e-mail via SMTP earlier this evening
You may have received errors trying to send e-mail through Rhombic Networks’ SMTP servers earlier this evening. This problem has been since rectified and all services are back online.
IMAP and SMTP login failures
For about an hour earlier this evening the authentication system for IMAP and SMTP connections was malfunctioning. You may have received a “Login failed” error during this time when trying to receive or send mail via Rhombic Networks servers. The problem is now fixed.
There should have been no mail loss associated with this event.
Database problems Saturday morning
Sometime this morning the MySQL server on caligula went down. It’s since been restarted and all services appear to be running normally.
Webmail down yesterday and this morning (Jan 23-24)
Samat -
Wednesday, January 24, 2007 - 15:52
Tags: Downtime
Both of our webmail clients were down yesterday (Jan 23) and this morning (Jan 24). The problem causing the outage has been found and rectified.
Apache problems on caligula; some modules disabled
There was a botched routine upgrade sometime earlier in the weekend, whose problems did not manifest till the Apache webserver on caligula was automatically restarted Sunday morning. Total downtime should have been only for a few hours Sunday morning; the server itself remained operational without any other problems.
The Apache problems have been temporarily resolved, but we had to disable some modules to get Apache working properly again.
If you encounter any problems, please send us a support ticket immediately.
caligula rebooted Sept 08; MySQL problems
caligula was rebooted Thursday, September 8 in response to a major SSL security bugfix. Rebooting was the only sure way to make sure all services dependent on SSL got restarted properly.
Across these reboots, MySQL had broken and was no longer starting automatically with the system. Downtime was slightly under 5 minutes till we noticed and corrected the problem. During this time any access to a database-powered site would have given an error about the inability to connect to our MySQL server.
All problems should now be resolved.
Contact form on main site broken
Samat -
Saturday, June 17, 2006 - 22:41
Tags: Downtime
The contact form of the main Rhombic Networks site is currently broken.
Setup as a temporary measure, if you go to that URL, you will be redirected to this site’s contact page.
Web mail, billing systems unavailable
Our web mail and billing systems were unavailable the last two days due to a small not-noticed, not-serious security issue.
The issue has been fixed, and all these systems are now available again.
