...continued.

July 31st, 2013
At 12:28 PM CDT today, as a result of a heavy burst of traffic, the Apache web server on smile.icompute.com, which hosts our most heavily accessed web sites, stopped responding to requests. Scripts to monitor and automatically restart apache failed to detect the hang, and so it took until 2:12 PM CDT to restart apache. We are sorry for tie interruption in service.

July 13th, 2013
At 8:14 AM CDT on Saturday July 13th, our upstream connection went down, due to an apparently major outage in the CenturyLink network. Service was restored at 11:37 AM CDT.

July 12th, 2013
At 4:10 AM CDT today. our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 5:12 AM CDT.

July 6th, 2013
At 6:14 AM CDT today. our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 6:40 AM CDT.

June 28th, 2013
At 12:08 PM CDT today. our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 12:25 PM CDT.

June 27rd, 2013
At 11:56 PM CDT this evening, our web server mercy.icompute.com was rebooted to install a test kernel that will help isolate and fix problems with stability. Mercy was returned to service at 12:15 AM CDT on June 28th.

June 27rd, 2013
At 4:21AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 4:57 AM CDT.

June 23rd, 2013
At 3:40 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 4:16 AM CDT, but crashed again at 4:21 AM CDT. Troubleshooting revealed an ongoing attack on the web server which appeared to be causing the trouble. The offending IP was blocked, and web server parameters were changed to avoid a recurrence.

June 21st, 2013
iCompute.com experienced a major service interruption on June 21st.

At 8:03 PM CDT June 21st, iCompute.com lost both power and upstream internet connectivity. Service was not restored until roughly 10 AM June 22nd, CDT. Our battery backups kept us running for several hours, but this power interruption was beyond our battery capacity, so even our Colo customers were shut down for part of the time of the interruption.

As of about 10:30 AM CDT June 22nd, all customer systems were back on line. The duration of the outage should not have been long enough to cause email to be "bounced", because most systems retry for 24 hours. No data was lost.

Please let us know of any additional concerns or problems.

We are sorry for the interruption of service.

See here for more information about the power problems in Minnesota.

June 19th, 2013
Our recent trouble with web statistics (http://stats.icompute.com) have been temporarily fixed by offloading the job of doing the analysys on another machine.

June 17th, 2013
At 3:35 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 4:12 AM CDT. During this time, the web server, ftp service, and shell access were unavailable.

June 16th, 2013
The upgrade of mercy transpired smoothly this morning, with mercy being down from 6:35 AM to 6:46 AM CDT. All systems are now running normally.

June 15th, 2013
At 8:07 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 8:23 AM CDT. During this time, the web server, ftp service, and shell access were unavailable.

June 13th, 2013
At 6:00 AM CDT on Sunday June 16th, we will be taking our main web and ftp server mercy.icompute.com off-line to complete a minor upgrade to NetBSD 6.0.2. This should take about 20 minutes, and the web server, ftp and shell access will be unavailable at that time.

May 29th, 2013
At 9:51 PM CDT on May 29th, we replaced our upstream router with an older model that the vendor believes will be more reliable, and upgraded the kernel on the main server mercy.icompute.com to NetBSD 6.0.2. There were two brief interruptions of service at 9:51 PM CDT and at 10:19 PM CDT, each of less than 2 minutes. These changes, and more that are planned, should result in fewer service interruptions going forward.

May 25th, 2013
At 7:00 AM CDT this afternoon, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 7:36 AM CDT. During this time, the web server, ftp service, and shell access were unavailable. We are sorry for the inconvenience. These crashes of our main server are unacceptable. We are working on stopping them.

May 23th, 2013
At 2:46 PM CDT this afternoon, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 2:59 PM CDT. During this time, the web server, ftp service, and shell access were unavailable. We are sorry for the inconvenience.

May 21st, 2013
At 4:09 AM CDT on Tuesday May 21st, our upstream connection went down, for unknown reasons. Service was restored at 4:36 AM CDT. During the downtime, all services were unavailable. No data or email was lost.

May 16th, 2013
At 6:51 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 7:27 AM CDT. During this time, the web server, ftp service, and shell access were unavailable. We are sorry for the inconvenience.

May 15th, 2013
@aol.com. is reporting that the problems with their new anti-spam system have been resolved. Please contact us if any more aol.com "bounces" are seen.

May 7th, 2013
We are seeing delivery problems with email going to the domain @aol.com. This is apparently caused by some problem with the new anti-spam system at AOL. Some information has been posted here: http://postmaster-blog.aol.com . A ticket has been opened with AOL, and the problem should be resolved quickly. In the mean time, messages appear to get through as long as they are not part of a mail list. If bounces are encountered, re-send the emails to each single addressee, one at a time.

May 7th, 2013
At 11:24 AM CDT on Tuesday May 7th, our upstream connection went down, for unknown reasons. Service was restored at 11:57 AM CDT. During the downtime, all services were unavailable. No data or email was lost.

May 6th, 2013
Due to administrative error, the MySQL server was not re-enabled after late night testing on May 5th. Monday morning, May 6th, at about 10:30 AM, a customer call triggered action to restart the service.

April 13th, 2013
At 6:14 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted at 6:47 AM CDT. Also at 8:37 AM CDT, mercy.icompute.com failed, with a reboot at 9:47 AM CDT. During these periods, the web server, ftp service, and shell access were unavailable. We are sorry for the inconvenience.

April 11th, 2013
At 8:35 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted at 8:47 AM CDT. During that period, the web server, ftp service, and shell access were unavailable. We are sorry for the inconvenience.

April 6th, 2013
At 6:37 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted at 7:20 AM CDT. During that period, the web server, ftp service, and shell access were unavailable. We are sorry for the inconvenience.

March 30th, 2013
At 3:12 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. Troubleshooting did not reveal the problem, and mercy was rebooted at 9:57 AM CDT. During that period, mercy was operating, but not responding to network requests. The web server, ftp service, and shell access were unavailable. Mercy is now operating properly. At the time of the reboot, the original NetBSD 6.0.1 was put back in service due to an additional problem being found in the "fixed" kernel. A kernel with a new fix for the security problem reported on the 26th (below) will be installed soon.

March 29th, 2013
At 6:09 AM CDT this morning, our web server mercy.icompute.com, crashed. mercy was rebooted at 6:40 AM CDT. During that period the web server, ftp service, and shell access were unavailable. Mercy is now operating properly. We are sorry for the inconvenience.

March 26th, 2013
At 11:10 PM CDT March 26th, a patched kernel was installed and mercy.icompute.com was rebooted. Downtime was less than 2 minutes. The security issue fixed was one of the quality of the ssh cryptography in connections to the server. More info can be seen here.

March 21st, 2013
At 12:20 AM CDT on Thursday March 21st, our upstream connection went down, for unknown reasons. Service was restored at 12:53 AM CDT. During the downtime, all services were unavailable. No data or email was lost.

March 2nd, 2013
At 12:11 AM CST this morning, our web server mercy.icompute.com, stopped responding to network requests. Troubleshooting did not reveal the problem, and mercy was rebooted at 12:40 AM CST. During that period, mercy was operating, but not responding to network requests. The web server, ftp service, and shell access were unavailable. Mercy is now operating properly. We are sorry for the inconvenience.

Click here for older news (May 2011 to February 2013)

Click here for older news (April 2009 to April 2011)

Click here for older news (July 2007 to March 2009)

Click here for older news (October 2006 to June 2007)

Click here for older news (April 2005 to September 2006)

Click here for older news (July 2003 to March, 2005)

Click here for older news (February 2003 to June 2003)

Click here for older news (October 2001 to December 2002)

Policies:

Any significant downtimes will be posted on this page.

If the downtime is unplanned, it will be posted here as soon as possible after the fact.

If the downtime is planned, it will be posted here at least one week in advance. Downtimes will be planned on Sunday morning between 6 and 8 AM whenver possible. Urgent, significant planned downtimes that cannot be scheduled a week in advance will still be posted here. If we deem it necessary, we will also send out e-mail to our customers with notification for the downtime as far in advance as possible.

-sysadmin