...continued.

February 16, 2013
Throughout the day, updates have been done to the perl installation on the web server mercy, which was seen to have issues with some of the perl scripts on web sites. This has now settled on a perl upgrade to version 5.16, which appears to be stableThe other upgrades done earlier in the day also appear to be stable.

February 16, 2013
At 9:35 AM CST AM this morning, our web server mercy.icompute.com, stopped responding to network requests. When mercy was rebooted, the web server refused to come up. Debugging efforts were unsuccessful, and at about 10:15 AM CST, the remaining upgrade packages that have been under test on charm were installed on mercy to fix the problem with the web server. This was successful, and the server was restored to full function at 10:45 AM CST. The new packages upgrade Apache to 2.2.23, PHP to 5.3.17, perl to 5.14.2 and MySQL to 5.1.65. We are sorry for the inconvenience.

February 12th, 2013
At 11:36 AM CST on Tuesday February 12th, our upstream connection went down, for unknown reasons. Service was restored at 11:44 AM CST, February 12th. During the downtime, all services were unavailable. No data or email was lost.

February 10th, 2013
The upgrade scheduled for this morning went smoothly. The main web server (mercy.icompute.com) was unavailable from 6 :08 AM to about 7:05 AM CST. We are now running NetBSD 6.0.1, but with the same software packages as before the upgrade. (Apache, PHP, Perl, etc) The packages will now be upgraded as testing and verification allows.

February 1st, 2013
An upgrade is planned for the main web server (mercy.icompute.com) on the February 10th, 2013. We will be upgrading the web server and supporting software to the latest stable, available packages. The web server Mercy.icompute.com will be unavailable from 6 AM CST to 7 AM CST, and possibly intermittently up until 9 AM CST. Please contact us if you have any questions or concerns.

January 30th, 2013
At about 1:30 AM CST this morning, our upstream router was replaced with another brand to fix problems of instability. This morning, the replacement modem was found to be blocking the ports used to connect to secure email (mailssl.icompute.com, ports 465, 993, and 995). The old modem was put back in service at 9:00:AM CST. Access to the mail server through SSL was not available from 1:30 AM to 9:00 AM CST today. We apologize for the inconvenience.

January 26th, 2013
Over the last few days, the transition from the old IP addresses and upstream internet connection to the new has been proceeding. All machines and services are now on the new address range. Throughout the week, there have been occasional brief intereuptions in some services as the transition was performed. On January 25th, there were two slightly longer interruptions, one from 5:12 PM CST to 5:52 PM CST where all services were unavailable, and another between midnight and 12:15 AM CST this morning, when the web server was not responding. The transition is now done, so these interruptions should stop. All servers will be closely monitored for the next couple of weeks to ensure that no new instabilities have been introduced. Thank you for your business, we apologize for the interruptions in service.

January 21st, 2013
On January 21st, as planned, a new upstream connection was installed and local configuration changes made. We did have some intermittent downtime between 10 PM and 2 AM CST, but otherwise the transition has been smooth. The plan remains to test this new line for the week of the 21st. If it meets all requirements, the old upstream will be disconnected. Our new IP range is 71.39.101.32/27. Please contact us if you have concerns about this transition.

January 20th, 2013
On Monday morning January 21st, a new upstream connection will be installed that is intended to replace the T1 that has been serving us. The new upstream connection will be over 3 times the speed of the old, and should also have good latency characteristics. The plan is to test this new line for the week of the 21st. If it meets all requirements, the old upstream will be disconnected. Our IP address range will change. Any users who "hard-code" IP addresses of our servers will have to adjust to the new range. Please contact us if you have concerns about this transition.

January 10th, 2013
Our upstream connection has been more stable, and the packet loss rate has fallen to normal levels since the maintenance incident on Jan 4th. We believe that the packet loss and performance problem has been fixed.

January 4th, 2013
At 1:03 AM CST on Friday January 4th, our upstream connection went down, due to vendor technical issues. This outage appears to have been caused by the efforts of the vendor to fix our packet loss problem. I have been told that some equipment has been swapped out. Service was restored at 1:45 AM CST, January 4th. During the downtime, all services were unavailable. No data or email was lost.

January 2nd, 2013
Our upstream connection continues to see intermittent packet loss. I'm seeing intermittent losses of around 3-5%. Our upstream vendor is reports that the problem is not specific to iCompute.com, but more widespread in their system. They are working to fix it. We will continue to monitor and post updates here.

December 31st, 2012
(11:00 AM CST) The investigation into packet loss and slow response has yielded some fixes in our upstream connection. The current packet loss is under 1%, which is acceptable. We consider the intermittent poor performance issue to be resolved, but will continue to monitor and address it as reuqired.

December 29th, 2012
At about 10:00 AM CST today, a customer reported slow response times in web site accesses. This has prompted an investigation, revealing elevated packet loss on our upstream connection (between 3 and 7% packet loss, intermittently.) This is being addressed. I will update this status as this issue progresses. No hard failures have been observed so far, only intermittent poor performance.

November 15th, 2012
At 10:15 PM CDT on Thursday November 15th, a combination of an unusually vigorous attack and/or software problems on mercy caused the web server to become unreliable. The web server was restored at 10:30 PM CDT, only to go down again 10 minutes later. At 11:40 PM CDT, mercy was rebooted, and normal operations resumed. We are sorry for the interruption of service.

October 13th, 2012
The problems encountered on October 11th delivering email to aol.com have cleared up, and nothing material was done on the iCompute.com side. It appears that the problem was caused by a problem on the aol.com mail servers.

October 11th, 2012
From roughly 5:30 AM CDT on Thursday October 11th, we experienced some significant problems delivering email to aol.com. It is unclear what the problem was, but email destined for aol.com was delayed because aol.com mail servers were refusing connections with a "temporary" error. (421) By approximately 5:00 PM CDT the errors from the aol.com servers ceased, and the queued mail was delivered. No data or email was lost.

September 11th, 2012
At 2:00 AM CDT on Tuesday September 11th, our upstream connection went down, due to vendor technical issues. Service was restored at 2:23 AM CDT, September 11th. During the downtime, all services were unavailable. No data or email was lost.

August 25st, 2012
At 10:44 AM CDT AM this morning, our web server mercy.icompute.com, stopped responding to network requests. Troubleshooting did not reveal the problem, and mercy was rebooted at 11:05 AM CDT. During that period, from 10:44 AM CDT to 11:05 AM CDT, mercy was operating, but not responding to network requests. The web server, ftp service, and shell access were unavailable. Mercy is now operating properly. We are sorry for the inconvenience.

June 21st, 2012
At 8:58 AM CDT AM this morning, our anti-spam seervice Postini, connected to our mailserver mailhost.icompute.com with an erroneous connection handshake that caused its internet address to be blacklisted. This was noticed by customers almost immediately, and at 9:56 AM CDT, the problem was corrected. In the hour that the problem persisted, 244 messages sent to 79 addresses at our domains were returned to sender. (bounced) Because some of this mail was spam, and sent to invalid addresses at that domain, not all of the bounced messages would have been delivered, but nevertheless a report was generated detailing the bounced mail was sent to a representative or administrator. The IP addresses involved have been "locked down" to prevent recurrence. We apologize for the interruption of service.

June 20th, 2012
From approximately 8:55 AM CDT to 9:20 AM CDT today, we were without electrical power. Our battery backups performed as inteded, and no services were affected.

April 29th, 2012
The memory upgrade of our web server mercy.icompute.com went smoothly. mercy was unavailable from 6:56 AM CDT to 7:18 AM CDT. No other services were affected.

April 19th, 2012
At 6:00 AM CDT on Sunday April 29th, our web server mercy.icompute.com will have its memory upgraded. This should involve downtime on the web server of about 20 minutes. Please contact us if this downtime will impose a hardship.

March 23rd, 2012
At 3:20 PM CDT on Friday March 23rd, our upstream connection went down, due to vendor technical issues. Service was not restored until 6:00 AM CDT, March 24th. During the downtime, all services were unavailable. Though no data or email was lost, email was held and delivered when the connection was restored as a required feature of standards-conforming email systems. We sincerely apologize for this lengthy interruption of service.

February 6th, 2012
From 9:57 AM CST this morning, our upstream connection has been unreliable. Most of the morning, the connection has been intermitten, mostly up, but down enough to be a problem to our users. The vendor has found a problem with the copper "pairs" that provide us service, and has repaired them, an operation that was completed at 2:20 PM CST. We apologize again for theinconvenience, and the interruption of service.

February 6th, 2012
At 9:57 AM CST this morning, our upstream connection went down, due to unidentified vendor technical issues. The vendor was contacted. Service was initially, and intermittently restored at 10:30 AM CST. Reliable service has not yet been restored. We are working with the vendor to restore reliable service. During the downtime, all services were unavailable. We apologize for the interruption of service.

January 4th, 2012

December 31st, 2011
At 10:45 PM CST December 31st, our electrical power was lost. We remained on battery power until power was restored at 12:13 AM January 1st. No services were affectted by this incident.

December 18th, 2011
At 2:41 AM CST this morning, our upstream connection went down, due to unidentified vendor technical issues. Service was restored at 5:00 AM CDT. During the downtime, all services were unavailable.

October 1st, 2011
At about 5:52 PM CDT this evening, the web server mercy, encountered a kernel panic. mercy was rebooted, and restored to service by 6:13 PM CDT. During the reboot, mercy was unavailable, with shell access, ftp and http access to the web server on mercy unavailable. email service was unaffected. We are sorry for the inconvenience.

August 31st, 2011
At about 9:45 PM CDT tonight, the web server mercy, started experiencing difficulty. Troubleshooting did not reveal any problems, but the web server continued to behave badly. The machine mercy was rebooted at 10:46 PM CDT to fix the problems and the reboot appears to have fixed the problem. During that hour between 9:45 PM CDT and 10:45 PM CDT, mercy was operating, but some network operations were not working properly. It was at best only partially operational. We are sorry for the inconvenience.

June 30th, 2011
At about 4:40 PM CDT today, our mail machine mailhost experienced a problem with the mail server software. The server software got "stuck" and used almost all the CPU power on that machine for about 25 minutes. This caused both the mail server and DNS services running on mailhost to give very poor response. The correction of the problem was delayed to gather debugging information for the mail server author. We apologize for the gap in good service.

Click here for older news (April 2009 to May 2011)

Click here for older news (July 2007 to March 2009)

Click here for older news (October 2006 to June 2007)

Click here for older news (April 2005 to September 2006)

Click here for older news (July 2003 to March, 2005)

Click here for older news (February 2003 to June 2003)

Click here for older news (October 2001 to December 2002)

Policies:

Any significant downtimes will be posted on this page.

If the downtime is unplanned, it will be posted here as soon as possible after the fact.

If the downtime is planned, it will be posted here at least one week in advance. Downtimes will be planned on Sunday morning between 6 and 8 AM whenver possible. Urgent, significant planned downtimes that cannot be scheduled a week in advance will still be posted here. If we deem it necessary, we will also send out e-mail to our customers with notification for the downtime as far in advance as possible.

-sysadmin