...continued.

Sept 23rd, 2014
From 12:15 AM CDT this morning, the Apache server had trouble connecting to MySQL for unknown reasons, causing some web sites (WordPress) to be inaccessible. This was fixed at about 8:45 AM CDT. Monitoring software is planned to prevent this from recurring.

Sept 13th, 2014
At 12:15 AM CDT this morning. the MySQL server crashed, leaving some of our web servers inoperative. This was firxed at about 9:12 AM CDT. Monitoring software is planned to prevent this from recurring.

Sept 7th, 2014
At 10:38 PM CDT, on Sept 7th the primary web server smile was upgraded with some minor system upgrades, and rebooted. smile was down less than 60 seconds.

Aug 7th, 2014
At 8:52 AM CDT, on Aug 7th the primary web server on smile stopped responding to network requests. The web server was restarted and restored to full service at 9:18 AM CDT.

Aug 5th, 2014
At 12:35 PM CDT, on Aug 5th our upstream connection experienced a brief interruption in service. Service was restored at 12:39 PM CDT.

June 23rd, 2014
The upstream connection troubles late last week have been resolved as temporary disruptions due to the thunderstorms and power outage in the area. The vendor says that "automatic" procedures corrected the problem(s), though no specific "fix" was performed.

June 21st, 2014
At 12:01 AM CDT this morning. the MySQL server crashed, leaving some of our web servers inoperative. This was firxed at about 3:40 PM CDT.

June 19th, 2014
The intermittent service problem on our upstream line "went away" at 11: 15 AM CDT this morning. Generally, problems that are not fixed, but "go away", come back. The tech coming out to analyze our upstream link is still scheduled for June 20th. Thank you for your patience.

June 18th, 2014
At 11:33 AM CDT, on June 18th our upstream connection experienced an interruption in service. Service was restored at 12:34 PM CDT. During this hour of interruption, all services were unavailable - web, email, shell and ftp. We apologize for the interruption in service.

June 4th, 2014
At 6:41 PM CDT, on June 4th our upstream connection experienced an interruption in service. Service was restored at 7:44 PM CDT. During this hour of interruption, all services were unavailable - web, email, shell and ftp.We apologize for the interruption in service.

May 23rd, 2014
At approximately 11:17 AM May 23rd the mail server Hope experienced a hang. The machine was rebooted at approximately 11:30 AM CDT and returned to full service. During this downtime, the web server smile, which depends on the mail server for DNS and NFS services, also stopped serving web pages. We apologize for the downtime.

May 11th, 2014
Between 9:12 PM May 11th and 1:05 AM CDT May 12th iCompute.com experienced some intermittent problems with our upstream connection. Thre were 14 gaps in service, totalling 21 minutes, the longest gap was about 3 1/2 minutes, and most of the gaps occurring after midnight CDT. The downtime corresponded to a local thunderstorm, and full service returned as the storm ended. We apologize for the outage.

May 10th, 2014
Between 12:17 PM and 1:41 PM CDT on May 10th mercy was upgraded to NetBSD 6.0.5, but the upgrade did not go smoothly as planned. Mercy was down for most of that time. Fortunately, almost all of the services that have been hosted on mercy have now moved to smile, so customer impact should have been minimal. We apologize for the outage.

March 26th, 2014
Between 4:45 PM and 5:09 PM CDT on March 26th a flurry of web site activity that may have been a DDOS (Distributed Denial of Service attack) caused many of our web sites to be inaccessible intermittenly. Settings changes on the web server and blocking attacking IP addresses corrected the problem. More tuning of the web server will be done to prevent future incidents. We apologize for the outage.

March 22nd, 2014
The maintenance period (system time) scheduled for the morning of March 23rd has been cancelled.

March 18th, 2014
At approximately 12:53 AM CDT this morning on March 18th the upstream link to iCompute was switched to a new router that should resolve the occasional intermittent connectivity problems that we have seen since early December. We will keep a close eye on this new equipment, but it shows every sign of being solid and reliable. Mercy and Smile (web servers) have also been upgraded to NetBSD 6.0.4. One administrative change is that ssh "passwordless" logins are slightly changed so that ".ssh/authorized_keys2" must have the "2" removed from the name to work.

March 17th, 2014
From 3:20 to 4:04 AM CDT on March 17th the upstream link to iCompute was brought down for maintenance to prepare for changes scheduled for March 23rd. Full service was restored at 4:04 AM CDT. Additional system time is still scheduled for next Sunday, March 23rd at 6:00 AM CDT to complete this work.

March 16th, 2014
The system time schedule for this morning was delayed. At 7:05 AM CDT on March 16th the upstream link to iCompute was brought down for maintenance. Full service was restored at 7:55 AM CDT. The maintenance was not successful. This maintenance is re-scheduled for next Sunday, March 23rd at 6:00 AM CDT.

March 13th, 2014
At 12:07 PM CDT on March 13th the web servers smile and mercy started failing some mysql transactions and operatons. This affects those customers who use MySQL as the "back end" of their web sites, such as Joomla or WordPress. A reboot of mercy was required to restore full function, but the actual problem has not yet been diagnosed. Full service was restored at 12:35 PM CDT. We are sorry for the interruption of service.

March 11th, 2014
This coming Sunday morning March 16th, between 6:00 AM CDT and 7:00 AM, we will change some router configurations to address intermittent instability in our upstream connection to the internet. The downtime should be limited to less than 15 minutes, but an hour has been set aside if the changes do not go smoothly.

February 13th, 2014
This evening, between 10:15 PM CST and 12:30 AM February 13th, some administrative errors caused intermittent instability in the web servers mercy and smile. The errors have been corrected, and should not recur.

February 12th, 2014
Our upstream connection to the internet has been plagued since Dec q2th with intermittent instability. This takes the shape of an inability to transfer data for 6-8 minutes. This has happened on average once every 4-5 days since early December. We are working with our vendor to track this down and fix it. In the last few days, the vendor has claimed to have found the problem. We'll see. We apologze for any disruption of service.

December 19th, 2013
Over the last 3 weeks, we have seen an unusual string of brief upstream outages. Normally, we do not report on one-time outages lasting less than 2 minutes, and even so, these are rare. That said, these should be entered in the record. On Sun Dec 8 23:20:33 CST, 6 minutes. On Mon Dec 9 14:31:54 CST, 10 minutes. On Thu Dec 12 16:58:35 CST, 7 minutes. On Sat Dec 14 15:04:07 CST, 11 minutes. On Sun Dec 15 22:53:58 CST, 11 minutes. We apologize for the interruptions, and are taking steps to identify and correct this instability.

December 3rd, 2013
At 1:51 PM CST our upstream connection experienced an interruption in service. Service was restored at 2:00 PM CST. We apologize for the interruption.

November 28th, 2013
The servers mailhost.icompute.com (email) and mercy.icompute.com (web hosting) were down this morning from 6:08 AM CST to 8:12 AM CST for scheduled maintenance. The maintenance work went smoothly, and performance should be improved on both of these servers.

Emergency Maintenance Announcement:

on 10/28/2013 - 6:00 AM to 9:00 AM CST - System time is being planned to upgrade the web server mercy.icompute.com and the disk on the mail server mailhost.icompute.com.. Mercy should be down about 30 minutes of this 3 hour window, and mailhost about 15, but occasional instability and/or unavilability of systems should be expected diring this time. We are sorry for any inconvenience this might cause.

November 24th, 2013
At 6:47 PM CST this evening, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 6:54 PM CST. During this interval, web sites hosted on mercy were unavailable.

November 11th, 2013
The MySQL server on mercy was not functioning correctly from 12:45 AM to 10:00 AM CST for unknown reasons. The failure was detected and corrected at 10:00 AM CST .

November 10th, 2013
The mail server hope (also mailhost.icompute.com and mailssl.icompute.com) was disabled from 8:30 PM to 8:40 PM CST to swap in new, faster hardware to improve mail server performance.

Maintenance Announcement:

on 10/20/2013 - 6:00 AM to 9:00 AM CDT - System time is being planned to upgrade software, and take other maintenance actions to improve security, upgrade software, and otherwise do things that might be disruptive to customers. We do not expect significant downtime during this 3 hour window, but instability and/or unavilability of systems should be expected diring this time.

October 23rd, 2013
The web server smile was disabled from 10:37 PM to 10:56 PM CDT to investigate and correct a management problem on the platform.

October 20th, 2013
The hastymail webmail service found at http://webmail.icompute.com has ben tweaked to fix a bug in displaying HTML messages.

October 10th, 2013
The hastymail webmail service found at http://webmail.icompute.com has been upgraded to version 2.1.1, with improvements in performance and stability and security improvements.

September 21st, 2013
Throughout the day, our web server mercy.icompute.com, has been occasionally and briefly unavailable to allow the movement of user data to network storage. This should improve reliability of the web server and all other services on mercy. Because all user data was moved, there is a possibility of administrator error. If anything is wrong or missing. Please contact us. As far as testing can determine, all systems and services are up and available as of 7:07 PM CDT.

September 21st, 2013
At 3:12 AM CDT Saturday, September 21st our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 4:00 AM CDT. During this interval, web sites hosted on both mercy and smile were unavailable.

September 1st, 2013
At 4:47 AM CDT Sunday, September 1st our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 7:14 AM CDT. During this interval, web sites hosted on both mercy and smile were unavailable.

August 19th, 2013
At 1:30 AM CDT Saturday, August 17th our MySQL server on mercy.icompute.com, unexpectedly shut down. The monitor script failed to restart it, or notify the system admin. A web visitor mention of one of the MySQL-driven databases being down triggered a correction, and MySQL was restarted at 6:09 PM on August 19th. During this interval, MySQL-driven web sites were not available. We apologize for the outage.

August 15th, 2013
At 4:42 AM CDT this morning, our web server mercy.icompute.com, stopped responding to network requests. mercy was rebooted and returned to service at 5:23 AM CDT. During this interval, web sites hosted on both mercy and smile were unavailable.

August 6th, 2013
At 1:14 PM CDT today, a heavy burst of web traffic to mercy.icompute.com, caused the web server to become partially non-responsive. By 1:25 PM CDT, the web server had been restored to full service, and the attack was being handled. (This was a Distributed Denial of Service Attack or DDoS from at least 600 uniqie IPs) The monitoring regime developed for smile will be applied to mercy to prevent this from happening in the future.

Click here for older news (August 2013 to June 2014)

Click here for older news (March 2013 to July 2013)

Click here for older news (May 2011 to February 2013)

Click here for older news (April 2009 to April 2011)

Click here for older news (July 2007 to March 2009)

Click here for older news (October 2006 to June 2007)

Click here for older news (April 2005 to September 2006)

Click here for older news (July 2003 to March, 2005)

Click here for older news (February 2003 to June 2003)

Click here for older news (October 2001 to December 2002)

Policies:

Any significant downtimes will be posted on this page.

If the downtime is unplanned, it will be posted here as soon as possible after the fact.

If the downtime is planned, it will be posted here at least one week in advance. Downtimes will be planned on Sunday morning between 6 and 8 AM whenver possible. Urgent, significant planned downtimes that cannot be scheduled a week in advance will still be posted here. If we deem it necessary, we will also send out e-mail to our customers with notification for the downtime as far in advance as possible.

-sysadmin