September 14th, 2006
At 23:45 CDT mailhost.icompute.com, the DNS and mail server was accidentally powered down. A reboot completed at 23:50 CDT. During the 5 minutes of downtime, the primary DNS for all domains served by iCompute.com was unavailable, and the mail server was not available. No mail should have been lost, because the mail server protocols should correctly handle the outage. We apologize for the interruption of service.

September 14th, 2006
At approximately 3:12 AM CDT mercy.icompute.com, our web, ftp, and ssh server, hung. (crashed) A reboot completed at 3:55 AM CDT. During the 43 minutes of downtime, the web server, ftp service, and ssh login were unavailable. We apologize for the interruption of service.

June 11th, 2006
A new "anti-SPAM" filter was added to the mail server at 10:30 AM CDT. This filter is called the "Strict Sender Domain" filter, and rejects incoming mail if the "MAIL FROM" specifier contains an invalid name/IP.

July 20st, 2006
Early in the morning of the 20th of July, the monitoring machine "wonder" lost power, apparently due to a faulty UPS. No services were affected.

June 11th, 2006
At 2:58 AM CDT mercy.icompute.com, our web, ftp, and ssh server, hung. (crashed) A reboot completed at 3:24 AM CDT. During the 24 minutes of downtime, the web server, ftp service, and ssh login were unavailable. We apologize for the interruption of service.

February 11th, 2006
Between 12:39 PM CST and 1:15 PM CST on Feb11th, our upstream internet connection was unreliable while wiring was being repaired. The purpose of the repair was to address some intermittent problems. The wiring work is now done, and the upstream connection is expected to be much more reliable. We apologize for the interruption of service.

February 5th, 2006
The maintenance period this morning was without incident, with a mercy reboot and an upgrade of the mail server to EIMS 3.2.8. All services were restored by 7 AM CST.

Maintenance Announcement:

Jan 29th, 2005 - System time on Feb 6th
A minor upgrade will be performed on the mail server - mailhost.icompute.com this comong Sunday morning at 6 AM CST February 5, 2006. We will also take the opportunity to do preventive maintenance on the web and ftp server "mercy", which will be rebooted. The upgrade and maintenance should take less than 30 minutes, but we have reserved a two hour time slot to handle unforseen problems. Please contact us if this outage is problematic.

January 28th, 2006
Over the last two weeks, we have had a problem with intermittent upstream connectivity. There have been over 20 outages, most of them lasting only a few seconds, but some lasting up to 10 minutes. This appears to be a problem with Qwest, but is not yet resolved. We are working on resolution, and apologize for the service interruptions.

January 17th, 2006
Our upstream connection went down unexpectdly at 4:03 PM and came back up again at 4:09 PM.

January 11th, 2006
Our service was interrupted briefly at approximately 9:55 AM to allow a Qwest technician to test our internal wiring. No problems were found.

January 5th, 2006
A Qwest DSL technician visited our site and verified that our external phone line is clear ("zero errors"). His testing caused a brief interruption of service shortly before 11:00 AM. Some question remains as to whether the wiring inside the house may have a problem. Another technician will visit next week.

January 3rd, 2006, late evening update.
Correction: The intermittent problems on our upstream connection were not caused by a failing router. We experienced two brief interruptions of our connection between 4 PM and 5 PM today. Qwest's trouble-shooter then believed he had tracked the problem to a heat-sink at the central office. However, after that was replaced, we again had an outage at approximately 9:30 PM. Qwest will be visiting our premises within the next 48 hours to check for problems in the wiring, both external and internal.

January 3rd, 2006
The intermittent problems on our upstream connection appear to be caused by a failing router on our end. The router has been swapped out, after some helpful Qwest diagnostic assistance. We will continue to monitor the situation, and apologize for any inconvenience.

January 3rd, 2006
Today we have experienced three interruptions of our upstream connection. The first occurred at 3:12 - 3:14 AM CST and the second at 7:54 - 8:05 AM CST, and the third was from 8:18 - 8:27 AM. We are checking into the cause.

Dec 4th, 2005
System maintenance this morning was successful. The mail server - mailhost.icompute.com is now running Mac OS X, and its two services - DNS and e-mail - are now upgraded to more modern versions of server software. DNS is now bind 9.3.1, and the e-mail server is the latest version of the mail server software (EIMS X 3.2.7). In addition, the email data is now on a "mirrored" RAID volume for data integrity. The result of these changes should be better reliability and performance for our customers, as well as improved administrative flexibility. Downtime for the morning was from 6:05 AM CST to 7:50 AM CST, and went according to plan.

Maintenance Announcement:

Nov 27th, 2005
The mail and DNS server - mailhost.icompute.com - will be upgraded to EIMS X 3.2.7, bind 9, a newer operating system (Mac OS X 10.2.8), and RAID disk for mail storage the morning of December 4th, at 6 AM CST. This upgrade will take no more than 3 hours, and may be done in as little as 30 minutes. The mail server will be unavailable during this upgrade.

Nov 4th, 2005
The problem reported yesterday with our upstream link continues. This is showing up for customers as delays, slow transfers, and lost packet traffic. The problem is apparently with the Qwest circuit that provides us with connectivity to the internet. Our upstream provider has a trouble ticket with Qwest, and reports that the failing "DS3 line card" is scheduled to be replaced the morning of Nov. 5th.

Nov 3rd, 2005
Upstream connectivity was lost between approximately 2:30 PM and 3 PM CST. This was caused by problems at our upstream provider. They are working on the problem, and there may be more brief interruptions of service later this afternoon. We apologize for the interruption of service.

Oct 26th, 2005
At some time in the morning of Oct. 26th, the first reports came in that something was wrong with the e-mail server. By noon, it was clear that something was not working correctly, and some diagnostic maintenance was performed. By evening, several customers were reporting e-mail downloads being "hung" or "very slow", and the symptoms pointed to a failing hard drive in the mail server.

At 5:40 PM CDT on the 26th, notice was sent out of emergency maintenance, and between 5:50 PM and 7:15 PM, the hard drive was replaced on hope.icompute.com (mailhost).

Files were dumped to backup and restored, but some of the errors reported hint of the possibility that some incoming or stored mail may have been lost. Nightly backups are performed on the mail server, so any mail stored on the server at midnight will be written to backup. However, mail that arrived on the 26th, was lost on the server, and had not been retrieved by any client may have been lost. We know of no specific instances of this, but it is possible.

If you believe you have lost e-mail, and want to track it down, please contact us at .

Aug 27th, 2005
The web and ftp server mercy.icompute.com hung today at 2:42 AM CDT. It was rebooted and brought back into full service at 3:09 AM CDT. We apologize for the downtime.

Aug 20th, 2005
The web and ftp server mercy.icompute.com hung today at 10:55 AM CDT. It was rebooted and brought back into full service at 11:14 AM CDT. We apologize for the downtime.

July 31st, 2005
System time was taken this morning and the following items were updated: EIMS upgraded to 3.2.7, Perl on mercy upgraded to 5.8.6, CPAN modules on mercy upgraded to latest, mysql removed from mercy, php 4.3.11 installed on mercy (no web server support, though). Downtime of the mail server was approximately 15 minutes, from 7:20 AM to 7:35 AM CDT.

July 22nd, 2005
On Sunday, July 31st 2005, we will be taking some system time between 6 and 8 AM CDT. We plan to upgrade the mail server to the latest EIMS software, and to add some additional email filtering.

July 22nd, 2005, 9:14 PM CDT
The mail server has been changed to require SMTP AUTH on mail submitted to the server on the SUBMIT port (port 587). This should be completely transparent to all users.

May 18th, 2005, 3:10 AM CDT
The main web server mercy.icompute.com crashed this morning at 2:28 AM CDT. The cause is unknown. It was restored to service at 2:48 AM CDT.

May 15th, 2005, 6:47 AM CDT
The work on the servers this morning is complete. Downtime was brief. The mail and DNS server - Hope - was down from 6:10 to 6:12 AM CDT. The web/ftp/shell server - Mercy - was down from 6:13 AM to 6:20 AM CDT. All servers and services are now running normally.

May 14th, 2005, 10:13 AM CDT
The system time tomorrow morning from 6-8 AM will include a brief interruption in service for the mail and DNS server of less than 5 minutes to upgrade the mail server to the latest version.

May 4th, 2005, 7:21 AM CST
The main web and ftp server, mercy.icompute.com will be unavailable between 6 AM and 8 AM CST on Sunday May 15th to make some minor hardware changes. Web and FTP sites will be unavailable, as will shell access to mercy. The mail server will continue to be available, as will DNS.

Click here for older news (July 2003 to March, 2005)

Click here for older news (February 2003 to June 2003)

Click here for older news (October 2001 to December 2002)

Policies:

Any significant downtimes will be posted on this page.

If the downtime is unplanned, it will be posted here as soon as possible after the fact.

If the downtime is planned, it will be posted here at least one week in advance. Downtimes will be planned on Sunday morning between 6 and 8 AM whenver possible. Urgent, significant planned downtimes that cannot be scheduled a week in advance will still be posted here. If we deem it necessary, we will also send out e-mail to our customers with notification for the downtime as far in advance as possible.

-sysadmin