...continued.

March 24nd, 2005, 7:21 AM CST
The main web server mercy.icompute.com crashed at 2:38 AM March 23rd. (yesterday morning) The cause is unknown. The server was returned to service by 3:05 AM CST.

March 22nd, 2005, 8:44 AM CST
Due to administrative error, our upstream connection was disabled this morning between 5:17 AM and 8:34 AM. All services were unavailable during this time. We apologize for any inconvenience this may have caused.

March 6th, 2005, 6:15 PM CST
The upgrade of the main web server went fairly smoothly this morning. The actual upgrade and downtime was between rouchly 6:40 AM and 7:40 AM CST. The web server was up and running again by 8:15 AM. There were two incidents this morning after the upgrade that caused an interruption in service. at 10:20 AM, the web server stopped responding briefly due to administrative error, and at 11:15 AM CST, the server crashed and was unavailable for approx. 15 minutes. Since 11:30 AM CST, the "new" mercy has been running smoothly.

There are a number of differences in the new server. The biggest change is the hardware, which is much faster, has more memory, and over ten times the storage space (360 GBytes). We have also upgraded the software on the server, including an upgrade of the OS to NetBSD 1.6.2, web server to Apache 2.0.53, a new password encryption algorithm, removal of telnet service (in favor of ssh), and upgrades of several other system services.

March 6th, 2005
The main web and ftp server, mercy.icompute.com will be unavailable from 6 AM to 9 AM CST on March 6th. The server will be replaced during this time. Web and FTP sites will be unavailable, as will shell access to mercy. The mail server will continue to be available, as will DNS.

Nov 10th, 2004, 7:47 PM
Due to administrator error, the web server was stopped at 7:47 PM this evening, CST. It was restarted at 8 PM. We apologize for the outage.

Oct 19th, 2004, 12;51 AM
The mail server - mailhost.icompute.com crashed at 12:21 AM CDT this morning. It was restored to service at 12:30 AM CDT.

Oct 6th, 2004, 2:14 PM
Our router crashed at 2:14 this afternoon. It was restarted within a few minutes. During the time it was down, all services were unavailable.

Sept 20th, 2004, 11 PM
At 10 PM CDT, the webserver was rebooted to clear up a problem with the ethernet port. About 90 seconds later, it crashed. A second reboot, (which took about 10 minutes to "fsck" the disks) brought the machine back into service. We apologize for any inconvenience.

Sept 20th, 2004, noon
The webserver failed to start up at reboot at 7:50AM CDT. Our web-hosting services remained down until the problem was noticed and corrected at 11:46AM CDT.

Sept 20th, 2004
At 7:35AM CDT today, mercy.icompute.com crashed. It was restored to service at 7:50 AM CDT.

Aug 17th, 2004
The NetBSD organization has announced a security vulnerability in ftpd that could affect the iCompute ftp server. You can read about it here. As a temporary measure, we have implemented a workaround: adding the "-r" option to ftpd(8). It is possible that this will interfere with some ftp accesses. If you have problems with access to mercy, please contact us.

July 18th, 2004
The scripts that generate the statistics pages at http://stats.icompute.com failed the night of the 16th. The scripts are being repaired. The statistics pages will be unstable and/or missing until about 1 PM CDT today (July 18th).

July 4th, 2004
The maintenance time planned for 6-8 AM CDT was successful, with a new version of the mail server installed (EIMS 3.2.5) An attempt to upgrade the memory in mercy failed, apparently due to bad memory parts. mercy.icompute.com was down from 6:01 AM CDT to 7:38 AM CDT. mailhost was down from 7:40 AM CDT to 7:49 AM CDT.

July 3rd, 2004
We have decided to take system time on mercy as well as the web server. In addition to mailhost, mercy.icompute.com will also be down some time between 6 AM and 8 AM CST on July 4th.

June 25th, 2004
On Sunday July 4th from 6 AM to 8 AM CST, we will be taking some system time to upgrade the mail server on "mailhost" to the latest maintenance patch level. (EIMS 3.2.5) We are reserving two hours of system time to allow for other maintenance items that may be required on the servers or network equipment. Though actual downtime should be minimal, all services (web, ftp, e-mail, telnet) may be unreliable during this 2-hour window. Please contact us if this downtime will cause you hardship.

March 27,2004
At 4:19 PM CST, today an operator error caused the web server mercy to reboot. Mercy was down approxamately 90 seconds. We apologize for the disruption of service.

March 11,2004
At 2:28 PM CST, today our upstream provider, Agiliti Inc., experienced an equipment failure which disrupted our connection. They replaced the failed equipment and our service was restored by 3:54 PM. During the time that the connection was down, our web server and email server were unable to communicate with the outside world. We apologize for the inconvenience this may have caused. Please contact us if you have concerns about this incident.

February 27th, 2004
At 3:26 AM CST this morning, the mail server reset its clock back 3 hours and 20 minutes. This behavior was apparently caused by a problem with the clock on the main server, mercy, which is also the main NTP server for iCompute.com. At about 9:30 AM, this was detected and corrected. Date and time stamps on events on the web and mail servers are likely to be inaccurate in this time period.

Maintenance Announcement:

February 29th, 2004
On Sunday February 29th from 6 AM to 8 AM CST, we will be taking some system time to upgrade the mail server on "mailhost" to the latest maintenance patch level. (EIMS 3.2.3) We are reserving two hours of system time to allow for other maintenance items that may be required on the servers or network equipment. Though actual downtime should be minimal, all services (web, ftp, e-mail, telnet) may be unreliable during this 2-hour window. Please contact us if this downtime will cause you hardship.

December 10th, 2003
At 5:35 AM CST, our upstream connection failed. At 5:44 AM, the problem was rectified. iCompute was unable to connect to the internet for these 10 minutes. The cause of this has not been determined.

September 21st, 2003
The mail server was upgraded to EIMS 3.2.2. Due to technical problems with the upgraded OpenSSH, and and the emergence of no credible evidence of any actual security threat from the OpenSSH vulnerability, the OpenSSH upgrade has been postponed.

Maintenance Announcement:

September 19th, 2003
On Sunday September 21st from 6 AM to 7 AM CDT, we will be taking some system time to upgrade OpenSSH on the main web server "mercy" to address recent security alerts. We will also upgrade the mail server on "mailhost" to the latest maintenance patch level. Each of these systems will be rebooted, and will therefore be briefly unavailable during this time.

September 18th, 2003
The recent security announcement e-mail was based on current information, and that information continues to emerge and be clarified. Fortunately, the vulnerability that was found in OpenSSH (http://www.cert.org/advisories/CA-2003-24.html) appears not to be as serious as feared. In particular, it appears not to be a major threat at iCompte.com. System time is still planned this Sunday to install a patched version of OpenSSH.

September 4th, 2003
Between 11:15 and 11:30 PM CDT last night, administrator error caused the mail server to have to be rebooted. The problem was caused by an attempt to manually manipulate a very large (almost 2 GByte) inbox. The mail server was unavailable for approximately 10 minutes during this time period. We apologize for the outage.

August 10th, 2003 9:00 AM CDT
Between 6:00 AM and 6:30 AM CDT today, we performed maintenance on the web server, mail server, and upstream router. The work went smoothly, and all services were restored by 6:30 AM .

Maintenance Announcement:

July 31st, 2003 9:15 PM CDT
On August 10th, from 6 AM to 8 AM CDT, we will be taking some maintenance time for the iCompute computer systems and network. Plans currently include testing a backup router, upgrading the mail server software, and rebooting the web server (mercy.icompute.com). Though actual downtime should be minimal, all services (web, ftp, e-mail, telnet) will be unreliable during this 2-hour window. Please contact us if this downtime will cause you hardship.

Click here for older news (February 2003 to June 2003)

Click here for older news (October 2001 to December 2002)

Policies:

Any significant downtimes will be posted on this page.

If the downtime is unplanned, it will be posted here as soon as possible after the fact.

If the downtime is planned, it will be posted here at least one week in advance. Downtimes will be planned on Sunday morning between 6 and 8 AM whenver possible. Urgent, significant planned downtimes that cannot be scheduled a week in advance will still be posted here. If we deem it necessary, we will also send out e-mail to our customers with notification for the downtime as far in advance as possible.

-sysadmin