...continued.

Mar 29th, 2009
The upgrade of the EIMS mail server to version 3.3.9 this morning went smoothly. The mail server was down intermittently between 6:15 AM CDT and 6:55 AM CDT, and connectivity to the internet was disabled completely for about 10 minutes at approximately 6:40 AM CDT. All services were restored by 7 AM.

Mar 17th, 2009
This evening at approximately 7 PM CDT, a "nationwide issue" caused our upstream provider to lose partial connectivity to the internet. From about 7:00 PM CDT to 7:45 PM CDT, connectivity was available to some areas, but not to others. According to our logs, this outage became more severe at about 7:45 PM CDT. At that time, it appears that all connectivity to the internet was lost. At approximately, 8:25 PM, full connectivity was restored. We apologize for this outage.

Mar 2nd, 2009
There will be a Planned Maintenance Period on Sunday March 29th from 6:00 AM until 9:00 AM CDT. The mail server mailhost.icompute.com will be upgraded and other maintenance performed as needed. Actual downtime in this period should be brief, but some downtime should be expected.

Feb 11th, 2009
This evening, our upstream provider will be doing emergency maintenance on a router in Chicago. This will cause intermittent problems between 11 PM CST and 3 AM CST on 12 Feb 09. We are sorry for any inconvenience.

Jan 3rd, 2009
At approximately 10:35 PM CST, a technical problem with a routine restart of the webserver, combined with an administrative error caused a crash of our main web server "mercy". The web/shell/ftp server mercy was restored to service by 10:53 PM CST. We apologize the outage.

Jan 2nd, 2009
At approximately 1:05 PM CST, our upstream connection was restored to full capacity. We apologize for today's outage.

Jan 2nd, 2009
At approximately 9:41 AM CST this morning, our upstream provider started seeing majr packet loss on circuits in the Twin Cities area. Connectivity is not broken, but packet traffic is much degraded, to the point where data transfer is from very slow to impossible. The vendor is working on fixing the problem. I will post again when I have more news.

Oct 28th, 2008
There will be a Planned Maintenance Period on Sunday November 9th from 6:00 AM until 9:00 AM CDT. The memory of the web server mercy.icompute.com will be upgraded and other maintenance performed as needed.

Oct 22nd, 2008
Interruption of Service: At approximately 1:40 PM CDT our upstream connection went down unexpectedly and remained down for about 9 minutes. Service was fully restored by 1:49 pm.

Aug 24th, 2008
The planned upgrade of the e-mail and DNS server this morning was performed as planned. The server hope was unavailable only briefly between 7:00 AM and 7:05 AM CDT. All services are now restored, and the ISC Bind that we run for DNS services is now updated to protect against the "kaminsky vulnerability. For more information, see:

http://www.unixwiz.net/techtips/iguide-kaminsky-dns-vuln.html

Aug 17th, 2008
Today our DNS and mail server was down from 6:15 AM until 9:00 AM CDT, as we attempted to perform the planned upgrade. The procedure did not go well, so at 9:00 AM we halted the upgrade attempt and brought the temporary server back online. We will run with the temporary server for the coming week, and will attempt the upgrade again next Sunday. Therefore, please note:

There will be a Planned Maintenance Period on Sunday August 24th from 6:00 AM until 9:00 AM CDT. The DNS and mail server may be down during significant portions of that time period.

Aug 10th, 2008
This morning we performed step one of the two-part security upgrade announced in our recent email message. Our DNS and mail server was down from 7:51 AM CDT until 8:20 AM CDT as we moved it temporarily to a different computer. Note that as of this move, the DNS server is secure. Next Sunday, 8/17/08, we will again have a planned maintenance period between 6:00 AM and 9:00 AM CDT. At that time we will upgrade the normal server and put it back into service. These steps are being taken due to security holes that have been recently discovered in the version of ISC Bind that we run. For all you speed demons out there, enjoy yourselves this week because the temporary server is slightly faster than the normal one.

June 9, 2008
At 6:32 PM CDT tonight, our upstream provider experienced a serious outage affecting their customers in the Minneapolis/St. Paul area. From that time to 9:30 PM CDT, our upstream link was down, and all iCompute.com services were unavailable. As of 9:35 PM CDT, all services have been restored. We apologize for this outage.

May 16, 2008
We are scheduling a system maintenance period this coming Sunday on the 25th of May 2008 from 6 AM to 9 AM CST. Service may be intermittent during this period, as we are making some improvements to the mail server.

December 11, 2007
After noticing some intermittency on our line, Qwest sent a technician to our location to perform some diagnostics and preventive maintenance. His efforts caused us to lose connectivity from 9:46 AM CST to 10:13 AM CST. We don't anticipate any further problems, but Qwest will continue monitoring the line closely for the next 24 hours.

November 25, 2007
The planned system downtime this morning at 6 AM CST took longer than anticipated, but otherwise went without incident. The mail server (mailhost.icompute.com) was disabled from 6:10 AM until 7:10 AM, and then was rebooted twice between 7:10 and 7:56 AM. The mail server was upgraded to EIMS 3.3.5, and a securty update was applied.

November 17, 2007
We are scheduling a system maintenance period this coming Sunday on the 25th of November 2007 from 6 AM to 9 AM CST. Current plans include the application of security patches to the mail server (mailhost.icompute.com). Downtime of the mail server is expected to be brief, lasting only a few minutes while the security update is applied and the system rebooted. Other maintenance may be added to this plan and service may be intermittent during this period. Check back here for further informaiton.

August 28, 2007
At approximately 3: 10 AM CDT on Aug 28th, a severe storm knocked out power to our server room. Fortunately, our UPS systems took over and for the next 90 minutes, all was well. Unfortunately, one of the UPS systems failed at 4:26 AM and some of our systems went off-line. For the next 7 hours, the network and servers were maintained on UPS, and at about 11:40 AM, power was restord. We apologize to those customers affected by the outage.

August 14, 2007
At 11:35 PM on the 13th, CDT, our main router experienced a failure, probably caused by power transients caused by a thunderstorm. A hardware failure also occurred in our monitoring system at the same time, and consequently we remained unaware of the down time until this morning. A reboot of the router and replacement of the failed modem at 9:05 AM CDT today restored all services.

July 20, 2007
Due to a problem with our upstream provider, we experienced several interruptions of service today. We were down from 1:52 - 2:04 PM CDT, from 4:12 PM to 4:24 PM, then again 4:58 PM to 5:10 PM CDT. Additional incidents occurred from 5:59PM to 6:11 PM, 6:20-6:30 PM and 8:44 - 9:04 PM. We apologize for the outage and are taking appropriate steps to avoid outages in the future.

June 27th, 2007
Our T1 connection failed at 5 PM CDT today and remained down until 6:50 PM, then was down again between 7:13 PM and 7:21 PM this evening. A technician was dispatched, and work was done on the circuits,junctions and terminals on our upstream line. The technician found and fixed corrosion that is probably the cause of our recent intermittent problems.We apologize for the interruption in service.

June 26th, 2007
Our T1 connection failed at 3:31 pm today and remained down until 4:28 pm, just under an hour. No information is available on the cause of the outage. Our upstream provider is continuing to monitor the situation.

Maintenance Announcement:

June 16th, 2007
On June 24th between 6 AM and 6:30 AM CDT, the mail server mailhost.icompute.com will be upgraded to EIMS 3.3.1. This is a bugfix release and should result in downtime for the mail server of approximately 5 minutes.

June 7th, 2007
At about 5:10 PM CDT this afternoon, our upstream connection failed briefly. The outage was approximately 9 minutes. We apologize for the interruption.

May 8th, 2007
This morning, at 6:33 a.m., our T1 line ceased to function. Qwest responded quickly and sent a technician to diagnose the problem. He traced it to a flaw in the "F2" copper wire running from our premises to the cross connection between here and the central office. He ordered a repair crew to string new wire. They completed their work and had us up and running by 12:13 pm.

We know this outage caused significant inconvenience to many of our customers. We thank you for your patience.

April 1st, 2007
The changes to the mail server have been made. We now greylist all incoming hosts, and have a filter regimen in place that should suppress spam for those domains that do not use Postini. Over the next few weeks, we will be moving non-postini domains to use these new facilities.

Click here for older news (October 2006 to March 2007)

Click here for older news (April 2005 to September 2006)

Click here for older news (July 2003 to March, 2005)

Click here for older news (February 2003 to June 2003)

Click here for older news (October 2001 to December 2002)

Policies:

Any significant downtimes will be posted on this page.

If the downtime is unplanned, it will be posted here as soon as possible after the fact.

If the downtime is planned, it will be posted here at least one week in advance. Downtimes will be planned on Sunday morning between 6 and 8 AM whenver possible. Urgent, significant planned downtimes that cannot be scheduled a week in advance will still be posted here. If we deem it necessary, we will also send out e-mail to our customers with notification for the downtime as far in advance as possible.

-sysadmin