...continued.

12/1/02 - At approximately 7:00 AM on 11/28/02, one of the "black hole" anti-spam list servers flagged mailhost.icompute.com as "relay tested". This caused a small amount of mail at select servers to bounce. (only those subscribers of njabl.org) This was noticed today (12/1) and corrective action taken. mailhost.icompute.com was removed from the blacklist in the mid-afternoon on 12/1. An announcement was sent out to detail the changes in e-mail security, including a new requirement to use SMTP secure authentication on all incoming SMTP traffic. We regret this added burden on our users, but is now necessary to maintain our good standing in the internet community. We apologize for the inconvenience.

There is good information about configuring e-mail clients for SMTP authentication here.

10/28/02 - 8/40 AM CST - At approximately 1:20 AM CST an apparent mail server attack with a port scanner and bad input disabled the mail server hope.icompute.com. Shortly after 8:00 am, the server was restarted and upgraded from version 3.1.3 to 3.1.4 (a recent patch). The mail server was back up and fully functional by 8:18 am.

10/27/02 - 12:39 AM CDT - At approximately 12:27 AM CDT the mail server hope.icompute.com crashed. The server was rebooted and restored to service by 12:39 AM.

8/17/02 - 11:55 PM CDT - To make the e-mail system easier to administer, monitor, and protect, we have decided to remove what are called the "secondary MX" records from the DNS database for the domains administered by iCompute.com. These changes should be completed within the next week. This change should have no impact on our users. Please contact us if you have questions about this.

7/31/02 - 9:31 PM CDT - Between 9:01 PM and 9:31 PM, our upstream link was unstable. Our logs tell us that we experienced a high rate of packet loss in this interval. It is unclear whether this appeared as "down" or appeared to only have poor performance. At 9:31 PM, the instability ended. We are investigating the cause, and apologize for the interruption of service.

6/5/02 - 10:40 AM CDT - Our mail server hung some time after 10:39 a.m. and remained out of service for somewhat less than an hour. No e-mail messages should have been lost. Web page hosting continued without incident.

5/26/02 - 6:05 AM CDT - Planned downtime to upgrade the mail server. The upgrade to EIMS 3.1.2 has gone smoothly and full service is restored by 6:35 AM.

4/25/02 - 11:10 PM CDT - Our upstream connection failed. iCompute connectivity to the internet was unavailable until 11:50 PM, when Qwest restored the circuit.

4/19/02 - 7:05 AM CDT - The web server required another reboot. The system was back up at 7:10 AM.

4/19/02 - 6:00 AM CDT - System reboot of mercy (web server) went smoothly. The web server was back on line by 6:10 AM.

Urgent Maintenance Announcement:

on 4/19/02 - 6:00 AM CDT - The Web server mercy.icompute.com will be re-booted at 6 AM the morning of 4/19/02 to correct some administrative problems. We anticipate the server will be down approximately 5 minutes.

4/18/02 - 7:19 PM CDT - Our upstream connection was disrupted by an unannounced maintenance period being taken by our upstream provider. Our upstream connection remained unavailable until 8:36 PM. All services were affected in this time period. We sincerely apologize for the downtime.

3/23/02 - 9:00 PM CST - The DNS server is upgraded to version 2.2.4 of QuickDNS Pro. This is intended to address recent stability problems on the mail server. The DNS server is unavailable during the upgrade for about 20 seconds.

3/17/02 - 12:50 PM CST - Our main router appears to have been the victim of some sort of DOS (Denial Of Service) attack. The router started reporting large numbers of incoming telnet connections at 12:41, and stopped routing packets at 12:50 PM. It is not clear that the router actually stopped functioning during this time, but the performance, at least, was severely degraded. We returned to full service at 13:00. Steps are being taken to prevent a recurrance.

3/15/02 - 12:26 AM CST - our mail and DNS servers crashed (again). Service was restored at 12:36 AM.

3/6/02 - 12:14 AM CST - our mail and DNS servers crashed. Service was restored at 12:19 AM. Secondary DNS and mail servers handled all incoming mail and DNS requests during the downtime. Anyone who tried to check mail during this time would have found the server unavailable. We are working on steps we can take to address the instability of this server.

2/16/02 - 12:03 AM CST - our mail and DNS servers crashed. Service was restored at 1:38 AM. Secondary DNS and mail servers handled all incoming mail and DNS requests during the downtime. Anyone who tried to check mail during this time would have found the server unavailable. We apologize for any inconvenience this may have caused.

2/8/02 - 10:40 AM CST - At 10:40 AM CST, and again at 4:30 PM, our upstream connection (to the internet) became unreliable. It appears that it was not a total outage, but very little data traffic was getting through. All internet services were affected. In both cases, it took roughly 20-25 minutes for our upstream provider to repair these problems.

1/23/02 - 7:12 PM CST - the web server "mercy" stopped responding to our monitor machine. Checkout revealed a bad ethernet cable, which was swapped out. Service was restored at about 7:22 PM.

1/12/02 - 9:39 PM CST - our upstream connection (to the internet) stopped responding. All internet services were affected. It took roughly 20 minutes to detect and diagnose this problem. Full service was restored by 10:07 PM CST.

1/8/02 - 4:59 PM CST - our upstream link to the internet started experiencing intermittent packet loss. From 4:59 PM to 6:24 PM, though access was possible, transfer rates were poor, and connections were unreliable. The worst period was between 5:17 PM and 6:14 PM, when packet loss rates were sometimes over 60%.

By 18:24, this problem, which was apparently caused by some routing problems in Qwest's network, was corrected.

12/02/01 - 12:15 AM CST - The mail and DNS server - hope crashed. The system was back up by 12:30 AM CST. We took the opportunity to add to the slightly stingy memory allotment in the system to try to improve the stability of the system.

11/10/01 - 12:07 AM CST - The mail and DNS server - hope crashed. The system was back up by 12:17 AM CST.

10/30/01 - 8:00 AM CST - Since the upgrade to NetBSD 1.5.2 on October 20th, systems have been running normally. No additional maintenance downtime is currently planned.

10/20/01- 6:00 AM to 8:00 AM CST - The main web server, mercy.icompute.com, will be upgraded from NetBSD 1.5 to NetBSD 1.5.2. This upgrade should be transparent to all users. Two hours have been alloted for the upgrade, but trial runs indicate that this should be more than sufficient. During the downtime the webserver, ftp, telnet access and dial-in will be unavailable. E-Mail and DNS will not be affected.

NetBSD 1.5.2 is a patch release containing a number of security and stability changes. Of particular interest to us are the patches to the networking code, and a number of responses to recent CERT security advisories. This release upgrades several of the standard network daemons to close security holes. Details can be obtained from http://www.netbsd.org

For those of you who wish to try out the new OS, a machine on our network - charm.icompute.com - has been set up running NetBSD 1.5.2. It has been restored to be an almost identical copy of mercy.icompute.com, including user data files, cron jobs, passwords, and (unfortunately) vi "recovery" files. This machine will be removed after the upgrade of mercy is complete.

Policies:

Any significant downtimes will be posted on this page.

If the downtime is unplanned, it will be posted here as soon as possible after the fact.

If the downtime is planned, it will be posted here at least one week in advance. Downtimes will be planned on Sunday morning between 6 and 8 AM whenver possible. Urgent, significant planned downtimes that cannot be scheduled a week in advance will still be posted here. If we deem it necessary, we will also send out e-mail to our customers with notification for the downtime as far in advance as possible.

-sysadmin