System Status

All Systems ActiveAll Systems Active: All Services are Online

Voyager Outage (Dec 4th, 2015)

Friday, December 4, 2015 - 9:04am

We have a batch job that has been failing this week and we have a ticket open with Ex Libris. We did not realize that the failed jobs were generating 4GB core dumps. At 2:10AM this morning the core dumps pushed the disk volume to 100% capacity. At approximately 8:50AM we identified and removed the core dump files to restore service.

This failure did not generate any automated alerts, so we will need to revise our monitoring.

Brandon Gant
CARLI

All CARLI Services scheduled downtime: Friday, October 23, 2015, 3am-5am

Wednesday, October 21, 2015 - 10:19am

UIUC networking staff will perform maintenance this week affecting CARLI services. Please notify your library staff working Friday morning.

  • Friday 10/23/2015, between 3AM and 5AM, all CARLI services including Voyager, I-Share, VuFind, SFX, CONTENTdm, and the CARLI website will be unavailable due to UIUC firewall maintenance.
    • This outage is expected to last approximately 15 minutes, but it could occur at any time during the scheduled hours.
    • UIUC patrons and staff may be unaffected.

Since this outage will affect the availability of the CARLI website, you will be unable to review the System Status updates on the CARLI homepage until the outages are resolved.

If the outages extend beyond the scheduled times, you can contact CARLI by telephone at 866-904-5843 or by email at and we will pass the outage information along to the UIUC networking staff.
 
Please send any questions about these upcoming outages to .

CARLI website scheduled downtime: Wednesday, October 21, 2015, 10pm-midnight

Wednesday, October 21, 2015 - 10:13am

UIUC networking staff will perform maintenance this week affecting CARLI services. Please notify your library staff working Wednesday night and Friday morning.

  • Tonight, Wednesday 10/21/2015, between 10pm and midnight, the CARLI website will be unavailable due to UIUC network maintenance.
     
  • Friday 10/23/2015, between 3AM and 5AM, all CARLI services including Voyager, I-Share, VuFind, SFX, CONTENTdm, and the CARLI website will be unavailable due to UIUC firewall maintenance.
    • This outage is expected to last approximately 15 minutes, but it could occur at any time during the scheduled hours.
    • UIUC patrons and staff may be unaffected.

Since both outages will affect the availability of the CARLI website, you will be unable to review the System Status updates on the CARLI homepage until the outages are resolved.

If the outages extend beyond the scheduled times, you can contact CARLI by telephone at 866-904-5843 or by email at and we will pass the outage information along to the UIUC networking staff.
 
Please send any questions about these upcoming outages to .  

 

Network Issues Friday (10/02) and Saturday (10/03)

Monday, October 5, 2015 - 4:41pm

The CARLI servers are in a Data Center on the University of Illinois at Urbana-Champaign (UIUC) campus. The UIUC campus firewalls automatically download the latest intrusion detection signatures and one of those signatures from the vendor was bad. At 9AM Friday the firewalls crashed and automatically restarted. A firewall restart drops all connections to our Voyager server (it probably has no effect on web traffic). At 3PM Friday, the firewalls crashed and restarted again, but now they were in a bad state that also slowed down network performance.

The poor network performance continued through Saturday evening until UIUC network staff and the firewall vendor identified the issue and removed the bad signature. Everything appears to be working normally again at this time.

Brandon Gant
CARLI

SFX Outage (Sept 11)

Friday, September 11, 2015 - 11:30pm

The Production SFX server (sfx.carli.illinois.edu) went offline at 10:41PM tonight. CARLI staff have taken the server down to apply software updates and expect service to be restored by midnight.

Our normal outage window is Midnight to 10AM on Sunday mornings with prior notification to our customers. This outage was not pre-approved, we will work with staff to prevent these type of outages in the future, and I apologize for this service interruption.

Brandon Gant
CARLI

VuFind Outage this Morning (July 31)

Friday, July 31, 2015 - 2:52pm

The vufind.carli.illinois.edu service stopped responding this morning at around 10:30AM. The Apache web server was restarted and the service was back online at 12:10AM. I apologize for the delay in getting the system online, but a few staff are on vacation today, so troubleshooting took longer than normal.

This is the third time Apache has stopped responding on this server since we upgraded to Voyager 9.1.1. We saw this time that certain Apache processes are not being released properly and are accumulating over time. This eventually exhausts the number of web processes available on the system causing it to ignore new requests.

We don't know exactly what is causing the processes to accumulate, but we know for certain that restarting Apache clears them out of the system. To prevent future outages, we have configured the server to restart the web server every day. The restart is quick and should not disrupt user searches.

Brandon Gant
CARLI

Voyager Maintenance this Sunday (July 19th)

Thursday, July 16, 2015 - 2:22pm

Starting at 12:01AM this Sunday, July 19th the Voyager and Oracle servers will be brought down so that their data can be transferred to a different storage array. The data transfer should take at least 10 hours. I will send an update before 10AM Sunday if it looks like the transfer is taking longer than anticipated.

To avoid confusion, we will also take VuFind offline while Voyager is offline.

Brandon Gant
CARLI

Voyager Outage at 6:36AM (July 10th)

Friday, July 10, 2015 - 11:14am

The Production Voyager server stopped responding at 6:36AM this morning. By 6:57AM, the VMware operating system decided that the server was really offline and initiated a restart. Services were back online by 7:05AM.

There is no indication that Voyager or the Linux operating system had any problem at that time. At 6:31AM, VMware automatically migrated some servers over to the same physical server that Production Voyager is on to rebalance the load. It looks like the Voyager virtual server lost access to the physcial CPU's, so this migration process either caused or contributed to the crash.

Since we have spare capacity and are in no danger of overloading the servers, I have configured the automated rebalance to be a more manual process.

Brandon Gant
CARLI

Sunday Server Patching (July 12th)

Wednesday, July 8, 2015 - 3:00pm

This Sunday morning starting at 12:01AM, Production servers will be patched and rebooted. All patching should be completed before 10AM Sunday, July 12th.

Voyager will be down from approximately 12:01AM to 12:30AM since the Oracle server needs to be patched and rebooted while Voyager is offline.

Brandon Gant
CARLI

VuFind web server outages

Tuesday, July 7, 2015 - 1:09pm

There have been two VuFind outages that appear to be caused by the Apache web server going into an odd state. The first outage was Friday, June 26th at 9PM and the second was at around 5AM yesterday (Monday, July 6th). In both cases, no obvious cause was found and the issue was corrected by restarting Apache.

Please continue to contact if you notice that VuFind has stopped responding or has slowed down dramatically. If the problem persists and no cause has been identified, we may need to implement a weekly or nightly restart of the application before Fall Semester start.

Brandon Gant
CARLI

Pages