System Status

SFX Downtime this Sunday, April 23rd

The Production SFX service will be moved to a new operating system between 6AM and 10AM this Sunday, April 23rd.

I expect the service to be down for less than an hour during this maintenance window.

Brandon Gant
CARLI

VuFind 0.6 Outage this Sunday, April 9th

An updated server used behind-the-scenes with VuFind 0.6 and VuFind 3.x will be put in place between 6AM and 10AM on Sunday, April 9th. The outage should be less than 20 minutes if everything works as expected. If not, we may remove the new server and put the older one back in place.

Brandon Gant
CARLI

Dropping Network Connections

This afternoon I was doing maintenance on redundant security devices. The devices are supposed to preserve network connections when one device in the pair is restarted, but I have learned that we are dropping those connections.

I am holding off on further maintenance until later tonight. I will also investigate why connections are not being preserved. I apologize for kicking you off of our servers twice this afternoon.

Brandon Gant
CARLI

Patching and Rebooting Production Servers (Mar 5th)

This Sunday, March 5th between 6AM and 10AM, Production servers will be patched and rebooted. Most servers will be back online quickly, but Voyager and Oracle will be down for about a half-hour.

Brandon Gant
CARLI

UIUC Network Outage Today (Feb 24)

At around 11:42AM today (Friday, Feb 24) the UIUC campus lost their network connection between the Internet and their firewalls. The outage lasted around 11 minutes and they are currently working with their firewall vendor to determine the cause. This outage affected all UIUC campus services including CARLI servers.

Brandon Gant
CARLI

Network Firmware Upgrades this Sunday, Jan 15

This Sunday, January 15 starting at 6:30AM, campus networking staff will be upgrading the firmware on a large number of network devices. This includes the devices in our Production Data Center. Since we have redundant routers and switches, I am not expecting any downtime during this work but I wanted to make you aware of it just in case.

Brandon Gant
CARLI

Z39.50 to Voyager should work now

All of the Z39.50 problems should be fixed now and the z3950.carli.illinois.edu service should be back online.

If you are still having any issues with Z39.50, please contact support@carli.illinois.edu to let us know.

Thanks,
Brandon Gant
CARLI

Z39.50 Access Offline

We are investigating a problem with Z39.50 to Voyager and have turned off Z39.50 to stabilize the server.

Once we figure out the cause of the problem, we will turn Z39.50 access back on and see if the problem is resolved.

Brandon Gant
CARLI

Voyager Maintenance this Sunday, Dec 4th

This Sunday, December 4th between 6AM and 10AM, the Production voyager.carli.illinois.edu server will go offline briefly to be moved behind a new network security device. Downtime should be around a half-hour.

Brandon Gant
CARLI

Patching Servers This Sunday, October 30

This Sunday, October 30th between 6AM and 10AM we will patch and reboot operating systems. Most security updates can be applied without rebooting, but the changes we are making require a server restart. We are also going to take this opportunity to patch the Oracle Database software as well.

Most server outages will be brief since a restart takes less than a minute. However, the larger Voyager and Oracle servers could be down for around a half-hour.

Applying Server Updates (Oct 30)

This Sunday, October 30th between 6AM and 10AM we will patch and reboot operating systems. Most security updates can be applied without rebooting, but the changes we are making require a server restart. We are also going to take this opportunity to patch the Oracle Database software as well.

Most server outages will be brief since a restart takes less than a minute. However, the larger Voyager and Oracle servers could be down for around a half-hour.

Brandon Gant
CARLI

UIUC Firewall Maintenance

The UIUC Campus Firewalls will be upgraded which could cause network outages for anyone off-campus connecting to CARLI services.

Network staff have indicated that the outages should be brief during this window.

Brandon Gant
CARLI

VuFind and SFX Maintenance this Sunday, Oct 2nd

This Sunday, October 2nd between 6AM and 10AM, the VuFind (https://vufind.carli.illinois.edu) and SFX (http://sfx.carli.illinois.edu) services will go offline briefly while servers are moved behind a network security device.

To report a problem with those services after 10:00 AM on Sunday, please call 866-904-5843, press 0, and leave a message. Emails sent to support at carli.illinois.edu may not be answered until normal business hours.

Brandon Gant
CARLI

Voyager and VuFind Problems this Afternoon (Sept 14th)

I apologize for the Voyager and VuFind outage this afternoon. I was doing prep work for this Sunday morning and made one change too many. The servers did not go offline but they were unable to talk across the network to the Oracle databases.

The prep work has been removed and no further changes will be made until the servers are powered down this Sunday morning.

Brandon Gant
CARLI

Scheduled I-Share/Voyager Downtime on Sunday, September 18, from 6-10am

On Sunday, September 18, all I-Share/Voyager-related CARLI-supported systems will go offline for maintenance between 6:00 AM and 10:00 AM.
 
I-Share/Voyager services include: VuFind local catalogs, the I-Share union catalog, WebVoyage local catalogs, Z39.50, Voyager staff clients, and Voyager MS Access Reporting.  (Voyager Offline Backup Circ [https://www.carli.illinois.edu/products-services/i-share/circ/usingobc] may be used during the downtime, if needed.)
 

SFX Outage (June 24, 2016)

Last night (June 23) at 10:30PM I staged a firewall rule in a new security device for the Production sfx.carli.illinois.edu server. This rule was disabled but it appears that the existance of the rule blocked network access to the server. The rule was removed at 7:00AM this morning restoring service.

Throughout the rest of June and July, I will be migrating more services behind the security device and there will most likely be more interruptions to service. I will try to make sure they are short disruptions and I apologize for the outage.

Brandon Gant
CARLI

Scheduled Downtime Saturday May 21st, 4AM-Noon

All CARLI services will experience interruptions in service while the University of Illinois network staff performs significant maintenance on the network exit infrastructure.

I-Share/Voyager services affected include: VuFind local catalogs, the I-Share union catalog, WebVoyage local catalogs, Z39.50, Voyager staff clients, and Voyager MS Access Reporting.  (Voyager Offline Backup Circulation may be used during the downtime, if needed.)

Emergency Patching at 11PM Tonight (Feb 17, 2016)

There is a new glibc vulnerability (CVE-2015-7547) that affects most Linux systems. It may allow remote code execution on the servers, so I will patch and reboot all CARLI servers starting at 11PM tonight (Wednesday, Feb 17).

For most servers this will be a quick reboot. For Voyager, it will probably be down for about 20 minutes while the Oracle database server is rebooted.

Brandon Gant
CARLI

Voyager Outage (Dec 4th, 2015)

We have a batch job that has been failing this week and we have a ticket open with Ex Libris. We did not realize that the failed jobs were generating 4GB core dumps. At 2:10AM this morning the core dumps pushed the disk volume to 100% capacity. At approximately 8:50AM we identified and removed the core dump files to restore service.

This failure did not generate any automated alerts, so we will need to revise our monitoring.

Brandon Gant
CARLI

All CARLI Services scheduled downtime: Friday, October 23, 2015, 3am-5am

UIUC networking staff will perform maintenance this week affecting CARLI services. Please notify your library staff working Friday morning.

Pages