System Status

System Status

System Status

All Systems ActiveAll Systems Active: All Services are Online.

VuFind and SFX Maintenance this Sunday, Oct 2nd

Monday, September 26, 2016 - 10:38am

This Sunday, October 2nd between 6AM and 10AM, the VuFind (https://vufind.carli.illinois.edu) and SFX (http://sfx.carli.illinois.edu) services will go offline briefly while servers are moved behind a network security device.

To report a problem with those services after 10:00 AM on Sunday, please call 866-904-5843, press 0, and leave a message. Emails sent to support at carli.illinois.edu may not be answered until normal business hours.

Brandon Gant
CARLI

Voyager and VuFind Problems this Afternoon (Sept 14th)

Wednesday, September 14, 2016 - 5:32pm

I apologize for the Voyager and VuFind outage this afternoon. I was doing prep work for this Sunday morning and made one change too many. The servers did not go offline but they were unable to talk across the network to the Oracle databases.

The prep work has been removed and no further changes will be made until the servers are powered down this Sunday morning.

Brandon Gant
CARLI

Scheduled I-Share/Voyager Downtime on Sunday, September 18, from 6-10am

Monday, September 12, 2016 - 9:09am

On Sunday, September 18, all I-Share/Voyager-related CARLI-supported systems will go offline for maintenance between 6:00 AM and 10:00 AM.
 
I-Share/Voyager services include: VuFind local catalogs, the I-Share union catalog, WebVoyage local catalogs, Z39.50, Voyager staff clients, and Voyager MS Access Reporting.  (Voyager Offline Backup Circ [https://www.carli.illinois.edu/products-services/i-share/circ/usingobc] may be used during the downtime, if needed.)
 
This service interruption is necessary in order to perform operating systems maintenance on these servers.
 
To report a problem connecting to a CARLI-supported system after 10:00 AM on Sunday September 18, please call 866-904-5843 and follow the directions to page the on-call technician. To report a problem during regular business hours, please email support at carli.illinois.edu.
 

SFX Outage (June 24, 2016)

Friday, June 24, 2016 - 9:08am

Last night (June 23) at 10:30PM I staged a firewall rule in a new security device for the Production sfx.carli.illinois.edu server. This rule was disabled but it appears that the existance of the rule blocked network access to the server. The rule was removed at 7:00AM this morning restoring service.

Throughout the rest of June and July, I will be migrating more services behind the security device and there will most likely be more interruptions to service. I will try to make sure they are short disruptions and I apologize for the outage.

Brandon Gant
CARLI

Scheduled Downtime Saturday May 21st, 4AM-Noon

Tuesday, May 10, 2016 - 3:16pm

All CARLI services will experience interruptions in service while the University of Illinois network staff performs significant maintenance on the network exit infrastructure.

I-Share/Voyager services affected include: VuFind local catalogs, the I-Share union catalog, WebVoyage local catalogs, Z39.50, Voyager staff clients, and Voyager MS Access Reporting.  (Voyager Offline Backup Circulation may be used during the downtime, if needed.)

Other services include: CARLI Digital Collections (CONTENTdm), SFX, the CARLI website, support@carli.illinois.edu, ILDS website, CARLI email lists, etc.

To report a problem connecting to a CARLI-supported system after noon on Saturday, May 21, please call 866-904-5873 and follow the directions to page a technician. To report a problem during regular business hours, please call the number above or email support@carli.illinois.edu.

Emergency Patching at 11PM Tonight (Feb 17, 2016)

Wednesday, February 17, 2016 - 3:42pm

There is a new glibc vulnerability (CVE-2015-7547) that affects most Linux systems. It may allow remote code execution on the servers, so I will patch and reboot all CARLI servers starting at 11PM tonight (Wednesday, Feb 17).

For most servers this will be a quick reboot. For Voyager, it will probably be down for about 20 minutes while the Oracle database server is rebooted.

Brandon Gant
CARLI

Voyager Outage (Dec 4th, 2015)

Friday, December 4, 2015 - 9:04am

We have a batch job that has been failing this week and we have a ticket open with Ex Libris. We did not realize that the failed jobs were generating 4GB core dumps. At 2:10AM this morning the core dumps pushed the disk volume to 100% capacity. At approximately 8:50AM we identified and removed the core dump files to restore service.

This failure did not generate any automated alerts, so we will need to revise our monitoring.

Brandon Gant
CARLI

All CARLI Services scheduled downtime: Friday, October 23, 2015, 3am-5am

Wednesday, October 21, 2015 - 10:19am

UIUC networking staff will perform maintenance this week affecting CARLI services. Please notify your library staff working Friday morning.

  • Friday 10/23/2015, between 3AM and 5AM, all CARLI services including Voyager, I-Share, VuFind, SFX, CONTENTdm, and the CARLI website will be unavailable due to UIUC firewall maintenance.
    • This outage is expected to last approximately 15 minutes, but it could occur at any time during the scheduled hours.
    • UIUC patrons and staff may be unaffected.

Since this outage will affect the availability of the CARLI website, you will be unable to review the System Status updates on the CARLI homepage until the outages are resolved.

If the outages extend beyond the scheduled times, you can contact CARLI by telephone at 866-904-5843 or by email at and we will pass the outage information along to the UIUC networking staff.
 
Please send any questions about these upcoming outages to .

CARLI website scheduled downtime: Wednesday, October 21, 2015, 10pm-midnight

Wednesday, October 21, 2015 - 10:13am

UIUC networking staff will perform maintenance this week affecting CARLI services. Please notify your library staff working Wednesday night and Friday morning.

  • Tonight, Wednesday 10/21/2015, between 10pm and midnight, the CARLI website will be unavailable due to UIUC network maintenance.
     
  • Friday 10/23/2015, between 3AM and 5AM, all CARLI services including Voyager, I-Share, VuFind, SFX, CONTENTdm, and the CARLI website will be unavailable due to UIUC firewall maintenance.
    • This outage is expected to last approximately 15 minutes, but it could occur at any time during the scheduled hours.
    • UIUC patrons and staff may be unaffected.

Since both outages will affect the availability of the CARLI website, you will be unable to review the System Status updates on the CARLI homepage until the outages are resolved.

If the outages extend beyond the scheduled times, you can contact CARLI by telephone at 866-904-5843 or by email at and we will pass the outage information along to the UIUC networking staff.
 
Please send any questions about these upcoming outages to .  

 

Network Issues Friday (10/02) and Saturday (10/03)

Monday, October 5, 2015 - 4:41pm

The CARLI servers are in a Data Center on the University of Illinois at Urbana-Champaign (UIUC) campus. The UIUC campus firewalls automatically download the latest intrusion detection signatures and one of those signatures from the vendor was bad. At 9AM Friday the firewalls crashed and automatically restarted. A firewall restart drops all connections to our Voyager server (it probably has no effect on web traffic). At 3PM Friday, the firewalls crashed and restarted again, but now they were in a bad state that also slowed down network performance.

The poor network performance continued through Saturday evening until UIUC network staff and the firewall vendor identified the issue and removed the bad signature. Everything appears to be working normally again at this time.

Brandon Gant
CARLI

Pages