System Status

System Status RSS Feed    System Status

All Systems ActiveAll Systems Active: All Services are Online

UIUC Database Outage April 15

Tuesday, April 15, 2014 - 4:28pm

At 10:51AM this morning, our Database Administrator changed the Oracle password for the University of Illinois at Urbana-Champaign (UIUC) database account to run some tests. He thought he was logged into the Oracle Test server, but he was actually logged into the Production Oracle server. This caused Voyager client errors, forced UIUC circulation clients into "offline circ" mode, displayed "The catalog is not available" message in UIUC's WebVoyage instance, and blocked UIUC's VuFind access. The problem was corrected and UIUC Voyager services were brought back online at 11:12AM and VuFind at 11:29AM.

I apologize for this outage. At our next IT staff meeting we will discuss ways to prevent this type of error from happening again.

Brandon Gant
CARLI

Oracle security changes caused Voyager and VuFind problems

Monday, April 14, 2014 - 4:08pm

At 9:30AM yesterday morning (Sunday, April 13th) we made a change to Production Oracle to enhance our database security. This change caused problems in VuFind, so it was backed out by 10AM Sunday morning. The change did not cause any issues in our Test server environment. We have identified what is different between Production and Test and are working to make sure they are identical for future testing.

The work on Sunday also introduced a permissions conflict on some database tables. The effect was that some of our weekend batch jobs did not run properly and will need to be submitted again. Some libraries were also not able to save changes to records or received errors in their Voyager clients. We identified this problem and corrected it at 10AM this morning (Monday, April 14th).

Hopefully we have identified the issues surrounding this security change, but we will wait until Spring semester classes have ended before applying it to Production Oracle again.

Brandon Gant
CARLI

No Heartbleed on CARLI Servers

Friday, April 11, 2014 - 4:31pm

The Heartbleed bug in OpenSSL has been all over the news this week. It is a serious enough problem that it even has its own website (www.heartbleed.com). We scanned our systems and we did not find this problem on any of them, so there is no need to worry about changing passwords on CARLI systems at this time.

We are always looking for ways to improve the performance and security of our services. For example, we were already planning to make some changes to our web servers this summer to improve the strength of our SSL connections (newer versions, better ciphers, Perfect Forward Secrecy). If you have suggestions for improving our services, please contact us at support@carli.illinois.edu.

Brandon Gant
CARLI

UIUC DNS Outage Today (April 4th)

Friday, April 4, 2014 - 6:03pm

The UIUC Domain Name Service (DNS) went offline at approximately 3:06PM today and campus networking staff report that it was brought back online at 3:42PM. The DNS service translates human-friendly names (i.e. voyager.carli.illinois.edu) into computer-friendly addresses (i.e. 192.17.55.247).

Without this service, some CARLI servers are unable to lookup addresses so that they can talk to other CARLI servers. The campus Voice-over-IP phone service also was impacted by this outage which resulted in busy signals when calling our office.

The UIUC campus maintains three DNS servers for redundancy: two located in Urbana and one in Chicago. A bad configuration was replicated across all three servers simultaneously causing them to go offline. The DNS manager has provided campus IT staff with a list of the issues that occurred today and the changes they will make to avoid these issues in the future. If needed, CARLI IT staff also have the option to setup our own caching DNS servers that we can use along with the UIUC DNS servers.

Brandon Gant
CARLI

 

RESOLVED: ILDS label problems when using Internet Explorer

Thursday, February 13, 2014 - 8:41am

The issue with Internet Explorer has been resolved. Users should now be able to create labels on the ILDS website when using the Internet Explorer browser.

Description of the problem that has been resolved: A problem creating ILDS labels when using Internet Explorer.  If you experience problems with the ILDS website, please try using Firefox or another browser to create labels.  CARLI staff are investigating the problem.

SSL Certificate Expiration - Update

Tuesday, January 21, 2014 - 8:42am

As of 10:00 pm on Tuesday, January 21, the SSL Certificate has been renewed and users should no longer see an "expired certificate" warning when accessing online catalogs.

Analysis of the Voyager Issues Thursday Morning (11/07/13)

Friday, November 8, 2013 - 10:45am

Yesterday morning (Thursday, Nov 7th) at around 4:45AM, the disk space on the Voyager server reached 100% usage. All services were still online, but it prevented requests in VuFind and WebVoyage, blocked record updates in the Voyager Staff Clients, and caused other issues. Two problems combined to create this service outage:

First, a non-MARC file was manually generated to diagnose a record loading issue. An automated batch job retrieved this file and began working on it. We have modified the batch job to only retrieve files with a very specific file name structure.

Second, the automated batch job tried to parse the non-MARC data as if it were MARC data. This caused the job to go into an infinite loop writing data to disk (which filled up the disk space). We have modified this batch job to ignore any non-MARC data it may encounter in the future.

Once we understood the problem and cleaned up files to free up disk space, the fastest way to restore service was to reboot the Voyager server.

Brandon Gant
CARLI

Network Maintenance

Thursday, November 7, 2013 - 9:35am

Planned network maintenance may cause intermittent outages for all CARLI services from midnight until 10AM Sunday morning.

RESOLVED: November 7, 2013: WebVoyage offline and problems working in Voyager clients

Thursday, November 7, 2013 - 7:57am

As of 9:15am, CARLI Staff rebooted the Voyager server and all systems should be back online and working properly.

WebVoyage stopped responding at approximately 5:00am on Thursday, November 7, 2013.  Voyager server usage has been unusually high and, in addition to affecting WebVoyage, libraries may also experience problems logging into or working in the Voyager clients.  CARLI Staff are aware of this issue and are working to resolve it.

I-Share Voyager offline October 2, 2013, midnight-6am

Tuesday, October 1, 2013 - 1:38pm

The CARLI Voyager system, I-Share, will be taken offline for all libraries from midnight until 6am Wednesday, October 2, to apply an emergency fix to a database index. During the outage VuFind will be available for searching the local and union catalogs, but VuFind will not be able to display item status information, nor will it support access to patron account records or the placing of requests.
 
During the outage Voyager staff client access will be limited to the Offline Backup function of the circulation client only. CARLI’s batch jobs that run circulation notices and the Voyager request promotion service will not run tonight; those data will be picked up in the batch run the following night. Other services such as SFX and discovery services that query Voyager will also be affected by this outage.
 
While we have successfully applied the fix on our test server, there is a slight risk that our work tonight may fail to resolve the problem. If that is the case, downtime may extend into the day on Wednesday. Please watch your email and the System Status News for the latest information and as always, if you have questions or experience any system problems not listed in our announcements, .

Pages