UIUC Database Outage April 15
Tuesday, April 15, 2014 - 4:28pm
At 10:51AM this morning, our Database Administrator changed the Oracle password for the University of Illinois at Urbana-Champaign (UIUC) database account to run some tests. He thought he was logged into the Oracle Test server, but he was actually logged into the Production Oracle server. This caused Voyager client errors, forced UIUC circulation clients into "offline circ" mode, displayed "The catalog is not available" message in UIUC's WebVoyage instance, and blocked UIUC's VuFind access. The problem was corrected and UIUC Voyager services were brought back online at 11:12AM and VuFind at 11:29AM.
I apologize for this outage. At our next IT staff meeting we will discuss ways to prevent this type of error from happening again.
Oracle security changes caused Voyager and VuFind problems
Monday, April 14, 2014 - 4:08pm
At 9:30AM yesterday morning (Sunday, April 13th) we made a change to Production Oracle to enhance our database security. This change caused problems in VuFind, so it was backed out by 10AM Sunday morning. The change did not cause any issues in our Test server environment. We have identified what is different between Production and Test and are working to make sure they are identical for future testing.
The work on Sunday also introduced a permissions conflict on some database tables. The effect was that some of our weekend batch jobs did not run properly and will need to be submitted again. Some libraries were also not able to save changes to records or received errors in their Voyager clients. We identified this problem and corrected it at 10AM this morning (Monday, April 14th).
Hopefully we have identified the issues surrounding this security change, but we will wait until Spring semester classes have ended before applying it to Production Oracle again.
No Heartbleed on CARLI Servers
Friday, April 11, 2014 - 4:31pm
The Heartbleed bug in OpenSSL has been all over the news this week. It is a serious enough problem that it even has its own website (www.heartbleed.com). We scanned our systems and we did not find this problem on any of them, so there is no need to worry about changing passwords on CARLI systems at this time.
We are always looking for ways to improve the performance and security of our services. For example, we were already planning to make some changes to our web servers this summer to improve the strength of our SSL connections (newer versions, better ciphers, Perfect Forward Secrecy). If you have suggestions for improving our services, please contact us at email@example.com.
UIUC DNS Outage Today (April 4th)
Friday, April 4, 2014 - 6:03pm
The UIUC Domain Name Service (DNS) went offline at approximately 3:06PM today and campus networking staff report that it was brought back online at 3:42PM. The DNS service translates human-friendly names (i.e. voyager.carli.illinois.edu) into computer-friendly addresses (i.e. 184.108.40.206).
Without this service, some CARLI servers are unable to lookup addresses so that they can talk to other CARLI servers. The campus Voice-over-IP phone service also was impacted by this outage which resulted in busy signals when calling our office.
The UIUC campus maintains three DNS servers for redundancy: two located in Urbana and one in Chicago. A bad configuration was replicated across all three servers simultaneously causing them to go offline. The DNS manager has provided campus IT staff with a list of the issues that occurred today and the changes they will make to avoid these issues in the future. If needed, CARLI IT staff also have the option to setup our own caching DNS servers that we can use along with the UIUC DNS servers.
RESOLVED: ILDS label problems when using Internet Explorer
Thursday, February 13, 2014 - 8:41am
The issue with Internet Explorer has been resolved. Users should now be able to create labels on the ILDS website when using the Internet Explorer browser.
Description of the problem that has been resolved: A problem creating ILDS labels when using Internet Explorer. If you experience problems with the ILDS website, please try using Firefox or another browser to create labels. CARLI staff are investigating the problem.
SSL Certificate Expiration - Update
Tuesday, January 21, 2014 - 8:42am
As of 10:00 pm on Tuesday, January 21, the SSL Certificate has been renewed and users should no longer see an "expired certificate" warning when accessing online catalogs.
Analysis of the Voyager Issues Thursday Morning (11/07/13)
Friday, November 8, 2013 - 10:45am
Yesterday morning (Thursday, Nov 7th) at around 4:45AM, the disk space on the Voyager server reached 100% usage. All services were still online, but it prevented requests in VuFind and WebVoyage, blocked record updates in the Voyager Staff Clients, and caused other issues. Two problems combined to create this service outage:
First, a non-MARC file was manually generated to diagnose a record loading issue. An automated batch job retrieved this file and began working on it. We have modified the batch job to only retrieve files with a very specific file name structure.
Second, the automated batch job tried to parse the non-MARC data as if it were MARC data. This caused the job to go into an infinite loop writing data to disk (which filled up the disk space). We have modified this batch job to ignore any non-MARC data it may encounter in the future.
Once we understood the problem and cleaned up files to free up disk space, the fastest way to restore service was to reboot the Voyager server.
Thursday, November 7, 2013 - 9:35am
Planned network maintenance may cause intermittent outages for all CARLI services from midnight until 10AM Sunday morning.
RESOLVED: November 7, 2013: WebVoyage offline and problems working in Voyager clients
Thursday, November 7, 2013 - 7:57am
As of 9:15am, CARLI Staff rebooted the Voyager server and all systems should be back online and working properly.
WebVoyage stopped responding at approximately 5:00am on Thursday, November 7, 2013. Voyager server usage has been unusually high and, in addition to affecting WebVoyage, libraries may also experience problems logging into or working in the Voyager clients. CARLI Staff are aware of this issue and are working to resolve it.
I-Share Voyager offline October 2, 2013, midnight-6am
Tuesday, October 1, 2013 - 1:38pm
The CARLI Voyager system, I-Share, will be taken offline for all libraries from midnight until 6am Wednesday, October 2, to apply an emergency fix to a database index. During the outage VuFind will be available for searching the local and union catalogs, but VuFind will not be able to display item status information, nor will it support access to patron account records or the placing of requests.
During the outage Voyager staff client access will be limited to the Offline Backup function of the circulation client only. CARLI’s batch jobs that run circulation notices and the Voyager request promotion service will not run tonight; those data will be picked up in the batch run the following night. Other services such as SFX and discovery services that query Voyager will also be affected by this outage.
While we have successfully applied the fix on our test server, there is a slight risk that our work tonight may fail to resolve the problem. If that is the case, downtime may extend into the day on Wednesday. Please watch your email and the System Status News for the latest information and as always, if you have questions or experience any system problems not listed in our announcements, please contact us at firstname.lastname@example.org.