System Status

All Systems ActiveAll Systems Active: All Systems Active

Voyager Slow this Afternoon (Wednesday, June 17)

Wednesday, June 17, 2015 - 3:58pm

As many of you probably noticed, the system was extremely slow this afternoon (Wednesday, June 17) starting at around 12:30PM. We have narrowed things down to a set of Course Reserve database queries that originated through VuFind from one member library. We killed those queries at 3PM and the system recovered immediately.

We do not know exactly what is causing this yet, but we now know what to look for and which queries to kill. We will need to do more analysis on the queries before we find a permanent solution.

Brandon Gant
CARLI

Additional info on the Universal Catalog outage

Wednesday, June 17, 2015 - 10:34am

This morning at 2:53AM, the Universal Catalog database stopped responding. The new Oracle version has a parameter that limits the amount of data that can be written to Archive Logs to 10GB. We have increased that setting for all databases and will monitor it to see if it should be increaed further. After making that change, the Universal Catalog started working again at 8:53AM this morning.

As we use the upgraded system over the next few weeks, we will most likely encounter more limits/thresholds in the database and operating system that need to be increased. Thanks for your patience and keep sending us information about any anomalies or outages you encounter.

Brandon Gant
CARLI

Resolved: Oracle Issues Affecting I-Share 6/17/2015

Wednesday, June 17, 2015 - 9:10am

This issue has been resolved. If you start seeing problems with holdings displays in I-Share please email support@carli.illinois.edu

Oracle Issues Affecting I-Share 6/17/2015

Wednesday, June 17, 2015 - 8:38am

This morning we are seeing Oracle issues affecting the display of holdings in I-Share. Searching is not affected. We are actively investigating this issue.

All Systems Back Online

Tuesday, June 16, 2015 - 10:18pm

A change made at 5PM had unintended consequences that didn't really manifest until about 7PM tonight. There were so many processes backed up waiting to run that we eventually had to take most systems offline to implement a fix.

We still have more tuning that needs to be done, but hopefully those changes will be scheduled on Sunday mornings from now on.

We have also put in place the threshold changes we were planning to make at 11PM, so there will be no need for another outage tonight.

Brandon Gant
CARLI

Oracle Performance Problems

Tuesday, June 16, 2015 - 7:28pm

We see some serious performance problems that are affecting Voyager, VuFind, and WebVoyage.

We have started investigating and will attempt to determine the cause and stabilize the system.

Voyager Restart Tonight at 11PM (Tuesday, June 16th)

Tuesday, June 16, 2015 - 4:31pm

We have found a threshold in Oracle that needs to be increased for some libraries. Unfortunately, this change requires a restart.

At 11PM tonight (Tuesday, June 16th), we will take Voyager down, make the Oracle changes, and then bring Voyager back online. We estimate about a half-hour of downtime for this process.

Brandon Gant
CARLI

I-Share Voyager Down for Software Upgrade

Friday, June 12, 2015 - 5:18pm

I-Share Voyager and its VuFind catalog interface are down this weekend (June 12-15) while CARLI staff upgrade its software to Voyager version 9.1.1 .

Note to library staff: Voyager batch jobs will be suspended until Monday night, June 15, so there will be no new circulation notices and no UB request promotion (promoted call slips) until Tuesday morning, June 16. Any bibliographic record you saved in cataloging on Friday, June 12, or updated in WorldCat using a bulk import production mode (e.g., "Express mode"), will not appear in your local VuFind catalog until Tuesday June 16, or in the I-Share union catalog until Wednesday, June 17.

We expect all systems to be back online at 7am, Monday June 15.

Please refer to the I-Share upgrade page for more information including a list of post-upgrade tasks I-Share library staff will want to review.

As always, send any questions or problem reports to us at support@carli.illinois.edu

 

VuFind Maintenance this Sunday Morning (04/04)

Wednesday, April 1, 2015 - 11:26am

This Sunday, April 4th between 6AM and 10AM, the VuFind service (http://vufind.carli.illinois.edu) will experience multiple outages while we upgrade three Linux servers running the SOLR indexes and MySQL database.

If you encounter any problems after 10AM, please report them to support@carli.illinois.edu.

Brandon Gant
CARLI

VuFind Outage, Saturday, Jan 31

Monday, February 2, 2015 - 10:28am

On Saturday, January 31st, at 11:53 AM we received reports of VuFind not responding. CARLI System Services staff investigated the problem and determined that it was due to one of VuFind's services exhausting its memory resources. More memory was added and VuFind returned to normal operation by 3:30 PM.

Todd Pavlik
CARLI

Pages