Updated June 2017

1.0 – Plan Introduction

The WISPALS Library Consortium, recognizing their operational dependency on computer systems, including Database Servers, Local Area Networks (LAN), Internet, and the potential loss of operational control that may occur in the event of a disaster, has authorized the preparation, implementation, and maintenance of a comprehensive disaster recovery plan.

The intent of a Disaster Recovery Plan is to provide a written plan directing the computer system recovery process in the event of an interruption in continuous service resulting from an unplanned or unexpected disaster.

The Disaster Recovery Plan preparation process includes several major steps.  They are as follows:

• Identify the Systems, Applications, and Hardware Configurations currently in use.
• Analyze the impact of computers on regular business and determine the critical recovery timeframes.
• Determine the Recovery Strategy.
• Develop and Document Emergency Recovery Procedures.
• Identify Recovery Team Members and Their Responsibilities.
• Identify Outside Vendor Support Responsibilities.

These steps have been conducted and this document represents the completed effort in the preparation of the WISPALS Library Consortium Disaster Recovery Plan.

1.1 – Mission and Objectives

The mission of the Disaster Recovery Plan is to establish defined responsibilities, actions, and procedures to recover the computer servers, communication, and network environment in the event of an unexpected and unscheduled interruption.  The plan is structured to attain the following objectives:

• Recover the physical network within the Critical Time Frames established and accepted by the user community.
• Recover the hardware and applications within the Critical Time Frames established and accepted by the user community.
• Minimize the impact on delivery of library services and operational interference.

1.2 – Scope of Plan

The scope of the plan is to recover computer information services provided by WISPALS Integrated Library Systems Vendor, Innovative Interfaces, Inc., and the data center and networks located at Waukesha County Technical College.  The scope of this plan is not intended to cover service outages at any of the consortium member college libraries that may occur as a direct result of failures in equipment or services located on-site at those college member campuses.

1.3 – Authorization

The Executive Committee of the WISPALS Library Consortium recognizes the need for a Disaster Recovery Plan for all library operations directly or indirectly dependent on information processing.   The WISPALS Executive Committee has authorized the development and ongoing maintenance of this plan.

1.4 – Responsibility

Responsibility for the development and maintenance of the plan is assumed by WiLS, WISPALS Consortium Manager.   Specific responsibility for ensuring that the plan is maintained rests with the Executive Committee.   In consideration of this responsibility, the library end user community is responsible for coordinating with WiLS on any changes or updates to their information technology requirements.

2.0 – Library Operations Impact Analysis

A Library Operations Impact Analysis was conducted to ascertain the impact of a disaster on the operations of the library.   This analysis drives the Disaster Recovery Plan by identifying and substantiating those applications systems whose loss has the greatest impact on library services in the event of a disaster.

The analysis concludes that there is only one server that is a part of the WISPALS local data center located at the Waukesha County Technical College (WCTC). The SFTP server is used by WISPALS libraries only to transfer data from their local schools to the hosted ILS server. The WCTC server is not used to store data. WISPALS Servers are hosted through Innovative Interfaces, Inc.

2.1 – Objectives

The Library Operations Impact Analysis is completed to determine the critical time frame in which library systems must be available after an interruption in service in order to minimize the potential operational loss of control in services.   The analysis also assists in identifying alternative manual procedures that may be used during the interruption of service.   Therefore, the objectives of the Library Operations Analysis are:

• Educate the user on the need for a disaster recovery plan.
• Identify which critical applications need to be restored in a timely manner in order to continue to support library services.
• Identify alternative manual procedures that may be temporarily utilized in order to minimize the impact of the computer service interruption.

2.2 – Conclusions

It is the libraries responsibility to have in place procedures, whether manual or automated, that the staff can utilize for the continuation of services should their automated systems become inoperable.  Staff should be trained not only in the use of these procedures but also have a clear understanding under what circumstances they should be deployed.

The consortium libraries have identified either WISCAT or EBSCO EDS as their backup online catalog to the Innovative Interfaces Inc., Sierra Library Information System.   If their local workstations are still operable, they would be able to search their online catalog at WISCAT or their EBSCO EDS catalog.

3.0 – Recovery Strategy and Organization

When an interruption in service occurs, it must be determined how best to address that interruption.   The scope of the interruption must be identified, a determination made of its severity, and the effect it will have on each consortium member library’s ability to continue operations.  Only when this has been determined, can a decision be made on how to best approach the problem.  What recovery procedures need to be exercised.   What staff/vendor are needed to effectively carry out those recovery procedures.

• Assess the severity of the disaster.
• Determine what recovery approach to follow.
• Based on the recovery approach, determine the type of staff/vendor are needed to recover from the problem.
• If the current recovery plan is not adequately addressing the problem, discuss alternative plans.

4.0 – Brief Overview of Server Sites

The SFTP server utilized by WISPALS is located on the Pewaukee Campus of Waukesha County Technical College in Pewaukee, Wisconsin.   The SFTP server is located along with other WCTC servers in a climate-controlled environment.

WISPALS utilizes multiple hosted dedicated servers for the consortium.  These are the Sierra Integrated Library System server and Encore Discovery Platform servers.    These servers reside at the hosting site of Innovative Interfaces, Inc.

5.0 – Operation and Backup of Supported Servers

This section identifies how each of the servers is operated and outlines the backup procedures currently in place.

WCTC SFTP Server Backup Scheme

• WCTC backups are VM-level and are stored on a disk-based backup appliance replicated offsite to another disk-based backup appliance.
• A full VM backup of the WISPALS SFTP server is taken daily between 10:00 pm and 8:00 am in WCTC’s primary datacenter.
• The backup is replicated to offsite storage at Carroll University every evening at 1:00 am.
• Backups are retained for 60 days on both backup appliances.

Innovative Hosted Sierra and Encore Servers Operation and Backup Scheme

• Server is operated and available to the WISPALS libraries and their users seven-days-a-week/24-hours a day.
• Technical Support via WiLS for server is available Monday – Friday from 8:00 am – 5:00 pm. Technical Support from vendor, Innovative Interfaces, Inc. is available 24 hours per day.
• Innovative uses enterprise monitoring solutions like Solarwinds and AppDynamics in addition to Cloud watch and flow logs that Amazon Web Service (AWS) provides as services. Innovative leverages these monitoring tools to ensure a proactive approach to identifying issues and addressing them immediately.
• Innovative performs daily backups using AWS’s snapshot based technology as a part of the standard hosting package.
• Snapshots are stored in geographically diverse AWS S3 locations that are replicated to other availability zones to meet the offsite storage requirements for all III customers.
• Random backups are periodically tested and restored to ensure compliance.

6.0 – Sierra and Encore Server Recovery

In the event of a hardware or software failure, immediate action will be taken to contact host vendor, Innovative Interfaces, Inc. to determine the extent of the issue and obtain a timeline for resolution.

Innovative employs a disaster recovery plan that was developed from the NIST SP800-34 framework. Standard hosting terms include recovery on a best effort basis. In the unlikely event of a disaster at the primary location, the latest backup is restored to a different availability zone on a best effort basis.

Innovative also offers enhanced disaster recovery services to meet specific Recovery Point Objective (RPO) and Recovery Time Objective (RTO). The disaster recovery is provided by replicating the bits to AWS instances in a different region within the continent. If there is no separate region available in the continent, then the replication is done to a different availability zone in the same region.

Innovative employs propriety technologies in combination with third party software to deliver optimized incremental replication to alternative availability zone or regions. In the event a disaster is declared at the primary location, the service is triggered to take the last successfully replicated instance and stand up the solution in that alternate location. With the solution available in the alternate location, WISPALS will need to reconfigure the systems to point to the AWS instance and any local DNS changes also will need to be updated.

Recovery Team

WISPALS Consortium Manager, WiLS

• Lead and coordinate the recovery effort.
• Provide recovery status updates to the consortium member libraries.

Innovative Interfaces Inc. Sierra and Encore Support

• Recover/rebuild Sierra and Encore software and hardware systems.

Appendix A: Contact Lists

This appendix contains the names, numbers, and other pertinent information for contacting the appropriate personnel as outlined in the WISPALS Library Consortium Disaster Recovery Plan.

The appendix contains the following lists:

• WISPALS Library Consortium Managers– WiLS Contacts
• WISPALS Library Consortium – Library Directors
• Hardware and Software Vendor Contacts

WISPALS Library Consortium Managers Contacts – WiLS Contacts

Melody Clark	mclark@wils.org	608-313-4175
Andi Coffin	acoffin@wils.org	414-979-9457

WISPALS Library Consortium – Library Directors or Contacts

See Current Contacts.

WCTC Information Technology Services

Matthew Hunter	Enterprise System Administrator	(262) 691-5588
Thomas Tousignant	Systems Programmer Manager	(262) 691-5205

Hardware and Software Vendor Contacts

Innovative, Inc.
5850 Shellmound Way
Emeryville, CA  94608

Phone: 1-510-450-6344 or 1-800444-2344

Appendix B: WISPALS Critical ILS Outage Procedure

In case of a Sierra outage:

2. Verify that it is not a local internet issue.
   1. Check to see if you can access the internet via a web browser such as FireFox or Chrome.
   2. Contact your campus IT Department to verify if there is a local internet connectivity issue.
3. Once identified that it is not a local issue, contact WiLS at wispals-wils@wils.org. You may also contact Melody directly. If you do so, be sure to copy the  wispals-wils@wils.org

Melody Clark
mclark@wils.org
608-313-4175

WiLS Availability: Monday through Friday, 8:00 AM to 5:00 PM. An email autoresponder will be set up for vacations and unplanned days out of the office.

If WiLS staff are unavailable:

1. Contact other WISPALS members by posting a message on Basecamp here.

In addition, another library may be contacted directly to determine the extent of the outage:

Chippewa Valley Technical College                           715-833-6285

Fox Valley Technical College                                        920-735-5653

Gateway Technical College                                           262-564-2786

Lakeshore Technical College                                        920-693-1130

Mid-State Technical College                                        715-422-5470

Moraine Park Technical College                                 920-924-3112

Northcentral Technical College                                  715-803-1115

Northeast Wisconsin Technical College                   920-498-5493

Northwood Technical College                                     715-246-6561 x4222

Waukesha Technical College                                       262-691-5316

Western Technical College                                           608-785-9142

2. Call Innovative’s help desk or log into Innovative’s support site, Supportal.
   The WISPALS site code is WPALS.
3. III Help Desk Phone: 800-444-2344
4. III Supportal
   Open a new ticket by clicking on the Add New button in the Help Desk section. Fill out the required information. In the Urgency section select System Down. Click the Submit button to complete the submission.
5. Update the WISPALS Community on Basecamp, notifying the consortium that a ticket with Innovative has been opened. The individual who opened the ticket is responsible to communicate updates to the rest of the WISAPLS community including any needed actions until resolved or a WiLS staff member becomes available.