To finish up year two of our IMLS-funded grant, the CC-PLUS team pulled together a pilot project to test out our software at a larger scale. Pilot participants included six consortia across the US and Canada representing a diverse group of library sizes and types. The overall goal of the pilot was to investigate the following questions:

  • Does CC-PLUS work at scale?
  • What issues still exist with the interface and tools?
  • What supporting documentation or tools are needed to make this effective?

The six pilot consortia met for the first time in January 2021 to discuss pilot goals and logistics. Instead of working on a shared research or comparison project, participants chose to set individual goals for their participation. Pilot participants agreed to meet weekly through mid-March to compare experiences and share information. After that, they would continue to have access to the pilot instance of CC-PLUS until the end of the grant period (June 30, 2021) but would meet only twice, at the mid-point in April and again at the end in June.

Individual goals among the pilot consortia varied. Some planned to compare CC-PLUS to their current workflows for gathering statistics to determine what advantages might be gained. Others developed collaborative workflows with their members institutions, and in effect ran a pilot-within-the-pilot. The ultimate goal for several was to develop a service for consortia members involving self-service for obtaining usage data. Still others planned to use the data gathered through CC-PLUS with other data on costs to create analyses and visualizations using other tools.

Poster delivered at the 2021 Code4Lib conference on the pilot project.
Poster delivered at the 2021 Code4Lib conference on the pilot project.

All of the pilot participants were able to make some progress on their goals within the initial six-week period, and some findings are already apparent. Most importantly, the pilot participants worked together to create documentation of the credentials needed to access specific vendors. In the world of COUNTER and SUSHI, there are different combinations of URLs, requestors, customers, and API, along with vendor-defined extension fields in some cases, that need to be combined for requests. Keeping a shared list of which vendors require which credentials, and which reports they produce was helpful. A community that supports this kind of information sharing would be immensely helpful to users going forward. As a result of the pilot a full suite of user documentation is being created and several templates and lists will be made available.

In addition to support outside of CC-PLUS itself, the pilot participants highlighted several areas where we can still do work to support the management of credentials. The process for uploading and updating credentials for various institutions and providers of data at present is complex and relies on very precisely formatted spreadsheets. But the act of even gathering the credentials in the first place can be time-consuming and require a lot of cooperation with consortium members. While we can’t do anything about the latter, a redesign of the processing of adding and managing the credentials within CC-PLUS could help to smooth the process. A participant from the Canadian Research Knowledge Network (CRKN) noted: “We recently completed a similar exercise for 20% of our members in a different context and it was a lengthy and challenging process. However, that both CRKN and members would then be able to access their reports in a single point of access would certainly facilitate usage data analysis for both the consortium and members long-term.”

Another area for improvement identified was the need for more nuance around user permissions and roles. CC-PLUS was designed for four different types of users: a consortial admin who can manage all institutions and statistics providers in the system, library managers who can manage their own institutions and credentials, universal viewers who can view the statistics gathered for all institutions and institutional users who can view the statistics for their own institution only.

The pilot presented us with our first very large consortium user: the Kentucky Virtual Library (KYVL), “ a consortium of nearly 300 Kentucky libraries and institutions, including colleges and universities, public libraries, K12 schools, hospitals, Department of Defense libraries, the Kentucky Department for Libraries and Archives (KDLA), and more.” The existing 4 user roles did not meet KYVL’s needs to have administrators and managers of various groups of libraries within their consortia, for example, a user who could manage all the high school libraries, or all the public libraries. Access could only be granted to manage all institutions and providers, or only one. This pilot feedback has been extremely valuable to our future design plans.

In March, informed by the pilot feedback and other project goals, the CC-PLUS Steering Committee submitted a proposal to the Institute for Museum and Library Services for an additional two years of funding. The new grant project will have three overarching goals:

  1. Improve CC-PLUS based on feedback: enhance user interface, administrative controls, and improve database design and speed.
  2. Fund a demonstration project that uses CC-PLUS as part of a larger data analysis project, combining usage data in a seamless pipeline with other library data for analysis.
  3. Build a strong CC-PLUS community.

The last goal is inspired by the pilot in several ways. Documentation and supporting information gathered by and for the pilot group will be used to create a hub for users. The listserv created to support the pilot participants communication will be repurposed into a general CC-PLUS users list. Finally, the model created by the pilot, wherein a consortium can have access to a hosted instance of CC-PLUS for several months to try it out, and then can export data to be ingested and used in a self-hosted version, will be extended to others.

The CC-PLUS Steering Committee would like to thank the pilot participants for their dedication and important contribution to the project:

  • CRKN – Émilie Laveleé-Funston and Jaclyn McLean
  • GALILEO – John Stephens and Ken Henslee
  • KYVL – Ilona Burdette and Enid Wohlstein
  • OhioLINK – Joanna Voss
  • SCELC – Jason Price
  • TRAILS – Hannah McKelvey

The CC-PLUS Advisory Board met on March 25th.  The Board was asked to review the current draft of the proposal for the IMLS National Leadership Grants for Libraries program, which is due tomorrow.  Document and information needs from the planned partners were identified.

The project design for this third part of the CC-PLUS program was described for the Board.  The project will be seeking three new volunteer roles:  Product Owner, Technical Lead, and Community Organizer, and there are two major areas of work: technology and community.  The technology component will be in two phases – core code development to add supplemental pieces of information, such as vendor-specific credentials, database processing efficiencies, etc., and an integration project, where CC-PLUS data is combined with another data source, such as cost.  The community component will have three relevant groups – a Practitioner Community, Pilot Team, and Partnerships Team.  The overarching goals of the third phase of this project are to reduce barriers to adoption of the software and build a community to sustain the program into the future.

The Board heard about the It Takes a Village beta testing coming up, which will involve facilitated activities to explore governance issues in the context of sustainability.

The CC-PLUS Advisory Board met on March 5th.  A second pilot with seven consortia has been underway since February, with the consortia asked to use it as much as possible over a six week period.  They have also been asked to develop a targeted question that they wanted to answer with the tool.  A number of bug reports and suggestions for enhancements have come through, and they have provided helpful improvements.  All pilot participants have been asked to write up a summary of their experience, including use of external tools and insights that have made the tool more usable for them.  Pilot consortia will have access to the tool through June 30th, and some plan to start local installations following the pilot.  The pilot has helped to develop ideas around what kind of user community could support the tool into the future.  An important next step is to have broad reviews of the code by other relevant communities.

CC-PLUS was selected by the LYRASIS It Takes a Village project to participate in developing sustainable governance.  The group will be testing out some of the templates included in the project.  This will be particularly helpful as the project is ending one grant term and hopefully beginning another.

A grant writing group is working on a full proposal for the IMLS National Leadership Grants for Libraries program.  The Advisory Board heard about the draft project plan, which includes a strong focus on future interoperability.  The first phase of the proposed project would center on improving the CC-PLUS software in a number of ways to make it more usable; the second phase would include a project with Index Data as a source for one of their library assessment and reporting tools to test out integration with other sources of library data and show the data being effective at scale; and the third phase would be community development, including practitioners in library and consortia as well as software developers.

The ICOLC Futures Group has narrowed down to six priorities, and a number of those fit well with CC-PLUS as a test case for a sustained and supported open project that consortia want to commit to.  More about this will be discussed at the April ICOLC meeting.  Ideas discussed creating a more formal structure within ICOLC to support initiatives like this as well as building in a strong and connected communications infrastructure.  Alignment to the ICOLC mission was noted as a critical piece.  Connections to e-resource librarians, COUNTER, LibLicense, and many other groups and resources are also deeply important to the future of CC-PLUS.  Could there a partnership to sustain interest in this project as well as others, such as a Library Data Alliance?  Are there ways to enable effective volunteer funding efforts?

The CC-PLUS Advisory Board met on January 28th.  After the Fall 2020 pilot, the project team took the feedback and prioritized the items by impact on the project and the difficulty of making the change.  Many issues related to the interface and a few related to functionality were solved.  Additional unresolved issues from the Fall 2020 pilot will be documented in GitHub in the future.

There is another six-week pilot with around six consortia underway, and there have already been suggestions from this group based on their work at scale.  This group will also provide feedback about useful community-building and documentation needs.  If CC-PLUS should be asked for a full proposal for the planned IMLS application, a small group has stepped forward to help draft and review the document under SCELC’s leadership.

The Board discussed the performance goals and measures of the current grant project.  Two goals were of particular focus: establishing community governance and building partnerships.  The importance of shared advocacy, such as for adherence to the COUNTER standard, was cited as important, as was a blended and robust community among libraries, consortia, and commercial entities – all those with a shared goal of efficiently processing usage statistics.  Connections to external tools – being where the users are as well as robust hosting solutions – are very important.  A small group is going to draft a sustainability and community roadmap for the future of CC-PLUS to bring back to the Board.

On November 20th, 2020, the CC-PLUS project did a webinar update for the ICOLC community, providing and update on progress, a demo, and a glimpse of future plans for the project. Video of the presentation is below and slides can be found here.

The CC-PLUS Advisory Board met on November 5th.  Ten institutions representing consortia and individual libraries participated in the Phase 1 pilot.  The goal of this first pilot was to put the software into practice and identify any bugs; it was not meant to be a production-level pilot.  One member pursued gathering data from 130 providers, and this was a good test of the system and process.  Only around 36 of these providers had COUNTER endpoints, some of which were not R5, and it was further evidence that tracking down valid credentials is time consuming and will continue to be a major hurdle to working with any usage statistics gathering systems.  Over the past month, the team went through the pilot feedback and compiled the key issues.

The requested grant extension was accepted, and the new end period is June 2021.  This allows time for additional prioritized development and improvements as well as a Phase 2 pilot in Spring 2021.  A small group of active participants who will meet weekly would be ideal for this second pilot, as the goal is to put the system into regular operation.  CC-PLUS has applied for a third round of IMLS National Leadership Grant funding for this project, led by SCELC.  If invited to submit a full proposal, there will be a March 31st deadline.

The webinar for ICOLC to be held on Friday, November 20th, 11am EDT currently has 42 registrants.  It will include an overview of the project, demo, and discussion about what will come next for the project.  Registration is available here:   https://vivalib.libcal.com/calendar/events/ccplus2020.

The Board reviewed a stakeholder mapping with an eye toward being inclusive in the development of this software, building community, and ensuring its long-term sustainability.

 

The CC-PLUS Advisory Board met in September.  CC-PLUS has applied to the LYRASIS It Takes a Village initiative, which would help the program in planning and managing sustainability.

The software pilot will be wrapping up on October 9th.  The second phase included changes to make the downloads closer to the standard COUNTER csv format, adding bulk upload of settings, and introducing multi-select functionality in the harvest logs and report generation.  There is a focus now on documentation and packaging the software for its first broad release.  CC-PLUS will apply for a third round of IMLS National Leadership Grant funding for this project, led by SCELC this time, with work prioritized by the Product Management Group and Steering Committee.

The first round of testing with ten institutions has been completed, and there was positive feedback both generally and about specific features. There is a list of priorities for a next release that will be tested next month, and many of these have been completed.  Improvements include a multi-select feature in drop-downs, changes to the harvest log to improve usability, the option to eliminate records with zero use from reports upon export, and more. A number of bugs were also identified and fixed during the pilot. The next release should be available to pilot participants next week, and there is a plan to develop a short demo video for the website.  One more round of testing with an additional release is planned for September.  The Product Management Team will be meeting next month to evaluate and prioritize future development directions.

The Steering Committee continued the discussion about the future framing of CC-PLUS governance models and approach, including the possible roles of ICOLC, COUNTER, and commercial partners.  The committee briefly reviewed the Community Charter & Governance document approved in 2019, draft Community Framework, and a draft proposed Memorandum of Understanding between CC-PLUS and COUNTER, and a discussion about these three documents is planned for the next meeting.  Following this discussion, a small group will develop draft documentation for what CC-PLUS should look like following the grant stage.

The ICOLC Futures presentation and conversation held at the recent ICOLC virtual meeting, and the variety of opinions present, were discussed.  The challenge of relying too heavily on volunteers was noted for both ICOLC and CC-PLUS, and a stakeholder mapping was proposed as a way to identify those with a vested interest in a sustainable future for CC-PLUS.  A virtual presentation about CC-PLUS for the ICOLC community is being planned.

Significant progress has been made on the development of CC-PLUS.  A demo of the current system, which includes ten pilot institutions/consortia, was shown to the Steering Committee.  Users of the software can bring in users, providers, and institutions, and the institutions can be clustered by groups or types (such as Carnegie Classification).  Reports can be automatically harvested each month or manually harvested if the time period has passed.  Institutions can have manager-level accounts to share some responsibility with the consortium’s central office.  Flexible reporting is available as well. 

Development directions under consideration include a web-based version of CC-PLUS that will harvest data and create downloadable reports without long-term data storage (similar to COUNTER’s need for a tool), integrating data from external sources like the GOKb service to enhance CC-PLUS data, more extensive data visualizations, incorporating non-COUNTER R5 data and/or providing a transformation service for R4 data, automated checks for restated data, and an API to make data within the CC-PLUS database available to other services.  There is a planned request to extend this project through June 30, 2021.

The Steering Committee had a discussion about the future framing of CC-PLUS and its associated community.  Active code development and a sustainable governance structure are deeply important to this program’s success.  Its strength is as an open source solution for gathering usage statistics, but high levels of use, partnerships/integrations, and supporting tools will be important to building and maintaining an active community with the resources needed for the software and any associated services.  There is also a valued role of such a community in advocating for COUNTER compliance and gathering information for the practical application of the standard and to inform future development of the standard.  It was suggested that the governance could have a similar structure to that of COUNTER, with layers of committees from the technical to the administrative.  Ongoing maintenance of any implementation of this software should not be underestimated, due to the demands of managing a usage statistics program.  Intentional commitments and shared goals will be important to sustaining CC-PLUS as it emerges into its next stage.

The development focus is on the front end of the software now,  and the UX Developer is building out the pages more fully. The goal is to have all basic pages able to display data by the ICOLC meeting in April.  The demonstration server is now automatically harvesting eight vendors’ data: ACS, BioOne, OUP, MUSE, Sage, T&F, Wiley, and JSTOR.  All four master reports (TR, PR, DR, and IR) are being harvested, subject to their availability from the vendors.

There has been significant interest in the upcoming CC-PLUS pilot, and the planned pilot participants will be finalized and invited soon.  The goal is to cover all typical use cases (individual library, self-hosting consortium, hosted consortium, ConsortiaManager users, etc.). 

Governance in place is for project, but bylaws that establish the way the community should function will be important to the project’s future and sustainability.  A draft of bylaws will be prepared for the next Advisory Board meeting.  A draft of a possible MOU was also presented to the Advisory Board to start a conversation about a more formal relationship between CC-PLUS and COUNTER.

A presentation about CC-PLUS is on the ICOLC April meeting agenda.  This will an opportunity to provide an update on the work of the project and check in with the larger community for feedback.