by Daniel Hensley, Adult Programming Coordinator, Carnegie Library of Pittsburgh, hensleyd@carnegielibrary.org.

Allen County Public Library has strategically incorporated logic models and outcome measurement in planning and evaluating programs and services in a relatively short amount of time by using Project Outcome tools as a central part of an overall shift to a more outward-facing approach to library services.

Judging by Allen County Public Library’s (ACPL) reputation as a national leader in using strategic metrics to improve library services, it would be easy to assume that they were among the earliest public library adopters of outcomes measurement. But, in fact, outcomes have only been a part of the library’s strategy for about three years.

“Not unlike other public libraries, ACPL had not used outcomes-based measurement prior to 2017,” reports Denise Davis, who, as ACPL’s Director of Strategic Initiatives, oversees data collection efforts. “The library relied almost entirely on outputs and anecdotes — customer feedback and quotes — to describe the value of public programming.”

This changed with ACPL’s 2018-2022 Strategic Plan, which established an outcomes-based framework for the development of all library services. According to Davis, this newly central role of outcomes required a great amount of commitment from staff at all levels of ACPL: “This has been a sea change for staff, and a good deal of education has been needed to help staff adjust to this outward-facing approach to planning, service delivery, and evaluation.”

ACPL has been able to quickly integrate outcomes measurement by making a commitment to outcomes at all levels of the organization, applying a strategic approach to planning and evaluation that includes a number of data inputs, and putting the data to work in ways that impact library users right away.

A Marathon, Not a Sprint

Outcomes feature prominently in ACPL’s 2018-2022 Strategic Plan. Tellingly, the document resembles a logic model in its structure – broad goals are connected to demonstrable outcomes, which in turn relate to “investments,” or activities that support the goals of the organization.

The ACPL Director at the time of this interview was Greta Southard. She saw the 2018 Plan, her first as Director of ACPL, as an opportunity to shake up the strategic planning process. “When I came, the strategic plan was not a strategic plan. It was a laundry list of to-do items that we did because you had to submit something to the state library.” Southard’s vision was to use the strategic planning process as a catalyst for organizational change and a way to focus services on community outcomes.

Project Outcome tools play a central part in ACPL’s commitment to using outcomes to improve services, but the surveys and visualization tools are only one part of a larger investment of time, resources, and energy required to make lasting organizational change.

ACPL began using Project Outcome in 2017 and continued training staff in outcomes-based measurement, the Project Outcome portal, and logic models in 2018 and 2019. To put the training into practice, staff had six months to work with program planning and logic models, survey a program, and then use the results to improve the program.

The big picture goal of using logic models, according to Davis, is to encourage professionals to “take the emotion out of program planning.”

“We know that our staff knows how to plan a program — we don’t question that. What we want people to question is their process for identifying topics, and their fallback if a program fails. How will you know if the topic resonates with the community, and how will you adjust it so that it meets community needs over time?”

Davis encourages staff to be strategic about what programs they survey. Rather than surveying every program, she advises staff to be selective when identifying programs to evaluate, using guiding questions such as “Is this a cornerstone program that has gotten stale?” and “Is this a new program that isn’t getting the traction you expected?”

For support, Davis is available for staff to contact to discuss evaluation; additionally, three managers who have received more in-depth training in logic models and outcomes-based measurement serve as point people for staff. Time is given on agendas at regular meetings to guide staff through logic model planning and evaluation processes.

Davis also speaks “almost daily” with staff who want to add or change questions on Project Outcomes surveys. “I ask back what they plan to do with that information.” Planning, evaluation, and development are all parts of the same process, and every decision in that process ultimately ties back to the intended impact on the community.

Avoiding A Colossal Waste of Time

Davis speaks plainly when it comes to the “measure everything” approach to surveying program participants. “[That] is a colossal waste of time.” Instead, Project Outcome surveys are just one of a number of methods that ACLP uses to get data about programs and services, each of which adds to a larger picture of progress on the Strategic Plan.

The experience of using Project Outcome surveys for programs across the library system has taught ACPL some lessons in getting a good response rate, though Davis admits that survey response is always a challenge. It helps to be selective about what programs to survey so that regular patrons do not get “survey fatigue.” Paper surveys continue to be the most effective for in-person programming. But perhaps the best way to encourage feedback is to show patrons that you are listening. ACPL’s Genealogy department regularly sees response rates of 80% or higher, and Director of Special Collections Curt Witcher credits that to a strong connection between customers and staff.

“Carefully crafted questions focusing on how the customer is benefiting from our programs invite responses that are more meaningful and better guide us in our offerings. Team members hearing directly from those experiencing our programs about their needs is powerful in both motivating and guiding our programming work.”

While Project Outcome surveys give valuable insight into the effectiveness of targeted programs, these surveys alone do not show the whole picture. To get more real-time feedback, staff are also encouraged to regularly use informal methods, such as posting a flip chart in the lobby with one question to get feedback about a program, service, or space change. Outreach events are also seen as opportunities to get feedback about what community members would like the library to do. Both of these methods provide real-time feedback, and give people who may not otherwise be vocal a chance to have their voices heard.

To complete the data picture at ACPL, staff have access to a highly developed warehouse of output data. ACPL also uses feedback cards to get customer satisfaction information, and questionnaires administered in print and through Survey Monkey to periodically get topical feedback. The responses to these targeted surveys can be impressive – a recent online-only survey, which was only open for a week, yielded nearly 7,000 responses. All of this is supplemented by reports from Gale Analytics (formerly Analytics on Demand), which help staff understand customer behavior and trends.

ACPL’s experience shows a good example of how to make the most out of Project Outcome. Project Outcome surveys are most effective as part of a more holistic program of measurement. Outputs show attendance trends and help staffing decisions; regular customer surveys provide a baseline of community attitudes and expectations; informal customer feedback gives frontline staff real-time data on a local level. In this context, ACPL’s targeted Project Outcome surveys are used to assess the quality of programs and services by measuring them against outcomes that are clearly defined within a logic model.

As a result of these coordinated efforts, staff at all levels have access to data to help guide decision making, improve services, and track progress on the strategic plan.

Working Smarter, Not Harder

ACPL’s experience in training and support to adopt Project Outcome and other elements of their evaluation strategy has been a slow process and required a major effort, but the investment has already shown returns in service improvements and community relations.

The Summer Learning Program is perhaps ACPL’s largest annual programming initiative, and so has been a major focus in efforts to plan, evaluate, and adjust programming using outcomes and other metrics. The program is supported by a significant local foundation in Fort Wayne. As a result of ACLP’s move to a more intentional planning process, the relationship with the foundation has strengthened; the foundation has offered longer grant cycles and has invited ACPL to apply for additional funding. Southard believes that this improved relationship played a big role in millions of dollars in funding.

Countless smaller changes have been made by staff across the system as they learned to use logic models and outcomes in their planning practice. Project Outcome results have become part of Board communications, too: visualizations, details, and quotes from Project Outcome are included in quarterly Board updates, which are also made available to all staff.

Southard has found that using outcomes has helped board members to better understand the library’s impact. Many board members come from the world of business and are accustomed to seeing profits or other quantitative statements; outcomes provide a relatable way to track progress. This approach can also help situate the library’s work in the context of greater community goals.

“One of the goals for the region is growth of population – we can link increased skill and knowledge to be good for the community in general for future economic development. We are helping build that pipeline for the workforce of the future. Helping make those linkages about the work that we’re doing, from preschoolers up, we can show how we’re feeding into the longer-term aspirational elements.”

To Davis, however, the biggest impact is more subtle.

“Overall, it would not be an overstatement that staff are using the feedback to understand how best to focus their time on program development and delivery to ensure that we are making the most of the limited capacity we have for program delivery – working smarter not harder. As with any evaluation tool, adoption and integration happens slowly. We continue to learn where it makes the most sense to apply the surveys and where not, and the why of evaluation. This is not meant to be a “do you like us” survey. It is meant to guide us in program development and delivery. Sometimes we learn simple but critical things – is the program at the best time, is the room the best location for the program, was the presenter effective. And, we do this in a neutral way through the outcomes surveys.”

As proud as Southard is in the work that has been done, she sees it as only the beginning of an ongoing process. “ We are a learning organization and we have to continue learning what the community wants and continue applying that learning.” By using Project Outcomes tools in the context of a larger strategy that is based on community impact, Southard is confident that her team will be energized as they see the impact of work that has been developing over the past few years.

“I think people are finally starting to understand that you have to have building blocks, and it takes time to put those building blocks in place. Things may not happen in the exact sequence that you would want, and you can’t necessarily predict when all of the variables will happen. But, with a plan, once the variables come into place, you can see the greater impact of your hard work.”

Advice to a Newcomer to Strategic Measurement

According to a recent survey of Project Outcome users, many libraries indicated that they used, or hoped to use, Project Outcome as a one-stop data collection tool for library services. While you may be able to add custom questions and make this work in theory, this approach is not likely to yield much valuable information, and it is sure to cause survey fatigue among staff and patrons alike.

Project Outcome is best used as part of an overall measurement strategy – an important part, but not the only part. Denise Davis, ACLP’s Director of Strategic Initiatives, offers some advice for libraries looking to get more out of Project Outcome by using it within the context of a larger planning and evaluation strategy.

Don’t survey everything. Spend time thinking about what you need to learn to improve a service, and pick one program to start.

Use a logic model to think through the process. They really do work.

Have a specific learning goal in mind for the surveys. Don’t burden staff and attendees with unfocused surveys.

Give it time. Don’t be discouraged if you have a slow start. This is new to your customers as well as staff, so they need to understand why you are asking for feedback and how you will use the information.

Consider an incentive. Don’t go overboard, but you may want to give out some chocolate or another inexpensive “thank you” for helping the library improve programs.

Look at the results…especially the open-ended responses. This is your baseline to guide program development.

Make a plan about what you will do. What can you act on now? What more do you need to know?

Follow up if you need to. If you need more information, find an easy way to get it, such as a flip chart or a sign inviting attendees to speak with staff to give more input.

Share the results with the community. Let them know what you changed, and thank them for helping the library to provide better service.

The post Work Smarter, Not Harder first appeared on Public Libraries Online.

PLA commissioned ORS Impact to interview participating libraries and community stakeholders to create five success case studies for its performance measurement initiative, Project Outcome (PO). PO is a free online toolkit designed to help public libraries understand and share the impact of essential library programs and services by providing simple surveys and an easy-to-use process for measuring and analyzing outcomes. What PLA learned from the case studies is that, even with limited survey responses, libraries are able to leverage their outcome data into actionable results. By using Project Outcome surveys, libraries are tracking their impact across time; improving and expanding programs and services to meet community needs; supporting new and deepen- ing existing partnerships; and increasing library championship.

Using Project Outcome with Summer Library Program to Track Impact Across Time and Strengthen Championship

Appleton (WI) Public Library (APL) first experienced the value of measuring outcomes through their ongoing use of the Impact Survey. Library leadership viewed Project Outcome as an opportunity to extend outcome measurement to their summer library program and support program improvement and communications with the library’s board.

Data-Driven Changes Strengthen Summer Library Program, and Evidence of Patron Outcomes Supports Library Championship

APL offers a range of activities for children, teens, and adults as part of its summer library program. Patrons can earn rewards through an online component by completing missions such as visiting a local nature preserve or the police department. These missions help patrons build connections with community resources and other community members through informal interactions. Project Outcome surveys conducted in 2016 showed that patrons benefited from the program, but found some of the missions confusing and hard to follow. APL has since improved the descriptions and layout of the missions, and they look forward to reviewing 2017 survey results.

APL reports on the impacts of library programs and services to its board based on the results of Project Outcome surveys and Impact Surveys. Library leaders have found that outcomes resonate with board members, strengthening how they voice their support for the library. One leader shared,

I think [reporting on outcomes] really transformed conversations with our board, so that they’re much more interested . . . It lets us tell a more complete story about the library, and what we’re offering . . . so they can understand it more fully . . . When you talk strictly in numbers, or outputs, it’s not something that people can hold on to as clearly. And so when you start talking about outcomes and the impact that a library is having, that’s where the true heart of what we’re doing is. And our board feels that, as well . . . It allows them to . . . have something more concrete to hold on to, and to talk to people about what the library does in a more complete way than “X number of people walk in the door every year” or “X number of books walk out.”

A board member shared, “The [outcome] data provides an objective story, backing up much more engaging stories from staff about serving the community with objective numbers . . . [and] includes things the board may not think to ask for. This adds dimensions to how the board considers the library’s success in serving the community.”

The post Project Outcome Results in Action first appeared on Public Libraries Online.

No matter the size of the library or the population it serves, all public libraries are working toward a common goal—providing relevant and impactful services in areas most important to patrons. As we strive to be a data-driven organization at Sno-Isle Libraries in Snohomish and Island Counties, WA, it is our job to make sure our programs are allocating the right amount of resources to our highest priority services and addressing the needs and interests of our communities. And we need the data to show it.

Data-Driven Change

Sno-Isle Libraries started its outcome data work in 2015 with project management. We began adopting project management methodologies including project charters, the role of project sponsor, and adherence to planning before execution. We considered how to evaluate quality control and user experience, and we were introduced to the outcomes-based logic model by Moe Hosseini-Ara and Rebecca Jones.¹ Continuing to look at systems and processes that would keep our organization moving forward, we saw that we needed better support for our programming services through support and evaluation. Of course, all of this needed to happen without any additional resources (sound like a familiar problem?). We needed to learn how our patrons are benefiting from our programs and we needed to do so quickly, easily, and inexpensively. With the release of the Public Library Association’s (PLA) Project Outcome² in June 2015, it’s as if the stars had perfectly aligned.

Sno-Isle Libraries started engaging with Project Outcome in Fall 2015, as the work of the Programming Support and Evaluation Project began, and was invited by PLA to participate in the 2016 pilot-testing of the Project Outcome Follow-Up Surveys. For us, the power of using Project Outcome is in its consistency and standardization. It is our belief that libraries don’t need nine thousand different ways of doing evaluation. Again, we’re all working toward that common goal, so why keep reinventing the wheel? Project Outcome is a free service available to all US and Canadian public libraries. It provides standardized outcome measures in seven core library services areas, measuring four key outcomes: knowledge, confidence, application/behavior change, and awareness. As a result of participating, Sno-Isle Libraries has shifted its programming purpose statements to align with Project Outcome’s seven core service areas and has fully adopted the key outcomes into our planning and surveying.

The Evaluation Plan

Knowing our evaluation plan and having clear, standardized outcome metrics (Project Outcome) helped us define our audit and evaluation objective deliverable: to produce a quarterly State of Programming Report to our organization.

The post Supporting Strategic Planning with Project and Data Management first appeared on Public Libraries Online.

This will be my last column as your PLA President, and I want to take the time to thank four special groups that have made my tenure so memorable. First, I want to thank my staff and board at the Cleveland Public Library for their support of my leadership journey. Second, I must thank my wife and daughters for their patience and love during the past twelve months. Third, I would be remiss if I didn’t thank you: the great people I have met this year, who work at and support libraries. You have been phenomenal. I’ve heard such great stories about how you are dedicated to making your communities better, it really reaffirmed my love for libraries. Finally, I want to thank the extraordinary PLA staff. Under the leadership of Executive Director Barb Macikas, the organization has made tremendous strides toward making PLA all that it can be, and I’m just thankful to have been a small part of this transformation.

Now transformation may seem like a strong word, but PLA is clearly transforming the way it serves its members and the way it represents public libraries as an industry. As the PLA spokesperson, I try not to be the hyperbolic salesman, but I’ve had the good fortune of presenting our new initiatives and services for the past year to library staff at all levels. I’ve received an overwhelmingly positive response to our new direction. The PLA Board has been advocating strongly for calculated risk taking, and PLA staff members have responded with a substantial effort to deepen PLA’s breadth. With the remarkable final gesture of the Bill and Melinda Gates Foundation’s (BMGF) Global Libraries program, we now have the resources and capacity to make these aspirations reality.

For twenty years, the Global Libraries program has collaborated with libraries and organizations like PLA, investing more than $1 billion globally to enhance the power of libraries to improve lives. Thanks to the exceptional vision of BMGF Global Libraries leader Deborah Jacobs, BMGF made a $10 million investment in PLA in May 2016.

Jacobs’ vision was to create a partnership among PLA, the International Federation of Library Associations (IFLA), and the Technology and Social Change Group at the University of Washington (TASCHA) to innovate the future of public libraries globally. We are excited to work with IFLA and TASCHA to this end.

You can expect these funds to be expended in three areas. First, we will continue our work to build a national platform for public library research, training, and practice. Initiatives such as Project Outcome will continue to grow and provide standardized measures of effectiveness for library services and promote training and implementation tools for using the data collected. Second, the funds will also be used to strengthen collaborations across organizations within the library profession and external to the profession. Our advocacy work is growing and PLA received a Power of (A) award from the American Society of Association Executives for the co-creation and implementation of Every Child Ready to Read. Our efforts are being recognized within and outside of the library world. The third area will be a focus on introducing U.S. public libraries to great work being done globally. Partnerships like the one between PLA and the African Library and Information Associations (AFLIA) will strengthen African libraries but also provide a connection for American libraries to learn from their African counterparts. This will build upon the PLA Library Academy and the Inclusive Internship Initiative, two programs committed to producing more diverse and prepared leaders for public libraries of the future.

As this wonderful year comes to a close, I must extend my gratitude to all the former PLA presidents who preceded me, especially the immediate past president Vailey Oehlke. They have assisted in the creation of a public library infrastructure association like no other. I’ve learned so much about leadership from these wonderful people and want them to know that I’m grateful. Pam Sandlian-Smith is such a thoughtful and determined leader that I know that PLA’s future is great in our next president’s hands. Good luck Pam, you’re going to love it.

The post The Transformation Will Not be Televised first appeared on Public Libraries Online.

Despite the limitations she faced in her one true love (dance), Kendra blossomed academically. The public library had become the family’s second home and the resources at the library had become the foundation for her success. Eighteen months ago, Kendra had walked into the Main Branch not knowing what to expect from the library. She shushed her brothers as they walked in and was initially surprised at how much activity surrounded her. A library staff member welcomed them at the front desk and directed her to a room in the back. As they walked through the library, they passed banks of computers that were filled with focused users. When they reached the meeting room, she was greeted by a young woman who introduced herself as Kailey. Kailey was a local college student who worked with a library partner organization to provide tutoring every afternoon after the local schools ended their day.

Kendra was a good student before she started working with Kailey, but the math tutoring that she received at the library transformed her from a middle-of-the road math student to the highest-scoring pupil in her algebra class. It hadn’t been easy. At first, Kendra was resistant to spending every afternoon watching her brothers and breaking down math equations, but her mother was thrilled with the arrangement. Kendra would sometimes be embarrassed when her mom would praise her as “my little genius” to her friends, but she was glad that her mother was proud of her. She could see changes in her brothers as well: her youngest brother was now reading all the time and the middle child was now an honor roll student.

While she still missed taking additional ballet classes, her mom would let her stay a little longer at the library for its Wednesday yoga class. Her mother also let her volunteer on the weekends. She was surprised by how many interesting programs the library offered and how they affected the lives of so many people. Sometimes, the librarian would let her introduce an author or someone important, and Kendra would think about how the library had changed her. Eighteen months ago, she had walked into the library as a student of the Anywhere School of the Arts. The library now allowed her to become a student of the world.

Kendra’s story is fictional, but we know that public libraries are transforming lives like Kendra’s across the United States every day. As I mentioned in my last column, I’m the poster child for public libraries changing lives, but beyond anecdotal stories, what can we do to document our work?

PLA’s Project Outcome seeks to solve this shortfall of information and the results are promising. Project Outcome started in 2013 as a task force of librarians and researchers who were brought together to consider outcome measurement. Then-PLA President Carolyn Anthony made outcome measurement her most important initiative and a task force was launched to examine using standardized measurements to judge the effectiveness of library programs. Chaired by Denise Davis, deputy director, Sacramento Public Library, the task force began with a question: “How do we go beyond simple attendance counts and enhance existing service data?” The task force met for over a year and in the fall of 2014 began to test surveys in seven core service program areas:

1. Civic/Community Engagement
2. Digital Learning
3. Early Childhood Literacy
4. Economic Development
5. Education/Lifelong Learning
6. Job Skills
7. Summer Reading

Each survey is flexible, easy to use, and accessible to libraries of all sizes. The feedback from the pilot libraries that tested the surveys were positive and patrons liked that the surveys were short and specific. The success of the pilot led to funding from the Bill and Melinda Gates Foundation. A three-year, three-million-dollar grant has allowed PLA to expand the program and currently 1,396 libraries are using Project Outcome. We are thrilled by this promising beginning, but recognize that we need many more participants to join us so libraries can build more persuasive arguments about the programs we provide.

I know that you are shaping the lives of young people like Kendra every day with the programs libraries are providing, but we must move beyond expressing our successes through inputs. Success today is judged by the outcomes displayed by those who attend our programs. Please join us as we seek to document how we make our communities better. Get more information at www.projectoutcome.org or email info@projectoutcome.org.

The post Measuring the Impact of Public Libraries first appeared on Public Libraries Online.

These outdated views of what libraries were create a negative impact on our libraries and tend to erode support for libraries with this logic: if libraries stand for books, and if books can be downloaded from the cloud at the drop of a dime, then why do we need libraries? Library members and frequent visitors know better. In fact, our regulars have been visiting public libraries for so long they have witnessed our transformation from pre-computers to free Wi-Fi and can testify to how libraries have impacted their lives.

Not only do the new videos align with the E’s of Libraries, they support the initiatives of Project Outcome, which focuses on outcome-based reporting favoring impact over numbers and helps librarians demonstrate to local funding authorities the difference the public library is making in the lives of their local residents. And if increased viewing, creating, and sharing of video content is the direction our society is headed (see Pew Research Center’s log of articles about online videos), then certainly our clever and creative use of these videos in our own libraries can be quite impactful.

So, how do we put these videos in the front of those who need to see them?

Because the films were intended for you to use in your library, they are devoid of library names or patron names, making them perfect for repurposing. Extend the E’s of Libraries brand by not only posting the videos to your existing media channels but also consider creating additional digital content. For example, when posting information about your library’s job events link to a locally created web page that displays the video created for employment and relevant statics[1] to accompany that video:

• Every day, 300,000 Americans get job-seeking help at their public library.
• In a year, 30 million people use library computers and internet access for employment or career purposes.
• Ninety-two percent of libraries provide access to online job databases and resources.
• Seventy-six percent of libraries help people complete job applications online.

Perhaps people who do not normally view the library as a place to get job help will think twice after seeing the video.

Consider using them in local PSAs, incorporate them into your talks and speeches, and play them at your next Friends meeting. The videos can also be created as a tool to request additional funding. For example, if your library is seeking to be that entrepreneurial spot in the community, show the video to your funders and let them know that you aspire to be that place, and that you need their support to get there.

The possibilities are endless! What’s most important is that you use them. If you found an impactful and powerful way to use the videos in your library or have some ideas on how you might use them, you are welcome to post in the comments below.

Resources

PLA Premieres Library Advocacy Video Series at National Conference

Project Outcome – Looking Back, Looking Forward

Public Library Association Conference in Denver showcases new advocacy series

Public Library Association Advocacy Videos

References

[1] “Employment,” Libraries Transform, accessed May 18, 2016.

The post Can Videos Change Perception that Libraries are Only About Books? first appeared on Public Libraries Online.

Measuring impact and outcomes is getting easier. Public librarians have an assortment of tools available to demonstrate the impact of library services in their communities. National initiatives like the Impact Study and PLA’s Project Outcome provide new standards and tools to measure library services.

Indicators like circulation or program attendance are valuable metrics. However, these are only indicators, a measure of volume rather than value. It’s important to align indicators with the desired goals and outcomes of a service to demonstrate its impact. Demonstrating impact is essential to the ongoing health of America’s public libraries. Measuring outcomes of library services helps to tell the story of why resources and funding are needed to sustain and grow existing services.

One nationally funded tool, the Impact Survey, was developed in 2009 to compile information for a study titled “Opportunity for All.” Researchers from the Technology & Social Change Group (TASCHA) and the U.S. Impact Study Research Group aimed to measure digital inclusion initiatives and their effects on public libraries across the country. In the six years since the study was initially conducted, the Impact Survey has shown some trends in responses from public library technology users. One key trend is that the majority of respondents have access to the Internet, but come to libraries to receive assistance from library staff or volunteers.

The Impact Survey has grown to become a standardized measurement tool for public libraries. It is now on its third iteration and offers many self-service tools that are easy to use. If you are looking use outcome measurement at your library, below are additional resources to investigate:

• Impact Survey: Tool to measure impact of public library technology services.
• PLA’s Project Outcome: Performance measurement tools for areas such as civic/community engagement, early childhood literacy, economic development, and job skills.
• Digital Inclusion Interactive Map : Comparison between American Community Census data and Digital Inclusion Survey results.
• Edge Initiative: Assessment tools to align services with growth and community priorities.

Tools like these help link goals with performance indicators. Through this comparison, public librarians can continually improve the services they offer. They can also take the next step of adjusting them to better fit local community needs.

Resources:

1. Samantha Becker et al., 2011. Opportunity for All: How the American Public Benefits from Internet Access at U.S. Libraries. Washington, D.C.: Institute of Museum and Library Services. Accessed October 25, 2015.
2. Becker, Samantha. 2015. “Outcomes, Impacts and Indicators” Library Journal 140, no. 15: 26. Accessed October 25, 2015.
3. Becker, Samantha. 2015. Impact Survey: Measuring Your Library’s Impact. Accessed October 25, 2015.
4. Teasdale, Rebecca. 2015. Project Outcome Launch – Seven Surveys To Measure Impact. Accessed October 25, 2015.

The post 21st Century Library Measuring Sticks first appeared on Public Libraries Online.

Denise is Deputy Director, Sacramento (CA) Public Library, ddavis@saclibrary.org. Emily Plagman is a Project Manager for the Public Library Association, eplagman@ala.org.

Project Outcome is PLA’s latest field-driven initiative, helping libraries to capture their impact in the communities they serve. In 2013, libraries and researchers formed a task force with a mission to develop and test a simple set of outcome-based surveys for any library type to use when measuring the outcomes of their services and programs. Project Outcome builds on the task force’s work by providing resources and support to help any library set strategic goals, measure outcomes, communicate their findings, and successfully achieve their goals.

Project Outcome started as just an idea when then-PLA President Carolyn Anthony announced that she would make outcome measurement a primary initiative during her tenure. “What is needed,” she wrote in an August 2013 Public Libraries Online article, “is a set of performance measures that can capture the services public libraries are currently providing in their communities, with guidelines for conducting the measures to ensure consistency and validity. We also need to measure the outcome or impact regarding the difference that some of these services make in the lives of individuals and the well-being of the community.”¹

Shortly thereafter, a PLA Task Force was formed comprised of librarians and researchers. Their mission: to develop standardized measures of effectiveness for widely offered public library programs and to promote the training for implementation and use of the measures across public libraries. They kicked off their work in earnest in July 2013 and spent the next year developing surveys specifically designed to be easy-to-use, flexible, and accessible for libraries of all sizes and demographics. The group pilot tested the surveys in the fall of 2014 and finalized them in early 2015.

The seven topics are:

1. civic/community engagement
2. digital inclusion
3. early childhood literacy
4. economic development
5. education and lifelong learning
6. job skills
7. summer reading

The Task Force’s work caught the eye of the Bill & Melinda Gates Foundation, resulting in a three-year, $3 million grant to PLA to build a project around the surveys—complete with resources, support, and trainings. Funding will allow PLA to expand the program with a goal of developing an online framework for public libraries to easily enroll and start using outcome measurement as well as creating a framework for a field-wide shift toward making outcome measurement a business-as-usual practice. In January 2015 the grant kicked off.

Project Outcome Implemented

By spring the project was given the name Project Outcome and development of content began. On June 26, 2015, Project Outcome launched publicly at a preconference during the American Library Association (ALA) Annual Conference in San Francisco.

At the project launch, more than sixty libraries met to learn how to pair strategic goals with service needs; to measure programming impact; and to identify ways they can take action such as communicating the findings of the surveys and clarifying their needs to funders, staff, patrons, or community. Project Outcome offers the resources needed to do all this work in one place and is designed to be comprehensive, customizable, simple, and available to any size public library interested in surveying one program once or many programs often. It sounds like Project Outcome is really easy, right? That’s because it’s designed that way.

Project Outcome helps libraries implement outcome measurement, capturing the change in patron’s beliefs or skills as a result of using library services and programs. It will help library staff measure the knowledge a patron gained after taking a job skills or résumé writing class. It helps librarians quantify the patron’s value of early childhood literacy programming, a bookmobile, or a cultural learning activity. It provides a centralized online location where library staff report and see the results of their surveys. It supports their efforts to use that data as evidence of what they already know in their gut to be true—that patrons find significant value in the services they offer and that libraries are not just buildings with books, but integral players in healthy and vibrant communities.

What Is an Outcome?

Public librarians are already accustomed to measuring their value in the form of outputs—they measure circulation, computer usage, and door counts. An outcome, however, is a specific benefit that results from a library program or service. Outcomes can be quantitative or qualitative, and are often expressed as changes that individuals perceive in themselves—like new or improved knowledge, skills, attitudes, behavior, or status. Project Outcome is designed to help libraries measure and identify the value they bring through critical services and programs (see table 1).

Why Measure Outcomes?

Outcome measurement helps libraries make an even stronger case for funding, partnerships, adding staff, or garnering community support—because it offers librarians methods for presenting the patron’s perspective. Project Outcome simplifies the steps to measure a crucial component, the services they provide to the public, and successfully uses the findings to achieve goals. Outcome measurement can help libraries:

• To better measure and improve upon the impact in the community. The services libraries and library staff members provide are all intended to improve patrons’ lives. Outcome measurement helps quantify what they hear every day from patrons: that library services make a difference in their well-being, level of understanding, and change in knowledge or skills.
• To support planning and assessment over time. Whether working toward a specific goal within a library’s strategic plan or developing new long-term plans, information about how current services benefit patrons today is essential.
• To help better manage services and resources. Outcome measurement helps ensure that the money and time spent on a program or service is the best use of those limited resources.
• To justify funding requests. This is one of the most important reasons! Anecdotal evidence can help illustrate your library’s value, but libraries need more to attain resources. Adding outcome measurement data alongside anecdotal evidence and output data will strengthen your library’s reports and proposals and allow you to better advocate, providing library leaders with a more compelling case for continued or increased library funding.

How Does It Work?

Project Outcome provides simple survey instruments that library staff can use to collect insights and data about all the ways libraries are meeting the needs of their communities. In a few steps, public library staff set goals; use the Project Outcome portal to enter data; view summary results; generate reports; and access Project Outcome resources to successfully take actionable steps to meet their advocacy, partnership development, and strategic decision-making goals.

The process is designed to align with services and activities that libraries are already conducting, and is simple enough that just about every library system can use it. Project Outcome uses an easy four-step process (see figure 1).

Some timing factors that may help libraries decide when to administer Project Outcome surveys include:

• anticipated changes in funding that will affect the library’s ability to provide its current services—whether it is a scheduled budget review or a special situation;
• an upcoming opportunity to seek funding to grow or add existing services;
• the library is beginning a new service and wants to set a baseline to compare outcomes;
• in order to show progress in a priority service area that aligns with a strategic plan, goal, or need;
• the library is planning to make service additions/changes or reductions and staff want to understand which services have the greatest outcome for the community; or
• library programming staff are required to report their performance and want to use outcomes evidence to demonstrate value.

The Role of Staff

Each member of a library team serves an important role in the success of implementing Project Outcome surveys. Pilot test-users reported that the greater the involvement and buy-in of staff at all levels, the greater success in both breadth and depth of survey distribution, use, and response rates—and, the greater the adoption of changes by the staff themselves because they were willing to make changes to improve the effectiveness and quality of their programs.

Examples of staff roles include:

• Librarians. Administer surveys at the end of programs that they manage.
• Library managers. Make decisions to change programs and services based on survey results.
• Library staff that interact with partners and community leaders. Package and communicate results appropriately for each audience.

Survey Topics and Sample Layout

Project Outcome provides seven topic areas to measure with short, easy-to-apply surveys (see table 2). Users are encouraged to select surveys based on the strategic focus or need of the library. It is not anticipated that libraries will administer all seven surveys. The data collection depth and periods are determined by the library based on how they envision using the results.

Each survey is six questions long and asks patrons to rate their level of agreement about what they learned or gained after attending a service or program on a five-point Likert scale, ranging from “strongly agree” to “strongly disagree,” with an option of “not applicable” (see table 3).

Field Test Findings

In 2014, the Project Outcome Task Force field-tested the surveys with twelve libraries and a total of twenty-seven locations. Each participating library was encouraged to use the surveys according to their needs. Some chose to enroll only a few test libraries within their systems. Others chose to test one or several survey topics. Still, others chose to measure a single program or a program series. Libraries responded very favorably to being able to choose the surveys and methodology that worked best according to their needs and objectives. A follow-up survey was distributed to the libraries after the testing phase and the findings are summarized below.

Patrons Willing to Help Out

Overall, pilot testers received a positive response from patrons when they were asked to complete the surveys. Some appreciated and wanted to give feedback on services. Houston Public Library tested two programs—a business competition and toddler storytime—and found in both instances that patrons were eager to participate and provide feedback for each one even though the topics and audiences were widely different. Both patrons and libraries found the limited length and specificity of the surveys valuable, especially patrons with kids who shared their appreciation for its shortness.

Champions Drive Success

With many effective initiatives, a champion helps drive successful implementation and system-wide adoption. As expected, libraries that had champions leading the effort had an easier time enlisting staff to help with testing the surveys. And if the champion is upper-level library management, engagement and participation by other staff is broader and deeper. Finally, response rates tended to be better when surveys were administered by committed staff and not volunteers.

Wide Range of Libraries in Pilot Test

Pilot-test libraries consisted of a wide range of public libraries as far as size, location, and level of outcome measurement experience. Most of these test libraries reported that they were able to use the surveys to meet their testing purposes. As an example, Holbrook Public Library in Arizona is a small library with only 2.5 staff serving a population of about 5,000 people. They surveyed six events and received a 31 percent response rate for their 120 attendees. While they did not have capacity to test as many events as they originally intended, they still found the survey responses to be valuable—affirming that the services they offered were meeting community needs.

Quantitative and Qualitative Data Were Valuable

Participating libraries were able to use the resulting data in board presentations, branch activity reports, and to underscore advocacy stories. Skokie (Ill.) Public Library (SPL) found the quantitative ratings of the surveys used after a business class helped them understand whether they were teaching useful information to patrons. Arizona libraries—participating at the request of their state library systems—found high scores to have a reinforcing effect on the good work they already do. The open-ended questions at the end of the survey were valuable, as well. SPL staff also found that the survey’s comments helped them understand why people choose to attend online business classes and what they want to get out of the session. The feedback they received will help them make the class more effective compared to only knowing attendance prior to the assessment.

Shift Toward Outcome Measurement

Prior to the launch, PLA staff worked with ORS Impact, a research consulting firm, to interview libraries about Project Outcome and outcome measurement in order to help guide content development. They conducted twelve interviews with library leaders, who, in the 2014 Public Library Data Service Survey, had expressed a high level of readiness and interest in outcome measurement.² The libraries represented a variety of size and geographic locations. The findings reinforced what the Task Force learned in their pilot testing: the library field is poised to make an industry-wide shift toward outcome measurement. The interviews revealed:

• Library leaders believe the field will benefit from having access to the types of resources Project Outcome is developing, particularly outcome surveys and best practices related to survey administration, data entry and analysis, and data-sharing partnerships.
• Engagement in outcome measurement at libraries is often part of broader organizational orientations toward outcome-based thinking and the library’s role in addressing community priorities.
• Data and outcome savvy libraries “lead by example” in their communities, gaining the esteem of stakeholders to whom they are accountable and attracting community partners.
• Many library leaders engaged in outcome-based measurement have successful data-sharing relationships with aligned community partners, particularly school systems. They also believe they and others can benefit from learning about different ways to partner for the purpose of data-sharing and different types of data sharing arrangements.
• Understanding what constitutes an outcome may be “fundamental” or “basic,” but it is also quite challenging.
• There is an appetite among data savvy library leaders to share their experiences with other libraries, either as part of learning communities or via efforts to cultivate more interest in outcome measurement, in the library field.
• Libraries on the leading edge look outside of the library field for information and partners to support their measurement and use of outcome data.³

Project Outcome Benefits All Libraries

Any library—one that has been collecting outcomes for years or just getting started—can benefit from participating in Project Outcome. Libraries in the beginning stages of measuring outcomes will have “Getting Started” resources available to them once they sign up on www.projectoutcome.org.
Mid-level enrollees—those who may have used some outcome measures in the past but have not operationalized them in their work—will have access to one central data entry and storage system with ready-to-use surveys that can take them from the occasional outcome measurer to adopting it as a regular practice. And finally, a champion user—someone who measures as a standard—can benefit from the webinars and other resources made available by Project Outcome that will examine topics like how to take data-sharing partnerships to the next level. Regardless of experience level, libraries have free access to all of these resources.

At all levels, Project Outcome offers a few distinct advantages over a standalone measurement system. It providesunique data visualizations of both the library-specific data and aggregate peer-level data for libraries to use to help build upon their own story. It also offers a comprehensive set of tools with regularly updated informational and training content based on the learnings of its users. It relies on peer-level engagement to develop content instead of a top-down approach to learning. The more libraries participate in Project Outcome the better it becomes.

References

1. Carolyn Anthony, “New Measures for a New Era,” Public Libraries Online, Aug. 21, 2013, accessed Apr. 30, 2015.
2. 2014 Public Library Data Service Statistical Report Special Section: Performance Measures, accessed July 27, 2015.
3. Interviews conducted via phone by ORS Impact with twelve library leaders in March 2015.

Further Information

Visit www.projectoutcome.org for more information or to sign up for the service. Information on webinars and other related events can also be found on the website.

The post Project Outcome: Helping Libraries Capture Their Community Impact first appeared on Public Libraries Online.