Ready, Set, Go! Practical Tips for Your Learning Agenda Journey
Kimberley Raue, Ph.D., William Rodick, Ph.D., and Stéphane Baldi, Ph.D.
February 2021
Since the signing of the Foundations for Evidence-Based Policymaking Act in 2019, strategic planning activities across federal agencies have been given new life. Better known as the Evidence Act, this law requires the 24 federal agencies covered by the Chief Financial Officers Act to focus on making data more open and goal-driven. A January 2021 memo from the new administration adds urgency to this requirement and further clarifies that evidence-building plans should include broad methodological approaches and the equitable delivery of policies, programs, and agency operations. The Evidence Act creates an opportunity for thinking through integrated and direct connections between data and evidence needs. It is a chance to make data use less ad hoc and better aligned to agency goals.
In many ways, the Evidence Act simply formalizes the practices that agencies already have in place, but it also pushes these practices into more purposeful directions. Thinking through what evidence means for an agency, where it comes from, and which types of evidence are most important can be dizzying. Even if agency staff have a concrete understanding of what evidence means for them and there is a good system in place for linking evidence to decision-making, aligning that work to the requirements of the Evidence Act requires ongoing, focused effort.
Trewon Technologies has been at the forefront of assisting government clients in prioritizing evidence needs. We have decades of experience conducting evaluations across the federal spectrum and have helped senior leadership prioritize data needs for program improvement and accountability. Most recently, Trewon assisted the National Science Foundation in implementing the early stages of the Evidence Act and developing learning agendas for the Office of Evaluation and Assessment Capability, the Office of International Science and Engineering, and the National Center for Science and Engineering Statistics (NCSES). We have developed the practical tips shared here to assist other agencies on their learning agenda journeys.
WHAT IS A LEARNING AGENDA?
A learning agenda has come to be seen as an essential first step for an agency’s response to the Evidence Act. Learning agendas are like many other strategic planning approaches that are already familiar to agencies, but what is unique about this strategy and what does it mean to put one in place? How does an agency make sure that this foundational step is going to lead to the kind of efficient data use practices that are expected from the Evidence Act?
An agency’s learning agenda is a roadmap for building evidence that expands and augments ongoing strategic planning. Through a process of clear prioritization and stakeholder engagement, the learning agenda focuses an agency on its most important needs, the questions that are most vital to answer, and a strategy for developing corresponding evidence.
As a multi-year strategy, the learning agenda can appear overwhelming and complex. It should be durable but flexible. It requires the engagement, cooperation, and long-term buy-in of a broad range of internal and external stakeholders. How many times has a program embarked on a similarly ambitious project, committed significant funds and staff time, only to find out later that it lacked certainty about where to go and how to get there?
WHAT SHOULD YOU KNOW BEFORE YOU GET STARTED?
Based on our work and experience supporting early adopters of learning agendas, we have come to recognize the importance of having clear steps in place at the outset for helping an agency narrow in on questions that directly meet its goals. The steps identified here are based on lessons learned about what worked and what didn’t. While by no means comprehensive, they represent a high-level overview of what to expect when embarking on this exciting journey. Here are a few tips we think will help you get started.
Demystify the Missing Middle
Before starting on a learning agenda, it’s crucial to have a clear vision of
your program—to know where you’re headed and the steps you believe will get you there. This seems straightforward, but we don’t always clearly articulate our assumptions about how a program is intended to work. Instead, we’re often left with what evaluators call “the missing middle,” or a black box around what happens between what a program does (its activities) and what it’s expected to achieve (its outcomes).
We find it helpful to create a theory of change, and if you already have one, to revisit it. Your program’s theory of change doesn’t need to be perfect, but it should be sufficiently detailed so that it makes the implicit assumptions about your program—the how and why—explicit. We recommend working backward from the program’s overarching goal by asking yourself what needs to happen for your program to reach that goal. As part of this process, you’ll identify the activities that lead to the shorter- and longer-term outcomes that ultimately lead to that overarching goal. Developing a theory of change will provide you with a solid foundation for identifying the key questions you have about your program and will want to address as a part of your learning agenda.
Know Your Stakeholders
Matthew Baker from USAID’s Learning Lab notes three factors that appear related to successful learning agendas, including the importance of consulting with a diverse set of stakeholders in the development and implementation of your learning agenda. The goal isn’t necessarily to achieve consensus across stakeholder groups but to understand your program from different perspectives. Different perspectives will prove invaluable to you as you formulate your learning agenda questions and seek the relevant data to answer them.
At this early stage, it’s important to identify who your
program’s stakeholders are and to make sure they include stakeholders inside and outside of your organization. Stakeholders may be individuals, or they may be organizations. They may have a role in the design or implementation of your program or the resources critical to your program’s success, or they may simply have an interest in your program’s results. You’ll want to include leadership and a designated evaluation officer. A brainstorming exercise is a quick and easy way to get your team’s thoughts on paper. We recommend being both specific and inclusive when first identifying your program’s stakeholders. That includes potential or future stakeholders as well as current ones. At a later stage, you’ll prioritize these stakeholders and determine the extent to which you involve them in the process.
Encourage Buy-In
We understand that stakeholders may have limited time to participate in learning agenda activities. Not all the stakeholders you’ve identified will be actively engaged in developing your program’s learning agenda, but those who are will need to have a clear understanding of the learning agenda’s purpose and process. If you don’t have the path forward clearly marked, you’ll lose their interest and buy-in. You’ll also want to show a sincere commitment to incorporating their input into your program’s learning agenda. Stakeholders have important perspectives to share and having their knowledge and experiences at the table will bring about a better, more well-rounded understanding of how your program works and can be improved.
We’ve used several strategies to maximize stakeholder engagement while limiting demands on their time. These strategies include leveraging existing meetings to address learning agenda items, facilitating discussions in ways that ensure they remain on topic and serve their intended purpose, providing creative ways for stakeholders to provide their input, and using virtual meetings to promote wider participation.
Focus on Key Decision Points
Ultimately, the Evidence Act is designed to help agencies generate the data they need and use the data they have to make better decisions about their work. This relies on data that is both timely and actionable. Most of us (maybe all of us) have been in a situation where we’ve received valuable information too late for it to influence our decisions.
Here we recommend another helpful brainstorming
exercise for your team that includes your program’s stakeholders: identify the key decision points related to your program. For grant programs, key decision points will center on decisions about solicitations and awards. As you develop your learning agenda, knowing these decision points will lead you to think about what evidence you need to make those decisions and when you need it. We’ve also found that this exercise is a great strategy for gaining stakeholder buy-in because it signals to them that you’re serious about using evidence (and their input) more strategically.
RECOMMENDED RESOURCES
United States Agency for International Development Learning Lab. CLA toolkit.
The Annie E. Casey Foundation. (2004). Theory of change: A practical tool for action, results, and learning. Baltimore, MD: Organizational Research Services.
ABOUT THE AUTHORS
Kimberley Raue, Ph.D. (kimberley.raue@trewon.com) is Director of Research and Evaluation at Trewon Technologies.
William Rodick, Ph.D. (william.rodick@trewon.com) is a Senior Research Analyst at Trewon Technologies.
Stéphane Baldi, Ph.D. (stephane.baldi@trewon.com) is Vice President of Research and Evaluation at Trewon Technologies.
RESEARCH AND EVALUATION AT TREWON
Trewon’s research and evaluation team is comprised of a range of experts with decades-long experience supporting government clients in all aspects of research and evaluation support services, such as developing learning agendas and evidence maps, facilitating stakeholder engagement, designing valid and reliable quick turnaround studies, collecting and analyzing data, providing technical assistance, developing engaging reports, and fostering a culture of learning and continuous improvement. Our clients include the National Science Foundation, the Department of State, the Department of Justice, the Department of Defense, City Gate Foundation, and the Center for Medicare and Medicaid Services.