Your Program Evaluation Questions Answered | Inside the Insights Inaugural Webinar

This transcript captures the first installment of Rosov Consulting’s Inside the Insights webinar series, held on July 30, 2026. The session, Your Evaluation Questions, Answered, was hosted by Frayda Gonshor Cohen, EdD, Managing Director at Rosov Consulting, and featured Rosov Consulting team members Matt Reingold, PhD, and Daniel Parmer, PhD. Together, they explored participants’ questions about evaluation, research, organizational learning, and data-informed decision-making.

Frayda Gonshor Cohen, EdD: Welcome, everyone! Hi, this is exciting. This is our very first Inside the Insights webinar for Rosov Consulting. I’m your host, Frayda Gonshor Cohen. And I’m here with my colleagues Matt Reingold and Daniel Parmer, who are going to introduce themselves in just a moment. I want to, welcome everyone and tell you a little bit about why we’re doing this. So at Rosov Consulting, our mission is to provide high-quality research, evaluation, and insights to Jewish organizations, enabling them to make informed decisions and effectively pursue their strategic goals. Our work is driven by a vision of diverse, thriving, data-informed ecosystems of Jewish organizations that understand and effectively meet the needs of the people and communities they serve. And so this webinar is really an opportunity to do just that, to connect like-minded professionals in this space, to introduce our team to all of you, and to open up a conversation and bring you in to the kinds of conversations we have with our clients and partners, and inside our project teams. I’m… Frayda, again, Managing Director of the firm. I’m joined today by Daniel and Matt. Daniel and Matt are two of our newest team members, but who come with vast experience in the space of research and Jewish communal work, and so I’m going to ask you to introduce yourselves. Matt, why don’t you go first?

Matt Reingold, PhD: Great, thank you so much, Frayda, and hi, everyone. It’s so nice to be able to join you today. My name is Matt Reingold, and I am one of Rosov Consulting’s Canadians, and the only one living in Canada. And so part of my work has been, over the last year, has been working with clients in Canada. Prior to joining Rosov, I did independent consulting along with teaching in secondary schools for a number of years. My areas of research interest, beyond the projects that I work on within Rosov, really revolve around day school, enrollment, student experiences, teacher experiences, camp, and how campers and staff at camp, experience Israel while at camp, and really anything to do with Israel education. Over the last couple years, I’ve done a lot of publishing and writing about Israel education in different settings, and I’ve been so fortunate and excited to bring my interests and curiosities to the Rosov team over the last year.

Frayda Gonshor Cohen, EdD: Glad to have you, Matt. Daniel, why don’t you…

Daniel Parmer, PhD: Absolutely, alright. Well, likewise, it’s great to join everybody here. So I joined Rosov Consulting very recently, just two months ago, actually, but I bring about 2 decades of experience in research, evaluation, strategic learning, within the Jewish nonprofit sector, to my work here at Rosov Consulting. Prior to joining the team here, I served as Senior Director of Research and Evaluation at Combined Jewish Philanthropies, which is Greater Boston’s Jewish Federation. And there I had led the organization’s long-term research agenda, and evaluation portfolio, including directing the design and implementation of the recently completed 2025 Greater Boston Jewish Community Study, which we had partnered with, Rosov Consulting. So, It’s interesting to be on both parts of the work here. And I also led evaluations of many of the major communal initiatives. My expertise is really in kind of mixed methods research with a focus on survey methodology, program evaluation, quantitative analysis, and applied research design.

Frayda Gonshor Cohen, EdD: Great. And, for those of you who don’t know me, I’m Frayda. I’m Managing Director of Rosov Consulting. I do a lot of, internal organizational work, and also get to do, wonderful project work. I’ve been with the company for 15 years. My research background is qualitative in nature. I have my doctorate in education, and a lot of my work at Rosov Consulting focuses on, working with clients on their theories of change, and managing large, complex research projects. So, but enough about me. We’re here to talk about your questions. We… asked those of you who are here to submit questions. We… you’ll note that we don’t put you all on screen together, partly because our goal is to kind of have an open mailbox conversation, and for you all to kind of observe, what it’s like to talk about these questions, and also not to put any of you on the spot, and hopefully for you to see the kind of universality of a lot of the issues that you, raised for discussion. I want to take a moment for those in the participant room, as it were, to actually take a second to introduce yourselves, though, even though you’re not on screen. We hope this is a good opportunity to get to know each other. You can tell us your name, the organization you’re from, why you’re here today, and then, if you’d like, you can… you’re welcome to post your LinkedIn or any link that other people can reach you at, if you’re… if you’re so inclined. So you can do that in the little chat box, below. So, welcome. And we should be able to see those pop up. You should be able to see that all, as you introduce yourselves. Give a second for that, go ahead. There we go, thanks. Hi, Pamela. Here from Cincinnati, can you all see that? Great.

Matt Reingold, PhD: Charles.

Frayda Gonshor Cohen, EdD: Excellent. All right, so you can continue to use that chat function to introduce yourselves, and I’ll keep chatting. Welcome, Alex, and Orna and Pamela. If you’re here because you’re wrestling with a question about evaluation, planning, learning, or strategy, or how to respond to a funder’s request, you’re not alone. One of the things we learn through our work is that organizations across the field are often asking many of the very same questions, which is why we created this opportunity to kind of open things up and to share some of those perseverating problems with each other and hear some of our thinking. I want to note that, while a lot of you are coming from very different kinds of organizations when it comes to your research and evaluation and planning and measurement and learning questions, you share a lot in common. And so we thought, why not create a space where we could have that conversation? The goal here is not for us to present a polished answer to the questions that you submitted. Rather, we want to kind of bring you into the messy experience that is inside our project team sometimes, and some of the kinds of conversations we have in response to the kinds of questions that you’re raising. So we had a chance to briefly look at the questions you all submitted, and we’re gonna kind of do a… Q&A mailbox session, and pull out a question that is representative of what we saw. And then I’m going to open it up for Matt and Daniel, and I’ll hop in where I can, to react to the question from their experience. So… We know that we won’t reach all of your questions today, so we’ll do our very best to get through as many as we can in the time that we have. And we are keeping all of the questions on a back burner, so if you don’t get your question answered today, we hope we’ll be able to address it in another session. And you might find that your question is answered by another question, because there’s a lot of similarity here.

Frayda Gonshor Cohen, EdD: And please do keep an eye on the chat throughout the session, and we will also be doing that to make sure you’re welcome to add questions in as we go. So let’s jump into our first question, which I really think is the, a very fundamental question here, which is, what is the value and cost of conducting an impact study? And do donors expect and value this type of research? Or put otherwise in a different question, but similarly, how do you frame program evaluation as a constructive tool for learning and improvement? and resource allocation, rather than as a signal that programs are at risk. So these questions are really getting to the heart of, what’s the value in doing this kind of research in Jewish organizations. Why do it? What is the value add? Or in… this… the word cost is here, too, that could be both literal or metaphorical here, so, Daniel, do you want to go first?

Daniel Parmer, PhD: Yeah, yeah, I can start us off. So I’m gonna, just start with this cost part of the question, and I’m gonna answer without giving a specific answer, and I know that can kind of be frustrating sometimes. But the cost, the cost really is, like, so variable on so many different, factors. You know, we can do a study, at a very low end, you know, for a few thousand dollars, and answer a question that a client is searching for, and we can also design a study, that costs in the hundreds of thousands of dollars, to answer sort of a similar question, but the, you know, the effort is different. Maybe we’re attacking it from multiple perspectives, maybe we have to, get data, that’s very hard to come by, and so there’s just a lot of extra effort. And so, this question of cost itself. really comes down to what is it that you’re trying to answer. Like, the purpose of evaluation Is to help make decisions. And in that, it differs from, you know, kind of academic research, that you might do. And it’s really to help an organization, help leadership, help programs, understand who they’re serving, understand what’s going on, and to be able to answer some specific questions. So the cost is really, you know. kind of a factor of, like, all of these different things. And, you know, I would almost flip it and say, what is the cost of not understanding.

Frayda Gonshor Cohen, EdD: True.

Daniel Parmer, PhD: your program, or your population, your constituencies, your community. And… and really think about it in those terms. Well, I’m gonna pause there, Matt, if you wanna… if you wanna jump in, I’m gonna come back to this, to this question of cost on a slightly different angle.

Matt Reingold, PhD: Yeah, thanks so much, Daniel. I second what Daniel said about that how could we not want to learn more about the work that we’re doing in order to then inform better decision making? And what I’m also seeing in that second question that you put up, Frayda, is that there’s a concern about how will people react when we’re asking questions, when we’re doing that digging into the learning. And I think that that is something that always needs to be balanced, and so the things that I sort of think about when I’m approaching evaluation Our understanding, like. what types of trust already exists within the organization, and what’s the culture of learning within the organization? And that if there is no prior culture surrounding evaluation and data, it can come across as scary or signaling that programs are at risk. Some of the ways that we can signal, though, that the goal of this is not to cut programs, but to actually, spur future growth and enhance programs is to, first of all, signal that funds are not being directly tied to the program evaluation itself, or at least not to a single program evaluation. And one of the ways that we’ve seen organizations do that effectively is to do midterm and final evaluations, and that when you really view that midterm evaluation as. a discovery, a learning opportunity, because what so many of us know is that when you launch a new initiative, and you have the program design, and it’s all written out beautifully, when you actually move to implementation, things… It’s not that they go off the rails, but they don’t always get implemented the way that you think they will. And so that programs need time to grow and evolve, and therefore, as a result, we want to have that mid-check-in to figure out. What are the changes? How are we learning? How is it growing, before we then move to something that might be more evaluative, in nature.

Frayda Gonshor Cohen, EdD: Yeah, yeah. Go ahead, Daniel.

Daniel Parmer, PhD: Well, I just want to pick on one piece, or pick up one piece that you said early on, which is this issue of trust and communication. And so when we think about both, do donors expect or value this type of research, and this other question about how do we communicate that evaluation isn’t about, isn’t about, like, putting something at risk. You know, it’s not something we do to a program, it’s something that we do with, the organization, with the leadership, with the participants, and it’s really about you know, again, sort of coming to a decision, and that decision may be, is this worth the effort? Is it providing the right kind of impact or the right direction that we need to to go, but oftentimes that question is, you know, we can frame it in a way that says, what can we do to make this program more effective, or better, or serve the right people? And so it’s understanding, are we even getting, like, coming back to the original goals and intent of a program. Are we even doing what we set out to do? Are we reaching the right people? Are we achieving the goals that we had aimed to achieve? And if not, the question isn’t always put it on the chopping block. The question often is, well, okay, how can we now retool this, retweak it, staff it differently, scale it up, whatever the question that we’re really trying to answer, yeah.

Frayda Gonshor Cohen, EdD: Right. I feel like it comes back to one of these, core distinctions in our work is the difference between… so the language here is the value and cost of conducting an impact study. I’m not sure for sure what the questioner was referencing there, but it’s, we often encounter the challenge of where do we measure? Do we measure the outputs or the outcomes? And, which is… outcomes are really often where impact is found, what’s changed as a result of your work. And, most… Jewish professionals are often busy with the work of tracking outputs, the measurable units of activity. How many people came? How many meals were served? How many, curricula written, and those are really, really important measures, but they don’t necessarily tell you much about the results of that work. And both are important, but for different reasons. And in fact, I do think that donors often are living in the world of impact, because they’re working… often, they’re partnering with Grantees, because the grantees are able to achieve or support the mission that the funder or donor is hoping will result from their dollars, right? So, a donor doesn’t know necessarily how to make sure that people are fed, but they partner with a grantee and a Jewish Family and Children’s Services to offer food services, because that organization is going to make sure that hunger is addressed. As a funder, they’re not… they don’t actually have the capacity, but they have the dollars the financial capacity, but not necessarily the expertise to make that happen. So it’s a really symbiotic relationship, and often the donors are sitting in the space of wanting to know, well, were people fed? And the grantees are often wondering, a little step earlier. They’re wanting to prove that they, you know, met the very basic outputs. And it’s really important that all of that learning is happening. Are we implementing the way we thought we would, or are we learning along the way about what it takes to implement well? And also, are we making the change? As Daniel said, like, are we doing the thing we set out to do? So I do think it’s really important to attend to that whole chain of design from, what are we doing? How are we going to know if we’re doing that well, just… and then, how are we seeing success?

Daniel Parmer, PhD: Yeah.

Frayda Gonshor Cohen, EdD: Great. Okay, so that’s, like, a foundation, like, why do this work? And then we had, I think the majority of the letters that came in, the questions that we’re going to pull out of the mailbox, were really about that culture of, and strategy of research in an organization. So we’re going to take one example, and we’re going to go through a number of these that really are about, like, that organizational-level commitment to research. So, how do you ensure that you’re collecting the right data and using it to tell the right stories to the right audiences. It’s an excellent question, and it’s so tricky. So Matt, do you, who started last time? I don’t remember.

Matt Reingold, PhD: I’m happy to. Great. So, I actually see in this question that there’s really, like, two parts to it. There’s the data collection piece, and then there’s the storytelling piece.

Matt Reingold, PhD: On the data collection piece, I think that that actually comes back to something that Daniel said right at the beginning, which is, what is our study design? And As part of the study design, is identifying Clear goals for our work, clear learning objectives for what the client wants to be understanding, having a solid set of research questions. And part of what our work involves is taking all of that and ensuring, then, that the collection tools map onto all of those desired learning goals and outcomes. And part of our work when we’re meeting with clients, it’s never, oh, we’re gonna design a survey and then just go and run it. It’s… let’s iterate that process with clients, and ensure that the survey matches the original RFP, but what we also see is that as clients begin to think more deeply, and they read a proposal, and then they have conversation with us. that RFP language and goals sort of shift, and so part of our work is to continue to iterate that process with the clients, so that… The final tools really align with what their learning goals are.

Matt Reingold, PhD: And then, when I think about the right stories to the right audiences, part of how I’m always approaching clients is asking, who’s the audience that you’re… that you want to be communicating to? Is this something that’s going to funders? Is this for internal use? Is this for being put on your LinkedIn, on your website, as, like. A foundational document, and… I’m thinking specifically about something, and we recently concluded a project in DC, and part of the work involved presenting the data at an academic conference, part of the work involved presenting it Internally, part of the work, within the organization, part of it included a presentation to their stakeholders. And what we did with that client was create 3 variants of the same material, one for each of the audiences, because what each was looking for were different questions and ultimate learnings that they could take home with them. And it’s thinking about, like, what’s the narrative that we’re building? within the data.

Frayda Gonshor Cohen, EdD: As you were talking, Matt, it made me think of, a tool we have on our website. I want to make sure we… Karen, maybe, behind the scenes, can find the link and post it for everyone, but we have this sort of, like, standard four questions to ask before entering a study—intentionally four questions, given our focus in the Jewish communal sector. These questions can guide our thinking around what data to collect, who it matters to, what stories we’re trying to tell that are so foundational before entering into any study. Daniel, what did you want to add here?

Daniel Parmer, PhD: Yeah, no, it’s good. You know, I was just thinking it’s, like, how critical it is on both ends, you know, at the beginning, doing a lot of, kind of, pre-work, really thinking about the kinds of questions, that you want to ask, and I don’t just mean, like, the survey questions, but, like, the big questions that you want to ask. And thinking about the decisions that, you know, this data, this information that you’re collecting, will help you solve. Like, that’s really at the heart of it. And so it’s kind of, you know. collect data with a decision in mind, and communicate it with an audience in mind. So you want to kind of be thinking about, both of those pieces. And Matt, it just made me think also, it’s not like when you were describing the three different audiences, it’s not that we’re just taking the same data and just sort of, like, packaging it slightly differently, because each of those audiences is also asking a very different question, right? When you think about what you’re presenting to your senior leadership team. It’s gonna be maybe more tactical. Than something that you might be presenting, to your board, which might be a little bit more strategic. You know, are we, are, like, are we… is this putting us in the right strategic direction that’s kind of aligned with you know, our organizational goals, whereas, you know, maybe a program manager or your senior leadership might be thinking more about the specific programs and the staffing and the other kinds of resources that are all wrapped up into this. So, it really is, there’s a lot of pre-work that, you know, we do with clients, that we ask our clients to kind of do. Some of that’s done through the RFP and proposal stage, as we kind of think about what’s being asked and how we’re going to respond to that. A lot of it’s done in the work that we do with the client, you know, at the… in those kind of initial conversations and launch. And I… and I’ll just, end here. I, like, I’m sure it’ll come up, at least once or twice, but it… it just reminds me of how critical some of the, tools like logic models and theories of change are, and something that, you know, every good, healthy organization should have and should kind of revisit, because it really helps you identify, like, where are we in this process? What are we looking at, you know, within the kind of life cycle or life timeline of, of this program. Or… or initiative, or whatever it is we’re evaluating. And, you know, it really helps you kind of identify what are the questions, what are the audiences, and kind of pinpoint, you know, where we need to be thinking about change and decision making.

Frayda Gonshor Cohen, EdD: Also, also a tool we have on our website here, so you can look that up. This conversation is making me think about, a book called The Goldilocks Challenge, which is about finding the right fit. Right, like, not too much, not too little around, evidence for the social sector, so I highly recommend looking at that book if anyone, you know, feel free to give me a thumbs up if you’ve read it, but I think they’re… they’re really talking about the matrix of learning and cost. And part of a well-designed study is the right fit. For the organization’s capacity. And, often, there’s, like, too little learning, not enough accountability, or too much accountability, and not enough learning, because we’re so focused on the little details. And it’s, like, finding that sweet spot where the cost is right, and the learning… amount of learning is right. We live in a day and age where you can collect A remarkable amount of data. But it’s, like, not necessarily… you don’t necessarily need all of that information, and we are often struggling, like, you have that one chance to get a survey, and so you want to cram as much information… as many questions as you can to try to figure out what you want to learn, but it’s the discipline of that Goldilocks, like, what is the right thing that we need to study? What’s right for our organization? That will help move our work forward. That’s… that takes discipline, it takes, a lot of stakeholder work, because different stakeholders will have different interests in that question. So it’s… it’s not a neutral question, it’s somewhat political, and people in the position of research agendas for an organization really have to constantly be… keeping an eye on the prize there, about that Goldilocks, right fit, right amount, right amount of learning, right amount of cost, so,

Daniel Parmer, PhD: I also want to make a comment on this, on this question of, like, within the question, you know, are we collecting the right data? And I think we sometimes… we put too much value on the data itself, right? Data are sort of impartial, like, there is no, like, one of my mentors used to say, there are no good data or bad data, there are only good inferences or bad inferences. So, the data themselves are sort of neutral, and it’s how we use them, how we think about them, how we interpret them, the inferences that we’re making, the questions that we’re asking, the decisions that they lead to, right? Those are all sort of imbued with that. that value, And… and obviously, we wanna… we wanna be asking the right questions. You know, I’m thinking about, like, the community study. We were very intentional about, designing the questionnaire shedding some questions that had been asked for decades, and asking new questions, because we were thinking in advance, where do we want to be as a community in the next 5 to 10 years? Not how do we want to compare ourselves to you know, how we’ve changed over the past 10 years, but where do we see ourselves, and then making sure we’re asking the right questions now. So we very intentionally framed it As a needs assessment, and… Making sure that we’re kind of asking the kinds of questions that we you know, to the extent that we could, that we sort of, saw ourselves answering, through the next stage of work, which was follow-up research and the strategic vision process, and, you know, how we’re going to invest our resources, and how we’re even going to raise the dollars for those resources. And so it was very. Intentional, again, kind of putting the effort in at the beginning. Asking, quote, the right questions, but really the… or collecting the right data, but really it was, like, with those questions in mind, so kind of linking those together.

Frayda Gonshor Cohen, EdD: That’s great. There’s this notion of, like, data, I don’t know if it’s good or bad, but it should be credible. So you want the… data to be high quality in the sense that people will trust it, and that it’s accurate. And I think one of the… especially for practitioners, like, it should be actionable. A lot of us are… a lot of us come from academic backgrounds, and we’re curious, and there’s a great question that we would just love to know the answer to, but sometimes the.

Daniel Parmer, PhD: Wouldn’t that be interesting?

Frayda Gonshor Cohen, EdD: Pretty interesting.

Matt Reingold, PhD: Right, I think that it’s also part of what organizations who are hiring us are looking for. They’re, in the same way that, Frayda, I think you said before, that funders might not have the capacity to implement the food service programs, but organizations are hiring us to help them, because our role is to help them better understand what are the actionable strategies that they can actually do once they have the data? Those are our areas of expertise, which is why, Daniel, you said that there’s no good or bad data, it’s just data. That’s why we’re not really just handing anyone, oh, here’s your data table, off you go and run with it. It’s part of the work of what we are doing is as part of the deliverable is… is curating it in some way, helping clients, wade through it, that having 50 pie charts is not something that most clients want. want, nor do they know what they could.

Daniel Parmer, PhD: I mean, pie charts is almost never helpful.

Frayda Gonshor Cohen, EdD: Sorry.

Matt Reingold, PhD: Data tables.

Daniel Parmer, PhD: Data…

Matt Reingold, PhD: tables, raw interview transcripts. Those aren’t the things that are going to help organizations grow in their work and learn from what it is that they’re doing.

Frayda Gonshor Cohen, EdD: I feel like we could keep going here, so I’m gonna, I’m gonna just… practice moving forward. There’s probably… I imagine there’s more to say in the next question. Okay, so this question is, how do we decide whether to evaluate programs individually or as part of a broader portfolio? So I… it’s without knowing too much about the case here, this might look like you can imagine from a funder’s perspective, there’s a number of grants, grantees that fit within a strategy area, for example, or an organization might have multiple, I don’t know, youth-serving a youth-serving organization might have multiple programs running at the same time. How much does the organization want to focus across the portfolio and study all of them, and what are the benefits of that, versus going deep about each one? And sometimes they share things, sometimes they don’t. So how does… how does one think… go about deciding where to focus? Is it… I think I would imagine, like, do you do one first? Do you do the broad first, and then go to the… then go to the narrow? Do you start narrow and then go wide? Like, how do you, think about that, Daniel, Matt? Who wants this time… I don’t know.

Daniel Parmer, PhD: Yeah, I’ll jump, I’ll jump in, I’ll jump in, because this is… this was a lot of the, you know, the sort of challenge that we faced, when I was, at Federation, Right, we had individual grants, we had individual grantees, I should say, and sometimes multiple grants within the grantee for multiple programs, initiatives, and so on. And then we had portfolios. So we had lots of grantees that, you know, fit within one of several portfolios. Caring for the vulnerable, social justice, Israel, and World Jewry, or Global Jewry, and we had, you know, Jewish education, and Jewish communal life, and so there were, like, lots of different portfolios, but sometimes comprised of 5, 10, 20, grantees, again, often multiple, programs, and so, you know, each one, you want to kind of, start to organize, how… if you’re doing, like, portfolio level, you really want to start to… to be thinking about. what some of these programs share in common, right? So, like, an individual program evaluation might be asking slightly different questions, like, who participates? Is the program being implemented well or as designed? What are participants experiencing? What are some of the outcomes, and how do they show up? how can we improve the program, right? So those are kind of, like, individual level, but then when we start to get to more of a portfolio level, evaluation or research design, we’re talking about, you know, collectively, are there gaps or overlaps or redundancies within the you know, individual, programs or grantees. Which populations are we serving? Which are we missing? Do we have kind of the right overall, like, mix of programs, right? So you might… think about, programs related to, education or targeted to a particular demographic group, like young adults, are we too heavy, for young adults on the social and not doing enough around learning, right? And so it gives you that perspective to be able to kind of shape some of the questions that you’re asking a little bit differently. And you want to be able to start to think about, like, developing some common metrics, and this doesn’t mean that the metrics have to be identical, but, you know, start with what they have, and then try and help shape the questions or the data that they’re collecting so that, you know, it is kind of getting at the same thing. Oftentimes, I think about, these specific questions more in terms of, like, constructs. So you can have a broader construct, and lots of different ways that you can ask the question that gets at that broader construct. And so if they’re all kind of, like, feeding into that, right, you want to test and validate and so on, but you can think… you can start to kind of think about your strategy in those terms.

Frayda Gonshor Cohen, EdD: And I think you’re pointing out, like, one of the tricky things about shared outcome efforts, and I’ve been through several of these, where there’s a desire when there’s, like, I don’t know, a teen initiative, or… Or a housing initiative, where you’ve got, like, 10 organizations each tackling a different angle on an issue, whether it’s locally or nationally, and there’s a desire to find a Implement a set of shared questions so that we can look across the 30,000-foot level around the success of that shared effort, and that’s always a really tricky endeavor, because each program often will come back and say, like, oh, that question doesn’t land right for our people. And so I think what we’re hearing in this question, that tension between sort of wanting that 30,000-foot planning perspective, and also really, staying sensitive to the uniqueness of each program and the user experience, like, the participant experience. Like, why are you asking me about my Jewish identity when I’m here for the, for the… for that food program, like, I’m picking up a food item, I don’t know if that connects me to my Jewish identity, but for some larger organization, that was part of a Jewish communal strategy, right? So, those are real challenges here. Matt?

Matt Reingold, PhD: there’s another, I think, layer of complexity. When you’re working within a federation model, or within a single organization, and it has multiple programs that we want to evaluate, it’s… It can be easier to implement that type of evaluation when it’s an external funder that’s funding a lot of different grantees, where each of those grantees has totally different capacities and number of staff and even the types of programs. It’s really important as the funder to be sensitive to those unique circumstances and conditions for each of the organizations, while also holding them to a certain standard to say, we need responses to these types of questions, because this is how we grow our portfolio, this is how we deepen our understanding of the work that we’re trying to do, and figuring out that balance of individual organizations’ autonomy and their need to do evaluations in their own way, alongside the funders’ need to do evaluation, is really an important balancing act. that funders for programs outside of their organization, bless you. is something that’s very important to be mindful of, and I don’t think there’s a one-size-fits-all answer, but I’m thinking of something that I’m working on right now, that a funder that is working with organizations that serve thousands of people, and organizations that serve a dozen people, and their capacity to do evaluation work is very different, and the funder needs to be sensitive to those differences, while still trying to find a baseline for understanding how do they understand their broader portfolio.

Frayda Gonshor Cohen, EdD: Right.

Daniel Parmer, PhD: Yeah, and I think thinking…

Frayda Gonshor Cohen, EdD: Go ahead, Daniel. Last comment.

Daniel Parmer, PhD: Well, I was just gonna say, I think thinking about it, like, in the same way you would think about investment, like an investment portfolio, you know, you wouldn’t sell, necessarily a poorly performing stock just because it’s not doing as well as all of the others, because you have to look at it in the totality, like, what is the purpose you know, that this stock serves? Is it diversifying in a very specific way? And so you really want to, like, think about it in that totality, and not just. kind of, you know, using a microscope and saying, like, this individual thing isn’t performing, cut it, right? So, it’s both kind of broader, but also thinking about how they all kind of work together, and… and thinking about it in the collective sense.

Frayda Gonshor Cohen, EdD: Yeah, I’ll just share one example.

Matt Reingold, PhD: to the idea… oh, sorry, Frayda. I think that comes back to the idea of having, like, clear understanding of The goal of evaluation is to learn more and make more informed decisions, not to make a decision of. cut or keep. It’s how can we strengthen the overall, programming and landscape of the work it is that organizations are doing.

Frayda Gonshor Cohen, EdD: Right. I will share one… one cautionary tale where… where if a program moves, too far away from individual evaluation. You go to these broad portfolio questions, like, you take what we call them in one case in a project, Narrow lenses and wide lenses. When you take a wide lens, by definition, you lose granularity on the… on the… details of the programs. And so, Sometimes, if you don’t take the individual approach while doing the broader of evaluation, you come back with findings that are very high level, because you’re asking a broad question. And then people want to say, but for whom? And in what circumstances? And you don’t always have that information, because you’ve asked, you know, you’ve asked for one question to be included in one survey somewhere, so it’s giving you a broad sense But then you don’t have that deep learning with cross tabs, and so sometimes my answer, if I was building that portfolio using Daniel’s metaphor, I’d say, like, you might want to invest in both in different ways. And if you’re wanting to build a culture of learning, maybe start with the individual so that the folks who are, who are learning to appreciate evaluation can see the benefit for their program, and not just for the portfolio.

Matt Reingold, PhD: Either or. What I’m hearing, Frayda is you’re saying that this isn’t, either or, it’s how do we do both, and how do we make choices about which one to do when?

Frayda Gonshor Cohen, EdD: Correct. Two things can be true. Absolutely. Okay, next question. We’re… I’m, I’m realizing we had hopes for maybe even 10, but I think we’ll get to maybe three. So… I think we might have, covered some of this already, so Karen, let’s go to the next one. Let’s keep pulling out of the box. Okay, so here, here we’re talking about, like, long-term impact. So in that case, we’re saying, like, if we want to track people, is what I’m hearing, because the next part of the question is, how can we rigorously measure the long-term impact while protecting privacy and avoiding survey fatigue? So if you want to do a 6-month 12-month post, or, like, an Up Series–style study, in which an individual is tracked for years and years. How do we rigorously do that without overburdening participants? Daniel, why don’t you start, mostly because I think you’ve had a bit more… more… historical experience with some of these long-term studies. Yeah.

Daniel Parmer, PhD: Well, I think part of it is just being very intentional in your design, in your research design, you know, and thinking about what your long-term goals or outcomes are, and how often you actually need to collect data, and how that data gets collected. So one of the things I really encourage Programs, initiatives, clients, whatever, to do. Is, if you have a registration process, that’s a great opportunity to collect some of the kinds of data that you need, so you don’t have to, like, survey them a second time. To, you know, you might actually only need for registration, name and email and, I don’t know if they have experience, you know, participating in something before, but you might also use that as one of your, like, pre, intervention or pre-program opportunities to collect some data about who they are and the experiences that they’ve had, and so it becomes, like, a data collection point. You don’t have to ask them again. You know, it’s like. if you have a, if you have children, your child comes home from school, and you ask them, like, how was school, and you ask them every day, how was school? You’re gonna start to get an answer that sounds something like, it was fine. And so the more you’re sort of asking the same kinds of questions, I think the less informative and less valuable they’re gonna be, and so you really want to be thoughtful about where you place those, those kind of data collection points. And, and again, always tying it to the design of the program, and to the decision-making, and whatever that looks like over the, over the course, you know, especially for long-term. Evaluation. You might not need, to collect data for, like, another year post in order to kind of see some of that long-term. You don’t have to collect it every… every 6 months, or 2 months, or whatever.

Matt Reingold, PhD: And I’m thinking from a qualitative perspective of what does it mean to be having frequent check-ins, and qualitatively, we’re talking not every 3 to 6 months, we’re talking maybe once a year, once every 2 years. I think it’s really about viewing the participants and In a different way than you might view a typical interviewee, that you view, really, the interview as a longer-term relationship, and that you really afford them the space to sort of take the conversation in the direction they want. And a few weeks ago, I led an interview, and someone said, oh, this is our fourth interview together in the last four years. And I said, oh, I hope you’re not bored. And she said, no, no, no, I love my Rosov yearly interviews. It’s a check-in for me to, like, get to talk about How have I changed? And this was part of a longer-term study that we’re working on, and I think… what she expressed was the idea that we allow her to sort of talk and to reflect, and it gives her space to sort of think about these common questions that we’ve been asking for years at a time, but also a chance to think about what are the new things that are happening in her life. And I think that it’s… It… that creates… it’s not survey fatigue so much as it avoids interview fatigue, is really seeing this as a longer-term relationship that we’re… we’re investing in and asking them to invest in also.

Daniel Parmer, PhD: I’ll just add, also, if you have a big database, you don’t always need to send a survey to everybody. You can, you know, utilize sampling methods and, you know, draw 100 people here, 100 people there, so you’re not always kind of relying on a big blast to everybody.

Frayda Gonshor Cohen, EdD: I want to point out that, I’m glad you said that. The other thing I’m thinking about is get serious about your incentives. If you’re finding that more and more, we’re kind of having to up the ante, that people… if you want people to continue to respond. maybe initially there’s a halo effect, but I’ll do… sure, I’ll do an interview for that thing. But as, you know, you get further and further out, you gotta think about what’s gonna actually capture their attention and time. So think about incentives seriously. I’m aware that we’re just about time, so there was a whole section on specific, methods, but I don’t think We’re gonna have the time, unless y’all think we could try a quick 30-second whip around on this one. So this one, I want to… I want to give you all, like, an on-one foot, quick answer, because I think this is, like, all the time question. How do we… basically, how do we understand people who are less connected, less engaged? That is often, like, the holy grail question.

Matt Reingold, PhD: So, I can share what we did, because this was also what was presented at the conference I mentioned earlier. We wanted to get to less engaged emerging adults, and we did multiple layers of snowball sampling, where we started with program providers, who then led us to engaged young adults in programming, who we then interviewed, who we then asked them for referrals to their friends who participated in… who were less engaged. And then asked them for their friends who were less engaged, and it created this, like, multiple path snowballing. And what it really allowed us to then determine was not just that there are less engaged emerging adults, but there were different barriers to why Different types of emerging adults were not participating, which allowed us to then identify different pathways back into Jewish organized life.

Frayda Gonshor Cohen, EdD: All right, and Daniel, what’s your on-one-foot, 30-second response to this very complex question?

Daniel Parmer, PhD: It’s really, really hard. I mean, you know, when we think about where we get most of our data from, it’s from institutional, you know, membership lists, and that shows the people who are already visible, so it’s sort of like, you know, looking for your keys under the streetlight, not because that’s where the keys are, but that’s because where the light is shining, and so you want to, you know, the keys may be somewhere else. So, like, the big question is, like, where Where do you look? How do you find them? And I think there’s two important things I want to say. One is that people move in and out of Jewish practice, Jewish observance, Jewish connection and identity throughout their life, and so somebody who looks disengaged or unengaged in one stage of their life may come back in another, and so, you know, people are, like, people are multitudes, and so the sample that you get probably includes, even if they are on a membership list, probably includes quite a few people who are… who would otherwise be considered, you know, the target population that you’re looking for, which is the disengaged, unengaged unconnected. And the second is there’s a lot of, there’s a lot of organizations out there that do a pretty good job of providing kind of services to people who have just, I’ll call it, low engagement. So if we think about, like, PJ Library, it’s kind of a gateway. So, it’s often on the sort of leading edge. Before people really start to get really engaged in Jewish communal life, they may just receive a book, and so it’s a really low-lift effort, and it’s a great resource, and it’s a great organization that kind of captures a lot of the the target population that many organizations are looking for, and that’s just one of many examples. There are lots of PJ Library-type organizations out there.

Frayda Gonshor Cohen, EdD: Right. So I want to make a plug for our podcast with, our, our, colleague Annie Jollymore with, in conversation with Josh Miller from the Jim Joseph Foundation, specifically talking about this notion of engagement and unpacking that language. So, Look that up on our website, too. We had a nice podcast episode there. We’re at time. I… we could keep going. This is fun. Welcome to our water cooler at Rosov Consulting. The things we get to chat about all day, and… really privileged to do so. So, thank you all for joining us for this. We look forward to more of these kinds of conversations where we get to wrestle with the kinds of questions you’re all dealing with, and hopefully, give you some… some additional insight, into our work, but also hopefully into your own. If our conversation sparked new questions you’d like to explore, don’t hesitate to reach out to us. You can reach us at communications@rosovconsulting.com. We encourage you to bring more questions to our other, upcoming webinars. And we’ll try different formats. We would appreciate your feedback on this format and how it worked for you. And…

Daniel Parmer, PhD: Survey coming to them?

Frayda Gonshor Cohen, EdD: They’ll be receiving a transcript of the discussion, so they can revisit the conversation, and the next one is on October 21st. Registration will open soon.

Matt Reingold, PhD: Daniel, the data is neutral.

Frayda Gonshor Cohen, EdD: Right. That’s right. And you can always follow us on LinkedIn and on Facebook, subscribe to our newsletter, receive our latest research and practical insights and updates for future webinars. So, thank you all, it’s been wonderful to, See you. I’m just checking one final question, seeing thank yous and links to the podcast, so great!

Daniel Parmer, PhD: Thanks, everyone.

Frayda Gonshor Cohen, EdD: See you again soon.