Challenge
Makom, the Education Lab of The Jewish Agency for Israel, spent a year developing an on-site program in Israel to help Jewish communal professionals and educators, in addition to Israel experience program participants such as Birthright, become better informed about pressing issues in Israel today—and especially the Israeli-Palestinian conflict. The program was unique from others in that it did not have a strong political orientation, offered voices from a very wide range of perspectives, and was designed to help participants translate their experiences into professional practice.
Following the outbreak of the COVID-19 pandemic, with the support of the program funders, Makom decided to experiment with running the first few of these “trips” remotely. Makom engaged Rosov Consulting to evaluate this experiment and help it determine whether to continue with a remote program-delivery model or to wait until travel to Israel resumes before running the program for additional cohorts.
Approach
When Makom decided to launch the professional development program virtually, it wanted to learn quickly from cohort one in order to inform the program design of subsequent cohorts. Rosov Consulting developed a post-program survey to be conducted within a few days of educators completing the program. The survey asked the participants what brought them to the program; whether they felt they gained new skills; what they thought of the structure of the program overall and the individual learning sessions; and where they stood on the political spectrum regarding Israel. The survey provided a few opportunities for participants to provide feedback that could contribute to the future design of the program. To that end, the intent was strongly formative, seeking to employ evaluation data in the service of continually improving the participant experience.
Results
Makom learned that the professional development program was, and is, very well received. All participants they felt they gained new knowledge and understanding even when they came to the program with extensive prior familiarity with the content. Almost all felt strongly that they gained an ability to incorporate new ideas into their education programs. The virtual program format was very effective, according to participants, and the length of the sessions (two hours) and the speakers were appreciated by a majority of participants.
Importantly for Makom, the data show that participants don’t know where Makom stands politically, and they don’t see it or its representatives as being aligned with any particular political orientation. Participants say they feel comfortable in the group sharing their thinking and perspectives on issues. Moreover, after the program, participants feel they move closer to the political center, whether they start on the right or left.
The evaluation has informed Makom’s thinking and approach to the program design of future cohorts:
- Makom decided to invest significantly in video production for the virtual sessions—with tours of parts of Israel and interviews with various people. Makom learned that the cohorts very much appreciate the high-quality videos and believe it adds to their learning experience.
- Makom is working to determine if homogeneous cohorts comprised of participants with more similar backgrounds make a difference in the impact of the program.
- Having been uncertain about the pace and intensity of the program in a remote format, Makom gained strong affirmation from participants that these features of the program were well constructed and appreciated.
- Makom affirmed that when participants hear anti-Zionist opinions from certain session guests, that experience does not make these educators any less Zionist themselves. This finding runs counter to the fears of some in the Jewish community. Makom believes it is vital the Jewish community and educators are not afraid to present different opinions to educators. More research, however, is still needed in order to understand when and for whom this is the case. An important working theory is that in the right context, certain opinions can and should be heard. This needs to be studied and understood further.
- To learn even more about the program’s impact, Makom and Rosov Consulting are now employing a pre- and post-program survey design, capturing in real time the extent to which participants’ understandings and sense of pedagogic self-efficacy changes as a result of this experience.
With this efficient, “right-sized” approach to evaluation, Rosov Consulting and Makom offer an example of how formative evaluations can help confirm educational choices or contribute to their adjustment and refinement. Next year, Makom is hoping to evaluate the in-person version of this program and then compare its impact to that of the virtual format. These learnings will inform Makom’s thinking about the frequency with which to make these experiences available to educators and Jewish community professionals.
