Zoom Logo

Evaluating/Assessing Reading Programs during COVID-19 - Shared screen with speaker view
Maureen Simunchembu
16:15
Hi Everyone, greetings from Zambia
Susan Hirsch-Ayari
16:42
Susan Ayari, Creative Associates, DC
Ildiko Laczko-Kerr
16:45
Hello from Phoenix Az
Joke Van Belle
16:47
Hi Maureen. Hi everyone. Greetings from Zambia.
Angela Keenan
16:47
hi, Angela Keenan, GEC Ethiopia, MEL lead
Pradeep Kumar Mishra
16:49
Hi This Pradeep from India
Luis Crouch
16:49
Luis Crouch, from RTI
Monica Fontana
16:50
Hi everyone, Monica Fontana from Uganda
Swadesh Maharjan
16:53
Hi, greeting from Nepal (EGRP)
Alice Eastwood
16:55
Hi All! Joining you from Bristol in the UK. I'm normally in Indonesia.
Katharina Hammler
16:56
Good afternoon from Munich, Germany!
Muhammad Fida Fida
16:58
Hi Everyone, Greetings from Pakistan.
Cledenin Veras
16:59
Hi! Cledenin Veras from the Dominican Republic
Carolyn Johnson
16:59
Hello from Portland Maine, USA
Christy Fellner
16:59
Christy Fellner, Director of Child Protection and Education at World Vision US
Allyson Krupar
16:59
Hello all! Ally Krupar, Save the Children
Elena Walls
17:00
Elena Walls, USAID/Office of Ed
karen tietjen
17:01
Karen Tietjen, Creative Associates
Felipe Hevia
17:02
Felipe Hevia, from México
Eve Brooks
17:02
Eve Brooks from Indonesia
Tania Santisteban López
17:02
Saludos desde Guatemala, Centroamérica
Isabelle McMahon
17:05
Isabelle McMahon, Et4d Charlottesville VA
Lisa Zook
17:05
Hi all, Lisa Zook from InfromEd International in Seattle, USA
corrie blankenbeckler
17:06
Corrie Blankenbeckler, New York, Creative Associates
Maby Palmisano
17:07
Hi Maby Palmisano, MEL Director Creative, Washington DC
Shawn Slakk
17:07
Hi there,
Communications Zambia
17:07
I’m Annabelle with Impact Network Zambia
David Waweru
17:08
David, Kenya
MaryFaith Mount-Cors
17:13
MaryFaith Mount-Cors from EdIntersect
Maria Dzula
17:14
Maria Dzula, RTI, Colorado
James Mburu
17:14
Hello everyone. James from Kenya
Cathy Davison
17:15
hello - Cathy Davison in Cordele, GA - working in Africa when I can get there...
Gopini Pandey
17:15
Hi, greetings from Nepal
Gertrude Niles
17:16
Hello. Greetings from Grenada
Moazzem Hossain
17:16
Hi everyone, Moazzem from Bangladesh, Save the Children
Sagar Neupane
17:17
Sagar Neupane, Nepal
Catherine Galloway
17:17
Hi everyone, calling from Kigali, Rwanda!
Meron Mamo- Creative
17:18
Greetings, Meron from Creative Ethiopia
Dereje Getahun [Creative Associates Int)
17:18
Hi I am Dereje Getahun from Ethiopia READ II project
Shawn Slakk
17:18
Shawn Slakk her in Chester VA
Mackenzie Matthews
17:18
Hi from NYC! Mackenzie Matthews, Education TA at IRC
Carina Omoeva
17:19
Carina Omoeva, FHI 360 / Washington DC
Marcia Davidson
17:19
Hello everyone. I am from the Ghana Learning Program and FHI360.
Ireen Katongo
17:25
Hi everyone
Jennifer Bister
17:26
Hello! Jeni Bister with SIL in South Africa
Chris McBurnie
17:27
Hi everyone! Chris McBurnie from the EdTech Hub (currently based in the UK).
Cristiana Amenabar
17:27
Good Morning from Guatemala
Linda Ng`ombe
17:28
from Linda Ng`ombe ,Zambia .Hello everyone
Sherry Sacino
17:29
Hi everyone....I’m Sherry Sacino from Global StoryMakers in Florida.
David Roba
17:32
David Roba from Kenya
A van Ginkel
17:33
Agatha van Ginkel, Afghan Children Read
Lindo Simelane
17:33
Greetings all, Lindo with SurveyCTO in Washington DC
Juliette Mendelovits
17:34
Hi. I"m Juliette Mendelovits from the Australian Council for Educational Research (Melbourne)
Elisabet Eikaas
17:42
Hello from Oslo, Norway
rosa vique
17:46
Hi, Rosa from Mozambique
Anna Farrell
17:47
Anna Farrell, Appleton, Wisconsin USA (about halfway between Chicago and Canada)
Maureen K
17:48
Hi everyone, from Relief International, Somalia
Raj K. Dulal
17:52
Hello all
Cosnat Ntenje (Malawi)
17:56
Cosnat Ntenje - Malawi
Raj K. Dulal
18:06
good evening from Nepal
Aurore Dorelien
18:13
Aurore from Ethiopia
Egidia Umutesi
18:15
Hello everyone. This is Egidia from Rwanda working for WV Rwanda
Kara McBride
18:19
Hi everyone, Kara McBride here, from World Learning. I'm here in connection with a program in Lebanon, typing from the US
Probak Karim
18:24
Hello, this is Probak from Rwanda
Aleligne Mengist
18:29
Hi everyone, this is Aleligne from Ethiopia, READ II, IR2, Quality Assurance Advisor.
Guy Bostock
18:30
Hello Everyone. This is Guy Bostock from Ethiopia READ II
Chala Legesse
18:37
Chala Legesse, From Ethiopia - CHADET
Chilobe Muchimba
18:46
Hi all. This is Chilobe Muchimba, from EDC - Let's Read Zambia.
Andres Navarrete-Berges
18:55
Hi Chala and Everyone! this is Andres Navarrete from One South
Melindah Chuma Michelo
19:06
Hello.
Penelope Bender
19:06
Penelope Bender, BurdaEducation, Washington D.C.
MUKANDAYISENGA VIRGINIA
19:18
Hello everyone,
Ireen Katongo
19:26
Hi everyone, joining you from Zambia
Khalilur Rahman
19:28
Hi Everyone. This is Khalilur Rahman, Basic Education program, Save the Children, Bangladesh country office.
Melindah Chuma Michelo
19:33
Melindah from Zambia.
Michael Greer
19:35
Greetings everyone! Michael Greer here with World Vision, working remotely form Austin, TX
Ireen Katongo
19:41
katongo lreen
MUKANDAYISENGA VIRGINIA
20:04
Hi everyone. This is Virginia from Rwanda.
Lisa Easterbrooks
20:08
Hi Everyone - Lisa Easterbrooks, Food for the Hungry
Carol da Silva
20:13
Hi Everyone, This is Carol da Silva joining from the Global Book Alliance
Paige Morency-Notario (Basic Education Coalition)
20:29
Welcome!\
Paige Morency-Notario (Basic Education Coalition)
20:31
Agenda:• BEC & GRN intros• Results• Logistics• Parental Engagement• Data Quality• Decision-Making• Audience Q&A• Closing remarks
Kinfed
20:48
HI Everyone, this Kinfe from CHADET Ethiopia
Paige Morency-Notario (Basic Education Coalition)
20:48
Please find the intervention summaries at the following links, which likely contain many of the answers to your questions around the program, school details, evaluation/assessment methods, etc.CARE International: https://drive.google.com/file/d/1OpSDV0_qnG517zRCWafQwmB-X52qZ9W-/view?usp=sharingCentre for the Study of African Economies: https://drive.google.com/file/d/12Ex75fueBsL0CX9xwyBwP1FLHC7GgmFf/view?usp=sharingCreative Associates: https://drive.google.com/file/d/1kVZviw0z1gTHkZxKpBoA9Rz9qx-xKRs5/view?usp=sharing
Tariq Omarshah
21:06
Hello All- this is Tariq Omarshah from One South an external evaluator for several literacy programs
Liana Gertsch
21:06
Liana Gertsch with RTI in Kyrgyz Republic
bwamey
21:41
HI everyone, Greetings from Beatrice Wamey at Nascent Solutions
ousmane diouf
22:42
hi everyone !
Semere Solomon
22:53
Semere Solomon, Creative Associates International
Truphena Choti
23:03
Hello everyone! Truphena Choti joining from Nascent Solutions.
Zannatun Nahar
23:06
Hi everyone, Greetings from Save the Children Bangladesh. GREAT to see you all. Zannatun Nahar
Olayinka Olagbegi-Adegbite
23:15
Hello everyone, this is Olayinka Olagbegi-Adegbite from Wisconsin, USA
Paige Morency-Notario (Basic Education Coalition)
23:17
Throughout the webinar, please write your questions in the chat box, indicating the country, project, organization, or individual so we know where to direct your question.
Flossie Kamulaga
23:54
Hello! Flossie Kamulaga from Nascent Solutions, Malawi.
saima malik
26:35
Hi everyone. Saima Malik from USAID Washington DC
Chris Cummiskey (BEC M&E Co-Chair, RTI)
29:07
Hello Everyone! If you have any questions for our panelists, please specify to whom the question is directed. You can specify either the individual presenter(s), the organization, the project, or the country.
Chris Cummiskey (BEC M&E Co-Chair, RTI)
29:46
Also, please review the following documents before asking your question. These documents might answer your question.CARE International: https://drive.google.com/file/d/1OpSDV0_qnG517zRCWafQwmB-X52qZ9W-/view?usp=sharing Centre for the Study of African Economies: https://drive.google.com/file/d/12Ex75fueBsL0CX9xwyBwP1FLHC7GgmFf/view?usp=sharing Creative Associates: https://drive.google.com/file/d/1kVZviw0z1gTHkZxKpBoA9Rz9qx-xKRs5/view?usp=sharing
Swadesh Maharjan
32:52
Can you share more about system use during SMS survey? That means any system was adopted or it's mannual entry by enumarator into kobo based on SMS receieved.
Maureen K
33:22
Thank you Lotte, how long was the assessments conducted for each child? Did they take both assessments?
Nafisa Baboo
34:10
were children with visible/known disabilities included, how and what were your experience & findings?
Sagar Neupane
35:13
@Noam, how did you get back data by using SMS only assessment?
Lotte Renault (CARE)
35:28
Hi Maureen, it depends on the child's fluency level, but the assessment is typically fast - less than 25 min. We have asked children to go through both assessments in each call.
Egidia Umutesi
35:48
At what level parents/caregivers are involved to support their children's learning?
Allyson Krupar
36:18
To the teams who used SMS, who responded, caregivers or children? GEC mentioned that phone availability was an issue, but what about response rate and identifying who is responding, who uses the phones? Was caregiver sensitization/training included?
Chris Cummiskey (BEC M&E Co-Chair, RTI)
36:21
The links to the papers for Low Tech Remote Learning can be found in the bottom of the document found here:
Chris Cummiskey (BEC M&E Co-Chair, RTI)
36:29
Centre for the Study of African Economies: https://drive.google.com/file/d/12Ex75fueBsL0CX9xwyBwP1FLHC7GgmFf/view?usp=sharing
Lotte Renault (CARE)
36:43
Hi Swadesh, we have used a basic system - sending the content for each task via SMS and asking the child to read it aloud/ respond the questions orally. The enumerator then entered the responses manually on Kobo.
Lotte Renault (CARE)
38:38
Hi Nafisa, CwDs were included in the sample. 10% of the students assessed had some type of disability (other than mental health issues).
Laura Mack
39:08
How was your response rate with telephonic interviews? Did you make these calls or did you use an automotive system?
Kinfed
39:47
In my opinion the stakeholders findings more reflect the realities at the ground.
Lotte Renault (CARE)
41:23
Hi Allyson, the children responded the survey directly. Caregiver consent was requested beforehand.
Jummai SARKI
41:33
Nice to be at this important session. Joining from Ibadan, Nigeria
annette Kiberu
42:32
Hello everyone from Annette in Uganda
Lotte Renault (CARE)
42:40
Allyson - on response rates, we probably got a high one (over 90%) because of two factors - parents are also involved in project interventions and students are used to receiving phone check-ins from project staff.
Tariq Omarshah
42:48
Thank you for the presentations. A quick question regarding the response rate question raised by Noam. Did you conduct any analysis to compare the composition of non-respondents to the respondents? With non-respondents did you have any data on who you were trying to reach e.g. marginalization criteria etc.?
Paige Morency-Notario (Basic Education Coalition)
43:13
For anyone just joining, please find the intervention summaries at the following links, which likely contain many of the answers to your questions around the program, school details, evaluation/assessment methods, etc.CARE International: https://drive.google.com/file/d/1OpSDV0_qnG517zRCWafQwmB-X52qZ9W-/view?usp=sharingCentre for the Study of African Economies: https://drive.google.com/file/d/12Ex75fueBsL0CX9xwyBwP1FLHC7GgmFf/view?usp=sharingCreative Associates: https://drive.google.com/file/d/1kVZviw0z1gTHkZxKpBoA9Rz9qx-xKRs5/view?usp=sharing
Paige Morency-Notario (Basic Education Coalition)
43:21
Throughout the webinar, please write your questions in the chat box, indicating the country, project, organization, or individual so we know where to direct your question.
Ferenc Koszorus
45:03
Question for CARE/Somalia - you mentioned that your team achieved gains in literacy and numeracy. Did you have benchmarks/goals for literacy and numeracy gains? If so, how did you determine those goals?
Ezra Simon
46:16
For all programs: Given light touch content approaches, do you anticipate distance learning to continue during the school break period or if schools reopen late? From parents' point of view, do they see this as a suitable way to keep kids busy for those who aren't completely engaged in agricultural or other pursuits?
Muhammad Tahir
46:32
Hello everyone, Muhammed Tahir Bello & Rukkaiyatu Bashir Ribadu from AUN/ SENSE.
Laura Mack
48:06
Laura Mack from USAID, South Africa. Another question about response rate for parents. One presenter mentioned calling repeatedly for responses but that is staff time and money. Did you have to increase your budget for this data collection?
Lotte Renault (CARE)
48:36
Hi Ferenc. We have benchmarks for both literacy and numeracy. Obviously, this abridged test is not directly comparable to the EGRA/EGMA results and respective benchmarks, but the results show that there were gains in the acquisition of skills, and such gains increased based on the self-reported time spent on remote learning and also on self-reported support received from teachers.
Cosnat Ntenje (Malawi)
48:53
I find the Telephone assessment model (IVR) implemented during COVID-19 to be more visible and appropriate post COVID-19 to enhance learning outcomes through community engagement. Assessment and Remediation sessions done through Interactive Voice Responses or calls made after school for parents and the community to participate in supporting struggling learners or all learners in general through these low tech approaches.
Lotte Renault (CARE)
50:06
Hi Ezra, given the social distancing requirements, there is a reduction of the time spent on class, resulting in the need for a blended approach. Therefore, we consider that remote learning is likely to continue for a long while.
Andres Navarrete-Berges
50:31
Hi, thank you for these insightful presentations. I wanted to ask, to Lotte... what would a decent sample size of a "temperature check" is for learning assessments? Are we aiming to make generalizations representative at a 0.8 power (as for the GEC)? Or are adjusting expectations about generalizability?What about the ethics of doing learning assessments with a control group or people who are not receiving the intervention. Do temperature checks aim include them? How we can draw conclusions about impact if we do not?
Sulleiman Adediran
52:12
Nice to be part of this session. Joining from Ibadan, NIGERIA
Lotte Renault (CARE)
52:39
Hi Andres, since we considered that this modality of testing is here to stay, we powered the sample to allow us to conduct repeated assessments for the next 6-8 months, detecting changes as expected.
Paige Morency-Notario (Basic Education Coalition)
53:08
Welcome! For anyone just joining, please find the intervention summaries at the following links, which likely contain many of the answers to your questions around the program, school details, evaluation/assessment methods, etc.CARE International: https://drive.google.com/file/d/1OpSDV0_qnG517zRCWafQwmB-X52qZ9W-/view?usp=sharingCentre for the Study of African Economies: https://drive.google.com/file/d/12Ex75fueBsL0CX9xwyBwP1FLHC7GgmFf/view?usp=sharingCreative Associates: https://drive.google.com/file/d/1kVZviw0z1gTHkZxKpBoA9Rz9qx-xKRs5/view?usp=sharingThroughout the webinar, please write your questions in the chat box, indicating the country, project, organization, or individual so we know where to direct your question.
karen tietjen
53:27
For Noam, what is the time commitment your program activities required from parents? Also, what directions did you give to parents to to enable them to gauge learning of their children? How did you deal with illterate households?
Chris McBurnie
53:49
@Dereje - thank you for sharing these results and insights! I like the idea of using basic questions on scheduling and story lines to assess access.
Virginia Ngindiru
54:30
To Noam, what was the dosage and duration of the intervention?
Chris McBurnie
54:33
@Dereje - do you have any ways or plans to assess learning outcomes from radio? How did you identify your sample?
Chris McBurnie
55:20
@Noam - I really liked your paper on the Young 1ove programme. I understand you are writing a follow-up paper after the next programme cycle. Do you have any indication of when this will come out?
Andres Navarrete-Berges
56:57
Thanks, Lotte, could you elaborate how the sample was powered? Was a control sample involved? if so, what has the response rate been like in this group?
Chris McBurnie
57:25
@Noam - do you have any examples of an effort riddle?
Kara McBride
58:45
@Noam, if there is a reward for effort, how do you make sure the answers come from the targeted child?
Andres Navarrete-Berges
59:40
@Lotte: was there an age below which you found children were not familiarized with the SMS?
Elena Walls
59:42
Question for all panelists: how much variability in learning gains did you find? Did you look into sample characteristics to explain this variability? Basically, for whom these programs seem to be working better?
Swadesh Maharjan
01:00:32
Thanks Lotte, If I understand correctly then you have some questionnaire initial regarding can child can open SMS or not? right
Andres Navarrete-Berges
01:04:36
@Dereje: thank you very much for your insights on the context of remote learning. If we were to do phone-based learning assessments in rural Ethiopia, do you have any major tips or advice?
Meron Mamo- Creative
01:06:52
Hi Andres, One tip to support remote data collection is having multiple contact details to ensure random assignments of respondents so that the result can be more representative of the situation on the ground, and other is collect parent details as part of the school profile data collection with parent contact details might help
Andres Navarrete-Berges
01:10:18
Thank you Meron, we are currently at a midline and have these details. But as Chris said, it may take up to 6 weeks to track a participant even with contact details so I was wondering if there were ways to increase the response rate in a rural context.
Angela Keenan
01:10:19
Andres & Meron, the issue we are faced with is some of our girls living in very remote parts of Ethiopia with no phone coverage so tele-interviews are not possible
Luis Crouch
01:10:32
On the data issue my sense is that in some situations it is fine to be able to distinguish between "almost all parents say X", "most", "around half" and "very few or almost none." As long as the achieved sample is not hugely biased, the CIs do not have to be very tight to get at these kinds of directional results. And that's way better than nothing. If it is also fast.
Angela Keenan
01:10:48
We want to ensure inclusivity but face these challenges
Lotte Renault (CARE)
01:11:13
Hi Elena, results are poorer among those who speak a language other than the LOI. Average scores increased as the amount of time spent on remote learning (self-reported) increased.
Luis Crouch
01:13:05
I don't mean to divert the conversation so please feel free to ignore me and then (if you wish) write me directly to lcrouch@rti.org afterwards. But especially for the presenters I have this question. As a policy person working at more of a global level it'd be good to have a sense of whether the policy makers in the countries where Noam and Dereje and others work are: 1) aware that these more lo-tech ways to do DL may be needed or at least worthwhile on an *ongoing* basis and that 2) it can be used especially to catch up the "covid" generation *especially* on the foundation skills? If this stuff is not making it to the policy level, is it wortwhile? Are your insights good enough to be communicated to those levels or do we need to learn more, still?
Meron Mamo- Creative
01:14:33
@ Anglia, yes that is one of the challenge we face to ensure random assignment , so we set criteria and interview respondents with phone, as indicated on the DHIS 2014 indicate 47% access, currently estimated that the mobile reach is more than 40 million. However, we collect regional data to understand where the challenge is intense so that we can devise possible intervention e.g 12 % of randomly selected respondents do not have any media with variation in different regions
Angela Keenan
01:15:29
Thanks @Meron - that makes sense
Angela Keenan
01:16:05
and target particular areas.
Laura Mack
01:18:42
What was the average attempts for your project?
Andres Navarrete-Berges
01:19:02
@Meron: thanks, that gives us an estimation for potential attrition.
Olayinka Olagbegi-Adegbite
01:22:23
Does any one know if any oragnization is evaluating the influences of COVID-19 on public primary education in Nigeria and what measures are being taken to alleviate learning loss?
Nafisa Baboo
01:25:54
Can you please post the link of the research that includes with the cost-effectiveness findings?
Meron Mamo- Creative
01:26:40
@ Laura the attempts were different for different regions, but we add 30% airtime assuming that the data collector might do multiple contact e.g some parents ask for the different time recall at their convenience and this are good things need to be planned
karen tietjen
01:26:42
for Noam, given the cot-effectiveness information you shared, do you think that the Ministry of Education will scale-up your program nation-wide?
Andres Navarrete-Berges
01:29:08
Thank you for that estimation Lotte.
Eve Brooks
01:29:19
Noam - Why do you think What’s App was not an effective tools for assessment , especially when parents requested this tool?
Elena Walls
01:29:54
Noam, I think it is really essential not to standardize CEA estimates to 1 SD since this would assume the relationship between cost and effect is linear. I think it'd be great to have the exact numbers for cost and the corresponding effect, to make sure folks understand what level of investment was made to produce what effect
Angela Keenan
01:34:43
Does anyone have any experience or tips on measuring numeracy outcomes remotely
Angela Keenan
01:35:42
akeenan@childhope.org.uk (as the focus here is on reading)
Muhammad Tahir
01:35:48
is it different set of people do you interview or the same set of people for duration
Angela Keenan
01:35:54
Thanks Lotte, will do
Meron Mamo- Creative
01:37:04
@ Muhammad for READ II we use successive independent sampling so each round of data collection will have different respondents
Muhammad Tahir
01:38:19
how do you measure if there is a change in one particular respondent
Paige Morency-Notario (Basic Education Coalition)
01:40:53
pmorencynotario@basiced.orgwww.basiced.orgBEC Twitter: https://twitter.com/BasicEdtweetsBEC LinkedIn: https://www.linkedin.com/company/15227360BEC Facebook: https://www.facebook.com/basiceducationcoalitionBEC Instagram: https://www.instagram.com/basicedcoalition/BEC Newsletter: https://mailchi.mp/basiced/international-basic-education-update-k8klfjnv5eGRN’s new website: https://www.globalreadingnetwork.net/GRN knowledge products on BEC’s website: https://www.basiced.org/globalreadingnetwork
Elena Walls
01:41:16
thank you to the panelists and a huge thanks to the organizers!!
Etty Hoesein
01:41:42
Thanks for sharing!!👏👏👏
Ezechiel Hategekimana
01:41:46
Thank you all!
Moazzem Hossain
01:41:50
Thanks great panelists andeveryone! Excellent presentations
Zaida Cabral
01:41:55
Thank you so much for the interesting webinar.
Cledenin Veras
01:41:57
Thank you all!
saima malik
01:41:59
Great webinar. Thank you to BEC-GRN and the amazing panelists!
annette Kiberu
01:42:06
Thank you for the webinar
Alice Eastwood
01:42:06
So interesting! Thank you all!
Renan Rapalo
01:42:06
Thank You!
Chala Legesse
01:42:13
Thank you very much!!!!
Joan Mwachi
01:42:21
Thank you. These sessions are very informative.
Eve Brooks
01:42:28
Thank you for this webinar. This is very helpful information.
Lilian Tzul
01:42:28
Thank you from Guatemala!
Swadesh Maharjan
01:42:30
Thank you ...
Khalilur Rahman
01:42:35
Great Webinar. Thank you all. Stay safe.
Raj K. Dulal
01:42:36
thank u
Meron Mamo- Creative
01:42:36
Thank you
Umuhoza Pascasie
01:42:38
Thank you so much!
Melindah Chuma Michelo
01:42:38
Thank you.
Andres Navarrete-Berges
01:42:39
Thank you everyone! Extremely relevant. Stay safe!
Sulleiman Adediran
01:42:40
Thanks for stimulating discussion
Olayinka Olagbegi-Adegbite
01:42:40
Thanks everyone!