Under a funnel approach, a program implementor casts a wide net and offers a basic level of services to many people, and then extends more intensive and expensive services to progressively smaller subsets. This was the approach used in the IFC’s Sourcing to Equal program in Colombia, designed to build the capacity of women-owned-or-led SMEs to bid for and win procurement contracts from large corporations. The funnel here has three stages:
1. A 2-hour online webinar: this provides a general overview of corporate procurement requirements, processes, and opportunities – and can be delivered at large scale to many firms (offered to 1,105 firms in our setting, with 612 taking it up)
2. Bootcamps: two half-day virtual sessions and one 2-day in-person session. These go into more depth, providing case studies, legal considerations, how to build a B2B value proposition, working capital solutions, negotiation skills, advice from buyers, putting together a pitch and proposal, etc. These are more expensive and intensive, but still delivered in a group setting, so not as much as the final stage. 399 firms attended the bootcamps. The combined cost of the bootcamp and masterclass was $628 per participant, with most of this for the bootcamp.
3. Mentoring: the most intensive phase, consisting of three group sessions, three individual sessions (10 hours). Here mentors developed customized workplans for the needs of each firm and helped them prepare for and bid for contracts, e.g., by designing measures to show a buyer they are capable, working on how to structure client engagements, and helping through the vendor registration and onboarding. 120 firms received mentoring, at a cost of $819 per participant.
This type of funnel approach is being used in other business support programs such as this one in Malawi, and also applies in many other settings: e.g. education programs that provide basic support to everyone, and then more intensive support to a subset; or job search assistance and counselling in which everyone gets a basic level, and then more help is given to those revealed to need more.
However, these programs raise several challenges for impact evaluation. First, because of the desire/need to draw many units into the funnel, you need a large sample which may make it hard to use an oversubscription design – in Colombia 1,105 was the total number of firms we could get after extensive outreach and advertising. Second, normally in impact evaluations we aim to minimize low take-up, given its implications for statistical power. But in funnel programs a reduction in take-up is a feature, not a bug – the funnel is used to reveal information (e.g. about who shows up, who demonstrates promise) that then determines where to focus more resources.
Evaluating funnel programs
In a new working paper with Rachel Cassidy and Kalyan Neelamraju, we evaluate the Sourcing to Equal program in Colombia, and use this to show how to address these issues and make progress on learning more from funnel programs.
Phased-in RCT
First, to deal with the issue of how to still conduct an experiment and have a control group when our entire sample of firms had to pass through the funnel, we used a phased-in RCT, randomly assigning firms to get the program in a first, second, or third wave. In each wave firms went through their respective funnel. We then conducted a midline survey in which wave 1 is treated (for 6 months), and waves 2 and 3 serve as control, and an endline survey in which waves 1 and 2 have been treated (for 12 and 6 months), and wave 3 as the control. The downside is this limits measuring longer-term impacts. An upside is that because we observe which firms get through to different phases of the funnel in later waves, under an assumption of temporal stability in funnel pass-through, we can then e.g., compare treated firms who get to the mentoring stage with control firms who subsequently do.
