Socioemotional skills – those we use to manage our emotions, collaborate, and strive toward goals – are central to impactful entrepreneurship programs, vocational training programs, and safe space clubs for youth and adults. Yet capturing improvements in these latent skills is hard and telling which skills drive economic empowerment and mental health is harder. Many studies rely on a few familiar concepts (self-efficacy, locus of control, grit) that reflect neither the range of socioemotional skills nor the skills the intervention teaches, often bundled into a single “noncognitive skills” index.
The World Bank’s Africa Gender Innovation Lab (GIL) and Innovations for Poverty Action (IPA) partnered to tackle this through the ESTEEM project (Effective Socioemotional skills To gain Economic EMpowerment) and associated research agenda. An earlier post sums up the agenda and findings so far. Here we focus on the nuts and bolts: how we developed self-report and behavioral measures of 14 skills for randomized evaluations in Côte d’Ivoire, Nigeria, and Tanzania, what we dropped along the way, and what we would tell a team adding socioemotional skill measures to its next survey.
Which skills to measure
We started from a review of 81 economics and psychology studies, including how skills differ by gender and relate to labor outcomes, and drew on experts, focus groups, and the CASEL and Goleman emotional intelligence frameworks. The result is a list of 14 nonoverlapping skills for youth and adults, grouped into awareness vs. management and intrapersonal vs. interpersonal skills (Figure 1). Note that some items from these measures also map to a measure of generalized self-efficacy. Also, other soft skills concepts such as beliefs and motivation were purposefully excluded. Detailed definitions and the theory of change can be found here.
How we built and tested each measure
For each skill, we developed a self-report and at least one behavioral assessment, so we could compare the two. Working with psychologists, assessment experts, and field coordinators, we took each measure through these steps across pilots and the three evaluations.
We began by reviewing existing measures to assess their fit with our skill definitions, links to labor outcomes, and evidence of validity in Africa or elsewhere. We also evaluated feasibility, prioritizing measures that could be administered in under five minutes per skill, used offline in household surveys, and limited in their overlap with other measures.
To establish face and content validity, we used expert review, focus groups, cognitive testing with 48 people across the three countries, and careful translation reviewed by a psychologist using this rubric (p. 6). We assessed reliability and validity using inter-item correlations (>0.2, <0.8), Cronbach’s alpha and McDonald’s omega, factor analysis, item response theory, and associations with employment, decision-making power, and mental health. We treated standard cutoffs (e.g., alpha ≥ 0.7; CFI ≥ 0.95; RMSEA ≤ 0.08) as rules of thumb, retaining items essential to content validity when a scale fell just short. We also iteratively improved measures, revising and removing items across rounds of data collection.
Coverage was incomplete in two countries: while we collected for both measures and all skills at multiple time points in Tanzania, COVID-19 limited baseline data collection in Nigeria, where we also collected only 7 self-reports at endline, and in Côte d’Ivoire we collected abbreviated behavioral measures (using only one scenario) for only 9 skills.
Self-reports: cheap and familiar, but prone to bias
Self-reports aggregate answers to a few survey questions (“items”) about how people see their own thoughts and behaviors (e.g., “I plan tasks carefully,” on a 5-point Likert scale). They are cheap and easy to add to a survey, but answers can depend on the setting respondents have in mind (household, work, community), who they compare themselves to, or the time period they consider. Respondents may agree with everything, give the obviously “best” answer, or lack the self-awareness to answer accurately. And behaviors that signal a skill in Western contexts may not apply elsewhere (eye contact as a sign of listening, for example).
Many self-report measures already existed, especially for intrapersonal skills, but few had been validated in African settings. Because existing measures often combined multiple interpersonal skills into a single construct, we worked with a psychologist developed several new interpersonal scales.
Two lessons emerged from this process. First, reverse-coded items did not work. We initially mixed positively and negatively framed items to detect and control for acquiescence bias, but respondents found the variation confusing, and the two sets of items loaded onto separate factors, a pattern also documented in other studies, particularly when translated. Second, a visual response scale, designed to reduce cognitive load, saved time. When shown an image of the scale, respondents could more easily remember the full range of options and quickly point to their answer.
The final self-report items and instructions (7 to 14 items per skill) are publicly available, as is the paper on their development and validation.
Behavioral measures: what we kept and what we dropped
Past Development Impact posts have covered group exercises in Jordan, number games in Türkiye, and EEGs, facial expression readings, and games in Central America. We tested four types: performance tasks, situational judgment tests, group activities, and household informant reports (see the paper on behavioral measures (Table M1) for all those we considered).
Performance tasks. At first, we saw performance tasks as the gold standard. But many took more than five minutes or could not run offline, and a task often captures only one facet of a skill, so its content can differ sharply from a self-report measure of the same skill. Short tasks are also harder to validate, since reliability and construct validity usually require repeated attempts at the same or related tasks. We kept continuous performance tasks for self-control, a triangle counting game for perseverance, and listening comprehension prompts. Enumerators also coded active listening, attention during the survey (self-control), and open-ended answers on respondent self-awareness and how respondents would solve problems and make decisions or influence others.
Situational judgment tests (SJTs). We also developed situational judgment tests, in which respondents indicate how they would act in hypothetical scenarios. By focusing on concrete situations rather than abstract judgments about oneself, SJTs leave less room for interpretation, reducing domain ambiguity and reference bias. The desirable response is also often less obvious.
Several design choices strengthened the SJTs. We developed scenarios through focus groups and drew response options from interviews with entrepreneurs and employees, socioemotional skills curricula, and self-report items, helping maintain common content validity across measure types. Rather than asking respondents to rank actions, we asked how likely they were to take each action, reducing cognitive load and literacy demands. We also developed separate household and workplace scenarios for each skill because behavior can vary across settings. As with the self-reports, the response scale was displayed visually.
Respondents liked the SJTs: they held their interest and were easy to understand. The tests also adapted well across contexts. Changing details such as businesses, currencies, and names made it possible to localize a scenario while keeping the response options broadly similar.
Group activities and household informant reports were dropped after testing with large samples, mostly for feasibility reasons. Replicating the group exercises used in Jordan (e.g., redesigning a failing amusement park) meant costly logistics to assemble participants and evaluators, and people behaved differently depending on their group, making individual skills hard to pin down. For informant reports, the household member who knew most about the respondent’s time use and income-generating activities answered the externally observable self-report items. Finding and waiting for informants slowed the survey or led to missing data, and the data quality (judged by correlation with labor outcomes) did not justify the cost.
Does the choice of measure matter? Yes
Behavioral measures we use take about twice as long to administer as self-reports. Self-reports required 2-3 minutes per skill, whereas performance tasks and SJTs required 4-6 minutes. Both types of measures also require extensive cognitive interviewing and enumerator training when introduced in a new setting.
Measures intended to capture the same skill often disagree. Correlations depended more on the type of measure than on the skill being assessed, and two measures of the same skill were often only weakly correlated, with coefficients no higher than about 0.4. The choice of measure can also produce different conclusions about gender gaps. Among jobless youth in Tanzania, young men scored about 0.2 standard deviations higher than young women on self-reports, significant across skills, whereas behavioral measures revealed only small gaps of 0.08 standard deviations or less, significant for only a few skills. This suggests that self-reports may overstate both men’s socioemotional skills and the corresponding gender gap.
The two types of measures also predicted different outcomes. Across all three countries, both were significantly correlated with labor market, mental health, and well-being outcomes, although the results were highly context dependent. Updated results reveal that behavioral measures varied more across skills and tended to be more strongly associated with labor force participation, decision-making power, mental health, and gender norms. Self-reports were more strongly associated with income aspirations, while findings for income were mixed. Relationship management, problem solving and decision making, and emotional regulation were often among the strongest predictors of outcomes. One important caveat is that socioemotional skills may not correlate with economic outcomes in contexts where work, or certain types of work, is not considered a desirable goal.
Finally, changes in behavioral measures tracked changes in outcomes over time. In Tanzania, where most measures were collected at several points, changes in behavioral measures of socioemotional skills significantly predicted changes in employment, hours worked, decision-making power, and well-being. Changes in self-reports performed better only in predicting reported job search.
