Essay 12
Recognizing the Power of Foundational Learning Data
By Silvia Montoya
Introduction
A child’s first steps are always the most difficult. As children begin to acquire the coordination needed between their two feet, the fact that the two feet each have the same name, foot and foot, makes it difficult to explain that you must not raise “the foot and the foot at the same time.“
In his short story "Instructions for Climbing a Staircase," Julio Cortazar used 381 words to describe what many of us would do using no more than 20 words, proving how difficult is to give instructions for an action that most adults perform unconsciously. As Cortazar did in his essay, Girin Beeharry calls for building on the foundational stages as the key to success, asking us to follow a long set of instructions to reach a successful ending.
In my view, this effort will only succeed after national, regional, and global stakeholders in Sustainable Development Goal 4 (SDG 4) enhance their cooperation; only then will it be possible to eliminate data gaps, monitor national learning levels, and monitor the implementation of missions responding to the review of progress on learning outcomes. Regular progress monitoring through the use of data will be the linchpin of reform. Regular monitoring will lead to the development of education initiatives based on data and help identify where investment is needed. Comparative data on progress to improve learning levels could lead to higher-quality policy dialogue and resource pledges and disbursements.
In this essay, I focus on the relevance of data, the factors that hinder data availability, and the needed actions and principles that can move us forward.
Good quality education data is critical to improve education outcomes
Since 2015, the SDG 4 monitoring framework, with its global and thematic indicators, has helped set a measurement agenda in education that includes the measurement of foundational learning through indicator 4.1.1. However, institutional capacity and financial constraints, related to, and partially caused by, insufficient demand-side pressure has meant that the international community (and more specifically, countries themselves) still lack a complete set of frequent data points from all countries to monitor levels or learning and trends. Ensuring such data are available would help set and, more importantly, track benchmarks to better assess progress on key policy areas.
Data on learning outcomes enables education leaders to understand which learners are making progress, which are not, and why. At the global level, learning outcomes data helps identify global trends, assist countries, and drive mutual accountability. Unfortunately, policymakers in the least developed countries often do not have access the right data.
The impact of the COVID-19 pandemic exacerbates these issues. At its peak, the pandemic impacted an estimated 1.5 billion learners and over 63 million teachers,2 and led to the cancellation or postponement of numerous tests or assessments, with a quarter of low- and lower-middle-income countries not tracking children’s learning at all in 2020.
Any solution to the data crisis should aim to support countries’ student assessment systems that to produce regular and comparable data for early primary, end-of-primary, and end-of-lower-secondary learning. Such data is required to enable policymakers to develop evidence-based policies to improve teaching and learning; to ensure reporting on SDG 4.1.1, with an initial focus on SDG 4.1.1 (a), (b), and (c);3 and to help monitor voluntarily-set national benchmarks. This objective will be achieved through the following activities:
- Strengthening national learning assessments, as well as related analysis and reporting frameworks, and supporting participation in cross-national assessments
- Strengthening capacity and capabilities in countries to design and implement learning assessments as well as to analyze and use the generated assessment data
- Improving coordination, oversight, and transparency of global efforts to use resources efficiently and effectively in supporting countries, notably by developing a coordinated plan for learning assessment data collection; “brokering” between countries, donors, and providers; and robust monitoring and evaluation with a focus on both data availability and national capacity development. Capacity-development efforts should fund the establishment of national evaluation agencies and training people abroad.
As Girin’s paper rightly points out, one crucial factor often forgotten is that these activities have to be tightly coordinated or have at least one point of very senior-level contact with activities aimed at improving learning outcomes (and improvement on other indicators, although those are to some degree already somewhat coordinated, e.g., through the Global Partnership for Education (GPE) Board).
Failures in the learning assessment market
Data for learning outcomes indicators are provided via a market that includes data producers; data consumers (countries, policymakers, international agencies, and researchers); and goods and services exchanged for money (prices) to produce the learning outcome data. However, not only are all the conditions required for a market to function violated in the market for learning assessments, but some market characteristics are diametrically opposed to those needed for efficient and equitable functioning, leading to actions that are always implemented in the presence of market failures. Rules for a well-functioning market are listed below, along with explanations about how the data market violates or complies with those rules.
1. There are many producers and consumers competing with each other over the same uniform and undifferentiated product
But with learning assessments, there is product differentiation. In fact, no important “product” sold in the learning assessment market is the same as any other; some assessments are about skills needed for the labor market, others are curriculum-based; some are designed for primary education, others focus on lower secondary; some are citizen-led, others are government-led. And so on.
2. Consumers have all the information they need about the products, including details on how they are produced (in economists’ jargon, there is no asymmetry of information)
The current lack of information means that countries must do their own analysis to evaluate the options. At a minimum, they need to understand how international assessments can be used to support the national agenda, to report on the SDGs, and to help their ministries make management decisions.
Finding the right fit can be difficult, which is why they need better guidance. For example, a country might be thinking about participating in a costly international assessment, even though it has a national assessment covering the same education level. To evaluate its options, a country needs to compare the overall costs and benefits of the decision, as well as more technical information to decide on the nitty gritty questions such as items and constructs. As with any expensive product, countries need a tool that allows them to compare the options by providing accurate and objective information.
3. The production techniques are known to all and can be copied
One would expect that with 25 years of intensive experience of preparing cross-national assessments, it would be straightforward to copy this process. Nobody argues that absolutely all information should be in the public domain, as it is important to ensure assessments are valid and keep certain aspects confidential. But the process of producing assessments, and the background knowledge needed, has been sufficiently standardized that the current high costs are not justified.
4. There is no price discrimination and prices are transparent and uniform
In the current data market, consumers often must haggle, like at a roadside vegetable stall, rather than choose their products at a market with prices clearly posted. While the haggling can lead to better prices for some, there tend to be hidden expenses and higher transactions costs, and maybe higher prices for others. We are all for negotiations, but they should be based on transparent pricing information.
In the learning assessment market there is price discrimination, but not in a positive-equity based way that can lead to better prices for the poorest countries. There is some negotiation on price and different levels of subsidies, and there is also intermediation. Prices in many cases are negotiated between third party payers (e.g., development partners) and the producer.
An official body could produce accurate, comprehensive, and up-to-date information on the current costs of assessments, while documenting steps or conditions that can help countries negotiate. And some of the information is actually contradictory. For example, countries are often led to believe that by joining an international assessment they will benefit from economies of scale. Yet why is it that the fees never seem to go down as the pool of participants grows?
5. There are no significant economies of scale or barriers to entry
In the learning assessment market, there are significant barriers to entry for possible competitors because it is costly to build a set of good learning assessment questions. New providers typically emerge only to provide a differentiated product. For example, there are assessments serving different geographies (such as initiatives in East Asia) or offering different ways of administering and engaging with the community (citizen-led assessments) as well as different education levels (e.g., the Collegiate Learning Assessment, a higher education standardized test in the United States).
Shaping the learning assessment market
When markets are inefficient, there is a “market failure” that can justify public or collective action. One possibility is the takeover of those functions by someone else. Another approach is “market-shaping”: the public sector can induce markets to be more efficient. Tools for shaping the market include:
1. Produce a consumer guide on the different types of assessments
One objective is to systematize market information by publicizing more information for users about which assessment is fit for their purpose, what is a reasonable price to pay, and so on. Various public agencies, such as the World Bank and UNESCO’s Institute for Statistics (UIS) have published such guides. More could be done, especially regarding price information and the dissemination of this knowledge. UIS, the World Bank, and other institutions have already taken some steps forward, and there have been open discussions through the Global Alliance to Monitor Learning and the Technical Cooperation Group.
Pricing information should, at a minimum, include the fees paid to the international assessment organization; opportunity costs of the technical knowledge required within national agencies; financial costs of field deployment; and the opportunity costs of field deployment if existing staff are used.
The second objective of a consumer guide is to provide more transparent price information about the cost of participation in cross-national assessments. Pricing information should at a minimum include the fees paid to the international assessment organization; the opportunity costs of the technical knowledge required within national agencies; the financial costs of field deployment; and the opportunity costs of field deployment if existing staff are used.
2. Develop methodologies to link national assessments to a global minimum proficiency level for global reporting
If countries and development agencies knew more about how one assessment “translates” into another, they would not feel the same pressure to “buy” every possible test in addition to their own national assessment. Countries can better evaluate the options and their relevance to policymaking if they can see the level at which an assessment is linked to a global yardstick. If one knows how to translate meters into yards, one does not need two measuring sticks. This information will also help to boost the technical skills of national staff while also supporting development partners that fund assessments as part of their contribution to SDG 4.
The UIS, through the Global Alliance to Monitor Learning, has worked intensively over the last five years to develop tools and linking strategies to allow reporting that uses all available sources of information. This has led to the definition of Minimum Proficiency Levels, the development of a Global Proficiency Framework (reading and mathematics), a Linking Strategy Portfolio based on a student-based linking through the Rosetta-Stone project, the Policy Linking Methodology to align national assessments, and, currently, the SDG4 test developed within the Monitoring Impact on Learning Outcomes project in Africa. An interim reporting strategy to increase reporting during the development phase, a Protocol for Reporting, and a detailed metadata complete the support to countries reporting. The UIS has also produced a proposal to report a standardized measure for SDG4 in 2017 that was used to estimate the number of children not learning and served as a basis for the World Bank Harmonized Learning Outcomes data.
3. Transfer and build knowledge in the Global South through a set of actions to create free information on how to craft a decent assessment, laying out technical policies and procedures for assessments, such as for citizen-led assessments or early-grade assessments.
An item bank of assessment questions that work well, and information on how to put such questions together, might be an example. Item banks can help to build sensible assessments that can be used for international reporting to deal with the technical barriers preventing producers (including countries themselves) from entering the market. One step towards generating capacity in the countries and lowering costs is the use of artificial intelligence applied to a bank of items properly aligned to the different education cycles of countries. The platform would embed a bank of questions with known and tested technical properties. The approach would also suggest ways to combine items in a sound manner, as needed for reporting. Countries would be able to get and contribute items, to build the whole assessment if needed, to run the modeling that produces the results without having to commit themselves to packages where their ownership and voice is reduced. A machine learning engine would, in an ambitious scenario, allow adaptive testing at low cost. Similarly, one way forward would be to develop a pool of items from existing assessments, as a free global public good that would empower national authorities to develop their own assessments. The item bank and associated software tools could help new providers, or countries who would self-provide, similar to how the global community has helped generic drug manufacturers develop products.
Is the learning assessment market equitable?
A market can have a major equity problem and still not be considered a failure. A market may simply not work for those without purchasing power. The problem with the learning assessment market is that all the conditions are violated and some characteristics are the diametrical opposite of those needed for efficient and equitable markets.
To make the learning assessment market more equitable, international education stakeholder may want to consider:
1. Make available resources to participate in cross-national assessments, and develop national capacities that are allocated unequally, transferring purchasing power to those who need it, either because they are very poor or simply because they need the behavioral nudge. But such subsidy mechanisms must be transparent, explicit, and designed carefully.
2. Transfer purchasing power and technical skills to countries. Assessment costs are high, relative to other discretionary quality assurance mechanisms, for poorer countries. They may need subsidies either to cover the cost of both participation and technical skills development, or as a behavioral nudge. As the price of cross-national assessments becomes standardized and transparent, international agencies should ensure grants are made available to countries with a long-term perspective, to ensure all countries benefit equally according to their need. The design of such a scheme is non-trivial and needs to be thought about carefully so as not to create further inequalities or perverse incentives (e.g., pretend to a lack of interest so as to merit the behavioral nudge).
3. Increased transparency of donor’s investment to improve the availability, reliability, and quality of learning assessments. Efforts by donors are undermined by the lack of transparency, coordination, or readily available data on donor support to these areas. As a result, the is no clarity about the value of donor support for learning assessments. Specifically, there is no readily available, up-to-date, or reliable data on the scale, focus, or direction of donor funding. This situation presents four main challenges to the global education community:
- There is no understanding of the full extent of donor support to the learning assessment agenda, at global, country, or donor levels
- There is no identification of the existing gaps (either thematic or financial) in support
- There is a lack of coordination of support across donors and countries leading to duplication, inefficiencies, or increase in country’s burden.
To resolve these challenges, UNESCO UIS and the World Bank have partnered in a Virtual Register for donor financing of learning assessments. The coordination in funding and new investments would function as a virtual fund where funding pledges and disbursements, as well as technical pledges, are brokered, coordinated, and tracked so that there is accountability. It is expected that funders would continue to provide support, and recipients continue to receive support, but in a more efficient, increased, intentional, and coordinated manner. It is estimated that this initiative would require about $259 million over the 10-year period from 2020 to 2030. This equates to an estimated $182 million more than if current estimated donor spending was maintained during this 10-year period (approx. $77 million). Given the significant lack of available financial data from donors and assessment providers, this costing is based on high-level estimates, built on initial assumptions and best-available data.
Towards a country-led, transparent, and efficient way
The discussion above is an important one for anyone working on SDG 4. The global desire to make progress on learning, and not just access, is a central pillar but may also have, perhaps unwillingly, contributed to the market we find ourselves with today. But markets don’t care about justice or equality; they are rarely set up to work for those without purchasing power. Something must be done, therefore, to level the playing field and the needed actions demands cooperation and duties from every actor if we do not want to repeat the same discussion of why we lack the basic inputs to achieve the outcomes.