By Linda A. White, Sumayya Saleem, Elizabeth Dhuey & Michal Perlman
Abstract
A growing number of academic studies and policy reports have identified a set of core skills considered crucial in the twenty-first century economy. This article critically examines the evidence base underpinning that ideational consensus among international organizations (IOs) and global management consulting firms (GMCFs). We collected 234 skills reports produced over the past decade by major IOs (European Commission, ILO, OECD, UNESCO, and World Bank) and GMCFs (BCG, Deloitte, Ernest and Young, KPMG, McKinsey, and PWC). We then extracted bibliographic references from each report and used the analytic technique of citation analysis to examine how the consensus around these core skills was generated in order to uncover the authoritative sources of knowledge and the pattern of ideational policy diffusion observed. Our analysis reveals substantial gaps in the evidence base used. Evidence drew largely on a few academic economists, along with strong use of grey literature, and high rates of self-citation. Given these characteristics, the consensus around twenty-first century skills appears less epistemic in nature and more like an ideational echo chamber, which raises concerns about the extent to which policymakers should rely on this evidence.
Conclusions and directions for future research
Our analysis reveals substantial gaps in the evidence base used in influential reports about the future of work. Rather than a diversity of scholarly sources and empirical bases to inform the consensus around twenty-first century skills, we find a concentrated set of sources, the strong use of grey literature, and high rates of self-citation. Given these characteristics, the consensus around twenty-first century skills appears less epistemic in nature and more an exercise in fast policymaking (Peck, 2011; Peck & Theodore, 2015).
We focused on applied/policy reports as our starting point and then traced the evidence base used in these reports. This does not speak to how much evidence exists, but rather only reveals what evidence is used. One area for future research is to conduct a similar citation analysis starting with the existing scholarly work in this area. It is also worth noting that we focused our analysis on a small number of papers that were cited ten or more times. This left more than 9000 sources that were rarely cited which we did not analyze in depth.
We also do not speak to the quality of the frequently cited research. Rather, in a companion paper (Saleem et al., 2022), we examine the quality of the sources being cited by IOs and GMCFs. We find that the quality of citations (operationalized as the number of sources being cited, rates of self-citation, and utilization of scholarly in comparison to grey literature) is significantly better for reports published by IOs compared to GMCFs. These findings suggest that there may be substantial variation across different types of organizations in the scope of the evidence that informs their claims, self-confirmatory biases and utilization of critically evaluated knowledge.
Another area of future research is to investigate the specific content of these reports and process traces the effects of these ideas (Jacobs, 2014) through systematic analysis of where these ideas go. For example, to what extent are these ideas taken up by domestic policy actors such as ministries of innovation, employment, and education. Such a study would build on research by scholars such as Durazzi (2019) who has found that in countries which assign a “strategic role to advanced manufacturing—and where employers have rather narrow skills needs—governments intervene more directly and actively to ensure the availability of STEM skills compared to countries pursuing knowledge-based growth centred on high-end services” (p. 1803; see also Durazzi, 2021). It would be fruitful to examine whether similar factors contribute to varied national uptake of IOs’ and GMCFs’ future skills reports.
Despite the limitations of the current research, it does allow us to draw a number of important policy conclusions. First, policymakers must be aware that the influential reports in the area of future skills are based on a very thin evidence base. Furthermore, the evidence that is reported in terms of academic sources is based on a small number of articles, written almost exclusively by male economists. This is especially alarming given that education and employment are areas that lend themselves to research conducted by researchers from different backgrounds—disciplinary and otherwise. Research tends to reflect the interests of researchers, and the lack of voices of women and people of color in the work being cited can lead to gaps in knowledge. May’s (2014) research, for example, based on a survey of American Economic Association (AEA) members found that female economists are more likely to favor government intervention; regard gender inequality as a problem; support the idea that employers should be required to provide health insurance for their employees; and support income equality, among other measurable differences (see also Hendengren et al., 2010; Kamas and Preston, 2019). May et al. (2018) found similar results in a survey of European economists. Given that this work is informing the reports of influential IOs and GMCFs, it is essential that they rely on scholarly work from a broader range of academic sources. Whether this reliance on such a small number of papers from such a unitary perspective is because no other literature exists, or whether this is a bias in what is cited, is an important question that needs to be explored.
Another alarming pattern that emerges from our analysis is the trend away from citing scholarly work with a concurrent reliance on grey literature. Since 2016 we can clearly see that influential reports in this area cite grey literature more often than they cite scholarly work. Given the greater quality control imposed on the scholarly literature than grey literature this is clearly of concern. As governments and other agencies are growing increasingly concerned about the issue of the skills needed to succeed in a new landscape, it is important to based reports used by governments on empirical/scholarly work.
A related issue is the heavy reliance by both the IOs and the GMCFs on self-citations. This tendency was substantially stronger for the GMCFs who cited their own work an average of 38% of the time compared to the IOs, for which it was 18% of the time. Since 20% or more in self-citations is considered high by scholarly standards (Epstein, 2007, p. 133; Aksnes, 2003; Mavrogenis et al., 2010), even the IO reports are approaching a rate considered high. To our knowledge, there is no standard for appropriate levels of self-citation in the grey literature. While we are not equating an academic citing themselves to a consultancy citing itself, we use the 20% standard as the one that is available. Future research should establish what are acceptable levels of self-citations in ILOs and GMCFs. Some level of self-citation makes sense as individuals and organizations with expertise in a given area would naturally cite their own earlier work. The risk, however, of high levels of self-citation is that it creates an echo chamber where the producers and readers of the work only encounter information that is consistent with opinions they already hold. The higher rates with which this occurs in GMCFs may not be surprising given that these are for-profit organizations that may use their reports as marketing tools to draw future business.
These findings serve as a call to researchers and policy analysts to ensure that there is more comprehensive evidence reflected in the reports used by government and other agencies to guide and inform policy. Looking back at the past 20 years of writing by IOs and GMCFs in this area, this has clearly not been the case. These findings have implications in terms of the influence of global actors and organizations on domestic policymaking. The OECD, for example, is clearly quite influential in this policy area based on its work as a knowledge producer, an activity which fits the core mandate of the organization. But, if IOs and GMCFs simply emulate the knowledge produced by a small number of actors and organizations, and the overall quality of those reports in terms of the evidence base used and the degree of self-referencing that can be found, then we must conclude that ‘fast policy’ needs to slow down.
