Learning From History: A Database of Prior Reform Proposals at State
By: Toby Weed | December 11, 2023
Click here to explore the interactive database of reform reports in a new window.
Practitioners and theorists alike have advocated for an institutional upgrade of U.S. foreign policy for decades, resulting in spasmodic waves of think tank reports, commissions, and committees on the topic. Left in their wake is an extensive backlog of recommendations that are too often ignored or quickly forgotten. Unfortunately, too many would-be reformers proceed without reviewing the large body of work on State Department reform that has accumulated from prior attempts.
This mirrors a broader problem in foreign policy. Without adequate knowledge management structures in place to support policymakers, we are doomed to repeat history’s lessons. fp21 has thought perhaps more methodically on this subject than any other organization, so we decided to leverage our expertise and set down an organized foundation for future research and advocacy to build upon.
fp21 created this database to be a resource for those working to reform US foreign policy process. We have thus developed a structured, annotated database of the most important prior work on State Department reform. Documents were carefully selected from a much broader body of literature based on scoping criteria intended to identify the highest-quality and most relevant work. The result is a set of resources curated specifically to be useful for those who seek to understand institutional U.S. foreign policy reform.
Even as US foreign policy reform is urgently needed, any reforms need to be carefully researched and based on the best available evidence. In addition to being a resource for designing reforms and seeing what has been proposed before, this dataset facilitates research into why certain reforms get implemented while most do not.
fp21 will continue working to improve this database. We have begun adding topic tags by hand to make it easier to identify documents most relevant to a particular reform. We have added tags to 17 documents. This subset has already covered 31 of the 35 possible topics we have identified; the most common topics are Interagency (7), Funding (6), Foreign Assistance (5), Organization Structure (5), and Training (5). Future additions may include a ranking of evidence use and a description of the target audience for the recommendations. This database builds on our pilot study of classifying individual recommendations to make that research process easier. Collecting the vast body of recommendations also makes it easier to see what areas and ideas are relatively underdeveloped.
This database is part of a suite of tools to facilitate reform. In the coming month we will release reports on reform legislation, foreign policy authorities, and foreign policy actors. All of these tools are in pursuit of fp21’s mission for a US foreign policy that makes the best use of evidence, learns from our success and failure, and applies those lessons to create a skilled, diverse, merit-driven staff.
fp21 collected 97 reports, books, op-eds, and other documents related to foreign policy reform and organized them into a dataset with details such as their date, type, and source. We also created a summary for each document using AI. The dataset can be accessed as a user-friendly Airtable database, Zotero database, or downloaded as a CSV file. The process for identifying, including, and coding is described below in the section on “Database Collection Methods.”
We find that there have been at least one or two significant publications on State Department reform almost every year for the last two decades, with many more in some years. This suggests sustained, continuous interest in the topic.
The bulk of the database is made up of 72 long-form pieces (66 formal reports and 6 books or book sections). These reports span a variety of topics and levels of specificity, including full-length, comprehensive reports specifically on State Department reform (e.g.) alongside reports that are focused on subtopics (e.g.) or more general ideas (e.g.). These reports are intended to be a mostly comprehensive sample, representing most of the major public research reports on State Department reform and closely related topics published within the last few decades.
The database also includes 22 op-eds from foreign policy venues such as The Foreign Service Journal or Foreign Affairs. These were selected because they contain interesting commentary on what we judged to be the most salient topics to foreign policy reformers. This collection is not comprehensive, however. The op-ed category should be regarded as an interesting sample that can be used to stimulate discussion and inspire further work.
There are 52 distinct publishing organizations represented in the database. The most common are The Foreign Service Journal with 9 entries, the American Academy of Diplomacy with 8 entries, and the Center for Strategic and International Studies with 7 entries.
Just over half of the documents are from think tanks and advocacy nonprofits. These are the institutions that are primarily responsible for carrying out the public conversation on this topic. There have also been a healthy number of reports coming out of government initiatives, for instance, prior commissions and legislative committees. It is worth noting that in our experience, government-sourced documents are much harder to find via traditional academic research methods. It is thus likely that there are more highly relevant government-produced documents that are not easily accessible via the internet and thus are not currently included in the database.
Database Collection Methods
Collection Methods
Candidate documents were gathered in 3 stages:
Leverage fp21’s existing expertise
A seed set of documents was collected from the sources used in fp21’s prior work on Unlocking Foreign Policy Reform recommendations. This Drive folder contains the candidate documents.
The seed set was expanded using documents from fp21’s existing Zotero resources.
This seed set was passed around internally, and fp21 employees and collaborators suggested additional documents to fill holes.
Using search engines
The Find Policy foreign policy think tank search engine were queried (“State department reform”) and (“State department” AND “reform”). All available pages were scanned for in-scope documents.
The Policy Commons search engine was queried (“state department reform”) and all 16 results were scanned.
The Harvard Kennedy School’s think tank search was queried (“state department reform”) and the first 10 pages were scanned.
Snowballing
Survey-level and particularly comprehensive or landmark documents were selected from the existing database. More resources were then identified from the citations and bibliographies of these documents.
In each of these steps, each candidate document was assessed for inclusion using three criteria:
Formality. Documents were only included if they represented finished, published work, rather than, for instance, internal Google docs or work-in-progress documents.
Applied orientation. Documents needed to be oriented to practical reform, rather than theory. For instance, academic work on performance management theory would not be included.
Relevance. Documents were required to have a high degree of explicit focus on the institutions of American foreign policy. For example, a document focused exclusively on reforming the Department of Defense would be deemed too off-topic.
The fp21 Zotero contains detailed records of which documents were obtained in which rounds. Please reach out to request more information.
Variable Details
The 9 metadata fields are:
Title
The document’s full title.
URL
A web link for the original document. If a full-text copy is not available online, this may link to the press release, Amazon page, or another descriptive resource.
Author
The main author or authors listed in the document. These are human names when those are available, but occasionally the name of an institution or organization is used.
Date
The year in which the document was released or published. There are at least one or two significant publications on State Department reform almost every year between 2000 and 2023, with many more in some years.
Organization or Publisher
The organization that was primarily responsible for producing or disseminating the document. In the case of major research reports, this is usually a think tank, but it may also be a government office or commission. For op-eds and journal articles, this is the publication venue. In the case of books written by independent authors, this field might be filled by the book’s publisher.
Summary (from ChatGPT)
A machine-generated summary of the full text of the document. These summaries were obtained by:
Extracting the full text of a subset of the PDFs using the R code in this Github repository.
Using Relevance.ai to enter each body of extracted text into GPT 3.5 along with the following text: ‘Write a one-paragraph summary of the text above. The summary should be aimed at a foreign policy expert audience. Note the most important specific policy and/or reform recommendations made by the document, if there are any. Please start the first sentence with an action word like "Discusses", "highlights", "recommends", etc..”’
Some of the documents did not have easily accessible, well-formatted PDFs. In those cases, this field is left empty. Since this field is machine-generated, it should not be taken as authoritative or complete. This is merely intended as a useful way for readers to quickly scan through documents.
Contains Recommendations
This Boolean field (TRUE/FALSE) indicates whether an easily identifiable portion of the document is dedicated to clearly enumerating policy recommendations. Note that if the Document Type is anything other than “Report”, this parameter is automatically set to FALSE. This field exists because a subset of the database is made up of reports with the express purpose of providing concrete policy advice. This particular subset might be useful for further research oriented around understanding what previous work on reform has advocated for, so this field was created to make that subset easily identifiable.
Document Type
The type of document, from the following 5 options:
Op-Ed
Report
Book
Book Section
Misc Document
Source Type
The type of organization responsible for the production and dissemination of the document, from the following 7 options:
Think Tank
Nonprofit
News Media
Government Committee
Government Commission
Government Office
Academic
Topics
The topics that the document addresses, from the following 35 options: Accountability, Analysis, Assignment, Authority, Clearance Process, Comprehensive, Consular, Data Collection, Decision-Making, Diversity, Economic, Embassies, Evaluation, Evidence Use, Forecasting, Foreign Assistance, Funding, Interagency, Knowledge Management, Learning, Management, Morale, Multilateral Diplomacy, Organization Structure, Personnel Organization, Promotion, Public Diplomacy, Recruitment, Research, Retention, Science and Technology Policy, Staff Wellbeing, Staffing, Strategy, Training.
