Research Participation Prospectus: Version 1.0, February 2026
Full details on research design, participation requirements, data handling, privacy rights, and what participants receive.
Context
Why This Research Is Needed Now
Agencies across federal and state government are operating under formal AI governance requirements. NIST AI Risk Management Framework adoption is accelerating. Agencies document, approve, and file oversight mechanisms. Executive orders and OMB guidance, currently Memorandum M-25-21,9 have created a compliance infrastructure that did not exist three years ago.
What is largely absent from the literature is empirical research on whether any of it is working. Most governance commentary is normative: what agencies should do, what frameworks they should adopt, what policies should require. Empirical research on what is happening in practice is sparse and mostly audit-by-audit. Are existing frameworks producing the accountability outcomes they were designed for?
The documentation exists. In three public reviews, evidence that the controls operated was missing. Each review found an evaluation or implementation step that had not been taken as of the report date. In 2012 GAO found that CBP had not assessed whether an updated set of rules in its Automated Targeting System was more effective.1 GAO described the system in 2006 as a cargo-inspection targeting tool,7 and DHS's 2006 notice and 2012 privacy assessment describe it as covering travelers as well as cargo.6,8 In June 2026 the Treasury Inspector General for Tax Administration found that the IRS had not taken sufficient steps to evaluate the effectiveness of its chat applications, which began as a pilot in 2017.2 In September 2026 the SBA Office of Inspector General found that SBA had not implemented a process to identify high-impact AI uses.3 DHS, the IRS, and SBA agreed with the recommendations.
Vendor disclosures raise the same question. A September 2026 news report described an OpenAI disclosure of six incidents of unexpected or concerning model behavior over the prior six months, and cited a Reuters report that related agent activity at Hugging Face began about two months before it was discovered.5
Looking good on paper does not make a framework function under operational conditions. Measurement discipline separates frameworks that hold from frameworks that only perform.
The Connective Thread: Measurement
Neither governance effectiveness nor oversight quality can be evaluated without agreed measurement criteria. ThinkCapital's contribution is methodological: applying systematic assessment frameworks developed over three decades of government IT research and informed by collaboration with IFPUG, NESMA, and Carnegie Mellon University. The goal is empirical findings that practitioners can use immediately.
Organizing Framework
The GIAG Governance Pipeline
Working Paper 4 defines six stages where agencies make, review, or record decisions about an AI system. A failure at one stage does not require a failure at another, so the program measures each stage separately. Four stages have a published instrument. Two are open research. GAO's 2021 accountability framework for AI audits is a reference point for the pipeline.14
| Stage | Question the stage must answer | Instrument or stream | Status |
|---|---|---|---|
| 1. Authorization & scope | Does the system still do only what it was authorized to do, and who would notice if it did not? | Stream Three: Authorization & Scope Integrity | Forming |
| 2. Technical specification | Could the testing claim have failed? | Testing Rigor: four conditions of a rigorous testing claim (Solicitation & Due-Diligence Review) | Published |
| 3. Deployment review | Did review change or stop anything, and did governance scale up when the pilot moved to production? | Stream Four: Deployment Review & the Pilot-to-Production Gate | Forming |
| 4. Operational oversight | Did the human reviewer have the authority, time, and information to exercise judgment? | Human Oversight Quality Index (Stream Two) | Published |
| 5. Audit | How long did an unauthorized action run before someone with authority knew? | Detection Clock (Working Paper 4) | Published |
| 6. Post-decision reconstruction | Can a single decision survive scrutiny after the fact? | Governance Evidence test (Working Paper 5) | Published |
Stream One (NIST AI RMF implementation) cuts across all six stages by testing whether framework adoption changes any of these outcomes.
Research Design
Research Streams
The initiative comprises two active streams and four forming streams. Two of the forming streams open the pipeline stages that have no instrument yet.
NIST AI RMF Implementation in Government Practice
The NIST AI Risk Management Framework,13 a voluntary framework, is rapidly becoming the default governance reference for federal and state agencies. Yet few empirical studies examine how it performs in practice across different organizational contexts.
This stream examines implementation patterns across agency types, sizes, and mission profiles through structured interviews with government IT leaders who have direct deployment or evaluation experience. Key dimensions include framework adaptation to agency context, measurement of implementation fidelity, and conditions under which RMF adoption produces durable governance versus audit-cycle compliance.
Human Oversight Quality in AI-Augmented Government Operations
Most AI governance requirements treat human oversight as binary: present or absent. The more consequential variable is whether oversight mechanisms constitute genuine controls or documented assumptions.
This stream develops a typology of oversight models observed across government AI deployments, tests them against real operational scenarios, and constructs a measurement framework for evaluating oversight quality. Key dimensions include oversight mechanism design across agentic and assistive AI systems, intervention point architecture, audit trail integrity, and the conditions under which oversight requirements produce functional accountability versus documentation without evidence of operation.
Authorization & Scope Integrity
Every later stage of the pipeline inherits the authorization decision. A system the agency approved for one purpose can grow into another through contract modifications, new data sources, and new users, while the original approval stays on file. The role responsible for the decision is sometimes vacant or contested.
This stream compares agencies that defined authorization before deployment with agencies that assembled it afterward to cover systems already running. It asks practitioners how a system's actual use compares with its authorized use, and who owns that comparison. It also examines chief AI officer designation and vacancy, the difference between an enforceable directive and a directive that only sits on paper, and the evidence that approval preceded launch. Participation is by structured interview with practitioners who have written, held, or audited authorization authority.
Deployment Review & the Pilot-to-Production Gate
Governance built for a pilot does not scale to production, and a pilot can run for years without a full production review. Recent reporting also describes pilots that stall in compliance review because testing, bias review, and monitoring arrive as late steps instead of design constraints.
This stream collects cases of deployment review from agencies and tests the Five Intervention Criteria (irreversibility, consequence transfer, distributional novelty, value conflict, and legal or regulatory accountability) and risk tiering against them. Key dimensions include what a review has the power to refuse, what evidence it requires, and how agencies resize governance when a pilot graduates to production.
Forming
Additional Streams Under Development
AI Adoption Thresholds & Organizational Tipping Points
Applying Schelling-Granovetter threshold models to explain why identical AI initiatives succeed in some agencies and stall in others, and what moves them past the tipping point.
Take part →AI Productivity Measurement for Government Missions
Developing empirical methods for measuring whether AI tools are delivering measurable improvements in government mission contexts, beyond technology metrics.
Take part →Participation Studies
Two Benchmarks You Can Contribute To
Each study takes under an hour. We remove names and agency identifiers from analysis files and do not publish participant identities without written consent. We cannot guarantee anonymity against legal process. Please do not share classified, controlled unclassified, or source-selection-sensitive material, and check your agency's rules first. Contributors receive aggregate findings before publication, not other participants' individual responses.
Detection Interval Benchmark
Government practitioners and contracting offices report three facts about one AI-enabled system:
- Whether the contract defines a maximum interval between an unauthorized action and notice to the agency.
- Whether the audit log retention window is long enough to reconstruct an event that surfaces months later.
- Who inside the agency owns escalation once an anomaly is flagged.
Solicitation & Due-Diligence Review
Procurement and contracting professionals share AI solicitation or contract language after removing identifying detail. GIAG scores each clause against the four conditions of a rigorous testing claim and reports the aggregate pattern: how often agencies require a vendor to state a measurable rate, and how often they accept a general assurance. Federal context includes the FTC's 2024 enforcement against deceptive AI claims11 and GSA's June 2026 request for comments on a proposed clause on large language model data.12 A second pass records whether the solicitation asks the vendor to disclose where it is organized, who certified its product, and any government ownership or funding interest. GIAG records whether such questions appear. It does not assess any vendor or country, and agencies should check with counsel before adding disclosure requirements.
Take part →How We Work
Methodology
The program applies comparative assessment methodology developed through three decades of government IT research. The approach is empirical and practitioner-oriented, grounded in systematic assessment disciplines pioneered at Software Productivity Research and refined through advisory engagements with federal agencies, defense organizations, and state governments.
Evidence Base: Two Phases
Phase 1, complete. Instrument development on documented public cases, together with informal conversations with IT and AI governance practitioners. The conversations shaped the questions and scoring criteria. They are not a sample, and GIAG draws no prevalence claims from them.
Phase 2, recruiting now. Structured interviews with government CIOs, chief AI officers, procurement leads, and audit staff. The target is 15 conversations completed by February 1, 2027. Findings from Phase 2 will state who was interviewed, how they were recruited, and what the sample can and cannot show. Until then, read the instruments as tested on public cases and not yet validated in the field.
Structured Practitioner Interviews (Phase 2)
Primary data collection through 30–45 minute structured interviews with government IT leaders, AI governance practitioners, and policy implementers with direct deployment or evaluation experience within the past 18 months.
Comparative Framework Analysis
Cross-agency analysis of governance documentation, implementation records, and self-reported outcomes using standardized assessment criteria drawn from international measurement standards work.
Public-Case Instrument Testing
ThinkCapital applies each measurement instrument to documented public cases before practitioners use it. We applied Testing Rigor to publicly available documentation about a state criminal sentencing tool10, the Governance Evidence test to litigation records, and the Detection Clock to publicly reported vendor disclosures. Scores describe documentation, not a product's accuracy or fairness. ThinkCapital publishes the results and scoring criteria with each working paper. These are tests on public cases, not field validation.
Pattern Identification and Framework Development
Synthesis of documentary findings and practitioner input into actionable decision frameworks for practitioners. The frameworks distinguish conditions that produce durable governance from those that produce documentation without evidence of operation.
Schedule
Research Timeline
Research Design and Participant Recruitment
ThinkCapital designed the interview instruments, began outreach to practitioners across both active streams, and established comparative assessment criteria and baseline documentation protocols. It released Working Paper 1.
Documentary Analysis and Practitioner Conversations
ThinkCapital held exploratory conversations with practitioners, collected and coded documentary evidence from public cases, and developed frameworks alongside the analysis. It released Working Papers 2 and 3.
Analysis and Draft Findings
Cross-case analysis produced the six-stage governance pipeline and four measurement instruments. ThinkCapital released Working Papers 4 and 5.
Publication, Dissemination, and New Streams
ThinkCapital publishes first findings and submits them to relevant practitioner and academic venues. Sector briefings are available on request. Phase 2 interviews with government practitioners (target: 15 conversations completed by February 1, 2027), Streams Three and Four, and the two participation studies are open for recruitment.
Get Involved
Participate in This Research
We are conducting structured interviews and running two benchmarks with government IT leaders, AI governance practitioners, contracting professionals, and policy implementers who have direct experience deploying AI or implementing governance frameworks in federal, state, or local government.
We aim to complete 15 conversations by February 1, 2027. Participation involves a single 30–45 minute structured interview. Participants receive aggregate preliminary findings and may be acknowledged by name or participate with identifiers removed. Anonymity cannot be guaranteed against legal process.
We are particularly interested in practitioners with direct experience implementing or evaluating AI governance frameworks in the past 18 months. Accounts of difficulty or partial implementation are as valuable as accounts of success.