Recent AI incidents have underscored the urgent need for robust governance to mitigate risks and ensure responsible development of AI-powered systems. For example, 4chan users leveraged AI tools to create violent and explicit images of female celebrities, and Google’s Gemini generated offensive images of “historical” figures. These incidents are part of a long history of AI failures, from chatbots spewing hate speech to algorithms exacerbating racial disparities in healthcare. Ongoing AI incidents raise a crucial question: Why do AI failures persist? The answer, while complex, centers on the inadequacy of current AI governance procedures.
Effective risk management and oversight of AI hinge on a critical, yet underappreciated tool: comprehensive documentation. Often, documentation is conceptualized as a tool for achieving transparency into AI systems, enabling accountability to external oversight bodies and the public. However, third-party visibility and accountability are only two of the many goals that documentation can facilitate. Documentation also serves as the backbone for effective AI risk management and governance and helps practitioners assess potential failure modes and proactively address these issues throughout the development and deployment lifecycle. Well-maintained documentation offers organizations ongoing insights into their systems’ strengths and weaknesses, fostering iterative improvements. And documentation informs decisions about whether to launch systems at all, given the potential benefits and risks they stand to pose. In essence, documentation is a tool that has the potential to—and is indeed necessary to—facilitate both external accountability and internal risk management practices.
Notwithstanding that potential, approaches that seem beneficial in theory are not always successful in practice. To ensure documentation can fully support robust AI governance, researchers, policymakers, and advocacy groups should consider insights from public- and private-sector practitioners experienced in creating and using documentation, as well as evidence of its efficacy in real-world AI contexts.
In ideal forms, AI documentation records fundamental details about AI systems, including the sources of training data, the hardware and software used to train the component AI models, and the evaluation methodologies used to assess the systems for efficacy and errors. Documentation can also describe the procedures a company has followed in designing, developing, and deploying these systems such as the original motivation for developing the system; whether it underwent an impact assessment or ethics review; how training or evaluation data were labeled.
Good documentation can provide insight into an AI system’s risks and improve development more generally. For example,"Wikipedia developers participating in a research study were asked to use a documentation framework... Through their engagement in the process...identified accuracy metrics more closely aligned with priorities." On the other hand,"when one group documented BookCorpus—a previously undocumented dataset—they revealed numerous duplications...and an overrepresentation of specific genres like romance."
Numerous researchers have proposed frameworks for documenting data models systems processes which have significantly influenced academic researchers policymakers seeking best practices for responsible development."For instance model cards adopted by many developers cited nearly 1 800 times last five years" "NIST references datasheets 26 times guide companies implementing effective risk management."
Improving governance outcomes can be achieved through artifacts created during documentation such as datasheets model cards system cards which help downstream stakeholders understand intended unintended uses compliance organizational legal requirements preventing non-compliant efforts alerting where mitigation techniques may be necessary before safe deployment.
The process itself fosters healthy risk management culture within organizations encouraging following best practices software development adopting rigorous scientific approaches facilitating collaboration among different stakeholders establishing common knowledge base helping cross-functional teams collectively examine strengths weaknesses risks multiple perspectives.
While proposed frameworks laid foundation norms emerging policy attention defining more specific standards necessitates careful evaluation which practices most effectively support goals real-world settings without adequate evidence understanding social organizational institutional dynamics influencing utility unproven potentially less effective methods could become norm more robust approaches better serve accountability go unadopted.
To contribute evidence-based understanding effective strategies reviewed 21 research papers presenting empirical findings related convened stakeholders technology companies consulting firms nonprofit organizations government agencies identifying challenges opportunities translating theoretical proposals real-world practice consulted individually those who use work sought identify challenges opportunities translating theoretical proposals real-world practice empirical studies identify variety challenges implementing effectively usability audience ambiguity misaligned incentives limited understanding ethical impacts poor integration existing workflows
Our convening consultations stakeholders likewise underscored implementing environments complex assumptions proposed often diverge conditions shaping real-world practice considerations raised included dynamic lifecycles complexity general-purpose relationship existing navigating legal reputational considerations
Given urgency preventing harms balance uncertainty around best support consequences continued inconsistency insufficiency current approaches meantime successes failures past implementations offer valuable lessons important lesson proposed valuable accompanied empirical evidence responding applied contexts extensive investigations practical short term smaller qualitative studies focus groups still provide helpful insights researcher Karen Boyd conducted study assessing effectiveness raising ethical awareness among practitioners participated given artifacts consult results indicated access likely recognize issues findings definitive Boyd represents some best available impact deliberation
Empirical evidence exists remains unpublished regrettable take example developed co-design approach iterate proposing gathering feedback relevant pilot implementations updating design converge final product collected generally shared publications authors believe primary contribution rather findings findings provide crucial context why adapted initial designs might succeed without context practitioners policymakers understand rationale behind specific choices necessary implementation moving forward prioritize evaluating proposed publicly sharing providing insight subtle tradeoffs significant implications efforts embrace evaluations necessary step proposing reaching consensus likely time meantime lack ambiguity weaponized actors looking water down rules ultimately expected follow nevertheless carefully review weaker basis informed robust build processes review revisit effectiveness recommendations guidelines ensuring gaps spotted filled help strategy live promise prioritize enhancing base guidance recommend mechanisms evaluating disclosing success particular adopt manage assess usability instrumental success any responsible grantmaking organizations National Science Foundation NSF safety initiatives promote requiring applicants detail empirically evaluate conferences journals shifted open science focus ethical implications publication venues further encourage adjusting review guidelines place significant emphasis presence absence evaluations acceptance decisions
As increasingly integral products establishing critical essential efforts evident equally important adopt validated methods improving outcomes settings learn successes failures studies insights defining refine fulfill theoretical potential collaborative emphasize harnessing benefits minimizing risks Governance Lab continue investigate highlight promising policy practitioner communities
