The Center for Democracy & Technology (CDT) has released a report titled "Assessing AI: Surveying the Spectrum of Approaches to Understanding and Auditing AI Systems." The report emphasizes the importance of clearly defining concepts such as auditing, impact assessment, red-teaming, evaluation, and assurance in the context of AI systems. These terms are often used interchangeably, risking their meaning without a stronger understanding of their specific goals.

The report aims to map out various approaches to AI assessment from narrowest to broadest and from least to most independent. This mapping helps identify which approaches best serve different goals. The goals for AI assessment generally fall into four categories: informing stakeholders about system characteristics and risks, evaluating the adequacy of systems or practices, communicating impacts to relevant parties, and supporting behavior change.

The scope of inquiry in assessments can vary widely. It may range from exploratory assessments that broadly explore potential harms without being bound by known risks to specific analyses using defined benchmarks or metrics. Similarly, independence in assessments varies; it ranges from low independence with direct access to an organization’s systems to high independence where efforts are impartial and unconstrained.

Recommendations from the CDT include aligning evaluation efforts with well-defined goals and ensuring transparency about methods used in assessments. They also emphasize including diverse participants in accountability efforts beyond just those with technical expertise.

"Ultimately," the report concludes, "no one set of accountability actors, single scope of assessment, or particular degree of auditor independence can accomplish all of the goals that stakeholders have for AI assessment and evaluation activities."