California’s legislature passed the AI safety bill, SB 1047.
The bill imposes safety requirements on AI models that required over $100 million to train.
As the bill itself summarizes (with headings added by me):
- OVERALL: This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, require that a developer, before beginning to initially train a covered model, as defined, comply with various requirements, including implementing the capability to promptly enact a full shutdown, as defined, and implement a written and separate safety and security protocol, as specified.
- SAFETY PROTOCOL: The bill would require a developer to retain an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years, including records and dates of any updates or revisions and would require a developer to grant to the Attorney General access to the unredacted safety and security protocol.
- PROHIBITED IF UNREASONABLE RISK: The bill would prohibit a developer from using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or making a covered model or a covered model derivative available for commercial or public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative will cause or materially enable a critical harm, as defined.
- ANNUAL AUDIT: The bill would require a developer, beginning January 1, 2026, to annually retain a third-party auditor to perform an independent audit of compliance with those provisions, as prescribed.
- The bill would require the auditor to produce an audit report, as prescribed, and would require a developer to retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years.
- The bill would require a developer to grant to the Attorney General access to the unredacted auditor’s report upon request. The bill would exempt from disclosure under the California Public Records Act the safety and security protocol and the auditor’s report described above.
- STATEMENT OF COMPLIANCE: This bill would require a developer of a covered model to submit to the Attorney General a statement of compliance with these provisions, as specified.
- REPORTING SAFETY INCIDENTS: The bill would also require a developer of a covered model to report each artificial intelligence safety incident affecting the covered model, or any covered model derivative controlled by the developer to the Attorney General, as prescribed.
- POLICIES FOR COMPUTING CLUSTERS: This bill would require a person that operates a computing cluster, as defined, to implement written policies and procedures to do certain things when a customer utilizes computer resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
- ENFORCEMENT: This bill would authorize the Attorney General to bring a civil action, as provided.
- WHISTLEBLOWERS: The bill would also provide for whistleblower protections, including by prohibiting a developer of a covered model or a contractor or subcontractor of the developer from preventing an employee from disclosing information, or retaliating against an employee for disclosing information, to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the information indicates the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm.
- NEW AGENCY: This bill would create the Board of Frontier Models within the Government Operations Agency, independent of the Department of Technology, and provide for the board’s membership. The bill would require the Government Operations Agency to, on or before January 1, 2027, and annually thereafter, issue regulations to, among other things, update the definition of a “covered model,” as provided, and would require the regulations to be approved by the board before taking effect.
- This bill would establish in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed.
- The bill would on or before January 1, 2026, require the Government Operations Agency to submit a report from the consortium to the Legislature with that framework. The bill would make those provisions operable only upon an appropriation in a budget act for its purposes.