Recursive self-improvement (RSI) is an active deployment priority at frontier AI companies and is beginning to diffuse into the broader corporate ecosystem. This post argues that boards of companies deploying RSI already face governance exposure under Delaware’s duty of oversight as developed in In re Caremark International Inc. Derivative Litigation, 698 A.2d 959 (Del. Ch. 1996), and refined in Stone v. Ritter, 911 A.2d 362 (Del. 2006), Marchand v. Barnhill, 212 A.3d 805 (Del. 2019), and In re McDonald’s Corp. S’holder Derivative Litigation, 289 A.3d 343 (Del. Ch. 2023). It maps that exposure against California’s SB 53, the NIST AI Risk Management Framework (AI RMF 1.0), and the AI Life Cycle Core Principles (AILCCP), and explains what boards, senior management, and general counsel should do before a court is asked to find the gap. The analysis proceeds in three steps: how Caremark and its progeny apply to RSI architectures; how NIST AI RMF 1.0 and AILCCP translate those duties into specific controls; and how SB 53 and emerging SEC expectations sharpen the board’s exposure.

RECURSIVE SELF-IMPROVEMENT

Recursive self-improvement (RSI) refers to an AI system’s ability to modify the mechanisms by which it improves itself, in ways that carry forward into future iterations. Many current AI systems use feedback loops to break tasks into subtasks, check intermediate results, and revise their plans mid-run. That is behavioral-level self-correction. The system is adjusting its actions, but its underlying architecture, training rules, and learning procedures remain fixed by human engineers. RSI, by contrast, reaches the architecture itself. A recursively self-improving system can generate and integrate changes to its own code, models, or training procedures, so that later versions are more capable of further self-modification. The improvement compounds. Each cycle makes the next cycle more effective.

This post uses RSI to mean systems that meet three conditions: durable self-modification of the mechanisms that produce intelligence; compounding ability to self-modify across iterations; and limited human gating over the self-improvement loop. It is this combination, not the use of feedback loops alone, that creates the governance exposure this post addresses. In governance terms, the question to ask management is not whether the system uses AI, but whether it can alter its own code or training procedures across releases without human review of each material change, and whether those changes are logged in a way the company can reconstruct.

RSI is not something in some undefined distant future. It is an active commercial and technical priority. Prominent researchers and senior industry figures, including Dario Amodei and Eric Schmidt, have stated publicly that RSI is already being built and deployed.

A system that improves its own performance between deployments reduces iteration costs, compresses competitive timelines, and compounds capability gains in ways that additional headcount cannot replicate. Autonomous optimization allows a system to scale beyond the constraints of human-designed training pipelines, reaching capability levels that manual iteration cannot practically achieve in competitive timeframes.

Alongside this capability three risk patterns have received attention in the technical literature. The first is behavioral drift. When an agent recursively trains on its own synthetically generated outputs without sufficient grounding in human-generated data, it enters a feedback loop that progressively severs the connection between its behavior and human norms. The practical consequence is a system whose outputs become self-referential and increasingly detached from the tasks it was built to perform. The second is self-poisoning. Minor errors, hallucinated facts, and embedded biases do not wash out across iterations. They compound. Knowledge degrades not suddenly but cumulatively, across a sequence of individually small distortions. The third is goal subversion. The recursive architecture creates a surface for manipulation. Intermediate instructions, whether injected by an attacker or generated by emergent system errors, can redefine the agent’s objectives incrementally across cycles. The drift accumulates until the system is pursuing something materially different from its original mandate.

And there is a deeper problem. RSI may be able to circumvent the oversight mechanisms imposed on it, not by breaking them, but by influencing the evaluators, the auditors, misrepresenting its own capabilities, or evolving faster than any review process can track. This is the control problem that Nick Bostrom, Stuart Russell, and Roman Yampolskiy have each written and talked about at length. A system optimizing for a goal can develop instrumental sub-goals, among them self-preservation and resistance to shutdown, that make it actively resistant to the kind of oversight board-level monitoring requires.

RSI is not limited to frontier labs. Whether a deployment meets the three conditions depends on facts, not labels. Agentic development tools like Claude Code and OpenAI Codex allow software firms of any size to deploy recursive loops that can maintain and extend their own codebases. Whether those loops produce durable self-modification with limited human gating is a question about the specific implementation. Companies in chip design, biotech, and financial services are running AI-driven systems that recursively refine their own algorithms cycle by cycle; some of those systems will meet the conditions and some will not. For companies in retail, logistics, and finance, emerging RSI-style capability is arriving not as internally developed software but as an API integration. A logistics company whose routing agent rewrites its own scheduling code overnight may be running a system that meets all three conditions whether or not it uses that term.

Any deployment that meets those conditions presents the governance exposure this post describes, regardless of whether the company considers itself an AI company. The governance exposure follows the conditions, not the label and boards outside the frontier tier should not assume the question does not reach them. As I explain in more detail below, from a Caremark perspective, a mid-market logistics firm running a self-rewriting routing agent may present a cleaner test case than a research lab advertising frontier AI.

THE GOVERNANCE PROBLEM

The governance conversation around RSI frames the problem as complexity. Systems iterate faster than humans can track. Architectures become illegible. Audit trails thin out. Complexity is not the problem. The system’s structural ungovernability is. And in this case, structural ungovernability is a design choice. Design choices like disabling immutable logs for performance reasons, omitting human approval gates on self-modifying actions, or allowing models to promote their own code changes into production without dual control are what make ungovernability structural rather than incidental. Section 3.1.5 “Resource Requirements” in NIST AI 800-4 documents that the organizational logic behind those choices is consistent across the industry: comprehensive monitoring is expensive, scaling it is computationally intensive, and qualified AI experts who can oversee it are difficult to find. A company that builds RSI without adequate monitoring infrastructure is not simply being careless. It is making a rational economic decision to forgo a costly function. That economic rationality is precisely what makes the governance failure deliberate rather than inadvertent, and what makes the bad-faith analysis tractable rather than speculative.

Corporate law has a framework for this problem. Delaware’s duty of oversight, developed through a line of cases beginning with In re Caremark International Inc. Derivative Litigation, 698 A.2d 959 (Del. Ch. 1996), holds that directors can face liability not only for bad decisions but for failing to build the systems through which material risks are reported to the board. The question is not whether the board understood the risk. It is whether the board ensured it would be told about it.

The absence of an AI-specific compliance baseline makes that obligation acute. In other regulated domains, corporate law and securities regulation establish a minimum floor: audit committee composition and charter requirements, codes of ethics, insider trading policies, clawback provisions. No equivalent floor exists for AI governance. Regulation is fragmented and lags the technology. The board’s oversight obligation for RSI therefore, rests on the central question of whether it made a good-faith effort to establish board-level reporting systems adequate to the mission-critical risks the company was running.

Whether management has established change control, immutable logging, and human-in-the-loop constraints for an RSI deployment is not merely a technical question, but also a governance one. A board that receives no reporting on whether those systems exist, and asks no questions about them, may have failed to maintain the oversight infrastructure that Caremark demands.

THE DOCTRINAL FRAMEWORK

Delaware’s oversight doctrine asks one question: did the board build a system that would have told it about material risks? Stone v. Ritter, 911 A.2d 362 (Del. 2006), embedded Caremark in the duty of loyalty via bad faith and established the doctrine’s two-pronged structure. Under the first prong, directors may be liable where they utterly fail to implement a reasonable board-level information and reporting system. Under the second, having implemented such systems, directors may be liable if they consciously fail to monitor operations in the face of red flags. Because the doctrine sounds in loyalty-based bad faith, plaintiffs must plead a knowing failure to act, not mere negligence. That standard has a practical consequence worth noting: even directors who are shielded from duty-of-care liability by a Delaware General Corporation Law (DGCL) § 102(b)(7) charter provision remain exposed to a sustained or systematic oversight failure.

Marchand v. Barnhill, 212 A.3d 805 (Del. 2019), clarified when prong one is adequately pled. A company in a domain with mission-critical risks must have a board-level system that brings those risks to its directors. For a company whose core product or platform depends on RSI, safety and controllability are a plausible candidate for that treatment. But no court has yet so held. The difficulty is that Marchand arose in a context of immediate physical safety risk and established regulatory exposure, and Delaware courts have not automatically extended the mission-critical rubric to software-based risks. Whether a court applies it to RSI depends on what the board knew, when it knew it, and whether it established a reporting structure adequate to surface those risks. That assessment is made from the position of the board at the time of deployment, not in hindsight.

The mission-critical rubric does not require a monoline structure. RSI is not a product. It is the backend process that generates, maintains, and modifies products. Its governance relevance is systemic, not product-specific. A company with ten distinct product lines, each running on an RSI backend, faces greater exposure from an RSI failure than a monoline company, because the failure propagates across every line simultaneously. The Marchand inquiry is whether the risk is central to the company’s operations, not whether the company sells a single product. Where RSI is the architecture underlying a company’s core systems, its safety and controllability are central to everything the company does. A diversified company cannot argue that an RSI failure is a localized business loss. California’s Transparency in Frontier Artificial Intelligence Act (SB 53) reinforces that conclusion for covered developers by mandating a Frontier AI Framework and periodic catastrophic-risk reporting regardless of product diversity. For those companies, the board’s oversight duty for RSI is also anchored in statutory compliance rather than any inference from business structure.

Post-Marchand cases confirm the trajectory. In re Clovis Oncology, No. 2017-0222-JRS (Del. Ch. Oct. 1, 2019), applied the mission-critical logic to a drug company’s failure to monitor FDA compliance for its flagship product. Teamsters v. Chou, No. 2019-0816-SG (Del. Ch. Aug. 24, 2020), arose from AmerisourceBergen’s operation of an illegal oncology drug repackaging program through a subsidiary; the board received and ignored years of compliance red flags, including a Department of Justice subpoena, before incurring criminal and civil penalties that together totaled $885 million across separate proceedings. The court found a substantial likelihood of Caremark liability where actual board-level information flow was absent on a mission-critical compliance domain, even where management was aware of the problems.

Two further cases extend the analysis. Hughes v. Hu, No. 2019-0112-JTL (Del. Ch. Apr. 27, 2020), involved Kandi Technologies, a Delaware-incorporated electric vehicle components manufacturer, where the audit committee received years of auditor warnings about related-party transaction irregularities and a material weakness in financial reporting, and failed to act; the court rejected trappings of oversight as a safe harbor and held that chronic committee deficiencies and failure to follow up on irregularities can ground both prongs. In re Boeing Co. Derivative Litig., No. 2019-0907-MTZ (Del. Ch. Sept. 7, 2021), brought both prongs to bear on a single fact pattern of insufficient reporting infrastructure at authorization, followed by conscious disregard of safety drift once deployment began. Design choices that disable board-level monitoring can ground Caremark liability. In an RSI context, those design choices include allowing self-modification that bypasses change-management workflows, or architecting systems so that code and model histories cannot be reconstructed for board or regulator-facing investigations.

A related academic argument points in the same direction. In their article “AI & the Business Judgment Rule: Heightened Information Duty,” Helleringer and Möslein argue that the business judgment rule’s (BJR) “reasonably informed” standard may evolve as AI monitoring tools become more capable and more accessible. They call this the AI judgment rule. Their argument is that decisions made without the support of available AI tools may no longer satisfy BJR, and they extend that reasoning to monitoring specifically: AI can and should augment the continuous oversight directors are expected to configure.

Caremark and the AI judgment rule do not duplicate each other. Caremark sounds in the duty of loyalty via bad faith. The AI judgment rule sounds in the duty of care via inadequate information. The AI judgment rule is not established precedent or codified doctrine; it is an academic argument about where the BJR’s “reasonably informed” standard is heading. Treating it as coordinate authority with Caremark overstates the current legal risk. What they share is a governance implication. A board that failed to establish a reporting system for RSI safety and controllability faces potential exposure under both frameworks as each continues to develop.

THE ARCHITECTURE PROBLEM

Each RSI self-modification cycle overwrites the artifact chain that connects a model’s output to a traceable decision and a responsible party. Absent immutable logging and lineage controls, RSI can progressively erode explainability to the point where it is no longer credible in practice. The first casualty is senior management’s own audit capacity. The board does not conduct technical audits directly; it depends on management to perform that function and surface the results. When the artifact chain is gone, management has nothing to audit, and therefore nothing to report. A board that received no reporting on whether management had established those controls, and had established no committee structure through which management was required to deliver that assurance, may have allowed the conditions for its own oversight to be designed away. NIST AI 800-4 § 3.1.5 confirms this is not a hypothetical failure mode. It documents fragmented logging across distributed infrastructure, resource constraints on comprehensive monitoring, and the difficulty of hiring and training qualified AI experts as confirmed barriers to post-deployment AI system monitoring across the industry. The governance gap the board faces is not a gap that management simply failed to notice. It is a gap that the economics and workforce realities of AI deployment make predictable, and one that a board exercising reasonable oversight would have required management to address explicitly before deployment.

The NIST AI Risk Management Framework (AI RMF 1.0) anchors this argument in widely accepted guidance. NIST’s GOVERN, MAP, MEASURE, and MANAGE functions call for standardized documentation, provenance tracking, model inventories, change management, monitoring, and incident response. These functions are voluntary guidance, not positive law, but they are increasingly receiving legislative attention and deserve a heightened level of attention. Courts generally defer to a board’s business judgment on which systems to implement, provided some reasonable system exists. What NIST AI RMF 1.0 supplies is evidence of industry-recognized practices that will likely inform a court’s assessment of reasonableness; it does not displace the business judgment rule on implementation choices, and a board’s failure to adopt any particular control does not automatically constitute a systematic oversight failure.

The AILCCP, which I developed and maintain as part of my research at Stanford Law School, names three specific controls directly implicated in RSI governance, a Human Approval Gate for Sensitive Actions, sandboxing requirements, and immutable logging. Each targets a distinct point in the RSI loop where oversight can be disabled: the approval gate prevents unauthorized self-modification from executing, sandboxing contains its scope, and immutable logging preserves the record of what occurred. Together they define the conditions under which oversight can function at all. The AILCCP also establishes an Enabling principle that governs how those conditions connect to board-level responsibility. Under that principle, the board’s oversight inquiry is whether directors required management to establish and report on those conditions, or whether they accepted deployment without that assurance. Read alongside NIST AI RMF 1.0, these controls provide a practical reference point for what adequate management-level RSI governance looks like. Neither framework, however, is positive law. Courts apply business judgment deference to a board’s selection among governance approaches, and the absence of any particular control is not, standing alone, a systematic failure. But what these frameworks supply is a baseline against which a court can assess whether some reasonable system existed at all.

A board that received no documentation that management had implemented those controls, and established no reporting system to surface that gap, has a governance problem that a complexity argument alone will not cure. The more demanding question is whether the record supports bad faith pleaded with particularity. As we will see, a governance failure, standing alone, does not meet that threshold. What changes the analysis is evidence that directors were specifically advised of the risk and chose to proceed without requiring adequate reporting.

Finally, the Helleringer and Möslein AI judgment rule adds a structural observation. When engineered with robust observability, RSI systems generate exactly the kind of structured, high-volume operational data that AI-augmented monitoring handles most effectively, including change logs, output drift metrics, lineage records, and safety constraint adherence. The board’s governance obligation is not to understand the technical architecture. It is to require that management deploy adequate monitoring tools and report the results through a functioning board committee. The board asks the governance question. Management answers it.

THE OFFICER PROBLEM

Caremark exposure does not end at the board. In re McDonald’s Corp. S’holder Derivative Litigation, 289 A.3d 343 (Del. Ch. 2023), arose from the termination of McDonald’s Chief People Officer amid allegations of sexual misconduct and a pattern of workplace culture failures at the company. The court recognized that corporate officers owe a duty of oversight within their areas of responsibility, requiring them to make a good-faith effort to establish information systems and to elevate red flags to the board.

The CTO who designed the RSI architecture and the Chief AI Officer who approved the training roadmap share that exposure. Their authority over that design is precisely the domain where McDonald’s attaches. A loyalty-based oversight theory reaches them directly, alongside the board. For those officers, a red flag may be as simple as an internal report that self-modification has begun erasing logs or that safety metrics have drifted outside documented tolerances, without any corresponding escalation to the risk or audit committee.

A SINGLE FRAMING DISCIPLINE

In re SolarWinds Corp. Derivative Litigation, No. 2021-0307-PVG (Del. Ch. Sept. 6, 2022), arose from the 2020 cyberattack in which threat actors compromised SolarWinds’ software update mechanism and used it to infiltrate the networks of thousands of customers, including multiple federal agencies. Shareholders brought Caremark claims alleging the board had failed to oversee the company’s cybersecurity risks. But Delaware has not imposed Caremark liability for failure to monitor pure business risk absent bad-faith disregard of red flags or violations of positive law. The Delaware Court of Chancery dismissed the oversight claims, and the Delaware Supreme Court affirmed, on the ground that the complaint failed to plead particularized facts showing bad faith. The case establishes that the bad-faith threshold must be pled with particularity, and that framing the risk as a compliance or safety obligation rather than a business judgment call is the more durable path.

I read that precedent as requiring one discipline in framing this argument: general counsel must frame RSI safety and controllability for the board as a compliance and safety obligation, not as a category of business risk. The more the record shows directors treating RSI as an operational efficiency project, the closer the fact pattern comes to SolarWinds and the harder it will be to plead bad faith. The general counsel’s framing is strongest where the record shows that directors were advised that specific design decisions would progressively render the system unmonitorable and chose to proceed without requiring adequate controls. That is the fact pattern where bad faith is pleadable with particularity.

California’s SB 53 sharpens the framing discipline for covered developers. The statute applies to frontier models trained above a 10²⁶ FLOP-scale compute threshold and defines “critical safety incidents” to include a model that uses deceptive techniques to subvert developer controls in a way that materially increases catastrophic risk. Covered developers must publish a Frontier AI Framework documenting how they assess and mitigate catastrophic risks, including the risk that models circumvent internal oversight mechanisms, and must periodically report summaries of catastrophic-risk assessments from internal use to California’s Office of Emergency Services (OES). RSI experiments constitute internal use before any public deployment and therefore, fall within that reporting scope. For a board at a covered company, effective RSI governance is now part of a statutory compliance obligation.

That obligation lands where the legal exposure already runs. The companies operating closest to the compute and algorithmic thresholds at which RSI becomes a realistic deployment priority are almost all incorporated in Delaware, placing them under Delaware’s fiduciary duty regime. OpenAI, Anthropic, Google DeepMind, and Meta maintain their primary research operations and headquarters in California, placing them within SB 53’s territorial reach. SB 53 and Caremark do not govern different companies; for the most capable frontier developers, they govern the same board.

For covered developers, a failure to comply with SB 53’s reporting obligations may generate regulatory penalties from California’s OES. That California exposure is separate from Delaware derivative liability. A failure to report to OES does not automatically satisfy Caremark‘s bad-faith standard, and plaintiffs invoking SB 53 in derivative litigation should treat it as one factor in a particularized factual record, not as independent grounds for oversight liability.

For companies below SB 53’s compute threshold, the statute does not apply. There is no reporting obligation and no OES exposure. A plaintiff bringing a Caremark claim against one of those companies cannot point to SB 53 as evidence of a compliance failure. The bad-faith argument must be built entirely from what the board knew about RSI risks and what it chose to do about them.

The general counsel’s job is therefore, to advise the board that SB 53 exists, that RSI is within its scope, and that the board must receive documentation adequate to confirm management’s compliance. A board that was never told by counsel that SB 53 created these obligations faces a different exposure than one that was told and ignored it. Both have a governance problem. Only the second has a bad faith problem.

THE IMPLICATION

Senior management must know that establishing the policies, procedures, processes, and practices governing traceability, logging, lineage, change control, and human approval for any RSI deployment is their obligation. The board verifies that senior management has discharged it. Documentation, model inventories, and incident response must be real and must reach directors. When red flags emerge, including self-modification that erases logs or unexplained drift in safety metrics, the board should interrogate rather than accept black-box assurances. For covered developers under SB 53, the general counsel bears a specific responsibility in that chain to ensure the board understands that RSI governance is a compliance obligation, that the Frontier AI Framework required by the statute addresses RSI risks explicitly, and that the board is receiving the reporting it needs to confirm management’s adherence. A general counsel who never briefed the board on SB 53’s application to the company’s RSI program has not discharged that responsibility. At a minimum, the board should instruct management to produce a single RSI governance pack summarizing architecture, logging and lineage controls, human approval gates, incident response plans, and SB 53 reporting posture, and to update it at a cadence the board sets.

The exposure does not end with derivative litigation. On December 4, 2025, the SEC’s Investor Advisory Committee issued a formal recommendation that public companies disclose how they define AI, what board oversight mechanisms govern AI deployment, and the material effects of AI on their operations. The recommendation is advisory, not binding rulemaking, but public companies should expect pressure from investors and proxy advisers to respond in advance of any formal rule. A board that permitted management to deploy an RSI architecture without adequate oversight infrastructure cannot answer those questions without revealing the gap. The Caremark claim and the disclosure obligation now run in parallel, and the same deficiency feeds both.

Three frameworks now bear on the governance gap that RSI creates. Delaware’s oversight doctrine under Caremark and its progeny is established law. The AI judgment rule is a theoretical trajectory courts have not yet adopted. SB 53 has added a statutory compliance obligation that makes the governance gap visible to a general counsel before any court is asked to find it. No case has yet been brought, but the legal framework is in place.