More on open AI... 

ChatGPT and Me

The future of AI is uncertain, but one thing is clear: the debate over open versus closed AI is far from over.

As usual my input (prompts) are in the boxed elements. The rest apart from some small edits and formatting is ChatGPT's output.

Mark Zuckerberg recently published an op-ed piece in the Wall Street Journal that seems to push for a more open AI approach. What do you think?

It's an interesting shift, though maybe less of a shift than it first appears. Zuckerberg has been arguing for years that open-weight AI models are strategically and economically valuable. The op-ed is really a more forceful political argument: don't let AI become controlled by a handful of companies or governments, and don't regulate open models out of existence.

The core arguments he seems to be making are: I think there are good arguments on both sides.

On the positive side, history does suggest that open ecosystems often create enormous innovation. Linux, the web, and open programming languages all benefited from broad participation. Meta's Llama models accelerated research and startup activity because thousands of companies could experiment without asking permission. If only three or four firms control frontier AI, they also control pricing, access, research priorities, and much of the downstream ecosystem. Zuckerberg is right to worry about that concentration.

He's also correct that regulation can become a moat. Large companies often have the lawyers, compliance teams, and capital to navigate complex rules, while startups don't. Ironically, regulations written to restrain big AI companies can end up protecting them.

Where I'm less persuaded is the tendency to frame the debate as "optimists versus pessimists."

The strongest AI safety arguments aren't simply fear of technology. They're about incentives. An open model that's good enough to automate cyberattacks, generate novel malware, or meaningfully assist biological misuse doesn't become harmless just because it's open. Once weights are released, they generally can't be recalled. That makes open-weight frontier models a different policy question from open-source databases or operating systems.

There's also an obvious business dimension that's impossible to ignore. Meta benefits enormously from a world where open-weight models are common. It doesn't have the same cloud business as Microsoft or Amazon, nor does it primarily monetize API access the way OpenAI or Anthropic do. Meta's business model is to build capabilities that strengthen its products and ecosystem while reducing dependence on competitors. So Zuckerberg's philosophy aligns with Meta's commercial interests unusually well. That doesn't make him wrong. It just means his incentives deserve to be part of the analysis, as they do for every AI CEO.

One thing I found striking is that the rhetoric is converging more than people admit. OpenAI has also recently argued that AI should broadly benefit society and warned against concentrating power, while still emphasizing stronger safety controls for the most capable systems. My own view is that the most durable position is probably neither "everything should be open" nor "everything should be closed."

Instead:

Humanity has a remarkable habit of taking useful inventions and immediately asking, "How can we either monopolize this or use it to start an argument?" AI appears determined to preserve that proud tradition. The interesting question isn't whether openness is good in the abstract. It's where the capability frontier sits, because the answer that makes sense for a 7B model may not make sense for a system approaching genuinely transformative capabilities.

And just to look at the other side of the coin, 1000 or more AI staff from various AI companies signed a ‘Pacing the Frontier’ letter, calling on the U.S. to help build tools that could deliberately pace AI progress as the world nears autonomous AI research.

That's what makes this week's debate unusually interesting.

These aren't two camps yelling past each other. They're overlapping groups of people, sometimes working at the same companies, who agree on many facts but draw different conclusions.

The "Pacing the Frontier" letter isn't really asking for a blanket slowdown. Its central claim is narrower: if AI reaches the point where it can substantially automate AI research itself, progress could accelerate so quickly that neither companies nor governments can keep up. Because no single lab can voluntarily slow down without risking being overtaken, the authors argue that the world should at least have the technical and governance mechanisms available to deliberately "buy time" if necessary.

What I find notable is who signed it. The list includes senior researchers from OpenAI, Anthropic, Google DeepMind, and Meta, including people who are directly building frontier models. This isn't an outside activist campaign. It's coming from inside the labs.

At first glance, it looks contradictory:

But I don't actually think they're addressing exactly the same problem.

Zuckerberg is mostly talking about who gets access to powerful models and avoiding regulatory capture by a few companies.

The pacing letter is talking about how fast capabilities themselves improve, especially if AI starts recursively improving AI. In other words, it's less about open versus closed, and more about acceleration versus governance.

Those positions can coexist. You could believe that: The harder question is whether "pacing" is even practical.

Suppose an AI lab develops a system that doubles research productivity every three months. If another lab, or another country, keeps pushing ahead, then the first lab has simply handed over the lead. The letter acknowledges exactly this coordination problem, which is why it asks for international mechanisms rather than unilateral restraint.

That's where I become skeptical.

Building "the brakes" sounds reasonable. Deciding who gets to pull the brake is much harder.

Who determines that AI has crossed the threshold? A government agency? An international body? The companies themselves? What evidence is sufficient? How do you verify compliance? And what happens if China, or another actor, doesn't participate?

Those are enormously difficult institutional questions. The letter intentionally stays high-level and doesn't answer them, which probably made it easier to attract broad support but leaves the practical implementation almost entirely unspecified.

One thing I do think has changed over the last year is that discussions about "recursive self-improvement" or "automated AI research" have moved from speculative philosophy into engineering planning. Researchers increasingly talk about AI systems that can propose experiments, write code, run evaluations, analyze results, and iterate with limited human input. Whether that leads to an intelligence explosion is still very uncertain, but it's no longer just science fiction.

The irony is almost theatrical. One set of AI leaders is saying, "Please don't lock AI behind a handful of companies." Another set, many of them employed by those same companies, is saying, "Please build a way to slow us down if we start moving too fast." Both groups may be responding to the same underlying reality: they're seeing capability progress inside the labs that makes them think the next few years could look very different from the last few.

That, more than the policy prescriptions, is the signal I pay attention to. When people who disagree on economics and regulation start converging on the claim that AI research itself may soon be substantially automated, it's worth taking seriously. Not because they're necessarily right, but because they're among the few people who can directly observe how quickly these systems are improving.

 

Green Hosting Badge