Stories
Former AI Researchers Warn NYC Lawmakers of Extinction Risks
- •Former researchers from Anthropic, Google DeepMind, and OpenAI testified before the New York City Council, warning that AI development is outpacing critical safety measures.
- •Jacob Coxon, formerly of Anthropic, stated that AI companies are taking reckless risks with systems whose future capabilities remain uncertain and potentially catastrophic.
- •Alex Turner of Google DeepMind estimated a one-in-three likelihood of an AI takeover, while Daniel Kokotajlo highlighted risks regarding the dangerous concentration of corporate power.
- •The hearing addressed severe concerns over recursive self-improvement, where AI systems could enhance their own capabilities with limited human involvement and oversight.
- •New York City is currently considering tougher legislation, including emergency kill switches and protections for whistleblowers, as state and local authorities ramp up oversight.
Key Questions Answered
Jacob Coxon warned lawmakers that the AI industry is taking unacceptable risks with systems whose future capabilities remain uncertain, arguing that rapid commercial development is poorly suited for technology with civilizational stakes. He also noted that researchers are already struggling to explain sophisticated model behavior and raised concerns about machine-generated code weakening human oversight.
New York City is discussing legislation that would grant local authorities greater oversight over AI businesses operating in the city. Key proposals include requirements for emergency kill switches and financial incentives for employees who report serious safety concerns.
New York State Governor Kathy Hochul announced that AI companies will be required to register through a government portal starting in November and report serious safety-related incidents within a 72-hour window.
