Humanity’s Last Exam: Experts Unite to Create Ultimate A.I. Challenge

In a bold move to gauge artificial intelligence’s true potential, the Center for AI Safety and Scale AI have unveiled an ambitious project.

Dubbed “Humanity’s Last Exam,” this initiative aims to create the world’s most demanding AI benchmark.

The project responds to the rapid strides made in AI capabilities, with recent models outperforming existing tests.

The exam’s primary goal is to measure AI progress towards expert-level abilities across various fields.

It seeks to address the shortcomings of current benchmarks that have become too simple for advanced AI models.

By setting a higher standard, the exam encourages the development of AI systems with deeper cognitive skills.

Experts from diverse fields are invited to contribute challenging questions to the exam. The project offers substantial incentives, including co-authorship opportunities and significant cash prizes.

This approach aims to attract high-quality submissions from experienced professionals and academics.

The exam will comprise at least 1,000 crowd-sourced questions, due by November 1, 2024. A rigorous peer review process will ensure the quality and relevance of these submissions.

To maintain the benchmark’s integrity, some questions will remain private, preventing AI systems from memorizing answers.

A $500,000 prize pool has been allocated to reward contributors. The top 50 questions will earn $5,000 each, while the next 500 will receive $500 each.

Beyond financial rewards, successful submissions will grant co-authorship on the resulting paper, offering recognition in academic circles.

The project has already attracted submissions from researchers at prestigious institutions like MIT, UC Berkeley, and Stanford.

This collaborative effort brings together experts from various fields to create a comprehensive and challenging benchmark.

Question guidelines emphasize complexity and originality. Submissions should be difficult for non-experts and not easily answerable through quick online searches.

Humanity’s Last Exam: Experts Unite to Create Ultimate A.I. Challenge

Ideally, contributors should have extensive experience in technical industries or advanced academic training.

The exam requires questions to be objective, with answers accepted by other experts in the field. It prohibits questions related to weaponization or sensitive topics.

The focus is on abstract reasoning skills, moving beyond simple memorization or undergraduate-level knowledge.

By setting a new bar for AI assessment, “Humanity’s Last Exam” aims to influence AI research and development.

This project could potentially drive significant investments in the field and provide valuable insights into the current capabilities of frontier AI models.