China's AI Breakthrough: A Technological Leap That Shook Silicon Valley
China's latest AI development, spearheaded by the research lab Deepseek, has caused a stir in the tech world. This breakthrough isn't from the usual suspects like OpenAI, Google, or Meta, making it a significant development to monitor closely. This model has opened the eyes of many to the advancements happening in AI within China.
Deepseek's Impressive Model: Cost and Performance
Deepseek claims it developed its model in just two months with less than $6 million, a stark contrast to the years and hundreds of millions of dollars spent by Google and OpenAI. The model has quickly become the best open-source option, attracting American developers to build upon it. The AI lab reportedly spent only $5.6 million to build Deepseek version 3. Consider OpenAI, which is spending $5 billion a year, and Google, which expects capital expenditures in 2024 to soar to over $50 billion. Microsoft has shelled out more than $13 billion just to invest in OpenAI.
Deepseek's model has outperformed lavishly funded American counterparts, beating Meta's Llama, OpenAI's GPT-4o, and Anthropic's Claude Sonnet 3.5 in accuracy across various tests. These tests included a subset of 500 math problems, AI math evaluation, coding competitions, and bug identification/fixing tasks. They followed it up with a new reasoning model called R1, which also outperformed OpenAI's o1 in some third-party tests.
Navigating Semiconductor Restrictions
Despite strict semiconductor restrictions imposed by the U.S. government, which limit China's access to computing power, Deepseek found a workaround. They utilized Nvidia's less performant H-800s, demonstrating that the intended chokehold from chip export controls wasn't as effective as planned. By using the available hardware more efficiently, Deepseek sidestepped the rules.
Unveiling Deepseek: Origins and Mission
Little is publicly known about Deepseek and its founder, Liang Wenfeng. Chinese media reports suggest Deepseek originated from a Chinese hedge fund, High Flyer Quant, which manages approximately $8 billion in assets. Their mission statement on the developer site reads: "unravel the mystery of AGI with curiosity. Answer the essential question with long-termism."
The Broader Chinese AI Landscape
Deepseek isn't the only Chinese AI model making waves. Kai Fu Lee's startup, Zero One Dot AI, has quickly become a unicorn, attracting attention and generating significant revenue. Alibaba's Qwen is also cutting costs on its large language models to attract developers, signaling an ongoing competitive race.
Shifting Perceptions of China's AI Capabilities
China's AI breakthrough challenges previous assumptions about the country's position in the AI race. Eric Schmidt, former Google CEO, initially estimated that China was 2-3 years behind the U.S. in early 2024. He has since revised his assessment, acknowledging that China has caught up remarkably quickly. This progress questions the true extent of the advantage held by U.S. AI labs.
The Impact of Open-Source Models
The availability of powerful open-source models like Deepseek's allows developers to bypass the capital-intensive process of building and training models from scratch. By building on existing models, developers can reach the forefront of AI with smaller budgets and teams. Deepseek's approach involves iterating on existing technology, using available datasets, and applying innovative tweaks. This has opened the eyes of Silicon Valley to what can be accomplished with 10, 15, 20, or 30 million dollars.
Deepseek's "Identity Crisis" and OpenAI's Response
Deepseek's model has exhibited an "identity crisis," sometimes identifying as a GPT-4 based model from OpenAI. This led OpenAI CEO Sam Altman to comment on the relative ease of copying existing work compared to creating something new and difficult. Deepseek's approach involves emulating GPT while adding its own enhancements.
Copying and Innovation in the AI Field
The AI field is characterized by widespread copying, with different companies drawing inspiration from each other. This raises the question of whether massive spending on individual LLMs is a worthwhile investment, especially considering that OpenAI, despite raising substantial funding, has yet to turn a profit.
Reasoning as the Key to Future AI
Reasoning is becoming increasingly crucial for AI models, requiring them to analyze, draw conclusions, and solve complex problems. While OpenAI's o1 reasoning model is currently cutting-edge, researchers are demonstrating the ability to create reasoning models for significantly less money. This suggests that creativity and innovation may be as important as capital for staying ahead.
The Implications of U.S. Chip Restrictions
U.S. chip restrictions, intended to slow down China's progress in AI, may have inadvertently spurred innovation and efficiency. By being forced to find workarounds, Chinese researchers have developed more efficient models.
The Danger of China Owning the Ecosystem
The open-source nature of Deepseek's model allows developers full access and customization, potentially leading to widespread adoption. This could undermine U.S. leadership and embed China more deeply into the global tech infrastructure. The fear is that the adoption of a Chinese open-source model at scale could lead to China owning the mindshare and ecosystem.
Concerns about Values and Censorship
AI models built in China are subject to state regulations, requiring them to adhere to "core socialist values." These models may censor historical events, deny human rights abuses, and filter criticism of Chinese leaders. This raises concerns about the development of autocratic AI versus democratic AI.
The Stakes of the AI Race
The U.S. and China are the only two countries currently capable of building AI at scale, making the stakes of the AI race incredibly high. The consequences of this competition will have enormous impacts on the global landscape.
Perplexity's CEO on China's Advantages and Disadvantages
Arvind Srinivas, co-founder and CEO of Perplexity, discussed China's advantages and disadvantages in competing with the U.S. China faces hardware limitations due to restricted access to advanced GPUs. However, necessity has driven them to find more efficient solutions. He noted that China's most dangerous move is creating the best open-source model on which American developers are building.
Overcoming Challenges Through Innovation
Srinivas highlighted that the surprise from Deepseek was the amount of clever solutions they came up with. Training a mixture of experts model is not easy, because of irregular loss spikes. They also figured out floating point-8 bit training.
The Changing Landscape of AI and Investment
The achievement of Deepseek has changed the AI landscape. China can now be said to be catching up to America. What was once thought to be behind, could be on par with the US. Spending will now be focused on reasoning.