jakiro
Author: Daniel Barabander, General Counsel and Partner of Variant Fund; Translated by: 0xjs@Bitchain Vision
Key points of this article
-
At present, basic AI development is led by a few technology companies and is in a closed and anti-competitive state.
-
Open source software development is another option, but basic AI cannot be developed as a traditional open source software project (such as Linux) because of its “resource problems”, and open source contributors are also asked to donate computing and data costs beyond their personal capabilities..
-
Encryption solves resource issues by contributing to a basic open source AI project through ownership incentive resource providers.
-
Open source AI combined with encryption can support larger models and drive more innovation, leading to better AI.
introduction
A 2024 Pew Research Center poll showed that 64% of Americans believe that social media has a negative rather than positive impact on the United States, and 78% say that social media companies have too much power in today’s politics andInfluence, 83% said these platforms are likely or likely to intentionally censor political views they disagree with.Disgust for social media platforms is one of the few issues that unite Americans.
Looking back at the progress of social media experiments over the past 20 years, it seems inevitable that we will end up in this situation.You all know this story.A few large tech companies initially attracted attention and, most importantly, got user data.While people initially hoped that the data would be made public, the companies quickly changed direction and turned off access after using it to build an unbreakable network effect.This basically leads to the current situation, where less than ten large technology social media companies exist like small feudal territories under oligopoly and have no motivation to change because the current situation is extremely profitable.It is closed, anti-competitive.
Looking at the current progress of the AI experiment, I feel like I’m repeating the same movie, but this time it involves more.A handful of large tech companies have accumulated GPUs and data to build basic AI models and block access to these models.For new entrants (not raising billions of dollars) it is no longer possible to build a competitive version because the barrier to entry is too high – the computational capital expenditure of pre-training a base model is as high as billions of dollars, and fromSocial media companies that benefited from the last technological boom are leveraging their control over proprietary user data to build models that competitors cannot do.We are all in the AI space to reproduce what we do on social media: closed and anti-competitive.If we continue along this path of closed AI, a handful of tech companies will have unlimited control over the access to information and opportunities.
Open Source AI and “Resource Issues”
If we don’t want a closed AI world, what’s our alternative?The answer is obvious, that is, to build the basic model into an open source software project.We have countless examples of open source projects that build the underlying software we rely on every day.If Linux shows that something as basic as an operating system can also be built open source, then what’s the difference with LLM?
Unfortunately, there are some limitations in the underlying AI models that make them different from traditional software, which seriously hinders their feasibility as traditional open source software projects.Specifically, basicAIThe model itself requires computing and data resources beyond any individual’s capabilities.The result is that unlike traditional open source software projects that rely on people to donate time (which is already a challenging issue), open source AI also requires people to donate resources in the form of computing and data.This is the “resource problem” of open source AI.
To better understand resource issues, let’s take a look at Meta’s LLaMa model.Meta differs from its competitors (OpenAI, Google, etc.) in that it does not hide the model behind paid APIs, but instead publicly provides the weight of LLaMa for free for anyone to use (with some restrictions).These weights represent what the model learns from the training process of Meta and are necessary to run the model.With weights, anyone can fine-tune the model or use the output of the model as input to the new model.
While Meta announced the weight of LLaMa to be praised, it is not a truly open source software project.Meta trains the model privately using its own calculations, data and decisions and unilaterally decides when to open it to the world.Meta does not invite independent researchers/developers to the community, because individual community members cannot afford the compute or data resources needed to train or retrain models—ten thousand high-memory GPUs, data centers that house them, a large cooling basefacilities, and trillions of training data tokens.As Stanford University’s 2024 AI Index report states, “The rising training costs have actually excluded universities (traditionally the center of AI research) from developing their own cutting-edge basic models.” To understand the cost, Sam Altman mentionedGPT-4’s training cost is $100 million, and it may not include capital expenditure; Meta’s capital expenditure increased by $2.1 billion year-on-year (Q2 2024 vs. Q2 2023), mainly from pairs and training AI modelsRelated investments in server, data center and network infrastructure.Therefore, while LLaMa’s community contributors may have the technical ability to contribute and iterate on the basic model architecture, they still lack the means to do so.
In summary, unlike traditional open source software projects, open source software projects only require contributors to contribute time, while contributors to open source AI projects are required to contribute time and substantial costs in the form of computing and data.It is unrealistic to rely on goodwill and volunteer services to inspire enough parties to provide these resources.They need further motivation.176B Parameter Open Source LLM The success of BLOOM may be the best counterexample of goodwill and volunteer services for the development of open source AI, which involves 1,000 volunteer researchers from more than 70 countries and over 250 institutions.While this is undoubtedly an impressive achievement (I fully support it), it took a year to coordinate a training run and received a €3 million funding from French research institutions (and this fee is not included in the use ofThe capital expenditure of the supercomputer trains the model, one of the French institutions can already use it).The process of coordinating and relying on new grants to iterate BLOOM is too cumbersome and bureaucratic to rival the pace of large tech labs.While BLOOM has been on the run for over two years, I don’t know if this group has made any follow-up models.
To make open source AI possible, we need to incentivize resource providers to contribute their computing and data without the expenses of open source contributors.
Why can Crypto solve the resource problem of open source AI
Crypto’s breakthrough is to use ownership to make open source software projects with high resource costs possible.Crypto solves the resource problems inherent in open source AI by incentivizing speculative resource providers with potential upside upside rather than requiring open source contributors to provide these resources in advance.
To prove this, just look at the original crypto project Bitcoin.Bitcoin is an open source software project; the code running it is completely open, and it has been the case since the day the project began.But the code itself is not a secret weapon; downloading and running Bitcoin node software to create a blockchain that only exists on the local computer is not very useful.The software is only useful if the number of computational mining blocks is sufficient to exceed the computing power of any single contributor.Only in this way can the value-added software be realized: maintaining an uncontrolled ledger.Like Foundation Open Source AI, Bitcoin also represents an open source software project that requires resources beyond the capabilities of any single contributor.They may need this calculation for different reasons—Bitcoin is to make the network tamper-proof, while Foundation AI is to iterate the model—but more broadly, they all require resources beyond the needs of any single contributor.Only by doing so can it work as a viable open source software project.
Bitcoin, or any crypto network, is a magic trick to motivate participants to provide resources for open source software projects to provide network ownership in the form of tokens.As Jesse wrote in his founding paper for Variant back in 2020, ownership incentive resource providers contribute resources to projects in exchange for potential upside space for the network.This is similar to how to use sweat equity to start a fledgling company—by paying early employees (such as founders) primarily through ownership of the business, startups can overcome startup problems by obtaining a workforce that they would otherwise not be able to afford.Crypto extends the concept of sweat equity to resource providers, not just those who devote their time.Therefore, Variant focuses on investing in projects that use ownership to build network effects, such as Uniswap, Morpho and World.
If we want to make open source AI possible, then realizing ownership through crypto is the solution to the resource problems it faces.Researchers are free to contribute their model design ideas to open source projects, because the resources required to implement their ideas will be provided by computing and data providers in exchange for their ownership of the project, rather than asking these researchers to pay highUpfront cost.Ownership can take many different forms in open source AI, but what I am most excited about is ownership of the model itself, like the approach proposed by Pluralis.
Pluralis calls this approach a protocol model, where computing providers can contribute computing resources to train a specific open source model and gain ownership of future inference revenue for that model.Since ownership belongs to a specific model and the value of ownership is based on inference income, computing providers are motivated to choose the best model rather than cheat training (because providing useless training reduces the expected value of future inference income).Then the question becomes: How to enforce ownership on Pluralis if you need to send weights to the compute provider for training?The answer is that model parallelism is used to allocate model shards among workers, allowing the exploitation of a key property of the neural network: it can contribute to training larger models while only seeing a small portion of the total weight,This ensures that the complete weight set remains unextractable.And because of the many different models being trained on Pluralis, the trainer will have many different sets of weights, which makes recreating the model extremely difficult.This is the core concept of protocol models: they are trainable, can be used, but cannot be extracted from the protocol (no more computational power is needed to train the model from scratch).This addresses a concern often raised by open source AI critics that closed AI competitors will take up the labor outcomes of open projects.
Why Crypto+Open Source=Better AI
I began this post with a problem with control of big tech companies to illustrate from a normative perspective why closed AI is bad.But in a world where our online experiences are fatalistic, I worry that this may make no sense to most readers.So in the end I want to give two reasons, that open source AI powered by encryption will actually lead to better artificial intelligence.
First, the combination of Crypto and open source AI will allow us to reach the next basic model, as it will coordinate more resources than closed AI.Our current research shows that more resources exist in the form of computational and data means better models, which is why the underlying models often get bigger and bigger.Bitcoin shows us what open source software plus encryption unlocks in terms of computing power.It is the world’s largest and most powerful computing network, orders of magnitude larger than the cloud of large tech companies.Encryption turns isolated competition into cooperative competition.Resource providers are motivated to contribute their resources to solve collective problems rather than hoarding their resources to solve the problem individually (and redundantly).Open source AI using encryption will be able to leverage the world’s collective computing and data to build model sizes that are far beyond the possible potential of closed AI.Companies like Hyperbolic have demonstrated the power of leveraging collective computing resources that anyone can rent out GPUs on their open markets for a lower price.
Second, combining Crypto and open source AI will drive more innovation.This is because, if we can overcome the resource problem, we can return to the highly iterative and innovative open source nature of machine learning research.Prior to the recent launch of the basic LLM, machine learning researchers have been publicly publishing blueprints for their models and replicating models for decades.These models often use more limited open datasets and have manageable computational requirements, meaning that anyone can iterate over them.It is through this iteration that we have made progress in sequence modeling, such as RNN, LSTM, and attention mechanisms, which makes it possible to construct the “Transformer” model architecture that the current underlying LLM depends on.But all this has changed with the introduction of GPT-3 (which reversed the trend of open source GPT-2) and ChatGPT’s huge success.This is because OpenAI proves that if you invest enough computation and data on massive models, you can build LLMs that seem to understand human language.This has created resource problems, making it impossible for academia to afford high prices and has caused the lab of large tech companies to basically stop publishing their model architectures publicly to maintain a competitive advantage.The current state of relying mainly on individual laboratories will limit our ability to break through the boundaries of state-of-the-art technology.Open source AI implemented through encryption will mean that researchers will once again be able to continue this iterative process on cutting-edge models to discover the “next transformer.”
