With the amount of new subnets being added it can be hard to get up to date information across all subnets, so data may be slightly out of date from time to time
With the amount of new subnets being added it can be hard to get up to date information across all subnets, so data may be slightly out of date from time to time
Bittensor Subnet 81 “Grail” is a decentralized AI training network that enables permissionless, collaborative training of large-scale machine learning models. In essence, Grail lets a global community of miners (contributors with hardware) collectively train a shared large language model (LLM) by pooling their compute power, with all coordination handled on-chain via the Bittensor protocol. The subnet was acquired and rebranded as “Grail” in mid-2025 (it was previously known as Patrol under Tensora Group).
Its core purpose is to crowdsource the training of AI models in an open, incentive-aligned way – often referred to as the “holy grail” of decentralized AI research, since cracking distributed training would allow communities to rival the giant centralized AI labs. In Grail’s network, any participant around the world with the required hardware can join to train the model and earn rewards, making it a permissionless and trust-minimized system for AI development. By leveraging Bittensor’s blockchain, Grail ensures that contributors are fairly incentivized in cryptocurrency for the value of their work, while producing an open-source model that is co-owned by the community. Ultimately, Grail aims to democratize AI – enabling state-of-the-art AI models to be built by the community rather than behind closed doors, and ensuring that the benefits (and token rewards) are distributed to the many contributors who help train these models.
Bittensor Subnet 81 “Grail” is a decentralized AI training network that enables permissionless, collaborative training of large-scale machine learning models. In essence, Grail lets a global community of miners (contributors with hardware) collectively train a shared large language model (LLM) by pooling their compute power, with all coordination handled on-chain via the Bittensor protocol. The subnet was acquired and rebranded as “Grail” in mid-2025 (it was previously known as Patrol under Tensora Group).
Its core purpose is to crowdsource the training of AI models in an open, incentive-aligned way – often referred to as the “holy grail” of decentralized AI research, since cracking distributed training would allow communities to rival the giant centralized AI labs. In Grail’s network, any participant around the world with the required hardware can join to train the model and earn rewards, making it a permissionless and trust-minimized system for AI development. By leveraging Bittensor’s blockchain, Grail ensures that contributors are fairly incentivized in cryptocurrency for the value of their work, while producing an open-source model that is co-owned by the community. Ultimately, Grail aims to democratize AI – enabling state-of-the-art AI models to be built by the community rather than behind closed doors, and ensuring that the benefits (and token rewards) are distributed to the many contributors who help train these models.
Grail is essentially the deployment of Templar’s decentralized training framework on Subnet 81, comprising a network of software miners and validators that cooperate via the blockchain to train a shared machine learning model. The “product” is twofold: (1) the protocol and infrastructure that perform this distributed training, and (2) the resulting AI model that the network produces. Technically, Grail’s build includes several key components and innovations:
Miners (Training Nodes): These are participant nodes running the Grail miner software on their GPUs to train the model. Each miner receives a deterministic slice of the training dataset and computes a “pseudo-gradient” update on the model using that data. Miners operate in synchronous rounds (e.g. ~84-second windows) where they intensively train on their data shard and then upload their gradient contributions to a decentralized storage bucket (Grail uses a cloud storage layer). The miners’ goal is to produce gradients that improve the global model’s performance (i.e. lower the loss on the given data) more effectively than other peers. They do this without any central coordinator – instead, coordination happens via the Bittensor chain and shared storage: miners fetch data, compute updates, and submit those updates all according to the timetable enforced by the blockchain.
Validators (Evaluation Nodes): Grail’s validators are specialized nodes that download the miners’ submitted gradients from the storage and evaluate their quality. A validator will test each gradient by applying it to a copy of the model and measuring the loss reduction on a validation dataset sample. Essentially, the validator serves as an impartial judge: checking that the gradient was submitted on time (using timestamps compared against blockchain block times) and that it actually provides a meaningful improvement to the model’s accuracy. If a miner’s gradient fails to beat a baseline (or if it’s submitted outside the allowed window), the validator will flag it as a poor contribution – such low-quality or late submissions lead to the miner getting slashed or receiving little to no reward for that round. High-quality contributions, on the other hand, earn the miner higher token rewards, proportional to the measured improvement their work provided. The validator then aggregates approved gradients and applies them to update the global model parameters. In Grail’s architecture, an aggregator component may also be used to accumulate the gradients and periodically save model checkpoints.
Bittensor Blockchain Integration: The Bittensor (TAO) chain underpins the whole coordination and incentive mechanism. Grail is built as a Bittensor subnet, meaning the blockchain keeps track of participant registrations (miners/validators join the subnet by bonding tokens and are identified on-chain) and handles weight setting and reward payouts based on contribution. For each training round, the validator posts scores or “weights” for miners to the chain, reflecting their performance. These on-chain weights determine how the block rewards (in the subnet’s native token) are allocated among miners – effectively paying each contributor in proportion to the utility of their gradient in improving the model. The use of blockchain provides a trustless ledger of contributions and rewards, ensuring transparency and deterring cheating in the absence of a central authority.
Communication & Data Pipeline: Grail implements a peer-to-peer communication system for exchanging model updates and data. Rather than directly sending gradients to each other (which would be bandwidth-intensive), miners upload their gradients to the shared storage bucket and use Bittensor’s built-in networking to signal availability. This design decouples the expensive data transfer from the blockchain’s limited bandwidth. The Grail software employs gradient compression techniques to reduce communication overhead – for example, using algorithms like Decoupled Momentum (DeMo) with Top-K sparse updates and even applying transforms (like discrete cosine transform) to compress gradients before upload. These techniques are crucial to make distributed training feasible over the internet by cutting down the data each miner must send while preserving the effectiveness of updates. Grail’s framework also ensures that each miner performs unique work (e.g. each miner gets a different data slice) to avoid redundancy; a mechanism is in place to assure miners aren’t all submitting the same gradient or copying one another’s work. This uniqueness check, combined with an OpenSkill rating system that tracks each miner’s performance over time, helps maintain healthy competition and continuous contribution quality.
In summary, Grail’s build is a distributed training pipeline. The system continuously iterates through training rounds where miners train the shared model on their local data shards and validators merge the useful results. The end “product” is a collaboratively-trained AI model (for instance, a language model with on the order of a billion+ parameters) that is co-owned by the community of contributors. Notably, Grail is one of the first examples of a fully incentive-driven, permissionless AI training run in practice – an extension of the pioneering work done on Subnet 3 (Templar) which proved that this concept can work at smaller scale. With Grail, the Templar team is pushing that vision further, using the lessons learned and improved software (sometimes dubbed “Gauntlet” for its incentive mechanism and other code optimizations) to tackle even larger models and more complex training regimes. All of the software is open-source (the Templar team’s code repositories are publicly available) and the network operates without centralized control – making Grail a truly decentralized AI training platform.
Grail is essentially the deployment of Templar’s decentralized training framework on Subnet 81, comprising a network of software miners and validators that cooperate via the blockchain to train a shared machine learning model. The “product” is twofold: (1) the protocol and infrastructure that perform this distributed training, and (2) the resulting AI model that the network produces. Technically, Grail’s build includes several key components and innovations:
Miners (Training Nodes): These are participant nodes running the Grail miner software on their GPUs to train the model. Each miner receives a deterministic slice of the training dataset and computes a “pseudo-gradient” update on the model using that data. Miners operate in synchronous rounds (e.g. ~84-second windows) where they intensively train on their data shard and then upload their gradient contributions to a decentralized storage bucket (Grail uses a cloud storage layer). The miners’ goal is to produce gradients that improve the global model’s performance (i.e. lower the loss on the given data) more effectively than other peers. They do this without any central coordinator – instead, coordination happens via the Bittensor chain and shared storage: miners fetch data, compute updates, and submit those updates all according to the timetable enforced by the blockchain.
Validators (Evaluation Nodes): Grail’s validators are specialized nodes that download the miners’ submitted gradients from the storage and evaluate their quality. A validator will test each gradient by applying it to a copy of the model and measuring the loss reduction on a validation dataset sample. Essentially, the validator serves as an impartial judge: checking that the gradient was submitted on time (using timestamps compared against blockchain block times) and that it actually provides a meaningful improvement to the model’s accuracy. If a miner’s gradient fails to beat a baseline (or if it’s submitted outside the allowed window), the validator will flag it as a poor contribution – such low-quality or late submissions lead to the miner getting slashed or receiving little to no reward for that round. High-quality contributions, on the other hand, earn the miner higher token rewards, proportional to the measured improvement their work provided. The validator then aggregates approved gradients and applies them to update the global model parameters. In Grail’s architecture, an aggregator component may also be used to accumulate the gradients and periodically save model checkpoints.
Bittensor Blockchain Integration: The Bittensor (TAO) chain underpins the whole coordination and incentive mechanism. Grail is built as a Bittensor subnet, meaning the blockchain keeps track of participant registrations (miners/validators join the subnet by bonding tokens and are identified on-chain) and handles weight setting and reward payouts based on contribution. For each training round, the validator posts scores or “weights” for miners to the chain, reflecting their performance. These on-chain weights determine how the block rewards (in the subnet’s native token) are allocated among miners – effectively paying each contributor in proportion to the utility of their gradient in improving the model. The use of blockchain provides a trustless ledger of contributions and rewards, ensuring transparency and deterring cheating in the absence of a central authority.
Communication & Data Pipeline: Grail implements a peer-to-peer communication system for exchanging model updates and data. Rather than directly sending gradients to each other (which would be bandwidth-intensive), miners upload their gradients to the shared storage bucket and use Bittensor’s built-in networking to signal availability. This design decouples the expensive data transfer from the blockchain’s limited bandwidth. The Grail software employs gradient compression techniques to reduce communication overhead – for example, using algorithms like Decoupled Momentum (DeMo) with Top-K sparse updates and even applying transforms (like discrete cosine transform) to compress gradients before upload. These techniques are crucial to make distributed training feasible over the internet by cutting down the data each miner must send while preserving the effectiveness of updates. Grail’s framework also ensures that each miner performs unique work (e.g. each miner gets a different data slice) to avoid redundancy; a mechanism is in place to assure miners aren’t all submitting the same gradient or copying one another’s work. This uniqueness check, combined with an OpenSkill rating system that tracks each miner’s performance over time, helps maintain healthy competition and continuous contribution quality.
In summary, Grail’s build is a distributed training pipeline. The system continuously iterates through training rounds where miners train the shared model on their local data shards and validators merge the useful results. The end “product” is a collaboratively-trained AI model (for instance, a language model with on the order of a billion+ parameters) that is co-owned by the community of contributors. Notably, Grail is one of the first examples of a fully incentive-driven, permissionless AI training run in practice – an extension of the pioneering work done on Subnet 3 (Templar) which proved that this concept can work at smaller scale. With Grail, the Templar team is pushing that vision further, using the lessons learned and improved software (sometimes dubbed “Gauntlet” for its incentive mechanism and other code optimizations) to tackle even larger models and more complex training regimes. All of the software is open-source (the Templar team’s code repositories are publicly available) and the network operates without centralized control – making Grail a truly decentralized AI training platform.
Grail is developed and managed by the Templar team, an experienced group of engineers and researchers dedicated to decentralized AI. The Templar organization (sometimes referred to as Templar AI or tplr.ai) previously launched Subnet 3 “Templar” – the world’s first distributed, permissionless LLM training subnet – and also created Subnet 39 “Basilica” focusing on trustless GPU compute. The team is led by Samuel Dare, a blockchain veteran and visionary who helped conceive the idea of incentive-driven AI training (“Sam” often goes by the handle “distributed” in the community). Alongside Sam, Templar includes AI researchers such as Joel Lidin, Amir Sarfi, and Evangelos Pappas, who co-authored a 2025 research paper detailing the Grail/Templar training framework and its incentive system. These individuals (affiliated with Templar AI) bring expertise from both machine learning and crypto – for example, one co-author, Eugene Belilovsky, is a research professor at Mila/Concordia who advises on the project, and Jacob Steeves (another co-author) is a founder of the OpenTensor Foundation.
In practice, the Templar team functions as the initial “custodians” of Subnet 81, meaning they set up the network, deploy code updates, and guide its development in these early stages. However, they emphasize that Grail is a community endeavor – the end goal is to decentralize control as the network matures. The team has a strong track record: their first subnet (Templar SN3) successfully demonstrated distributed training with ~200 GPUs on a 1.2B parameter model, overcoming numerous technical hurdles and exploits through rapid iteration and community collaboration. They have fostered an open developer community around these subnets, encouraging miners and validators to contribute not just compute but also improvements to the code. Key members like Sam (“distributed”) are active in public forums, sharing progress and inviting feedback. The team also operates under a thematic ethos (drawn from Knights Templar lore) – with project names like Templar, Basilica, and Grail, reflecting their mission to build “sacred” infrastructure for AI. Templar’s core contributors are passionate about AI decentralization, and they are continually recruiting talent (researchers, ML engineers) to join the cause of “building the future of decentralized AI training,” as noted on their website.
Grail is developed and managed by the Templar team, an experienced group of engineers and researchers dedicated to decentralized AI. The Templar organization (sometimes referred to as Templar AI or tplr.ai) previously launched Subnet 3 “Templar” – the world’s first distributed, permissionless LLM training subnet – and also created Subnet 39 “Basilica” focusing on trustless GPU compute. The team is led by Samuel Dare, a blockchain veteran and visionary who helped conceive the idea of incentive-driven AI training (“Sam” often goes by the handle “distributed” in the community). Alongside Sam, Templar includes AI researchers such as Joel Lidin, Amir Sarfi, and Evangelos Pappas, who co-authored a 2025 research paper detailing the Grail/Templar training framework and its incentive system. These individuals (affiliated with Templar AI) bring expertise from both machine learning and crypto – for example, one co-author, Eugene Belilovsky, is a research professor at Mila/Concordia who advises on the project, and Jacob Steeves (another co-author) is a founder of the OpenTensor Foundation.
In practice, the Templar team functions as the initial “custodians” of Subnet 81, meaning they set up the network, deploy code updates, and guide its development in these early stages. However, they emphasize that Grail is a community endeavor – the end goal is to decentralize control as the network matures. The team has a strong track record: their first subnet (Templar SN3) successfully demonstrated distributed training with ~200 GPUs on a 1.2B parameter model, overcoming numerous technical hurdles and exploits through rapid iteration and community collaboration. They have fostered an open developer community around these subnets, encouraging miners and validators to contribute not just compute but also improvements to the code. Key members like Sam (“distributed”) are active in public forums, sharing progress and inviting feedback. The team also operates under a thematic ethos (drawn from Knights Templar lore) – with project names like Templar, Basilica, and Grail, reflecting their mission to build “sacred” infrastructure for AI. Templar’s core contributors are passionate about AI decentralization, and they are continually recruiting talent (researchers, ML engineers) to join the cause of “building the future of decentralized AI training,” as noted on their website.
Grail’s roadmap is ambitious, targeting both technical improvements and massive scale-up in model size. In the near term, the team has been focused on perfecting the current training runs on smaller models (around 1.2B parameters) to ensure stability, reproducibility, and robust learning curves. This involves fine-tuning the incentive algorithms and synchronization mechanisms so that the network can train smoothly without stalling or being gamed by adversarial miners. One key upgrade on the horizon is implementing asynchronous training (and gradient accumulation) instead of the strictly synchronous round-by-round approach. Asynchronous training would allow miners and validators to operate in a more pipeline-parallel fashion – not all waiting for each global sync point – which is expected to massively boost throughput and efficiency. The team believes that by reducing idle time and letting gradients propagate continuously, the decentralized network could even outperform traditional centralized training “pound-for-pound” in terms of hardware utilization. Achieving this will require careful engineering (to handle out-of-order updates, staleness, etc.), but it’s a high priority on the roadmap given its potential to speed up training dramatically.
In parallel, the Templar developers are rolling out other optimizations and new features. For example, they introduced a novel optimizer called “CCLoco” (Chunked Compressed Local Optimizer) to improve communication efficiency. CCLoco combines local-update methods with gradient compression, reducing the bandwidth required per training step. Such algorithmic advances will be integrated into Grail to ensure it can scale to larger models without overwhelming the network. Additionally, security and integrity remain a focus – the team continuously refines mechanisms to prevent cheating (e.g. validators now ensure each miner’s work is unique and timely, as noted earlier) and to penalize bad actors. There is also an expectation that Subnet 39 Basilica (Templar’s sister subnet) will play a role in Grail’s future: Basilica is building a “trustless marketplace for GPU compute” with on-chain hardware verification. In the future, Grail could leverage Basilica’s tech to verify miners’ hardware proofs or even to rent external GPUs securely, thereby expanding its compute capacity while maintaining trust. This would allow Grail to tap into a larger pool of GPUs (potentially even cloud or data center resources) in a decentralized manner, accelerating training further.
Looking a bit further ahead, Grail’s ultimate goal is to tackle truly large-scale models. The team has openly stated their intention to train models on the order of 70 billion parameters and beyond – essentially to produce the world’s largest open-source LLMs through the decentralized approach. This is a bold target (comparable to GPT-3/GPT-4 scales) and will require significant advances in both software and the number of participating miners. To get there, the roadmap includes: increasing the network’s capacity (more miners with more powerful GPUs, facilitated by things like Basilica), refining the incentive design to handle long training runs, and possibly sharding or modularizing the model training to scale out (e.g. techniques like model parallelism or multi-subnet collaboration could come into play in the long run). As the network scales, governance will also become important – the Templar team plans to gradually decentralize the governance of Grail, likely by involving the community in decision-making and possibly introducing on-chain governance mechanisms for the subnet. They’ve described themselves only as “current custodians” and stress that subnet decentralization is the end goal. This means over time we can expect Grail to transition to community control, with miners/validators and token holders voting on upgrades or parameters, much like how some blockchain projects hand off to their communities.
In summary, the roadmap for Grail involves scaling up and opening up: scaling to larger models and faster training via technical breakthroughs (asynchronous updates, better optimizers, more compute), and opening up the network through decentralization of both infrastructure (more participants, possibly leveraging Basilica’s GPU network) and governance. If successful, Grail will not only train cutting-edge AI models in a decentralized way, but also prove that a global collaborative network can achieve what only giant tech companies have done so far. The vision is that within the next couple of years, Grail could produce a state-of-the-art, community-owned AI model that rivals the best from industry labs – a milestone that would validate the entire Bittensor paradigm of incentive-aligned decentralized AI. Each step on the roadmap – from near-term stability improvements to long-term multi-billion-parameter training – is bringing the project closer to that “Holy Grail” of AI: an open, distributed, and autonomous process for creating intelligence.
Grail’s roadmap is ambitious, targeting both technical improvements and massive scale-up in model size. In the near term, the team has been focused on perfecting the current training runs on smaller models (around 1.2B parameters) to ensure stability, reproducibility, and robust learning curves. This involves fine-tuning the incentive algorithms and synchronization mechanisms so that the network can train smoothly without stalling or being gamed by adversarial miners. One key upgrade on the horizon is implementing asynchronous training (and gradient accumulation) instead of the strictly synchronous round-by-round approach. Asynchronous training would allow miners and validators to operate in a more pipeline-parallel fashion – not all waiting for each global sync point – which is expected to massively boost throughput and efficiency. The team believes that by reducing idle time and letting gradients propagate continuously, the decentralized network could even outperform traditional centralized training “pound-for-pound” in terms of hardware utilization. Achieving this will require careful engineering (to handle out-of-order updates, staleness, etc.), but it’s a high priority on the roadmap given its potential to speed up training dramatically.
In parallel, the Templar developers are rolling out other optimizations and new features. For example, they introduced a novel optimizer called “CCLoco” (Chunked Compressed Local Optimizer) to improve communication efficiency. CCLoco combines local-update methods with gradient compression, reducing the bandwidth required per training step. Such algorithmic advances will be integrated into Grail to ensure it can scale to larger models without overwhelming the network. Additionally, security and integrity remain a focus – the team continuously refines mechanisms to prevent cheating (e.g. validators now ensure each miner’s work is unique and timely, as noted earlier) and to penalize bad actors. There is also an expectation that Subnet 39 Basilica (Templar’s sister subnet) will play a role in Grail’s future: Basilica is building a “trustless marketplace for GPU compute” with on-chain hardware verification. In the future, Grail could leverage Basilica’s tech to verify miners’ hardware proofs or even to rent external GPUs securely, thereby expanding its compute capacity while maintaining trust. This would allow Grail to tap into a larger pool of GPUs (potentially even cloud or data center resources) in a decentralized manner, accelerating training further.
Looking a bit further ahead, Grail’s ultimate goal is to tackle truly large-scale models. The team has openly stated their intention to train models on the order of 70 billion parameters and beyond – essentially to produce the world’s largest open-source LLMs through the decentralized approach. This is a bold target (comparable to GPT-3/GPT-4 scales) and will require significant advances in both software and the number of participating miners. To get there, the roadmap includes: increasing the network’s capacity (more miners with more powerful GPUs, facilitated by things like Basilica), refining the incentive design to handle long training runs, and possibly sharding or modularizing the model training to scale out (e.g. techniques like model parallelism or multi-subnet collaboration could come into play in the long run). As the network scales, governance will also become important – the Templar team plans to gradually decentralize the governance of Grail, likely by involving the community in decision-making and possibly introducing on-chain governance mechanisms for the subnet. They’ve described themselves only as “current custodians” and stress that subnet decentralization is the end goal. This means over time we can expect Grail to transition to community control, with miners/validators and token holders voting on upgrades or parameters, much like how some blockchain projects hand off to their communities.
In summary, the roadmap for Grail involves scaling up and opening up: scaling to larger models and faster training via technical breakthroughs (asynchronous updates, better optimizers, more compute), and opening up the network through decentralization of both infrastructure (more participants, possibly leveraging Basilica’s GPU network) and governance. If successful, Grail will not only train cutting-edge AI models in a decentralized way, but also prove that a global collaborative network can achieve what only giant tech companies have done so far. The vision is that within the next couple of years, Grail could produce a state-of-the-art, community-owned AI model that rivals the best from industry labs – a milestone that would validate the entire Bittensor paradigm of incentive-aligned decentralized AI. Each step on the roadmap – from near-term stability improvements to long-term multi-billion-parameter training – is bringing the project closer to that “Holy Grail” of AI: an open, distributed, and autonomous process for creating intelligence.
A big thank you to Tao Stats for producing these insightful videos in the Novelty Search series. We appreciate the opportunity to dive deep into the groundbreaking work being done by Subnets within Bittensor! Check out some of their other videos HERE.
Recorded August 2025: This Tao Stats Novelty Search session wrangles a lively panel to unveil Covenant’s three-part push across Bittensor—Templar (pre-training), Basilica (compute), and Grail (post-training/RL). The team spotlights a big research leap: Sparse-LoCo, a decentralized training optimizer that combines top-k compression with 2-bit quantization to slash communication while improving accuracy, enabling a permissionless 70B-parameter run. Basilica is positioned as a compute network that will evolve beyond “rentals” into value-added services like verifiable inference and hardware-efficiency tricks to cut the “Jensen tax.” Grail targets single- then multi-turn RL, plus a fast hidden-state “fingerprint” to verify miners’ outputs and model usage. Together—with a coming rebrand to Covenant.ai—the trio aims to turn open research into production pipelines while keeping incentives aligned and results shareable.
Introducing SparseLoCo: a communication-efficient method for LLM pre-training.
TL;DR: We leverage Top-k sparsification + error feedback with DiLoCo’s infrequent outer steps—communicating only 1–3% gradients with 2-bit quantization—outperforming DiLoCo and DeMo. 1/N,
ArXiv:…
Very cool work with @amir_sarfi @benjamintherien and @tplr_ai
What if I told you how to outperform DiLoCo while only communicating 1-3% of the pseudogradient?
https://x.com/amir_sarfi/status/1958714182750077215
The Covenant is unveiled.
Our new sanctuary is live:
One covenant, many orders.
Templar. Basilica. Grail.
@tplr_ai x @basilic_ai x @grail_ai
Covenant
One Covenant, Many Orders.
covenant.ai
If i spent the rest of my life building on @opentensor , it would be a life well lived.
Our sacred pact.
Bittensor will claim its place in history
Covenant
One Covenant, Many Orders.
www.covenant.ai
Ladies and gentlmen….. it’s with a great pleasure….. Covenant
$TAO
Covenant
One Covenant, Many Orders.
www.covenant.ai
