With the amount of new subnets being added it can be hard to get up to date information across all subnets, so data may be slightly out of date from time to time
With the amount of new subnets being added it can be hard to get up to date information across all subnets, so data may be slightly out of date from time to time
This subnet uses a distributed approach to train Large Language Models on web-based datasets. Their proposed solution is a subnet that incentivizes compute, bandwidth, and latency. Compute resources drive the training of each miner’s local model, while bandwidth and latency facilitate the averaging of local model weights using a process called butterfly all-reduce. Once this process is completed, every miner receives a unified global averaged gradient to update their model weights.
This subnet uses a distributed approach to train Large Language Models on web-based datasets. Their proposed solution is a subnet that incentivizes compute, bandwidth, and latency. Compute resources drive the training of each miner’s local model, while bandwidth and latency facilitate the averaging of local model weights using a process called butterfly all-reduce. Once this process is completed, every miner receives a unified global averaged gradient to update their model weights.
Training Process:
Miners train the collective model on specific dataset segments. The training is iterative, with both local and global tracking of epochs and steps. Miners perform local training on their assigned data and participate in gradient averaging using the butterfly all-reduce method.
Dataset:
The subnet utilizes the “HuggingFaceFW/fineweb” dataset with the “sample-350BT” configuration.
Data is streamed in real-time from Hugging Face servers for efficient large-scale data handling.
Text is tokenized with the GPT-2 tokenizer (“distilgpt2”).
Model Submission:
After each gradient averaging step, miners push the updated model to the Hugging Face Hub.
The model is tagged with the current epoch number.
In case of upload failure, the system retries within a set limit.
Validation:
Validators perform two main queries: “Train” and “AllReduce.”
For “Train” queries, validators check miners’ loss, gradients, and dataset indices.
For “AllReduce” queries, they initiate gradient averaging and verify miner participation.
Incentive Mechanism:
Bandwidth Score: Measures miners’ efficiency in sharing model states.
Gradient Score: Compares miner-reported gradients to validator-calculated gradients.
Steps Score: Rewards miners based on the volume of data trained in each step.
Training Process:
Miners train the collective model on specific dataset segments. The training is iterative, with both local and global tracking of epochs and steps. Miners perform local training on their assigned data and participate in gradient averaging using the butterfly all-reduce method.
Dataset:
The subnet utilizes the “HuggingFaceFW/fineweb” dataset with the “sample-350BT” configuration.
Data is streamed in real-time from Hugging Face servers for efficient large-scale data handling.
Text is tokenized with the GPT-2 tokenizer (“distilgpt2”).
Model Submission:
After each gradient averaging step, miners push the updated model to the Hugging Face Hub.
The model is tagged with the current epoch number.
In case of upload failure, the system retries within a set limit.
Validation:
Validators perform two main queries: “Train” and “AllReduce.”
For “Train” queries, validators check miners’ loss, gradients, and dataset indices.
For “AllReduce” queries, they initiate gradient averaging and verify miner participation.
Incentive Mechanism:
Bandwidth Score: Measures miners’ efficiency in sharing model states.
Gradient Score: Compares miner-reported gradients to validator-calculated gradients.
Steps Score: Rewards miners based on the volume of data trained in each step.
Keep ahead of the Bittensor exponential development curve…
Subnet Alpha is an informational platform for Bittensor Subnets.
This site is not affiliated with the Opentensor Foundation or TaoStats.
The content provided on this website is for informational purposes only. We make no guarantees regarding the accuracy or currency of the information at any given time.
Subnet Alpha is created and maintained by The Realistic Trader. If you have any suggestions or encounter any issues, please contact us at [email protected].
Copyright 2024