The core purpose, as reiterated across multiple sources including the official GitHub repository, news analyses, and community discussions, is to incentivize the decentralized creation and continuous improvement of high-quality, open-source LLMs specifically tailored for role-playing scenarios. This is achieved by fostering a competitive environment where miners contribute models and validators evaluate them based on sophisticated scoring mechanisms.

Key goals highlighted include:

• Bridging the Quality Gap: Elevating open-source roleplay LLMs to be competitive with, or even surpass, closed-source alternatives.
• Fostering Empathy in AI: Developing LLMs that can engage in more natural, empathetic, and human-like interactions suitable for companionship and roleplay.
• Leveraging Decentralization: Utilizing the Bittensor network’s incentive structure to attract diverse talent and approaches to LLM development for roleplay.
• Powering the Dippy App: Integrating the top-performing models from the subnet into the Dippy.ai application, providing a real-world testing ground and direct user feedback loop.
• Open-Source Leadership: Establishing Impel Intelligence and the Dippy subnet as leaders in the open-source roleplaying LLM space.

Mechanism and Functionality

The Dippy Roleplay subnet (SN11) operates on a competitive, incentive-based mechanism inherent to the Bittensor network, specifically tailored to foster the creation of high-quality open-source roleplaying LLMs. The core of its functionality revolves around the interaction between two key participants: miners and validators, with the ultimate product being the refined LLMs themselves and the data/insights generated through their evaluation.

Miner Participation:

Miners on Subnet 11 are tasked with developing and submitting LLMs optimized for roleplay. Their process generally involves:

• Model Creation/Adaptation: Miners can employ various strategies to produce their models. This includes training models from scratch, fine-tuning existing open-source foundational models, or using advanced techniques like model merging (e.g., using tools like MergeKit) to combine the strengths of different architectures. The emphasis is on creating unique and effective roleplay LLMs.
• Model Submission: Once a miner has a candidate model, they submit it to a shared Hugging Face model pool designated by the subnet. Each miner typically registers one active model per unique identifier (UID) on the subnet.

Validator Participation and Evaluation Process:

Validators are crucial for assessing the quality and performance of the models submitted by miners. Their role involves a multi-faceted evaluation process:

• Model Retrieval and Assessment: Validators retrieve models from the submission pool and evaluate them using the Dippy protocol. This protocol outlines a specific, open scoring format detailed in the subnet’s documentation.
• Dynamic Scoring Mechanism: The scoring system is not static; it is designed to evolve and adapt based on advancements in SOTA model benchmarks and data. This ensures that the evaluation criteria remain relevant and continue to push the boundaries of roleplay LLM capabilities.
• Multi-Phase Scoring: The evaluation is a sophisticated process, often involving several phases to comprehensively assess a model’s suitability for roleplay:

1. Evaluation Score: Models are graded against a streaming dataset. While a reference dataset (DippyAI/dippy_synthetic_dataset_v1 on Hugging Face) is provided from previous day’s data, the actual evaluation occurs on a real-time generated sample. This score can be modified by a Creativity Score to penalize models that are overfit and lack originality. Model size and latency also contribute a small portion to this score.
2. Judge Score (Experimental Multiplier): An LLM judge (e.g., GPT-4o or similar) evaluates the model. If the submitted model wins or ties against others above a certain threshold (e.g., 30%, subject to change), it receives a significant score boost. This introduces a qualitative, comparative assessment.
3. Coherence Score (Binary Factor): Models generate conversations based on augmented data (e.g., from proj-persona/PersonaHub). The output is compared against a high-performing baseline model (like GPT-4o) for coherence. If a model fails to meet a certain coherence threshold, it automatically scores zero for that evaluation round, effectively filtering out nonsensical or irrelevant outputs.
4. Post Evaluation Score (Experimental Multiplier): This is an experimental phase aimed at aligning model performance with industry-standard benchmarks, though its exact implementation and weight were still being formalized at the time of documentation.

• Win Rate Calculation: After individual model scores are determined, validators compare each miner’s score against all other active models on the subnet to calculate a win rate. This relative performance metric is critical for determining incentives.
• Weight Assignment: Validators assign weights to miners based primarily on their win rate. To discourage simple model copying and encourage continuous innovation, modifiers such as a time penalty are applied. This means newer models generally need to score significantly better than established top models to achieve a high ranking quickly.

Incentive Mechanism and Product Output:
The overarching incentive mechanism follows the Bittensor standard: miners who produce high-performing models (as determined by validator scoring and win rates) receive a greater share of the TAO emissions allocated to Subnet 11. The collective output of this process is a continuously improving, open-source roleplay LLM, with the best-performing models potentially being integrated into the Dippy.ai application and made available to its large user base. This also generates valuable data on model performance and user preferences in roleplay scenarios.

Technical Architecture

The Dippy Roleplay subnet (SN11) is built using Python (version 3.9+) and leverages several key components and integrations to facilitate its operations. The technical architecture is designed to support the submission, evaluation, and ranking of roleplay LLMs in a decentralized manner.

Core Components (from the impel-intelligence/dippy-bittensor-subnet GitHub repository):

Neuron Implementations:

• miner.py: Contains the primary logic for miners. This script enables miners to prepare their roleplay LLMs, connect to the Bittensor network, register their models, and respond to validation requests from the subnet’s validators.
• validator.py: Houses the code for validator nodes. Validators use this to query miners, receive model outputs (or access submitted models), execute the scoring protocol, and set weights on the Bittensor network based on model performance.
• model_queue.py: An internal component, likely used by the subnet operators or core validation infrastructure for managing the queue of models submitted for evaluation.

Scoring Mechanism (scoring/ directory): This directory is critical as it contains all the code related to the LLM scoring process. This includes the implementation of the multi-phase evaluation (Evaluation Score, Creativity Score, Judge Score, Coherence Score, Post Evaluation Score) and the logic for comparing models against datasets and benchmarks. It also includes Jinja templates (scoring/prompt_templates) for common foundational models to handle different token systems and instruct syntaxes.

Utilities (utilities/ directory): Provides a collection of common utility functions essential for the subnet’s operation. This can include helper functions for interacting with the Bittensor network, managing Hugging Face repository interactions, and other validation support functions.

Documentation (docs/ directory): A comprehensive set of Markdown files offering guidance for participants. Key documents include:

• miner.md: Instructions for setting up and running a miner.
• validator.md: Instructions for setting up and running a validator.
• FAQ.md: Answers to frequently asked questions regarding mining, evaluation, troubleshooting, and contributions.
• llm_scoring.md: Detailed explanation of the LLM scoring criteria and methodology.

Worker API (worker_api/ directory): This component defines an API for model validation tasks. It is primarily utilized by validators and the subnet operators to manage the evaluation pipeline. Dockerfiles (evaluator.Dockerfile for the scoring worker and vapi.Dockerfile for the worker API) are provided to facilitate the deployment of these services. The scoring worker is responsible for executing the computationally intensive model evaluation tasks.

Key Dependencies and Integrations:

• Hugging Face: The subnet heavily relies on Hugging Face for model submissions. Miners are required to submit their models to a designated Hugging Face model pool, from which validators can access them for evaluation.
• Bittensor Network: The entire subnet operates within the Bittensor ecosystem, utilizing its underlying blockchain for registration, incentive distribution (TAO emissions), and decentralized consensus.
• Python Ecosystem: Built on Python, it likely uses common data science and machine learning libraries such as PyTorch, Transformers (from Hugging Face), and others for model handling and evaluation.
• PM2 (Recommended for Validators): The documentation recommends using PM2, a process manager for Node.js applications (but also widely used for Python scripts), for running auto-updating validator nodes. This helps ensure validators stay online and keep their software up-to-date.
• Local Subtensor Node: Validators have the option to run with a local Subtensor node for more direct interaction with the Bittensor network, potentially offering lower latency and greater control.

Model Worker Orchestration:

An important architectural detail noted in the GitHub repository is that, at the time of the documentation, general validators call a model worker orchestration service hosted by the Dippy subnet owners (Impel Intelligence). This service likely manages the distribution of evaluation tasks to a pool of scoring workers. While the code for local worker orchestration might exist, it was disabled for general validators, centralizing the execution of the core evaluation computation to some extent, likely for consistency, resource management, and to prevent abuse during the initial phases of the subnet.

Evolution from Other Subnets:
The Dippy subnet’s codebase initially drew inspiration from concepts found in Nous Research’s and MyShell’s subnets. However, it has reportedly diverged significantly to cater to its specific focus on roleplay LLM evaluation and development.