# hanabi-1

By [0xReisearch](https://paragraph.com/@0xreisearch) · 2025-03-06

---

_By REI Network | March 6, 2025_

_Hanabi-1 is the the first in our "Catalog" series, which specializes in Financial Prediction. Catalog will be a series of transformer models designed to serve a variety of different specialized purposes. The majority of these models will be open-sourced, making them freely available to the developer community. For those requiring programmatic integration, our API will provide a seamless way to incorporate these capabilities into existing workflows as well as plugging them to CORE for improved efficiency._

Introducing Hanabi-1: A Transformer model for Financial Market Analysis
=======================================================================

While the industry gravitates toward increasingly large models, our research has revealed that financial market prediction benefits from a more specialized, compact architecture. Hanabi-1 demonstrates how targeted design can outperform brute-force approaches in specific domains like financial time series analysis

With 16.4 million parameter model consists of:

*   8 transformer layers with multi-head attention mechanisms
    
*   384-dimensional hidden states throughout the network
    
*   Multiple specialized predictive pathways for direction, volatility, price change, and spread
    
*   Batch normalization rather than layer normalization for better training dynamics
    
*   Focal loss implementation to address inherent class imbalance
    

The compact size enables faster inference times and allows us to deploy models at the edge for real-time decision making—critical for high-frequency market environments.

Mathematical Foundations: Functions and Formulas
------------------------------------------------

### Positional Encoding

To help the transformer understand sequence ordering, we implement sinusoidal positional encoding:

$$PE\_{(pos,2i)} = \\sin\\left(pos \\cdot \\frac{1}{10000^{2i/d\_{model}}}\\right)$$

$$PE\_{(pos,2i+1)} = \\cos\\left(pos \\cdot \\frac{1}{10000^{2i/d\_{model}}}\\right)$$

Where $pos$ is the position within the sequence and $i$ is the dimension index.

### Focal Loss for Direction Prediction

To address the severe class imbalance in financial market direction prediction, we implemented Focal Loss:

$$FL(p\_t) = -(1 - p\_t)^\\gamma \\log(p\_t)$$

Where $p\_t$ is the model's estimated probability for the correct class and $\\gamma$ is the focusing parameter (set to 2.0 in Hanabi-1). This loss function down-weights the contribution of easy examples, allowing the model to focus on harder cases.

### Confidence Calibration

A key innovation in Hanabi-1 is our confidence-aware prediction system:

$$Confidence = 2 \\cdot |p - threshold|$$

Where $p$ is the predicted probability and $threshold$ is our calibrated decision boundary (0.5). This allows users to filter predictions based on confidence levels, dramatically improving accuracy in high-confidence scenario.

![Confidence vs Accuracy](https://storage.googleapis.com/papyrus_images/e0accc5a0add776a21fd994f4fe793649ccc1da3d167c5e3249edee30cdece76.png)

Confidence vs Accuracy

As shown above, predictions with "High" confidence achieve nearly 100% accuracy, while "Very Low" confidence predictions are barely above random chance.

Training Dynamics and Balanced Validation
-----------------------------------------

Training financial models presents unique challenges, particularly the tendency to collapse toward predicting a single class. Our novel validation scoring function addresses this:

$$ValScore = F1 + 0.5 \\cdot Accuracy + 0.5 \\cdot PR\_{balance} - 0.1 \\cdot Loss - Balance\_{penalty}$$

Where $PR\_{balance}$ is the precision-recall balance metric:

$$PR\_{balance} = \\frac{\\min(Precision, Recall)}{\\max(Precision, Recall)}$$

And $Balance\_{penalty}$ applies severe penalties for extreme prediction distributions:

    if precision == 0 or recall == 0:
        # Heavy penalty for predicting all one class
        balance_penalty = 0.5
    elif precision < 0.2 or recall < 0.2:
        # Moderate penalty for extreme imbalance
        balance_penalty = 0.3
    

This scoring function drives the model toward balanced predictions that maintain high accuracy:

![Training Dynamics](https://storage.googleapis.com/papyrus_images/6f94470913e4eb23d1d589280ef8443bb57f8e2684bebea1c02e45d1b8ad0833.png)

Training Dynamics

The plot above reveals how training progresses through multiple phases, with early fluctuations stabilizing into consistent improvements after epoch 80.

Model Architecture Details
--------------------------

Hanabi-1 employs a specialized architecture with several innovative components:

*   **Feature differentiation through multiple temporal aggregations:**
    
    *   Last hidden state capture (most recent information)
        
    *   Average pooling across the sequence (baseline signal)
        
    *   Attention-weighted aggregation (focused signal)
        
*   **Direction pathway with BatchNorm for stable training:**
    
    *   Three fully-connected layers with BatchNorm1d
        
    *   LeakyReLU activation (slope 0.1) to prevent dead neurons
        
    *   Xavier initialization with small random bias terms
        
*   **Specialized regression pathways:**
    
    *   Separate networks for volatility, price change, and spread prediction
        
    *   Reduced complexity compared to the direction pathway
        
    *   Independent optimization focuses training capacity where needed
        

The model's multi-task design forces the transformer encoder to learn robust representations that generalize across prediction tasks.

Prediction Temporal Distribution
--------------------------------

![Direction Probabilities](https://storage.googleapis.com/papyrus_images/575ff3d93bee2fed2241805d35aad45a7521b555928c82580a101d744313fc7d.png)

Direction Probabilities

The distribution of predictions over time shows Hanabi-1's ability to generate balanced directional signals across varying market conditions. Green dots represent correct predictions, and red dots are incorrect predictions.

Performance and Future Directions
---------------------------------

Current performance metrics:

*   **Direction accuracy:** 73.9%
    
*   **F1 score:** 0.67
    
*   **Balanced predictions:** 54.2% positive / 45.8% negative
    

Hanabi-1 currently operates on two primary configurations:

*   4-hour window model (w4\_h1)
    
*   12-hour window model (w12\_h1)
    

Both predict market movements for the next hour, with the 12-hour window model showing superior performance in more volatile conditions.

Future developments include:

*   Extending prediction horizons to 4, 12 and 24 hours
    
*   Implementing adaptive thresholds based on market volatility
    
*   Adding meta-learning approaches for hyperparameter optimization
    
*   Integrating on-chain signals for cross-domain pattern recognition
    

Conclusion
----------

Hanabi-1 demonstrates that specialized, compact transformers can achieve remarkable results in financial prediction tasks. By focusing on addressing the unique challenges of financial data—class imbalance, temporal dynamics, and confidence calibration—we've created a model that delivers reliable signals even in challenging market conditions.

While the model can still be refined, we found that it’s a robust and important first step towards the definition and creation of even more capable financial models.

Follow the github repo for the current implementation and future upgrades:

[https://github.com/0xReisearch/hanabi-1](https://github.com/0xReisearch/hanabi-1)

---

*Originally published on [0xReisearch](https://paragraph.com/@0xreisearch/hanabi-1)*