# hanabi-1

**Published by:** [0xReisearch](https://paragraph.com/@0xreisearch/)
**Published on:** 2025-03-06
**URL:** https://paragraph.com/@0xreisearch/hanabi-1

## Content

By REI Network | March 6, 2025 Hanabi-1 is the the first in our "Catalog" series, which specializes in Financial Prediction. Catalog will be a series of transformer models designed to serve a variety of different specialized purposes. The majority of these models will be open-sourced, making them freely available to the developer community. For those requiring programmatic integration, our API will provide a seamless way to incorporate these capabilities into existing workflows as well as plugging them to CORE for improved efficiency.Introducing Hanabi-1: A Transformer model for Financial Market AnalysisWhile the industry gravitates toward increasingly large models, our research has revealed that financial market prediction benefits from a more specialized, compact architecture. Hanabi-1 demonstrates how targeted design can outperform brute-force approaches in specific domains like financial time series analysis With 16.4 million parameter model consists of:8 transformer layers with multi-head attention mechanisms384-dimensional hidden states throughout the networkMultiple specialized predictive pathways for direction, volatility, price change, and spreadBatch normalization rather than layer normalization for better training dynamicsFocal loss implementation to address inherent class imbalanceThe compact size enables faster inference times and allows us to deploy models at the edge for real-time decision making—critical for high-frequency market environments.Mathematical Foundations: Functions and FormulasPositional EncodingTo help the transformer understand sequence ordering, we implement sinusoidal positional encoding: $$PE_{(pos,2i)} = \sin\left(pos \cdot \frac{1}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos,2i+1)} = \cos\left(pos \cdot \frac{1}{10000^{2i/d_{model}}}\right)$$ Where $pos$ is the position within the sequence and $i$ is the dimension index.Focal Loss for Direction PredictionTo address the severe class imbalance in financial market direction prediction, we implemented Focal Loss: $$FL(p_t) = -(1 - p_t)^\gamma \log(p_t)$$ Where $p_t$ is the model's estimated probability for the correct class and $\gamma$ is the focusing parameter (set to 2.0 in Hanabi-1). This loss function down-weights the contribution of easy examples, allowing the model to focus on harder cases.Confidence CalibrationA key innovation in Hanabi-1 is our confidence-aware prediction system: $$Confidence = 2 \cdot |p - threshold|$$ Where $p$ is the predicted probability and $threshold$ is our calibrated decision boundary (0.5). This allows users to filter predictions based on confidence levels, dramatically improving accuracy in high-confidence scenario.Confidence vs AccuracyAs shown above, predictions with "High" confidence achieve nearly 100% accuracy, while "Very Low" confidence predictions are barely above random chance.Training Dynamics and Balanced ValidationTraining financial models presents unique challenges, particularly the tendency to collapse toward predicting a single class. Our novel validation scoring function addresses this: $$ValScore = F1 + 0.5 \cdot Accuracy + 0.5 \cdot PR_{balance} - 0.1 \cdot Loss - Balance_{penalty}$$ Where $PR_{balance}$ is the precision-recall balance metric: $$PR_{balance} = \frac{\min(Precision, Recall)}{\max(Precision, Recall)}$$ And $Balance_{penalty}$ applies severe penalties for extreme prediction distributions:if precision == 0 or recall == 0: # Heavy penalty for predicting all one class balance_penalty = 0.5 elif precision < 0.2 or recall < 0.2: # Moderate penalty for extreme imbalance balance_penalty = 0.3 This scoring function drives the model toward balanced predictions that maintain high accuracy:Training DynamicsThe plot above reveals how training progresses through multiple phases, with early fluctuations stabilizing into consistent improvements after epoch 80.Model Architecture DetailsHanabi-1 employs a specialized architecture with several innovative components:Feature differentiation through multiple temporal aggregations:Last hidden state capture (most recent information)Average pooling across the sequence (baseline signal)Attention-weighted aggregation (focused signal)Direction pathway with BatchNorm for stable training:Three fully-connected layers with BatchNorm1dLeakyReLU activation (slope 0.1) to prevent dead neuronsXavier initialization with small random bias termsSpecialized regression pathways:Separate networks for volatility, price change, and spread predictionReduced complexity compared to the direction pathwayIndependent optimization focuses training capacity where neededThe model's multi-task design forces the transformer encoder to learn robust representations that generalize across prediction tasks.Prediction Temporal DistributionDirection ProbabilitiesThe distribution of predictions over time shows Hanabi-1's ability to generate balanced directional signals across varying market conditions. Green dots represent correct predictions, and red dots are incorrect predictions.Performance and Future DirectionsCurrent performance metrics:Direction accuracy: 73.9%F1 score: 0.67Balanced predictions: 54.2% positive / 45.8% negativeHanabi-1 currently operates on two primary configurations:4-hour window model (w4_h1)12-hour window model (w12_h1)Both predict market movements for the next hour, with the 12-hour window model showing superior performance in more volatile conditions. Future developments include:Extending prediction horizons to 4, 12 and 24 hoursImplementing adaptive thresholds based on market volatilityAdding meta-learning approaches for hyperparameter optimizationIntegrating on-chain signals for cross-domain pattern recognitionConclusionHanabi-1 demonstrates that specialized, compact transformers can achieve remarkable results in financial prediction tasks. By focusing on addressing the unique challenges of financial data—class imbalance, temporal dynamics, and confidence calibration—we've created a model that delivers reliable signals even in challenging market conditions. While the model can still be refined, we found that it’s a robust and important first step towards the definition and creation of even more capable financial models. Follow the github repo for the current implementation and future upgrades: https://github.com/0xReisearch/hanabi-1

## Publication Information

- [0xReisearch](https://paragraph.com/@0xreisearch/): Publication homepage
- [All Posts](https://paragraph.com/@0xreisearch/): More posts from this publication
- [RSS Feed](https://api.paragraph.com/blogs/rss/@0xreisearch): Subscribe to updates
- [Twitter](https://twitter.com/0xreisearch): Follow on Twitter