How to design a real-time ML system to predict crypto prices
May 18, 2024
Today you will learn how to design a modular and scalable real-time ML system that predicts short-term crypto prices.
Let’s get to it!
The problem
We want to build a real-time API that serves in real-time short-term predictions on crypto prices.
For example
To predict the price of Ethereum (ETH) in the next 10 seconds.
This API can be consumed by a human trader, or (even better) an algorithmic trading bot to improve the return of a trading strategy.
For example
A very naive strategy to map predictions to actions is to
Buy ETH when the predicted price return by our API goes up
Sell ETH when the predicted price return by our API goes down
A more sophisticated algorithmic trading strategy would consume the predictions generated by our API, and take them into account, together with other factors, to come up wit the final action.
Today we will NOT focus on constructing trading strategies, but on designing a modular and scalable machine learning system that delivers these predictions in real time.
The ML system behind our API
Our API is just one component of an ML system that continuously
-
Ingests raw market data in real-time and
-
Generates new predictions using a Machine Learning model.
Under the hood, this system can be decomposed into 3 types of pipelines:
→ Feature pipelines → ingest raw market data and generate predictive features.
→ The Training pipeline → ingests training data and generates an ML model.
→ The Inference pipeline → loads the ML model and the most recent features from the Feature Store, and generates fresh predictions.
These are the steps to build the system ↓
Step 1. Build the feature pipeline
A Feature pipeline that
→ ingests real-time price data from an third party API.
→ transforms price data into 10-second Open-High-Low-Close prices, and
→ pushes it to the Feature Store.
Step 2. Build the training pipeline
A Training pipeline that
→ ingests training data from the feature store,
→ trains a predictive ML model, and
→ pushes the model to the model registry
Step 3. Build the inference pipeline
An Inference pipeline that
→ load the model from the registry and recent features, and
→ serves fresh predictions through our API.
Step 4. Iteratively improve the system
Once this first version is working, you start improving each component of your system, for example by
-
Building more feature pipelines, to provide more signals to your ML model.
-
Improving the underlying ML model
-
Optimizing the inference pipeline to produce faster predictions.
Wanna build this real-time ML system with me?
A just launched a new live, interactive course in which you will learn step-by-step how to design, build and deploy this real-time ML system.
How is this course gonna work?
We will meet on June 3rd and spend 3 intense, super-rewarding weeks building a Real-Time ML system to predict crypto prices.
After completing this course (+ A LOT of hard work on your end) you will know how to
-
Build modular and scalable real-time ML systems
-
For the business problem you care about
-
Using any real-time data source
-
Following MLOps best practices, for fast iteration and short-time-to-market.
And the best part?
You pay now and have access to all future sessions of the program.
Looking forward to having you on board
Have a great weekend,
Talk to you next week
Pau
