quant datapoller library
We last announced our quantpylib library with its official host documentation:
We also discussed about the future directives, one of which includes a new and enhanced data polling library for financial data and quantitative features.
We discussed prior as to how the existing data_service class has some concerns with regards to engineering principles such as modularity and separation of concerns.
Speaking of which - we are going over programming concepts in our market notes now, and intend to develop those specific chapters in extensive detail over the coming months - our last notes were here:
If you wish to evolve from a novice/intermediate programmer to an advanced user of Python; I strongly encourage you to peruse those material.
About the datapoller - here are some engineering considerations and concerns we are going to address in the upcoming module:
i) First, from a user (the quant researcher) perspective, data is often categorized (mentally) into asset classes - equities, crypto, fx and so on.
ii) End users tend to prefer one data source over the other pertaining to the asset class - for instance, one might prefer to use Oanda brokerage for FX data, open source yfinance for equities and binance for crypto. They tend to have a preference source endpoint, or priority sort for data sources by asset class.
These considerations are on the user side - but there are many considerations on the business logic side - for instance, most Data APIs provide different endpoints for different asset classes. Simultaneously, across these different endpoints, it is common for APIs to do rate/resource pool limiting for GET/POST requests.
Exceeding the maximum request throughput, can result in IP blacklisting and downgrading of performance for future requests. This is often handled via a TooManyRequests code, and the time to wait is given in the packet headers - but this is data source specific. Additionally, there could already be other requests en route, which would fail with possible adverse consequences aforementioned.
We want to handle these operations gracefully, and recognize these business logic features into our new and improved datapoller. So the external-API facing logic would have a nomenclature that supports resource sharing at the datasource level, whereas the user-facing APIs would be by asset classes.
I am going to be developing it in my branch first, so you can check it out here:
https://github.com/hangukquant/quantpylib/commit/0a23266f5d19e7a78c424bf778af1f5020b3521e
For those without Github repo access:
Cheers~