forex_data_aggregator package
Subpackages
- forex_data.config package
- forex_data.data_management package
- Submodules
- forex_data.data_management.common module
DEFAULT_PATHSDATA_TYPECOLUMN_NAMECOLUMN_NAME.TIMESTAMPCOLUMN_NAME.OPENCOLUMN_NAME.HIGHCOLUMN_NAME.LOWCOLUMN_NAME.CLOSECOLUMN_NAME.ASKCOLUMN_NAME.BIDCOLUMN_NAME.VOLUMECOLUMN_NAME.ASK_VOLUMECOLUMN_NAME.BID_VOLUMECOLUMN_NAME.VWMPCOLUMN_NAME.VWMP_AVGCOLUMN_NAME.P_VALUECOLUMN_NAME.TRANSACTIONSCOLUMN_NAME.OTCCOLUMN_NAME.VWAP
DATA_FILE_COLUMN_INDEXASSET_TYPEDTYPE_DICTPYARROW_DTYPE_DICTPOLARS_DTYPE_DICTDATA_COLUMN_NAMESFILENAME_TEMPLATEDATA_KEYYearMonthListNormalizedDictSQL_COMPARISON_OPERATORSSQL_CONDITION_AGGREGATION_MODESSUPPORTED_HISTORICAL_DATA_PROVIDERScollect_lazyframe()validator_file_path()validator_dir_path()validate_timedelta_str()get_attrs_names()check_timeframe_str()any_date_to_datetime64()normalize_to_naive_utc()empty_dataframe()is_empty_dataframe()shape_dataframe()get_dataframe_column()get_dataframe_row()get_dataframe_element()astype()read_csv()sort_dataframe()concat_data()list_remove_duplicates()reframe_data()write_csv()write_parquet()read_parquet()to_pandas_dataframe()get_pair_symbols()to_source_symbol()get_date_interval()random_date_between()get_histdata_tickers()TickerNotFoundErrorTickerDataNotFoundErrorTickerDataBadTypeExceptionTickerDataInvalidExceptionbusiness_days_data()update_ticker_years_dict()get_class_attr_keys()get_class_attr_values()get_class_attr_dict()get_forex_holidays()adjust_end_date_to_business_days()adjust_start_date_to_business_days()estimate_start_date_to_business_days()estimate_end_date_to_business_days()TimestampTimestamp.astimezone()Timestamp.ceil()Timestamp.combine()Timestamp.ctime()Timestamp.date()Timestamp.daysinmonthTimestamp.dst()Timestamp.floor()Timestamp.fromordinal()Timestamp.fromtimestamp()Timestamp.isocalendar()Timestamp.isoweekday()Timestamp.now()Timestamp.replace()Timestamp.round()Timestamp.strftime()Timestamp.strptime()Timestamp.time()Timestamp.timetuple()Timestamp.timetz()Timestamp.to_julian_date()Timestamp.today()Timestamp.toordinal()Timestamp.tzTimestamp.tz_convert()Timestamp.tz_localize()Timestamp.tzinfoTimestamp.tzname()Timestamp.utcfromtimestamp()Timestamp.utcnow()Timestamp.utcoffset()Timestamp.utctimetuple()Timestamp.weekday()Timestamp.weekofyearTimestamp.clsTimestamp.ts_inputTimestamp.return
PolarsDatetimePolarsFloat32PolarsDataFramePolarsLazyFrame
- forex_data.data_management.database module
DatabaseConnectorDatabaseConnector.data_pathDatabaseConnector.data_typeDatabaseConnector.engineDatabaseConnector.polars_gpu_engineDatabaseConnector.__init__()DatabaseConnector.connect()DatabaseConnector.check_connection()DatabaseConnector.write_data()DatabaseConnector.read_data()DatabaseConnector.read_data_year()DatabaseConnector.read_data_window()DatabaseConnector.read_last_timestamp()DatabaseConnector.get_tickers_list()DatabaseConnector.get_ticker_keys()DatabaseConnector.get_ticker_timeframes_list()DatabaseConnector.get_ticker_years_list()DatabaseConnector.create_tickers_years_dict()DatabaseConnector.save_tickers_years_info()DatabaseConnector.add_tickers_years_info_to_file()DatabaseConnector.clear_tickers_years_info()DatabaseConnector.clear_database()DatabaseConnector.load_tickers_years_info()
LocalDBConnectorLocalDBYearConnector
- forex_data.data_management.historicaldata module
HistoricalManagerDBHistoricalManagerDB.configHistoricalManagerDB.data_typeHistoricalManagerDB.engineHistoricalManagerDB.data_pathHistoricalManagerDB.db_files_year_partitioningHistoricalManagerDB.volume_dataHistoricalManagerDB.ssl_verifyHistoricalManagerDB.polars_gpu_engineHistoricalManagerDB.connector_idHistoricalManagerDB.max_discrepancy_with_nowHistoricalManagerDB.offlineHistoricalManagerDB.__init__()HistoricalManagerDB.get_source_connectors()HistoricalManagerDB.clear_database()HistoricalManagerDB.add_timeframe()HistoricalManagerDB.get_data()HistoricalManagerDB.get_data_window()HistoricalManagerDB.close()HistoricalManagerDB.patch_raw_tick_holes()
- forex_data.data_management.remoteconnector module
RemoteConnectorRemoteConnector.data_pathRemoteConnector.data_typeRemoteConnector.engineRemoteConnector.polars_gpu_engineRemoteConnector.volume_dataRemoteConnector.__init__()RemoteConnector.connect()RemoteConnector.check_connection()RemoteConnector.clear_temporary_folder()RemoteConnector.get_available_tickers()RemoteConnector.get_data()RemoteConnector.get_recent_data()RemoteConnector.download_month_raw()
HistDataConnectorDukascopyConnectorTwelveDataConnectorTiingoConnectorTiingoConnector.api_keyTiingoConnector.planTiingoConnector.tierTiingoConnector.chunk_sizeTiingoConnector.max_requests_per_hourTiingoConnector.__init__()TiingoConnector.get_top_of_book()TiingoConnector.get_realtime_price()TiingoConnector.get_data()TiingoConnector.get_recent_data()TiingoConnector.data_pathTiingoConnector.data_typeTiingoConnector.engineTiingoConnector.polars_gpu_engineTiingoConnector.volume_data
CTraderConnectorCTraderConnector.client_idCTraderConnector.client_secretCTraderConnector.access_tokenCTraderConnector.refresh_tokenCTraderConnector.broker_account_idCTraderConnector.broker_account_is_demoCTraderConnector.chunk_sizeCTraderConnector.__init__()CTraderConnector.connect()CTraderConnector.close()CTraderConnector.check_connection()CTraderConnector.get_available_tickers()CTraderConnector.get_data()CTraderConnector.get_recent_data()CTraderConnector.download_day_raw()CTraderConnector.download_range_raw()CTraderConnector.download_month_raw()
cTraderDataConnector
- forex_data.data_management.charts package
- Module contents
HistoricalManagerDBHistoricalManagerDB.configHistoricalManagerDB.data_typeHistoricalManagerDB.engineHistoricalManagerDB.data_pathHistoricalManagerDB.db_files_year_partitioningHistoricalManagerDB.volume_dataHistoricalManagerDB.ssl_verifyHistoricalManagerDB.polars_gpu_engineHistoricalManagerDB.connector_idHistoricalManagerDB.max_discrepancy_with_nowHistoricalManagerDB.offlineHistoricalManagerDB.__init__()HistoricalManagerDB.get_source_connectors()HistoricalManagerDB.clear_database()HistoricalManagerDB.add_timeframe()HistoricalManagerDB.get_data()HistoricalManagerDB.get_data_window()HistoricalManagerDB.close()HistoricalManagerDB.patch_raw_tick_holes()
COLUMN_NAMECOLUMN_NAME.TIMESTAMPCOLUMN_NAME.OPENCOLUMN_NAME.HIGHCOLUMN_NAME.LOWCOLUMN_NAME.CLOSECOLUMN_NAME.ASKCOLUMN_NAME.BIDCOLUMN_NAME.VOLUMECOLUMN_NAME.ASK_VOLUMECOLUMN_NAME.BID_VOLUMECOLUMN_NAME.VWMPCOLUMN_NAME.VWMP_AVGCOLUMN_NAME.P_VALUECOLUMN_NAME.TRANSACTIONSCOLUMN_NAME.OTCCOLUMN_NAME.VWAP
YearMonthListNormalizedDictDATA_FILE_COLUMN_INDEXDEFAULT_PATHSSQL_COMPARISON_OPERATORSSQL_CONDITION_AGGREGATION_MODESempty_dataframe()is_empty_dataframe()shape_dataframe()get_dataframe_column()get_dataframe_row()get_dataframe_element()get_attrs_names()any_date_to_datetime64()normalize_to_naive_utc()get_db_key_elements()check_timeframe_str()DatabaseConnectorDatabaseConnector.data_pathDatabaseConnector.data_typeDatabaseConnector.engineDatabaseConnector.polars_gpu_engineDatabaseConnector.__init__()DatabaseConnector.connect()DatabaseConnector.check_connection()DatabaseConnector.write_data()DatabaseConnector.read_data()DatabaseConnector.read_data_year()DatabaseConnector.read_data_window()DatabaseConnector.read_last_timestamp()DatabaseConnector.get_tickers_list()DatabaseConnector.get_ticker_keys()DatabaseConnector.get_ticker_timeframes_list()DatabaseConnector.get_ticker_years_list()DatabaseConnector.create_tickers_years_dict()DatabaseConnector.save_tickers_years_info()DatabaseConnector.add_tickers_years_info_to_file()DatabaseConnector.clear_tickers_years_info()DatabaseConnector.clear_database()DatabaseConnector.load_tickers_years_info()
LocalDBConnectorconcat_data()reframe_data()validator_dir_path()TickerNotFoundErrorTickerDataNotFoundErrorTickerDataBadTypeExceptionTickerDataInvalidExceptionget_histdata_tickers()POLARS_DTYPE_DICTbusiness_days_data()get_class_attr_keys()get_class_attr_values()get_class_attr_dict()random_date_between()RemoteConnectorRemoteConnector.data_pathRemoteConnector.data_typeRemoteConnector.engineRemoteConnector.polars_gpu_engineRemoteConnector.volume_dataRemoteConnector.__init__()RemoteConnector.connect()RemoteConnector.check_connection()RemoteConnector.clear_temporary_folder()RemoteConnector.get_available_tickers()RemoteConnector.get_data()RemoteConnector.get_recent_data()RemoteConnector.download_month_raw()
TwelveDataConnectorHistDataConnectorDukascopyConnectorTiingoConnectorTiingoConnector.api_keyTiingoConnector.planTiingoConnector.tierTiingoConnector.chunk_sizeTiingoConnector.max_requests_per_hourTiingoConnector.__init__()TiingoConnector.get_top_of_book()TiingoConnector.get_realtime_price()TiingoConnector.get_data()TiingoConnector.get_recent_data()TiingoConnector.data_pathTiingoConnector.data_typeTiingoConnector.engineTiingoConnector.polars_gpu_engineTiingoConnector.volume_data
CTraderConnectorCTraderConnector.client_idCTraderConnector.client_secretCTraderConnector.access_tokenCTraderConnector.refresh_tokenCTraderConnector.broker_account_idCTraderConnector.broker_account_is_demoCTraderConnector.chunk_sizeCTraderConnector.__init__()CTraderConnector.connect()CTraderConnector.close()CTraderConnector.check_connection()CTraderConnector.get_available_tickers()CTraderConnector.get_data()CTraderConnector.get_recent_data()CTraderConnector.download_day_raw()CTraderConnector.download_range_raw()CTraderConnector.download_month_raw()
cTraderDataConnectorDataChartsDataCharts.__init__()DataCharts.plot_historical()DataCharts.start_realtime_plot()DataCharts.close()DataCharts.configDataCharts.data_typeDataCharts.engineDataCharts.data_pathDataCharts.db_files_year_partitioningDataCharts.volume_dataDataCharts.ssl_verifyDataCharts.polars_gpu_engineDataCharts.connector_idDataCharts.max_discrepancy_with_nowDataCharts.offline
create_historical_figure()RealtimeDashAppSupportedChartType
Module contents
Created on Sun Jul 17 17:07:39 2022
@author: fiora
- forex_data.read_config_folder(folder_path=None, file_pattern='appconfig.yaml')[source]
- Return type:
- class forex_data.HistoricalManagerDB(**kwargs)[source]
Bases:
object- config: str
- data_type: str
- engine: str
- db_files_year_partitioning: bool
- volume_data: bool
- ssl_verify: bool
- polars_gpu_engine: bool
- connector_id: str
- max_discrepancy_with_now: str
- offline: bool
- __init__(**kwargs)[source]
- get_source_connectors()[source]
- Return type:
- add_timeframe(timeframe)[source]
Add and cache a new timeframe to the database.
Creates aggregated data for the specified timeframe from tick data and caches it in the database for faster future access. The timeframe is added to the internal list of available timeframes.
- Parameters:
timeframe (
Union[str,List[str]]) – Timeframe(s) to add. Can be a single string or list of strings. Supported values: ‘1m’, ‘5m’, ‘15m’, ‘30m’, ‘1h’, ‘4h’, ‘1D’, ‘1W’, ‘1M’- Return type:
- Returns:
None
- Raises:
TypeError – If timeframe is not a string or list of strings
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> manager.add_timeframe('1W') # Add weekly timeframe >>> manager.add_timeframe(['4h', '1D']) # Add multiple timeframes
Note
Only new timeframes (not already in the list) will be processed
Aggregation can take time for large datasets
Once added, the timeframe is permanently cached in the database
- get_data(ticker, timeframe, start, end, comparison_column_name=None, check_level=None, comparison_operator=None, aggregation_mode=None)[source]
Retrieve OHLC historical data for the specified ticker and timeframe.
Fetches historical forex data from the database, automatically downloading and aggregating data if not already available. Supports multiple timeframes and date ranges.
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘NZDUSD’). Case-insensitive.timeframe (
str) – Candle timeframe for data aggregation. Supported frames: 1s (1 second) 1m (1 minute) 1h (1 hour) 1d (1 calendar day) 1w (1 calendar week) 1mo (1 calendar month) 1q (1 calendar quarter) 1y (1 calendar year)integer (and any multiple of these values by a positive) – ‘2m’, ‘3m’, ‘2h’, ‘3h’, etc.
e.g. – ‘2m’, ‘3m’, ‘2h’, ‘3h’, etc.
start (
str | datetime) – Start date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime objectend (
str | datetime) – End date for data retrieval. Same format as start. Must be after start date.comparison_column_name (
Union[List[str],str,None]) – Column names to retrieve. Default is None.check_level (
Union[List[int|float],int,float,None]) – Check level for conditions. Default is None.comparison_operator (
Union[List[Literal['>','<','>=','<=','==','!=']],Literal['>','<','>=','<=','==','!='],None]) – Condition for data retrieval. Default is None.aggregation_mode (
Optional[Literal['AND','OR']]) – Aggregation mode for data retrieval. Default is None.
- Returns:
- DataFrame containing OHLC data with columns:
timestamp: datetime column with candle timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)
- Return type:
Union[DataFrame,LazyFrame]- Raises:
TickerNotFoundError – If the ticker is not available in the historical database
ValueError – If timeframe is invalid or end date is before start date
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> data = manager.get_data( ... ticker='EURUSD', ... timeframe='1h', ... start='2020-01-01', ... end='2020-01-31' ... ) >>> print(f"Retrieved {len(data)} hourly candles") Retrieved 744 hourly candles
Note
Data is automatically downloaded from histdata.com if not cached locally
First call for a new timeframe may take longer as it builds the aggregation
Downloaded data is cached for faster subsequent access
Ticker names are case-insensitive and automatically normalized
- get_data_window(ticker, date, timeframe, periods, direction, comparison_column_name=None, check_level=None, comparison_operator=None, comparison_aggregation_mode=None)[source]
Retrieve OHLC historical window data for the specified ticker. The unit resoluton of the window is set equal to the timeframe. Unit resolution is the timespan between two candles (rows) in normal conditions: during weekends the rule does not apply. The window total number of candles (rows) is specified by timeframe * periods.
Fetches historical forex data from the database, automatically downloading and aggregating data if not already available. Supports multiple timeframes
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘NZDUSD’). Case-insensitive.date (
date) – date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime objecttimeframe (
str) – Candle timeframe for data aggregation. Supported frames: 1s, 1m, 1h, 1d, 1w, 1mo, 1q, 1y.periods (
int) – Number of timeframe units to look back or forward.direction (
Literal['backward','forward']) – Direction to look back (‘backward’ or ‘forward’).comparison_column_name (
Union[List[str],str,None]) – List of column names to compare.check_level (
Union[List[int|float],int,float,None]) – List of values to compare against.comparison_operator (
Union[List[Literal['>','<','>=','<=','==','!=']],Literal['>','<','>=','<=','==','!='],None]) – Comparison operators.comparison_aggregation_mode (
Optional[Literal['AND','OR']]) – Aggregation mode.
- Returns:
- DataFrame with the historical
data.
- Return type:
Union[DataFrame,LazyFrame]- Raises:
TickerNotFoundError – If the ticker is not found.
TickerDataNotFoundError – If the ticker data is not found.
TickerDataBadTypeException – If the ticker data is not of the expected type.
TickerDataInvalidException – If the ticker data is invalid.
Examples
>>> get_data_window( ... ticker='EURUSD', ... date='2022-01-01', ... timeframe='1m', ... window=10, ... direction='backward' ... )
- close()[source]
- patch_raw_tick_holes(ticker, year, start_date=None, end_date=None, min_volume=0.0)[source]
Scans origin raw tick parquet (forex_{ticker}_tick_{year}.parquet) for data holes: rows with nulls, NaNs, or non-positive values (<= 0.0) across ANY feature column during business hours. Optionally accepts start_date/end_date to force patch a specific range, or min_volume threshold. Fetches clean replacement chunks from cTrader in 1-day chunks, splices them into the raw tick file, and re-reframes all downstream timeframes.
- Return type:
- class forex_data.COLUMN_NAME[source]
Bases:
object- TIMESTAMP = 'timestamp'
- OPEN = 'open'
- HIGH = 'high'
- LOW = 'low'
- CLOSE = 'close'
- ASK = 'ask'
- BID = 'bid'
- VOLUME = 'volume'
- ASK_VOLUME = 'ask_volume'
- BID_VOLUME = 'bid_volume'
- VWMP = 'vwmp'
- VWMP_AVG = 'vwmp_avg'
- P_VALUE = 'p'
- TRANSACTIONS = 'transactions'
- OTC = 'otc'
- VWAP = 'vwap'
- class forex_data.YearMonthList(years_dict=None)[source]
Bases:
list- __init__(years_dict=None)[source]
- add_month(year, month)[source]
- remove(year)[source]
Remove first occurrence of value.
Raises ValueError if the value is not present.
- append(year)[source]
Append object to the end of the list.
- extend(years)[source]
Extend list by appending elements from the iterable.
- class forex_data.NormalizedDict(*args, **kwargs)[source]
Bases:
dict- __init__(*args, **kwargs)[source]
- get(key, default=None)[source]
Return the value for key if key is in the dictionary, else default.
- setdefault(key, default=None)[source]
Insert key with a value of default if key is not in the dictionary.
Return the value for key if key is in the dictionary, else default.
- update([E, ]**F) None. Update D from mapping/iterable E and F.[source]
If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
- class forex_data.DEFAULT_PATHS[source]
Bases:
object- BASE_PATH = '/home/runner/.database'
- HIST_DATA_FOLDER = 'HistoricalData'
- REALTIME_DATA_FOLDER = 'RealtimeData'
- class forex_data.SQL_COMPARISON_OPERATORS[source]
Bases:
object- GREATER_THAN = '>'
- LESS_THAN = '<'
- GREATER_THAN_OR_EQUAL = '>='
- LESS_THAN_OR_EQUAL = '<='
- EQUAL = '=='
- NOT_EQUAL = '!='
- forex_data.empty_dataframe(engine)[source]
- forex_data.is_empty_dataframe(dataframe)[source]
- forex_data.shape_dataframe(dataframe)[source]
- forex_data.get_dataframe_column(dataframe, column)[source]
- forex_data.get_dataframe_row(dataframe, row)[source]
- forex_data.get_dataframe_element(dataframe, column, row)[source]
- forex_data.get_attrs_names(instance_object, **kwargs)[source]
- forex_data.any_date_to_datetime64(any_date, date_format='ISO8601', unit=None, to_pydatetime=False)[source]
- forex_data.normalize_to_naive_utc(date_val, to_pydatetime=False)[source]
Normalize any date/timestamp representation (str, int/float unix timestamp, datetime, pd.Timestamp, pl.Datetime) into a timezone-naive UTC timestamp.
If ‘now’, resolves to current UTC time stripped of tzinfo.
If already timezone-aware, converts to UTC and strips tzinfo.
If timezone-naive, retains the naive value assuming UTC.
If to_pydatetime=True, returns Python datetime instead of pd.Timestamp.
- Return type:
- forex_data.get_db_key_elements(key)[source]
- forex_data.check_timeframe_str(tf, engine='pandas')[source]
- class forex_data.DatabaseConnector(**kwargs)[source]
Bases:
object- data_type: str
- engine: str
- polars_gpu_engine: bool
- __init__(**kwargs)[source]
- check_connection()[source]
Check database connection - must be implemented by subclasses.
- Return type:
- write_data(target_table, dataframe, clean=False)[source]
Write data to database - must be implemented by subclasses.
- Return type:
- read_data(market, ticker, timeframe, start, end)[source]
Read data from database - must be implemented by subclasses.
- Return type:
LazyFrame
- read_data_year(market, ticker, timeframe, years)[source]
Read data for specific year(s) - must be implemented by subclasses.
- Return type:
LazyFrame
- read_data_window(market, ticker, timeframe, date, periods, direction, comparison_column_name=None, check_level=None, comparison_operator=None, comparison_aggregation_mode=None)[source]
Read window of data - must be implemented by subclasses.
- Return type:
LazyFrame
- read_last_timestamp(market, ticker, timeframe=None)[source]
Read last timestamp from database - must be implemented by subclasses.
- Return type:
- get_ticker_timeframes_list(ticker)[source]
Get timeframes list for ticker - must be implemented by subclasses.
- create_tickers_years_dict()[source]
Create a dictionary containing ticker years data, structured as: {ticker: {timeframe: YearMonthList}}
If no data files exist yet, returns an empty dictionary.
- Return type:
Dict[str,Dict[str,YearMonthList]]
- save_tickers_years_info(ticker_years_dict)[source]
Save ticker years list to a JSON file.
- Return type:
Parameters
- ticker_years_dictDict[str, Dict[str, Any]]
Dictionary containing ticker years data
- add_tickers_years_info_to_file(ticker, timeframe, year)[source]
In local info filepath, update the years/months list of the given ticker and timeframe by reading the actual months present in the DB files and updating the file.
- Return type:
- clear_tickers_years_info(filter=None)[source]
Clear the tickers years info file. If filter is specified, it has to be a ticker value and so only the tickers years info related to the filter are cleared. If filter is not specified, the entire file is cleared.
- Return type:
Parameters
- filterOptional[str], optional
Filter to apply to the tickers years info file, by default None Filter has to be a ticker value
- clear_database(filter=None)[source]
Clear database files If filter is provided and is a ticker present in database (files present) delete only files related to that ticker
- Return type:
- class forex_data.LocalDBConnector(**kwargs)[source]
Bases:
DatabaseConnector- __init__(**kwargs)[source]
- write_data(target_table, dataframe, clean=False)[source]
Write data to database - must be implemented by subclasses.
- Return type:
- read_data(market, ticker, timeframe, start, end, comparison_column_name=None, check_level=None, comparison_operator=None, comparison_aggregation_mode=None)[source]
Read data from database - must be implemented by subclasses.
- Return type:
LazyFrame
- read_data_year(market, ticker, timeframe, years)[source]
Read data for specific year(s) using SQL YEAR() filter.
- Return type:
LazyFrame
- read_data_window(market, ticker, timeframe, date, periods, direction, comparison_column_name=None, check_level=None, comparison_operator=None, comparison_aggregation_mode=None)[source]
Read window of data specified by input requirements: the data window has timespan in order to return a dataframe with rows size equal to periods. Query the local db to calculate the start date of the window if direction is forward, or end date if backward.
- Return type:
LazyFrame
- read_last_timestamp(market, ticker, timeframe=None)[source]
Read last timestamp from database. If timeframe is not set (None), retrieve the smallest timeframe available in local database
- class forex_data.TwelveDataConnector(**kwargs)[source]
Bases:
RemoteConnectorClass to read real-time data from the database using TwelveData API.
- api_key: str
- plan: str
- property tier: str
self.tier.
- Type:
Alias for plan to avoid AttributeError
- __init__(**kwargs)[source]
- property chunk_size: int
Max number of data points per request.
- property max_requests_per_minute: int
Max number of requests per minute.
- get_realtime_price(symbol)[source]
Fetches the instantaneous real-time price and outputs as a 1-row LazyFrame.
- Return type:
LazyFrame
- get_data(symbol, timeframe, start_date, end_date)[source]
Fetches historical data for a specific date range. Automatically checks boundaries for maximum output limits per call.
- Return type:
LazyFrame
- get_recent_data(symbol, timeframe, interval_window)[source]
Fetches recent data relative to the current time minus the interval_window. Example: Pass timedelta(days=90) to get the most recent rolling 3 months.
- Return type:
LazyFrame
- class forex_data.TiingoConnector(**kwargs)[source]
Bases:
RemoteConnectorReal-time and intraday market data connector for Forex using Tiingo API. Provides top-of-book bid/ask quotes and size liquidity in base currency units.
- api_key: str
- plan: str
- property tier: str
- property chunk_size: int
Max number of data points per request.
- property max_requests_per_hour: int
Max number of requests per hour.
- __init__(**kwargs)[source]
- get_top_of_book(symbol)[source]
Fetches current top-of-book data (bidPrice, bidSize, askPrice, askSize, midPrice).
- get_realtime_price(symbol)[source]
Fetches current real-time top-of-book price and returns it as a PolarsLazyFrame.
- Return type:
LazyFrame
- get_data(symbol, timeframe, start_date, end_date)[source]
Fetches intraday historical candles and enriches them with top-of-book spread and volume.
- Return type:
LazyFrame
- get_recent_data(symbol, timeframe, interval_window)[source]
Fetches recent data up to current time minus interval_window.
- Return type:
LazyFrame
- data_type: str
- engine: str
- polars_gpu_engine: bool
- volume_data: bool
- class forex_data.CTraderConnector(**kwargs)[source]
Bases:
RemoteConnectorConnector class to retrieve market data from cTrader OpenAPI. Fetches real broker trendbars and ticks, applying a 50/50 volume split formula to derive ask_volume and bid_volume from total tick volume.
- client_id: str
- client_secret: str
- access_token: str
- refresh_token: str
- broker_account_id: str
- broker_account_is_demo: bool
- chunk_size: int
- __init__(**kwargs)[source]
- check_connection()[source]
Check database connection - must be implemented by subclasses.
- Return type:
- get_available_tickers()[source]
Get available tickers - must be implemented by subclasses.
- get_data(symbol, timeframe, start_date, end_date)[source]
Get data - must be implemented by subclasses.
- Return type:
LazyFrame
- get_recent_data(symbol, timeframe, interval_window)[source]
Fetches recent data up to current time minus interval_window.
- Return type:
LazyFrame
- download_day_raw(ticker, day_date, engine='polars_lazy')[source]
Download raw tick data synthesized from 1-minute trendbars for a single day. Uses 4-point OHLC tick synthesis at seconds :00, :15, :30, :45 to preserve the exact Open, High, Low, Close, and regular VWMP with volumes when reframed.
- Return type:
Union[DataFrame,LazyFrame]
- download_range_raw(ticker, start_date, end_date, engine='polars_lazy')[source]
Download raw tick data across a date range in 1-day chunks.
- Return type:
Union[DataFrame,LazyFrame]
- download_month_raw(ticker, year, month_num, temp_filepath='', engine='polars_lazy')[source]
Download raw tick data for an entire month from cTrader in 1-day chunks. Matches the interface of DukascopyConnector.download_month_raw.
- Return type:
Union[DataFrame,LazyFrame]
- forex_data.cTraderDataConnector
alias of
CTraderConnector
- class forex_data.DukascopyConnector(**kwargs)[source]
Bases:
RemoteConnectorConnector class that encapsulates all HTTP interactions with Dukascopy’s historical datafeed.
Wraps connectivity checks, ticker registry, and tick data downloading via tick_vault library behind a single RemoteConnector-derived interface.
- ssl_verify: bool
- __init__(**kwargs)[source]
- get_available_tickers()[source]
Get list of available tickers from tick_vault’s registry and Dukascopy’s tools page.
Returns
- List[str]
Sorted, deduplicated list of ticker symbols (e.g. [‘BTCUSD’, ‘EURUSD’, …]).
- download_month_raw(ticker, year, month_num, temp_filepath='', engine='polars_lazy')[source]
Downloads tick data for a specific year and month from Dukascopy using tick_vault.
- Return type:
Union[DataFrame,LazyFrame]
Parameters
- ticker: str
The currency pair (e.g., ‘EURUSD’).
- year: int
Year of the data to download.
- month_num: int
Month of the data (1-12).
- engine: str
Either ‘polars’ or ‘polars_lazy’.
Returns
- Union[PolarsDataFrame, PolarsLazyFrame]
Polars DataFrame or LazyFrame with column names matching HistDataConnector.
- get_recent_data(symbol, timeframe, interval_window, engine='polars_lazy')[source]
Fetches recent data relative to the current time minus the interval_window.
- Return type:
Union[DataFrame,LazyFrame]
Parameters
- symbol: str
The currency pair (e.g. ‘EURUSD’).
- timeframe: str
The target timeframe (e.g. ‘TICK’, ‘1m’, ‘5m’).
- interval_window: timedelta
The duration of data to fetch (e.g., timedelta(days=90)).
- engine: str
Either ‘polars’ or ‘polars_lazy’.
Returns
- Union[PolarsDataFrame, PolarsLazyFrame]
Polars DataFrame or LazyFrame containing recent data.
- forex_data.concat_data(data_list=_CountingAttr(counter=27, _default=NOTHING, repr=True, eq=True, order=True, hash=None, init=True, on_setattr=None, alias=None, metadata={}))[source]
- forex_data.validator_dir_path(create_if_missing=False)[source]
- forex_data.get_histdata_tickers(verify=True)[source]
Get all available tickers from HistData.com.
Parameters
- verifybool, optional
Whether to verify SSL certificates. Default is True.
Returns
- List[str]
List of all available tickers (e.g., [‘EURUSD’, ‘GBPUSD’, …]).
- class forex_data.POLARS_DTYPE_DICT[source]
Bases:
object- TICK_DTYPE = {'ask': Float32, 'ask_volume': Float32, 'bid': Float32, 'bid_volume': Float32, 'vwmp': Float32}
- TF_DTYPE = {'ask': Float32, 'ask_volume': Float32, 'bid': Float32, 'bid_volume': Float32, 'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32, 'vwmp': Float32, 'vwmp_avg': Float32}
- TIME_TICK_DTYPE = {'ask': Float32, 'ask_volume': Float32, 'bid': Float32, 'bid_volume': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None), 'vwmp': Float32}
- TIME_TF_DTYPE = {'ask': Float32, 'ask_volume': Float32, 'bid': Float32, 'bid_volume': Float32, 'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None), 'vwmp': Float32, 'vwmp_avg': Float32}
- forex_data.business_days_data(dataframe)[source]
Remove non-business days data from the input dataframe. Filter out weekends data: saturday and sunday. Use holidays to get list of country holidays. Consider dataframe always have a column named ‘timestamp’ of type datetime.
- Return type:
DataFrame|LazyFrame
- forex_data.random_date_between(start_date, end_date)[source]
Get a random datetime between two datetime objects.
- Parameters:
start_date (
datetime) – The start date.end_date (
datetime) – The end date.
- Returns:
A random datetime between start_date and end_date.
- Return type:
datetime
- class forex_data.DataCharts(config='', realtime_connector=None, data_path=None, **kwargs)[source]
Bases:
HistoricalManagerDB- __init__(config='', realtime_connector=None, data_path=None, **kwargs)[source]
Initializes DataCharts instance.
- plot_historical(ticker, timeframe, start_date, end_date, chart_type=SupportedChartType.OHLC)[source]
Plots historical data for the specified ticker and date range.
- start_realtime_plot(ticker, timeframe, start_date, update_interval_ms=1000, chart_type=SupportedChartType.OHLC, port=8050)[source]
Starts a realtime charting server for the specified ticker.
- close()[source]
- config: str
- data_type: str
- engine: str
- db_files_year_partitioning: bool
- volume_data: bool
- ssl_verify: bool
- polars_gpu_engine: bool
- connector_id: str
- max_discrepancy_with_now: str
- offline: bool
- forex_data.create_historical_figure(df, ticker, chart_type=SupportedChartType.OHLC)[source]
Creates a plotly figure based on the provided dataframe.
- class forex_data.RealtimeDashApp(data_fetcher_callback, ticker, chart_type=SupportedChartType.OHLC, interval_ms=1000, host='127.0.0.1', port=8050)[source]
Bases:
object- __init__(data_fetcher_callback, ticker, chart_type=SupportedChartType.OHLC, interval_ms=1000, host='127.0.0.1', port=8050)[source]
- start()[source]
Starts the Dash app in a daemon thread.
- stop()[source]
Dash does not provide a graceful shutdown for the development server
run via app.run(). Because it is started in a daemon thread, it will
terminate when the main program exits.