forex_data.data_management package
Submodules
forex_data.data_management.common module
Created on Sat Apr 30 09:23:19 2022
@author: fiora
- class forex_data.data_management.common.DEFAULT_PATHS[source]
Bases:
object- BASE_PATH = '/home/runner/.database'
- HIST_DATA_FOLDER = 'HistoricalData'
- REALTIME_DATA_FOLDER = 'RealtimeData'
- class forex_data.data_management.common.DATA_TYPE[source]
Bases:
object- CSV_FILETYPE = 'csv'
- PARQUET_FILETYPE = 'parquet'
- DUCKDB = 'duckdb'
- class forex_data.data_management.common.BASE_DATA_COLUMN_NAME[source]
Bases:
object- TIMESTAMP = 'timestamp'
- OPEN = 'open'
- HIGH = 'high'
- LOW = 'low'
- CLOSE = 'close'
- ASK = 'ask'
- BID = 'bid'
- VOL = 'vol'
- P_VALUE = 'p'
- TRANSACTIONS = 'transactions'
- VWAP = 'vwap'
- OTC = 'otc'
- class forex_data.data_management.common.DATA_FILE_COLUMN_INDEX[source]
Bases:
object- TIMESTAMP = 0
- class forex_data.data_management.common.ASSET_TYPE[source]
Bases:
object- STOCK = 'STOCK'
- ETF = 'ETF'
- FOREX = 'FOREX'
- class forex_data.data_management.common.DTYPE_DICT[source]
Bases:
object- TICK_DTYPE = {'ask': 'float32', 'bid': 'float32', 'p': 'float32', 'vol': 'float32'}
- TF_DTYPE = {'close': 'float32', 'high': 'float32', 'low': 'float32', 'open': 'float32'}
- TIME_TICK_DTYPE = {'ask': 'float32', 'bid': 'float32', 'p': 'float32', 'timestamp': 'datetime64[ms]', 'vol': 'float32'}
- TIME_TF_DTYPE = {'close': 'float32', 'high': 'float32', 'low': 'float32', 'open': 'float32', 'timestamp': 'datetime64[ms]'}
- class forex_data.data_management.common.PYARROW_DTYPE_DICT[source]
Bases:
object- TICK_DTYPE = {'ask': DataType(float), 'bid': DataType(float), 'p': DataType(float), 'vol': DataType(float)}
- TF_DTYPE = {'close': DataType(float), 'high': DataType(float), 'low': DataType(float), 'open': DataType(float)}
- TIME_TICK_DTYPE = {'ask': DataType(float), 'bid': DataType(float), 'p': DataType(float), 'timestamp': TimestampType(timestamp[ms]), 'vol': DataType(float)}
- TIME_TF_DTYPE = {'close': DataType(float), 'high': DataType(float), 'low': DataType(float), 'open': DataType(float), 'timestamp': TimestampType(timestamp[ms])}
- class forex_data.data_management.common.POLARS_DTYPE_DICT[source]
Bases:
object- TICK_DTYPE = {'ask': Float32, 'bid': Float32, 'p': Float32, 'vol': Float32}
- TF_DTYPE = {'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32}
- TIME_TICK_DTYPE = {'ask': Float32, 'bid': Float32, 'p': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None), 'vol': Float32}
- TIME_TF_DTYPE = {'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None)}
- class forex_data.data_management.common.DATA_COLUMN_NAMES[source]
Bases:
object- TICK_DATA_NO_PVALUE = ['timestamp', 'ask', 'bid', 'vol']
- TICK_DATA = ['timestamp', 'ask', 'bid', 'vol', 'p']
- TF_DATA = ['timestamp', 'open', 'high', 'low', 'close']
- TICK_DATA_TIME_INDEX = ['ask', 'bid', 'vol', 'p']
- TF_DATA_TIME_INDEX = ['open', 'high', 'low', 'close']
- POLYGON_IO_AGGS = ['open', 'high', 'low', 'close', 'volume', 'vwap', 'timestamp', 'transactions']
- class forex_data.data_management.common.FILENAME_TEMPLATE[source]
Bases:
object- TICKER_INDEX = 0
- YEAR_INDEX = 1
- YEAR_NUMERICAL_CHAR = 1
- TF_INDEX = 2
- FILETYPE_INDEX = 3
- class forex_data.data_management.common.DATA_KEY[source]
Bases:
object- MARKET = 0
- TICKER_INDEX = 1
- TF_INDEX = 2
- class forex_data.data_management.common.REALTIME_DATA_PROVIDER[source]
Bases:
object- ALPHA_VANTAGE = 'ALPHA_VANTAGE'
- POLYGON_IO = 'POLYGON-IO'
- class forex_data.data_management.common.CANONICAL_INDEX[source]
Bases:
object- AV_LATEST_DATA_INDEX = 0
- AV_DF_DATA_INDEX = 0
- AV_DICT_INFO_INDEX = 1
- class forex_data.data_management.common.SQL_COMPARISON_OPERATORS[source]
Bases:
object- GREATER_THAN = '>'
- LESS_THAN = '<'
- GREATER_THAN_OR_EQUAL = '>='
- LESS_THAN_OR_EQUAL = '<='
- EQUAL = '=='
- NOT_EQUAL = '!='
- class forex_data.data_management.common.SQL_CONDITION_AGGREGATION_MODES[source]
Bases:
object- AND = 'AND'
- OR = 'OR'
- forex_data.data_management.common.any_date_to_datetime64(any_date, date_format='ISO8601', unit=None, to_pydatetime=False)[source]
- forex_data.data_management.common.concat_data(data_list=_CountingAttr(counter=27, _default=NOTHING, repr=True, eq=True, order=True, hash=None, init=True, on_setattr=None, alias=None, metadata={}))[source]
- forex_data.data_management.common.reframe_data(dataframe, tf)[source]
Parameters
- dataTYPE
DESCRIPTION.
- tfTYPE
DESCRIPTION.
Raises
- ValueError
DESCRIPTION.
Returns
- Dataframe
DESCRIPTION.
- forex_data.data_management.common.get_date_interval(start=None, end=None, interval_start_mode=None, interval_end_mode='now', interval_timespan=None, freq=None, normalize=False, bdays=False)[source]
- forex_data.data_management.common.random_date_between(start_date, end_date)[source]
Get a random datetime between two datetime objects.
- Parameters:
start_date (
datetime) – The start date.end_date (
datetime) – The end date.
- Returns:
A random datetime between start_date and end_date.
- Return type:
datetime
- forex_data.data_management.common.get_histdata_tickers()[source]
Get all available tickers from HistData.com.
Returns
- List[str]
List of all available tickers (e.g., [‘EURUSD’, ‘GBPUSD’, …]).
- forex_data.data_management.common.business_days_data(dataframe)[source]
Remove non-business days data from the input dataframe. Filter out weekends data: saturday and sunday. Use holidays to get list of country holidays. Consider dataframe always have a column named ‘timestamp’ of type datetime.
- Return type:
DataFrame|LazyFrame
- forex_data.data_management.common.update_ticker_years_dict(ticker_years_dict, ticker, timeframe, years_to_add)[source]
Update a ticker years dictionary with new years for a specific ticker and timeframe.
This function modifies the dictionary in place and returns whether any changes were made.
- Return type:
Parameters
- ticker_years_dictDict[str, Dict[str, List[int]]]
Dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
- tickerstr
The ticker symbol to update
- timeframestr
The timeframe for the ticker data
- years_to_addList[int]
List of years to add to the years list
Returns
- bool
True if any changes were made, False otherwise
forex_data.data_management.database module
Created on Sun Feb 23 00:02:36 2025
@author: fiora
- class forex_data.data_management.database.DatabaseConnector(**kwargs)[source]
Bases:
object- check_connection()[source]
Check database connection - must be implemented by subclasses.
- Return type:
- write_data(target_table, dataframe, clean=False)[source]
Write data to database - must be implemented by subclasses.
- Return type:
- read_data(market, ticker, timeframe, start, end)[source]
Read data from database - must be implemented by subclasses.
- Return type:
LazyFrame
- get_ticker_keys(ticker, timeframe=None)[source]
Get ticker keys - must be implemented by subclasses.
- class forex_data.data_management.database.DuckDBConnector(**kwargs)[source]
Bases:
DatabaseConnector- check_connection()[source]
Check database connection - must be implemented by subclasses.
- Return type:
- get_ticker_keys(ticker, timeframe=None)[source]
Get ticker keys - must be implemented by subclasses.
- get_ticker_years_list(ticker, timeframe='tick')[source]
Get years list for ticker - must be implemented by subclasses.
- write_data(target_table, dataframe, clean=False)[source]
Write data to database - must be implemented by subclasses.
- Return type:
- class forex_data.data_management.database.LocalDBConnector(**kwargs)[source]
Bases:
DatabaseConnector- get_ticker_keys(ticker, timeframe=None)[source]
Get ticker keys - must be implemented by subclasses.
- get_ticker_years_list(ticker, timeframe='tick')[source]
Get years list for ticker - must be implemented by subclasses.
- create_tickers_years_dict()[source]
Create a dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
If no data files exist yet, returns an empty dictionary.
- save_tickers_years_info(ticker_years_dict)[source]
Save ticker years list to a JSON file.
- Return type:
Parameters
- ticker_years_dictDict[str, Dict[str, List[int]]]
Dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
- filenamestr, optional
Name of the JSON file to save the data, by default ‘tickers_years.json’
Raises
- TypeError
If ticker_years_dict is not a dictionary
- IOError
If there’s an error writing the file
- add_tickers_years_info_to_file(ticker, timeframe, year)[source]
In local info filepath, update just the years list of the given ticker and timeframe by adding the year(s) specified if not already present
- Return type:
Parameters
- tickerstr
The ticker symbol to update
- timeframestr
The timeframe for the ticker data
- yearUnion[int, List[int]]
The year or list of years to add to the years list
Raises
- TypeError
If year is not an integer or list of integers
- clear_tickers_years_info(filter=None)[source]
Clear the tickers years info file. If filter is specified, it has to be a ticker value and so only the tickers years info related to the filter are cleared. If filter is not specified, the entire file is cleared. Parameters ———- filter : Optional[str], optional
Filter to apply to the tickers years info file, by default None Filter has to be a ticker value
- Return type:
- clear_database(filter=None)[source]
Clear database files If filter is provided and is a ticker present in database (files present) delete only files related to that ticker
- Return type:
- load_tickers_years_info()[source]
Load ticker years list from a JSON file.
Returns
- Dict[str, Dict[str, List[int]]]
Dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
Raises
- FileNotFoundError
If the JSON file doesn’t exist
- IOError
If there’s an error reading the file
forex_data.data_management.historicaldata module
- class forex_data.data_management.historicaldata.HistoricalManagerDB(**kwargs)[source]
Bases:
object- add_timeframe(timeframe)[source]
Add and cache a new timeframe to the database.
Creates aggregated data for the specified timeframe from tick data and caches it in the database for faster future access. The timeframe is added to the internal list of available timeframes.
- Parameters:
timeframe (
str | List[str]) – Timeframe(s) to add. Can be a single string or list of strings. Supported values: ‘1m’, ‘5m’, ‘15m’, ‘30m’, ‘1h’, ‘4h’, ‘1D’, ‘1W’, ‘1M’- Return type:
- Returns:
None
- Raises:
TypeError – If timeframe is not a string or list of strings
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> manager.add_timeframe('1W') # Add weekly timeframe >>> manager.add_timeframe(['4h', '1D']) # Add multiple timeframes
Note
Only new timeframes (not already in the list) will be processed
Aggregation can take time for large datasets
Once added, the timeframe is permanently cached in the database
- get_data(ticker, timeframe, start, end, comparison_column_name=None, check_level=None, comparison_operator=None, aggregation_mode=None)[source]
Retrieve OHLC historical data for the specified ticker and timeframe.
Fetches historical forex data from the database, automatically downloading and aggregating data if not already available. Supports multiple timeframes and date ranges.
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘NZDUSD’). Case-insensitive.timeframe (
str) – Candle timeframe for data aggregation. Supported values: ‘1m’, ‘5m’, ‘15m’, ‘30m’, ‘1h’, ‘4h’, ‘1D’, ‘1W’, ‘1M’start (
str | datetime) – Start date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime objectend (
str | datetime) – End date for data retrieval. Same format as start. Must be after start date.comparison_column_name (
List[str] | str | None) – Column names to retrieve. Default is None.check_level (
List[int | float] | int | float | None) – Check level for conditions. Default is None.comparison_operator (
List[SUPPORTED_SQL_COMPARISON_OPERATORS] | SUPPORTED_SQL_COMPARISON_OPERATORS | None) – Condition for data retrieval. Default is None.aggregation_mode (
SUPPORTED_SQL_CONDITION_AGGREGATION_MODES | None) – Aggregation mode for data retrieval. Default is None.
- Returns:
- DataFrame containing OHLC data with columns:
timestamp: datetime column with candle timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)
- Return type:
polars.DataFrame | polars.LazyFrame
- Raises:
TickerNotFoundError – If the ticker is not available in the historical database
ValueError – If timeframe is invalid or end date is before start date
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> data = manager.get_data( ... ticker='EURUSD', ... timeframe='1h', ... start='2020-01-01', ... end='2020-01-31' ... ) >>> print(f"Retrieved {len(data)} hourly candles") Retrieved 744 hourly candles
Note
Data is automatically downloaded from histdata.com if not cached locally
First call for a new timeframe may take longer as it builds the aggregation
Downloaded data is cached for faster subsequent access
Ticker names are case-insensitive and automatically normalized
- plot(ticker, timeframe, start_date, end_date)[source]
Plot candlestick chart for the specified ticker and date range.
Generates an interactive candlestick chart using mplfinance, displaying OHLC (Open, High, Low, Close) data for the specified time period.
- Parameters:
- Returns:
Displays the chart using matplotlib
- Return type:
None
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> manager.plot( ... ticker='EURUSD', ... timeframe='1D', ... start_date='2020-01-01', ... end_date='2020-12-31' ... )
Note
The chart will be displayed in a matplotlib window. The data is automatically fetched using get_data() and converted to the appropriate format for plotting.
forex_data.data_management.realtimedata module
Created on Mon Apr 25 18:07:21 2022
@author: fiora
- class forex_data.data_management.realtimedata.RealtimeManager(**kwargs)[source]
Bases:
object- get_daily_close(ticker, last_close=False, recent_days_window=None, day_start=None, day_end=None)[source]
Retrieve daily OHLC data for the specified ticker.
Fetches daily forex data from Alpha Vantage API. Supports three modes of operation: last close only, recent N days window, or specific date range.
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘USDJPY’). Case-insensitive.last_close (
bool, optional) – If True, returns only the most recent daily close. Default is False.recent_days_window (
int, optional) – Number of recent days to retrieve. Mutually exclusive with day_start/day_end. Default is None.day_start (
str, optional) – Start date for data retrieval in ‘YYYY-MM-DD’ format. Used with day_end to specify exact date range. Default is None.day_end (
str, optional) – End date for data retrieval in ‘YYYY-MM-DD’ format. Used with day_start to specify exact date range. Default is None.
- Returns:
DataFrame containing daily OHLC data with columns:
timestamp: datetime column with daily timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)
Returns empty DataFrame if API call fails.
- Return type:
polars.DataFrame | polars.LazyFrame
- Raises:
AssertionError – If recent_days_window is not an integer when provided
BadResponse – If Alpha Vantage API request fails (handled internally)
Example:
# Get last close only manager = RealtimeManager(config='data_config.yaml') latest = manager.get_daily_close(ticker='EURUSD', last_close=True) # Get last 10 days recent = manager.get_daily_close(ticker='EURUSD', recent_days_window=10) # Get specific date range range_data = manager.get_daily_close( ticker='EURUSD', day_start='2024-01-01', day_end='2024-01-31' )
Note
Requires valid Alpha Vantage API key in configuration
Free tier has 25 requests per day limit
outputsize=’compact’ returns ~100 most recent data points
outputsize=’full’ can return several years of data
Use last_close=True for minimal data transfer
- get_data(ticker, start=None, end=None, timeframe=None)[source]
Retrieve real-time OHLC data for the specified ticker and timeframe.
Fetches intraday forex data from Polygon.io API for the specified date range and timeframe. Data is automatically reframed to the requested timeframe.
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘USDJPY’). Case-insensitive.start (
str | datetime, optional) – Start date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime object Default is None.end (
str | datetime, optional) – End date for data retrieval. Same format as start. Must be after start date. Default is None.timeframe (
str, optional) – Target timeframe for aggregation. If specified, minute data will be reframed to this timeframe (e.g., ‘5m’, ‘1h’, ‘1D’). Default is None (returns minute data).
- Returns:
DataFrame containing OHLC data with columns:
timestamp: datetime column with candle timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)
Returns empty DataFrame if API call fails.
- Return type:
polars.DataFrame | polars.LazyFrame
- Raises:
BadResponse – If Polygon.io API request fails (handled internally, returns empty DataFrame)
Example:
# Get hourly data for 5 days manager = RealtimeManager(config='data_config.yaml') data = manager.get_data( ticker='EURUSD', start='2024-01-10', end='2024-01-15', timeframe='1h' ) print(f"Retrieved {len(data)} hourly candles") # Output: Retrieved 120 hourly candles
Note
Requires valid Polygon.io API key in configuration
Free tier has rate limits and historical data restrictions
Data is fetched at 1-minute resolution and aggregated to requested timeframe
Failed requests return an empty DataFrame with a warning logged
Module contents
Created on Sat Jul 16 01:31:35 2022
@author: fiora
- class forex_data.data_management.HistoricalManagerDB(**kwargs)[source]
Bases:
object- add_timeframe(timeframe)[source]
Add and cache a new timeframe to the database.
Creates aggregated data for the specified timeframe from tick data and caches it in the database for faster future access. The timeframe is added to the internal list of available timeframes.
- Parameters:
timeframe (
str | List[str]) – Timeframe(s) to add. Can be a single string or list of strings. Supported values: ‘1m’, ‘5m’, ‘15m’, ‘30m’, ‘1h’, ‘4h’, ‘1D’, ‘1W’, ‘1M’- Return type:
- Returns:
None
- Raises:
TypeError – If timeframe is not a string or list of strings
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> manager.add_timeframe('1W') # Add weekly timeframe >>> manager.add_timeframe(['4h', '1D']) # Add multiple timeframes
Note
Only new timeframes (not already in the list) will be processed
Aggregation can take time for large datasets
Once added, the timeframe is permanently cached in the database
- get_data(ticker, timeframe, start, end, comparison_column_name=None, check_level=None, comparison_operator=None, aggregation_mode=None)[source]
Retrieve OHLC historical data for the specified ticker and timeframe.
Fetches historical forex data from the database, automatically downloading and aggregating data if not already available. Supports multiple timeframes and date ranges.
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘NZDUSD’). Case-insensitive.timeframe (
str) – Candle timeframe for data aggregation. Supported values: ‘1m’, ‘5m’, ‘15m’, ‘30m’, ‘1h’, ‘4h’, ‘1D’, ‘1W’, ‘1M’start (
str | datetime) – Start date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime objectend (
str | datetime) – End date for data retrieval. Same format as start. Must be after start date.comparison_column_name (
List[str] | str | None) – Column names to retrieve. Default is None.check_level (
List[int | float] | int | float | None) – Check level for conditions. Default is None.comparison_operator (
List[SUPPORTED_SQL_COMPARISON_OPERATORS] | SUPPORTED_SQL_COMPARISON_OPERATORS | None) – Condition for data retrieval. Default is None.aggregation_mode (
SUPPORTED_SQL_CONDITION_AGGREGATION_MODES | None) – Aggregation mode for data retrieval. Default is None.
- Returns:
- DataFrame containing OHLC data with columns:
timestamp: datetime column with candle timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)
- Return type:
polars.DataFrame | polars.LazyFrame
- Raises:
TickerNotFoundError – If the ticker is not available in the historical database
ValueError – If timeframe is invalid or end date is before start date
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> data = manager.get_data( ... ticker='EURUSD', ... timeframe='1h', ... start='2020-01-01', ... end='2020-01-31' ... ) >>> print(f"Retrieved {len(data)} hourly candles") Retrieved 744 hourly candles
Note
Data is automatically downloaded from histdata.com if not cached locally
First call for a new timeframe may take longer as it builds the aggregation
Downloaded data is cached for faster subsequent access
Ticker names are case-insensitive and automatically normalized
- plot(ticker, timeframe, start_date, end_date)[source]
Plot candlestick chart for the specified ticker and date range.
Generates an interactive candlestick chart using mplfinance, displaying OHLC (Open, High, Low, Close) data for the specified time period.
- Parameters:
- Returns:
Displays the chart using matplotlib
- Return type:
None
Example
>>> manager = HistoricalManagerDB(config='data_config.yaml') >>> manager.plot( ... ticker='EURUSD', ... timeframe='1D', ... start_date='2020-01-01', ... end_date='2020-12-31' ... )
Note
The chart will be displayed in a matplotlib window. The data is automatically fetched using get_data() and converted to the appropriate format for plotting.
- class forex_data.data_management.RealtimeManager(**kwargs)[source]
Bases:
object- get_daily_close(ticker, last_close=False, recent_days_window=None, day_start=None, day_end=None)[source]
Retrieve daily OHLC data for the specified ticker.
Fetches daily forex data from Alpha Vantage API. Supports three modes of operation: last close only, recent N days window, or specific date range.
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘USDJPY’). Case-insensitive.last_close (
bool, optional) – If True, returns only the most recent daily close. Default is False.recent_days_window (
int, optional) – Number of recent days to retrieve. Mutually exclusive with day_start/day_end. Default is None.day_start (
str, optional) – Start date for data retrieval in ‘YYYY-MM-DD’ format. Used with day_end to specify exact date range. Default is None.day_end (
str, optional) – End date for data retrieval in ‘YYYY-MM-DD’ format. Used with day_start to specify exact date range. Default is None.
- Returns:
DataFrame containing daily OHLC data with columns:
timestamp: datetime column with daily timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)
Returns empty DataFrame if API call fails.
- Return type:
polars.DataFrame | polars.LazyFrame
- Raises:
AssertionError – If recent_days_window is not an integer when provided
BadResponse – If Alpha Vantage API request fails (handled internally)
Example:
# Get last close only manager = RealtimeManager(config='data_config.yaml') latest = manager.get_daily_close(ticker='EURUSD', last_close=True) # Get last 10 days recent = manager.get_daily_close(ticker='EURUSD', recent_days_window=10) # Get specific date range range_data = manager.get_daily_close( ticker='EURUSD', day_start='2024-01-01', day_end='2024-01-31' )
Note
Requires valid Alpha Vantage API key in configuration
Free tier has 25 requests per day limit
outputsize=’compact’ returns ~100 most recent data points
outputsize=’full’ can return several years of data
Use last_close=True for minimal data transfer
- get_data(ticker, start=None, end=None, timeframe=None)[source]
Retrieve real-time OHLC data for the specified ticker and timeframe.
Fetches intraday forex data from Polygon.io API for the specified date range and timeframe. Data is automatically reframed to the requested timeframe.
- Parameters:
ticker (
str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘USDJPY’). Case-insensitive.start (
str | datetime, optional) – Start date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime object Default is None.end (
str | datetime, optional) – End date for data retrieval. Same format as start. Must be after start date. Default is None.timeframe (
str, optional) – Target timeframe for aggregation. If specified, minute data will be reframed to this timeframe (e.g., ‘5m’, ‘1h’, ‘1D’). Default is None (returns minute data).
- Returns:
DataFrame containing OHLC data with columns:
timestamp: datetime column with candle timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)
Returns empty DataFrame if API call fails.
- Return type:
polars.DataFrame | polars.LazyFrame
- Raises:
BadResponse – If Polygon.io API request fails (handled internally, returns empty DataFrame)
Example:
# Get hourly data for 5 days manager = RealtimeManager(config='data_config.yaml') data = manager.get_data( ticker='EURUSD', start='2024-01-10', end='2024-01-15', timeframe='1h' ) print(f"Retrieved {len(data)} hourly candles") # Output: Retrieved 120 hourly candles
Note
Requires valid Polygon.io API key in configuration
Free tier has rate limits and historical data restrictions
Data is fetched at 1-minute resolution and aggregated to requested timeframe
Failed requests return an empty DataFrame with a warning logged
- class forex_data.data_management.BASE_DATA_COLUMN_NAME[source]
Bases:
object- TIMESTAMP = 'timestamp'
- OPEN = 'open'
- HIGH = 'high'
- LOW = 'low'
- CLOSE = 'close'
- ASK = 'ask'
- BID = 'bid'
- VOL = 'vol'
- P_VALUE = 'p'
- TRANSACTIONS = 'transactions'
- VWAP = 'vwap'
- OTC = 'otc'
- class forex_data.data_management.DEFAULT_PATHS[source]
Bases:
object- BASE_PATH = '/home/runner/.database'
- HIST_DATA_FOLDER = 'HistoricalData'
- REALTIME_DATA_FOLDER = 'RealtimeData'
- class forex_data.data_management.SQL_COMPARISON_OPERATORS[source]
Bases:
object- GREATER_THAN = '>'
- LESS_THAN = '<'
- GREATER_THAN_OR_EQUAL = '>='
- LESS_THAN_OR_EQUAL = '<='
- EQUAL = '=='
- NOT_EQUAL = '!='
- class forex_data.data_management.SQL_CONDITION_AGGREGATION_MODES[source]
Bases:
object- AND = 'AND'
- OR = 'OR'
- forex_data.data_management.any_date_to_datetime64(any_date, date_format='ISO8601', unit=None, to_pydatetime=False)[source]
- class forex_data.data_management.DatabaseConnector(**kwargs)[source]
Bases:
object- check_connection()[source]
Check database connection - must be implemented by subclasses.
- Return type:
- write_data(target_table, dataframe, clean=False)[source]
Write data to database - must be implemented by subclasses.
- Return type:
- read_data(market, ticker, timeframe, start, end)[source]
Read data from database - must be implemented by subclasses.
- Return type:
LazyFrame
- get_ticker_keys(ticker, timeframe=None)[source]
Get ticker keys - must be implemented by subclasses.
- class forex_data.data_management.DuckDBConnector(**kwargs)[source]
Bases:
DatabaseConnector- check_connection()[source]
Check database connection - must be implemented by subclasses.
- Return type:
- get_ticker_keys(ticker, timeframe=None)[source]
Get ticker keys - must be implemented by subclasses.
- get_ticker_years_list(ticker, timeframe='tick')[source]
Get years list for ticker - must be implemented by subclasses.
- write_data(target_table, dataframe, clean=False)[source]
Write data to database - must be implemented by subclasses.
- Return type:
- class forex_data.data_management.LocalDBConnector(**kwargs)[source]
Bases:
DatabaseConnector- get_ticker_keys(ticker, timeframe=None)[source]
Get ticker keys - must be implemented by subclasses.
- get_ticker_years_list(ticker, timeframe='tick')[source]
Get years list for ticker - must be implemented by subclasses.
- create_tickers_years_dict()[source]
Create a dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
If no data files exist yet, returns an empty dictionary.
- save_tickers_years_info(ticker_years_dict)[source]
Save ticker years list to a JSON file.
- Return type:
Parameters
- ticker_years_dictDict[str, Dict[str, List[int]]]
Dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
- filenamestr, optional
Name of the JSON file to save the data, by default ‘tickers_years.json’
Raises
- TypeError
If ticker_years_dict is not a dictionary
- IOError
If there’s an error writing the file
- add_tickers_years_info_to_file(ticker, timeframe, year)[source]
In local info filepath, update just the years list of the given ticker and timeframe by adding the year(s) specified if not already present
- Return type:
Parameters
- tickerstr
The ticker symbol to update
- timeframestr
The timeframe for the ticker data
- yearUnion[int, List[int]]
The year or list of years to add to the years list
Raises
- TypeError
If year is not an integer or list of integers
- clear_tickers_years_info(filter=None)[source]
Clear the tickers years info file. If filter is specified, it has to be a ticker value and so only the tickers years info related to the filter are cleared. If filter is not specified, the entire file is cleared. Parameters ———- filter : Optional[str], optional
Filter to apply to the tickers years info file, by default None Filter has to be a ticker value
- Return type:
- clear_database(filter=None)[source]
Clear database files If filter is provided and is a ticker present in database (files present) delete only files related to that ticker
- Return type:
- load_tickers_years_info()[source]
Load ticker years list from a JSON file.
Returns
- Dict[str, Dict[str, List[int]]]
Dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
Raises
- FileNotFoundError
If the JSON file doesn’t exist
- IOError
If there’s an error reading the file
- forex_data.data_management.concat_data(data_list=_CountingAttr(counter=27, _default=NOTHING, repr=True, eq=True, order=True, hash=None, init=True, on_setattr=None, alias=None, metadata={}))[source]
- forex_data.data_management.get_histdata_tickers()[source]
Get all available tickers from HistData.com.
Returns
- List[str]
List of all available tickers (e.g., [‘EURUSD’, ‘GBPUSD’, …]).
- class forex_data.data_management.POLARS_DTYPE_DICT[source]
Bases:
object- TICK_DTYPE = {'ask': Float32, 'bid': Float32, 'p': Float32, 'vol': Float32}
- TF_DTYPE = {'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32}
- TIME_TICK_DTYPE = {'ask': Float32, 'bid': Float32, 'p': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None), 'vol': Float32}
- TIME_TF_DTYPE = {'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None)}
- forex_data.data_management.business_days_data(dataframe)[source]
Remove non-business days data from the input dataframe. Filter out weekends data: saturday and sunday. Use holidays to get list of country holidays. Consider dataframe always have a column named ‘timestamp’ of type datetime.
- Return type:
DataFrame|LazyFrame