forex_data_aggregator package

Creates aggregated data for the specified timeframe from tick data and caches it in the database for faster future access. The timeframe is added to the internal list of available timeframes.

Parameters:: timeframe (str | List[str]) – Timeframe(s) to add. Can be a single string or list of strings. Supported values: ‘1m’, ‘5m’, ‘15m’, ‘30m’, ‘1h’, ‘4h’, ‘1D’, ‘1W’, ‘1M’
Return type:: None
Returns:: None
Raises:: TypeError – If timeframe is not a string or list of strings

Example

>>> manager = HistoricalManagerDB(config='data_config.yaml')
>>> manager.add_timeframe('1W')  # Add weekly timeframe
>>> manager.add_timeframe(['4h', '1D'])  # Add multiple timeframes

Note

Only new timeframes (not already in the list) will be processed
Aggregation can take time for large datasets
Once added, the timeframe is permanently cached in the database

get_data(ticker, timeframe, start, end, comparison_column_name=None, check_level=None, comparison_operator=None, aggregation_mode=None)[source]

Retrieve OHLC historical data for the specified ticker and timeframe.

Fetches historical forex data from the database, automatically downloading and aggregating data if not already available. Supports multiple timeframes and date ranges.

Parameters:

ticker (str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘NZDUSD’). Case-insensitive.
timeframe (str) – Candle timeframe for data aggregation. Supported values: ‘1m’, ‘5m’, ‘15m’, ‘30m’, ‘1h’, ‘4h’, ‘1D’, ‘1W’, ‘1M’
start (str | datetime) – Start date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime object
end (str | datetime) – End date for data retrieval. Same format as start. Must be after start date.
comparison_column_name (List[str] | str | None) – Column names to retrieve. Default is None.
check_level (List[int | float] | int | float | None) – Check level for conditions. Default is None.
comparison_operator (List[SUPPORTED_SQL_COMPARISON_OPERATORS] | SUPPORTED_SQL_COMPARISON_OPERATORS | None) – Condition for data retrieval. Default is None.
aggregation_mode (SUPPORTED_SQL_CONDITION_AGGREGATION_MODES | None) – Aggregation mode for data retrieval. Default is None.

Returns:

DataFrame containing OHLC data with columns:

timestamp: datetime column with candle timestamps
open: Opening price (float32)
high: Highest price (float32)
low: Lowest price (float32)
close: Closing price (float32)

Return type:

polars.DataFrame | polars.LazyFrame

Raises:

TickerNotFoundError – If the ticker is not available in the historical database
ValueError – If timeframe is invalid or end date is before start date

Example

>>> manager = HistoricalManagerDB(config='data_config.yaml')
>>> data = manager.get_data(
...     ticker='EURUSD',
...     timeframe='1h',
...     start='2020-01-01',
...     end='2020-01-31'
... )
>>> print(f"Retrieved {len(data)} hourly candles")
Retrieved 744 hourly candles

Note

Data is automatically downloaded from histdata.com if not cached locally
First call for a new timeframe may take longer as it builds the aggregation
Downloaded data is cached for faster subsequent access
Ticker names are case-insensitive and automatically normalized

plot(ticker, timeframe, start_date, end_date)[source]

Plot candlestick chart for the specified ticker and date range.

Generates an interactive candlestick chart using mplfinance, displaying OHLC (Open, High, Low, Close) data for the specified time period.

Parameters:

ticker (str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’)
timeframe (str) – Candle timeframe (e.g., ‘1m’, ‘5m’, ‘1h’, ‘1D’, ‘1W’)
start_date (str) – Start date in ISO format ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’
end_date (str) – End date in ISO format ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’

Returns:

Displays the chart using matplotlib

Return type:

None

Example

>>> manager = HistoricalManagerDB(config='data_config.yaml')
>>> manager.plot(
...     ticker='EURUSD',
...     timeframe='1D',
...     start_date='2020-01-01',
...     end_date='2020-12-31'
... )

Note

The chart will be displayed in a matplotlib window. The data is automatically fetched using get_data() and converted to the appropriate format for plotting.

close()[source]

class forex_data.RealtimeManager(**kwargs)[source]

Bases: object

config: str

providers_key: dict

data_type: str

engine: str

data_path: str | Path

__init__(**kwargs)[source]

tickers_list(data_source, asset_class=None)[source]

Return type:: List[str]

get_realtime_quote(ticker)[source]

Return type:: Any

get_daily_close(ticker, last_close=False, recent_days_window=None, day_start=None, day_end=None)[source]

Retrieve daily OHLC data for the specified ticker.

Fetches daily forex data from Alpha Vantage API. Supports three modes of operation: last close only, recent N days window, or specific date range.

Parameters:

ticker (str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘USDJPY’). Case-insensitive.
last_close (bool, optional) – If True, returns only the most recent daily close. Default is False.
recent_days_window (int, optional) – Number of recent days to retrieve. Mutually exclusive with day_start/day_end. Default is None.
day_start (str, optional) – Start date for data retrieval in ‘YYYY-MM-DD’ format. Used with day_end to specify exact date range. Default is None.
day_end (str, optional) – End date for data retrieval in ‘YYYY-MM-DD’ format. Used with day_start to specify exact date range. Default is None.

Returns:

DataFrame containing daily OHLC data with columns:

timestamp: datetime column with daily timestamps

open: Opening price (float32)

high: Highest price (float32)

low: Lowest price (float32)

close: Closing price (float32)

Returns empty DataFrame if API call fails.

Return type:

polars.DataFrame | polars.LazyFrame

Raises:

AssertionError – If recent_days_window is not an integer when provided
BadResponse – If Alpha Vantage API request fails (handled internally)

Example:

# Get last close only
manager = RealtimeManager(config='data_config.yaml')
latest = manager.get_daily_close(ticker='EURUSD', last_close=True)

# Get last 10 days
recent = manager.get_daily_close(ticker='EURUSD', recent_days_window=10)

# Get specific date range
range_data = manager.get_daily_close(
    ticker='EURUSD',
    day_start='2024-01-01',
    day_end='2024-01-31'
)

Note

Requires valid Alpha Vantage API key in configuration
Free tier has 25 requests per day limit
outputsize=’compact’ returns ~100 most recent data points
outputsize=’full’ can return several years of data
Use last_close=True for minimal data transfer

get_data(ticker, start=None, end=None, timeframe=None)[source]

Retrieve real-time OHLC data for the specified ticker and timeframe.

Fetches intraday forex data from Polygon.io API for the specified date range and timeframe. Data is automatically reframed to the requested timeframe.

Parameters:

ticker (str) – Currency pair symbol (e.g., ‘EURUSD’, ‘GBPUSD’, ‘USDJPY’). Case-insensitive.
start (str | datetime, optional) – Start date for data retrieval. Accepts: - ISO format: ‘YYYY-MM-DD’ or ‘YYYY-MM-DD HH:MM:SS’ - datetime object Default is None.
end (str | datetime, optional) – End date for data retrieval. Same format as start. Must be after start date. Default is None.
timeframe (str, optional) – Target timeframe for aggregation. If specified, minute data will be reframed to this timeframe (e.g., ‘5m’, ‘1h’, ‘1D’). Default is None (returns minute data).

Returns:

DataFrame containing OHLC data with columns:

timestamp: datetime column with candle timestamps

open: Opening price (float32)

high: Highest price (float32)

low: Lowest price (float32)

close: Closing price (float32)

Returns empty DataFrame if API call fails.

Return type:

polars.DataFrame | polars.LazyFrame

Raises:

BadResponse – If Polygon.io API request fails (handled internally, returns empty DataFrame)

Example:

# Get hourly data for 5 days
manager = RealtimeManager(config='data_config.yaml')
data = manager.get_data(
    ticker='EURUSD',
    start='2024-01-10',
    end='2024-01-15',
    timeframe='1h'
)
print(f"Retrieved {len(data)} hourly candles")
# Output: Retrieved 120 hourly candles

Note

Requires valid Polygon.io API key in configuration
Free tier has rate limits and historical data restrictions
Data is fetched at 1-minute resolution and aggregated to requested timeframe
Failed requests return an empty DataFrame with a warning logged

class forex_data.BASE_DATA_COLUMN_NAME[source]

Bases: object

TIMESTAMP = 'timestamp'

OPEN = 'open'

HIGH = 'high'

LOW = 'low'

CLOSE = 'close'

ASK = 'ask'

BID = 'bid'

VOL = 'vol'

P_VALUE = 'p'

TRANSACTIONS = 'transactions'

VWAP = 'vwap'

OTC = 'otc'

class forex_data.DATA_FILE_COLUMN_INDEX[source]

Bases: object

TIMESTAMP = 0

class forex_data.DEFAULT_PATHS[source]

Bases: object

BASE_PATH = '/home/runner/.database'

HIST_DATA_FOLDER = 'HistoricalData'

REALTIME_DATA_FOLDER = 'RealtimeData'

class forex_data.SQL_COMPARISON_OPERATORS[source]

Bases: object

GREATER_THAN = '>'

LESS_THAN = '<'

GREATER_THAN_OR_EQUAL = '>='

LESS_THAN_OR_EQUAL = '<='

EQUAL = '=='

NOT_EQUAL = '!='

class forex_data.SQL_CONDITION_AGGREGATION_MODES[source]

Bases: object

AND = 'AND'

OR = 'OR'

forex_data.empty_dataframe(engine)[source]

forex_data.is_empty_dataframe(dataframe)[source]

forex_data.shape_dataframe(dataframe)[source]

forex_data.get_dataframe_column(dataframe, column)[source]

forex_data.get_dataframe_row(dataframe, row)[source]

forex_data.get_dataframe_element(dataframe, column, row)[source]

forex_data.get_attrs_names(instance_object, **kwargs)[source]

forex_data.any_date_to_datetime64(any_date, date_format='ISO8601', unit=None, to_pydatetime=False)[source]

forex_data.get_db_key_elements(key)[source]

forex_data.check_timeframe_str(tf, engine='pandas')[source]

class forex_data.DatabaseConnector(**kwargs)[source]

Bases: object

data_path: str | Path

__init__(**kwargs)[source]

connect()[source]

Connect to database - must be implemented by subclasses.

Return type:: Any

check_connection()[source]

Check database connection - must be implemented by subclasses.

Return type:: bool

write_data(target_table, dataframe, clean=False)[source]

Write data to database - must be implemented by subclasses.

Return type:: None

read_data(market, ticker, timeframe, start, end)[source]

Read data from database - must be implemented by subclasses.

Return type:: LazyFrame

exec_sql()[source]

Execute SQL - must be implemented by subclasses.

Return type:: None

get_tickers_list()[source]

Get list of tickers - must be implemented by subclasses.

Return type:: List[str]

get_ticker_keys(ticker, timeframe=None)[source]

Get ticker keys - must be implemented by subclasses.

Return type:: List[str]

get_ticker_years_list(ticker, timeframe='tick')[source]

Get years list for ticker - must be implemented by subclasses.

Return type:: List[int]

clear_database(filter=None)[source]

Clear database - must be implemented by subclasses.

Return type:: None

class forex_data.DuckDBConnector(**kwargs)[source]

Bases: DatabaseConnector

__init__(**kwargs)[source]

connect()[source]

Connect to database - must be implemented by subclasses.

Return type:: Any

check_connection()[source]

Check database connection - must be implemented by subclasses.

Return type:: bool

get_tickers_list()[source]

Get list of tickers - must be implemented by subclasses.

Return type:: List[str]

get_ticker_keys(ticker, timeframe=None)[source]

Get ticker keys - must be implemented by subclasses.

Return type:: List[str]

get_ticker_years_list(ticker, timeframe='tick')[source]

Get years list for ticker - must be implemented by subclasses.

Return type:: List[int]

write_data(target_table, dataframe, clean=False)[source]

Write data to database - must be implemented by subclasses.

Return type:: None

read_data(market, ticker, timeframe, start, end)[source]

Read data from database - must be implemented by subclasses.

Return type:: LazyFrame

clear_database(filter=None)[source]

Clear database tables If filter is provided, delete only tables related to that ticker

Return type:: None

class forex_data.LocalDBConnector(**kwargs)[source]

Bases: DatabaseConnector

data_type: str

engine: str

__init__(**kwargs)[source]

get_tickers_list()[source]

Get list of tickers - must be implemented by subclasses.

Return type:: List[str]

get_ticker_keys(ticker, timeframe=None)[source]

Get ticker keys - must be implemented by subclasses.

Return type:: List[str]

get_ticker_timeframes_list(ticker)[source]

Return type:: List[str]

get_ticker_years_list(ticker, timeframe='tick')[source]

Get years list for ticker - must be implemented by subclasses.

Return type:: List[int]

create_tickers_years_dict()[source]

Create a dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}

If no data files exist yet, returns an empty dictionary.

Return type:: Dict[str, Dict[str, List[int]]]

save_tickers_years_info(ticker_years_dict)[source]

Save ticker years list to a JSON file.

Return type:: None

Parameters

ticker_years_dictDict[str, Dict[str, List[int]]]: Dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}
filenamestr, optional: Name of the JSON file to save the data, by default ‘tickers_years.json’

Raises

TypeError: If ticker_years_dict is not a dictionary
IOError: If there’s an error writing the file

add_tickers_years_info_to_file(ticker, timeframe, year)[source]

In local info filepath, update just the years list of the given ticker and timeframe by adding the year(s) specified if not already present

Return type:: None

Parameters

tickerstr: The ticker symbol to update
timeframestr: The timeframe for the ticker data
yearUnion[int, List[int]]: The year or list of years to add to the years list

Raises

TypeError: If year is not an integer or list of integers

clear_tickers_years_info(filter=None)[source]

Clear the tickers years info file. If filter is specified, it has to be a ticker value and so only the tickers years info related to the filter are cleared. If filter is not specified, the entire file is cleared. Parameters ———- filter : Optional[str], optional

Filter to apply to the tickers years info file, by default None Filter has to be a ticker value

Return type:: None

clear_database(filter=None)[source]

Clear database files If filter is provided and is a ticker present in database (files present) delete only files related to that ticker

Return type:: None

load_tickers_years_info()[source]

Load ticker years list from a JSON file.

Return type:: Dict[str, Dict[str, List[int]]]

Returns

Dict[str, Dict[str, List[int]]]: Dictionary containing ticker years data, structured as: {ticker: {timeframe: [year1, year2, …]}}

Raises

FileNotFoundError: If the JSON file doesn’t exist
IOError: If there’s an error reading the file

write_data(target_table, dataframe, clean=False)[source]

Write data to database - must be implemented by subclasses.

Return type:: None

read_data(market, ticker, timeframe, start, end, comparison_column_name=None, check_level=None, comparison_operator=None, comparison_aggregation_mode=None)[source]

Read data from database - must be implemented by subclasses.

Return type:: LazyFrame

forex_data.concat_data(data_list=_CountingAttr(counter=27, _default=NOTHING, repr=True, eq=True, order=True, hash=None, init=True, on_setattr=None, alias=None, metadata={}))[source]

forex_data.validator_dir_path(create_if_missing=False)[source]

exception forex_data.TickerNotFoundError[source]: Bases: Exception

exception forex_data.TickerDataNotFoundError[source]: Bases: Exception

exception forex_data.TickerDataBadTypeException[source]: Bases: Exception

exception forex_data.TickerDataInvalidException[source]: Bases: Exception

forex_data.get_histdata_tickers()[source]

Get all available tickers from HistData.com.

Return type:: List[str]

Returns

List[str]: List of all available tickers (e.g., [‘EURUSD’, ‘GBPUSD’, …]).

class forex_data.POLARS_DTYPE_DICT[source]

Bases: object

TICK_DTYPE = {'ask': Float32, 'bid': Float32, 'p': Float32, 'vol': Float32}

TF_DTYPE = {'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32}

TIME_TICK_DTYPE = {'ask': Float32, 'bid': Float32, 'p': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None), 'vol': Float32}

TIME_TF_DTYPE = {'close': Float32, 'high': Float32, 'low': Float32, 'open': Float32, 'timestamp': Datetime(time_unit='ms', time_zone=None)}

forex_data.business_days_data(dataframe)[source]

Remove non-business days data from the input dataframe. Filter out weekends data: saturday and sunday. Use holidays to get list of country holidays. Consider dataframe always have a column named ‘timestamp’ of type datetime.

Return type:: DataFrame | LazyFrame

forex_data.random_date_between(start_date, end_date)[source]

Get a random datetime between two datetime objects.

Parameters:

start_date (datetime) – The start date.
end_date (datetime) – The end date.

Returns:

A random datetime between start_date and end_date.

Return type:

datetime