Release 2.4.1

2021-01-21

This is a patch release that fixes a regression affecting data collection of the US Stock minute bundle. Users of the US Stock minute bundle who collected the bundle prior to the release of version 2.4.0 and are now collecting bundle updates using version 2.4.0 are affected by this regression and should update to 2.4.1. Users who first collected the minute bundle using version 2.4.0 are not affected. Users who are only collecting the daily bundle are also not affected.

quantrocket/zipline:2.4.1

• Fix a regression introduced in version 2.4.0 where collecting updates for an existing minute bundle resulted in only the daily updates being synced from the cloud. Background: before version 2.4.0, the stored US Stock bundle config did not specify "minute" or "daily" frequency, since only "minute" was supported. With the introduction of "daily" support starting in 2.4.0, "daily" was incorrectly being assumed as the data frequency for existing bundles that did not specify a data frequency.

Important: After updating to 2.4.1, users should run a "force" update of their minute bundle to sync any missing minute data, which can be done as follows by substituting the correct bundle name:

$ BUNDLE_NAME=usstock-1min
$ curl -X POST "http://houston/zipline/ingestions/$BUNDLE_NAME?force=true"

Release 2.4.0

2021-01-14

Release 2.4.0 includes a number of enhancements related to real-time data, the US Stock dataset, and closer feature parity with Quantopian.

Highlights

• There are two new additions to the Code Library:
  • Quant Finance Lectures. This is the Quantopian Lecture Series updated for QuantRocket.
  • Pipeline tutorial. This tutorial provides an in-depth walkthrough of the Pipeline API.
• It is now possible to collect real-time data for large universes, such as the entire US stock market, by collecting minute or second aggregate data from Polygon.io instead of full tick data. See the usage guide and the updated configuration guidance for using Polygon.io data with Zipline.
• There are several new securities master fields for the US Stock dataset:
  • Sector and Industry. See an example record in the usage guide.
  • Primary share. A new field distinguishes primary from secondary share classes for companies with multiple share classes. See the usage guide.
• You can now query daily data from a minute bundle using the new data_frequency parameter on quantrocket.get_prices and quantrocket.zipline.download_bundle_file. See the usage guide.
• Users who don't want to collect the full US Stock minute bundle can ingest only the daily portion of the bundle for use in Zipline, the Pipeline API, and throughout QuantRocket. See the usage guide for a comparison of the different ways to access the US Stock dataset.
• There are two Python API enhancements for the securities master service:
  • a new convenience function, quantrocket.master.get_securities, queries the securities master database and loads the results into a DataFrame. (Previously, downloading the master file and loading it into a DataFrame were two separate steps.) See the API Reference.
  • The function quantrocket.master.create_universe now accepts a sids parameter that makes it easier to create a universe from a DataFrame without having to write the DataFrame to a CSV. See the API Reference for an example.

Documentation

• A new section of the usage guide documents how to replicate the QTradableStocksUS universe, a popular Pipeline filter on Quantopian.
• A detailed note has been added to the docstring for quantrocket.master.download_master_file and quantrocket.master.get_securities explaining how multiple parameters are combined when querying the securities master. See the API Reference.
• A note has been added to usage guide mentioning that print statements will show up in the detailed logs.

API Changes

• the function quantrocket.zipline.download_minute_file has been renamed download_bundle_file since it can now be used to download minute or daily data, using the new data_frequency parameter. The old function is still available but will print a deprecation warning.

Sample Data

• SPY is now included in the free sample data

Complete 2.4.0 release notes

quantrocket/blotter:2.4.0

• reduce memory usage in PNL calculation when there is a large number of executions

quantrocket/flightlog:2.4.0

• make quantrocket.flightlog.FlightlogHandler a singleton to prevent users from unintentionally logging duplicate messages in their code

quantrocket/fundamental:2.4.0

• better error handling when the IBKR API returns invalid XML indicating no Reuters estimates data is available

quantrocket/master:2.4.0

• add Sector and Industry fields to US Stock dataset. See an example record in the usage guide.
• add a field to distinguish primary vs secondary share class in the US Stock dataset. See the usage guide.
• add a new convenience function quantrocket.master.get_securities, which queries the securities master database and loads the results into a DataFrame. (Previously, downloading the master file and loading it into a DataFrame were two separate steps.) See the API Reference.
• add a sids parameter to quantrocket.master.create_universe to make it easier to create a universe from a DataFrame without having to write the DataFrame to a CSV. See the API Reference for an example.

quantrocket/moonshot:2.4.0

• add new paramater DB_DATA_FREQUENCY which can used for querying daily data from a Zipline minute bundle. See the API Reference.

quantrocket/postgres:2.4.0

• changes the default PostgreSQL log level in order to silence the excessively noisy TimescaleDB logging related to aggregate databases seen in previous versions

quantrocket/realtime:2.4.0

• add support for collecting Polygon.io minute or second aggregates, which facilitates real-time data collection for large universes. See the usage guide, the updated note on database performance, and the updated configuration guidance for using Polygon.io data with Zipline.

quantrocket/zipline:2.4.0

• support for querying daily data from a minute bundle using the new data_frequency parameter on quantrocket.get_prices and quantrocket.zipline.download_bundle_file. See the usage guide.
• support for ingesting only the daily portion of the US Stock minute bundle. This is provided as a convenience for users who don't want to collect the full minute bundle. See the usage guide for a comparison of the different ways to access the US Stock dataset.
• add new US Stock master fields (usstock_Sector, usstock_Industry, usstock_PrimaryShareSid, and usstock_CIK) to SecuritiesMaster Pipeline dataset
• roll-up real-time minute data in live trading to support queries for partial day daily data using data.hist(... '1d'). See related forum post.

quantrocket_trading_calendars:2.4.0

• add a trading calendar for US extended hours which runs from 4 AM to 8 PM: us_extended_hours. This can be used when querying trading hours from the master service or when ingesting a history database with extended hours data into Zipline.

Release 2.3.2

2020-12-21

This is a patch release that fixes a difference in behavior between Zipline backtesting and live trading. See the fuller note below.

quantrocket/master:2.3.2

• allow querying calendar without first collecting listings. This is a minor change to support the QuickStart. See related forum post.

quantrocket/zipline:2.3.2

• In live trading, return real-time data from the previously completed minute, not from the current, partially completed minute.
  • Background: In Zipline backtesting, the most recent data available to strategies is from the previously completed minute, but in live trading, real-time data from the current, partially completed minute was being returned, if available. For example: suppose a Zipline strategy queries data.current(asset, ['open', 'high', 'low', 'close', 'volume']) at 9:32:00. In a backtest, the OHLCV of the 9:31:00 bar is returned, representing the trades that occurred from 09:31:00-09:31:59. In live trading, prior to this update, the same request would return the OHLCV of the 9:32:00 bar, representing trades that occurred from 09:32:00-09:32:59. Since this data was returned only a few seconds after 09:32:00, there wasn't much data to return (because the minute had just started), which often resulted in NaNs. This update aligns the behavior of live trading with that of backtesting so that a query at 9:32:00 always returns the 09:31:00-09:31:59 OHLCV. See related forum post.

Release 2.3.1

2020-12-02

This is a patch release that fixes an issue preventing some Mac users from using the Zipline environment kernel in JupyterLab. If you are not a Mac user or do not use the Zipline environment in JupyterLab, this update will not affect you and is optional.

quantrocket/jupyter:2.3.1

• Fix an issue affecting some Mac users where, after starting the Zipline environment kernel in JupyterLab and importing the zipline module, the kernel would immediately restart.

Release 2.3.0

2020-10-22

Release 2.3.0 includes improvements for teams and financial advisors, and other enhancements and bug fixes.

Highlights

• improvements for teams, including a dedicated documentation section, support for linked license keys, and support for read-only S3 credentials. See the teams documentation
• new dedicated documentation section for financial advisors, and support for placing FA Group/FA Profile orders through Interactive Brokers. See the documentation.
• new endpoint: quantrocket flightlog wait/quantrocket.flightlog.wait_for_message. This function allows you to search for a log message and block until the message appears. This can be useful when you need your code to wait until a background process such as data collection finishes. See the usage guide.
• add continuous_future function to Zipline research environment. See the API Reference.

Documentation

• Document continuous futures in Zipline
• Document how to use Linux programs on your crontab
• Document how to incorporate non-Pipeline datasets in Zipline strategies

Complete 2.3.0 release notes

quantrocket/blotter:2.3.0

• support for placing FA Group/FA Profile orders through Interactive Brokers. See the usage guide.
• fix an issue where only a subset of order IDs might be returned when running Alpaca live and paper trading simultaneously and querying order status with the --open/open_orders=True parameter

quantrocket/db:2.3.0

• add a --region/region parameter to quantrocket db s3config/quantrocket.db.set_s3_config to support pushing to and pulling from S3 buckets in regions other than us-east-1. See the API Reference.
• read-only S3 credentials are now supported and can be used for pulling databases from S3 (without allowing pushing to S3). Intended for teams.

quantrocket/flightlog:2.3.0

• new endpoint: quantrocket flightlog wait/quantrocket.flightlog.wait_for_message. This function allows you to search for a log message and block until the message appears. See the usage guide.

quantrocket/history:2.3.0

• fix an issue reported in a forum post where IBKR data collection failed because ibkr_ConIds were not treated as integers.
• fix an issue where multiple corporate actions occurring for the same security on the same day resulted in the second corporate action being incorrectly adjusted by the amount of the first adjustment factor, in EDI history databases

quantrocket/license-service:2.3.0

• license keys are now encrypted at rest and are obfuscated in display output.
• support linked license keys for teams.

quantrocket/realtime:2.3.0

• fix an issue reported in a forum post where IBKR data collection failed because ibkr_ConIds were not treated as integers.
• automatically re-connect when PostgreSQL connection is disconnected after being idle

quantrocket/zipline:2.3.0

• add continuous_future function to research environment. See the API Reference.
• fix futures contracts being out of order in data.current_chain(). See related forum post.
• modify continuous futures "calendar" roll to roll on RolloverDate, not LastTradeDate. See related forum post.
• fix an issue reported in a forum post where Zipline backtests could fail with KeyError due to stale trading calendars in the calendar cache.
• make REPORTPERIOD and CALENDARDATE Sharadar fundamentals fields available in Pipeline API

Release 2.2.0

2020-09-03

Release 2.2.0 contains Zipline enhancements, performance improvements, usability improvements, and bug fixes.

Highlights

• live trading of end-of-day strategies that use daily data. Previously live trading required the use of minute data. See the usage guide.
• add Zipline pipeline integration for Alpaca easy-to-borrow and IBKR shortable shares datasets. See the usage guide.
• data quality enhancements for US Stock dataset. See Data Corrections below.

API Changes

• The preferred syntax for constructing Zipline pipelines with Sharadar or Reuters data has changed. The new syntax utilizes slicing to provide a cleaner syntax that better aligns with the conventions used on quantopian.com. For example, the syntax SharadarQuarterlyFundamentals.REVENUE is now deprecated in favor of sharadar.Fundamentals.slice(dimension="ARQ", period_offset=0).REVENUE. However, the deprecated conventions will still work. See the usage guide.

Documentation

• a new Zipline example strategy in the Code Library demonstrates live trading of an intraday sell-on-gap strategy

Data Corrections

Corrections have been applied to the US Stock dataset to link together certain price series that were previously split up due to corporate restructurings. For example, in 2015 Google restructured and formed a new parent company, Alphabet. The US Stock dataset previously provided Google's price history under two separate sids: one for the pre-2015 Google and one for the post-2015 Alphabet. These are now linked together under one sid. This also applies to a number of other securities. For additional background, see related forum post.

To update your local copy of the dataset with these corrections, follow these instructions:

US Stock end-of-day dataset

Please drop and re-collect the dataset.

US Stock intraday bundle

Since version 2.1.0, the US Stock Zipline bundle can ingest data either sid by sid or day by day. The sid by sid method is used for initial collection or if there are more than 5 days' worth of updates to ingest. The day by day method is used if there are fewer than 5 days' worth of updates to ingest, as it is much faster in these cases.

The data corrections will be automatically ingested the next time the sid by sid method is used. To force this to happen, you can use the force parameter. This parameter is not included in the client library, so use curl as shown below:

$ BUNDLE_NAME=usstock-1min
$ curl -X POST "http://houston/zipline/ingestions/$BUNDLE_NAME?force=true"

Complete 2.2.0 release notes

quantrocket/blotter:2.2.0

• fix an issue where the blotter would continue to request the order status of orders already rejected by Alpaca

quantrocket/codeload:2.2.0

• add a GIT_KEEP_METADATA environment variable which can be used to automatically load a Git repository when your deployment first launches. (This is an advanced use case.) See the usage guide.

quantrocket/fundamental:2.2.0

• gracefully handle a new situation where the IBKR API may indicate that no Reuters estimates data is available by returning an invalid XML file instead of the usual API error code. See support forum post for background.

quantrocket/history:2.2.0

• improve error handling when you request historical data from IBKR and have not yet collected the security listings from IBKR

quantrocket/jupyter:2.2.0

• Add the QuantRocket version number to the JupyterLab menu bar, with an indicator showing when new updates are available.
• fix a performance issue with quantrocket.fundamental.get_sharadar_sp500_reindexed_like. See forum post for background.

quantrocket/postgres:2.2.0

• update to TimescaleDB 1.7.3

quantrocket/realtime:2.2.0

• improve error handling when you request real-time data from IBKR and have not yet collected the security listings from IBKR

quantrocket/zipline:2.2.0

• live trading of end-of-day strategies that use daily data. Previously live trading required the use of minute data. See the usage guide.
• add Zipline pipeline integration for Alpaca easy-to-borrow and IBKR shortable shares datasets. See the usage guide.
• improve query performance of quantrocket.zipline.download_bundle_file
• add Order.open attribute as a shortcut for checking if an order is open. See API reference.
• The preferred syntax for constructing Zipline pipelines with Sharadar or Reuters data has changed. The new syntax utilizes slicing to provide a cleaner syntax that better aligns with the conventions used on quantopian.com. For example, the syntax SharadarQuarterlyFundamentals.REVENUE is now deprecated in favor of sharadar.Fundamentals.slice(dimension="ARQ", period_offset=0).REVENUE. However, the deprecated conventions will still work. See the usage guide.
• fix an error when trying to load a context file containing a ContinuousFuture in live trading
• fix a bug that could cause the incremental ingestion of daily usstock updates to terminate early if encountering sids with no data for a particular date
• update pandas version from 0.20.1 to 0.22.0 on the zipline service. This matches the pandas version in the "Zipline environment" kernel in JupyterLab.

Release 2.1.1

2020-07-27

This is a patch release that extends Zipline pipeline support to include futures and non-US equities.

quantrocket/jupyter:2.1.1

• update Zipline environment with the changes described under quantrocket/zipline:2.1.1

quantrocket/zipline:2.1.1

• add pipeline support for futures
• fix an issue preventing the use of pipelines with non-US equities. See related forum post.
• automatically derive the pipeline domain from the trading calendar so that a domain need not be specified

Release 2.1.0

2020-07-23

Release 2.1.0 contains a number of enhancements and usability improvements for Zipline strategy development, as well as bug fixes.

Highlights

• This release introduces a new Zipline Research API which greatly improves usability by allowing you to perform much of your Zipline strategy development within the interactive environment of a research notebook. It includes support for running pipelines and accessing the data objects used in Zipline strategies. Along with this change, a Zipline kernel has been added to JupyterLab. See the usage guide.
• A progress meter has been added to Zipline backtests that provides progress and performance statistics during long-running backtests. See the usage guide.
• The new Zipline Intro tutorial in the Code Library walks you through the improved Zipline workflow.
• The runtime when ingesting daily updates of the US Stock 1-minute dataset has been signficantly improved. See the fuller note in the API Changes section.

API Changes

• The runtime for collecting daily updates of the US Stock 1-minute bundle has significantly improved. Previously, collecting an update took several hours but should now take only 10-15 minutes per day if updated daily (longer if collecting multiple days' updates). To take advantage of this speed improvement, you must drop and re-ingest the entire bundle. We regret this inconvenience, but previously ingested bundles did not store enough metadata to take advantage of the newly redesigned ingestion strategy. (Background: Previously, data was ingested by syncing one security at a time from the cloud to your local deployment. On updates, only the diff needed to be synced but determining the diff still took time. Now, if the Zipline service determines it will be faster, it will utilize an alternate strategy of loading individual daily update files, that is, ingesting data day by day instead of sid by sid. This is signficantly faster when you only need to ingest a day or a few day's worth of updates.)
• In Zipline strategies, commissions and slippage are now disabled by default. See the usage guide for reasoning and examples. The examples are based on the previous default behavior and can be used by users who wish to preserve the previous default behavior. (Note for futures traders: the dictionary of exchange fees by root symbol in the linked example only includes a subset of root symbols for simplicity. The full list of exchange fees by root symbol that was previously used for default futures commissions is available in GitHub.)
• In Zipline strategies, storing asset objects to context in the initialize() function is no longer supported and will throw an error. Instead, if you need to store asset objects to context, please do so in before_trading_start (or another function called after initialize). Storing asset objects to context in initialize() causes the asset objects to become stale in live trading because the asset objects are always loaded from the stored context instead of being re-instantiated each day. Daily re-instantiation of asset objects is necessary to ensure that the asset's metadata (particularly as concerns the asset's end date) stays up-to-date. Otherwise, assets will not be tradeable in live trading because Zipline will (incorrectly) think they are stale. We are choosing to immediately enforce this change instead of printing a deprecation warning due to the potential adverse effect on live trading.

The following is no longer allowed:

def initialize(context):

    context.aapl = algo.sid("FIBBG000B9XRY4")
    ...

Replace it with this:

def before_trading_start(context, data):

    context.aapl = algo.sid("FIBBG000B9XRY4")
    ...

Documentation

• Added Zipline Intro tutorial to Code Library.
• Added API reference sections for Alphalens, Pyfolio, and quantrocket-trading-calendars, and improved API reference for Zipline.

Complete 2.1.0 release notes

quantrocket/account:2.1.0

• fix a bug where checking your portfolio before collecting securities master listings for your broker caused an error

quantrocket/blotter:2.1.0

• improve reliability of Alpaca execution monitoring by always requesting an overlapping buffer of executions in case execution records arrive out of order. See related forum post.

quantrocket/history:2.1.0

• fix an issue caused by a change in the IBKR API behavior which resulted in missing dates when collecting 1-day bars from IBKR. See related forum post.
• query results will now respect the use of a datetime string (YYYY-MM-DD HH:MM:SS) for the start_date or end_date parameter when querying an intraday history database. Previously, only the date part was extracted and used in the query, even if a datetime string was passed.

quantrocket/ibg:2.1.972.0

• improve error handling during rollback when setting IBKR credentials fails. See related forum post.

quantrocket/jupyter:2.1.0

• add Zipline Research API, with support for running pipelines and accessing data objects in interactive research. See the usage guide.
• add a "Zipline environment" kernel to support using the Research API for Zipline. See the usage guide.
• add example templates to JupyterLab Launcher for Zipline strategies, Moonshot and Zipline allocation files, countdown crontabs, rollover rules configuration files, and IB Gateway permission files. Previously Launcher templates were only available for Moonshot and MoonshotML strategies.

quantrocket/postgres:2.1.0

• update to TimescaleDB 1.7.2

quantrocket/realtime:2.1.0

• fix an unhandled exception that occurred when you try to drop ticks from a real-time database that has no data

quantrocket/zipline:2.1.0

• add progress meter to Zipline backtests. See the usage guide.
• add endpoint to check bundle config. See API reference.
• use multithreading during initial ingestion of US Stock 1-minute data bundle, reducing runtime from 15-18 hours previously to 12-15 hours currently.
• improve runtime when ingesting daily updates of the US Stock 1-minute dataset. See the fuller note in the API Changes section above.
• ingest data one security at a time when ingesting from a daily history database. Previously the entire database was ingested in one go which failed on large databases. See related forum post.
• default to applying no commissions or slippage on backtests. See the usage guide for reasoning and examples. The examples are based on the previous default behavior and can be used by users who wish to preserve the previous default behavior.
• fix an issue where the presence of illiquid securities with no price data caused an error when querying the US Stock 1-minute data bundle without specifying sids or universes. See related forum post.
• fix an issue where ingestion failed if a security had a NULL symbol. See related forum post.
• improve error message when you query a Zipline bundle using a date range that is not present in the trading calendar
• fix an issue where ingesting a history database failed if you hadn't previously collected listings from Interactive Brokers
• provide a helpful error message if user tries to ingest two distinct futures chains (for example that trade on different exchanges) that use the same root symbol

Release 2.0.0

2020-06-09

Highlights

• New data: this release includes numerous new data providers and datasets, including end-of-day, intraday, real-time, and fundamental data. See the Data Library for available datasets.
• Zipline live trading. See the usage guide.
• New broker: Alpaca. See how to connect and how to place and manage orders.

API changes

QuantRocket version 2 is a major product upgrade from version 1. Due to the scale of changes, there are many breaking API changes that version 1 users should be aware of. See the migration guide.

Complete 2.0.0 release notes

quantrocket/account:2.0.0

• support for querying account balances and portfolios from Alpaca
• store all available exchange rates from ECB
• derive GBX exchange rate from GBP

quantrocket/blotter:2.0.0

• support for Alpaca. See usage guide
• fix a SQL error that happens when you query summary pnl with a date range only and no other parameters

quantrocket/db:2.0.0

• encrypt S3 credentials at rest

quantrocket/fundamental:2.0.0

• new data: Sharadar insider holdings
• new data: Sharadar institutional ownership
• new data: Sharadar SEC Form 8-K
• new data: Sharadar S&P 500 constituents
• new data: Alpaca easy-to-borrow stocks

quantrocket/history:2.0.0

• new data: US Stock prices (end-of-day and intraday)
• new data: EDI global end-of-day stock prices
• new data: Sharadar end-of-day fund prices

quantrocket/ibg:2.0.972.0

• encrypt IBKR credentials at rest

quantrocket/ibgrouter:2.0.0

• renamed service from launchpad to ibgrouter

quantrocket/jupyter:2.0.0

• update to JupyterLab 1.2.7
• add support for a .bashrc file. See usage guide
• document attaching VS Code to the jupyter container

quantrocket/license-service:2.0.0

• add endpoint to connect to Alpaca
• add endpoint to connect to Polygon.io
• add endpoint to connect to Quandl

quantrocket/master:2.0.0

• new vendor: collect Alpaca listings
• new vendor: collect EDI listings
• new vendor: collect FIGI listings
• new vendor: collect US Stock listings
• consolidate securities master file from multiple vendors

quantrocket/postgres:2.0.0

• update to PostgreSQL 12 and TimescaleDB 1.7.1

quantrocket/realtime:2.0.0

• new real-time data provider: Polygon.io
• add ability to drop ticks from tick databases. See usage guide
• automatic compression of real-time databases

quantrocket/satellite:2.0.0

• support for custom setup scripts to customize environment

quantrocket/theia:2.0.0

• update to latest version of Eclipse Theia

quantrocket/zipline:2.0.0

• Zipline live trading. See the usage guide
• redesign data collection API to mirror the API of the history service. See usage guide
• support for incremental ingestion of history databases. See usage guide
• Add pipeline support for Sharadar fundamentals, institutions, S&P 500 constituents, Reuters financials and estimates, and securities master. See the usage guide