Backtesting 12+ Years Data in #MT4 Using Minute Bars #quant

Source(s):  4xTrader, Forex Tester Software
Date: 7/11/2013
by: Admin

Here is a step-by-step guide to setting up a decent backtesting environment in MT4 (MetaTrader 4) using 1 minute GMT (Greenwich Mean Time, non-DST) bars from the Forex Tester Software site.  The data set goes back to 1/1/2001 (12.5 years of data), consists of 1 minute bars, is regularly updated and can be downloaded for FREE here.

1. Freeze and/or Record the Bid/Ask Spread
Download the SpreadIndicator.mq4 indicator and compile/install it on your MT4 platform.  This will show you the current Bid/Ask spread in points.  For five-digit brokers, this will be a two digit number, such as 37 (or 3.7 pips) for EURJPY.

Note that when running the backtester in MT4, the MT4 Strategy Tester will default to using the current spread (e.g. last known spread of any given currency pair, at the exact moment the backtest was started).  Spreads can vary based on trading times, especially at market close on weekends, where the spreads will widen significantly and cause backtest results to be way off, or even produce errors within the MT4 Strategy Tester when you try to run the backtest with a really wide, invalid spread.  Therefore, if backtesting on weekends, make sure to key in the spread manually, or take your trading platform offline before (say an hour or so) market close to freeze a reasonable spread price.

The Bid/Ask spread can have a large effect on backtest results so make sure you lock in a good one based on actual quotes or key in a valid one manually!

Record what the normal spreads are for the currency pairs you will be backtesting, so that you can key this information in manually when backtesting.  This will be a two-digit number for 5 digit brokers and a one digit with decimal number for four-digit brokers.

Alternatively, you can freeze all current spread prices by taking the MT4 trading platform offline at a time when spreads tend to be at normal levels.

2.  Take MT4 Offline
In the MT4 Trading Platform you will use for backtesting, take the platform offline by right clicking your MT4 backtesting account in the Navigator panel, and choosing the delete option.

Navigator panel can be activated with the Ctrl+N key or via the View menu.

This does not harm your system, as the next time the platform is re-started, MT4 will first prompt you to open a new [demo] account (which you can cancel out of) and next prompts you for your existing account number and login password information (which you can key in or cancel out of as well).  Just make sure you have your original account number and login password information stored somewhere to be able to key in the information later on.

3.  Delete Existing History Data
Navigate to the MT4 installation folder ...\history sub-directory and delete all *.hst files.  If you are using Windows Vista, you might need to click "Compatibility Files" to be able to see these files.

4. Download 1 Minute Bar Data Files from Forex Tester Software
In your web browser, navigate to http://www.forextester.com/data/datasources.html and download whichever data set(s) that suit your backtesting needs.  Unzip the file(s) to a known location.

5. Re-Start MT4 and Import History Data
Restart your MT4 backtesting platform and cancel out of the prompt asking you if you want to create a new demo account.  Also, cancel out of the prompt asking you to log in with your account and password from the original account you delete from Navigator in step 2.

From the Options dialog box, which can be activated by pressing Ctrl+O or from the MT4 Tools menu, click on the Charts tab and change the max bars in history and max bars in history fields from the default values to eight 9s, e.g. 999999999.

Activate by pressing Ctrl-O or from Tools|Options

Activate History Center, which can be launched by pressing the F2 key or opened from the Tools menu. Expand the tree view under the currency pair you would like to import backtest data to. Double click on the 1 minute timeframe to make sure there is not any existing data.

Hit F2 to activate History Center, or go to Tools|History Center to open

If there is existing history data it means that when you deleted the *.hst files in step 2. Delete History Data one or more *.hst files were not removed.  It is possible that "Compatibility Files" in Windows Explorer was not selected and caused the *.hst files to be invisible due to file permissions.  In any case, all is not lost.  Instead, just double click each timeframe and manually highlight/delete all existing bar data for each currency pair you are interested in backtesting until you are sure there is no more data left.  This method is just slightly more time consuming than deleting the *.hst files themselves.

Click on the Import button to launch the Import dialog.

Import Dialog: Make sure Skip 1 columns and 1 rows is set as appropriate.

From the File field, use the Browse button to navigate to the unzipped 1 minute bar file you downloaded in Step 3. Download 1 Minute Bar Data Files from Forex Tester Software.  Note that once you have selected the import file, the screen will freeze for a while.  Just wait for MT4 to recover.

Now set the Skip fields columns and rows to 1 column and 1 row respectively, as circled in red above.

Click OK to import the data.

Repeat this process for all currency pairs you would like to backtest.  Note that subsequent launches of the Import dialog will be slow, as it defaults to loading whatever previous import file you had selected previously.

Once you are done importing history files, shut down MT4 and wait about a minute after the platform has been shut down to allow for the history file(s) to be imported to completion properly.  Do not launch MT4 yet...Go to step 6.

6. Convert the 1 Min Bar Data to Other Timeframes
The 4xTrader site provides a good tool which I have personally used and tested, which converts the 1 minute bar data in to other timeframes.  Download Auto_converter.ex4 here and copy it to your ...\experts\scripts\ folder.  Unfortunately, no source code was provided so I might need to update this article to provide a substitute if the .ex4 file stops working for whatever reason.  In case 4xTrader goes down, I have saved a backup copy of their Auto_converter.ex4 script here.

Now launch MT4, and again cancel out of any prompts that ask you to create a new [demo] account or log in with your existing account number and password.
Create a 1 minute chart with the currency pair whose 1 minute bar data you want to convert.
From Navigator under Scripts, drag the Auto_converter script on to the chart.  Be patient, the script should progressively show it is "Running" for 5, 15, 30, 60 (hourly), 240 (4 hour) and then 1440 (daily) bars.

Example: Auto_converter in middle converting 1 minute to 15 minute bars

Wait until the script completes and displays "Done..." on the screen.
Repeat the process with other currency pairs you are backtesting.

The 4xTrader site recommends not running backtests with data outside of the dataset provided by Forex Tester software (so for the latest dataset, the latest date is 6/30/2013).  However, since my broker's server is set to GMT, I am not too concerned about mismatched bars.

I am still investigating tick-based backtesting in MT4.  Currently my strategies all operate on the 4 hour or higher timeframes, but if I ever decide to write a scalping EA, the tick-based backtesting will become a necessity.

The 4xTrader site which this article is sourced from recommends going to Birt's EA Review|Tick data article for information on tick-based backtesting.  I tried to run the free tools but without much luck.  Birt does have a paid suite of tools for $90 bucks with a free trial period and money back guarantee.

I am currently considering evaluating TickStory's FREE tick downloading/backtest software for MT4 and will provide updates as I discover more details.

Editor's Note:  Here is my current ZuluTrade forwards testing performance for 4 different trading robots running on 4 or more currency pairs on the same demo account.  My previous forwards test bombed out due to over-leverage plus I tried some manual trades as an experiment which blew up the account (not the EA's fault). I re-started the forwards test with 1/4 leverage on each robot.