Signal Tester
Overview
Attention!
The Signal Tester module is available for Enterprise Services clients only. Trader/Professional Services clients may purchase the module from the Tool Store.
Scientists working on The Manhattan Project used Monte Carlo simulations for risk analysis before putting their ideas into practice. They had a limited supply of uranium and wanted to explore all likely outcomes before depleting resources.
You can use Monte Carlo simulation to preserve your own resources and get a look at investments and your strategy before you act. The signal tester allows you to move beyond averages and towards thinking in terms of probabilities and ranges of possible outcomes.
And, you don’t have to be a scientist to understand and use it!
Any custom criteria can be tested, generating a report of the price action that occurs before and after the criteria is met. This can be done for a single code, a custom universe or an entire exchange.
Once the report is generated, the numbers will help you see the usefulness of the criteria. Also, it is quick and easy to explore variations of the criteria. And, to get a full picture, you can peer inside the averages into the probabilities along any point during the test period.
The Signal Tester takes every signal over the specified time period and calculates the average returns X periods after each signal.
Unlike the Backtester there are no exit signals or capital/position size constraints - the results are based purely on the number of periods after each signal.
The Trade Tester is very similar to the Signal Tester, but it allows you to have exit signals.
Videos
Click on the images below to watch three videos on using the Signal Tester presented by Mathew Verdouw, CMT, CFTe:
Signal Testing Part 1
Signal Testing Part 2
Signal Testing Part 3
Actions & Properties
Actions
- Generate Report: Only available after a Signal Test has been ran. Will produce a PDF report of the Signal Test which will include any commentary you have added via the Properties section.
- Copy to Clipboard: Creates an image of the completed Signal Test (either as a Bitmap or Vector) which can be pasted into other applications such as Paint or MS Word.
- Restore Default Settings: If the default settings of the Signal tests has been adjusted, left clicking this action will restore the defaults back to those that were originally installed with Optuma.
- Save Settings as Default: If you have setup a new Signal Test and adjusted the settings used, and you want all new tests to be setup with the same settings left click this action.
Properties
Note
All properties with a * are required to be set by the user to run a Signal Test.
- * Script: Allows the user to set the script the Signal Test will run against the selected symbol(s). When clicked, the Script Manager will open allowing the user to select an existing script, or enter in a new custom script. To learn about scripting see the free video courses here. If you need help with scripting contact us for a one-on-one consulting session.
Security Selection
- Comparison Index: Allows the user to select a code (usually an Index) which will display alongside the Signal Test results to be used as a comparison to measure performance.
In the example above the Orange Band represents the performance of the XJO over the same time period.
- * Signal Test On: Allows the user to select running the Signal Test on a Single Code, or Multiple Codes.
- * Codes to Test: Used to enter the Single Code, or Select a list of codes (via the Security Selector) the Signal Test will use.
- * Date Range: Used to select the length of time the Signal Test will run over when looking for the selected Criteria to be met. Options range from Last Day to Everything (Uses Last 10 Years by Default).
- * Data Timeframe: Sets the timeframe the criteria is used on (1 Day, 1 Week, 1 Month, etc). For Example, If when manually analysing a chart you apply your criteria / tools to a 1 Week chart, you would set the Timeframe in the Signal Tester to 1 Week.
Multi Code Signal Tests
Multi code scans include an additional results tab along the top of the Signal Test once the test has been ran.
Clicking on the Code Results tab will display the signal tests statistics on a code-by-code level.
You can adjust the order to ascending / descending by clicking on the header of any of the columns in the list.
Grouping
By default, the results of the Signal Test will be condensed into a single line (with high low bands). The Grouping option allows you to display results for individual years, or other custom groups (such as GICS Sectors, if you have Bloomberg data).
This gives you the ability to see the results of price movement after the signal has been found through different market conditions (Bull, Bear, Sideways, etc).
Calculation Rules
- Average Style: Allows the user to select how the combined results are averaged. The options are Median (default) or Mean. The Mean is the basic average of the results. The Median value takes the numeric value separating the higher half of a sample from the lower half to produce the average line.
- Periods Before: Allows the user to select how many bars before each signal are displayed on the Signal Test results. Set to 30 bars by default.
- * Periods After: Allows the user to select how many bars after each signal are displayed on the Signal Test results. Set to 252 bars by default (a typical trading year for equities).
Results Display
- Comparison Index: Allows you to display the statistics of the selected comparison index along side the statistics of the selected Signal test for direct comparison.
The Comparison Index allows you to select other signal tests as well. This allows you to directly compare the statistics / results of two signal tests side by side.
- Draw Style: Allows the user to select between Line (default) which is a composite of all the results averaged out as a Median or Mean, or individual components (a line for each signal).
- Show High Low Bands: Allows the user to display / hide the upper and lower bands on the results graph.
In the above example, High Low Bands have been turned off.
- Auto Scale: The scale on the Signal Test results page will automatically recalculate to show the Max / Minimum values. If this option is disabled the scales can be manually set.
Profit Analysis
A graph of the Profit Analysis is displayed below the main Signal Test graph.
- Profit Factor Display Style: Allows the user to adjust the type of graph displayed for the Profit Analysis. The options are Distribution (default) and Ranked.
- Logarithmic Profit: Allows the user to set the display of the X Axis of the Profit Analysis chart to use a Logarithmic scale.
An example of the Profit Analysis graph set to Log Scale
- Stop Loss: Add a Stop Loss (as a percentage value) to the Signal Test. The Signal Test will automatically recalculate when this value is entered / adjusted.
- Profit Target: Add a Profit Target (as a percentage value) to the Signal Test. The Signal Test will automatically recalculate when this value is entered / adjusted.
Monte Carlo Simulation
Monte Carlo simulations are used to model the probability of different outcomes in a process that cannot easily be predicted due to the intervention of random variables.
By analysing historical price data from a random selection of Signals, you can determine the drift, standard deviation, variance and average price movement for a random selection of signals within the total number of results generated. These are the foundations of a Monte Carlo simulation.
- Simulation Display Style: Allows the user to adjust the type of graph displayed for the Monte Carlo Simulation. The options are Distribution (default), Ranked and Win / Loss Count.
- Logarithmic Simulation: Allows the user to set the display of the X Axis of the Monte Carlo Simulation chart to use a Logarithmic scale.
An example of the Monte Carlo Simulation graph set to Log Scale
- * Number of Tests: Sets the number of tests used to create the random samples used to generate the Monte Carlo Simulation. Setting the value higher will improve the statistical results generated by the simulation (Mean and Median returns, etc). Has a minimum value of 100.
- * Signals per Test: Sets the number of results per Test used to create the random samples to generate the Monte Carlo Simulation. Has a minimum value of 2.
Report Options
The properties contained in this section are used when generating a PDF report of the selected Signal Test.
- Description: Enter a general description of the signal test which will display at the top of the 1st page of the PDF report.
- Returns Commentary: The text entered here will display under the Signal Distribution graph at the bottom of the 1st page of the PDF report.
- Distribution Commentary: The text entered here will display under the Profit Analysis Distribution and Ranked graphs on the second page of the PDF report.
- Monte Carlo Commentary: The text entered here will display under the Monte Carlo Simulation Distribution, Ranked and Win / Loss Count graphs on the second page of the PDF report.
- Include Script: Tick this checkbox to include the details of the script used to generate the Signal Test. If selected the script will display on the top right of the 1st page of the PDF report.
- Report Header Colour: Allows the user to adjust the colour of the header bar displayed at the top of the 1st page of the PDF. By default, dark blue will be used.
- Report Header Logo: Allows the user to replace the default Optuma Logo displayed on the top right corner of the 1st page with a custom logo image (we recommend using a PNG image file). The logo image size will be adjusted automatically to fit within the report header.
Statistics
A table of statistical values will be displayed to the left of the Profit Analysis graph of each Signal Test. These values include:
- Probability of Gain (%): The percentage of signals that saw an increase in value once triggered.
- Probability of Loss (%): The percentage of signals that saw a decrease in value once triggered.
- Mean Return: Displays the value of the Signal Tests average line (using the Mean calculation method).
- Median Return: Displays the value of the Signal Tests average line (using the Median calculation method).
- 80th Percentile: Shows the 80th percentile value. For example, if the 4th highest return in a sample of 20 Signals is 5%, 80% of the values will be lower than 5%.
- 20th Percentile: Shows the 20th percentile value. For example, if the 4th lowest return in a sample of 20 Signals is -5%, 20% of the values will be lower than -5%.
- Skewness: Skewness describes asymmetry from the normal distribution in a set of statistical data. Skewness can come in the form of “negative skewness” or “positive skewness”.
- An asymmetrical distribution with a long tail to the right (higher values) has a positive skew.
- An asymmetrical distribution with a long tail to the left (lower values) has a negative skew.
- Kurtosis: The Kurtosis value describes the trend / curve across the Signal Test / Profit Analysis and Monte Carlo Simulation graphs. A high kurtosis value indicates a chart with fat tails and a low, even distribution, whereas a low kurtosis indicates a chart with skinny tails and a distribution concentrated toward the mean.
- Standard Deviation: The Standard Deviation of all individual component results for the selection period.