In this chapter we describe the possible inconsistencies regarding the historical file and how to deal with them.
Introduction
Definition
Inconsistencies are warnings and errors that Trading Conceiver found in the original file.
You can look at them in the text area contained in this pane.
Two Categories
When parsing an historical data file, two main categories of warnings and errors can be found:
Issues stemming from erroneous parsing.
Real issues of the original file.
In the former case, you need to refine the settings in the Format pane,
to give Trading Conceiver exact information about the file structure and number format.
In the latter case you need to deal with the faultiness of the file.
Faulty Data Are in the Nature of Things
No matter how authoritative and reliable the historical data provider might be, historical data can definitely be faulty, contain inconsistent or missing data.
For example, the open price might not have been recorded and appears as 0 or with a wording such as null.
As another example, the high quote might be not really the highest value and is lower than the close price.
Rules
So we should give for granted that in an historical file there could be imperfections.
And yet, we can't dismiss the whole file for that, we must deal with them.
Trading Conceiver offers various methods, or rules, to apply to the file in order to make it usable anyway, despite its shortcomings.
These rules allow to handle and manage inconsistencies, and to work even with faulty data.
Arbitrariness
No matter how sound and reasonable these rules might be, by applying them, we are introducing something arbitrary.
Whatever we do, we are altering the reality recorded in the file, dismissing or modifying data in subjective ways.
So keep this in mind when applying these rules.
There is no magic that can retrieve wrong or missing data in the file.
Threshold on the Number of Changes to the File
As a consequence of the previous point, Trading Conceiver lets you apply rules a maximum number of times per file.
The purpose is to make usable some data with imperfections, to take care of just a few problematic lines, not to invent or fabricate contrived data.
This is why if the rules should be applied a number of times beyond a certain threshold, Trading Conceiver will disable the button
Apply
The Original File Is Not Modified
Remember that only the internal representation of the file will be affected by the application of the rules, while the historical file will not be changed in any way.
Historical File Rules Window
Rules are methods or routines that can be applied to an historical file in order to amend its inadequacies or to simply inform the user about something unusual.
Open the Historical File Rules Window
To see the available rules, press the button
Edit
As an alternative, you can open it through the menu:
Tools → Options → Historical file
The Options tab, dedicated to the rules, pops up.
Dismiss Lines with Undefined Date/Time
By checking this option, lines where date and/or time are undefined are ignored.
This might be the case in the following example, where one entry is missing in the file and this fact is reported with a wording such as:
date1, close1
null
date3, close3
The line with the null is a faulty line and will be dismissed.
Dismiss Lines with Unparsable Numbers
An unparsable number is a number written in such a way that it is incomprehensible.
By checking this option, lines where even just one of the following is unparsable are dismissed:
Open
High
Low
Close
Volume
This might be the case in the following examples, where one entry for the close is missing in the file and this fact is reported as:
The number with the - or the null is unparsable and the whole line will be dismissed, even if the other data on the same line are fine.
Values at 0
The values
open, high, low, close, volume
if they exist, should never be 0. If they are, there are three options to deal with them, selectable through dedicated radio buttons:
Leave them as they are.
Dismiss the line.
Linear interpolation.
Leave Them as They Are
No actions will be taken if a value is 0.
For instance, this could be fine if a lot of 'volume' values are 0, which happens for some quotes dating way back,
when volume was not recorded and the volume appears only from a certain date onward.
In such a case, you can select this option and either just not use algorithms involving volumes or use them knowing they will be active only when volume is present.
Dismiss Lines with Values at 0
Lines where at least one of the values corresponding to the selected checkboxes are 0, will be ignored.
In a situation like in the previous paragraph, where the volume values are present only from a certain point onward,
you might unselect the volume checkbox and select all the others.
Linear Interpolation
Values at 0 will be substituted by computing a linear interpolation.
The linear interpolation is a straight line.
The left point of the line is found by looking back, until a non 0 value is found for the same field (e.g. open).
The right point of the line is found by looking forward, until a non 0 value is found for the same field.
All the values at 0 will be given the value of the straight line, i.e. the linear interpolation.
Adjustment
The computed value will then be adjusted to make it coherent with the other ones.
E.g. the 'low' value can't be higher than open, high or close.
So if the interpolated value doesn't satisfy that relation, it will be set at the minimum between open and close.
Force High to Max and Low to Min
If the high of the price is not really the greatest value and the low is not the least, by checking this option they are forced to be.
So e.g. high is forced to be the maximum between open, low and close.
When it is applied
This option is applied to 'high' only if the 'high' field exists, i.e. the correspondent spinner has a value other than 0:
High spinner ≠ 0
and it is
high ≠ 0
because it is conceived to deal only with light incongruities of data.
Analogously for low.
Volume Gap from Price
This setting helps to pinpoint the right volume column.
A warning is signaled if the volume has a value similar to the price;
more precisely if the volume differs from the price less than the percentage indicated in the spinner.
This is so because usually the volume has a very different value from the price.
However, it is perfectly legitimate to have the volume in the same range of the price.
This rule option differs from the others in that the warning is signaled in the volume spinner, not in the Inconsistencies pane.
To Omit the Check
If you want to omit this check, set the spinner to 0.
Split Threshold
Split and Reverse Split
A split happens when a stock is split in two, each having half the value as before.
The ratio can be any, not necessarily 1:2.
It is usually done when the stock price gets too high.
On the contrary, a reverse split happens when two stocks are merged into one, having the double of the previous value.
The ratio can be any, not necessarily 2:1.
It is usually done when the stock price gets too low.
When this happens, the price seems to halve or to double, but it is only a consequence of the split or reverse split, respectively.
In actuality, the price remains the same, save for the normal change from one entry to the other.
Trading Conceiver Informs You
As the abrupt change of price in case of splits can be misleading, Trading Conceiver gives you a warning in the Inconsistencies pane.
The Spinner Value
The value in the spinner is the threshold used to identify probable splits and reverse splits.
To see the available rules, press the button
By checking this option, lines where date and/or time are undefined are ignored.
This might be the case in the following example, where one entry is missing in the file and this fact is reported with a wording such as:
An unparsable number is a number written in such a way that it is incomprehensible.
By checking this option, lines where even just one of the following is unparsable are dismissed:
No actions will be taken if a value is 0.
For instance, this could be fine if a lot of 'volume' values are 0, which happens for some quotes dating way back,
when volume was not recorded and the volume appears only from a certain date onward.
In such a case, you can select this option and either just not use algorithms involving volumes or use them knowing they will be active only when volume is present.
Lines where at least one of the values corresponding to the selected checkboxes are 0, will be ignored.
In a situation like in the previous paragraph, where the volume values are present only from a certain point onward,
you might unselect the volume checkbox and select all the others.
Values at 0 will be substituted by computing a linear interpolation.
The linear interpolation is a straight line.
The left point of the line is found by looking back, until a non 0 value is found for the same field (e.g. open).
The right point of the line is found by looking forward, until a non 0 value is found for the same field.
All the values at 0 will be given the value of the straight line, i.e. the linear interpolation.
If the high of the price is not really the greatest value and the low is not the least, by checking this option they are forced to be.
So e.g. high is forced to be the maximum between open, low and close.
This setting helps to pinpoint the right volume column.
A warning is signaled if the volume has a value similar to the price;
more precisely if the volume differs from the price less than the percentage indicated in the spinner.
This is so because usually the volume has a very different value from the price.
However, it is perfectly legitimate to have the volume in the same range of the price.
This rule option differs from the others in that the warning is signaled in the volume spinner, not in the Inconsistencies pane.
If you edit the historical file options, make them effective by pressing the 
Rules can be applied by pressing, in the Inconsistencies pane, the button
See the user interface chapter for recommendations.