|
<< Click to Display Table of Contents >> ETL > 14.0 > Implementation Guide > Tutorial and implementation guide Optional and required tables |
In many places, it is possible to define a source table or a source step for operations (for example a source table for map step, source tables for join/enrich operations etc.).
Unless otherwise specified, the source is by default optional. This means that the following two examples:
{
"id": 1, // unique ID
"type": "map",
"name": "Description of the step",
"description": "Long description of what this step does",
"source": "Sourcetable",
"columns": {
// a dictionary of columns
[...]
},
"target": "The name of the output table"
}
or
{
"id": 1, // unique ID
"type": "map",
"name": "Description of the step",
"description": "Long description of what this step does",
"source": 1, // step id
"columns": {
// a dictionary of columns
[...]
},
"target": "The name of the output table"
}
will not fail if the table Sourcetable or table created by step 1 do not exist.
To mark a table or a step as "required", use the object notation in the following form:
{
"id": 1, // unique ID
"type": "map",
"name": "Description of the step",
"source": {
"step": "Sourcetable",
"required": true
},
"columns": {
// a dictionary of columns
[...]
},
"target": "The name of the output table"
}
or
{
"id": 1, // unique ID
"type": "map",
"name": "Description of the step",
"source": {
"step": 1, // step id
"required": true
},
"columns": {
// a dictionary of columns
[...]
},
"target": "The name of the output table"
}
By setting the property required to true, the ETL engine will check if the table exists before starting the execution.
The following rules apply:
•If the source is a required table and the table does not exist in the source, the process will throw an exception before it is even started.
•If the source is a required wildcard, then at least one table matching the pattern must exist in the source, otherwise the process will throw an exception before it is even started.
•If the source is a required step, then the referenced step must produce a table. Some steps will not produce any output if their dependencies are not satisfied. For example, if your step transforms the data from a filter step, then it implicitly means that the source table for that filter step must exist, otherwise the filter step will not produce any value. More information about linking the steps together can be found in the following chapter: Chaining steps.