How to load incrementally my data?

Do you have a large volume of source data to load into your DWH (Data Warehouse)? The incremental loading of this data is adapted to your needs. You can base your increment on any numeric or date column contained in your source data.

By Claire Pierreclos

Noob Hack

You can configure your stage objects to have an incremental load. And this in your ODS as well as in your DWH depending on the project configuration in place.

For this, you need to configure the Custom Property "IncrementalFilterColumn" - uses any field from the source object to load data incrementally (date as increment, number as increment):

Each load will start from the last increment already loaded (here according to the UpdatedAt date field). Several load strategies can be configured to adapt the incremental load to your needs: LastRun, LastExecution, LastExecutionPlan or Fixed.

All the details can be found in our community site.

Pro Hack

You may want to base your incremental load on several columns. For example, your source object contains two logging columns for modifications: CreatedDate and ModifiedDate. The CreatedDate is always filled when you create a new record but the ModifiedDate is only filled if you do an update. So, for a lot of records, no ModifiedDate filled.

You can base your incremental load on a rule like COALESCE(ModifiedDate, CreatedDate).
You need to use an existent term on a stage and add the rule as a term rule.

All the details can be found in our community site.

Hint: If you already are a biGENIUS user and have no access to our community site, please ask the biGENIUS support.