TOPIC: Append Only New Records

I have a table with over 550K records and would like to bring over only the new ones. The table has an auto number field so I only need records where Read.NextRecord > Write.NextRecord.

Sounds easy but I am just not sure how work this in FlowHeater

Hi Niel,

Usually you just switch on then "Insert data" and "Ignore existing records" options on the WRITE side. With this FlowHeater decide by the primary key on the WRITE side if the record exits.

Note: It´s necessary to connect the primary key fields from the READ side to the WRITE side.

In your case the "auto number field" can be a problem! Please post your table schema (READ and WRITE) and I could make a little example for you.

The problem with this method is that FlowHeater must check all 550k records on the WRITE side for existing! This can be take a while.

Forecast for the next Version 2.1.1 coming soon in October) Here you can use dynamical filters like this.

1. Determinate the last ID in a previous SQL statement
2. Use this id in the SQL select statement on the READ side like this
   select * from your_table where id > %ID from WRITE$ side%

Has this enhancement been setup in the new version?

Hi Niel,

Yes this feature is released with the current version 2.2.1. There’s only a short time gap between new versions and the online help translation! Please excuse the delay.

First you have to define a parameter $MAX_ID$. You can find the parameter configuration dialog in the menu Edit->Parameters.

In this parameter the SQL Heater (new in version 2.2.1) stores the result from the SQL statement

Select max(id) from t_import

Note: The SQL Heater runs the SQL statement before the definition starts!

On the READ side you can now use this parameter to build a select SQL statement like this

Select * from t_export where id > $MAX_ID$

I’ve made a brief example; please have a look to the attachment.

This works perfectly, Thanks!