PutSql Loading is slow with Support Fragmented Transactions as True...

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

PutSql Loading is slow with Support Fragmented Transactions as True...

rey26
I have a flow where i am putting data into Teradata database from csv files.I
am reading with csv and using convert record processor with source as
CSVReader and Target as JSON Reader and than JSONtoSql and then Putsql, but
loading is quite slow.

Since in the csv each file , each line is independent of each other , i
though to mark Support Fragmented Transactions as False and it improved  the
load performance.

Can someone explains why its default value is True ?
Does it will have any impact on data?
Whats the exact use of this parameter?




--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: PutSql Loading is slow with Support Fragmented Transactions as True...

Mark Payne
Hello,

That property will basically determine whether pushing the data is transactional or not.
For example, if you want to ensure that all data from your CSV is pushed or none of it is,
then you'll want that to be true. If you're okay with pushing the data in a bit at a time, then
you can set it to false.

Now, that said, the flow that you're using to convert CSV to JSON, then JSON to SQL, then
put SQL is fairly laborious and slow. Instead, I would recommend replacing all 3 ConvertRecord,
ConvertJSONToSql, and PutSQL Processor with a single PutDatabaseRecord processor. Then
you can put the CSV data directly to the database without the need to convert to JSON or SQL,
and this should be faster.

Thanks
-Mark


> On Aug 17, 2018, at 3:02 PM, rey26 <[hidden email]> wrote:
>
> I have a flow where i am putting data into Teradata database from csv files.I
> am reading with csv and using convert record processor with source as
> CSVReader and Target as JSON Reader and than JSONtoSql and then Putsql, but
> loading is quite slow.
>
> Since in the csv each file , each line is independent of each other , i
> though to mark Support Fragmented Transactions as False and it improved  the
> load performance.
>
> Can someone explains why its default value is True ?
> Does it will have any impact on data?
> Whats the exact use of this parameter?
>
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/