Incremental Fetch for RestAPI

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Incremental Fetch for RestAPI

Manee
Hi Team,

I am new to NiFi .I have a task like .We need to fetch a data from  Client
API and stored into Postgresql
My Flow ;

InvokeHTTP -->>--SplitJson-->>--EvaluateJsonPath
-->>ConvertJSONTosql--->>PutSql

This is my flow is working fine but i need to make this as a incremental
fetch from API ..whenever client API made changes it should reflect in our
databases .How i can process for incremental fetch in API call ?.Please
guide me to fix this problem .


Thanks in Advance,
Manikandan K






-----
Thanks,
Manee
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Thanks,
Manee
Reply | Threaded
Open this post in threaded view
|

Re: Incremental Fetch for RestAPI

Koji Kawamura-2
Hi Manee,

It depends on the Client API on how to tell what the next response
data set should be.
That may be an additional query parameter such as last fetch
timestamp, or something like HTTP etag header in most APIs.
You can pass FlowFiles to InvokeHTTP to tell such parameters.

Also, I recommend using Record processors over splitting dataset to process.
Because Record processors work more efficiently and the data set unit
will be more meaningful in your case.

I think your flow would be something like:

InvokeHTTP (Assuming the API result contains value to make next
incremental request)
 -> Do some JSON transformation (QueryRecord, UpdateRecord or
JoltTransformRecord)
 -> PutDatabaseRecord
 -> Then connect success back to InvokeHTTP to fetch next dataset

Hope this helps.

Thanks,
Koji

On Mon, Nov 12, 2018 at 9:06 PM Manee <[hidden email]> wrote:

>
> Hi Team,
>
> I am new to NiFi .I have a task like .We need to fetch a data from  Client
> API and stored into Postgresql
> My Flow ;
>
> InvokeHTTP -->>--SplitJson-->>--EvaluateJsonPath
> -->>ConvertJSONTosql--->>PutSql
>
> This is my flow is working fine but i need to make this as a incremental
> fetch from API ..whenever client API made changes it should reflect in our
> databases .How i can process for incremental fetch in API call ?.Please
> guide me to fix this problem .
>
>
> Thanks in Advance,
> Manikandan K
>
>
>
>
>
>
> -----
> Thanks,
> Manee
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/