In nifi usgae of Evaluate jsonpath processor will it affect performance impact because of attribute creation

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

In nifi usgae of Evaluate jsonpath processor will it affect performance impact because of attribute creation

vikramas
I'm trying to integrate nifi REST API's with my application. So by mapping
input and output from my application, I am trying to call nifi REST api for
flow creation. So, in my use case most of the times I will extract the JSON
values and will apply expression languages.

So, for simplifying all the use-cases I am using evaluate JSONpath processor
for fetching all attributes using jsonpath and apply expression language
function on that in extract processor. Below is the flow diagram regarding
that.

<http://apache-nifi-developer-list.39713.n7.nabble.com/file/t1186/jsonpath_attributes.png>

Is it the right approach because for JSON to JSON manipulation having 30
keys this is the simplest way, and as I am trying to integrate nifi REST
API's with my application I cannot generate JOLT transformation logic
dynamically based on the user mapping.

So, in this case, does the usage of evaluating JSONpath processor creates
any performance issues for about 50 use case with different transformation
logic because as I saw in documentation attribute usage creates
performance(regarding memory) issues.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: In nifi usgae of Evaluate jsonpath processor will it affect performance impact because of attribute creation

Andy LoPresto-2
Hi Vikram,

I replied to your question on Stack Overflow. I’m pasting the same answer here for the list.

> Your concern about having too many attributes in memory should not be an issue here; having 30 attributes per flowfile is higher than usual, but if these are all strings between 0 - ~100-200 characters, there should be minimal impact. If you start trying to extract KB worth of data from the flowfile content to the attributes on each flowfile, you will see increased heap usage, but the framework should still be able to handle this until you reach very high throughput (1000's of flowfiles per second on commodity hardware like a modern laptop).
>
> You may want to investigate ReplaceTextWithMapping [1], as that processor can load from a definition file and handle many replace operations using a single processor.
>
> It is usually a flow design "smell" to have multiple copies of the same flow process with different configuration values (with the occasional exception of database interaction). Rather, see if there is a way you can genericize the process and populate the relevant values for each flowfile using variable population [2] (from the incoming flowfile attributes, the variable registry [3], environment variables, etc.).

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ReplaceTextWithMapping/
[2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Using_Custom_Properties
[3] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables_Window


Andy LoPresto
[hidden email]
[hidden email]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 26, 2019, at 7:51 AM, vikramas <[hidden email]> wrote:
>
> I'm trying to integrate nifi REST API's with my application. So by mapping
> input and output from my application, I am trying to call nifi REST api for
> flow creation. So, in my use case most of the times I will extract the JSON
> values and will apply expression languages.
>
> So, for simplifying all the use-cases I am using evaluate JSONpath processor
> for fetching all attributes using jsonpath and apply expression language
> function on that in extract processor. Below is the flow diagram regarding
> that.
>
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/t1186/jsonpath_attributes.png>
>
> Is it the right approach because for JSON to JSON manipulation having 30
> keys this is the simplest way, and as I am trying to integrate nifi REST
> API's with my application I cannot generate JOLT transformation logic
> dynamically based on the user mapping.
>
> So, in this case, does the usage of evaluating JSONpath processor creates
> any performance issues for about 50 use case with different transformation
> logic because as I saw in documentation attribute usage creates
> performance(regarding memory) issues.
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/