Need help in designing the solution.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Need help in designing the solution.

Irfan Basha Sheik
Hi,

I have a use case where "I am reading bunch of rows from a database table,
apply some rules(eg. amount > 100) and add additional columns depending on
the rules that got satisfied, and then store the updated rows into elastic
search".

The approach I have in mind at the moment is :

1. Use "EvaluateJsonPath" processor to convert "content" into "attributes"
2. Use "UpdateAttribute" processor to run all the rules(defined using
Nifi's Expression Language (EL)) and add the additional attributes.

*Drawbacks:*

1. If the rows contain data of type "blob" or "clob" then EvaluateJsonPath
processor may lead to memory out of bounds as it fetches all the content on
to heap for processing
2. Not sure how to update the additional attributes added by
"UpdateAttribute" processor to the actual content(or append to original
rows)

Probably, "ExecuteScript" processor with Nifi's EL support can be at
rescue. But wanted to know which option will be better. Any suggestion/help
on this will be very much appreciated.

Thanks.