NiFi processor to fetch attribute value dynamically from file or table

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi processor to fetch attribute value dynamically from file or table

Mothi86
Hey Hi,

I want NiFi processor to fetch attribute value on run time. Example - if I
am filtering twitter feeds by specific keywords, i want to maintain the
list of keywords in a separate repository like file or table and not
confined as a text box value. In that case, how will NiFi processor fetch
that values from external file or table to attribute value.

Warm Regards,
M. Mothilal
Reply | Threaded
Open this post in threaded view
|

Re: NiFi processor to fetch attribute value dynamically from file or table

Joey Frazee
There's definitely more than one way to do this, so I'll throw out some of the approaches I've taken:

- Wrap the config store in a web service and use InvokeHttp with Put Response Body in Attribute set to true, followed by an ExecuteScript processor that evaluates the web service response and merges the attributes onto the FlowFile.

- Use ExecuteScript to read a file, e.g., YAML, and merge the values into attributes.

- Use multiple ScanAttributes with routing rules. ScanAttribute lets you setup a reloadable dictionary, so you can treat each dictionary as the result of the lookup key. This will only make since if there's a small number of values.

- Use FetchDistributedMapCache with Put Cache Value in Attribute set to true.

- Put the config store in a database and use ExtractText and ReplaceText to first move the content to an attribute, get the config, put the config in an attribute and return the original contents to the FlowFile.

- Use the variable registry. One caveat however: for now, the variable registry cannot be updated dynamically so you'd have to restart NiFi if you need to change something.

- Last, I've been doing some work on a LookupAttribute processor [1] that uses controller services to abstract what the config store is. It either fetches all keys or specific keys by name if you use dynamic properties. Right now it has a reloadable properties file implementation and I've started on a JSON REST interface but haven't pushed the code yet. As the variable registry progresses, however, this could very likely become obsolete.
-joey

1. https://github.com/jfrazee/nifi-lookup-service/tree/file-based-lookup-service

On Nov 15, 2016, at 12:44 PM, Mothilal M <[hidden email]> wrote:

Hey Hi,

I want NiFi processor to fetch attribute value on run time. Example - if I
am filtering twitter feeds by specific keywords, i want to maintain the
list of keywords in a separate repository like file or table and not
confined as a text box value. In that case, how will NiFi processor fetch
that values from external file or table to attribute value.

Warm Regards,
M. Mothilal
Reply | Threaded
Open this post in threaded view
|

Re: NiFi processor to fetch attribute value dynamically from file or table

Akshay6787
Hi,
i have declared  my custom properties file path  in nifi.properties  to
variable 'nifi.variable.registry.properties=custom.properties'
.i want my processors to pickup the new value of the property when the
chnages are made to a property.I have loaded the updated properties in the
code(FileBasedVariableRegistry.java).but processors takes old values..

Thanks.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/