Temporary FlowFile storage.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Temporary FlowFile storage.

Toivo Adams
I wonder what is the best way to store FlowFile few days.
For example we do some processing, then suspend processing and continue after few days.
Where FlowFile’s can be stored meanwhile?
Should I write custom processors for storing and restoring FlowFile content with attributes?
Or is it better to use some existing functionality?

Thanks,
Toivo
Reply | Threaded
Open this post in threaded view
|

Re: Temporary FlowFile storage.

Mark Payne
Hey Toivo,

The MergeContent Processor can be used for this. It has a Merge Format of FlowFile Stream v3. This will
create a FlowFile where the content contains both the original content and the attributes, bundled up in a
wrapper. If you want each FlowFile to be packaged up separately you can set the min & max bin size to 1.
Otherwise, you can configure it however you'd like to bundle up multiple FlowFiles. Then, use PutFile,
PutHDFS, PutAmazonS3, or whatever else makes sense for your scenario for storage. To bring the data
back into the flow, of course you can use GetFile, GetHDFS, FetchS3Object, etc. And this will bring in the
'wrapped' FlowFile. UnpackContent can then be used to unwrap this FlowFile back into its original form.

The only caveat here is that NiFi does not allow you to set a new UUID on a FlowFile. It is automatically
generated by the framework. So once you unpack the content you will still have a new 'uuid' attribute than
the original. So as long as you're not relying on that 'uuid' attribute this should handle your use case quite
nicely.

-Mark


> On Dec 13, 2016, at 2:21 AM, Toivo Adams <[hidden email]> wrote:
>
> I wonder what is the best way to store FlowFile few days.
> For example we do some processing, then suspend processing and continue
> after few days.
> Where FlowFile’s can be stored meanwhile?
> Should I write custom processors for storing and restoring FlowFile content
> with attributes?
> Or is it better to use some existing functionality?
>
> Thanks,
> Toivo
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Temporary-FlowFile-storage-tp14210.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Temporary FlowFile storage.

Toivo Adams
Hi Mark,

Great, this should be sufficient.


Thanks,
Toivo