Forcing FlowFiles to Expire at a Connection

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Forcing FlowFiles to Expire at a Connection

Michael Hogue
All,

   I'm working on NIFI-620 <https://issues.apache.org/jira/browse/NIFI-620>,
which is to add expired FlowFile count to the Status History UI for
connections, and i've gotten to a point that i'm ready to test this but i'm
having trouble forcing FlowFiles to expire. I've created this flow [1] with
1ns FlowFile expirations, but no luck.

   Is there an easy way to force FlowFiles to be expired at a connection so
i can verify that my changes are being presented in the UI? Reading through
the code, it seems that it shouldn't be too difficult but possibly i'm
missing something.

Thanks,
Mike

[1] https://gist.github.com/m-hogue/a115c1bbc2423e4d9f7010acd408b0ad
Reply | Threaded
Open this post in threaded view
|

Re: Forcing FlowFiles to Expire at a Connection

Pierre Villard
Hi Mike,

What I would suggest is :
GFF (scheduled every 1s) -> LogAttribute (scheduled every 5s)
And expiration set to 1s on success relationship.
You should have about 1FF for 5 going through LogAttribute.

Pierre


2017-11-10 15:32 GMT+01:00 Michael Hogue <[hidden email]>:

> All,
>
>    I'm working on NIFI-620 <https://issues.apache.org/jira/browse/NIFI-620
> >,
> which is to add expired FlowFile count to the Status History UI for
> connections, and i've gotten to a point that i'm ready to test this but i'm
> having trouble forcing FlowFiles to expire. I've created this flow [1] with
> 1ns FlowFile expirations, but no luck.
>
>    Is there an easy way to force FlowFiles to be expired at a connection so
> i can verify that my changes are being presented in the UI? Reading through
> the code, it seems that it shouldn't be too difficult but possibly i'm
> missing something.
>
> Thanks,
> Mike
>
> [1] https://gist.github.com/m-hogue/a115c1bbc2423e4d9f7010acd408b0ad
>
Reply | Threaded
Open this post in threaded view
|

Re: Forcing FlowFiles to Expire at a Connection

Mark Payne
In reply to this post by Michael Hogue
Mike,

FlowFile expiration can happen in two ways:

1) There is a background thread that runs every 30 seconds or so that can trigger Expiration
2) Whenever a Processor (or other component) pulls from a queue, any expired FlowFiles are
expired instead of being handed to the Processor.

So if the destination of the connection is not running, you will see it expire only once every 30
seconds.

Thanks
-Mark


> On Nov 10, 2017, at 9:32 AM, Michael Hogue <[hidden email]> wrote:
>
> All,
>
>   I'm working on NIFI-620 <https://issues.apache.org/jira/browse/NIFI-620>,
> which is to add expired FlowFile count to the Status History UI for
> connections, and i've gotten to a point that i'm ready to test this but i'm
> having trouble forcing FlowFiles to expire. I've created this flow [1] with
> 1ns FlowFile expirations, but no luck.
>
>   Is there an easy way to force FlowFiles to be expired at a connection so
> i can verify that my changes are being presented in the UI? Reading through
> the code, it seems that it shouldn't be too difficult but possibly i'm
> missing something.
>
> Thanks,
> Mike
>
> [1] https://gist.github.com/m-hogue/a115c1bbc2423e4d9f7010acd408b0ad