NiFi Queues FlowFile Expiration

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi Queues FlowFile Expiration

Brian Ghigiarelli
Hi all,

In regards to the FlowFile expiration setting on queues between processors,
is that time based on:
- the arrival timestamp of the FlowFile into the queue,
- the first created / received timestamp of the FlowFile in the overall
flow, or
- something else entirely?

Thanks,

--
Brian Ghigiarelli
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Queues FlowFile Expiration

Jennifer Barnabee
Brian,
It's based on the timestamp of when the FlowFile was first received/created
in the overall flow.
-Jenn

On Wed, Apr 15, 2015 at 2:04 PM, Brian Ghigiarelli <[hidden email]>
wrote:

> Hi all,
>
> In regards to the FlowFile expiration setting on queues between processors,
> is that time based on:
> - the arrival timestamp of the FlowFile into the queue,
> - the first created / received timestamp of the FlowFile in the overall
> flow, or
> - something else entirely?
>
> Thanks,
>
> --
> Brian Ghigiarelli
>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Queues FlowFile Expiration

Corey Flowers
In reply to this post by Brian Ghigiarelli
Good afternoon Mr. Ghigiarelli,

          The flowfile expiration is based on the  timestamp of the file into the Nifi repo. Not the previous processor or original age of the file.

Hope that helps!

Sent from my iPhone

> On Apr 15, 2015, at 2:04 PM, Brian Ghigiarelli <[hidden email]> wrote:
>
> Hi all,
>
> In regards to the FlowFile expiration setting on queues between processors,
> is that time based on:
> - the arrival timestamp of the FlowFile into the queue,
> - the first created / received timestamp of the FlowFile in the overall
> flow, or
> - something else entirely?
>
> Thanks,
>
> --
> Brian Ghigiarelli
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Queues FlowFile Expiration

Brian Ghigiarelli
Thanks for the responses, Corey and Jennifer.

Corey - is "the timestamp of the file into the NiFi repo" set when the
FlowFile is first created/received, or when the FlowFile is enqueued?
Jennifer's answer indicates that it's set when the FlowFile is first
created/received.

Along these lines, if I clone and transfer a FlowFile, does it use the
timestamp of when the FlowFile was cloned or does it use the first
timestamp of the original FlowFile tracked by provenance?

Thanks,
Brian

On Wed, Apr 15, 2015 at 2:36 PM, Corey Flowers <[hidden email]>
wrote:

> Good afternoon Mr. Ghigiarelli,
>
>           The flowfile expiration is based on the  timestamp of the file
> into the Nifi repo. Not the previous processor or original age of the file.
>
> Hope that helps!
>
> Sent from my iPhone
>
> > On Apr 15, 2015, at 2:04 PM, Brian Ghigiarelli <[hidden email]>
> wrote:
> >
> > Hi all,
> >
> > In regards to the FlowFile expiration setting on queues between
> processors,
> > is that time based on:
> > - the arrival timestamp of the FlowFile into the queue,
> > - the first created / received timestamp of the FlowFile in the overall
> > flow, or
> > - something else entirely?
> >
> > Thanks,
> >
> > --
> > Brian Ghigiarelli
>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Queues FlowFile Expiration

Mark Payne
Brian,

Currently it would use the time stamp of when the clone happened. We do have the time stamp of when it was created, so we have considered changing to that but as of right now it is based on when the clone was created.

Thanks
-Mark

Sent from my iPad

> On Apr 15, 2015, at 2:01 PM, Brian Ghigiarelli <[hidden email]> wrote:
>
> Thanks for the responses, Corey and Jennifer.
>
> Corey - is "the timestamp of the file into the NiFi repo" set when the
> FlowFile is first created/received, or when the FlowFile is enqueued?
> Jennifer's answer indicates that it's set when the FlowFile is first
> created/received.
>
> Along these lines, if I clone and transfer a FlowFile, does it use the
> timestamp of when the FlowFile was cloned or does it use the first
> timestamp of the original FlowFile tracked by provenance?
>
> Thanks,
> Brian
>
> On Wed, Apr 15, 2015 at 2:36 PM, Corey Flowers <[hidden email]>
> wrote:
>
>> Good afternoon Mr. Ghigiarelli,
>>
>>          The flowfile expiration is based on the  timestamp of the file
>> into the Nifi repo. Not the previous processor or original age of the file.
>>
>> Hope that helps!
>>
>> Sent from my iPhone
>>
>>>> On Apr 15, 2015, at 2:04 PM, Brian Ghigiarelli <[hidden email]>
>>> wrote:
>>>
>>> Hi all,
>>>
>>> In regards to the FlowFile expiration setting on queues between
>> processors,
>>> is that time based on:
>>> - the arrival timestamp of the FlowFile into the queue,
>>> - the first created / received timestamp of the FlowFile in the overall
>>> flow, or
>>> - something else entirely?
>>>
>>> Thanks,
>>>
>>> --
>>> Brian Ghigiarelli
>>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Queues FlowFile Expiration

Corey Flowers
In reply to this post by Brian Ghigiarelli
Hey Brian,

        It would be the created/received time for the orig flowfile and like Mark said, for clones it would be the time of cloning because in essence a clone is a new flowfile. Jen just beat me to the punch and worded it a little better!

Corey

Sent from my iPhone

> On Apr 15, 2015, at 3:00 PM, Brian Ghigiarelli <[hidden email]> wrote:
>
> Thanks for the responses, Corey and Jennifer.
>
> Corey - is "the timestamp of the file into the NiFi repo" set when the
> FlowFile is first created/received, or when the FlowFile is enqueued?
> Jennifer's answer indicates that it's set when the FlowFile is first
> created/received.
>
> Along these lines, if I clone and transfer a FlowFile, does it use the
> timestamp of when the FlowFile was cloned or does it use the first
> timestamp of the original FlowFile tracked by provenance?
>
> Thanks,
> Brian
>
> On Wed, Apr 15, 2015 at 2:36 PM, Corey Flowers <[hidden email]>
> wrote:
>
>> Good afternoon Mr. Ghigiarelli,
>>
>>          The flowfile expiration is based on the  timestamp of the file
>> into the Nifi repo. Not the previous processor or original age of the file.
>>
>> Hope that helps!
>>
>> Sent from my iPhone
>>
>>>> On Apr 15, 2015, at 2:04 PM, Brian Ghigiarelli <[hidden email]>
>>> wrote:
>>>
>>> Hi all,
>>>
>>> In regards to the FlowFile expiration setting on queues between
>> processors,
>>> is that time based on:
>>> - the arrival timestamp of the FlowFile into the queue,
>>> - the first created / received timestamp of the FlowFile in the overall
>>> flow, or
>>> - something else entirely?
>>>
>>> Thanks,
>>>
>>> --
>>> Brian Ghigiarelli
>>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Queues FlowFile Expiration

Brian Ghigiarelli
Great - thanks for the clarification, everybody!  That's consistent with
what we're seeing while running it too.

On Wed, Apr 15, 2015 at 4:11 PM, Corey Flowers <[hidden email]>
wrote:

> Hey Brian,
>
>         It would be the created/received time for the orig flowfile and
> like Mark said, for clones it would be the time of cloning because in
> essence a clone is a new flowfile. Jen just beat me to the punch and worded
> it a little better!
>
> Corey
>
> Sent from my iPhone
>
> > On Apr 15, 2015, at 3:00 PM, Brian Ghigiarelli <[hidden email]>
> wrote:
> >
> > Thanks for the responses, Corey and Jennifer.
> >
> > Corey - is "the timestamp of the file into the NiFi repo" set when the
> > FlowFile is first created/received, or when the FlowFile is enqueued?
> > Jennifer's answer indicates that it's set when the FlowFile is first
> > created/received.
> >
> > Along these lines, if I clone and transfer a FlowFile, does it use the
> > timestamp of when the FlowFile was cloned or does it use the first
> > timestamp of the original FlowFile tracked by provenance?
> >
> > Thanks,
> > Brian
> >
> > On Wed, Apr 15, 2015 at 2:36 PM, Corey Flowers <[hidden email]>
> > wrote:
> >
> >> Good afternoon Mr. Ghigiarelli,
> >>
> >>          The flowfile expiration is based on the  timestamp of the file
> >> into the Nifi repo. Not the previous processor or original age of the
> file.
> >>
> >> Hope that helps!
> >>
> >> Sent from my iPhone
> >>
> >>>> On Apr 15, 2015, at 2:04 PM, Brian Ghigiarelli <[hidden email]>
> >>> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> In regards to the FlowFile expiration setting on queues between
> >> processors,
> >>> is that time based on:
> >>> - the arrival timestamp of the FlowFile into the queue,
> >>> - the first created / received timestamp of the FlowFile in the overall
> >>> flow, or
> >>> - something else entirely?
> >>>
> >>> Thanks,
> >>>
> >>> --
> >>> Brian Ghigiarelli
> >>
>