clear all flowfiles in all queues upon NiFi restart

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

clear all flowfiles in all queues upon NiFi restart

尹文才
Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is
restarted, but I don't know the correct way to do this. I checked all
NiFi's guide documentation,
it seems there're 2 possible solutions:
1. write a custom notification service: a notification service could be
notified when NiFi is restarted and then inside the service, delete all the
files inside content_repository, flowfile_repository and
provenance_repository.
   I know there're now 2 existing services: email and http. But I'm not
quite sure how to correctly write one and deploy it into my NiFi
environment, is there a tutorial on writing one notification service?

2. I know from the developer guide that by using the annotation @Shutdown
in a custom processor, the method could be called when NiFi is successfully
shut down. The problem with this approach is the method could
   not be guaranteed to be called when NiFi dies unexpectedly.

Does anyone know what is the correct way to implement it? Thanks.

Regards,
Ben
Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

Mark Payne
Ben,

I have to admit - that’s kind of an odd request :) I’m curious what the use case is, if you can share?

Regardless, the easiest way would be to update nifi.properties so that the FlowFile repo that is used is the VolatileFlowFileRepository. This would avoid writing the FlowFile state to disk, so ok restart you will lose all FlowFiles. The content will still be present, but nifi will delete it all on startup because there is no FlowFile associated with it.

I’m on my phone right now so can’t easily tell you the exact name of the property to change but you’ll probably find it pretty quickly. The Admin Guide may well explain the different repositories as well.

Thanks
-Mark

Sent from my iPhone

> On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
>
> Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is
> restarted, but I don't know the correct way to do this. I checked all
> NiFi's guide documentation,
> it seems there're 2 possible solutions:
> 1. write a custom notification service: a notification service could be
> notified when NiFi is restarted and then inside the service, delete all the
> files inside content_repository, flowfile_repository and
> provenance_repository.
>   I know there're now 2 existing services: email and http. But I'm not
> quite sure how to correctly write one and deploy it into my NiFi
> environment, is there a tutorial on writing one notification service?
>
> 2. I know from the developer guide that by using the annotation @Shutdown
> in a custom processor, the method could be called when NiFi is successfully
> shut down. The problem with this approach is the method could
>   not be guaranteed to be called when NiFi dies unexpectedly.
>
> Does anyone know what is the correct way to implement it? Thanks.
>
> Regards,
> Ben
Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

尹文才
Thanks Mark, my case is that I'm using NiFi to do some ETL work and it's
possible that NiFi dies unexpectedly due to lack of system resources. After
NiFi restarts itself,
I will re-extract all the data from database and re-perform all the
operations, so I need to clear all possible FlowFiles that might exist in
any queue.

Regards,
Ben

2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:

> Ben,
>
> I have to admit - that’s kind of an odd request :) I’m curious what the
> use case is, if you can share?
>
> Regardless, the easiest way would be to update nifi.properties so that the
> FlowFile repo that is used is the VolatileFlowFileRepository. This would
> avoid writing the FlowFile state to disk, so ok restart you will lose all
> FlowFiles. The content will still be present, but nifi will delete it all
> on startup because there is no FlowFile associated with it.
>
> I’m on my phone right now so can’t easily tell you the exact name of the
> property to change but you’ll probably find it pretty quickly. The Admin
> Guide may well explain the different repositories as well.
>
> Thanks
> -Mark
>
> Sent from my iPhone
>
> > On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
> >
> > Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is
> > restarted, but I don't know the correct way to do this. I checked all
> > NiFi's guide documentation,
> > it seems there're 2 possible solutions:
> > 1. write a custom notification service: a notification service could be
> > notified when NiFi is restarted and then inside the service, delete all
> the
> > files inside content_repository, flowfile_repository and
> > provenance_repository.
> >   I know there're now 2 existing services: email and http. But I'm not
> > quite sure how to correctly write one and deploy it into my NiFi
> > environment, is there a tutorial on writing one notification service?
> >
> > 2. I know from the developer guide that by using the annotation @Shutdown
> > in a custom processor, the method could be called when NiFi is
> successfully
> > shut down. The problem with this approach is the method could
> >   not be guaranteed to be called when NiFi dies unexpectedly.
> >
> > Does anyone know what is the correct way to implement it? Thanks.
> >
> > Regards,
> > Ben
>
Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

尹文才
Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, if I
switch to use VolatileFlowFileRepository, will NiFi swap out all the other
FlowFiles to disk if a queue is already full?
Is it just simply keeping all FlowFiles in memory?

Regards,
Ben

2018-01-12 12:07 GMT+08:00 尹文才 <[hidden email]>:

> Thanks Mark, my case is that I'm using NiFi to do some ETL work and it's
> possible that NiFi dies unexpectedly due to lack of system resources. After
> NiFi restarts itself,
> I will re-extract all the data from database and re-perform all the
> operations, so I need to clear all possible FlowFiles that might exist in
> any queue.
>
> Regards,
> Ben
>
> 2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:
>
>> Ben,
>>
>> I have to admit - that’s kind of an odd request :) I’m curious what the
>> use case is, if you can share?
>>
>> Regardless, the easiest way would be to update nifi.properties so that
>> the FlowFile repo that is used is the VolatileFlowFileRepository. This
>> would avoid writing the FlowFile state to disk, so ok restart you will lose
>> all FlowFiles. The content will still be present, but nifi will delete it
>> all on startup because there is no FlowFile associated with it.
>>
>> I’m on my phone right now so can’t easily tell you the exact name of the
>> property to change but you’ll probably find it pretty quickly. The Admin
>> Guide may well explain the different repositories as well.
>>
>> Thanks
>> -Mark
>>
>> Sent from my iPhone
>>
>> > On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
>> >
>> > Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is
>> > restarted, but I don't know the correct way to do this. I checked all
>> > NiFi's guide documentation,
>> > it seems there're 2 possible solutions:
>> > 1. write a custom notification service: a notification service could be
>> > notified when NiFi is restarted and then inside the service, delete all
>> the
>> > files inside content_repository, flowfile_repository and
>> > provenance_repository.
>> >   I know there're now 2 existing services: email and http. But I'm not
>> > quite sure how to correctly write one and deploy it into my NiFi
>> > environment, is there a tutorial on writing one notification service?
>> >
>> > 2. I know from the developer guide that by using the annotation
>> @Shutdown
>> > in a custom processor, the method could be called when NiFi is
>> successfully
>> > shut down. The problem with this approach is the method could
>> >   not be guaranteed to be called when NiFi dies unexpectedly.
>> >
>> > Does anyone know what is the correct way to implement it? Thanks.
>> >
>> > Regards,
>> > Ben
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

Andrew Grande-2
Perhaps you could delete the repository directories when you need to
restart with no data?

On Thu, Jan 11, 2018, 9:16 PM 尹文才 <[hidden email]> wrote:

> Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, if I
> switch to use VolatileFlowFileRepository, will NiFi swap out all the other
> FlowFiles to disk if a queue is already full?
> Is it just simply keeping all FlowFiles in memory?
>
> Regards,
> Ben
>
> 2018-01-12 12:07 GMT+08:00 尹文才 <[hidden email]>:
>
> > Thanks Mark, my case is that I'm using NiFi to do some ETL work and it's
> > possible that NiFi dies unexpectedly due to lack of system resources.
> After
> > NiFi restarts itself,
> > I will re-extract all the data from database and re-perform all the
> > operations, so I need to clear all possible FlowFiles that might exist in
> > any queue.
> >
> > Regards,
> > Ben
> >
> > 2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:
> >
> >> Ben,
> >>
> >> I have to admit - that’s kind of an odd request :) I’m curious what the
> >> use case is, if you can share?
> >>
> >> Regardless, the easiest way would be to update nifi.properties so that
> >> the FlowFile repo that is used is the VolatileFlowFileRepository. This
> >> would avoid writing the FlowFile state to disk, so ok restart you will
> lose
> >> all FlowFiles. The content will still be present, but nifi will delete
> it
> >> all on startup because there is no FlowFile associated with it.
> >>
> >> I’m on my phone right now so can’t easily tell you the exact name of the
> >> property to change but you’ll probably find it pretty quickly. The Admin
> >> Guide may well explain the different repositories as well.
> >>
> >> Thanks
> >> -Mark
> >>
> >> Sent from my iPhone
> >>
> >> > On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
> >> >
> >> > Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is
> >> > restarted, but I don't know the correct way to do this. I checked all
> >> > NiFi's guide documentation,
> >> > it seems there're 2 possible solutions:
> >> > 1. write a custom notification service: a notification service could
> be
> >> > notified when NiFi is restarted and then inside the service, delete
> all
> >> the
> >> > files inside content_repository, flowfile_repository and
> >> > provenance_repository.
> >> >   I know there're now 2 existing services: email and http. But I'm not
> >> > quite sure how to correctly write one and deploy it into my NiFi
> >> > environment, is there a tutorial on writing one notification service?
> >> >
> >> > 2. I know from the developer guide that by using the annotation
> >> @Shutdown
> >> > in a custom processor, the method could be called when NiFi is
> >> successfully
> >> > shut down. The problem with this approach is the method could
> >> >   not be guaranteed to be called when NiFi dies unexpectedly.
> >> >
> >> > Does anyone know what is the correct way to implement it? Thanks.
> >> >
> >> > Regards,
> >> > Ben
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

尹文才
Hi Andrew, sorry I didn't follow your idea, could you please elaborate with
more details?
What I want to do is to be able to clear all the FlowFiles when NiFi dies
unexpectedly and restarts itself.

Regards,
Ben

2018-01-12 12:44 GMT+08:00 Andrew Grande <[hidden email]>:

> Perhaps you could delete the repository directories when you need to
> restart with no data?
>
> On Thu, Jan 11, 2018, 9:16 PM 尹文才 <[hidden email]> wrote:
>
> > Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned,
> if I
> > switch to use VolatileFlowFileRepository, will NiFi swap out all the
> other
> > FlowFiles to disk if a queue is already full?
> > Is it just simply keeping all FlowFiles in memory?
> >
> > Regards,
> > Ben
> >
> > 2018-01-12 12:07 GMT+08:00 尹文才 <[hidden email]>:
> >
> > > Thanks Mark, my case is that I'm using NiFi to do some ETL work and
> it's
> > > possible that NiFi dies unexpectedly due to lack of system resources.
> > After
> > > NiFi restarts itself,
> > > I will re-extract all the data from database and re-perform all the
> > > operations, so I need to clear all possible FlowFiles that might exist
> in
> > > any queue.
> > >
> > > Regards,
> > > Ben
> > >
> > > 2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:
> > >
> > >> Ben,
> > >>
> > >> I have to admit - that’s kind of an odd request :) I’m curious what
> the
> > >> use case is, if you can share?
> > >>
> > >> Regardless, the easiest way would be to update nifi.properties so that
> > >> the FlowFile repo that is used is the VolatileFlowFileRepository. This
> > >> would avoid writing the FlowFile state to disk, so ok restart you will
> > lose
> > >> all FlowFiles. The content will still be present, but nifi will delete
> > it
> > >> all on startup because there is no FlowFile associated with it.
> > >>
> > >> I’m on my phone right now so can’t easily tell you the exact name of
> the
> > >> property to change but you’ll probably find it pretty quickly. The
> Admin
> > >> Guide may well explain the different repositories as well.
> > >>
> > >> Thanks
> > >> -Mark
> > >>
> > >> Sent from my iPhone
> > >>
> > >> > On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
> > >> >
> > >> > Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi
> is
> > >> > restarted, but I don't know the correct way to do this. I checked
> all
> > >> > NiFi's guide documentation,
> > >> > it seems there're 2 possible solutions:
> > >> > 1. write a custom notification service: a notification service could
> > be
> > >> > notified when NiFi is restarted and then inside the service, delete
> > all
> > >> the
> > >> > files inside content_repository, flowfile_repository and
> > >> > provenance_repository.
> > >> >   I know there're now 2 existing services: email and http. But I'm
> not
> > >> > quite sure how to correctly write one and deploy it into my NiFi
> > >> > environment, is there a tutorial on writing one notification
> service?
> > >> >
> > >> > 2. I know from the developer guide that by using the annotation
> > >> @Shutdown
> > >> > in a custom processor, the method could be called when NiFi is
> > >> successfully
> > >> > shut down. The problem with this approach is the method could
> > >> >   not be guaranteed to be called when NiFi dies unexpectedly.
> > >> >
> > >> > Does anyone know what is the correct way to implement it? Thanks.
> > >> >
> > >> > Regards,
> > >> > Ben
> > >>
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

Brett Ryan
Is there a plugin hook that can be written to allow one to see the current state of all flows?

One thought I have is that this process could look for a configured property, like volatile.delete.strategy that a hook on startup could delete this based on given factors.

One use case would be that something being ingested if is still in the pipeline for over a given period after a restart it may be considered stale. I could think of other use cases similar to this.

> On 12 Jan 2018, at 15:48, 尹文才 <[hidden email]> wrote:
>
> Hi Andrew, sorry I didn't follow your idea, could you please elaborate with
> more details?
> What I want to do is to be able to clear all the FlowFiles when NiFi dies
> unexpectedly and restarts itself.
>
> Regards,
> Ben
>
> 2018-01-12 12:44 GMT+08:00 Andrew Grande <[hidden email]>:
>
>> Perhaps you could delete the repository directories when you need to
>> restart with no data?
>>
>>> On Thu, Jan 11, 2018, 9:16 PM 尹文才 <[hidden email]> wrote:
>>>
>>> Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned,
>> if I
>>> switch to use VolatileFlowFileRepository, will NiFi swap out all the
>> other
>>> FlowFiles to disk if a queue is already full?
>>> Is it just simply keeping all FlowFiles in memory?
>>>
>>> Regards,
>>> Ben
>>>
>>> 2018-01-12 12:07 GMT+08:00 尹文才 <[hidden email]>:
>>>
>>>> Thanks Mark, my case is that I'm using NiFi to do some ETL work and
>> it's
>>>> possible that NiFi dies unexpectedly due to lack of system resources.
>>> After
>>>> NiFi restarts itself,
>>>> I will re-extract all the data from database and re-perform all the
>>>> operations, so I need to clear all possible FlowFiles that might exist
>> in
>>>> any queue.
>>>>
>>>> Regards,
>>>> Ben
>>>>
>>>> 2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:
>>>>
>>>>> Ben,
>>>>>
>>>>> I have to admit - that’s kind of an odd request :) I’m curious what
>> the
>>>>> use case is, if you can share?
>>>>>
>>>>> Regardless, the easiest way would be to update nifi.properties so that
>>>>> the FlowFile repo that is used is the VolatileFlowFileRepository. This
>>>>> would avoid writing the FlowFile state to disk, so ok restart you will
>>> lose
>>>>> all FlowFiles. The content will still be present, but nifi will delete
>>> it
>>>>> all on startup because there is no FlowFile associated with it.
>>>>>
>>>>> I’m on my phone right now so can’t easily tell you the exact name of
>> the
>>>>> property to change but you’ll probably find it pretty quickly. The
>> Admin
>>>>> Guide may well explain the different repositories as well.
>>>>>
>>>>> Thanks
>>>>> -Mark
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
>>>>>>
>>>>>> Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi
>> is
>>>>>> restarted, but I don't know the correct way to do this. I checked
>> all
>>>>>> NiFi's guide documentation,
>>>>>> it seems there're 2 possible solutions:
>>>>>> 1. write a custom notification service: a notification service could
>>> be
>>>>>> notified when NiFi is restarted and then inside the service, delete
>>> all
>>>>> the
>>>>>> files inside content_repository, flowfile_repository and
>>>>>> provenance_repository.
>>>>>>  I know there're now 2 existing services: email and http. But I'm
>> not
>>>>>> quite sure how to correctly write one and deploy it into my NiFi
>>>>>> environment, is there a tutorial on writing one notification
>> service?
>>>>>>
>>>>>> 2. I know from the developer guide that by using the annotation
>>>>> @Shutdown
>>>>>> in a custom processor, the method could be called when NiFi is
>>>>> successfully
>>>>>> shut down. The problem with this approach is the method could
>>>>>>  not be guaranteed to be called when NiFi dies unexpectedly.
>>>>>>
>>>>>> Does anyone know what is the correct way to implement it? Thanks.
>>>>>>
>>>>>> Regards,
>>>>>> Ben
>>>>>
>>>>
>>>>
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

Mark Payne
In reply to this post by 尹文才
Ben,

No, the FlowFiles will still be swapped out to avoid heap exhaustion, as happens by default.
This is controlled by the "nifi.swap.manager.implementation" property, which you can still leave
set to "org.apache.nifi.controller.FileSystemSwapManager"

Thanks
-Mark

> On Jan 11, 2018, at 11:15 PM, 尹文才 <[hidden email]> wrote:
>
> Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, if I
> switch to use VolatileFlowFileRepository, will NiFi swap out all the other
> FlowFiles to disk if a queue is already full?
> Is it just simply keeping all FlowFiles in memory?
>
> Regards,
> Ben
>
> 2018-01-12 12:07 GMT+08:00 尹文才 <[hidden email]>:
>
>> Thanks Mark, my case is that I'm using NiFi to do some ETL work and it's
>> possible that NiFi dies unexpectedly due to lack of system resources. After
>> NiFi restarts itself,
>> I will re-extract all the data from database and re-perform all the
>> operations, so I need to clear all possible FlowFiles that might exist in
>> any queue.
>>
>> Regards,
>> Ben
>>
>> 2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:
>>
>>> Ben,
>>>
>>> I have to admit - that’s kind of an odd request :) I’m curious what the
>>> use case is, if you can share?
>>>
>>> Regardless, the easiest way would be to update nifi.properties so that
>>> the FlowFile repo that is used is the VolatileFlowFileRepository. This
>>> would avoid writing the FlowFile state to disk, so ok restart you will lose
>>> all FlowFiles. The content will still be present, but nifi will delete it
>>> all on startup because there is no FlowFile associated with it.
>>>
>>> I’m on my phone right now so can’t easily tell you the exact name of the
>>> property to change but you’ll probably find it pretty quickly. The Admin
>>> Guide may well explain the different repositories as well.
>>>
>>> Thanks
>>> -Mark
>>>
>>> Sent from my iPhone
>>>
>>>> On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
>>>>
>>>> Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is
>>>> restarted, but I don't know the correct way to do this. I checked all
>>>> NiFi's guide documentation,
>>>> it seems there're 2 possible solutions:
>>>> 1. write a custom notification service: a notification service could be
>>>> notified when NiFi is restarted and then inside the service, delete all
>>> the
>>>> files inside content_repository, flowfile_repository and
>>>> provenance_repository.
>>>>  I know there're now 2 existing services: email and http. But I'm not
>>>> quite sure how to correctly write one and deploy it into my NiFi
>>>> environment, is there a tutorial on writing one notification service?
>>>>
>>>> 2. I know from the developer guide that by using the annotation
>>> @Shutdown
>>>> in a custom processor, the method could be called when NiFi is
>>> successfully
>>>> shut down. The problem with this approach is the method could
>>>>  not be guaranteed to be called when NiFi dies unexpectedly.
>>>>
>>>> Does anyone know what is the correct way to implement it? Thanks.
>>>>
>>>> Regards,
>>>> Ben
>>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

Russell Bateman-2
In reply to this post by 尹文才
Andrew just meant that if you smoke the contents of all the repository
subdirectories under ${NIFI_ROOT}, it will result in what you seem to be
asking for.

Hope this helps.

On 01/11/2018 09:48 PM, 尹文才 wrote:

> Hi Andrew, sorry I didn't follow your idea, could you please elaborate with
> more details?
> What I want to do is to be able to clear all the FlowFiles when NiFi dies
> unexpectedly and restarts itself.
>
> Regards,
> Ben
>
> 2018-01-12 12:44 GMT+08:00 Andrew Grande <[hidden email]>:
>
>> Perhaps you could delete the repository directories when you need to
>> restart with no data?
>>
>> On Thu, Jan 11, 2018, 9:16 PM 尹文才 <[hidden email]> wrote:
>>
>>> Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned,
>> if I
>>> switch to use VolatileFlowFileRepository, will NiFi swap out all the
>> other
>>> FlowFiles to disk if a queue is already full?
>>> Is it just simply keeping all FlowFiles in memory?
>>>
>>> Regards,
>>> Ben
>>>
>>> 2018-01-12 12:07 GMT+08:00 尹文才 <[hidden email]>:
>>>
>>>> Thanks Mark, my case is that I'm using NiFi to do some ETL work and
>> it's
>>>> possible that NiFi dies unexpectedly due to lack of system resources.
>>> After
>>>> NiFi restarts itself,
>>>> I will re-extract all the data from database and re-perform all the
>>>> operations, so I need to clear all possible FlowFiles that might exist
>> in
>>>> any queue.
>>>>
>>>> Regards,
>>>> Ben
>>>>
>>>> 2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:
>>>>
>>>>> Ben,
>>>>>
>>>>> I have to admit - that’s kind of an odd request :) I’m curious what
>> the
>>>>> use case is, if you can share?
>>>>>
>>>>> Regardless, the easiest way would be to update nifi.properties so that
>>>>> the FlowFile repo that is used is the VolatileFlowFileRepository. This
>>>>> would avoid writing the FlowFile state to disk, so ok restart you will
>>> lose
>>>>> all FlowFiles. The content will still be present, but nifi will delete
>>> it
>>>>> all on startup because there is no FlowFile associated with it.
>>>>>
>>>>> I’m on my phone right now so can’t easily tell you the exact name of
>> the
>>>>> property to change but you’ll probably find it pretty quickly. The
>> Admin
>>>>> Guide may well explain the different repositories as well.
>>>>>
>>>>> Thanks
>>>>> -Mark
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
>>>>>>
>>>>>> Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi
>> is
>>>>>> restarted, but I don't know the correct way to do this. I checked
>> all
>>>>>> NiFi's guide documentation,
>>>>>> it seems there're 2 possible solutions:
>>>>>> 1. write a custom notification service: a notification service could
>>> be
>>>>>> notified when NiFi is restarted and then inside the service, delete
>>> all
>>>>> the
>>>>>> files inside content_repository, flowfile_repository and
>>>>>> provenance_repository.
>>>>>>    I know there're now 2 existing services: email and http. But I'm
>> not
>>>>>> quite sure how to correctly write one and deploy it into my NiFi
>>>>>> environment, is there a tutorial on writing one notification
>> service?
>>>>>> 2. I know from the developer guide that by using the annotation
>>>>> @Shutdown
>>>>>> in a custom processor, the method could be called when NiFi is
>>>>> successfully
>>>>>> shut down. The problem with this approach is the method could
>>>>>>    not be guaranteed to be called when NiFi dies unexpectedly.
>>>>>>
>>>>>> Does anyone know what is the correct way to implement it? Thanks.
>>>>>>
>>>>>> Regards,
>>>>>> Ben
>>>>

Reply | Threaded
Open this post in threaded view
|

Re: clear all flowfiles in all queues upon NiFi restart

尹文才
Thanks Mark, Andrew and Russell, I think using the Volatile-implementations
repositories mentioned by Mark should be sufficient for me.

Regards,
Ben

2018-01-12 22:59 GMT+08:00 Russell Bateman <[hidden email]>:

> Andrew just meant that if you smoke the contents of all the repository
> subdirectories under ${NIFI_ROOT}, it will result in what you seem to be
> asking for.
>
> Hope this helps.
>
>
> On 01/11/2018 09:48 PM, 尹文才 wrote:
>
>> Hi Andrew, sorry I didn't follow your idea, could you please elaborate
>> with
>> more details?
>> What I want to do is to be able to clear all the FlowFiles when NiFi dies
>> unexpectedly and restarts itself.
>>
>> Regards,
>> Ben
>>
>> 2018-01-12 12:44 GMT+08:00 Andrew Grande <[hidden email]>:
>>
>> Perhaps you could delete the repository directories when you need to
>>> restart with no data?
>>>
>>> On Thu, Jan 11, 2018, 9:16 PM 尹文才 <[hidden email]> wrote:
>>>
>>> Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned,
>>>>
>>> if I
>>>
>>>> switch to use VolatileFlowFileRepository, will NiFi swap out all the
>>>>
>>> other
>>>
>>>> FlowFiles to disk if a queue is already full?
>>>> Is it just simply keeping all FlowFiles in memory?
>>>>
>>>> Regards,
>>>> Ben
>>>>
>>>> 2018-01-12 12:07 GMT+08:00 尹文才 <[hidden email]>:
>>>>
>>>> Thanks Mark, my case is that I'm using NiFi to do some ETL work and
>>>>>
>>>> it's
>>>
>>>> possible that NiFi dies unexpectedly due to lack of system resources.
>>>>>
>>>> After
>>>>
>>>>> NiFi restarts itself,
>>>>> I will re-extract all the data from database and re-perform all the
>>>>> operations, so I need to clear all possible FlowFiles that might exist
>>>>>
>>>> in
>>>
>>>> any queue.
>>>>>
>>>>> Regards,
>>>>> Ben
>>>>>
>>>>> 2018-01-12 11:49 GMT+08:00 Mark Payne <[hidden email]>:
>>>>>
>>>>> Ben,
>>>>>>
>>>>>> I have to admit - that’s kind of an odd request :) I’m curious what
>>>>>>
>>>>> the
>>>
>>>> use case is, if you can share?
>>>>>>
>>>>>> Regardless, the easiest way would be to update nifi.properties so that
>>>>>> the FlowFile repo that is used is the VolatileFlowFileRepository. This
>>>>>> would avoid writing the FlowFile state to disk, so ok restart you will
>>>>>>
>>>>> lose
>>>>
>>>>> all FlowFiles. The content will still be present, but nifi will delete
>>>>>>
>>>>> it
>>>>
>>>>> all on startup because there is no FlowFile associated with it.
>>>>>>
>>>>>> I’m on my phone right now so can’t easily tell you the exact name of
>>>>>>
>>>>> the
>>>
>>>> property to change but you’ll probably find it pretty quickly. The
>>>>>>
>>>>> Admin
>>>
>>>> Guide may well explain the different repositories as well.
>>>>>>
>>>>>> Thanks
>>>>>> -Mark
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Jan 11, 2018, at 10:31 PM, 尹文才 <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi
>>>>>>>
>>>>>> is
>>>
>>>> restarted, but I don't know the correct way to do this. I checked
>>>>>>>
>>>>>> all
>>>
>>>> NiFi's guide documentation,
>>>>>>> it seems there're 2 possible solutions:
>>>>>>> 1. write a custom notification service: a notification service could
>>>>>>>
>>>>>> be
>>>>
>>>>> notified when NiFi is restarted and then inside the service, delete
>>>>>>>
>>>>>> all
>>>>
>>>>> the
>>>>>>
>>>>>>> files inside content_repository, flowfile_repository and
>>>>>>> provenance_repository.
>>>>>>>    I know there're now 2 existing services: email and http. But I'm
>>>>>>>
>>>>>> not
>>>
>>>> quite sure how to correctly write one and deploy it into my NiFi
>>>>>>> environment, is there a tutorial on writing one notification
>>>>>>>
>>>>>> service?
>>>
>>>> 2. I know from the developer guide that by using the annotation
>>>>>>>
>>>>>> @Shutdown
>>>>>>
>>>>>>> in a custom processor, the method could be called when NiFi is
>>>>>>>
>>>>>> successfully
>>>>>>
>>>>>>> shut down. The problem with this approach is the method could
>>>>>>>    not be guaranteed to be called when NiFi dies unexpectedly.
>>>>>>>
>>>>>>> Does anyone know what is the correct way to implement it? Thanks.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ben
>>>>>>>
>>>>>>
>>>>>
>