Adding Failure handling to ListFile

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding Failure handling to ListFile

Toivo Adams
Hi

ListFile has only SUCCESS relationship.

It would useful to add also FAILURE.
For example when input directory does not exist or can’t be read.

What do you think?

Thanks
Toivo



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: Adding Failure handling to ListFile

Joe Witt
Toivo

What would be sent to FAILURE?  What is the content or attributes of
the flowfile in that case?

Thanks

On Mon, Oct 30, 2017 at 8:42 AM, Toivo Adams <[hidden email]> wrote:

> Hi
>
> ListFile has only SUCCESS relationship.
>
> It would useful to add also FAILURE.
> For example when input directory does not exist or can’t be read.
>
> What do you think?
>
> Thanks
> Toivo
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: Adding Failure handling to ListFile

Toivo Adams
Content would be empty.
One or more attributes contain error message, maybe error code, etc.
 



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: Adding Failure handling to ListFile

Joe Witt
toivo

with all source processors there is the potential for failure.  The
challenge is what to capture and more generally what one would do with a
flowfile representing not data but the fact of a failure.  Can you share
more of your thoughts on that for this case and perhaps generally?

thanks

On Oct 30, 2017 8:56 AM, "Toivo Adams" <[hidden email]> wrote:

> Content would be empty.
> One or more attributes contain error message, maybe error code, etc.
>
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: Adding Failure handling to ListFile

Toivo Adams
We are run separate processes  (implemented as Process Group) which have
start and end. Same process should same unique id.
We want to monitor process from start to end. Summary include only process
name, when started, when ended, unique id and whatever it was successful or
not. In case of failure we want to have reason and where it failed.
So we route all processors FAILURE relationships to Failure handler which
will log some data and sometimes do some cleanup or finishing actions.

Our own processor all have FAILURE relationships, but same NiFi processors
don’t. And is is the problem.

Actually if NiFi have some other general kind failure handling which is not
based on FAILURE relationships this would most welcome.

I know we can use Reporting tasks, but reporting task don’t allow to use
standard NiFi processors for some cleanup tasks. Also we can’t cancel
execution of failed process (Fail Fast)

And we already use FAILURE relationships heavily.

And another problem is - some processors doesn't have inputs, so we cant
link all processors to single process.

Thanks



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/