[Discuss] DeleteSFTP

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Discuss] DeleteSFTP

Andre
Folks,

I was recently asked by a user if we have a DeleteSFTP processor to handle
file deletion AFTER some logic has been performed.

I noticed we don't have a processor covering that, and while ExecuteScript
could be used to make this happen I wondered if it we should support that?

The logic is simple:

ListSFTP -> FetchSFTP  - (if success) -> DoSomething - (if success) ->
DeleteSFTP


I think this pattern would fit those who are just too cautious to trust the
"Delete file" "Completion strategy" available on FetchSFTP.  The reason for
caution is the assumption that although NiFi queues are persistent, things
can still go wrong between the file being sourced and the "DoSomething"
success obtained.

What do you think?
Reply | Threaded
Open this post in threaded view
|

Re: [Discuss] DeleteSFTP

Pierre Villard
Hi Andre,

I don't see any issue but if something goes wrong with a file, this file
won't be listed anymore by ListSFTP. So I guess the "move" strategy is
better suited to remove files already listed by the ListSFTP processor. But
I agree that someone might want to delete the file in the "move" directory
once the file has been successfully processed (I follow a similar approach
with DeleteHDFS processor).

Anyway... yes, I do see use cases where it'd be useful :)

Pierre

2017-10-26 5:14 GMT+02:00 Andre <[hidden email]>:

> Folks,
>
> I was recently asked by a user if we have a DeleteSFTP processor to handle
> file deletion AFTER some logic has been performed.
>
> I noticed we don't have a processor covering that, and while ExecuteScript
> could be used to make this happen I wondered if it we should support that?
>
> The logic is simple:
>
> ListSFTP -> FetchSFTP  - (if success) -> DoSomething - (if success) ->
> DeleteSFTP
>
>
> I think this pattern would fit those who are just too cautious to trust the
> "Delete file" "Completion strategy" available on FetchSFTP.  The reason for
> caution is the assumption that although NiFi queues are persistent, things
> can still go wrong between the file being sourced and the "DoSomething"
> success obtained.
>
> What do you think?
>