Thoughts on an internal "Terminate" handler for special processors

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Thoughts on an internal "Terminate" handler for special processors

Peter Wicks (pwicks)
I've run into a few cases where a processor that works with external resources (DBCP especially) will hang and can't be properly terminated. I was thinking it would make sense to have an interface or annotation that these processors could have that would flag them for "special" termination.

For example. In DBCP processors we could put a reference to the current statement, and then when a terminate is received we can call into the processor and ask it to perform it's special termination process, which in this case would be to cancel the statement.

I've heard similar complaints concerning ExecuteStreamCommand, but have not experienced it.

Any thoughts on this? It would only be for the Terminate option, it would not affect normal operation states such as Stop.

Thanks,
  Peter Wicks
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on an internal "Terminate" handler for special processors

Mark Payne
Peter,

Without thinking through this too much, I am not opposed to the idea, but there are certainly a lot of things that would have to be carefully thought through around the lifecycle:

- When a processor is terminated, its threads are interrupted. They may or may not ignore the interrupt. When should this handler be triggered? After the threads return from the interrupt? Or after the interrupt is triggered and before they return (if they ever will)?
- What if the Termination handler never returns? Does this just eat up an additional thread and get reported as a "Terminated thread"? What thread pool, if any, would this thread come from?

Just some thoughts that would have to be worked through...


> On Jun 19, 2019, at 11:17 AM, Peter Wicks (pwicks) <[hidden email]> wrote:
>
> I've run into a few cases where a processor that works with external resources (DBCP especially) will hang and can't be properly terminated. I was thinking it would make sense to have an interface or annotation that these processors could have that would flag them for "special" termination.
>
> For example. In DBCP processors we could put a reference to the current statement, and then when a terminate is received we can call into the processor and ask it to perform it's special termination process, which in this case would be to cancel the statement.
>
> I've heard similar complaints concerning ExecuteStreamCommand, but have not experienced it.
>
> Any thoughts on this? It would only be for the Terminate option, it would not affect normal operation states such as Stop.
>
> Thanks,
>  Peter Wicks

Reply | Threaded
Open this post in threaded view
|

RE: [EXT] Re: Thoughts on an internal "Terminate" handler for special processors

Peter Wicks (pwicks)
Mark,

Lots of good questions. I'll write up a JIRA on it at least, and finish reviewing your original PR on Terminate (https://github.com/apache/nifi/pull/2555/).

Thanks,
  Peter


-----Original Message-----
From: Mark Payne <[hidden email]>
Sent: Wednesday, June 19, 2019 9:27 AM
To: [hidden email]
Subject: [EXT] Re: Thoughts on an internal "Terminate" handler for special processors

Peter,

Without thinking through this too much, I am not opposed to the idea, but there are certainly a lot of things that would have to be carefully thought through around the lifecycle:

- When a processor is terminated, its threads are interrupted. They may or may not ignore the interrupt. When should this handler be triggered? After the threads return from the interrupt? Or after the interrupt is triggered and before they return (if they ever will)?
- What if the Termination handler never returns? Does this just eat up an additional thread and get reported as a "Terminated thread"? What thread pool, if any, would this thread come from?

Just some thoughts that would have to be worked through...


> On Jun 19, 2019, at 11:17 AM, Peter Wicks (pwicks) <[hidden email]> wrote:
>
> I've run into a few cases where a processor that works with external resources (DBCP especially) will hang and can't be properly terminated. I was thinking it would make sense to have an interface or annotation that these processors could have that would flag them for "special" termination.
>
> For example. In DBCP processors we could put a reference to the current statement, and then when a terminate is received we can call into the processor and ask it to perform it's special termination process, which in this case would be to cancel the statement.
>
> I've heard similar complaints concerning ExecuteStreamCommand, but have not experienced it.
>
> Any thoughts on this? It would only be for the Terminate option, it would not affect normal operation states such as Stop.
>
> Thanks,
>  Peter Wicks