Current state of Kafka processors

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Current state of Kafka processors

Woschitz, Janosch
Hi,

I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:

"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”

I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?

Could somebody provide more insights into the current state?

Thanks,
Janosch

Reply | Threaded
Open this post in threaded view
|

Re: Current state of Kafka processors

Woschitz, Janosch
Hi everyone,

Has anyone an idea what's the technical background regarding the warning in the Kafka processors documentation?

It is not clear to me how the processor can end up in a "indefinite stuck state”.

Thanks,
Janosch



On 14.11.17, 11:01, "Woschitz, Janosch" <[hidden email]> wrote:

>Hi,
>
>I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:
>
>"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”
>
>I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?
>
>Could somebody provide more insights into the current state?
>
>Thanks,
>Janosch
>
Reply | Threaded
Open this post in threaded view
|

Re: Current state of Kafka processors

Joe Witt
Janosch

Sorry for the lack of response.  It was there previously because it
was observed that timeout conditions could cause threads to stay in a
hung state and as the caller of the client we weren't in a position to
resolve it.  However, newer client libraries are better and I no
longer have observed these issues personally.  For the 0_10 or newer
clients I think we can adjust the docs to indicate that these should
work just fine.  Please feel free to file a JIRA if you dont mind.

Thanks

On Mon, Nov 20, 2017 at 6:22 PM, Woschitz, Janosch
<[hidden email]> wrote:

> Hi everyone,
>
> Has anyone an idea what's the technical background regarding the warning in the Kafka processors documentation?
>
> It is not clear to me how the processor can end up in a "indefinite stuck state”.
>
> Thanks,
> Janosch
>
>
>
> On 14.11.17, 11:01, "Woschitz, Janosch" <[hidden email]> wrote:
>
>>Hi,
>>
>>I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:
>>
>>"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”
>>
>>I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?
>>
>>Could somebody provide more insights into the current state?
>>
>>Thanks,
>>Janosch
>>
Reply | Threaded
Open this post in threaded view
|

Re: Current state of Kafka processors

Woschitz, Janosch
Hi Joe,

Thanks a lot for the explanation, this was also my assumption by looking at the code, though I assumed I missed something.

I filed https://issues.apache.org/jira/browse/NIFI-4623 and added a simple PR which just removes the warning in the doc comments for the newer Kafka processors.

Regards,
Janosch




On 21.11.17, 00:24, "Joe Witt" <[hidden email]> wrote:

>Janosch
>
>Sorry for the lack of response.  It was there previously because it
>was observed that timeout conditions could cause threads to stay in a
>hung state and as the caller of the client we weren't in a position to
>resolve it.  However, newer client libraries are better and I no
>longer have observed these issues personally.  For the 0_10 or newer
>clients I think we can adjust the docs to indicate that these should
>work just fine.  Please feel free to file a JIRA if you dont mind.
>
>Thanks
>
>On Mon, Nov 20, 2017 at 6:22 PM, Woschitz, Janosch
><[hidden email]> wrote:
>> Hi everyone,
>>
>> Has anyone an idea what's the technical background regarding the warning in the Kafka processors documentation?
>>
>> It is not clear to me how the processor can end up in a "indefinite stuck state”.
>>
>> Thanks,
>> Janosch
>>
>>
>>
>> On 14.11.17, 11:01, "Woschitz, Janosch" <[hidden email]> wrote:
>>
>>>Hi,
>>>
>>>I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:
>>>
>>>"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”
>>>
>>>I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?
>>>
>>>Could somebody provide more insights into the current state?
>>>
>>>Thanks,
>>>Janosch
>>>