Re: [jira] [Commented] (NIFI-220) Build Processors to interact with Kafka

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: [jira] [Commented] (NIFI-220) Build Processors to interact with Kafka

Joe Witt
Mark

Thanks.  My review then is a +1.  The 0x could be added later if of use.

Thanks!
Joe

On Mon, Jan 5, 2015 at 7:56 PM, Mark Payne (JIRA) <[hidden email]> wrote:

>
>     [
> https://issues.apache.org/jira/browse/NIFI-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265447#comment-14265447
> ]
>
> Mark Payne commented on NIFI-220:
> ---------------------------------
>
> Joe,
>
> PutKafka:
>  - Delivery Guarantee does switch to a numeric value when running, as
> Kafka wants them as -1, 0, or 1. But I didn't want to expose those
> arbitrary values in the UI, so I provided names.
>  - Message Delimiter is a UTF-8 String. I considered requiring a
> hex-encoded byte string, but that's pretty terrible if the data is text.
> Could make it configurable but then it's a bit of a weird property to add.
> Could check if it starts with '0x' and if so expect a hex-encoded byte
> sequence?
>
> GetKafka:
>  - We do not store these values locally. If NiFi is restarted, it could
> result in duplication of 'up to X amount of time worth of data'. Kafka's
> default is 60 seconds, so I chose to use their default.
>  - Message Demarcator: Same as Message Delimiter in PutKafka.
>
> Combined:
>  - The Known Brokers vs. ZooKeeper Connection String, unfortunately, is
> what Kafka exposes. Can't make them consistent with their API.
>  - GetKafka is configured with a Demarcator, as it is demarcating the
> messages that it receives. PutKafka has a delimiter, as the incoming data
> is delimited using this delimiter. I can see your viewpoint, too, though...
> honestly, I think either way works.
>  - Client Name is mapped to 'client.id' property in Kafka config. Their
> docs state "The client id is a user-specified string sent in each request
> to help trace calls. It should logically identify the application making
> the request." So modifying that should be fine.
>
>
> I think the biggest thing that we want to consider here is the
> delimiter/demarcator: What do you think of using a prefix of "0x" to
> indicate hex byte sequence, otherwise using a UTF-8 Encoded string?
>
> > Build Processors to interact with Kafka
> > ---------------------------------------
> >
> >                 Key: NIFI-220
> >                 URL: https://issues.apache.org/jira/browse/NIFI-220
> >             Project: Apache NiFi
> >          Issue Type: Task
> >          Components: Extensions
> >            Reporter: Mark Payne
> >
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>