Reading avro encoded message key from kafka

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Reading avro encoded message key from kafka

ashwin.konale@gmail.com
Hi,
I have key-value pair of avro messages in kafka topic I want to consume
from. I can easily do modifications on message value using nifi
consumeKafkaRecord processor, but it doesnt show key of the message.
ConsumeKafka processor has kafka.key attribute but I am not sure how to
read its contents(Since it is avro encoded) and add certain values as
attributes to flowfile. Any pointers will be much helpful.

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Reading avro encoded message key from kafka

Mike Thomsen
If you are having encoding-related issues with reading that attribute, try
switching to the Kafka string serializer in your producer.

On Fri, Nov 23, 2018 at 10:12 AM ashwin konale <[hidden email]>
wrote:

> Hi,
> I have key-value pair of avro messages in kafka topic I want to consume
> from. I can easily do modifications on message value using nifi
> consumeKafkaRecord processor, but it doesnt show key of the message.
> ConsumeKafka processor has kafka.key attribute but I am not sure how to
> read its contents(Since it is avro encoded) and add certain values as
> attributes to flowfile. Any pointers will be much helpful.
>
> Thanks
>
Reply | Threaded
Open this post in threaded view
|

Re: Reading avro encoded message key from kafka

Mike Thomsen
Configure the producer to encode the key with that, not the message
contents.

On Fri, Nov 23, 2018 at 10:14 AM Mike Thomsen <[hidden email]>
wrote:

> If you are having encoding-related issues with reading that attribute, try
> switching to the Kafka string serializer in your producer.
>
> On Fri, Nov 23, 2018 at 10:12 AM ashwin konale <[hidden email]>
> wrote:
>
>> Hi,
>> I have key-value pair of avro messages in kafka topic I want to consume
>> from. I can easily do modifications on message value using nifi
>> consumeKafkaRecord processor, but it doesnt show key of the message.
>> ConsumeKafka processor has kafka.key attribute but I am not sure how to
>> read its contents(Since it is avro encoded) and add certain values as
>> attributes to flowfile. Any pointers will be much helpful.
>>
>> Thanks
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Reading avro encoded message key from kafka

Ashwin Konale-2
In reply to this post by ashwin.konale@gmail.com
Hi,
Its not encoding issue. I am not able to figure out how to read Kafka key itself.
eg.
Kafka key = {type: foo, meta: etc, etc }
Kafka message = {Avro Payload}

I want to use RouteOnAttribute processor based on type = foo or bar. For this to happen, I need to extract value foo from Kafka.key to flow file.  Basically I am not able to figure out how to read key and extract attributes from it from Kafka message in nifi. Could you suggest me something here.

Thanks


On 2018/11/23 15:14:53, Mike Thomsen <[hidden email]> wrote:

> If you are having encoding-related issues with reading that attribute, try>
> switching to the Kafka string serializer in your producer.>
>
> On Fri, Nov 23, 2018 at 10:12 AM ashwin konale <[hidden email]>>
> wrote:>
>
> > Hi,>
> > I have key-value pair of avro messages in kafka topic I want to consume>
> > from. I can easily do modifications on message value using nifi>
> > consumeKafkaRecord processor, but it doesnt show key of the message.>
> > ConsumeKafka processor has kafka.key attribute but I am not sure how to>
> > read its contents(Since it is avro encoded) and add certain values as>
> > attributes to flowfile. Any pointers will be much helpful.>
> >>
> > Thanks>
> >>
>

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Reading avro encoded message key from kafka

Bryan Bende
Unfortunately I don't think there is a good way to interpret the value
of the key when it is Avro because we don't have any expression
language functions that understand Avro or record-oriented values.

The main option would be to change how the data is being produced in some way...

- Put the value you are interested in in a message header, then it
will come across as a string key/value pair in a flow file attribute
and use RouteOnAttrbute
- Put the value you are interested in in the message body somewhere,
use PartitionRecord to route on the value of the field in the message
- Use a different kind of key serialization like json which can then
be parsed with expression language functions

A possible improvement we could make is to add some kind of
"avro-to-json" EL function, then from there use the EL jsonPath
function.

On Tue, Nov 27, 2018 at 1:01 PM Ashwin Konale
<[hidden email]> wrote:

>
> Hi,
> Its not encoding issue. I am not able to figure out how to read Kafka key itself.
> eg.
> Kafka key = {type: foo, meta: etc, etc }
> Kafka message = {Avro Payload}
>
> I want to use RouteOnAttribute processor based on type = foo or bar. For this to happen, I need to extract value foo from Kafka.key to flow file.  Basically I am not able to figure out how to read key and extract attributes from it from Kafka message in nifi. Could you suggest me something here.
>
> Thanks
>
>
> On 2018/11/23 15:14:53, Mike Thomsen <[hidden email]> wrote:
> > If you are having encoding-related issues with reading that attribute, try>
> > switching to the Kafka string serializer in your producer.>
> >
> > On Fri, Nov 23, 2018 at 10:12 AM ashwin konale <[hidden email]>>
> > wrote:>
> >
> > > Hi,>
> > > I have key-value pair of avro messages in kafka topic I want to consume>
> > > from. I can easily do modifications on message value using nifi>
> > > consumeKafkaRecord processor, but it doesnt show key of the message.>
> > > ConsumeKafka processor has kafka.key attribute but I am not sure how to>
> > > read its contents(Since it is avro encoded) and add certain values as>
> > > attributes to flowfile. Any pointers will be much helpful.>
> > >>
> > > Thanks>
> > >>
> >
Reply | Threaded
Open this post in threaded view
|

Re: Reading avro encoded message key from kafka

Ashwin Konale-2
In reply to this post by ashwin.konale@gmail.com
Hi,
Thanks a lot for the suggestion. I didn’t know about jsonpath EL functions. I can easily implement that in my use case.

- Ashwin

On 2018/11/27 18:52:05, Bryan Bende <[hidden email]> wrote:

> Unfortunately I don't think there is a good way to interpret the value>
> of the key when it is Avro because we don't have any expression>
> language functions that understand Avro or record-oriented values.>
>
> The main option would be to change how the data is being produced in some way...>
>
> - Put the value you are interested in in a message header, then it>
> will come across as a string key/value pair in a flow file attribute>
> and use RouteOnAttrbute>
> - Put the value you are interested in in the message body somewhere,>
> use PartitionRecord to route on the value of the field in the message>
> - Use a different kind of key serialization like json which can then>
> be parsed with expression language functions>
>
> A possible improvement we could make is to add some kind of>
> "avro-to-json" EL function, then from there use the EL jsonPath>
> function.>
>
> On Tue, Nov 27, 2018 at 1:01 PM Ashwin Konale>
> <[hidden email]> wrote:>
> >>
> > Hi,>
> > Its not encoding issue. I am not able to figure out how to read Kafka key itself.>
> > eg.>
> > Kafka key = {type: foo, meta: etc, etc }>
> > Kafka message = {Avro Payload}>
> >>
> > I want to use RouteOnAttribute processor based on type = foo or bar. For this to happen, I need to extract value foo from Kafka.key to flow file.  Basically I am not able to figure out how to read key and extract attributes from it from Kafka message in nifi. Could you suggest me something here.>
> >>
> > Thanks>
> >>
> >>
> > On 2018/11/23 15:14:53, Mike Thomsen <[hidden email]> wrote:>
> > > If you are having encoding-related issues with reading that attribute, try>>
> > > switching to the Kafka string serializer in your producer.>>
> > >>
> > > On Fri, Nov 23, 2018 at 10:12 AM ashwin konale <[hidden email]>>>
> > > wrote:>>
> > >>
> > > > Hi,>>
> > > > I have key-value pair of avro messages in kafka topic I want to consume>>
> > > > from. I can easily do modifications on message value using nifi>>
> > > > consumeKafkaRecord processor, but it doesnt show key of the message.>>
> > > > ConsumeKafka processor has kafka.key attribute but I am not sure how to>>
> > > > read its contents(Since it is avro encoded) and add certain values as>>
> > > > attributes to flowfile. Any pointers will be much helpful.>>
> > > >>>
> > > > Thanks>>
> > > >>>
> > >>
>

smime.p7s (4K) Download Attachment