ListenTcpRecord

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ListenTcpRecord

Clay Teahouse
Hi All,

Does anyone have an example of ListenTcpRecord processor in action, say for
example, with a json or protobuf reader? I am specifically wondering about
the record length/prefix.

thanks
Clay
Reply | Threaded
Open this post in threaded view
|

Re: ListenTcpRecord

Bryan Bende
Hello,

I wrote a post that shows an example of using ListenTCPRecord with a
GrokReader to receive multi-line log messages. There is a link to a
template of the flow at the very end.

You could easily change the example so that PutTCP is sending a single
JSON document, or an array of JSON documents, and ListenTCPRecord is
using a JsonTreeReader.

We don't have a protobuf record reader so that currently isn't an option.

Let us know if you have any other questions.

-Bryan

[1] https://bryanbende.com/development/2017/10/04/apache-nifi-processing-multiline-logs

On Wed, Oct 4, 2017 at 8:39 AM, Clay Teahouse <[hidden email]> wrote:
> Hi All,
>
> Does anyone have an example of ListenTcpRecord processor in action, say for
> example, with a json or protobuf reader? I am specifically wondering about
> the record length/prefix.
>
> thanks
> Clay
Reply | Threaded
Open this post in threaded view
|

Re: ListenTcpRecord

Clay Teahouse
Thanks Bryan, for the feedback.

I don't seem to be able  to replicate the example in your blog. So, is the
flow  GenerateFlowFile -> TCPPUT --> LogAttribute and ListenTCPRecord
-->LogAttribute?  Both TCPPut and ListenTCPRecord  are sending data to
LogAttribute?  Shouldn't ListenTCPRecord be receiving the data data? Also,
I am not able to link ListenTCPRecord to any other processor.

In any case, I tried to send logs to ListenTCPRecord using netcat and have
ListenTCPRecord send the processed data to LogAttribute, but
ListenTCPRecord doesn't seem to be listening (I don't see any activity on
the processor). I'd appreciate if you let me what I am doing wrong.

thanks
Clay

On Wed, Oct 4, 2017 at 9:14 AM, Bryan Bende <[hidden email]> wrote:

> Hello,
>
> I wrote a post that shows an example of using ListenTCPRecord with a
> GrokReader to receive multi-line log messages. There is a link to a
> template of the flow at the very end.
>
> You could easily change the example so that PutTCP is sending a single
> JSON document, or an array of JSON documents, and ListenTCPRecord is
> using a JsonTreeReader.
>
> We don't have a protobuf record reader so that currently isn't an option.
>
> Let us know if you have any other questions.
>
> -Bryan
>
> [1] https://bryanbende.com/development/2017/10/04/apache-
> nifi-processing-multiline-logs
>
> On Wed, Oct 4, 2017 at 8:39 AM, Clay Teahouse <[hidden email]>
> wrote:
> > Hi All,
> >
> > Does anyone have an example of ListenTcpRecord processor in action, say
> for
> > example, with a json or protobuf reader? I am specifically wondering
> about
> > the record length/prefix.
> >
> > thanks
> > Clay
>
Reply | Threaded
Open this post in threaded view
|

Re: ListenTcpRecord

Bryan Bende
Have you tried using the template?

https://gist.githubusercontent.com/bbende/fa2bff34e721fef21453986336664cb2/raw/db658c64f75fec47785ab63920ee23582bf1492f/multi_line_log_processing.xml

If you import that it will give you the exact flow from my post and
all you have to do is start everything.

PutTCP is sending to ListenTCP, the connection coming out of PutTCP to
LogAttribute is for the failure relationship just to easily see if
anything fails.

-Bryan


On Thu, Oct 5, 2017 at 7:05 AM, Clay Teahouse <[hidden email]> wrote:

> Thanks Bryan, for the feedback.
>
> I don't seem to be able  to replicate the example in your blog. So, is the
> flow  GenerateFlowFile -> TCPPUT --> LogAttribute and ListenTCPRecord
> -->LogAttribute?  Both TCPPut and ListenTCPRecord  are sending data to
> LogAttribute?  Shouldn't ListenTCPRecord be receiving the data data? Also,
> I am not able to link ListenTCPRecord to any other processor.
>
> In any case, I tried to send logs to ListenTCPRecord using netcat and have
> ListenTCPRecord send the processed data to LogAttribute, but
> ListenTCPRecord doesn't seem to be listening (I don't see any activity on
> the processor). I'd appreciate if you let me what I am doing wrong.
>
> thanks
> Clay
>
> On Wed, Oct 4, 2017 at 9:14 AM, Bryan Bende <[hidden email]> wrote:
>
>> Hello,
>>
>> I wrote a post that shows an example of using ListenTCPRecord with a
>> GrokReader to receive multi-line log messages. There is a link to a
>> template of the flow at the very end.
>>
>> You could easily change the example so that PutTCP is sending a single
>> JSON document, or an array of JSON documents, and ListenTCPRecord is
>> using a JsonTreeReader.
>>
>> We don't have a protobuf record reader so that currently isn't an option.
>>
>> Let us know if you have any other questions.
>>
>> -Bryan
>>
>> [1] https://bryanbende.com/development/2017/10/04/apache-
>> nifi-processing-multiline-logs
>>
>> On Wed, Oct 4, 2017 at 8:39 AM, Clay Teahouse <[hidden email]>
>> wrote:
>> > Hi All,
>> >
>> > Does anyone have an example of ListenTcpRecord processor in action, say
>> for
>> > example, with a json or protobuf reader? I am specifically wondering
>> about
>> > the record length/prefix.
>> >
>> > thanks
>> > Clay
>>
Reply | Threaded
Open this post in threaded view
|

Re: ListenTcpRecord

Clay Teahouse
Thank you, Bryan. I was able to set up the flow using your template and
then fix my own.

 Can ListenTCPRecord deal with records spread across multiple packets? More
specifically, can it handle payloads prefixed with the length? I receive
JSON and protobuf messages from a client and each record is prefixed with
the length of the message.

-Clay

On Thu, Oct 5, 2017 at 8:22 AM, Bryan Bende <[hidden email]> wrote:

> Have you tried using the template?
>
> https://gist.githubusercontent.com/bbende/fa2bff34e721fef21453986336664c
> b2/raw/db658c64f75fec47785ab63920ee23582bf1492f/multi_line_log_
> processing.xml
>
> If you import that it will give you the exact flow from my post and
> all you have to do is start everything.
>
> PutTCP is sending to ListenTCP, the connection coming out of PutTCP to
> LogAttribute is for the failure relationship just to easily see if
> anything fails.
>
> -Bryan
>
>
> On Thu, Oct 5, 2017 at 7:05 AM, Clay Teahouse <[hidden email]>
> wrote:
> > Thanks Bryan, for the feedback.
> >
> > I don't seem to be able  to replicate the example in your blog. So, is
> the
> > flow  GenerateFlowFile -> TCPPUT --> LogAttribute and ListenTCPRecord
> > -->LogAttribute?  Both TCPPut and ListenTCPRecord  are sending data to
> > LogAttribute?  Shouldn't ListenTCPRecord be receiving the data data?
> Also,
> > I am not able to link ListenTCPRecord to any other processor.
> >
> > In any case, I tried to send logs to ListenTCPRecord using netcat and
> have
> > ListenTCPRecord send the processed data to LogAttribute, but
> > ListenTCPRecord doesn't seem to be listening (I don't see any activity on
> > the processor). I'd appreciate if you let me what I am doing wrong.
> >
> > thanks
> > Clay
> >
> > On Wed, Oct 4, 2017 at 9:14 AM, Bryan Bende <[hidden email]> wrote:
> >
> >> Hello,
> >>
> >> I wrote a post that shows an example of using ListenTCPRecord with a
> >> GrokReader to receive multi-line log messages. There is a link to a
> >> template of the flow at the very end.
> >>
> >> You could easily change the example so that PutTCP is sending a single
> >> JSON document, or an array of JSON documents, and ListenTCPRecord is
> >> using a JsonTreeReader.
> >>
> >> We don't have a protobuf record reader so that currently isn't an
> option.
> >>
> >> Let us know if you have any other questions.
> >>
> >> -Bryan
> >>
> >> [1] https://bryanbende.com/development/2017/10/04/apache-
> >> nifi-processing-multiline-logs
> >>
> >> On Wed, Oct 4, 2017 at 8:39 AM, Clay Teahouse <[hidden email]>
> >> wrote:
> >> > Hi All,
> >> >
> >> > Does anyone have an example of ListenTcpRecord processor in action,
> say
> >> for
> >> > example, with a json or protobuf reader? I am specifically wondering
> >> about
> >> > the record length/prefix.
> >> >
> >> > thanks
> >> > Clay
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: ListenTcpRecord

Bryan Bende
Clay,

Multiple packets should not be an issue since it is reading a stream of data from the socket, but I don’t think the prefixed length will work.

The data coming across has to be in a format that one of the record readers can understand. If you have JSON data and then have something additional added to it like the length, then its not going to be readable by a standard JSON library.

You could potentially create a custom record reader to handle your format.

-Bryan

> On Oct 7, 2017, at 10:37 AM, Clay Teahouse <[hidden email]> wrote:
>
> Thank you, Bryan. I was able to set up the flow using your template and
> then fix my own.
>
> Can ListenTCPRecord deal with records spread across multiple packets? More
> specifically, can it handle payloads prefixed with the length? I receive
> JSON and protobuf messages from a client and each record is prefixed with
> the length of the message.
>
> -Clay
>
> On Thu, Oct 5, 2017 at 8:22 AM, Bryan Bende <[hidden email]> wrote:
>
>> Have you tried using the template?
>>
>> https://gist.githubusercontent.com/bbende/fa2bff34e721fef21453986336664c
>> b2/raw/db658c64f75fec47785ab63920ee23582bf1492f/multi_line_log_
>> processing.xml
>>
>> If you import that it will give you the exact flow from my post and
>> all you have to do is start everything.
>>
>> PutTCP is sending to ListenTCP, the connection coming out of PutTCP to
>> LogAttribute is for the failure relationship just to easily see if
>> anything fails.
>>
>> -Bryan
>>
>>
>> On Thu, Oct 5, 2017 at 7:05 AM, Clay Teahouse <[hidden email]>
>> wrote:
>>> Thanks Bryan, for the feedback.
>>>
>>> I don't seem to be able  to replicate the example in your blog. So, is
>> the
>>> flow  GenerateFlowFile -> TCPPUT --> LogAttribute and ListenTCPRecord
>>> -->LogAttribute?  Both TCPPut and ListenTCPRecord  are sending data to
>>> LogAttribute?  Shouldn't ListenTCPRecord be receiving the data data?
>> Also,
>>> I am not able to link ListenTCPRecord to any other processor.
>>>
>>> In any case, I tried to send logs to ListenTCPRecord using netcat and
>> have
>>> ListenTCPRecord send the processed data to LogAttribute, but
>>> ListenTCPRecord doesn't seem to be listening (I don't see any activity on
>>> the processor). I'd appreciate if you let me what I am doing wrong.
>>>
>>> thanks
>>> Clay
>>>
>>> On Wed, Oct 4, 2017 at 9:14 AM, Bryan Bende <[hidden email]> wrote:
>>>
>>>> Hello,
>>>>
>>>> I wrote a post that shows an example of using ListenTCPRecord with a
>>>> GrokReader to receive multi-line log messages. There is a link to a
>>>> template of the flow at the very end.
>>>>
>>>> You could easily change the example so that PutTCP is sending a single
>>>> JSON document, or an array of JSON documents, and ListenTCPRecord is
>>>> using a JsonTreeReader.
>>>>
>>>> We don't have a protobuf record reader so that currently isn't an
>> option.
>>>>
>>>> Let us know if you have any other questions.
>>>>
>>>> -Bryan
>>>>
>>>> [1] https://bryanbende.com/development/2017/10/04/apache-
>>>> nifi-processing-multiline-logs
>>>>
>>>> On Wed, Oct 4, 2017 at 8:39 AM, Clay Teahouse <[hidden email]>
>>>> wrote:
>>>>> Hi All,
>>>>>
>>>>> Does anyone have an example of ListenTcpRecord processor in action,
>> say
>>>> for
>>>>> example, with a json or protobuf reader? I am specifically wondering
>>>> about
>>>>> the record length/prefix.
>>>>>
>>>>> thanks
>>>>> Clay
>>>>
>>


signature.asc (849 bytes) Download Attachment