Error log in ListenTCP

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Error log in ListenTCP

Rajesh Biswas
Hello Team,

I am facing one issue to get binary data into Nifi.

As of now I am using ListenTCP processor, when I am sending character string
data I am able to see the data in flow file.

But when I am trying to send Binary data, I am not able to see any data thru
Data province.

Would you please guide me:

1.       Which log files to see if there are any error (I checked
nifi-app.log and nifi-user.log, did not see anything)

2.       Which processor to use to read binary data thru TCP protocol (I am
not able to change Character Set other than UTF-8)

 

Would you please help me with your experience?

 

Thanks and Regards,

Rajesh Biswas | +91 9886433461 |  <http://www.bridgera.com/>
www.bridgera.com

 



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Reply | Threaded
Open this post in threaded view
|

Re: Error log in ListenTCP

Bryan Bende
ListenTCP uses new-line characters to determine logical message boundaries,
so the data itself does not have to be characters, but there has to be
new-lines indicating the end of a logical piece of data.

Imagine streaming the content of several images over the same connection,
the client would have to send something like the following... bytes of
image 1, new line, bytes of image 2, new line, bytes of image 3, new line,
etc.


On Mon, Oct 8, 2018 at 1:52 PM Rajesh Biswas <[hidden email]>
wrote:

> Hello Team,
>
> I am facing one issue to get binary data into Nifi.
>
> As of now I am using ListenTCP processor, when I am sending character
> string data I am able to see the data in flow file.
>
> But when I am trying to send Binary data, I am not able to see any data
> thru Data province.
>
> Would you please guide me:
>
> 1.       Which log files to see if there are any error (I checked
> nifi-app.log and nifi-user.log, did not see anything)
>
> 2.       Which processor to use to read binary data thru TCP protocol (I
> am not able to change Character Set other than UTF-8)
>
>
>
> Would you please help me with your experience?
>
>
>
> Thanks and Regards,
>
> *Rajesh Biswas* | +91 9886433461 | www.bridgera.com
>
>
>
>
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29> I’m
> protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29>
> <#m_-3931440512116331171_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Reply | Threaded
Open this post in threaded view
|

RE: Error log in ListenTCP

Rajesh Biswas
Hello Bryan,

We need to read binary stream data thru TCP.
As the default chatacter encoding is UTF-8 for ListenTCP, the binary data is getting altered after passing through the processors.
Would you please suggest any other alternative?
Below is the sample data:
"00010001990010000015B00100019900101001"
The processor group we are using
ListenTCP -> PutFile

Configuration for ListenTCP
Local Network InterfaceNo value set  No Value
Port 9111
Receive Buffer Size 65507 B
Max Size of Message Queue 10000
Max Size of Socket Buffer 1 MB
Character Set UTF-8
Max Batch Size 1
Batching Message Delimiter \n
Max Number of TCP Connections 2
SSL Context ServiceNo value set
Client Auth NONE


Thank you very much:


Thanks and Regards,
Rajesh Biswas | +91 9886433461 | www.bridgera.com

-----Original Message-----
From: Bryan Bende [mailto:[hidden email]]
Sent: Tuesday, October 9, 2018 1:20 AM
To: [hidden email]
Subject: Re: Error log in ListenTCP

ListenTCP uses new-line characters to determine logical message boundaries,
so the data itself does not have to be characters, but there has to be
new-lines indicating the end of a logical piece of data.

Imagine streaming the content of several images over the same connection,
the client would have to send something like the following... bytes of
image 1, new line, bytes of image 2, new line, bytes of image 3, new line,
etc.


On Mon, Oct 8, 2018 at 1:52 PM Rajesh Biswas <[hidden email]>
wrote:

> Hello Team,
>
> I am facing one issue to get binary data into Nifi.
>
> As of now I am using ListenTCP processor, when I am sending character
> string data I am able to see the data in flow file.
>
> But when I am trying to send Binary data, I am not able to see any data
> thru Data province.
>
> Would you please guide me:
>
> 1.       Which log files to see if there are any error (I checked
> nifi-app.log and nifi-user.log, did not see anything)
>
> 2.       Which processor to use to read binary data thru TCP protocol (I
> am not able to change Character Set other than UTF-8)
>
>
>
> Would you please help me with your experience?
>
>
>
> Thanks and Regards,
>
> *Rajesh Biswas* | +91 9886433461 | www.bridgera.com
>
>
>
>
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29> I’m
> protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29>
> <#m_-3931440512116331171_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Reply | Threaded
Open this post in threaded view
|

Re: Error log in ListenTCP

Bryan Bende
ListenTCP only uses the Character Set property for writing the
demarcator to the output stream of the flow file, it takes the String
value of Character Set and calls value.getBytes(charset).

For the actual data it is never converted from bytes to characters, so
it takes byte[] from the incoming stream and writes byte[] directly to
the flow file output stream.
On Tue, Oct 9, 2018 at 11:17 AM Rajesh Biswas
<[hidden email]> wrote:

>
> Hello Bryan,
>
> We need to read binary stream data thru TCP.
> As the default chatacter encoding is UTF-8 for ListenTCP, the binary data is getting altered after passing through the processors.
> Would you please suggest any other alternative?
> Below is the sample data:
> "00010001990010000015B00100019900101001"
> The processor group we are using
> ListenTCP -> PutFile
>
> Configuration for ListenTCP
> Local Network InterfaceNo value set  No Value
> Port 9111
> Receive Buffer Size 65507 B
> Max Size of Message Queue 10000
> Max Size of Socket Buffer 1 MB
> Character Set UTF-8
> Max Batch Size 1
> Batching Message Delimiter \n
> Max Number of TCP Connections 2
> SSL Context ServiceNo value set
> Client Auth NONE
>
>
> Thank you very much:
>
>
> Thanks and Regards,
> Rajesh Biswas | +91 9886433461 | www.bridgera.com
>
> -----Original Message-----
> From: Bryan Bende [mailto:[hidden email]]
> Sent: Tuesday, October 9, 2018 1:20 AM
> To: [hidden email]
> Subject: Re: Error log in ListenTCP
>
> ListenTCP uses new-line characters to determine logical message boundaries,
> so the data itself does not have to be characters, but there has to be
> new-lines indicating the end of a logical piece of data.
>
> Imagine streaming the content of several images over the same connection,
> the client would have to send something like the following... bytes of
> image 1, new line, bytes of image 2, new line, bytes of image 3, new line,
> etc.
>
>
> On Mon, Oct 8, 2018 at 1:52 PM Rajesh Biswas <[hidden email]>
> wrote:
>
> > Hello Team,
> >
> > I am facing one issue to get binary data into Nifi.
> >
> > As of now I am using ListenTCP processor, when I am sending character
> > string data I am able to see the data in flow file.
> >
> > But when I am trying to send Binary data, I am not able to see any data
> > thru Data province.
> >
> > Would you please guide me:
> >
> > 1.       Which log files to see if there are any error (I checked
> > nifi-app.log and nifi-user.log, did not see anything)
> >
> > 2.       Which processor to use to read binary data thru TCP protocol (I
> > am not able to change Character Set other than UTF-8)
> >
> >
> >
> > Would you please help me with your experience?
> >
> >
> >
> > Thanks and Regards,
> >
> > *Rajesh Biswas* | +91 9886433461 | www.bridgera.com
> >
> >
> >
> >
> > <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29> I’m
> > protected online with Avast Free Antivirus. Get it here — it’s free
> > forever.
> > <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29>
> > <#m_-3931440512116331171_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>