using avro flume source in nifi

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

using avro flume source in nifi

Chris Herssens
Hello All,

I want to configure a flume source processor using avro as flume source.
When I start the ExecuteFlumeSource, I don't get any errors, but I can't
connect to the source port.

telnet localhost 5151 gives connection refused.

The configuration is as follows

type = avro

agent name = a1

source name = batchSource

Flume configuration

   a1.sources.batchSource.type = avro

   a1.sources.batchSource.bind = 0.0.0.0

   a1.sources.batchSource.port=5151


Using netcat as flume source works.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: using avro flume source in nifi

Joe Witt
Chris,

I dont believe there are many in the community here familiar enough
with Flume so that might make getting help here a bit tougher.  Are
you wanting NiFi to act as a TCP socket listener on a given port
receiving Avro objects?

Thanks

On Thu, Aug 3, 2017 at 11:22 PM, Chris Herssens
<[hidden email]> wrote:

> Hello All,
>
> I want to configure a flume source processor using avro as flume source.
> When I start the ExecuteFlumeSource, I don't get any errors, but I can't
> connect to the source port.
>
> telnet localhost 5151 gives connection refused.
>
> The configuration is as follows
>
> type = avro
>
> agent name = a1
>
> source name = batchSource
>
> Flume configuration
>
>    a1.sources.batchSource.type = avro
>
>    a1.sources.batchSource.bind = 0.0.0.0
>
>    a1.sources.batchSource.port=5151
>
>
> Using netcat as flume source works.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: using avro flume source in nifi

Chris Herssens
Hello Joe,

Currently we use Flume for sending avro message to our big data platform.
Now I want to replace flume with NIfi. Since the sender is a  flume agent,
I need to use the flume source.
If I configure this sourcs for receiving netcat, I can access the tcp port.
Using avro as type doesn't work. If I set "run schedule" ti 60 sec, then it
work, I can telnet to the configured tcp port.

Regards,

Chris

On Fri, Aug 11, 2017 at 4:03 AM, Joe Witt <[hidden email]> wrote:

> Chris,
>
> I dont believe there are many in the community here familiar enough
> with Flume so that might make getting help here a bit tougher.  Are
> you wanting NiFi to act as a TCP socket listener on a given port
> receiving Avro objects?
>
> Thanks
>
> On Thu, Aug 3, 2017 at 11:22 PM, Chris Herssens
> <[hidden email]> wrote:
> > Hello All,
> >
> > I want to configure a flume source processor using avro as flume source.
> > When I start the ExecuteFlumeSource, I don't get any errors, but I can't
> > connect to the source port.
> >
> > telnet localhost 5151 gives connection refused.
> >
> > The configuration is as follows
> >
> > type = avro
> >
> > agent name = a1
> >
> > source name = batchSource
> >
> > Flume configuration
> >
> >    a1.sources.batchSource.type = avro
> >
> >    a1.sources.batchSource.bind = 0.0.0.0
> >
> >    a1.sources.batchSource.port=5151
> >
> >
> > Using netcat as flume source works.
>
Loading...