Blog post

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Blog post

Joey Echeverria
I wrote a quick getting started blog post on NiFi for the ingest.tips blog here:

http://ingest.tips/2014/12/22/getting-started-with-apache-nifi/


I focused on intefration with Hadoop and the flexibility of NiFi to integrate existing ETL flows and tools. 


-Joey


Joey Echeverria
Reply | Threaded
Open this post in threaded view
|

RE: Blog post

Mark Payne
Joey,
Nice - thanks for putting that together & sending out!
Only feedback I would give is that for the PutHDFS, in the typical use case you wouldn't want to Auto-terminate the failure relationship. Instead, you'd route the failure relationship back to the PutHDFS Processor. This way, if NiFi can't push the file to HDFS (network problem, typo in config file, whatever), NiFi will be able to retry until it's successful.
Thanks-Mark

> Date: Mon, 22 Dec 2014 10:06:28 -0800
> From: [hidden email]
> To: [hidden email]
> Subject: Blog post
>
> I wrote a quick getting started blog post on NiFi for the ingest.tips blog here:
>
> http://ingest.tips/2014/12/22/getting-started-with-apache-nifi/
>
>
> I focused on intefration with Hadoop and the flexibility of NiFi to integrate existing ETL flows and tools.
>
>
> -Joey
>
> —
> Joey Echeverria
     
Reply | Threaded
Open this post in threaded view
|

RE: Blog post

Joey Echeverria
Neat, I hadn't thought of using that as a retry mechanism. Is there any way to tell how often a file has cycled through failure so you could distinguish between a transient failure and a permanent failure, like a permissions issue?


-Joey



Joey Echeverria

On Mon, Dec 22, 2014 at 10:18 AM, Mark Payne <[hidden email]> wrote:

> Joey,
> Nice - thanks for putting that together & sending out!
> Only feedback I would give is that for the PutHDFS, in the typical use case you wouldn't want to Auto-terminate the failure relationship. Instead, you'd route the failure relationship back to the PutHDFS Processor. This way, if NiFi can't push the file to HDFS (network problem, typo in config file, whatever), NiFi will be able to retry until it's successful.
> Thanks-Mark
>> Date: Mon, 22 Dec 2014 10:06:28 -0800
>> From: [hidden email]
>> To: [hidden email]
>> Subject: Blog post
>>
>> I wrote a quick getting started blog post on NiFi for the ingest.tips blog here:
>>
>> http://ingest.tips/2014/12/22/getting-started-with-apache-nifi/
>>
>>
>> I focused on intefration with Hadoop and the flexibility of NiFi to integrate existing ETL flows and tools.
>>
>>
>> -Joey
>>
>> —
>> Joey Echeverria
>      
Reply | Threaded
Open this post in threaded view
|

Re: Blog post

Joe Witt
Joey,

On every connection and process now you can right-click and see stats.
Those stats reveal how often things are failing, etc.  Right now you'd not
be able to distinguish between a permissions issue as the cause and
something more transient.  But, we should put the failure reason as an
attribute of a flow file so one could route on that should the distinction
help them make a decision or better track.

Thanks
JOe

On Mon, Dec 22, 2014 at 1:21 PM, Joey Echeverria <[hidden email]> wrote:

> Neat, I hadn't thought of using that as a retry mechanism. Is there any
> way to tell how often a file has cycled through failure so you could
> distinguish between a transient failure and a permanent failure, like a
> permissions issue?
>
>
> -Joey
>
>
> —
> Joey Echeverria
>
> On Mon, Dec 22, 2014 at 10:18 AM, Mark Payne <[hidden email]> wrote:
>
> > Joey,
> > Nice - thanks for putting that together & sending out!
> > Only feedback I would give is that for the PutHDFS, in the typical use
> case you wouldn't want to Auto-terminate the failure relationship. Instead,
> you'd route the failure relationship back to the PutHDFS Processor. This
> way, if NiFi can't push the file to HDFS (network problem, typo in config
> file, whatever), NiFi will be able to retry until it's successful.
> > Thanks-Mark
> >> Date: Mon, 22 Dec 2014 10:06:28 -0800
> >> From: [hidden email]
> >> To: [hidden email]
> >> Subject: Blog post
> >>
> >> I wrote a quick getting started blog post on NiFi for the ingest.tips
> blog here:
> >>
> >> http://ingest.tips/2014/12/22/getting-started-with-apache-nifi/
> >>
> >>
> >> I focused on intefration with Hadoop and the flexibility of NiFi to
> integrate existing ETL flows and tools.
> >>
> >>
> >> -Joey
> >>
> >> —
> >> Joey Echeverria
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: Blog post

johny casanova
In reply to this post by Joey Echeverria
Hey Joey,

 

what tools you use to make the blogs? thats a nice site. :)
 

> Date: Mon, 22 Dec 2014 10:06:28 -0800
> From: [hidden email]
> To: [hidden email]
> Subject: Blog post
>
> I wrote a quick getting started blog post on NiFi for the ingest.tips blog here:
>
> http://ingest.tips/2014/12/22/getting-started-with-apache-nifi/
>
>
> I focused on intefration with Hadoop and the flexibility of NiFi to integrate existing ETL flows and tools.
>
>
> -Joey
>
> —
> Joey Echeverria
     
Reply | Threaded
Open this post in threaded view
|

Re: Blog post

Joey Echeverria
It's a wordpress site. I'm not sure what was used to make the skin or
if it's just a wordpress default.

FYI, anyone that has interesting ingest/data integration related
topics is welcome to contribute to that blog. Just shoot me an e-mail.

-Joey

On Mon, Dec 22, 2014 at 1:24 PM, johny casanova <[hidden email]> wrote:

> Hey Joey,
>
>
>
> what tools you use to make the blogs? thats a nice site. :)
>
>
>> Date: Mon, 22 Dec 2014 10:06:28 -0800
>> From: [hidden email]
>> To: [hidden email]
>> Subject: Blog post
>>
>> I wrote a quick getting started blog post on NiFi for the ingest.tips blog here:
>>
>> http://ingest.tips/2014/12/22/getting-started-with-apache-nifi/
>>
>>
>> I focused on intefration with Hadoop and the flexibility of NiFi to integrate existing ETL flows and tools.
>>
>>
>> -Joey
>>
>> —
>> Joey Echeverria
>



--
Joey Echeverria