Fault tolerance

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Fault tolerance

Toivo Adams
From Joe mail Dec 17, 2014; 6:14am Re: NiFi architecture
„Fault tolerance:  If a node dies the data on the node at this time is 'as
dead as the node'. “

Dow I understand correctly data in queues (waiting processed by processors) are lost?

Are repositories implementations pluggable?
Is it possible to implement Flowfile repository and content repository using fault tolerant data store – for example Cassandra?
Can this help to avoid losing data in queues?

Thanks
Toivo
Reply | Threaded
Open this post in threaded view
|

Re: Fault tolerance

Mark Payne
Toivo,


The data in the queues would be lost if the disk were completely lost. Generally, for a production flow, we would highly recommend using a RAID storage device that provides redundancy. This way, if a disk goes bad the data is still available.


The repositories are pluggable, but there are currently only two implementations available: Volatile (in-memory) and File-backed storage. You could also back the repository by a NAS device. I’ve experimented with doing this in EC2, for instance, with an EBS drive, and everything worked well with really good throughput. In this way, if a node were lost, the EBS instance can simply be mapped to a new node, and all is good.


Thanks

-Mark






From: [hidden email]
Sent: ‎Friday‎, ‎March‎ ‎20‎, ‎2015 ‎11‎:‎27‎ ‎AM
To: [hidden email]





From Joe mail Dec 17, 2014; 6:14am Re: NiFi architecture
„Fault tolerance:  If a node dies the data on the node at this time is 'as
dead as the node'. “

Dow I understand correctly data in queues (waiting processed by processors)
are lost?

Are repositories implementations pluggable?
Is it possible to implement Flowfile repository and content repository using
fault tolerant data store – for example Cassandra?
Can this help to avoid losing data in queues?

Thanks
Toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Fault-tolerance-tp1023.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.