puts3object super slow

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

puts3object super slow

pradeepbill
Hello folks,
I have a listenSyslogProcessor -> puts3object processor, and our network bandwidth is 1GBPS, but I see a super slow performance on the s3 writes,our s3 bucket has no policy, here are the stats

Total Size written – 17MB
Total Time for Queue to empty – 21 mins
Min/Max/Mean – 0.0/9.29MB/1.08MB

Its absolutely unusable, I dont know where the slowness is , either the puts3processor api?, or network ? , or environment ? .Where can I debug, the nifi-app.log has no errors/exceptions.Please advice.

Thanks
Pradeep
Reply | Threaded
Open this post in threaded view
|

Re: puts3object super slow

Bryan Rosander-3
Hi Pradeep,

How does the upload speed compare to uploading without NiFi?  Maybe try the
enhanced uploader [1] to the same bucket from the same machine to see if
the problem is internal to NiFi or maybe due to something else?

[1] http://docs.aws.amazon.com/AmazonS3/latest/UG/enhanced-uploader.html

Thanks,
Bryan

On Tue, Apr 25, 2017 at 4:41 PM, pradeepbill <[hidden email]> wrote:

> Hello folks,
> I have a listenSyslogProcessor -> puts3object processor, and our network
> bandwidth is 1GBPS, but I see a super slow performance on the s3 writes,our
> s3 bucket has no policy, here are the stats
>
> Total Size written – 17MB
> Total Time for Queue to empty – 21 mins
> Min/Max/Mean – 0.0/9.29MB/1.08MB
>
> Its absolutely unusable, I dont know where the slowness is , either the
> puts3processor api?, or network ? , or environment ? .Where can I debug,
> the
> nifi-app.log has no errors/exceptions.Please advice.
>
> Thanks
> Pradeep
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/puts3object-super-slow-tp15554.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: puts3object super slow

pradeepbill
thanks Bryan, i took your hint, and looks like the network might be the culprit.I did a test and 95M took almost an hour.