ListFile is slow when it scan NFS mount

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

ListFile is slow when it scan NFS mount

Milan Das
Hello All,

We are using NIFI 1.5 for one of the client. Seems like Listfile is super slow when it scan a directory hosted on NFS mount.

Performance on `ls` seems to be around 200 files/sec. But Listfile takes 20 mins to list 200 files.

 

Is there any known issue or custom settings need to be conjured on ListS3 ?

 

Thanks,

Milan Das

 

 

Reply | Threaded
Open this post in threaded view
|

Re: ListFile is slow when it scan NFS mount

Joe Witt
If you look at https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.1/org.apache.nifi.processors.standard.ListFile/index.html

The property 'List File Attributes' should be false or else
performance will be slow.  If that wasn't in NiFi 1.5 you'll want to
upgrade.

Thanks
On Thu, Sep 20, 2018 at 2:48 PM Milan Das <[hidden email]> wrote:

>
> Hello All,
>
> We are using NIFI 1.5 for one of the client. Seems like Listfile is super slow when it scan a directory hosted on NFS mount.
>
> Performance on `ls` seems to be around 200 files/sec. But Listfile takes 20 mins to list 200 files.
>
>
>
> Is there any known issue or custom settings need to be conjured on ListS3 ?
>
>
>
> Thanks,
>
> Milan Das
>
>
>
>
>