File.listFiles() and large directories. [NIFI-3423]

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

File.listFiles() and large directories. [NIFI-3423]

Benjamin Wood
Has anyone looked into this?
https://issues.apache.org/jira/browse/NIFI-3423

I found a lot of stuff suggesting to replace java.io.File with java.nio.file

Seems like "newDirectoryStream" could achieve what this JIRA is asking.

Here is an example from SO:
https://stackoverflow.com/questions/31706058/get-large-directory-content-faster-java-io-file-alternatives

I actually just encountered this problem when I (stupidly) tried to run
ListFiles processor on a directory with 500,000+ files in it. Oops!

Thanks,
Ben
Reply | Threaded
Open this post in threaded view
|

Re: File.listFiles() and large directories. [NIFI-3423]

Joe Witt
ben

I think you're right that we should switch and don't think anyone is
actively working it.

Thanks

On Fri, Sep 29, 2017 at 5:58 PM, Benjamin Wood <[hidden email]> wrote:

> Has anyone looked into this?
> https://issues.apache.org/jira/browse/NIFI-3423
>
> I found a lot of stuff suggesting to replace java.io.File with java.nio.file
>
> Seems like "newDirectoryStream" could achieve what this JIRA is asking.
>
> Here is an example from SO:
> https://stackoverflow.com/questions/31706058/get-large-directory-content-faster-java-io-file-alternatives
>
> I actually just encountered this problem when I (stupidly) tried to run
> ListFiles processor on a directory with 500,000+ files in it. Oops!
>
> Thanks,
> Ben