java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

shweta
Hi All,

I'm getting a java.lang.UnsatisfiedLinkError while adding data into PutHDFS processor with compression codec as snappy. The error message says "Failed to write to HDFS due to org.apache.hadoop.util.NativeCodeloader.build.SupportsSnappy()Z.

Inspite of this error, .snappy files are being written in my Hdfs.

Has anyone faced a similar issue before or can provide any pointers.

Thanks,
Shweta
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

Joe Witt
Can you show what is in your core-site.xml and the proc properties.
Also can you show the full log output?

Thanks
Joe

On Sat, Feb 6, 2016 at 9:11 AM, shweta <[hidden email]> wrote:

> Hi All,
>
> I'm getting a java.lang.UnsatisfiedLinkError while adding data into PutHDFS
> processor with compression codec as snappy. The error message says "Failed
> to write to HDFS due to
> org.apache.hadoop.util.NativeCodeloader.build.SupportsSnappy()Z.
>
> Inspite of this error, .snappy files are being written in my Hdfs.
>
> Has anyone faced a similar issue before or can provide any pointers.
>
> Thanks,
> Shweta
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/java-lang-UnsatisfiedLinkError-in-PutHDFS-with-snappy-compression-tp7182.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

Jeremy Dyer
Shweta,

Looks like your missing the snappy native library. I have seen this several
times before. Assuming your on a linux machine you have 2 options. You can
copy the libsnappy.so native library to your JAVA_HOME/jre/lib native
directory. Or you can set LD_LIBRARY_PATH to point to where your
libsnappy.so native library is located on the machine.

I believe if you closely examine the files that are being written to HDFS
with a .snappy extension you will see that in fact that are not actually
snappy compressed.

Jeremy Dyer

On Sat, Feb 6, 2016 at 1:04 PM, Joe Witt <[hidden email]> wrote:

> Can you show what is in your core-site.xml and the proc properties.
> Also can you show the full log output?
>
> Thanks
> Joe
>
> On Sat, Feb 6, 2016 at 9:11 AM, shweta <[hidden email]> wrote:
> > Hi All,
> >
> > I'm getting a java.lang.UnsatisfiedLinkError while adding data into
> PutHDFS
> > processor with compression codec as snappy. The error message says
> "Failed
> > to write to HDFS due to
> > org.apache.hadoop.util.NativeCodeloader.build.SupportsSnappy()Z.
> >
> > Inspite of this error, .snappy files are being written in my Hdfs.
> >
> > Has anyone faced a similar issue before or can provide any pointers.
> >
> > Thanks,
> > Shweta
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/java-lang-UnsatisfiedLinkError-in-PutHDFS-with-snappy-compression-tp7182.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

Matt Burgess
To add to Jeremy's last point, even after the library is present, the files must be greater than the HDFS block size (default is 64 MB I think?) or Hadoop-snappy will also not compress them.

Sent from my iPhone

> On Feb 6, 2016, at 5:41 PM, Jeremy Dyer <[hidden email]> wrote:
>
> Shweta,
>
> Looks like your missing the snappy native library. I have seen this several
> times before. Assuming your on a linux machine you have 2 options. You can
> copy the libsnappy.so native library to your JAVA_HOME/jre/lib native
> directory. Or you can set LD_LIBRARY_PATH to point to where your
> libsnappy.so native library is located on the machine.
>
> I believe if you closely examine the files that are being written to HDFS
> with a .snappy extension you will see that in fact that are not actually
> snappy compressed.
>
> Jeremy Dyer
>
>> On Sat, Feb 6, 2016 at 1:04 PM, Joe Witt <[hidden email]> wrote:
>>
>> Can you show what is in your core-site.xml and the proc properties.
>> Also can you show the full log output?
>>
>> Thanks
>> Joe
>>
>>> On Sat, Feb 6, 2016 at 9:11 AM, shweta <[hidden email]> wrote:
>>> Hi All,
>>>
>>> I'm getting a java.lang.UnsatisfiedLinkError while adding data into
>> PutHDFS
>>> processor with compression codec as snappy. The error message says
>> "Failed
>>> to write to HDFS due to
>>> org.apache.hadoop.util.NativeCodeloader.build.SupportsSnappy()Z.
>>>
>>> Inspite of this error, .snappy files are being written in my Hdfs.
>>>
>>> Has anyone faced a similar issue before or can provide any pointers.
>>>
>>> Thanks,
>>> Shweta
>>>
>>>
>>>
>>> --
>>> View this message in context:
>> http://apache-nifi-developer-list.39713.n7.nabble.com/java-lang-UnsatisfiedLinkError-in-PutHDFS-with-snappy-compression-tp7182.html
>>> Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

shweta
In reply to this post by Joe Witt
Hi Joe,

Please find the contents of core-site.xml  which contains following property.

<property>
  <name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

mapred-site.xml

 <property>
      <name>mapreduce.map.output.compress</name>
      <value>true</value>
    </property>

    <property>
     <name>mapred.map.output.compress.codec</name> 
     <value>org.apache.hadoop.io.compress.SnappyCodec</value>
    </property>


    <property>
      <name>mapreduce.admin.user.env</name>
      <value>LD_LIBRARY_PATH=/usr/hdp/2.2.0.0-1084/hadoop/lib/native</value>
    </property>

The logs snapshot as following:-

2016-02-09 15:03:14,713 INFO [pool-26-thread-8] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=1a396460-dca5-4adf-af4a-8bc828c2e395] Initialized a new HDFS File System with working dir: hdfs://localhost:9000/user/shweta default block size: 134217728 default replication: 1 config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, /home/shweta/software/hadoop/hadoop-2.4.0/etc/hadoop/hdfs-site.xml, /home/shweta/software/hadoop/hadoop-2.4.0/etc/hadoop/core-site.xml
2016-02-09 15:03:14,724 INFO [pool-26-thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutHDFS[id=1a396460-dca5-4adf-af4a-8bc828c2e395] to run with 1 threads
2016-02-09 15:03:15,999 INFO [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ReplaceText ReplaceText[id=59f04d7a-c230-4774-a78f-90f44bcc24d5] Transferred StandardFlowFileRecord[uuid=cb84f64f-1bc1-4922-8380-b2692afaf1f5,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1455009982642-1, container=default, section=1], offset=724356, length=3710],offset=0,name=test2.csv,size=3710] to 'success'
2016-02-09 15:03:16,015 INFO [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ReplaceText ReplaceText[id=59f04d7a-c230-4774-a78f-90f44bcc24d5] Transferred StandardFlowFileRecord[uuid=f7156b6b-0a5d-4ea8-9494-0d7bcd2cf83d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1455009982642-1, container=default, section=1], offset=728066, length=7],offset=0,name=test.txt,size=7] to 'success'
2016-02-09 15:03:16,282 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=1a396460-dca5-4adf-af4a-8bc828c2e395] Failed to write to HDFS due to java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
2016-02-09 15:03:17,508 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=1a396460-dca5-4adf-af4a-8bc828c2e395] Failed to write to HDFS due to java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

Thanks,
Shweta
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

shweta
In reply to this post by Jeremy Dyer
Hi Jeremy,

Even after copying libsnappy.so to java_home/jre/lib it did not help much. I also pointed LD_LIBRARY_PATH to the location of libsnappy.so. Even went to the extent of modyfying bootstrap.conf with jvm params
 -Djava.library.path=//<path for libsnappy.so>.

But received the same error again. I have configured following properties in  Hadoop files as following:-

core-site.xml

<property>
  <name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

mapred-site.xml

 <property>
      <name>mapreduce.map.output.compress</name>
      <value>true</value>
    </property>

    <property>
     <name>mapred.map.output.compress.codec</name> 
     <value>org.apache.hadoop.io.compress.SnappyCodec</value>
    </property>


    <property>
      <name>mapreduce.admin.user.env</name>
      <value>LD_LIBRARY_PATH=/usr/hdp/2.2.0.0-1084/hadoop/lib/native</value>
    </property>

Anything else I'm missing on to get this issue fixed??

Thanks,
Shweta
Reply | Threaded
Open this post in threaded view
|

Re: java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

Matt Burgess
When you say you pointed LD_LIBRARY_PATH to the location of libsnappy.so, do you mean just the setting of the “mapreduce.admin.user.env” property in mapred-site.xml, or the actual environment variable before starting NiFi?  The mapred-site settings won’t be used as PutHDFS does not use MapReduce. If you do something like:

export LD_LIBRARY_PATH=/usr/hdp/2.2.0.0-1084/hadoop/lib/native
bin/nifi.sh start

That should let PutHDFS know about the appropriate libraries.




On 2/9/16, 4:38 AM, "shweta" <[hidden email]> wrote:

>Hi Jeremy,
>
>Even after copying libsnappy.so to java_home/jre/lib it did not help much. I
>also pointed LD_LIBRARY_PATH to the location of libsnappy.so. Even went to
>the extent of modyfying bootstrap.conf with jvm params
> -Djava.library.path=//<path for libsnappy.so>.
>
>But received the same error again. I have configured following properties in
>Hadoop files as following:-
>
>core-site.xml
>
><property>
>  <name>io.compression.codecs</name>
><value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
></property>
>
>mapred-site.xml
>
> <property>
>      <name>mapreduce.map.output.compress</name>
>      <value>true</value>
>    </property>
>
>    <property>
>     <name>mapred.map.output.compress.codec</name>  
>     <value>org.apache.hadoop.io.compress.SnappyCodec</value>
>    </property>
>
>
>    <property>
>      <name>mapreduce.admin.user.env</name>
>      <value>LD_LIBRARY_PATH=/usr/hdp/2.2.0.0-1084/hadoop/lib/native</value>
>    </property>
>
>Anything else I'm missing on to get this issue fixed??
>
>Thanks,
>Shweta
>
>
>
>--
>View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/java-lang-UnsatisfiedLinkError-in-PutHDFS-with-snappy-compression-tp7182p7236.html
>Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.