Binaries in embedded redis jar files

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Binaries in embedded redis jar files

Joe Skora-2
I noticed a new jar dependency for Redis includes binaries.

$ jar -xf

> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/embedded-redis-0.6.jar
> $ ls -l
> total 6292
> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
> $ file redis-server*
> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
> redis-server-2.8.19.app: Mach-O 64-bit executable
> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS Windows
>

So, installing NiFi will include externally built binaries, not just
scripts but full ELF, MacOS, and Windows PE binaries.

Is this a problem for Apache?  Does this cause concern for anyone else?

Regards,
Joe
Reply | Threaded
Open this post in threaded view
|

Re: Binaries in embedded redis jar files

Joe Witt
It is not a problem.  It would be an issue if those were part of our
source release which is the official apache release artifact.



On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora <[hidden email]> wrote:

> I noticed a new jar dependency for Redis includes binaries.
>
> $ jar -xf
>> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/embedded-redis-0.6.jar
>> $ ls -l
>> total 6292
>> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
>> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
>> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
>> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
>> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
>> $ file redis-server*
>> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
>> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
>> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
>> redis-server-2.8.19.app: Mach-O 64-bit executable
>> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS Windows
>>
>
> So, installing NiFi will include externally built binaries, not just
> scripts but full ELF, MacOS, and Windows PE binaries.
>
> Is this a problem for Apache?  Does this cause concern for anyone else?
>
> Regards,
> Joe
Reply | Threaded
Open this post in threaded view
|

Re: Binaries in embedded redis jar files

Michael Moser
Whoa, this causes serious concerns for me.  While I can understand
that the source code is the official ASF artifact, the nifi.apache.org
site provides convenience binaries which I'm sure many people use.
When NiFi 1.4.0 releases, we should write very conspicuous warnings on
the Downloads page to advise users about this.

-- Mike


On Wed, Jul 19, 2017 at 2:04 PM, Joe Witt <[hidden email]> wrote:

> It is not a problem.  It would be an issue if those were part of our
> source release which is the official apache release artifact.
>
>
>
> On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora <[hidden email]> wrote:
>> I noticed a new jar dependency for Redis includes binaries.
>>
>> $ jar -xf
>>> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/embedded-redis-0.6.jar
>>> $ ls -l
>>> total 6292
>>> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
>>> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
>>> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
>>> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
>>> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
>>> $ file redis-server*
>>> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
>>> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
>>> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
>>> redis-server-2.8.19.app: Mach-O 64-bit executable
>>> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS Windows
>>>
>>
>> So, installing NiFi will include externally built binaries, not just
>> scripts but full ELF, MacOS, and Windows PE binaries.
>>
>> Is this a problem for Apache?  Does this cause concern for anyone else?
>>
>> Regards,
>> Joe
Reply | Threaded
Open this post in threaded view
|

Re: Binaries in embedded redis jar files

Joe Skora
My concern is that an unknown, unsigned binary could be downloaded and
executed with each build regardless of the operating system.

On Wed, Jul 19, 2017 at 2:19 PM, Michael Moser <[hidden email]> wrote:

> Whoa, this causes serious concerns for me.  While I can understand
> that the source code is the official ASF artifact, the nifi.apache.org
> site provides convenience binaries which I'm sure many people use.
> When NiFi 1.4.0 releases, we should write very conspicuous warnings on
> the Downloads page to advise users about this.
>
> -- Mike
>
>
> On Wed, Jul 19, 2017 at 2:04 PM, Joe Witt <[hidden email]> wrote:
> > It is not a problem.  It would be an issue if those were part of our
> > source release which is the official apache release artifact.
> >
> >
> >
> > On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora <[hidden email]> wrote:
> >> I noticed a new jar dependency for Redis includes binaries.
> >>
> >> $ jar -xf
> >>> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/
> embedded-redis-0.6.jar
> >>> $ ls -l
> >>> total 6292
> >>> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
> >>> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
> >>> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
> >>> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
> >>> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
> >>> $ file redis-server*
> >>> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
> >>> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
> >>> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
> >>> redis-server-2.8.19.app: Mach-O 64-bit executable
> >>> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS
> Windows
> >>>
> >>
> >> So, installing NiFi will include externally built binaries, not just
> >> scripts but full ELF, MacOS, and Windows PE binaries.
> >>
> >> Is this a problem for Apache?  Does this cause concern for anyone else?
> >>
> >> Regards,
> >> Joe
>
Reply | Threaded
Open this post in threaded view
|

Re: Binaries in embedded redis jar files

Pierre Villard
In reply to this post by Michael Moser
Mike, I think this is just something that will be downloaded in your Maven
repository in case you're building NiFi. But I don't think this will be
included in the NiFi binary itself. At least, that's what I understand.

Pierre

2017-07-19 20:19 GMT+02:00 Michael Moser <[hidden email]>:

> Whoa, this causes serious concerns for me.  While I can understand
> that the source code is the official ASF artifact, the nifi.apache.org
> site provides convenience binaries which I'm sure many people use.
> When NiFi 1.4.0 releases, we should write very conspicuous warnings on
> the Downloads page to advise users about this.
>
> -- Mike
>
>
> On Wed, Jul 19, 2017 at 2:04 PM, Joe Witt <[hidden email]> wrote:
> > It is not a problem.  It would be an issue if those were part of our
> > source release which is the official apache release artifact.
> >
> >
> >
> > On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora <[hidden email]> wrote:
> >> I noticed a new jar dependency for Redis includes binaries.
> >>
> >> $ jar -xf
> >>> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/
> embedded-redis-0.6.jar
> >>> $ ls -l
> >>> total 6292
> >>> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
> >>> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
> >>> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
> >>> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
> >>> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
> >>> $ file redis-server*
> >>> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
> >>> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
> >>> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
> >>> redis-server-2.8.19.app: Mach-O 64-bit executable
> >>> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS
> Windows
> >>>
> >>
> >> So, installing NiFi will include externally built binaries, not just
> >> scripts but full ELF, MacOS, and Windows PE binaries.
> >>
> >> Is this a problem for Apache?  Does this cause concern for anyone else?
> >>
> >> Regards,
> >> Joe
>
Reply | Threaded
Open this post in threaded view
|

Re: Binaries in embedded redis jar files

Michael Moser
In reply to this post by Michael Moser
OK, I see that embedded-redis.jar is only used in the test scope, so
this isn't really relevant to the convenience binaries.  But I cannot
in good conscience allow my NiFi users to execute these unit tests if
they want to build on my corporate network.

-- Mike

On Wed, Jul 19, 2017 at 2:19 PM, Michael Moser <[hidden email]> wrote:

> Whoa, this causes serious concerns for me.  While I can understand
> that the source code is the official ASF artifact, the nifi.apache.org
> site provides convenience binaries which I'm sure many people use.
> When NiFi 1.4.0 releases, we should write very conspicuous warnings on
> the Downloads page to advise users about this.
>
> -- Mike
>
>
> On Wed, Jul 19, 2017 at 2:04 PM, Joe Witt <[hidden email]> wrote:
>> It is not a problem.  It would be an issue if those were part of our
>> source release which is the official apache release artifact.
>>
>>
>>
>> On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora <[hidden email]> wrote:
>>> I noticed a new jar dependency for Redis includes binaries.
>>>
>>> $ jar -xf
>>>> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/embedded-redis-0.6.jar
>>>> $ ls -l
>>>> total 6292
>>>> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
>>>> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
>>>> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
>>>> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
>>>> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
>>>> $ file redis-server*
>>>> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
>>>> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
>>>> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
>>>> redis-server-2.8.19.app: Mach-O 64-bit executable
>>>> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS Windows
>>>>
>>>
>>> So, installing NiFi will include externally built binaries, not just
>>> scripts but full ELF, MacOS, and Windows PE binaries.
>>>
>>> Is this a problem for Apache?  Does this cause concern for anyone else?
>>>
>>> Regards,
>>> Joe
Reply | Threaded
Open this post in threaded view
|

Re: Binaries in embedded redis jar files

trkurc
Administrator
In reply to this post by Michael Moser
Mike -
Is there something about these binaries versus all of the other myriad
binaries (jars, .class files, dlls and .so files) included in the
convenience jar? I wouldn't think so, these are just more binary artifacts.
I honestly can't think of another apache project that provides similar
convenience binaries that have warnings such as those you mentioned.

Tony

On Wed, Jul 19, 2017 at 2:19 PM, Michael Moser <[hidden email]> wrote:

> Whoa, this causes serious concerns for me.  While I can understand
> that the source code is the official ASF artifact, the nifi.apache.org
> site provides convenience binaries which I'm sure many people use.
> When NiFi 1.4.0 releases, we should write very conspicuous warnings on
> the Downloads page to advise users about this.
>
> -- Mike
>
>
> On Wed, Jul 19, 2017 at 2:04 PM, Joe Witt <[hidden email]> wrote:
> > It is not a problem.  It would be an issue if those were part of our
> > source release which is the official apache release artifact.
> >
> >
> >
> > On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora <[hidden email]> wrote:
> >> I noticed a new jar dependency for Redis includes binaries.
> >>
> >> $ jar -xf
> >>> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/
> embedded-redis-0.6.jar
> >>> $ ls -l
> >>> total 6292
> >>> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
> >>> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
> >>> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
> >>> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
> >>> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
> >>> $ file redis-server*
> >>> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
> >>> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
> >>> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
> >>> redis-server-2.8.19.app: Mach-O 64-bit executable
> >>> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS
> Windows
> >>>
> >>
> >> So, installing NiFi will include externally built binaries, not just
> >> scripts but full ELF, MacOS, and Windows PE binaries.
> >>
> >> Is this a problem for Apache?  Does this cause concern for anyone else?
> >>
> >> Regards,
> >> Joe
>
Reply | Threaded
Open this post in threaded view
|

Re: Binaries in embedded redis jar files

Joey Frazee
In addition to only being for testing, these binaries are only used for an integration test, so we don't even need to worry about whether they're going to work on everyone's OS/cause test failures if the binaries don't work for you.

Now it is packaging up the entire Redis server binary, though, which is a little different than a class file or dll, so I think a different question is whether we want ITs that bring their own server in in this way?

A mock server might be sufficient or I've seen projects using docker-maven to provide all the server resources needed for ITs. This could be helpful for us for the Elastic, C*, etc. ITs too if we did it.

So binaries = ok and no warnings needed, but it might still be a valid question about whether we want to use this redis-embedded lib; but it doesn't strike me as a very urgent thing.

> On Jul 19, 2017, at 1:54 PM, Tony Kurc <[hidden email]> wrote:
>
> Mike -
> Is there something about these binaries versus all of the other myriad
> binaries (jars, .class files, dlls and .so files) included in the
> convenience jar? I wouldn't think so, these are just more binary artifacts.
> I honestly can't think of another apache project that provides similar
> convenience binaries that have warnings such as those you mentioned.
>
> Tony
>
>> On Wed, Jul 19, 2017 at 2:19 PM, Michael Moser <[hidden email]> wrote:
>>
>> Whoa, this causes serious concerns for me.  While I can understand
>> that the source code is the official ASF artifact, the nifi.apache.org
>> site provides convenience binaries which I'm sure many people use.
>> When NiFi 1.4.0 releases, we should write very conspicuous warnings on
>> the Downloads page to advise users about this.
>>
>> -- Mike
>>
>>
>>> On Wed, Jul 19, 2017 at 2:04 PM, Joe Witt <[hidden email]> wrote:
>>> It is not a problem.  It would be an issue if those were part of our
>>> source release which is the official apache release artifact.
>>>
>>>
>>>
>>>> On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora <[hidden email]> wrote:
>>>> I noticed a new jar dependency for Redis includes binaries.
>>>>
>>>> $ jar -xf
>>>>> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/
>> embedded-redis-0.6.jar
>>>>> $ ls -l
>>>>> total 6292
>>>>> drwxr-xr-x. 3 user1 users      36 Apr 12  2015 META-INF/
>>>>> drwxr-xr-x. 3 user1 users      21 Apr 12  2015 redis/
>>>>> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
>>>>> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
>>>>> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
>>>>> $ file redis-server*
>>>>> redis-server-2.8.19:     ELF 64-bit LSB executable, x86-64, version 1
>>>>> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
>>>>> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
>>>>> redis-server-2.8.19.app: Mach-O 64-bit executable
>>>>> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS
>> Windows
>>>>>
>>>>
>>>> So, installing NiFi will include externally built binaries, not just
>>>> scripts but full ELF, MacOS, and Windows PE binaries.
>>>>
>>>> Is this a problem for Apache?  Does this cause concern for anyone else?
>>>>
>>>> Regards,
>>>> Joe
>>