NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Erik Anderson
I have a slight improvement to Environment variables that the NiFI docker system uses to bootstrap its nifi.properties files

I documented it under

https://issues.apache.org/jira/browse/NIFI-6013

The commit is here.

https://github.com/dprophet/nifi/commit/ea31ac6bd8f00944166e3e230af2040636c0505c

*****

There are times, when learning NiFi, that I want to stop and start NiFi with debugging on. If NiFi was running inside a Docker container it was hard.

What I did was I created a new environment variable within the NiFi bootstrapping system it uses to start itself up inside a docker container. FYI, some of those environment variables are documented here.

https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/README.md

What I did was introduce a new environment variable

NIFI_DEBUG

And in file
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh

I added

if [[ -n $NIFI_DEBUG ]] ; then
    . "${scripts_dir}/update_logback.sh"
fi

Basically what it does, if the NIFI_DEBUG is set
Example:

NIFI_DEBUG="org.apache.nifi.web.security=TRACE,org.apache.nifi.authorization=DEBUG,org.apache.nifi.web.api.config=TRACE"

It updates the
${NIFI_HOME}/conf/logback.xml

File to enable the debugging modes of NiFi

This is a very fast mechanism to restart NiFi with debugging turned on. Very developer and workflow friendly.

If of interest I will create a JIRA, put the code on a fork, and give a pull request.

Erik Anderson
Bloomberg
Reply | Threaded
Open this post in threaded view
|

Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Andrew Grande-2
Erik,

I was reading that email and was thinking of JVM debug options, with
suspend y/n. I guess it just shows we meant very different things by debug
mode. Maybe you could incorporate those into a PR too?

Andrew

On Fri, Feb 8, 2019, 2:15 PM Erik Anderson <[hidden email]> wrote:

> I have a slight improvement to Environment variables that the NiFI docker
> system uses to bootstrap its nifi.properties files
>
> I documented it under
>
> https://issues.apache.org/jira/browse/NIFI-6013
>
> The commit is here.
>
>
> https://github.com/dprophet/nifi/commit/ea31ac6bd8f00944166e3e230af2040636c0505c
>
> *****
>
> There are times, when learning NiFi, that I want to stop and start NiFi
> with debugging on. If NiFi was running inside a Docker container it was
> hard.
>
> What I did was I created a new environment variable within the NiFi
> bootstrapping system it uses to start itself up inside a docker container.
> FYI, some of those environment variables are documented here.
>
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/README.md
>
> What I did was introduce a new environment variable
>
> NIFI_DEBUG
>
> And in file
>
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
>
> I added
>
> if [[ -n $NIFI_DEBUG ]] ; then
>     . "${scripts_dir}/update_logback.sh"
> fi
>
> Basically what it does, if the NIFI_DEBUG is set
> Example:
>
>
> NIFI_DEBUG="org.apache.nifi.web.security=TRACE,org.apache.nifi.authorization=DEBUG,org.apache.nifi.web.api.config=TRACE"
>
> It updates the
> ${NIFI_HOME}/conf/logback.xml
>
> File to enable the debugging modes of NiFi
>
> This is a very fast mechanism to restart NiFi with debugging turned on.
> Very developer and workflow friendly.
>
> If of interest I will create a JIRA, put the code on a fork, and give a
> pull request.
>
> Erik Anderson
> Bloomberg
>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Erik Anderson
> I was reading that email and was thinking of JVM debug options, with
> suspend y/n. I guess it just shows we meant very different things by debug
> mode. Maybe you could incorporate those into a PR too?
>
> Andrew

Good point Andrew. and sorry for the slow response. I had to look at how NiFi set the JVM properties.

We had issues with Java and needed to manually set env Java flags.

It seems all of the Java debugging flags are set in the bootstrap.conf, located here.

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/bootstrap.conf

For me, I chained together the Dockerfile
ENTRYPOINT ["my_company_entry_point.sh"]
then had it call the original entry point
In short, all of NiFi properties are set via environment variables in a /etc/profile.d/nifi.sh on the host and propogate throughout the NiFi system. Like I listed in
https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md

I normalized all NIFI specific environment variables to start with NIFI_

Why? Because you set the

export NIFI_FOO="my foo"

and its directly passed into the container and into the NiFi role account used to start the container. You now have access to the environment variables throughout the NiFi system.

Example:
docker run --name nifi --env-file <(env | grep NIFI_) --hostname nifi

IMO, the Dockerfile entry point should allow a plug and play script so you can set these custom behaviors (both enterprise behaviors and custom developer/debugging). I doubt any enterprise will blindly pull a DockerHub container and run it. From my experience, a public container isnt enterprise friendly.

Andrew, define what you want for JVM debugging, what you would want to set (and unset), and I will take a look.

Erik Anderson
Bloomberg
https://www.linkedin.com/in/erikanderson/
Reply | Threaded
Open this post in threaded view
|

Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Andrew Grande-2
Here's what I did previously, Erik. A user only had to specify an
additional -debug flag for the main shell script which would in turn take
care of any bootstrap rewriting/generation for allowing for the remote JVM
debug session to connect. Maybe it could give a few ideas.

Andrew

On Wed, Feb 13, 2019, 7:34 AM Erik Anderson <[hidden email]> wrote:

> > I was reading that email and was thinking of JVM debug options, with
> > suspend y/n. I guess it just shows we meant very different things by
> debug
> > mode. Maybe you could incorporate those into a PR too?
> >
> > Andrew
>
> Good point Andrew. and sorry for the slow response. I had to look at how
> NiFi set the JVM properties.
>
> We had issues with Java and needed to manually set env Java flags.
>
> It seems all of the Java debugging flags are set in the bootstrap.conf,
> located here.
>
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/bootstrap.conf
>
> For me, I chained together the Dockerfile
> ENTRYPOINT ["my_company_entry_point.sh"]
> then had it call the original entry point
> In short, all of NiFi properties are set via environment variables in a
> /etc/profile.d/nifi.sh on the host and propogate throughout the NiFi
> system. Like I listed in
>
> https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md
>
> I normalized all NIFI specific environment variables to start with NIFI_
>
> Why? Because you set the
>
> export NIFI_FOO="my foo"
>
> and its directly passed into the container and into the NiFi role account
> used to start the container. You now have access to the environment
> variables throughout the NiFi system.
>
> Example:
> docker run --name nifi --env-file <(env | grep NIFI_) --hostname nifi
>
> IMO, the Dockerfile entry point should allow a plug and play script so you
> can set these custom behaviors (both enterprise behaviors and custom
> developer/debugging). I doubt any enterprise will blindly pull a DockerHub
> container and run it. From my experience, a public container isnt
> enterprise friendly.
>
> Andrew, define what you want for JVM debugging, what you would want to set
> (and unset), and I will take a look.
>
> Erik Anderson
> Bloomberg
> https://www.linkedin.com/in/erikanderson/
>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Erik Anderson
Andrew,

Sorry, was skiing.

> A user only had to specify an
> additional -debug flag for the main shell script which would in turn take
> care of any bootstrap rewriting/generation for allowing for the remote JVM
> debug session to connect.

So you dont want logging, you want a remote debugger session?

1) I expect many environments wont let devs just connect a debugger into a production network. In a prod network, we start with logs.
2) What about NiFi clusters? Distributed debugging, wow, now that gives me a headache.  How would you connect a java debugger into a 3 node cluster?
3) Maybe you want to trap/breakpoint a list of exceptions and stop the java process to wait for a debugger to be attached?

As our NiFi usage grows, the #1 most important feature is debug-ability of an issue in any environment.

Andrew, as you have a lot more experience with issues you have seen in the past, how about a 30 minute call? This way can I figure out what debugging means to everyone, what information they need, and the best way to bootstrap it.

Erik Anderson
Bloomberg
https://www.linkedin.com/in/erikanderson/

Sent from my iPad

> On Feb 14, 2019, at 12:47 AM, Andrew Grande <[hidden email]> wrote:
>
> Here's what I did previously, Erik. A user only had to specify an
> additional -debug flag for the main shell script which would in turn take
> care of any bootstrap rewriting/generation for allowing for the remote JVM
> debug session to connect. Maybe it could give a few ideas.
>
> Andrew
>
> On Wed, Feb 13, 2019, 7:34 AM Erik Anderson <[hidden email]> wrote:
>
>>> I was reading that email and was thinking of JVM debug options, with
>>> suspend y/n. I guess it just shows we meant very different things by
>> debug
>>> mode. Maybe you could incorporate those into a PR too?
>>>
>>> Andrew
>>
>> Good point Andrew. and sorry for the slow response. I had to look at how
>> NiFi set the JVM properties.
>>
>> We had issues with Java and needed to manually set env Java flags.
>>
>> It seems all of the Java debugging flags are set in the bootstrap.conf,
>> located here.
>>
>>
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/bootstrap.conf
>>
>> For me, I chained together the Dockerfile
>> ENTRYPOINT ["my_company_entry_point.sh"]
>> then had it call the original entry point
>> In short, all of NiFi properties are set via environment variables in a
>> /etc/profile.d/nifi.sh on the host and propogate throughout the NiFi
>> system. Like I listed in
>>
>> https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md
>>
>> I normalized all NIFI specific environment variables to start with NIFI_
>>
>> Why? Because you set the
>>
>> export NIFI_FOO="my foo"
>>
>> and its directly passed into the container and into the NiFi role account
>> used to start the container. You now have access to the environment
>> variables throughout the NiFi system.
>>
>> Example:
>> docker run --name nifi --env-file <(env | grep NIFI_) --hostname nifi
>>
>> IMO, the Dockerfile entry point should allow a plug and play script so you
>> can set these custom behaviors (both enterprise behaviors and custom
>> developer/debugging). I doubt any enterprise will blindly pull a DockerHub
>> container and run it. From my experience, a public container isnt
>> enterprise friendly.
>>
>> Andrew, define what you want for JVM debugging, what you would want to set
>> (and unset), and I will take a look.
>>
>> Erik Anderson
>> Bloomberg
>> https://www.linkedin.com/in/erikanderson/
>>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Mike Thomsen
Erik,

I think a lot of people would benefit from having both the JVM debugger
option AND your debug logging. Go ahead and send us a PR for your debug
logging and I'll do a ticket to add the JVM debugger. I just merged a
change to the Docker setup that will make the JVM debugger much easier to
enable.

Thanks,

Mike

On Wed, Feb 20, 2019 at 9:48 AM Erik Anderson <[hidden email]> wrote:

> Andrew,
>
> Sorry, was skiing.
>
> > A user only had to specify an
> > additional -debug flag for the main shell script which would in turn take
> > care of any bootstrap rewriting/generation for allowing for the remote
> JVM
> > debug session to connect.
>
> So you dont want logging, you want a remote debugger session?
>
> 1) I expect many environments wont let devs just connect a debugger into a
> production network. In a prod network, we start with logs.
> 2) What about NiFi clusters? Distributed debugging, wow, now that gives me
> a headache.  How would you connect a java debugger into a 3 node cluster?
> 3) Maybe you want to trap/breakpoint a list of exceptions and stop the
> java process to wait for a debugger to be attached?
>
> As our NiFi usage grows, the #1 most important feature is debug-ability of
> an issue in any environment.
>
> Andrew, as you have a lot more experience with issues you have seen in the
> past, how about a 30 minute call? This way can I figure out what debugging
> means to everyone, what information they need, and the best way to
> bootstrap it.
>
> Erik Anderson
> Bloomberg
> https://www.linkedin.com/in/erikanderson/
>
> Sent from my iPad
>
> > On Feb 14, 2019, at 12:47 AM, Andrew Grande <[hidden email]> wrote:
> >
> > Here's what I did previously, Erik. A user only had to specify an
> > additional -debug flag for the main shell script which would in turn take
> > care of any bootstrap rewriting/generation for allowing for the remote
> JVM
> > debug session to connect. Maybe it could give a few ideas.
> >
> > Andrew
> >
> > On Wed, Feb 13, 2019, 7:34 AM Erik Anderson <[hidden email]> wrote:
> >
> >>> I was reading that email and was thinking of JVM debug options, with
> >>> suspend y/n. I guess it just shows we meant very different things by
> >> debug
> >>> mode. Maybe you could incorporate those into a PR too?
> >>>
> >>> Andrew
> >>
> >> Good point Andrew. and sorry for the slow response. I had to look at how
> >> NiFi set the JVM properties.
> >>
> >> We had issues with Java and needed to manually set env Java flags.
> >>
> >> It seems all of the Java debugging flags are set in the bootstrap.conf,
> >> located here.
> >>
> >>
> >>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/bootstrap.conf
> >>
> >> For me, I chained together the Dockerfile
> >> ENTRYPOINT ["my_company_entry_point.sh"]
> >> then had it call the original entry point
> >> In short, all of NiFi properties are set via environment variables in a
> >> /etc/profile.d/nifi.sh on the host and propogate throughout the NiFi
> >> system. Like I listed in
> >>
> >>
> https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md
> >>
> >> I normalized all NIFI specific environment variables to start with NIFI_
> >>
> >> Why? Because you set the
> >>
> >> export NIFI_FOO="my foo"
> >>
> >> and its directly passed into the container and into the NiFi role
> account
> >> used to start the container. You now have access to the environment
> >> variables throughout the NiFi system.
> >>
> >> Example:
> >> docker run --name nifi --env-file <(env | grep NIFI_) --hostname nifi
> >>
> >> IMO, the Dockerfile entry point should allow a plug and play script so
> you
> >> can set these custom behaviors (both enterprise behaviors and custom
> >> developer/debugging). I doubt any enterprise will blindly pull a
> DockerHub
> >> container and run it. From my experience, a public container isnt
> >> enterprise friendly.
> >>
> >> Andrew, define what you want for JVM debugging, what you would want to
> set
> >> (and unset), and I will take a look.
> >>
> >> Erik Anderson
> >> Bloomberg
> >> https://www.linkedin.com/in/erikanderson/
> >>
>