Running Python Scripts from NiFi

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Running Python Scripts from NiFi

Rob Weiss
Any thoughts on how to accomplish this?
Reply | Threaded
Open this post in threaded view
|

Re: Running Python Scripts from NiFi

Mark Payne
Rob,

We used to have a processor for running scripts directly in NiFi.
However, we have to remove that due to licensing conflicts with the
libraries that we were trying to use - they were not Apache friendly,
unfortunately.

You could use the ExecuteStreamCommand processor in order to accomplish
this though. It invokes whatever OS command you give it, so you can use
it to run a python script.

Does that do what you need?

Thanks
-Mark

------ Original Message ------
From: "Rob Weiss" <[hidden email]>
To: [hidden email]
Sent: 4/28/2015 10:31:37 AM
Subject: Running Python Scripts from NiFi

>Any thoughts on how to accomplish this?
Reply | Threaded
Open this post in threaded view
|

Re: Running Python Scripts from NiFi

Rob Weiss
Mark,
How would that processor get access to the data in the flowfile?

On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <[hidden email]> wrote:

> Rob,
>
> We used to have a processor for running scripts directly in NiFi. However,
> we have to remove that due to licensing conflicts with the libraries that
> we were trying to use - they were not Apache friendly, unfortunately.
>
> You could use the ExecuteStreamCommand processor in order to accomplish
> this though. It invokes whatever OS command you give it, so you can use it
> to run a python script.
>
> Does that do what you need?
>
> Thanks
> -Mark
>
>
> ------ Original Message ------
> From: "Rob Weiss" <[hidden email]>
> To: [hidden email]
> Sent: 4/28/2015 10:31:37 AM
> Subject: Running Python Scripts from NiFi
>
>  Any thoughts on how to accomplish this?
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Running Python Scripts from NiFi

Mark Payne
Rob,

The content of the FlowFile would be streamed to the Python's STDIN.

So the Python script would have to be written to read from std in. If it
instead is intended to run against a file, you could use PutFile to
write the contents to some temporary directory.

Thanks
-Mark

------ Original Message ------
From: "Rob Weiss" <[hidden email]>
To: [hidden email]
Sent: 4/28/2015 10:43:54 AM
Subject: Re: Running Python Scripts from NiFi

>Mark,
>How would that processor get access to the data in the flowfile?
>
>On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <[hidden email]>
>wrote:
>
>>  Rob,
>>
>>  We used to have a processor for running scripts directly in NiFi.
>>However,
>>  we have to remove that due to licensing conflicts with the libraries
>>that
>>  we were trying to use - they were not Apache friendly, unfortunately.
>>
>>  You could use the ExecuteStreamCommand processor in order to
>>accomplish
>>  this though. It invokes whatever OS command you give it, so you can
>>use it
>>  to run a python script.
>>
>>  Does that do what you need?
>>
>>  Thanks
>>  -Mark
>>
>>
>>  ------ Original Message ------
>>  From: "Rob Weiss" <[hidden email]>
>>  To: [hidden email]
>>  Sent: 4/28/2015 10:31:37 AM
>>  Subject: Running Python Scripts from NiFi
>>
>>   Any thoughts on how to accomplish this?
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Running Python Scripts from NiFi

Rob Weiss
Mark,
Thx for the heads up; we are trying not to touch disk during this process.
Can I assume that STDOUT would be where we send the modified data to get it
back into NiFi?

Thanks Again!

On Tue, Apr 28, 2015 at 10:51 AM, Mark Payne <[hidden email]> wrote:

> Rob,
>
> The content of the FlowFile would be streamed to the Python's STDIN.
>
> So the Python script would have to be written to read from std in. If it
> instead is intended to run against a file, you could use PutFile to write
> the contents to some temporary directory.
>
> Thanks
> -Mark
>
> ------ Original Message ------
> From: "Rob Weiss" <[hidden email]>
> To: [hidden email]
> Sent: 4/28/2015 10:43:54 AM
> Subject: Re: Running Python Scripts from NiFi
>
>  Mark,
>> How would that processor get access to the data in the flowfile?
>>
>> On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <[hidden email]>
>> wrote:
>>
>>   Rob,
>>>
>>>  We used to have a processor for running scripts directly in NiFi.
>>> However,
>>>  we have to remove that due to licensing conflicts with the libraries
>>> that
>>>  we were trying to use - they were not Apache friendly, unfortunately.
>>>
>>>  You could use the ExecuteStreamCommand processor in order to accomplish
>>>  this though. It invokes whatever OS command you give it, so you can use
>>> it
>>>  to run a python script.
>>>
>>>  Does that do what you need?
>>>
>>>  Thanks
>>>  -Mark
>>>
>>>
>>>  ------ Original Message ------
>>>  From: "Rob Weiss" <[hidden email]>
>>>  To: [hidden email]
>>>  Sent: 4/28/2015 10:31:37 AM
>>>  Subject: Running Python Scripts from NiFi
>>>
>>>   Any thoughts on how to accomplish this?
>>>
>>>>
>>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Running Python Scripts from NiFi

Mark Payne
Rob,

The processor has 2 relationships: "original" where the original
flowfile goes and "output stream" where a new FlowFile is routed. The
contents of this new flowfile are the output from the command's stdout.

Thanks
-Mark


------ Original Message ------
From: "Rob Weiss" <[hidden email]>
To: [hidden email]
Sent: 4/28/2015 10:54:11 AM
Subject: Re: Running Python Scripts from NiFi

>Mark,
>Thx for the heads up; we are trying not to touch disk during this
>process.
>Can I assume that STDOUT would be where we send the modified data to
>get it
>back into NiFi?
>
>Thanks Again!
>
>On Tue, Apr 28, 2015 at 10:51 AM, Mark Payne <[hidden email]>
>wrote:
>
>>  Rob,
>>
>>  The content of the FlowFile would be streamed to the Python's STDIN.
>>
>>  So the Python script would have to be written to read from std in. If
>>it
>>  instead is intended to run against a file, you could use PutFile to
>>write
>>  the contents to some temporary directory.
>>
>>  Thanks
>>  -Mark
>>
>>  ------ Original Message ------
>>  From: "Rob Weiss" <[hidden email]>
>>  To: [hidden email]
>>  Sent: 4/28/2015 10:43:54 AM
>>  Subject: Re: Running Python Scripts from NiFi
>>
>>   Mark,
>>>  How would that processor get access to the data in the flowfile?
>>>
>>>  On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <[hidden email]>
>>>  wrote:
>>>
>>>    Rob,
>>>>
>>>>   We used to have a processor for running scripts directly in NiFi.
>>>>  However,
>>>>   we have to remove that due to licensing conflicts with the
>>>>libraries
>>>>  that
>>>>   we were trying to use - they were not Apache friendly,
>>>>unfortunately.
>>>>
>>>>   You could use the ExecuteStreamCommand processor in order to
>>>>accomplish
>>>>   this though. It invokes whatever OS command you give it, so you
>>>>can use
>>>>  it
>>>>   to run a python script.
>>>>
>>>>   Does that do what you need?
>>>>
>>>>   Thanks
>>>>   -Mark
>>>>
>>>>
>>>>   ------ Original Message ------
>>>>   From: "Rob Weiss" <[hidden email]>
>>>>   To: [hidden email]
>>>>   Sent: 4/28/2015 10:31:37 AM
>>>>   Subject: Running Python Scripts from NiFi
>>>>
>>>>    Any thoughts on how to accomplish this?
>>>>
>>>>>
>>>>>
>>>>