Controller service question

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Controller service question

Yuri Krysko
Hello Devs,

I discovered Apache NiFi several days ago, parsed through all the docs and would like to ask for your opinion on my use-case. I plan to have several custom processors to do HTTP POSTs to a 3rd party ticketing system via its REST API based on the Flowfiles fed by other processors. In order to compose JSON payload messages and POST to this ticketing system, in addition to the FlowFiles, these custom processors require metadata info from the ticketing system. The metadata may or may not change over time, as it is potentially configurable by the ticketing system admins. I am wondering whether I should create a custom Controller service to pull the metadata and make it available to all interested processors? Can I include a Runnable task in the Controller service and have a configurable parameter of how often to pull the data?

Thank you,
Yuri

________________________________

LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
Reply | Threaded
Open this post in threaded view
|

Re: Controller service question

Bryan Bende
Hi Yuri,

You could definitely create a controller service to obtain the metadata.

The controller service API doesn't provide anything to run a task on a
timer, its meant to be something that is accessed when needed by a
processor or reporting task, but you could certainly start a
background thread when your controller service is enabled, and have
the thread refresh the metadata periodically.

You could also have a property like "Refresh Interval" so users can
specify how often to refresh the metadata on the service, and then use
the value of this property to schedule the refresh thread.

Another option to avoid needing a thread is to lazily refresh the
metadata when needed, so keep track of the last time you fetched it
and then have a threshold for how old it needs to be to trigger a
refresh, and whenever a method of the controller service is called you
can check if the threshold is exceeded and refresh if necessary.

Let us know if you have more questions.

Thanks,

Bryan


On Fri, May 12, 2017 at 11:43 AM, Yuri Krysko <[hidden email]> wrote:

> Hello Devs,
>
> I discovered Apache NiFi several days ago, parsed through all the docs and would like to ask for your opinion on my use-case. I plan to have several custom processors to do HTTP POSTs to a 3rd party ticketing system via its REST API based on the Flowfiles fed by other processors. In order to compose JSON payload messages and POST to this ticketing system, in addition to the FlowFiles, these custom processors require metadata info from the ticketing system. The metadata may or may not change over time, as it is potentially configurable by the ticketing system admins. I am wondering whether I should create a custom Controller service to pull the metadata and make it available to all interested processors? Can I include a Runnable task in the Controller service and have a configurable parameter of how often to pull the data?
>
> Thank you,
> Yuri
>
> ________________________________
>
> LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
Reply | Threaded
Open this post in threaded view
|

Re: Controller service question

Yuri Krysko
Thank you Bryan,

I will try to incorporate your advice on lazy-load. Good idea.

On 5/12/17, 11:57 AM, "Bryan Bende" <[hidden email]> wrote:

>Hi Yuri,
>
>You could definitely create a controller service to obtain the metadata.
>
>The controller service API doesn't provide anything to run a task on a
>timer, its meant to be something that is accessed when needed by a
>processor or reporting task, but you could certainly start a
>background thread when your controller service is enabled, and have
>the thread refresh the metadata periodically.
>
>You could also have a property like "Refresh Interval" so users can
>specify how often to refresh the metadata on the service, and then use
>the value of this property to schedule the refresh thread.
>
>Another option to avoid needing a thread is to lazily refresh the
>metadata when needed, so keep track of the last time you fetched it
>and then have a threshold for how old it needs to be to trigger a
>refresh, and whenever a method of the controller service is called you
>can check if the threshold is exceeded and refresh if necessary.
>
>Let us know if you have more questions.
>
>Thanks,
>
>Bryan
>
>
>On Fri, May 12, 2017 at 11:43 AM, Yuri Krysko <[hidden email]>
>wrote:
>> Hello Devs,
>>
>> I discovered Apache NiFi several days ago, parsed through all the docs
>>and would like to ask for your opinion on my use-case. I plan to have
>>several custom processors to do HTTP POSTs to a 3rd party ticketing
>>system via its REST API based on the Flowfiles fed by other processors.
>>In order to compose JSON payload messages and POST to this ticketing
>>system, in addition to the FlowFiles, these custom processors require
>>metadata info from the ticketing system. The metadata may or may not
>>change over time, as it is potentially configurable by the ticketing
>>system admins. I am wondering whether I should create a custom
>>Controller service to pull the metadata and make it available to all
>>interested processors? Can I include a Runnable task in the Controller
>>service and have a configurable parameter of how often to pull the data?
>>
>> Thank you,
>> Yuri
>>
>> ________________________________
>>
>> LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this
>>e-mail and any files transmitted with it to be protected, proprietary or
>>privileged information intended solely for the use of the named
>>recipient(s). Any disclosure of this material or the information
>>contained herein, in whole or in part, to anyone outside of the intended
>>recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts
>>no liability for the content of this e-mail or for the consequences of
>>any actions taken on the basis of the information contained in it,
>>unless that information is subsequently confirmed in writing. Employees
>>of M.C. Dean, Inc. are instructed not to infringe on any rights of the
>>recipient; any such communication violates company policy. If you are
>>not the intended recipient, any disclosure, copying, distribution, or
>>action taken or omitted in reliance on this information is strictly
>>prohibited by M.C. Dean, Inc.; please notify the sender immediately by
>>return e-mail, delete this communication and destroy all copies.


________________________________

LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.