NiFi Workflow deployment

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi Workflow deployment

Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US)
Hi.

Can someone describe the best way to maintain, configure and deploy
workflows to various NiFi instances?  The project I work on has three DEV
and three TEST instances.  Our workflows have host-specific values and
flow-specific values so we want to be able to automate configuring the flows
and deploying them to the various servers.  NiFi out of the box doesn't seem
to support doing what we need to do.

We are, currently, using NiFi 1.1.2.

Any suggestions would be greatly appreciated!

Thank you,

Steven K. Byers



smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Workflow deployment

Andy LoPresto-2
Hi Steven,

This is a common problem which we are actively working to improve. Currently, the “Variable Registry” [1] feature is a stop-gap solution in conjunction with publishing and loading templates [2] into the various environments. The new Flow Registry [3] feature will make this much easier and automate the process for you. You can read about the feature roadmap and if you search these mailing lists [4][5] for “SDLC” or “flow registry” you can find many other users discussing their solutions and tips for similar experiences. Hope this helps. 



Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 15, 2017, at 2:50 PM, Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) <[hidden email]> wrote:

Hi.

Can someone describe the best way to maintain, configure and deploy
workflows to various NiFi instances?  The project I work on has three DEV
and three TEST instances.  Our workflows have host-specific values and
flow-specific values so we want to be able to automate configuring the flows
and deploying them to the various servers.  NiFi out of the box doesn't seem
to support doing what we need to do.

We are, currently, using NiFi 1.1.2.

Any suggestions would be greatly appreciated!

Thank you,

Steven K. Byers




signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: [Non-DoD Source] Re: NiFi Workflow deployment

Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US)

Andy,

 

Thank you for your reply.  I have come up with a scheme which should meet our needs.  It involves using a key/value approach similar to the idea behind the Variable Registry but is file based.  Key/value pairs are stored in JSON files for each template  and I have a Python program which matches the keys in the templates and substitutes the value.  This allows me to target configurations for each server we will be deploying to.

 

Thank you,

 

Steven K. Byers

DXC Technology - Contractor

Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC)

Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC

1681 Nelson Street, Fort Detrick, MD  21702

(443) 538-7575 | (410) 872-4923

Email:     [hidden email]

 

 

From: Andy LoPresto [mailto:[hidden email]]
Sent: Tuesday, August 15, 2017 3:02 PM
To: [hidden email]
Subject: [Non-DoD Source] Re: NiFi Workflow deployment

 

All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser.


 

Hi Steven,

 

This is a common problem which we are actively working to improve. Currently, the “Variable Registry” [1] feature is a stop-gap solution in conjunction with publishing and loading templates [2] into the various environments. The new Flow Registry [3] feature will make this much easier and automate the process for you. You can read about the feature roadmap and if you search these mailing lists [4][5] for “SDLC” or “flow registry” you can find many other users discussing their solutions and tips for similar experiences. Hope this helps. 

 

[1] Caution-https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry < Caution-https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry > 

[2] Caution-https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates < Caution-https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates > 

[3] Caution-https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows < Caution-https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows > 

[4] Caution-https://lists.apache.org/list.html?[hidden email] < Caution-https://lists.apache.org/list.html?[hidden email] > 

[5] Caution-https://lists.apache.org/list.html?[hidden email] < Caution-https://lists.apache.org/list.html?[hidden email] > 

 

 

Andy LoPresto

[hidden email] < Caution-mailto:[hidden email] > 

[hidden email] < Caution-mailto:[hidden email] > 

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 

On Aug 15, 2017, at 2:50 PM, Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) <[hidden email] < Caution-mailto:[hidden email] > > wrote:

 

Hi.

Can someone describe the best way to maintain, configure and deploy
workflows to various NiFi instances?  The project I work on has three DEV
and three TEST instances.  Our workflows have host-specific values and
flow-specific values so we want to be able to automate configuring the flows
and deploying them to the various servers.  NiFi out of the box doesn't seem
to support doing what we need to do.

We are, currently, using NiFi 1.1.2.

Any suggestions would be greatly appreciated!

Thank you,

Steven K. Byers

 


smime.p7s (7K) Download Attachment