Automated Testing of Nifi Flow by injecting data into Processors

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Automated Testing of Nifi Flow by injecting data into Processors

Zenin, Alexei
Hi,

Is there any way in Nifi to test out your whole flow without having the source systems online and running. I need this kind of capability to run automated integration tests which tests if the flow behaves as expected. None of the tools or libraries I have seen accomplish this and the Nifi API does not seem to allow to inject data directly into a processor (Nifi's TestRunner seems to be able to create FlowFiles but can we inject mock data to a processor through the Nifi REST API?). I think this would be a critical feature to make this system more production ready and abide by best practices for CICD.

Regards,
Alexei
To unsubscribe from receiving further commercial electronic messages from The Bank of Nova Scotia in Canada, please click here: https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK

To unsubscribe from receiving further commercial electronic messages from certain other senders set out in the attached list, please click here: https://www.unsubscribe.gwm.scotiabank.com?page=gwm 

This email may contain confidential information the use of which by an unintended recipient is unauthorized. This email may also contain important disclosure information for the records of the intended recipient(s).  For details please click here: http://www.scotiabank.com/email_disclaimer/email_english.html

Pour vous désabonner et cesser de recevoir des messages électroniques commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici : https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK

Pour vous désabonner et cesser de recevoir des messages électroniques commerciaux de certains autres expéditeurs figurant dans la liste ci-jointe, veuillez cliquer ici : https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr 

Cette transmission peut contenir de l'information confidentielle et son utilisation par toute personne autre que la personne à laquelle cette transmission est destinée est interdite. Le présent courriel peut aussi contenir des renseignements importants pour les dossiers du ou des destinataires prévus.  Pour plus de détails, veuillez vous diriger vers: http://www.scotiabank.com/email_disclaimer/email_francais.html

Para dejar de recibir mensajes electrónicos comerciales de The Bank of Nova Scotia en Canadá, haga clic aquí: https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK

Para dejar de recibir mensajes electrónicos comerciales de otras entidades incluidas en la lista adjunta, haga clic aquí: https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es

Este correo electrónico puede contener información confidencial cuyo uso por parte de personas distintas de los destinatarios del mismo está prohibido. El mensaje puede tener también datos importantes para los archivos   del(de los) destinatario(s) a quien(es) está dirigido.  Para más detalles, por favor dirigirse a: http://www.scotiabank.com/email_disclaimer/email_espanol.html
Reply | Threaded
Open this post in threaded view
|

Re: Automated Testing of Nifi Flow by injecting data into Processors

Andrew Psaltis
Hi Alexei,
If you're talking about testing your flow with simulated data as if it's
deployed, basically not unit tests. I've seen this done in several
different ways:
1. Using the GenerateFlowFile processor and adding in custom text. At
first, that may seem obvious, however, if you think about it in that custom
text field you can add in a block of JSON data that then in a subsequent
processor you can manipulate to populate various values and then convert it
to the format you need. From here you can have the data continue on to the
rest of your flow, or if you want to test the reading of data from a type
of source, perhaps File then you can write the data out to test location
and have your regular flow read it in. There are various ways you can take
this to make it as elaborate as necessary.

2. Build a custom processor to do the data generation. Here is an example
for TimeSeries data:
https://medium.com/hashmapinc/its-here-an-apache-nifi-simulator-for-generating-random-and-realistic-time-series-data-d7e463aa5e78

Does this help with idea on how to do it?



On Wed, Mar 6, 2019 at 11:50 AM Zenin, Alexei <[hidden email]>
wrote:

> Hi,
>
> Is there any way in Nifi to test out your whole flow without having the
> source systems online and running. I need this kind of capability to run
> automated integration tests which tests if the flow behaves as expected.
> None of the tools or libraries I have seen accomplish this and the Nifi API
> does not seem to allow to inject data directly into a processor (Nifi's
> TestRunner seems to be able to create FlowFiles but can we inject mock data
> to a processor through the Nifi REST API?). I think this would be a
> critical feature to make this system more production ready and abide by
> best practices for CICD.
>
> Regards,
> Alexei
> To unsubscribe from receiving further commercial electronic messages from
> The Bank of Nova Scotia in Canada, please click here:
> https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK
>
> To unsubscribe from receiving further commercial electronic messages from
> certain other senders set out in the attached list, please click here:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm
>
> This email may contain confidential information the use of which by an
> unintended recipient is unauthorized. This email may also contain important
> disclosure information for the records of the intended recipient(s).  For
> details please click here:
> http://www.scotiabank.com/email_disclaimer/email_english.html
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici :
> https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de certains autres expéditeurs figurant dans la liste
> ci-jointe, veuillez cliquer ici :
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr
>
> Cette transmission peut contenir de l'information confidentielle et son
> utilisation par toute personne autre que la personne à laquelle cette
> transmission est destinée est interdite. Le présent courriel peut aussi
> contenir des renseignements importants pour les dossiers du ou des
> destinataires prévus.  Pour plus de détails, veuillez vous diriger vers:
> http://www.scotiabank.com/email_disclaimer/email_francais.html
>
> Para dejar de recibir mensajes electrónicos comerciales de The Bank of
> Nova Scotia en Canadá, haga clic aquí:
> https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK
>
> Para dejar de recibir mensajes electrónicos comerciales de otras entidades
> incluidas en la lista adjunta, haga clic aquí:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es
>
> Este correo electrónico puede contener información confidencial cuyo uso
> por parte de personas distintas de los destinatarios del mismo está
> prohibido. El mensaje puede tener también datos importantes para los
> archivos   del(de los) destinatario(s) a quien(es) está dirigido.  Para más
> detalles, por favor dirigirse a:
> http://www.scotiabank.com/email_disclaimer/email_espanol.html
Reply | Threaded
Open this post in threaded view
|

Re: Automated Testing of Nifi Flow by injecting data into Processors

Bronislav Jitnikov
In reply to this post by Zenin, Alexei
I think, you can use https://pypi.org/project/nipyapi/ to create test
scenario for integration testing of complete flow without access data in
source system. I work on template for similar task, but only in start of my
way.
As an idea Create Python script that
1. deploy flow from Bucket to testing enviroment
2. load test data to quey after source system access processor (source)
3. Run flow except source step by step
4. Check result in queue after each processor.

May be create some module that can smiplify this task.

ср, 6 мар. 2019 г. в 19:50, Zenin, Alexei <[hidden email]>:

> Hi,
>
> Is there any way in Nifi to test out your whole flow without having the
> source systems online and running. I need this kind of capability to run
> automated integration tests which tests if the flow behaves as expected.
> None of the tools or libraries I have seen accomplish this and the Nifi API
> does not seem to allow to inject data directly into a processor (Nifi's
> TestRunner seems to be able to create FlowFiles but can we inject mock data
> to a processor through the Nifi REST API?). I think this would be a
> critical feature to make this system more production ready and abide by
> best practices for CICD.
>
> Regards,
> Alexei
> To unsubscribe from receiving further commercial electronic messages from
> The Bank of Nova Scotia in Canada, please click here:
> https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK
>
> To unsubscribe from receiving further commercial electronic messages from
> certain other senders set out in the attached list, please click here:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm
>
> This email may contain confidential information the use of which by an
> unintended recipient is unauthorized. This email may also contain important
> disclosure information for the records of the intended recipient(s).  For
> details please click here:
> http://www.scotiabank.com/email_disclaimer/email_english.html
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici :
> https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de certains autres expéditeurs figurant dans la liste
> ci-jointe, veuillez cliquer ici :
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr
>
> Cette transmission peut contenir de l'information confidentielle et son
> utilisation par toute personne autre que la personne à laquelle cette
> transmission est destinée est interdite. Le présent courriel peut aussi
> contenir des renseignements importants pour les dossiers du ou des
> destinataires prévus.  Pour plus de détails, veuillez vous diriger vers:
> http://www.scotiabank.com/email_disclaimer/email_francais.html
>
> Para dejar de recibir mensajes electrónicos comerciales de The Bank of
> Nova Scotia en Canadá, haga clic aquí:
> https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK
>
> Para dejar de recibir mensajes electrónicos comerciales de otras entidades
> incluidas en la lista adjunta, haga clic aquí:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es
>
> Este correo electrónico puede contener información confidencial cuyo uso
> por parte de personas distintas de los destinatarios del mismo está
> prohibido. El mensaje puede tener también datos importantes para los
> archivos   del(de los) destinatario(s) a quien(es) está dirigido.  Para más
> detalles, por favor dirigirse a:
> http://www.scotiabank.com/email_disclaimer/email_espanol.html