Interest in a time series simulator processor for NiFi

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Interest in a time series simulator processor for NiFi

Chris Herrera
Hi All,

I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.

Regards,
Chris

[1] - https://github.com/cetic/TSimulus
Reply | Threaded
Open this post in threaded view
|

Re: Interest in a time series simulator processor for NiFi

Matt Burgess
Chris,

This sounds great! IMO Realistic data generation in all forms is a great addition, looking forward to your contribution!

Regards,
Matt


> On Jul 7, 2017, at 10:18 AM, Chris Herrera <[hidden email]> wrote:
>
> Hi All,
>
> I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.
>
> Regards,
> Chris
>
> [1] - https://github.com/cetic/TSimulus
Reply | Threaded
Open this post in threaded view
|

Re: Interest in a time series simulator processor for NiFi

Lee Laim (leelaim)
Yes.   It will be very useful in testing flows that will run in non-perfect environments.   This will be much more elegant than using several generate flow files in parallel with different settings.  

Thanks!
Lee

[hidden email]



> On Jul 7, 2017, at 8:44 AM, Matt Burgess <[hidden email]> wrote:
>
> Chris,
>
> This sounds great! IMO Realistic data generation in all forms is a great addition, looking forward to your contribution!
>
> Regards,
> Matt
>
>
>> On Jul 7, 2017, at 10:18 AM, Chris Herrera <[hidden email]> wrote:
>>
>> Hi All,
>>
>> I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.
>>
>> Regards,
>> Chris
>>
>> [1] - https://github.com/cetic/TSimulus
Reply | Threaded
Open this post in threaded view
|

Aw: Re: Interest in a time series simulator processor for NiFi

Uwe Geercken
Just my two cents...

I have created a processor that generates CSV data.
 
You can see an example here: http://datamelt.weebly.com/blog/nifi-processor-generatedata
 
It generates data based on word lists, regular expressions or purely random. Date data can be generated in a way that the columns generated make sense. So e.g. a real date 2017-07-08 is generated randomly and you can have columns for year, month, day, weekday, week of year, etc that corespond to this date.
 
And when you use word lists, it will generate data for catagories such as airlines, seasons, car manufacturers, etc.
 
Hope this helps,
 
Uwe
 

Gesendet: Freitag, 07. Juli 2017 um 17:21 Uhr
Von: "Lee Laim (leelaim)" <[hidden email]>
An: "[hidden email]" <[hidden email]>
Betreff: Re: Interest in a time series simulator processor for NiFi
Yes. It will be very useful in testing flows that will run in non-perfect environments. This will be much more elegant than using several generate flow files in parallel with different settings.

Thanks!
Lee

[hidden email]



> On Jul 7, 2017, at 8:44 AM, Matt Burgess <[hidden email]> wrote:
>
> Chris,
>
> This sounds great! IMO Realistic data generation in all forms is a great addition, looking forward to your contribution!
>
> Regards,
> Matt
>
>
>> On Jul 7, 2017, at 10:18 AM, Chris Herrera <[hidden email]> wrote:
>>
>> Hi All,
>>
>> I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.
>>
>> Regards,
>> Chris
>>
>> [1] - https://github.com/cetic/TSimulus[https://github.com/cetic/TSimulus]