Re: Example Load Profile

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Example Load Profile

Stephen Rathnam
Hi everyone,

I am also starting to do some performance testing of a similar nature. Did
anyone find a good load profile/xml flow to use for this?

Regards,
Stephen

-----------------------------------------------------------------
From: James Farrington <[hidden email]>
Date: Mon, Jun 19, 2017 at 11:26 AM
Subject: Example Load Profile
To: [hidden email]


Hello All,

I am trying to get a performance benchmark for what size load my server can
handle. Currently we are running 3 nodes each with the following
configuration: 4 CPUs, 30.5GB RAM, and 80GB SSD.

Additionally, we are currently using an example XML file from online
(attached).

First off, would anyone have a good idea of a sample load profile that
would give a good baseline of how much my setup can handle? Secondly, would
anyone suggest a different XML file to use for a bench marking test (this
one was just found online).

In my search thus far I have not found any documentation on an example load
profile for a given configuration, so it would perhaps be helpful to others
in the future to have this formally documented. As well as adding a good
sample XML file for getting some benchmark data.

Any advice would be great!

Thanks,
James
Reply | Threaded
Open this post in threaded view
|

Re: Example Load Profile

Mark Payne
James,

Sorry, I let this slip through the cracks. Just took a look at the template that you attached.

What you're doing here should certainly work okay with NiFi. However, the pattern that you are
using is to take a large XML file, split it into small chunks, process each chunk individually, and
then merge the small chunks back into a large chunk. This tends to be quite expensive and may
not yield the best performance.

NiFi would actually perform *far* better if you perform your processing without splitting that data
apart and then re-joining it. To that end, in NiFi 1.2.0 and 1.3.0 we have introduced several new
processors for handling record-oriented data [1] [2]. Unfortunately, we do not yet have a Record
Reader for XML data. However, there is a ScriptedRecordReader that could be used for scripting
one out, and I've seen one float around somewhere. Perhaps someone else on the list is able to
provide something there?

This approach, though, should make the flow much simpler to use and maintain. You'd be able to
unzip the data, then use QueryRecord to filter out the records you don't want, and then write the
results in JSON. So your flow would probably be as simple as:

GetFile -> UnpackContent -> QueryRecord -> PutFile

Thanks
-Mark

[1] https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
[2] https://blogs.apache.org/nifi/entry/real-time-sql-on-event




On Jun 28, 2017, at 10:12 AM, Stephen Rathnam <[hidden email]<mailto:[hidden email]>> wrote:

Hi everyone,

I am also starting to do some performance testing of a similar nature. Did
anyone find a good load profile/xml flow to use for this?

Regards,
Stephen

-----------------------------------------------------------------
From: James Farrington <[hidden email]<mailto:[hidden email]>>
Date: Mon, Jun 19, 2017 at 11:26 AM
Subject: Example Load Profile
To: [hidden email]<mailto:[hidden email]>


Hello All,

I am trying to get a performance benchmark for what size load my server can
handle. Currently we are running 3 nodes each with the following
configuration: 4 CPUs, 30.5GB RAM, and 80GB SSD.

Additionally, we are currently using an example XML file from online
(attached).

First off, would anyone have a good idea of a sample load profile that
would give a good baseline of how much my setup can handle? Secondly, would
anyone suggest a different XML file to use for a bench marking test (this
one was just found online).

In my search thus far I have not found any documentation on an example load
profile for a given configuration, so it would perhaps be helpful to others
in the future to have this formally documented. As well as adding a good
sample XML file for getting some benchmark data.

Any advice would be great!

Thanks,
James