How to calculate and set fragment.count attribute for MergeRecord

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to calculate and set fragment.count attribute for MergeRecord

Hemal Padhiar
Hi All,

Below is my use case:
Flow:
1. I have multiple zip files and read it from a folder
2. I use CompressContent processor unzip content -> contains multiple json
files
3. Each json file is an array of json object I use split json processor to
extract individual json object
4. Each json object contains nested json array, I extract each nested array
object and write to a single file using mergeRecord processor

MergeRecord with defragment, csvReader, csvRecordSetWriter and
schemaRegistry and updating fragment.identifier (using updateAttribute
processor prior to mergeRecord) as filename so that all records from single
seed file are kept in single file. My question is how to set fragment.count
(giving round figure, say 1000 creates multiple files each with 1000
records but the remainder remains in the queue )

Also, how can I get summary stats like number of nested array records
exratcted across all json files.

Thanks & regards,
Hemal