For the latter problem, the new possibilities of configuring shared (JMS
2.0) subscriptions introduced by
https://issues.apache.org/jira/browse/NIFI-3983 seemed promising, especially
for our clustered setup (3 Nifi instances, which could then share the load
of JMS consumption equally).
However, we are experiencing message loss in the following configuration of
the ConsumeJMS Processor:
We're currently running Nifi 1.7.0. The message loss occurs for
comparatively high-traffic topics. We'v established a controlled
reproducible setup of sending 1000 messages, from which we are losing
roughly 200-400. When increasing the concurrent tasks, we are loosing much
less (even down to 0-9 messages), but still substantially.
Interestingly, in the following to situations, we do NOT experience message
(1) non-shared subscription
(2) shared durable subscription
Both are not really an option for our case, as (1) still has the problem
when the primary node switches and (2) is currently not desired by our Tibco
My current impression is that the way how Nifi schedules / triggers the JMS
consumption introduces little "breaks" in the shared non-durable JMS
subscriptions; messages which occur during these breaks are consequently
lost. Before going deeper into this direction by myself, it would be great
to have an opinion if this direction is promising? Or any other thoughts /
experiences on this topic?