LivySessionController - Unexpected Behavior

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

LivySessionController - Unexpected Behavior

Peter Wicks (pwicks)
I've been working on a few enhancements to the LivySessionController as part of my ticket NIFI-6175 : Livy - Add Support for All Missing Session Startup Features.

I've been running into a lot of... interesting? features in the code, and I was hoping someone (Matt?) who was involved with it from the beginning could shed some light on these questions before I change any default behavior/fix any perceived bugs.

  *   If NiFi is running in a cluster then a race condition is entered where you can't really predict how many Livy sessions will be created.
     *   If the controller service was recently running, and you just restarted the controller service, then everything is fine. The existing Livy sessions will be found and used. But even in this scenario it's not working correctly, because if I asked the controller service to use two instances, it will use all available sessions in Livy for the "Kind" (more on this farther down).
     *   The logic checks if any sessions exist on controller startup (and on each update interval), since all instances of the controller service start up roughly at the same time, you might end up with full duplication across the cluster, or you might end up with 50% duplication, or anything in between, depending on how quickly session create requests get sent in.
  *   The controller service will "steal" any open Livy session that it can see, so long as the "Kind" of session matches the configuration. It will also over reach in session allocation if more sessions are available then it needs.
     *   If there are 10 Livy sessions open, it will load all 10 as available for use, even if I only wanted 2. If some of those sessions die off it does not create new ones, but it will keep using them as long as they are available.
     *   If you have multiple Livy Controller Services, it's very hard (impossible?) to keep sessions separate if they are running under the same account (and maybe even if they are not, have not spent much time testing the separate account option).
     *   The code does not block the sessions/mark it as used. It relies on the Livy Session state value of "Idle" to designate a session as available. This is another race condition where running multiple threads of ExecuteSparkInteractive, either because your in a cluster, or because you just have multiple threads, would easily dual assign a Livy session instead of using the expected WAIT relationship.
  *   The Controller Service is unable to delete existing sessions. So even though there is a Controller Service shutdown hook in the code, it does not clean up it's open sessions and they have to time-out.

I don't have resolutions for all these issues. But one thing I was thinking about doing was using the Livy Session Name parameter to tag the session when it's created so it's associated with a specific Controller Service instance by UUID (so it would work across a cluster too), and maybe only manage Livy sessions from the master node? (not sure how to find out if your on Master, but a thought).