[OOD-users] Performance issues

[Franz, Eric]

Dan,

On the first request to the dashboard (anything to /pun/sys/dashboard) Passenger loads the dashboard app. It is a Rails app and part of the startup includes loading the initializers. We added loading of custom initializers from /etc for configuration that is easiest to setup via some Ruby code and that we haven’t added a special configuration file for.  Also, arbitrary code or monkey patches can be added to the app there (thus my suggestion below).

User logging out and back in would not trigger it. I think that the default configuration is after 5 minutes of inactivity the Passenger app process is killed. If the user clicked Help -> Restart Web Server then their PUN would restart, and thus all Passenger app processes would be killed as well.

I hope that helps.

Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu

From: Dan Dietz <ddietz at purdue.edu>
Date: Friday, November 9, 2018 at 2:12 PM
To: "Franz, Eric" <efranz at osc.edu>
Cc: "ood-users at lists.osc.edu" <ood-users at lists.osc.edu>
Subject: Re: [OOD-users] Performance issues

Yes, our qlist will return different results between users and clusters. When exactly does a "Dashboard load" happen? Would a user logging out and reopening the browser trigger that or are we talking about at the server level restart? Refreshing the cache at login for a user would be plenty good enough I think. Either way, I think your patch will work for the time being - we are likely 2-3 months away from going to any level of production with this so no hurry on a robust solution for us.

I will give this a try and let you know.

Thanks again!

Dan

On Fri, Nov 9, 2018 at 1:55 PM Franz, Eric <efranz at osc.edu<mailto:efranz at osc.edu>> wrote:
Dan,

If it does, this is one option. Add an /etc/ood/config/apps/dashboard/initializers/qlist.rb with contents like this:

module Purdue
  class Qlist
    def self.qlist(cluster)
      @qlist ||= Hash.new do |h, key|
        Rails.logger.warn "executing expensive qlist"
        h[key] = build_qlist(key)
      end

      @qlist[cluster]
    end

    def self.build_qlist(cluster)
      ["#{cluster}-debug", "#{cluster}-serial", "#{cluster}-parallel" ]
    end
  end
end

# call to build cache on initialize (optional)
Purdue::Qlist.qlist('owens')
Purdue::Qlist.qlist('ruby')
Purdue::Qlist.qlist('pitzer')
Purdue::Qlist.qlist('oakley')


You would replace build_qlist body with the method that executes qlist for the specified cluster. The other method qlist just employs per-argument memoization using Ruby’s Hash.new feature (the block provided to the hash is executed to generate the value if the value does not exist in the hash).

Then you would add calls below to initialize the cache (though this might not be necessary, since memoziation would take care of this).

In a form.yml.erb you would use <% Purdue::Qlist.qlist('your_cluster').each do |queue| %> to iterate over your queue list for that cluster.

If you need something more robust because your requirements dictate you need to have the qlist executed at the start of the web form load, then I’ll have an alternative solution for you early next week.

Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu<mailto:efranz at osc.edu>

From: OOD-users <ood-users-bounces+efranz=osc.edu at lists.osc.edu<mailto:osc.edu at lists.osc.edu>> on behalf of "Franz, Eric via OOD-users" <ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>>
Reply-To: "Franz, Eric" <efranz at osc.edu<mailto:efranz at osc.edu>>, User support mailing list for Open OnDemand <ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>>
Date: Friday, November 9, 2018 at 1:29 PM
To: Dan Dietz <ddietz at purdue.edu<mailto:ddietz at purdue.edu>>
Cc: "ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>" <ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>>
Subject: Re: [OOD-users] Performance issues

Dan,

Does the value of qlist vary from cluster to cluster/user to user, and is it something that could be cached on Dashboard load instead of something that is executed every time you open an iHPC app’s web form?

Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu<mailto:efranz at osc.edu>

From: Dan Dietz <ddietz at purdue.edu<mailto:ddietz at purdue.edu>>
Date: Friday, November 9, 2018 at 1:10 PM
To: "Franz, Eric" <efranz at osc.edu<mailto:efranz at osc.edu>>
Cc: "ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>" <ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>>
Subject: Re: [OOD-users] Performance issues

Eric,

Thanks for the quick response. You are right, it does not affect job composer, although there is a bit of delay opening it at first.

A temporary patch would be most appreciated.

Thanks,
Dan

On Fri, Nov 9, 2018 at 11:29 AM Franz, Eric <efranz at osc.edu<mailto:efranz at osc.edu>> wrote:
Dan,

From what you are describing, I would not expect it to affect the job composer. But I apologize, this is a big oversight on our part. I opened issues https://github.com/OSC/ood-dashboard/issues/417 and https://github.com/OSC/ood-dashboard/issues/418 to track this work.

We knew that the dashboard is inefficient in the menu building process and it is something we were planning to fix in the next three months. But it looks like this is higher priority.

The dashboard builds the menu dynamically on each request, instead of caching the information. This is great when you have multiple people doing app sharing or app development as the users PUN doesn’t have to be restarted to see a new app. It is also useful for an interactive app where description or title information changes system state. But those are edge cases, not the common case for most Open OnDemand sites. And as you observe it is problematic when the form.yml.erb contains expensive code.

I can provide a patch you can use till the next release. I’ll follow up with that in another email.

Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu<mailto:efranz at osc.edu>

From: OOD-users <ood-users-bounces+efranz=osc.edu at lists.osc.edu<mailto:osc.edu at lists.osc.edu>> on behalf of Dan Dietz via OOD-users <ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>>
Reply-To: Dan Dietz <ddietz at purdue.edu<mailto:ddietz at purdue.edu>>, User support mailing list for Open OnDemand <ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>>
Date: Friday, November 9, 2018 at 10:59 AM
To: "ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>" <ood-users at lists.osc.edu<mailto:ood-users at lists.osc.edu>>
Subject: [OOD-users] Performance issues

I'm currently testing implementing OOD here and all has gone quite smoothly so far (really, very nice work, I'm really enjoying working with it). I'm having one issue now though. As I started making the apps (I've got 6 so far) use form.yml.erb and dynamically display the user's available queues (select instead of free-form text box) and dynamically pull in versions from modules, things have gotten progressively slower in the dashboard (in apps, job composer, just loading the home page, anything) as I've improved more apps. It probably takes 10 seconds to load any page now.

Watching `top` on the server running OOD what I see are a ton of `qlist` (in-house command that lists eligible queues for the user the form template is using) processes flying by. What it feels like is the dashboard is building every dynamic form on every page load, even if I'm nowhere near any of the interactive apps.

Am I doing something wrong or has anyone else seen this?

Thanks!
Dan

--
Dan Dietz
Senior Scientific Applications Analyst
ITaP Research Computing, Purdue University


--
Dan Dietz
Senior Scientific Applications Analyst
ITaP Research Computing, Purdue University


--
Dan Dietz
Senior Scientific Applications Analyst
ITaP Research Computing, Purdue University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/ood-users/attachments/20181109/0d821f6a/attachment-0001.html>