[mvapich-discuss] Re: Running MVAPICH2 with SLURM Process Manager

[Kevin Ball]

Hi Don,

  SLURM propagates a number of variables to remote nodes from your host
node, and the ulimit appears to be one of them.  I'm not 100% sure from
reading your log, but do you have ulimit set on your remote node or the
one you are launching slurm from?  You will need to set it on the local
node or explicitly set it within your slurm job (e.g. sourcing a script
in your .bashrc).  Let me know if this solves your problem.

Kevin Ball
Senior Engineer
SIG, QLogic Corp.

On Thu, 2006-05-25 at 18:48, Don.Dhondt at bull.com wrote:
> No joy yet...
> 
> [molson] (root) security> cat limits.conf
> # /etc/security/limits.conf
> #
> #Each line describes a limit for a user in the form:
> #
> #<domain>        <type>  <item>  <value>
> #
> #Where:
> #<domain> can be:
> #        - an user name
> #        - a group name, with @group syntax
> #        - the wildcard *, for default entry
> #        - the wildcard %, can be also used with %group syntax,
> #                 for maxlogin limit
> #
> #<type> can have the two values:
> #        - "soft" for enforcing the soft limits
> #        - "hard" for enforcing hard limits
> #
> #<item> can be one of the following:
> #        - core - limits the core file size (KB)
> #        - data - max data size (KB)
> #        - fsize - maximum filesize (KB)
> #        - memlock - max locked-in-memory address space (KB)
> #        - nofile - max number of open files
> #        - rss - max resident set size (KB)
> #        - stack - max stack size (KB)
> #        - cpu - max CPU time (MIN)
> #        - nproc - max number of processes
> #        - as - address space limit
> #        - maxlogins - max number of logins for this user
> #        - priority - the priority to run user process with
> #        - locks - max number of file locks the user can hold
> #
> 
> * soft    nproc   2047
> * hard    nproc   16384
> * soft    nofile  4096
> * hard    nofile  65536
> 
> * hard    memlock unlimited
> * soft    memlock unlimited
> 
> 
> # End of file
> [molson] (root) security>
> 
> THE SLURM DEAMONS WERE RESTARTED BETWEEN THESE TWO SNIPS
> 
> [molson] (slurm) slurm> srun sh -c 'ulimit -l'
> 512
> [molson] (slurm) slurm> 
> 
> 
> 
> 
> wei huang
> <huanwei at cse.ohio-state.edu>
> 
> 05/25/2006 09:38 AM
>                To
> Don.Dhondt at Bull.com
>                cc
> mvapich-discuss at cse.ohio-state.edu
>           Subject
> Re: Running
> MVAPICH2 with
> SLURM Process
> Manager
> 
> 
> 
> 
> Hi,
> 
> > The latest suggestion from LLNL is to make sure we have unlimited
> max
> > locked memory for our MPI tasks with:
> 
> > srun sh -c 'ulimit -l'
> 
> We wonder if you have tried this command? If things are setting up
> properly, then for every code it should return "unlimited" or the
> maximum
> size of physical memory you have on each node.
> 
> > Below are the latest "traces" of the error.
> 
> > 1: cannot create cq
> 
> In most cases the "ulimit" is the cause for failing to create cq. So
> please let us know once you've tried it.
> 
> Thanks
> 
> 
> Regards,
> Wei Huang
> 
> 774 Dreese Lab, 2015 Neil Ave,
> Dept. of Computer Science and Engineering
> Ohio State University
> OH 43210
> Tel: (614)292-8501
> 
> 
> On Thu, 25 May 2006 mvapich-discuss-owner at cse.ohio-state.edu wrote:
> 
> > As list administrator, your authorization is requested for the
> > following mailing list posting:
> >
> >     List:    mvapich-discuss at cse.ohio-state.edu
> >     From:    don.dhondt at bull.com
> >     Subject: Re: [openib-general] Running MVAPICH2 with SLURM
> Process Manager
> >     Reason:  Post by non-member to a members-only list
> >
> > At your convenience, visit:
> >
> >     http://mail.cse.ohio-state.edu/mailman/admindb/mvapich-discuss
> >
> > to approve or deny the request.
> >
> 
> 
> 
> 
> 
> ______________________________________________________________________
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss