[mvapich-discuss] Re: Running MVAPICH2 with SLURM Process Manager

[wei huang]

> * hard    nproc   16384
> * soft    nofile  4096
> * hard    nofile  65536
>
> * hard    memlock unlimited
> * soft    memlock unlimited
>

This only says you are allowed to set to unlimited. But it doesn't mean
the memlock limit has been set to that value.
>
> # End of file
> [molson] (root) security>
>
> THE SLURM DEAMONS WERE RESTARTED BETWEEN THESE TWO SNIPS
>
> [molson] (slurm) slurm> srun sh -c 'ulimit -l'
> 512

It shows that memlock limit is set to 512kB, which is not enough to run
mvapich2. You need to have the value set correctly so that the command
shows `unlimited'. Here is a brief instruction:

http://nowlab.cse.ohio-state.edu/projects/mpi-iba/mvapich_user_guide.html#x1-350007.2.3

Hope that works.

Thanks.

--Wei

> [molson] (slurm) slurm>
>
>
>
>
>
> wei huang <huanwei at cse.ohio-state.edu>
> 05/25/2006 09:38 AM
>
> To
> Don.Dhondt at Bull.com
> cc
> mvapich-discuss at cse.ohio-state.edu
> Subject
> Re: Running MVAPICH2 with SLURM Process Manager
>
>
>
>
>
>
> Hi,
>
> > The latest suggestion from LLNL is to make sure we have unlimited max
> > locked memory for our MPI tasks with:
>
> > srun sh -c 'ulimit -l'
>
> We wonder if you have tried this command? If things are setting up
> properly, then for every code it should return "unlimited" or the maximum
> size of physical memory you have on each node.
>
> > Below are the latest "traces" of the error.
>
> > 1: cannot create cq
>
> In most cases the "ulimit" is the cause for failing to create cq. So
> please let us know once you've tried it.
>
> Thanks
>
>
> Regards,
> Wei Huang
>
> 774 Dreese Lab, 2015 Neil Ave,
> Dept. of Computer Science and Engineering
> Ohio State University
> OH 43210
> Tel: (614)292-8501
>
>
> On Thu, 25 May 2006 mvapich-discuss-owner at cse.ohio-state.edu wrote:
>
> > As list administrator, your authorization is requested for the
> > following mailing list posting:
> >
> >     List:    mvapich-discuss at cse.ohio-state.edu
> >     From:    don.dhondt at bull.com
> >     Subject: Re: [openib-general] Running MVAPICH2 with SLURM Process
> Manager
> >     Reason:  Post by non-member to a members-only list
> >
> > At your convenience, visit:
> >
> >     http://mail.cse.ohio-state.edu/mailman/admindb/mvapich-discuss
> >
> > to approve or deny the request.
> >
>
>
>
>