[mvapich-discuss] ( Нет темы )

[Egor Tur]

 Hi All

> > > This error occur after applying this patch.
> > > Now I get latest version of mpiexec from svn, compile and install
> > > but errors are the same.
> >
> > The problem of memory locking is dependant on the startup method used. As
> > you noted, it works for mpirun_rsh, so the general setup sounds correct.
> > Per the discussion on this list regarding other process managers, there
> > should be a startup parameters file for mpiexec that will allow setting
> > the lockable memory limit. Unfortunately, I am not very familiar with
> > mpiexec and don't know any details for it. Perhaps someone else with more
> > knowledge of mpiexec can give their insights.
>
> Egor, I lost the thread here.  Were you having problems running out
> of locked memory for IB comms?  Or protocol issues due to changes in
> mvapich from 0.9.<older> to 0.9.9?  Maybe Matt has some insight I
> missed.
>
> For memory locking, /etc/security/limits.conf on redhat:
>    * hard memlock 4194304
>    * soft memlock 4194304
> or do "ulimit -l 4194304" in PBS startup script.
> 
> For protocol issues, you should run with "-v -v" and try to figure
> out what's going wrong.

I have solved this my problem of "Abort: Error creating CQ" when runing jobs from
mpiexec over IB. This is my stupid mistake. This realy is problem of max locked memory values
(as many people tell here) and batch system (in my case - torque) and not mpiexec or mvapich. 
I installed and configured batch system on running cluster and I not rebooted our cluster more 
than 5 months. After reboot I have this problem: when I submit jobs over batch the memlock velue 
takes 32K limit. After reboot pbs_mom server on each node mom server for memlock I think read 
/etc/security/limits.conf file and set velue to unlimited. I add "ulimit -l unlimited" to my pbs_mom 
startup script and this solve limits problem during start mom server.

Sorry for my question and
Thanx all for assistance.