[mvapich-discuss] MVAPICH2 Error - Assertion
'current_bytes[vc->smp.local_nodes]==0' failed.
Sylvain Jeaugey
sylvain.jeaugey at bull.net
Thu Jun 7 03:28:08 EDT 2007
The workaround is to use another way to communicate inside a node. The
first and possibly only idea is to disable _SMP_, of course, but
performance may be worse (or better ... depending on your architecture and
the size of your nodes).
We could workaround this problem since we use a bull-customized version of
MVAPICH2 embedding another shared-memory mecanism for intra-node
communication. Using this "other" device also made the problem disappear,
which confirms that this problem is related to the _SMP_ code.
I don't know if MVAPICH 1 would perform better, but you may want to give
it a try.
Sylvain
On Wed, 6 Jun 2007, Thomas O'Shea wrote:
> Did you ever find a work-around?
>
> Thanks,
> Tom
> ----- Original Message -----
> From: "Sylvain Jeaugey" <sylvain.jeaugey at bull.net>
> To: "Thomas O'Shea" <toshea at trg.saic.com>
> Cc: "wei huang" <huanwei at cse.ohio-state.edu>;
> <mvapich-discuss at cse.ohio-state.edu>
> Sent: Monday, June 04, 2007 9:29 AM
> Subject: Re: [mvapich-discuss] MVAPICH2 Error - Assertion
> 'current_bytes[vc->smp.local_nodes]==0' failed.
>
>
>> Hi all,
>>
>> For the record, this is an error I already encountered. [I didn't report
>> it since I'm still using an old mvapich tree.]
>> Unfortunately, we also don't have a simple way to reproduce it.
>>
>> Sylvain
>>
>> On Mon, 4 Jun 2007, Thomas O'Shea wrote:
>>
>>> We migrated over to gen2 (OpenFabrics ) and we are still getting the
> same
>>> errors. I was wondering if you found anything, or have any ideas of what
> to
>>> try next.
>>>
>>> Thanks,
>>> Tom
>>> ----- Original Message -----
>>> From: "wei huang" <huanwei at cse.ohio-state.edu>
>>> To: "Thomas O'Shea" <toshea at trg.saic.com>
>>> Cc: <mvapich-discuss at cse.ohio-state.edu>
>>> Sent: Friday, May 04, 2007 3:06 PM
>>> Subject: Re: [mvapich-discuss] MVAPICH2 Error - Assertion
>>> 'current_bytes[vc->smp.local_nodes]==0' failed.
>>>
>>>
>>>> Hi Thomas,
>>>>
>>>> Thanks for your reply.
>>>>
>>>> Because the source code of your application is not available to us, we
>>>> will do a code review of our code (or do you have a piece of code which
>>>> shows the problem that can be sent to us?)
>>>>
>>>> The reason I ask you to try gen2 (OpenFabrics) stack is because the
> whole
>>>> InfiniBand community is moving towards this. So actually most of our
>>>> efforts is spent on this front (though we still maintain certain
> necessary
>>>> maintenance and bug fixes for the vapi stack). You can find useful
>>>> information to install the OFED stack (OpenFabrics Enterprise
>>>> Distribution) here:
>>>>
>>>> http://www.openfabrics.org/downloads.htm
>>>>
>>>> And the information to compile mvapich2 with OFED stack is avaialable
>>>> through our website.
>>>>
>>>> Anyway, we will get back to you once we find something.
>>>>
>>>> Thanks.
>>>>
>>>> Regards,
>>>> Wei Huang
>>>>
>>>> 774 Dreese Lab, 2015 Neil Ave,
>>>> Dept. of Computer Science and Engineering
>>>> Ohio State University
>>>> OH 43210
>>>> Tel: (614)292-8501
>>>>
>>>>
>>>> On Fri, 4 May 2007, Thomas O'Shea wrote:
>>>>
>>>>> Thanks for the response.
>>>>>
>>>>> 1) Turns out we are using mvapich2-0.9.8p1 already.
>>>>>
>>>>> 2) Yes, the standard compiling scripts were used.
>>>>>
>>>>> 3) You are correct, most of the communication involves one sided
>>> operations
>>>>> with passive synchronization. The code also uses a few other MPI
>>> commands.
>>>>>
>>>>> We define MPI vector types:
>>>>>
>>>>> CALL MPI_TYPE_VECTOR(xlen,nguard,iu_bnd,MPI_DOUBLE_PRECISION,
>>>>> & xtype,ierr)
>>>>>
>>>>> CALL MPI_TYPE_COMMIT(xtype,ierr)
>>>>>
>>>>> Create MPI Windows:
>>>>>
>>>>> CALL MPI_WIN_CREATE(work,winsize,8,MPI_INFO_NULL,
>>>>> & MPI_COMM_WORLD,win,ierr)
>>>>>
>>>>> Synch our gets with lock and unlock:
>>>>>
>>>>> CALL MPI_WIN_LOCK(MPI_LOCK_SHARED,get_pe,0,win,ierr)
>>>>> CALL MPI_GET(wget,1,xtype,get_pe,
>>>>> & targ_disp,1,xtype,win,ierr)
>>>>> CALL MPI_WIN_UNLOCK(get_pe,win,ierr)
>>>>>
>>>>> We use one broadcast call
>>>>>
>>>>> call MPI_BCAST(qxyz,3*maxpan,MPI_DOUBLE_PRECISION,0,
>>>>> 1 MPI_COMM_WORLD,ierr)
>>>>>
>>>>> And of course barriers and freeing the windows and vector types.
>>>>>
>>>>> The error we are getting happens on a MPI_WIN_UNLOCK after a GET call
>>> that
>>>>> does not use the MPI_TYPE_VECTOR that we created though. The ierr from
>>> the
>>>>> GET call is 0 as well.
>>>>>
>>>>>
>>>>> 4) I talked with the IT person in charge of this cluster and he said
>>> that we
>>>>> could try that, but he said the documentation he found on gen2 and
> udapl
>>> was
>>>>> somewhat sparse in that he wasn't sure exactly how to set that up and
>>> what
>>>>> the different compilations actually do differently. Is there any
>>> resource
>>>>> you can point us towards?
>>>>>
>>>>> Thanks,
>>>>> Tom
>>>>>
>>>>>
>>>>>> Hi Thomas,
>>>>>>
>>>>>> We will look into this issue. Would you please let us know the
>>> following:
>>>>>>
>>>>>> 1) We have recently made a couple of bug fixes and released
>>>>>> mvapich2-0.9.8p1. Would you first try that version?
>>>>>>
>>>>>> And if it is not working:
>>>>>>
>>>>>> 2) Did you use the standard compiling scripts (you mentioned ib gold
>>>>>> release, is it on vapi? And did you use make.mvapich2.vapi?)
>>>>>>
>>>>>> 3) Would you provide us some information on how the comunication
>>> patterns
>>>>>> of your application are? It seems like one sided operations with
>>> passive
>>>>>> synchronization (lock, get, unlock). Did you use other operations?
>>>>>>
>>>>>> 4) Will it possible for you to try gen2 (make.mvapich2.ofa) or udapl
>>> on
>>>>>> your stack, if they are available on your systems?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Regards,
>>>>>> Wei Huang
>>>>>>
>>>>>> 774 Dreese Lab, 2015 Neil Ave,
>>>>>> Dept. of Computer Science and Engineering
>>>>>> Ohio State University
>>>>>> OH 43210
>>>>>> Tel: (614)292-8501
>>>>>>
>>>>>>
>>>>>> On Thu, 3 May 2007, Thomas O'Shea wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm running the MVAPICH2-0.9.8 using the IB Gold Release. I've got 2
>>>>>>> 16 processor nodes (each has 8 dual-core AMD Opterons) hooked up
>>>>>>> through infiniband. I started off running this parallel Fortran code
>>>>>>> on just one node with MPICH2 and had no problems. It scaled decently
>>>>>>> to 8 processors but didn't see much improvement with the jump to 16
>>>>>>> (possibly due to cache coherency or something). Now, when trying to
>>>>>>> get it running across the infiniband connect I get this error:
>>>>>>>
>>>>>>> current bytes 4, total bytes 28, remote id 1
>>>>>>> nfa_opt: ch3_smp_progress.c:2075: MPIDI_CH3I_SMP_pull_header:
>>> Assertion
>>>>> 'current_bytes[vc->smp.local_nodes] == 0' failed.
>>>>>>> rank 0 in job 1 nessie_32906 caused collective abort of all ranks
>>>>>>> exit status of rank 0: killed by signal 9
>>>>>>>
>>>>>>> This happens right after a one sided communication (MPI_GET) but
>>>>>>> before the MPI_WIN_UNLOCK call that follows. Also this is only with
>>> a
>>>>>>> process that is on the same node as the calling process, The MPI_GET
>>>>>>> call exits with no errors also.
>>>>>>>
>>>>>>> All the osu_benchmarks run with no problems. There were also no
>>>>>>> problems if I make a local mpd (mpd &) ring on a single node and run
>>>>>>> the code with MVAPICH2 with 2,4,8,or 16 processors. If I compile
>>> with
>>>>>>> the MPICH2 libraries there are no problems on a single node or
>>> running
>>>>>>> processes spread out on both nodes.
>>>>>>>
>>>>>>> Ever seen this before? Any help would be greatly appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas O'Shea
>>>>>>> SAIC
>>>>>
>>>
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>
>
>
More information about the mvapich-discuss
mailing list