[mvapich-discuss] MVAPICH2 Error - Assertion
'current_bytes[vc->smp.local_nodes]==0' failed. (fwd)
Gopal Santhanaraman
santhana at cse.ohio-state.edu
Sun Jun 10 14:28:37 EDT 2007
Hi Thomas
Thanks for your reply.
We have tried out the communication patterns that you had reported
and they run fine even with the case where the two processes are
running on the same node.
I have attached the tests with this mail. These tests are
are from the mpich2 test suite (test4.c test4_am.c transpose.c).
You can also try out these tests on your system.
We have not been able to reproduce the error that you are reporting.
Can you give more insights into the application code that you are
running or if it is possible give us the application code.
Thanks
Gopal
> Date: Mon, 4 Jun 2007 08:37:23 -0700
> From: Thomas O'Shea <THOMAS.T.O'SHEA at saic.com>
> Reply-To: Thomas O'Shea <toshea at trg.saic.com>
> To: wei huang <huanwei at cse.ohio-state.edu>
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: Re: [mvapich-discuss] MVAPICH2 Error - Assertion
> 'current_bytes[vc->smp.local_nodes]==0' failed.
>
> We migrated over to gen2 (OpenFabrics ) and we are still getting the same
> errors. I was wondering if you found anything, or have any ideas of what to
> try next.
>
> Thanks,
> Tom
> ----- Original Message -----
> From: "wei huang" <huanwei at cse.ohio-state.edu>
> To: "Thomas O'Shea" <toshea at trg.saic.com>
> Cc: <mvapich-discuss at cse.ohio-state.edu>
> Sent: Friday, May 04, 2007 3:06 PM
> Subject: Re: [mvapich-discuss] MVAPICH2 Error - Assertion
> 'current_bytes[vc->smp.local_nodes]==0' failed.
>
>
> > Hi Thomas,
> >
> > Thanks for your reply.
> >
> > Because the source code of your application is not available to us, we
> > will do a code review of our code (or do you have a piece of code which
> > shows the problem that can be sent to us?)
> >
> > The reason I ask you to try gen2 (OpenFabrics) stack is because the whole
> > InfiniBand community is moving towards this. So actually most of our
> > efforts is spent on this front (though we still maintain certain necessary
> > maintenance and bug fixes for the vapi stack). You can find useful
> > information to install the OFED stack (OpenFabrics Enterprise
> > Distribution) here:
> >
> > http://www.openfabrics.org/downloads.htm
> >
> > And the information to compile mvapich2 with OFED stack is avaialable
> > through our website.
> >
> > Anyway, we will get back to you once we find something.
> >
> > Thanks.
> >
> > Regards,
> > Wei Huang
> >
> > 774 Dreese Lab, 2015 Neil Ave,
> > Dept. of Computer Science and Engineering
> > Ohio State University
> > OH 43210
> > Tel: (614)292-8501
> >
> >
> > On Fri, 4 May 2007, Thomas O'Shea wrote:
> >
> > > Thanks for the response.
> > >
> > > 1) Turns out we are using mvapich2-0.9.8p1 already.
> > >
> > > 2) Yes, the standard compiling scripts were used.
> > >
> > > 3) You are correct, most of the communication involves one sided
> operations
> > > with passive synchronization. The code also uses a few other MPI
> commands.
> > >
> > > We define MPI vector types:
> > >
> > > CALL MPI_TYPE_VECTOR(xlen,nguard,iu_bnd,MPI_DOUBLE_PRECISION,
> > > & xtype,ierr)
> > >
> > > CALL MPI_TYPE_COMMIT(xtype,ierr)
> > >
> > > Create MPI Windows:
> > >
> > > CALL MPI_WIN_CREATE(work,winsize,8,MPI_INFO_NULL,
> > > & MPI_COMM_WORLD,win,ierr)
> > >
> > > Synch our gets with lock and unlock:
> > >
> > > CALL MPI_WIN_LOCK(MPI_LOCK_SHARED,get_pe,0,win,ierr)
> > > CALL MPI_GET(wget,1,xtype,get_pe,
> > > & targ_disp,1,xtype,win,ierr)
> > > CALL MPI_WIN_UNLOCK(get_pe,win,ierr)
> > >
> > > We use one broadcast call
> > >
> > > call MPI_BCAST(qxyz,3*maxpan,MPI_DOUBLE_PRECISION,0,
> > > 1 MPI_COMM_WORLD,ierr)
> > >
> > > And of course barriers and freeing the windows and vector types.
> > >
> > > The error we are getting happens on a MPI_WIN_UNLOCK after a GET call
> that
> > > does not use the MPI_TYPE_VECTOR that we created though. The ierr from
> the
> > > GET call is 0 as well.
> > >
> > >
> > > 4) I talked with the IT person in charge of this cluster and he said
> that we
> > > could try that, but he said the documentation he found on gen2 and udapl
> was
> > > somewhat sparse in that he wasn't sure exactly how to set that up and
> what
> > > the different compilations actually do differently. Is there any
> resource
> > > you can point us towards?
> > >
> > > Thanks,
> > > Tom
> > >
> > >
> > > > Hi Thomas,
> > > >
> > > > We will look into this issue. Would you please let us know the
> following:
> > > >
> > > > 1) We have recently made a couple of bug fixes and released
> > > > mvapich2-0.9.8p1. Would you first try that version?
> > > >
> > > > And if it is not working:
> > > >
> > > > 2) Did you use the standard compiling scripts (you mentioned ib gold
> > > > release, is it on vapi? And did you use make.mvapich2.vapi?)
> > > >
> > > > 3) Would you provide us some information on how the comunication
> patterns
> > > > of your application are? It seems like one sided operations with
> passive
> > > > synchronization (lock, get, unlock). Did you use other operations?
> > > >
> > > > 4) Will it possible for you to try gen2 (make.mvapich2.ofa) or udapl
> on
> > > > your stack, if they are available on your systems?
> > > >
> > > > Thanks.
> > > >
> > > > Regards,
> > > > Wei Huang
> > > >
> > > > 774 Dreese Lab, 2015 Neil Ave,
> > > > Dept. of Computer Science and Engineering
> > > > Ohio State University
> > > > OH 43210
> > > > Tel: (614)292-8501
> > > >
> > > >
> > > > On Thu, 3 May 2007, Thomas O'Shea wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I'm running the MVAPICH2-0.9.8 using the IB Gold Release. I've got 2
> > > > > 16 processor nodes (each has 8 dual-core AMD Opterons) hooked up
> > > > > through infiniband. I started off running this parallel Fortran code
> > > > > on just one node with MPICH2 and had no problems. It scaled decently
> > > > > to 8 processors but didn't see much improvement with the jump to 16
> > > > > (possibly due to cache coherency or something). Now, when trying to
> > > > > get it running across the infiniband connect I get this error:
> > > > >
> > > > > current bytes 4, total bytes 28, remote id 1
> > > > > nfa_opt: ch3_smp_progress.c:2075: MPIDI_CH3I_SMP_pull_header:
> Assertion
> > > 'current_bytes[vc->smp.local_nodes] == 0' failed.
> > > > > rank 0 in job 1 nessie_32906 caused collective abort of all ranks
> > > > > exit status of rank 0: killed by signal 9
> > > > >
> > > > > This happens right after a one sided communication (MPI_GET) but
> > > > > before the MPI_WIN_UNLOCK call that follows. Also this is only with
> a
> > > > > process that is on the same node as the calling process, The MPI_GET
> > > > > call exits with no errors also.
> > > > >
> > > > > All the osu_benchmarks run with no problems. There were also no
> > > > > problems if I make a local mpd (mpd &) ring on a single node and run
> > > > > the code with MVAPICH2 with 2,4,8,or 16 processors. If I compile
> with
> > > > > the MPICH2 libraries there are no problems on a single node or
> running
> > > > > processes spread out on both nodes.
> > > > >
> > > > > Ever seen this before? Any help would be greatly appreciated.
> > > > >
> > > > > Thanks,
> > > > > Thomas O'Shea
> > > > > SAIC
> > >
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
-------------- next part --------------
/* -*- Mode: C; c-basic-offset:4 ; -*- */
/*
* (C) 2001 by Argonne National Laboratory.
* See COPYRIGHT in top-level directory.
*/
#include "mpi.h"
#include "stdio.h"
#include "stdlib.h"
#include "mpitest.h"
/* tests passive target RMA on 2 processes. tests the lock-single_op-unlock
optimization. */
#define SIZE1 100
#define SIZE2 200
int main(int argc, char *argv[])
{
int rank, nprocs, A[SIZE2], B[SIZE2], i;
MPI_Win win;
int errs = 0;
MTest_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
if (nprocs != 2) {
printf("Run this program with 2 processes\n");
MPI_Abort(MPI_COMM_WORLD,1);
}
if (rank == 0) {
for (i=0; i<SIZE2; i++) A[i] = B[i] = i;
MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);
for (i=0; i<SIZE1; i++) {
MPI_Win_lock(MPI_LOCK_SHARED, 1, 0, win);
MPI_Put(A+i, 1, MPI_INT, 1, i, 1, MPI_INT, win);
MPI_Win_unlock(1, win);
}
for (i=0; i<SIZE1; i++) {
MPI_Win_lock(MPI_LOCK_SHARED, 1, 0, win);
MPI_Get(B+i, 1, MPI_INT, 1, SIZE1+i, 1, MPI_INT, win);
MPI_Win_unlock(1, win);
}
MPI_Win_free(&win);
for (i=0; i<SIZE1; i++)
if (B[i] != (-4)*(i+SIZE1)) {
printf("Get Error: B[%d] is %d, should be %d\n", i, B[i], (-4)*(i+SIZE1));
errs++;
}
}
else { /* rank=1 */
for (i=0; i<SIZE2; i++) B[i] = (-4)*i;
MPI_Win_create(B, SIZE2*sizeof(int), sizeof(int), MPI_INFO_NULL,
MPI_COMM_WORLD, &win);
MPI_Win_free(&win);
for (i=0; i<SIZE1; i++) {
if (B[i] != i) {
printf("Put Error: B[%d] is %d, should be %d\n", i, B[i], i);
errs++;
}
}
}
/* if (rank==0) printf("Done\n");*/
MTest_Finalize(errs);
MPI_Finalize();
return 0;
}
-------------- next part --------------
/* -*- Mode: C; c-basic-offset:4 ; -*- */
/*
* (C) 2001 by Argonne National Laboratory.
* See COPYRIGHT in top-level directory.
*/
#include "mpi.h"
#include "stdio.h"
#include "stdlib.h"
#include "mpitest.h"
/* tests passive target RMA on 2 processes. tests the lock-single_op-unlock
optimization. */
/* same as test4.c but uses alloc_mem */
#define SIZE1 100
#define SIZE2 200
int main(int argc, char *argv[])
{
/* int rank, nprocs, A[SIZE2], B[SIZE2], i; */
int rank, nprocs, *A, *B, i;
MPI_Win win;
int errs = 0;
MTest_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
if (nprocs != 2) {
printf("Run this program with 2 processes\n");
MPI_Abort(MPI_COMM_WORLD,1);
}
i = MPI_Alloc_mem(SIZE2 * sizeof(int), MPI_INFO_NULL, &A);
if (i) {
printf("Can't allocate memory in test program\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}
i = MPI_Alloc_mem(SIZE2 * sizeof(int), MPI_INFO_NULL, &B);
if (i) {
printf("Can't allocate memory in test program\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}
if (rank == 0) {
for (i=0; i<SIZE2; i++) A[i] = B[i] = i;
MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);
for (i=0; i<SIZE1; i++) {
MPI_Win_lock(MPI_LOCK_SHARED, 1, 0, win);
MPI_Put(A+i, 1, MPI_INT, 1, i, 1, MPI_INT, win);
MPI_Win_unlock(1, win);
}
for (i=0; i<SIZE1; i++) {
MPI_Win_lock(MPI_LOCK_SHARED, 1, 0, win);
MPI_Get(B+i, 1, MPI_INT, 1, SIZE1+i, 1, MPI_INT, win);
MPI_Win_unlock(1, win);
}
MPI_Win_free(&win);
for (i=0; i<SIZE1; i++)
if (B[i] != (-4)*(i+SIZE1)) {
printf("Get Error: B[%d] is %d, should be %d\n", i, B[i], (-4)*(i+SIZE1));
errs++;
}
}
else { /* rank=1 */
for (i=0; i<SIZE2; i++) B[i] = (-4)*i;
MPI_Win_create(B, SIZE2*sizeof(int), sizeof(int), MPI_INFO_NULL,
MPI_COMM_WORLD, &win);
MPI_Win_free(&win);
for (i=0; i<SIZE1; i++) {
if (B[i] != i) {
printf("Put Error: B[%d] is %d, should be %d\n", i, B[i], i);
errs++;
}
}
}
MPI_Free_mem(A);
MPI_Free_mem(B);
/* if (rank==0) printf("Done\n");*/
MTest_Finalize(errs);
MPI_Finalize();
return 0;
}
-------------- next part --------------
/* -*- Mode: C; c-basic-offset:4 ; -*- */
/*
* (C) 2001 by Argonne National Laboratory.
* See COPYRIGHT in top-level directory.
*/
#include "mpi.h"
#include "stdio.h"
#include "mpitest.h"
/* transposes a matrix using passive target RMA and derived
datatypes. Uses vector and hvector (Example 3.32 from MPI 1.1
Standard). Run on 2 processes. */
#define NROWS 100
#define NCOLS 100
int main(int argc, char *argv[])
{
int rank, nprocs, A[NROWS][NCOLS], i, j;
MPI_Win win;
MPI_Datatype column, xpose;
int errs = 0;
MTest_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
if (nprocs != 2) {
printf("Run this program with 2 processes\n");
MPI_Abort(MPI_COMM_WORLD,1);
}
if (rank == 0) {
for (i=0; i<NROWS; i++)
for (j=0; j<NCOLS; j++)
A[i][j] = i*NCOLS + j;
/* create datatype for one column */
MPI_Type_vector(NROWS, 1, NCOLS, MPI_INT, &column);
/* create datatype for matrix in column-major order */
MPI_Type_hvector(NCOLS, 1, sizeof(int), column, &xpose);
MPI_Type_commit(&xpose);
MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);
MPI_Win_lock(MPI_LOCK_SHARED, 1, 0, win);
MPI_Put(A, NROWS*NCOLS, MPI_INT, 1, 0, 1, xpose, win);
MPI_Type_free(&column);
MPI_Type_free(&xpose);
MPI_Win_unlock(1, win);
MPI_Win_free(&win);
}
else { /* rank = 1 */
for (i=0; i<NROWS; i++)
for (j=0; j<NCOLS; j++)
A[i][j] = -1;
MPI_Win_create(A, NROWS*NCOLS*sizeof(int), sizeof(int), MPI_INFO_NULL,
MPI_COMM_WORLD, &win);
MPI_Win_free(&win);
for (j=0; j<NCOLS; j++)
for (i=0; i<NROWS; i++)
if (A[j][i] != i*NCOLS + j) {
printf("Error: A[%d][%d]=%d should be %d\n", j, i,
A[j][i], i*NCOLS + j);
errs++;
}
}
/* if (rank==0) printf("Done\n"); */
MTest_Finalize(errs);
MPI_Finalize();
return 0;
}
More information about the mvapich-discuss
mailing list