[mvapich-discuss] Assumptions about page size in OSU benchmarks

[Mark Debbage]

I was running the OSU benchmarks on a SiCortex system recently, and hit
a bus error with osu_bw.c. The MPI folks at SiCortex discovered the
cause of the problem, and it was due to the value of MAX_ALIGNMENT being
set too low. These systems have a 64K page size and the effect of the
alignment adjustment code is to cause a buffer over-run:

#define MAX_ALIGNMENT 16384
#define MAX_MSG_SIZE (1<<22)
#define MYBUFSIZE (MAX_MSG_SIZE + MAX_ALIGNMENT)

...

char s_buf1[MYBUFSIZE];
char r_buf1[MYBUFSIZE];

...

    align_size = getpagesize();
    s_buf =
        (char *) (((unsigned long) s_buf1 + (align_size - 1)) /
                  align_size * align_size);
    r_buf =
        (char *) (((unsigned long) r_buf1 + (align_size - 1)) /
                  align_size * align_size);

I suggest increasing MAX_ALIGNMENT to at least 65536 bytes, and add
a run-time check that gives a suitable diagnostic if getpagesize() 
is larger than MAX_ALIGNMENT. Alternatively you could dynamically
allocate the arrays, perhaps using posix_memalign, to avoid
the problem. This affects osu_bw.c,  osu_bibw.c and osu_mbw_mr.c 
though I did not check all of the benchmarks for this.

Regards,

Mark Debbage.