[mvapich-discuss] PCI-X vs PCI-Express compile options:

[Sayantan Sur]

Hello Troy,

* On Jun,1 Troy Telford<ttelford at linuxnetworx.com> wrote :
> I've never really concerned myself much with how MVAPICH has different  
> configurations for PCI-X and PCI Express; that is changing however.
> 
> It seems that the difference is in a particular define (_PCI_X_ vs.  
> _PCI_EX_); and the only differences are #ifdefs in vbuf.h.  Looking at  
> vbuf.h, it seems there are only two differences:
> 	1.)  VBUF_TOTAL_SIZE is different
> 	2.)  viadev_vbuf_total_size is also different.
> 
> A casual observation is that when MVAPICH is compiled to use PCI-X, it  
> still works on PCI Express hardware (with the same speed for the small  
> number of nodes I was using).
> 
> Could somebody more knowledgeable of the workings of MVAPICH enlighten me  
> as to how important it is to compile it for the correct bus?

Thanks for taking a deeper look into MVAPICH. It seems to me that you
are either looking at the 0.9.7 release version (vapi & ch_gen2) OR at
the latest trunk (vapi). The latest trunk/ch_gen2 device doesn't require
the knowledge of the bus as such, rather the type of HCA being used.
This information is then used for choosing optimal MTU, good features
(like SRQ) in addition to the vbuf size.

To answer your question -- if you used the correct bus type while
compiling MVAPICH (the version you have), it will result in optimized
bandwidth for medium sized messages.


Thanks,
Sayantan.

-- 
http://www.cse.ohio-state.edu/~surs