[mvapich-discuss] EP attributes' values

[jasjit singh]

 Hi 


I am using mvapich2-1.0.1 

While running  more than 64 processes on 8 nodes (each with 8 cores, 
64-bit, RHEL-2.6.9-42.ELsmp), I have observed some changes in certain 
attributes. 

1) 
Value of max_rdma_write_iov changes from 0 to 42. 
Value of max_rdma_read_iov also changes from 0 to a non-zero value. 
I want to know why there is such a dramatic change in these values.How 
should we proceed if we want to run more than 64 processes successfully ?

2) 
Value of max_message_size attribute in our stack is 4294967296 (i.e 4GB) 
that is returned in dat_ia_query(). So we are expecting MVAPICH to set 
the same value for max_message_size while setting DAT_EP_ATTR in EP 
creation. It is doing so if we run upto 64 processes. But if number of 
processes exceed 64, MVAPICH sets this value to 1024(i.e 1K). This is 
again a drastic change. And what is more surprising is it does post recv 
for size larger than 1K. MVAPICH, it seems, is on one hand limiting MAX 
MESSAGE SIZE and on the other hand posting larger data size. 

I am sure that changes in these values have nothing to do with the 
number of nodes (or oversubscription, I essentially mean).(CMIIW) 
These changes are only due to increase in number of processes. And one 
thing more I want to confirm is this has nothing to do with cluster type 
whether this is small, medium or large as the limit for number of 
processes for small cluster is 128. 

Regards, 
Jasjit Singh 


      __________________________________________________________
Not happy with your email address?.
Get the one you really want - millions of new email addresses available now at Yahoo! http://uk.docs.yahoo.com/ymail/new.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20080707/22842a17/attachment-0001.html