[mvapich-discuss] Same nodes different time?

[Tony Ladd]

It can be large in my experience. I found in some cases a 10G infiniband 
network (in that case our universities HPC system) could be slower than 
gigabit ethernet (with highly optimized drivers and a flat network). Its 
true you have sole access to the nodes but not sole access to the switch 
network. One thing is to ask the sys admins if they will set aside a 
portion of the network for you sometime.

However before doing that you might consider the following. A CFD code 
should not be communication bound - the surface to volume ratio works in 
your favor if you have enough nodes per processor. I notice you were 
using 64 cores - at a rough guess I would say your problem should be 
5-10 million cells to get good parallel performance. If you want to test 
the compiler I would run on just 1 node (perhaps with a smaller 
problem). That is all you need to check the compiler - or even a single 
process. I doubt you will find much difference between O2 and O3. Better 
to check on the web for some magic flags. I found --no-prec-div (I think 
that was it) sped up one of my codes by a factor of 2. It prevents it 
from doing the IEEE error checking (or something like that) which can 
sometimes make a big difference. Someone at TACC put me on to that. 
There may well be others.

Tony


On 03/27/2014 05:58 PM, Daniel WEI wrote:
>
> On Thu, Mar 27, 2014 at 5:45 PM, Tony Ladd <tladd at che.ufl.edu 
> <mailto:tladd at che.ufl.edu>> wrote:
>
>     So your performance can vary depending on what else is going on
>     with the other nodes in the system
>
>
> Thank you Tony. I see.
>
> (1) But how much variance?! My results shows some very disturbing 
> difference, on one case, to initialize the case, it takes 37s, another 
> 5s, yet another 2s!!!
> (2) How can I do my best, or somebody else to do their best, in order 
> to reduce this variance? (there is 16 cores/node, so there should be 
> nobody using the nodes I was calling, this seems to be guaranteed)
> (3) I goal is to compare intel compiler's -O3 and -O2 difference on 
> building my CFD code concerning speed, but now if my performance vary 
> even in the same case, same hosts, how can I trust my results anymore....?
>
>
>
>
>
>
> Zhigang Wei
> ----------------------
> /University of Notre Dame/

-- 
Tony Ladd

Chemical Engineering Department
University of Florida
Gainesville, Florida 32611-6005
USA

Email: tladd-"(AT)"-che.ufl.edu
Web    http://ladd.che.ufl.edu

Tel:   (352)-392-6509
FAX:   (352)-392-9514