[mvapich-discuss] Same nodes different time?

[Gus Correa]

On 03/27/2014 05:58 PM, Daniel WEI wrote:
>
> On Thu, Mar 27, 2014 at 5:45 PM, Tony Ladd <tladd at che.ufl.edu
> <mailto:tladd at che.ufl.edu>> wrote:
>
>     So your performance can vary depending on what else is going on with
>     the other nodes in the system
>
>
> Thank you Tony. I see.
>
> (1) But how much variance?! My results shows some very disturbing
> difference, on one case, to initialize the case, it takes 37s, another
> 5s, yet another 2s!!!
> (2) How can I do my best, or somebody else to do their best, in order to
> reduce this variance? (there is 16 cores/node, so there should be nobody
> using the nodes I was calling, this seems to be guaranteed)
> (3) I goal is to compare intel compiler's -O3 and -O2 difference on
> building my CFD code concerning speed, but now if my performance vary
> even in the same case, same hosts, how can I trust my results anymore....?
> Zhigang Wei
> ----------------------
> /University of Notre Dame/
>

Hi Zhigang

What time are you measuring?
Wall time from the job scheduler for the whole job?
Wall time for the application only (say with Unix time utility or 
MPI_Wtime)?
Something else?

Have you tried to run your test simulations for a longer time (several 
minutes, one hour perhaps, not just a few seconds)
to see if the outcome shows less spread?
Say, you could change the number of time steps to 100x
or perhaps 10,000x what you are currently using,
depending of course on the max walltime allowed by your cluster queue.

My wild guess is that with short-lived simulations
what may count most is the job or application
startup and wrapup times, which may vary significantly in a cluster,
specially in a big cluster, overwhelming and obscuring your program 
execution time.
Most MPI and benchmark implementations recommend
that you "warm up" your own tests/benchmarks
for a time long enough to reduce such startup/wrapup effects.

My two cents,
Gus Correa