[mvapich-discuss] Large jitter in message transfer

[Dan Kokron]

You might consider using TAU in TRACE mode to see what each process is
doing and when it's doing it.

(http://www.cs.uoregon.edu/research/tau/home.php)

Dan

On Fri, 2011-04-15 at 11:46 -0500, Sashi Balasingam wrote:
> Hi all, 
> 
> I recently started on an MPI-based, 'real-time', pipelined-processing
> application, and the application fails due to large time-jitter in
> sending and receiving messages. 
> 
> Here are related info - 
> 
>   
> 
> 1) Platform: 
> 
> a) Intel Box: Two Hex-core, Intel Xeon, 2.668 GHz (...total of 12
> cores/box) 
> 
> b) OS: SUSE Linux Enterprise Server 11 (x86_64) - Kernel \r (\l) 
> 
> c) MPI Rev:  mvapich2-1.6rc2 
> 
> d) HCA: InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR,
> PCIe 2.0 5GT/s] (rev a0) 
> 
>   
> 
> 2) Application detail: 
> 
>   
> 
> a) My app uses two of these boxes, with IB connection. I launch 11
> processes for a "Pipelined" Image processing application, where Box-1
> has 7, and Box-2 has 4 processes. 
> 
>   
> 
> b) Each process waits for a message (sizes vary between 1 KBytes to 26
> KBytes), 
> 
> then process the data, and outputs a message (sizes vary between 1
> KBytes to 26 KBytes), to next process. 
> 
> There is an input pkt every 80 micro-sec into this pipeline, and
> typically a job will entails about 500,000 pkts. 
> 
>   
> 
> c) MPI transport functions used : "MPI_Isend", MPI_Irecv, MPI_Test. 
> 
>    i)  For Receiving messages, I first make an MPI_Irecv call,
> followed by a busy-loop on MPI_Test, waiting for message 
> 
>    ii) For Sending message, there is a busy-loop on MPI_Test to ensure
> prior buffer was sent, then use MPI_Isend. 
> 
>   
> 
> d) When the job starts, all these 11 process are put in High priority
> mode ( SCHED_FIFO policy, with priority setting of 99). 
> 
> The Job entails an input data packet stream (and a series of MPI
> messages), continually at 40 micro-sec rate, for a few minutes.    
> 
>   
> 
> 3) The Problem: 
> 
> a) Once the job starts, about 10% of MPI data xfer occurring between
> the two boxes have a large jitter, ranging from 8 ~ 30 millisec. 
> 
> This causes some of my internal application input queues to fill-up
> and cause a failure. 
> 
>   
> 
> b) I have used a few basic tools to look at CPU usage during job
> running, nothing significant is running other than my app. 
> 
>   
> 
> - Any suggestions on improved MPI config settings or OS config/issues
> will be much appreciated. 
> 
>   
> 
> - Is there any performance monitoring tool that can monitor CPU
> activity for a few seconds, at 1 milli-sec level resolution ? 
> 
>   
> 
> Thanks in advance.SashiBala
> 
> 
-- 
Dan Kokron
Global Modeling and Assimilation Office
NASA Goddard Space Flight Center
Greenbelt, MD 20771
Daniel.S.Kokron at nasa.gov
Phone: (301) 614-5192
Fax:   (301) 614-5304