FAQ:
Tuning the run-time characteristics of MPI TCP communications

In general, you specify that the tcp BTL component should be used. However, note that you should also specify that the self BTL component should be used. self is for loopback communication (i.e., when an MPI process sends to itself), and is technically a different communication channel than TCP. For example:

shell$ mpirun --mca btl tcp,self ...

Failure to specify the self BTL may result in Open MPI being unable to complete send-to-self scenarios (meaning that your program will run fine until a process tries to send to itself).

Note that if the tcp BTL is available at run time (which it should be on most POSIX-like systems), Open MPI should automatically use it by default (ditto for self). Hence, it's usually unnecessary to specify these options on the mpirun command line. They are typically only used when you want to be absolutely positively definitely sure to use the specific BTL.

No. Following the so-called "Law of Least Astonishment", Open MPI assumes that if you have both a TCP network and at least one high-speed network (such as Myrinet or InfiniBand), you will likely only want to use the high-speed network(s) for MPI message passing. Hence, the tcp BTL component will sense this and automatically deactivate itself.

That being said, Open MPI may still use TCP for setup and teardown information -- so you'll see traffic across your TCP network during startup and shutdwon of your MPI job. This is normal and does not affect the MPI message passing channels.

The ompi_info command can display all the parameters available for the tcp BTL component (i.e., the component that uses TCP for MPI communications):

shell$ ompi_info --param btl tcp

Usually not.

In general message passing usage, there are two scenarios where using the TCP loopback interface could be used:

1. Sending a message from one process to itself
2. Sending a message from one process to another process on the same machine

The TCP BTL does not handle "send-to-self" scenarios in Open MPI; indeed, it is not even capable of doing so. Instead, the self BTL component is used for all send-to-self MPI communications (this allows all Open MPI BTL components to avoid special case code for send-to-self scenarios). The self component uses its own mechanisms for send-to-self scenarios; it does not use network interfaces.

When sending to processes on the same machine, Open MPI will default to using the shared memory (sm) BTL. If the user has deactivated this BTL, depending on what other BTL components are available, it is possible that the TCP BTL will be chosen for message passing to processes on the same node, in which case the TCP lookback device will likely be used. But this is not the default; either shared memory has to fail to startup properly or the user must specifically request not to use the shared memory BTL.

TCP-related errors are usually reported by Open MPI in a message similar to these:

btl_tcp_endpoint.c:572:mca_btl_tcp_endpoint_complete_connect] connect() failed with errno=113
mca_btl_tcp_frag_send: writev failed with errno=104 

If an error number is displayed with no explanation string, you can see what that specific error number means on your operating system with the following command (the following example was run on Linux; results may be different on other operating systems):

linux_shell$ perl -e 'die$!=113'
No route to host at -e line 1.

linux_shell$ perl -e 'die$!=104'
Connection reset by peer at -e line 1.

Two types of errors are commonly reported to the Open MPI user's mailing list:

In some parallel environments, it is not uncommon to have multiple TCP interfaces on each node -- for example, one TCP network may be "slow" and used for control information such as a batch scheduler, a networked filesystem, and/or interactive logins. Another TCP network (or networks) may be "fast" and be intended for parallel applications to use during their runs.

Although Open MPI is likely to open multiple TCP sockets during MPI_INIT, the tcp BTL component does not open one socket per MPI peer process during MPI_INIT. Open MPI opens sockets as they are required -- so the first time a process sends a message to a peer and there is a TCP connection between the two, Open MPI will automatically open a new socket.

Hence, you should not have scalability issues with running large numbers of processes (e.g., running out of per-process file descriptors) if your parallel application is sparse in its communication with peers.

This is a fairly complicated question -- there can be ambiguity when hosts have multiple TCP NICs and/or there are multiple TCP networks that are not routable to each other in a single MPI job.

It is important to note that Open MPI's atomic unit of routing is a process -- not an IP address. Hence, Open MPI makes connections between processes, not nodes (these processes are almost always on remote nodes, but it's still better to think in terms of processes, not nodes).

Specifically, since OMPI can span multiple TCP networks, each MPI process may be able to use multiple IP addresses to each to each other MPI process (and vice versa). So for each process, Open MPI needs to determine which IP address -- if any - to use to connect to a peer MPI process.

For example, say that you have a cluster with 16 nodes on a private ethernet network. One of these nodes doubles as the head node for the cluster and therefore has 2 ethernet NICs -- one to the external network and one to the internal cluster network. But since 16 is a nice number, you also want to use it for computation as well. So when you mpirun spanning all 16 nodes, OMPI has to figure out to not use the external NIC on the head node and only use the internal NIC.

The 1.3 series assumptions about routability are much different than in the 1.2 series assumption. In the 1.3 series, we assume that all interfaces are routable as long as they have the same address family, IPv4 or IPv6. We use graph theory and give each possible connection a weight depending on the quality of the connection. This allows the library to select the best connections between nodes. This method also supports striping but prevents more than one connection to any interface.

The quality of the connection is defined as follows, with a higher number meaning better connection. Note that when giving a weight to a connection consisting of a private address and a public address, it will give it the weight of PRIVATE_DIFFERENT_NETWORK.

            NO_CONNECTION = 0
PRIVATE_DIFFERENT_NETWORK = 1
PRIVATE_SAME_NETWORK      = 2
PUBLIC_DIFFERENT_NETWORK  = 3
PUBLIC_SAME_NETWORK       = 4

At this point, an example will best illustrate how two processes on two different nodes would connect up. Here we have two nodes with a variety of interfaces.

       
        NodeA                NodeB
   ---------------       ---------------
  |     lo0       |     |     lo0       |
 '|  127.0.0.1    |     |  127.0.0.1    |
  |  255.0.0.0    |     |  255.0.0.0    |
  |               |     |               |
  |     eth0      |     |    eth0       |
  |   10.8.47.1   |     |   10.8.47.2   |
  | 255.255.255.0 |     | 255.255.255.0 |
  |               |     |               |
  |     ibd0      |     |     ibd0      |
  |  192.168.1.1  |     |  192.168.1.2  |
  | 255.255.255.0 |     | 255.255.255.0 |
  |               |     |               |
  |     ibd1      |     |               |
  |  192.168.2.2  |     |               |
  | 255.255.255.0 |     |               |
   ---------------       ---------------

     NodeA         NodeB
eth0 --------- 2 -------- eth0
    \
     \
      \------- 1 -------- ibd0

ibd0 --------- 1 -------- eth0
    \
     \
      \------- 2 -------- ibd0

ibd1 --------- 1 -------- eth0
    \
     \
      \------- 1 -------- ibd0

The library then examines all the connections and picks the optimal ones. This leaves us with two connections being established between the two nodes.

If you are curious about the actual connect() calls being made by the processes, then you can run with --mca btl_base_verbose 30. This can be useful if you notice your job hanging and believe it may be the library trying to make connections to unreachable hosts.

# Here is an example with some of the output deleted for clarity.
# One can see the connections that are attempted.

> mpirun --mca btl self,sm,tcp --mca btl_base_verbose 30 -np 2 -host NodeA,NodeB a.out
[...snip...]
[NodeA:18003] btl: tcp: attempting to connect() to address 10.8.47.2 on port 59822
[NodeA:18003] btl: tcp: attempting to connect() to address 192.168.1.2 on port 59822
[NodeB:16842] btl: tcp: attempting to connect() to address 192.168.1.1 on port 44500
[...snip...]

As of v1.2, no.

Although TCP sockets are opened "lazily" (meaning that MPI connections / TCP sockets are only opened upon demand -- as opposed to opening all possible sockets between MPI peer processes during MPI_INIT), they are never closed.