1. What is MPI?
MPI (Message Passing Interface) is a standardized, portable library for writing parallel programs in distributed computing environments. It enables processes to communicate by sending and receiving messages, making it ideal for high-performance computing (HPC) on clusters or supercomputers. Key features include:
- Distributed Memory: Each process has its own memory space.
- SPMD Model: Single Program, Multiple Data (all processes run the same code but operate on different data).
- Portability: Works across diverse hardware systems.
2. Key Concepts
- Communicator: A group of processes that can communicate. The default is
MPI_COMM_WORLD. - Rank: A unique identifier (integer) for each process in a communicator (e.g., rank 0, 1, 2…).
- Point-to-Point Communication: Direct message exchange between two processes.
- Collective Communication: Operations involving all processes in a communicator (e.g., broadcast, reduce).
3. Basic MPI Functions
Initialization and Finalization
1
2
| MPI_Init(&argc, &argv); // Initialize MPI
MPI_Finalize(); // Shut down MPI
|
1
2
3
| int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank); // Get current process rank
MPI_Comm_size(MPI_COMM_WORLD, &size); // Get total number of processes
|
4. Hello World Example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| #include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
MPI_Init(&argc, &argv);
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
printf("Hello from process %d of %d\n", rank, size);
MPI_Finalize();
return 0;
}
|
Compile and Run
1
2
| mpicc -o hello hello.c # Compile with MPI wrapper
mpirun -np 4 ./hello # Run with 4 processes
|
Output (order may vary):
1
2
3
4
| Hello from process 0 of 4
Hello from process 1 of 4
Hello from process 3 of 4
Hello from process 2 of 4
|
5. Point-to-Point Communication
Blocking Send/Receive
1
2
3
4
5
6
7
8
| if (rank == 0) {
int data = 42;
MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD); // Send to rank 1
} else if (rank == 1) {
int data;
MPI_Recv(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); // Receive from rank 0
printf("Received %d\n", data);
}
|
Key Arguments:
MPI_Send(void* data, int count, MPI_Datatype, int dest, int tag, MPI_Comm).MPI_Recv(void* buffer, int count, MPI_Datatype, int source, int tag, MPI_Comm, MPI_Status*).
6. Collective Communication
Broadcast Example
1
2
3
4
5
6
| int data;
if (rank == 0) {
data = 100; // Root process sets the value
}
MPI_Bcast(&data, 1, MPI_INT, 0, MPI_COMM_WORLD); // Broadcast to all
printf("Process %d received %d\n", rank, data);
|
Output:
1
2
3
| Process 0 received 100
Process 1 received 100
...
|
Other Collective Operations:
- Scatter/Gather: Distribute/collect data across processes.
- Reduce: Combine data from all processes (e.g., sum, max).
7. Common Pitfalls
- Deadlocks: Caused by mismatched
MPI_Send/MPI_Recv (e.g., two processes waiting to receive first). - Mismatched Tags: Ensure tags match between sender and receiver.
- Data Type Errors: Use consistent
MPI_Datatype (e.g., MPI_INT for integers). - Forgetting MPI_Init/Finalize: Always initialize and shut down MPI properly.
8. When to Use MPI?
- Distributed Systems: When working across multiple nodes (not just multi-core CPUs).
- Large-Scale Problems: For tasks requiring massive parallelism (e.g., fluid dynamics, climate modeling).
9. Further Resources
- Books: Using MPI by Gropp et al.
- Documentation: MPI Forum
- Tools: Debug with
mpirun -np 4 xterm -e gdb ./your_program.
By mastering MPI, you can efficiently parallelize complex computations across thousands of processors! Start small, test often, and explore advanced features like non-blocking communication as you progress.