Record roll in MPI application

Question

I would like to collect statistics from an MPI application

To save the application running on a single machine, I can use the command:

perf record mpirun -np $NUMBER_OF_CORES app_name

However, when executing this command to distribute the task on more than one machine, the recorded events are only from the local machine - which I triggered the execution

To register each of the processes, I created a script that calls the perf in each of the sub-processes

#!/bin/bash
app=$1
perf record -m 512G -o "${app}-${RANDOM}.perf.data ${app}

And I do it as follows:

mpirun -np $NUMBER_OF_CORES perf_record.sh app_name

However, the generated files did not record any event

The app_name-123.perf.data file has no samples!

1 - What I’m doing wrong?

2 - Is there any other tool that is more suitable to use with MPI applications?

Obs, I’m really interested in the time spent on each of the program functions and not the time of communication

Answer 1

Actually the problem was in -m 512G, passing a high value like this, no event was recorded, as described here.

That one flag had been added to get around the problem

Permission error mapping pages.
Consider increasing /proc/sys/kernel/perf_event_mlock_kb,
or try again with a smaller value of mmap_pages.

But actually the correct solution would be to add write permissions to the user, as described here. This could be done through the commands:

Temporally:

sudo sh -c 'echo -1 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 > /proc/sys/kernel/kptr_restrict'

Persistently:

sudo sh -c 'echo kernel.perf_event_paranoid=-1 >> /etc/sysctl.d/local.conf'
sudo sh -c 'echo kernel.kptr_restrict=0 >> /etc/sysctl.d/local.conf'