scout - Man Page

Scalasca parallel trace analyzer

Synopsis

scout.ser [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
scout.omp [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
scout.mpi [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
scout.hyb [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)

Description

scout is the scalable automatic trace-analysis component of the Scalasca Trace Tools. In particular, it provides the ability to

Usually, scout is launched automatically by the Scalasca measurement collection and analysis nexus scan(1) after a successful measurement if event tracing is configured. However, it can also be run manually on an existing event trace measurement.

scout currently supports trace experiments in two different event trace formats: OTF2 traces generated by the Score-P instrumentation and measurement system, and legacy traces in EPILOG format generated by the measurement system of the Scalasca 1.x release series. For OTF2 event traces, scout has to be provided with the corresponding ANCHOR_FILE (e.g., 'traces.otf2'), for EPILOG traces with the experiment directory name EXPERIMENT_DIR.

Depending on the build configuration and the capabilities of the target platform, the scout analyzer may be available in up to four forms:

scout.ser

is always built. It is used to analyze event traces generated by serial applications. It can also be used to analyze event traces from multi-threaded applications, however, it will then only provide information about the master thread.

scout.omp

is built whenever the Scalasca Trace Tools are configured with OpenMP support. It is used to analyze event traces generated by pure multi-threaded applications (e.g., using OpenMP or POSIX threads). It can also be used to analyze event traces from serial applications, though analysis incurs a higher overhead than using scout.ser.

scout.mpi

is built whenever the Scalasca Trace Tools are configured with MPI support. It is used to analyze event traces generated by pure MPI applications. It can also be used on traces from multi-threaded MPI applications, however, it will then only provide information about the master thread of each process and its MPI activities.

scout.hyb

is built if the Scalasca Trace Tools are configured with both MPI and OpenMP support. It is used to analyze event traces generated by multi-threaded MPI applications (e.g., MPI+OpenMP or MPI+Pthreads), providing information about all OpenMP/POSIX threads of each MPI process. It can also be used on traces from pure MPI applications, though analysis incurs a slightly higher overhead than using scout.mpi.  

Note that scout.mpi and scout.hyb are implemented as MPI programs, and therefore have to be executed using appropriate MPI launch commands and flags. Also, the number of MPI processes for scout must be identical to the number of MPI processes used for the target application execution.

If successful, scout produces the following output files in the measurement archive directory:

Options

scout accepts a number of command-line options to enable/disable particular analysis features. When scout is launched automatically from the Scalasca measurement collection and analysis nexus scan(1), these options can be passed to the analyzer via the SCAN_ANALYZE_OPTS environment variable.

Common options

--statistics

Enables most-severe instance tracking and wait-state statistics. This is the default.

--no-statistics

Disables most-severe instance tracking and wait-state statistics.

--critical-path

Enables critical-path analysis. This is the default.

--no-critical-path

Disables critical-path analysis.

--rootcause

Enables root-cause analysis. This is the default.

--no-rootcause

Disables root-cause analysis.

--single-pass

Use single-pass forward analysis only. Disables both critical-path and root-cause analysis, as well as the detection of Late Receiver wait states.

-v,  --verbose

Increase verbosity.

--help

Print a brief usage summary, then exit.  

MPI options (scout.mpi/scout.hyb only)

--time-correct

Enables enhanced timestamp correction. Event traces collected on clusters without a synchronized clock may contain logical clock condition violations (such as a receive completing before the corresponding send is initiated). When scout detects such situations, it issues a warning that the analysis may be inconsistent and recommends (re-)running trace analysis with its integrated timestamp correction algorithm activated.

--no-time-correct

Disables enhanced timestamp correction. This is the default.  

Exit Status

scout exits with status 0 if automatic trace analysis was successful, and greater than 0 if errors occur.

Notes

scout poses a number of requirements on the input event trace data, which are documented in the OPEN_ISSUES file installed as part of the Scalasca Trace Tools documentation. It is also available online at https://www.scalasca.org. If those requirements are not met, scout may abort, deadlock, or crash.

If scout crashes or deadlocks even though the documented requirements are met (which usually indicates a bug), restricting the scope of the analysis by disabling certain features (e.g., critical-path and/or root-cause analysis) may help as a workaround. In any case, please report such issues for further investigation (see Bugs below).

Bugs

Please report bugs to scalasca@fz-juelich.de. Make sure to include at least the following information in your bug report:

Also, please archive a copy of the entire experiment archive directory including the event trace data, as this may be required to aid in debugging. However, ONLY PROVIDE TRACE DATA IF EXPLICITLY REQUESTED, as the data volume may be excessive.

Examples

scout.omp scorep_foo_Ox4_trace/traces.otf2
Perform the Scalasca OpenMP event trace analysis on the OTF2 event trace with anchor file scorep_foo_Ox4_trace/traces.otf2.

mpiexec -n 16 scout.mpi --time-correct scorep_bar_16_trace/traces.otf2
Apply the enhanced timestamp correction and perform the Scalasca MPI event trace analysis on the OTF2 event trace with anchor file scorep_bar_16_trace/traces.otf.

mpiexec -n 4 scout.hyb epik_foobar_4x4_trace
Perform the hybrid Scalasca MPI+OpenMP event trace analysis on the EPILOG event trace in the experiment archive epik_foobar_4x4_trace generated by the Scalasca 1.x release series.

See Also

scalasca(1), scan(1), square(1)

The full Scalasca Trace Tools documentation is available online at https://www.scalasca.org.

Referenced By

The man pages scout.hyb(1), scout.mpi(1), scout.omp(1) and scout.ser(1) are aliases of scout(1).

Fri Mar 22 2019 Version 2.5 Scalasca Trace Tools