postfix-logwatch [options] [logfile ...]
The postfix-logwatch(1) utility is a Postfix MTA log parser that produces summaries, details, and statistics regarding the operation of Postfix.
This utility can be used as a standalone program, or as a Logwatch filter module to produce Postfix summary and detailed reports from within Logwatch.
Postfix-logwatch is able to produce a wide range of reports with data grouped and sorted as much as possible to reduce noise and highlight patterns. Brief summary reports provide a quick overview of general Postfix operations and message delivery, calling out warnings that may require attention. Detailed reports provide easy to scan, hierarchically-arranged and organized information, with as much or little detail as desired.
Postfix-logwatch outputs two principal sections: a Summary section and a Detailed section. For readability and quick scanning, all event or hit counts appear in the left column, followed by brief description of the event type, and finally additional statistics or count representations may appear in the rightmost column.
The following segment from a sample Summary report illustrates:
****** Summary ******************************************** 81 *Warning: Connection rate limit reached (anvil) 146 Warned 68.310M Bytes accepted 71,628,177 97.645M Bytes delivered 102,388,245 ======== ================================================ 3464 Accepted 41.44% 4895 Rejected 58.56% -------- ------------------------------------------------ 8359 Total 100.00% ======== ================================================
The report warns that anvil's connection rate was hit 81 times, a Postfix access check WARN action was logged 146 times, and a total of 68.310 megabytes (71,628,177 bytes) were accepted into the Postfix system, delivering 97.645 megabytes of data (due to multiple recipients). The Accepted and Rejected lines show that Postfix accepted 3464 (41.44% of the total messages) and rejected 4895 (the remaining 58.56%) of the 8359 total messages (temporary rejects show up elsewhere).
There are dozens of sub-sections available in the Detailed report, each of whose output can be controlled in various ways. Each sub-section attempts to group and present the most meaningful data at superior levels, while pushing less useful or noisy data towards inferior levels. The goal is to provide as much benefit as possible from smart grouping of data, to allow faster report scanning, pattern identification, and problem solving. Data is always sorted in descending order by count, and then numerically by IP address or alphabetically as appropriate.
The following MX errors segment from a sample Detailed report illustrates the basic hierarchical level structure of postfix-logwatch:
****** Detailed ******************************************* 261 MX errors -------------------------------------- 261 Unable to look up MX host 222 Host not found 73 foolishspammer.local 60 completely.bogus.domain.example 11 friend.example.com 39 No address associated with hostname 23 dummymx.sample.net 16 pushn.spam.sample.com
The postfix-logwatch utility reads from STDIN or from the named Postfix logfile. Multiple logfile arguments may be specified, each processed in order. The user running postfix-logwatch must have read permission on each named log file.
The options listed below affect the operation of postfix-logwatch. Options specified later on the command line override earlier ones. Any option may be abbreviated to an unambiguous length.
- -f config_file
- --config_file config_file
Use an alternate configuration file config_file instead of the default. This option may be used more than once. Multiple configuration files will be processed in the order presented on the command line. See Configuration File below.
- --debug keywords
Output debug information during the operation of postfix-logwatch. The parameter keywords is one or more comma or space separated keywords. To obtain the list of valid keywords, use --debug xxx where xxx is any invalid keyword.
Enables (disables) output of the message delays percentiles report. The delays percentiles report shows percentiles for each of the 4 delivery latency times reported by Postfix (available in version 2.3 and later) in the form delays=a/b/c/d, where a is the amount of time before the active queue (includes time for previous delivery attempts and time in the deferred queue), b is the amount of time in the active queue up to delivery agent handoff, c is the amount of time spent making connections (including DNS, HELO and TLS) and d is the amount of time spent delivering the message. The total delay shown comes from the delay= field in a message delivery log line.
Note: This report may consume a large amount of memory; if you have no use for it, disable the delays report.
- --delays_percentiles p1 [p2 ...]
Specifies the percentiles to be used in the message delays percentiles report. The percentiles p1, p2, ... range from 0 to 100, inclusively. The order of the list is not sorted - the report will output the percentiles columns in the order you specify.
- --detail level
Sets the maximum detail level for postfix-logwatch to level. This option is global, overriding any other output limiters described below.
The postfix-logwatch utility produces a Summary section, a Detailed section, and additional report sections. With level less than 5, postfix-logwatch will produce only the Summary section. At level 5 and above, the Detailed section, and any additional report sections are candidates for output. Each incremental increase in level generates one additional hierarchical sub-level of output in the Detailed section of the report. At level 10, all levels are output. Lines that exceed the maximum report width (specified with max_report_width) will be cut. Setting level to 11 will prevent lines in the report from being cut (see also --line_style).
Print usage information and a brief description about command line options.
- --ignore_service pattern
Ignore log lines that contain the postfix service name postfix/service. The parameter service is a regular expression.
Note: if you use parenthesis in your regular expression, be sure they are cloistering and not capturing: use (?:pattern) instead of (pattern).
- --ipaddr_width width
Specifies that IP addresses in address/hostname pairs should be printed with a field width of width characters. Increasing the default may be useful for systems using long IPv6 addresses.
- -l limiter=levelspec
- --limit limiter=levelspec
Sets the level limiter limiter with the specification levelspec.
- --line_style style
Specifies how to handle long report lines. Three styles are available: full, truncate, and wrap. Setting style to full will prevent cutting lines to max_report_width; this is what occurs when detail is 11 or higher. When style is truncate (the default), long lines will be truncated according to max_report_width. Setting style to wrap will wrap lines longer than max_report_width such that left column hit counts are not obscured. This option takes precedence over the line style implied by the detail level. The options --full, --truncate, and --wrap are synonyms.
Enables (disables) interpretation of long queue IDs in Postfix (>= 2.9) logs.
Disables the Detailed section of the report, and all supplemental reports. This option provides a convenient mechanism to quickly disable all sections under the Detailed report, where subsequent command line options may re-enable one or more sections to create specific reports.
Enables (disables) displaying of the the Summary section of the report. The variable postfix_Show_Summary in used in a configuration file.
- --recipient_delimiter delimiter
Split email delivery addresses using the recipient delimiter character delimiter. This should generally match the recipient_delimiter specified in the Postfix parameter file main.cf, or the default value indicated in postconf -d recipient_delimiter. This is very useful for obtaining per-alias statistics when a recipient delimiter is used for mail delivery.
- --reject_reply_patterns r1 [r2 ...]
Specifies the list of reject reply patterns used to create reject groups. Each entry in the list r1 [r2 ...] must be either a three character regular expression reply code of the form [0-9.][0-9.], or the word "Warn". The "." in the regular expression is a literal dot which matches any reject reply subcode; this wildcarding allows creation of broad rejects groups. List order is preserved, in that reject reports will be output in the same order as the entries in the list. Specific reject reply codes will take priority over wildcard patterns, regardless of the list order.
The default list is "5.. 4.. Warn", which creates three groups of rejects: permanent rejects, temporary reject failures, and reject warnings (as in warn_if_reject).
This feature allows, for example, distinguishing 421 transmission channel closures from 45x errors (eg. 450 mailbox unavailable, 451 local processing errors, 452 insufficient storage). Such a grouping would be configured with the list: "421 4.. 5.. Warn". See RFC 2821 for more information about reply codes.
See also Configuration File regarding using reject_reply_patterns within a configuration file.
- --show_sect_vars boolean
Enables (disables) supplementing each Detailed section title with the name of that section's level limiter. The name displayed is the command line option (or configuration file variable) used to limit that section's output. With the large number of level limiters available in postfix-logwatch, this a convenient mechanism for determining exactly which level limiter affects a section.
- --syslog_name namepat
Specifies the syslog service name that postfix-logwatch uses to match syslog lines. Only log lines whose service name matches the perl regular expression namepat will be used by postfix-logwatch; all non-matching lines are silently ignored. This is useful when a pre-installed Postfix package uses a name other than the default (postfix), or when multiple Postfix instances are in use and per-instance reporting is desired.
The pattern namepat should match the syslog_name configuration parameter specified in the Postfix parameter file main.cf, the master control file master.cf, or the default value as indicated by the output of postconf -d syslog_name.
Note: if you use parenthesis in your regular expression, be sure they are cloistering and not capturing: use (?:pattern) instead of (pattern).
- --show_unknown boolean
Enables (disables) display of the postfix-generated name of 'unknown' in formated IP/hostname pairs in Detailed reports. Default: enabled.
Print postfix-logwatch version information.
The output of every section in the Detailed report is controlled by a level limiter. The name of the level limiter variable will be output when the sect_vars option is set. Level limiters are set either via command line in standalone mode with --limit limiter=levelspec option, or via configuration file variable $postfix_limiter=levelspec. Each limiter requires a levelspec argument, which is described below in Level Control.
The list of level limiters is shown below.
There are several level limiters that control reject sub-sections (eg. rejectbody, rejectsender, etc.). Because the list of reject variants is not known until runtime after reject_reply_patterns is seen, these reject limiters are shown below generically, with the prefix ###. To use one of these reject limiters, substitute ### with one of the reject reply codes in effect, replacing each dot with an x character. For example, using the default reject_reply_patterns list of "5.. 4.. Warn", three rejectbody variants are valid: --limit 5xxrejectbody, --limit 4xxrejectbody and --limit warnrejectbody. As a convenience, you may entirely eliminate the ### prefix, and instead use the bare rejectXXX option, and all reject level limiter variations will be auto-generated based on the reject_reply_patterns list. For example, the command line segment:
... --reject_reply_patterns "421 5.." \ --limit rejectrbl="1:10:"
would automatically become:
... --reject_reply_patterns "421 5.." \ --limit 421rejectrbl="1:10:" --limit 5xxrejectrbl="1:10:"
See reject_reply_patterns above, and comments in the configuration file postfix-logwatch.conf.
[ THIS SECTION IS NOT YET COMPLETE ]
Errors obtaining attribute data from service.
Messages that triggered access, header_checks or body_checks BCC action. (postfix 2.6 experimental branch)
Local and remote bounces. A bounce is considered a local bounce if the relay was one of none, local, virtual, avcheck, maildrop or 127.0.0.1.
Regrouping by client host IP address of all 5xx (permanent) reject variants.
Postfix errors talking to one of its services.
Anvil rate or concurrency limits.
Connections made to the smtpd server.
Connections lost to the smtpd server.
Connections lost during smtp communications with remote MTA.
Failures reported by smtp when connecting to remote MTA.
Warnings noted when binary database map file requires postmap update from newer source file.
Message delivery deferrals. A single deferred message will have one or more deferrals many times.
Address verification indicates recipient address is deliverable.
Number of messages handed-off to a delivery agent such as local or virtual.
Messages that triggered access, header_checks or body_checks DISCARD action.
Any one of several errors encountered during DNS lookups.
List of sending domains. (2 levels: envelope sender domain, localpart)
List of envelope senders. (1 level: envelope sender)
Postfix general error messages.
Fatal main.cf or master.cf configuration errors.
Postfix general fatal messages.
Messages that triggered access, header_checks or body_checks FILTER action.
Messages forwarded by MDA for one address class to another (eg. local -> virtual).
Messages that were placed on hold by postsuper, or triggered by access, header_checks or body_checks HOLD action.
Invalid hostname detected.
Lookup of hostname does not map back to the IP of the peer (ie. the remote system connecting to smtpd). Also known as forward-confirmed reverse DNS (FCRDNS). When the reverse name has no DNS entry, the message "host not found, try again" is included; otherwise, it is not (e.g. when the reverse has some IP address, but not the one Postfix expects).
Illegal syntax in an email address provided during the MAIL FROM or RCPT TO dialog.
Any LDAP errors during LDAP lookup.
An MX lookup for the best mailer to use to deliver mail would result in a sending to ourselves.
Problem with an access table map that needs correcting.
Postfix encountered an error when trying to create a message file somewhere in the spool directory.
A hostname was found that was numeric, instead of alphabetic.
Postfix general panic messages.
Workarounds were enabled to avoid remote Cisco PIX SMTP "fixups".
Summarization of policyweight/policydweight results.
Summarization of PolicySPF results.
Summarization of Postgrey results.
Summarization of 2.7's postscreen and verify services.
Summarization of 2.7's dnsblog service.
Messages that triggered header_checks or body_checks PREPEND action.
Postfix services that exited unexpectedly.
A Postfix service has reached or exceeded the maximum number of processes allowed.
Problems writing a Postfix queue file.
Lookup errors for RBLs.
Messages that triggered access, header_checks or body_checks REDIRECT action.
Messages that triggered body_checks REJECT action.
Messages rejected by client access controls (smtpd_client_restrictions).
Message rejected due to server configuration errors.
Messages rejected by message_reject_characters.
Messages rejected at DATA stage in SMTP conversation (smtpd_data_restrictions).
Messages rejected at ETRN stage in SMTP conversation (smtpd_etrn_restrictions).
Messages that triggered header_checks REJECT action.
Messages rejected at HELO/EHLO stage in SMTP conversation (smtpd_helo_restrictions).
Messages rejected due to insufficient storage space.
Messages rejected due to temporary DNS lookup failures.
Milter rejects. No reject reply code is available for these rejects, but an extended 5.7.1 DSN is provided. These rejects are forced into the generic 5xx rejects group. If you redefine reject_reply_patterns such that it does not contain the pattern 5.., milter rejects will not be output.
Messages rejected by an RBL hit.
Messages rejected by recipient access controls (smtpd_recipient_restrictions).
Messages rejected by relay access controls.
Messages rejected by sender access controls (smtpd_sender_restrictions).
Messages rejected due to excessive message size.
Messages rejected by unknown client access controls.
Messages rejected by unknown reverse client access controls.
Messages rejected by unknown user access controls.
Messages rejected by unverified client access controls.
Messages rejected dueo to address verification failures.
Messages that triggered header_checks or body_checks REPLACE action.
Messages returned to sender due to exceeding queue lifetime (maximal_queue_lifetime).
SASL authentication successes, includes SASL method, username, and sender when present.
SASL authentication failures.
Messages sent via the SMTP delivery agent.
Messages sent via the LMTP delivery agent.
Errors during the SMTP/ESMTP dialog.
Protocol violation during the SMTP/ESMTP dialog.
Errors during Postfix server startup.
Connections to smtpd that timed out.
TLS client connections.
TLS communication offered.
TLS server connections.
Unverified TLS connections.
Address verification indicates recipient address is undeliverable.
Messages that triggered access, header_checks or body_checks WARN action.
Warnings regarding Postfix configuration errors.
Postfix general warning messages.
The Detailed section of the report consists of a number of sub-sections, each of which is controlled both globally and independently. Two settings influence the output provided in the Detailed report: a global detail level (specified with --detail) which has final (big hammer) output-limiting control over the Detailed section, and sub-section specific detail settings (small hammer), which allow further limiting of the output for a sub-section. Each sub-section may be limited to a specific depth level, and each sub-level may be limited with top N or threshold limits. The levelspec argument to each of the level limiters listed above is used to accomplish this.
It is probably best to continue explanation of sub-level limiting with the following well-known outline-style hierarchy, and some basic examples:
level 0 level 1 level 2 level 3 level 4 level 4 level 2 level 3 level 4 level 4 level 4 level 3 level 4 level 3 level 1 level 2 level 3 level 4
The simplest form of output limiting suppresses all output below a specified level. For example, a levelspec set to "2" shows only data in levels 0 through 2. Think of this as collapsing each sub-level 2 item, thus hiding all inferior levels (3, 4, ...), to yield:
level 0 level 1 level 2 level 2 level 1 level 2
Sometimes the volume of output in a section is too great, and it is useful to suppress any data that does not exceed a certain threshold value. Consider a dictionary spam attack, which produces very lengthy lists of hit-once recipient email or IP addresses. Each sub-level in the hierarchy can be threshold-limited by setting the levelspec appropriately. Setting levelspec to the value "2::5" will suppress any data at level 2 that does not exceed a hit count of 5.
Perhaps producing a top N list, such as top 10 senders, is desired. A levelspec of "3:10:" limits level 3 data to only the top 10 hits.
With those simple examples out of the way, a levelspec is defined as a whitespace- or comma-separated list of one or more of the following:
Specifies the maximum level to be output for this sub-section, with a range from 0 to 10. if l is 0, no levels will be output, effectively disabling the sub-section (level 0 data is already provided in the Summary report, so level 1 is considered the first useful level in the Detailed report). Higher values will produce output up to and including the specified level.
Same as above, with the addition that n limits this section's level 1 output to the top n items. The value for n can be any integer greater than 1. (This form of limiting has less utility than the syntax shown below. It is provided for backwards compatibility; users are encouraged to use the syntax below).
This triplet specifies level l, top n, and minimum threshold t. Each of the values are integers, with l being the level limiter as described above, n being a top n limiter for the level l, and t being the threshold limiter for level l. When both n and t are specified, n has priority, allowing top n lists (regardless of threshold value). If the value of l is omitted, the specified values for n and/or t are used for all levels available in the sub-section. This permits a simple form of wildcarding (eg. place minimum threshold limits on all levels). However, specific limiters always override wildcard limiters. The first form of level limiter may be included in levelspec to restrict output, regardless of how many triplets are present.
All three forms of limiters are effective only when postfix-logwatch's detail level is 5 or greater (the Detailed section is not activated until detail is at least 5).
See the Examples section for usage scenarios.
Postfix-logwatch can read configuration settings from a configuration file. Essentially, any command line option can be placed into a configuration file, and these settings are read upon startup.
Because postfix-logwatch can run either standalone or within Logwatch, to minimize confusion, postfix-logwatch inherits Logwatch's configuration file syntax requirements and conventions. These are:
- White space lines are ignored.
- Lines beginning with # are ignored
Settings are of the form:
option = value
- Spaces or tabs on either side of the = character are ignored.
- Any value protected in double quotes will be case-preserved.
- All other content is reduced to lowercase (non-preserving, case insensitive).
- All postfix-logwatch configuration settings must be prefixed with "$postfix_" or postfix-logwatch will ignore them.
- When running under Logwatch, any values not prefixed with "$postfix_" are consumed by Logwatch; it only passes to postfix-logwatch (via environment variable) settings it considers valid.
- The values True and Yes are converted to 1, and False and No are converted to 0.
- Order of settings is not preserved within a configuration file (since settings are passed by Logwatch via environment variables, which have no defined order).
To include a command line option in a configuration file, prefix the command line option name with the word "$postfix_". The following configuration file setting and command line option are equivalent:
$postfix_Line_Style = Truncate --line_style Truncate
Level limiters are also prefixed with $postfix_, but on the command line are specified with the --limit option:
$postfix_Sent = 2 --limit Sent=2
The order of command line options and configuration file processing occurs as follows: 1) The default configuration file is read if it exists and no --config_file was specified on a command line. 2) Configuration files are read and processed in the order found on the command line. 3) Command line options override any options already set either via command line or from any configuration file.
Command line options are interpreted when they are seen on the command line, and later options will override previously set options. The notable exception is with limiter variables, which are interpreted in the order found, but only after all other options have been processed. This allows --reject_reply_patterns to determine the dynamic list of the various reject limiters.
See also --reject_reply_patterns.
The postfix-logwatch utility exits with a status code of 0, unless an error occurred, in which case a non-zero exit status is returned.
Note: postfix-logwatch reads its log data from one or more named Postfix log files, or from STDIN. For brevity, where required, the examples below use the word file as the command line argument meaning /path/to/postfix.log. Obviously you will need to substitute file with the appropriate path.
To run postfix-logwatch in standalone mode, simply run: postfix-logwatch file
A complete list of options and basic usage is available via: postfix-logwatch --help
To print a summary only report of Postfix log data: postfix-logwatch --detail 1 file
To produce a summary report and a one-level detail report for May 25th:
grep 'May 25' file | postfix-logwatch --detail 5
To produce only a top 10 list of Sent email domains, the summary report and detailed reports are first disabled. Since commands line options are read and enabled left-to-right, the Sent section is re-enabled to level 1 with a level 1 top 10 limiter:
postfix-logwatch --nosummary --nodetail --limit sent='1 1:10:' file
The following command and its sample output shows a more complex level limiter example. The command gives the top 3 Sent email addresses from the top 5 domains, in addition, all level 3 items with a hit count of 2 or less are suppressed (in the Sent sub-section, this happens to be email's Original To address). Ellipses indicate top N or threshold-limited data:
postfix-logwatch --nosummary --nodetail \ --limit sent '1:5: 2:3: 3::2' file 1762 Sent via SMTP ----------------------------------- 352 example.com 310 joe 255 email@example.com 7 firstname.lastname@example.org 21 pooryoda3 11 hot93uh ... 244 sample.net 97 buzz 26 leroyjones 14 sally ... 152 example.net 40 jim_jameson 23 sam_sampson 19 paul_paulson ... 83 sample.us 44 root 39 jenny1 69 dom3.example.us 10 kay 7 ron 6 mrsmith ... ...
The next command uses both reject_reply_patterns and level limiters to see 421 RBL rejects, threshold-limiting level 2 output to hits greater than 5 (level 2 in the Reject RBL sub-section is the client's IP address / hostname pair). This makes for a very nice RBL offenders list, shown in the sample output (note the use of the unambiguous, abbreviated command line option reject_reply_pat):
postfix-logwatch --reject_reply_pat '421 4.. 5.. Warn' \ --nosummary --nodetail --limit 421rejectrbl='2 2::5' file 300 421 Reject RBL --------------------------------------- 243 zen.spamhaus.org=127.0.0.2 106 10.0.0.129 129.0.0.example.com 41 192.168.10.70 hostx10.sample.net 40 192.168.42.39 hostz42.sample.net 15 10.1.1.152 dsl-10-1-1-152.example.us 14 10.10.10.122 mail122.sample.com 7 192.168.3.44 smalltime-spammer.example.com ... 48 zen.spamhaus.org=127.0.0.4 17 10.29.124.92 10-29-124-92.adsl-static.sample.us ... 8 zen.spamhaus.org=127.0.0.11 ... 1 zen.spamhaus.org=127.0.0.10 ...
Running within Logwatch
Note: Logwatch versions prior to 7.3.6, unless configured otherwise, required the --print option to print to STDOUT instead of sending reports via email. Since version 7.3.6, STDOUT is the default output destination, and the --print option has been replaced by --output stdout. Check your configuration to determine where report output will be directed, and add the appropriate option to the commands below.
To print a summary report for today's Postfix log data:
logwatch --service postfix --range today --detail 1 To print a report for today's Postfix log data, with one level of detail in the Detailed section: logwatch --service postfix --range today --detail 5
To print a report for yesterday, with two levels of detail in the Detailed section:
logwatch --service postfix --range yesterday --detail 6
To print a report from Dec 12th through Dec 14th, with four levels of detail in the Detailed section:
logwatch --service postfix --range \ 'between 12/12 and 12/14' --detail 8 To print a report for today, with all levels of detail: logwatch --service postfix --range today --detail 10
Same as above, but leaves long lines uncut:
logwatch --service postfix --range today --detail 11
The postfix-logwatch program uses the following (automatically set) environment variables when running under Logwatch:
This is the detail level specified with the Logwatch command line argument --detail or the Detail setting in the ...conf/services/postfix.conf configuration file.
This is the debug level specified with the Logwatch command line argument --debug.
The Logwatch program passes all settings postfix_xxx in the configuration file ...conf/services/postfix.conf to the postfix filter (which is actually named .../scripts/services/postfix) via environment variable.
The postfix-logwatch program
The postfix-logwatch configuration file in standalone mode
The Logwatch postfix filter
The Logwatch postfix filter configuration file
logwatch(8), system log analyzer and reporter
README, an overview of postfix-logwatch Changes, the version change list history Bugs, a list of the current bugs or other inadequacies Makefile, the rudimentary installer LICENSE, the usage and redistribution licensing terms
Covered under the included MIT/X-Consortium License: http://www.opensource.org/licenses/mit-license.php
The original postfix Logwatch filter was written by Kenneth Porter, and has had many contributors over the years. They are entirely not responsible for any errors, problems or failures since the current author's hands have touched the source code.