PMDAOPENMETRICS(1) General Commands Manual PMDAOPENMETRICS(1)
pmdaopenmetrics - OpenMetrics PMDA
$PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics [-D] [-n] [-c config] [-d
domain] [-l logfile] [-r root] [-t timeout] [-u user]
pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA) which
creates PCP metrics from OpenMetrics endpoints, which provide HTTP
based access to application metrics. The default config directory is
$PCP_PMDAS_DIR/openmetrics/config.d/, see ``CONFIGURATION SOURCES''
below. The default URL fetch timeout is 2 seconds. The default
user, if not specified with the -u option, is the current user. If
the -n option is given, the list of configuration files will not be
sorted prior to processing. This list is sorted by default but that
can be expensive if there are a large number of configuration files
(URLs and/or scripts).
If the -D option is given, additional diagnostic messages will be
written to the PMDA log file, which is
$PCP_LOG_DIR/pmcd/openmetrics.log by default (see also -lbelow). In
addition, the metric openmetrics.control.debug controls the same
debug flag and can be set with the following command:
pmstore openmetrics.control.debug value
where value is either 1 (to enable verbose log messages) or 0 (to
disable verbose log messages). This is particularly useful for
examining the http headers passed to each fetch request, filter
settings and other processing details that are logged when the
debugging flag is enabled.
The -d option may be used to override the default performance metrics
domain number, which defaults to 144. It is strongly recommended not
to change this. The domain number should be different for every PMDA
on the one host, and the same domain number should be used for
pmdaopenmetrics PMDA on all hosts. See also the -r option, which
allows the root of the dynamic namespace to be changed from the
default openmetrics.
The -l option may be used to specify logfile as the destination for
PMDA messages instead of the default,
$PCP_LOG_DIR/pmcd/openmetrics.log. As a special case, logfile may be
"-" to send messages to the stderr stream instead, e.g. -l-. This
would normally be the stderr stream for the parent process, pmcd(1),
which may itself have redirected stderr. This redirection is
normally most useful in a containerized environment, or when using
dbpmda(1).
The -r option allows the root of the dynamic namespace to be changed
to root from the default, openmetrics. In conjunction with other
command line options, this allows pmdaopenmetrics to be deployed as a
different PMDA with distinct metrics namespace and metrics domain on
the same host system. Note that all PMDAs require a unique domain
number so the -d option must also be specified. Use of the -r option
may also change the defaults for some other command line options,
e.g. the default log file name and the default configuration
directory.
As it runs, pmdaopenmetrics periodically recursively scans the
$PCP_PMDAS_DIR/openmetrics/config.d directory (or the directory
specified with the -c option), looking for source URL files (*.url)
and executable scripts or binaries. Any files that do not have the
.url suffix or are not executable, are ignored - this allows
documentation files such as "README" and non-executable "common"
script function definitions to be present without being considered as
config files.
A remote server does not have to be up or stay running - the PMDA
tolerates remote URLs that may come and go over time. The PMDA will
relay data and metadata when/if they are available, and will return
errors when/if they are down. PCP metric IDs, internal and external
instance domain identifiers are persisted and will be restored when
individual metric sources become available and/or when the PMDA is
restarted. In addition, the PMDA checks directory modification times
and will rescan for new or changed configuration files dynamically.
It is not necessary to restart the PMDA when adding, removing or
changing configuration files.
Each file with the .url suffix found in the config directory or sub-
directory contains one complete HTTP or HTTPS URL at which
pmdaopenmetrics can reach a OpenMetrics endpoint. Local file access
is also supported with a conventional file:///somepath/somefile URL,
in which case somepath/somefile should contain openmetrics formatted
metric data.
The first line of a .url config file should be the URL, as described
above. Subsequent lines, if any, are prefixed with a keyword that
can be used to alter the http GET request. A keyword must end with
':' (colon) and the text extends to the end of the line. Comment
lines that start with # and blank lines are ignored. The only
currently supported keywords are HEADER: and FILTER:.
HEADER: headername: value ... to end of line
Adds headername and its value to the headers passed in the http GET
request for the configured URL. An example configuration file that
provides 3 commonly used headers and an authentication token might be
:
http://somehost/path/endpoint.html
# this is a comment
HEADER: Accept: text/html
HEADER: Keep-Alive: 300
HEADER: Connection: keep-alive
HEADER: Authorization: token ABCDEF1234567890
As mentioned above, header values extend to the end of the line.
They may contain any valid characters, including colons. Multiple
spaces will be collapsed to a single space, and leading and trailing
spaces are trimmed. A common use for headers is to configure a proxy
agent and the assorted parameters it may require.
FILTER: INCLUDE METRIC regex
or
FILTER: EXCLUDE METRIC regex
Dynamically created metric names that match regex will be either
included or excluded in the name space, as specified. The simple
rule is that the first matching filter regex for a particular metric
name is the rule that prevails. If no filter regex matches (or there
are no filters), then the metric is included by default, i.e. the
default filter if none are specified is FILTER: INCLUDE METRIC .*
This is backward compatible with older versions of the configuration
file that did not support filters. Multiple FILTER: lines would
normally be used, e.g. to include some metrics but exclude all
others, use FILTER: EXCLUDE METRIC .* as the last of several filters
that include the desired metrics. Conversely, to exclude some
metrics but include all others, use FILTER: EXCLUDE METRIC regex. In
this case it's not necessary (though doesn't hurt) to specify the
final FILTER: INCLUDE METRIC .* because, as stated above, any metric
that does not match any filter regex will be included by default.
Label filtering uses similar FILTER: syntax and semantics. FILTER:
EXCLUDE LABEL regex will delete all labels matching regex from all
metrics defined in the configuration file. The same rules as for
metrics apply for labels too - an implicit rule: FILTER: INCLUDE
LABEL .* applies to all labels that do not match any earlier filter
rule.
Caution is needed with label filtering because by default, all labels
are used to construct the PCP instance name. By excluding some
labels, the instance names will change. Excluding all labels for a
particular metric changes that metric to be singular, i.e. have no
instance domain. In addition, by excluding some labels, different
instances of the same metric may become duplicates. When such
duplicates occur, the last duplicate instance returned by the end-
point URL prevails over any earlier instances. For these reasons, it
is recommended that label filtering rules be configured when the
configuration file is first defined, and not changed thereafter. If
a label filtering change is required, the configuration file should
be renamed, which effectively defines a new metric, with the new (or
changed) instance naming.
Unrecognized keywords in configuration files are reported in the PMDA
log file but otherwise ignored.
Executable scripts present in the $PCP_PMDAS_DIR/openmetrics/config.d
directory or sub-directories will be executed and the stdout stream
containing openmetrics formatted metric data will be parsed as though
it had come from a URL or file. The stderr stream from a script will
be sent to the PMDA log file, which by default can be found in
$(PCP_LOG_DIR)/pmcd/openmetrics.log.
Note that scripted sources do not support label or metric filtering
(as described above for URL sources) - they can simply do their own
filtering in the script itself with sed(1), awk(1), or whatever tool
is desired.
A simple example of a scripted config entry follows:
#! /bin/sh
awk '{
print("# HELP loadavg local load average")
print("# Type loadavg gauge")
printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
}' /proc/loadavg
This script produces the following OpenMetrics-formatted metric data
when run:
# HELP loadavg local load average
# Type loadavg gauge
loadavg {interval="1-minute"} 0.12
loadavg {interval="5-minute"} 0.27
loadavg {interval="15-minute"} 0.54
If the above script was saved and made executable in a file named
$PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh then this would
result in a new PCP metric named openmetrics.local.system.loadavg
which would have three instances for the current load average values:
1-minute, 5-minute and 15-minute.
Scripted config entries may produce more than one PCP leaf metric
name. For example, the above "system.sh" script could also export
other metrics such as CPU statistics, by reading /proc/stat on the
local system. Such additional metrics would appear as peer metrics
in the same PCP metric subtree. In the case of CPU counters, the
metric type definition should be counter, not gauge. For full
details of the openmetrics exposition formats, see
https://openmetrics.io/docs/instrumenting/exposition_formats .
All metrics from a file named JOB.* will be exported as PCP metrics
with the openmetrics.JOB metric name prefix. Therefore, the JOB name
must be a valid non-leaf name for PCP PMNS metric names. If the JOB
name has multiple dot-separated components, the resulting PMNS names
will include those components and care is needed to ensure there are
no overlapping definitions, e.g. metrics returned by JOB.response may
overlap or conflict with metrics returned by JOB.response.time.
Config file entries (URLs or scripts) found in subdirectories of the
config directory will also result in hierarchical metric names. For
example, a config file named
$PCP_PMDAS_DIR/openmetrics/config.d/mysource/latency/get.url will
result in metrics being created (by fetching that source URL) below
openmetrics.mysource.latency.get in the PCP namespace. Scripts found
in subdirectories of the config directory similarly result in
hierarchical PCP metric names.
As described above, changes and new additions can be made to files in
the configuration directory without having to restart the PMDA.
These changes are detected automatically and the PCP metric names
below openmetrics in the PMNS will be updated accordingly, i.e. new
metrics will be dynamically added and/or existing metrics removed.
In addition, pmdaopenmetrics honors the PMCD_NAMES_CHANGE pmFetch(3)
protocol that was introduced in PCP version 4.0. In particular, if
openmetrics metrics are being logged by a PCP version 4.0 or later
pmlogger(1), new metrics that appear as a result of changes in the
PMDA configuration directory will automatically start to be logged,
provided the root of the openmetrics PMDA namespace is configured for
logging in the pmlogger configuration file. See pmlogger(1) for
details. An example of such a pmlogger configuration file is :
log mandatory on 2 second {
# log all metrics below the root of the openmetrics namespace
openmetrics
}
The PMDA maintains special control metrics, as described below.
Apart from openmetrics.control.debug, each of these metrics has one
instance for each configured metric source. All of these metrics
have integer values with counter semantics, except
openmetrics.control.status, which has a string value. It is
important to note that fetching any of the openmetrics.control
metrics will only update the counters and status values if the
corresponding URL is actually fetched. If the source URL is not
fetched, the control metric values do not trigger a refresh and the
control values reported represent the most recent fetch of each
corresponding source.
The instance domain for the openmetrics.control metrics is adjusted
dynamically as new sources are discovered. If there are no sources
configured, the metric names are still defined but the instance
domain will be empty and a fetch will return no values.
openmetrics.control.status
A string representing the status of the last fetch of the
corresponding source. This will generally be success for an
http response code of 200. This metric can be used for
service availability monitoring - provided, as stated above,
the corresponding source URL is fetched too.
openmetrics.control.status_code
This metric is similar to openmetrics.control.status except
that it is the integer response code of the last fetch. A
value of 200 usually signifies success and any other value
failure. This metric can also be used for service
availability monitoring, with the same caveats as
openmetrics.control.status.
openmetrics.control.calls
total number of times each configured metric source has been
fetched (if it's a URL) or executed (if it's a script), since
the PMDA started.
openmetrics.control.fetch_time
Total time in milliseconds that each configured metric source
has taken to return a document, excluding the time to parse
the document.
openmetrics.control.parse_time
Total time in milliseconds that each configured metric source
has taken to parse each document, excluding the time to fetch
the document.
When converted to a rate, the calls metric represents the average
fetch rate of each source over the sampling interval (time delta
between samples). The fetch_time and parse_time counters, when
converted to a rate, represent the average fetch and parsing latency
(respectfully), during the sampling interval.
The openmetrics.control.debug metric has a singular value, defaulting
to 0. If a non-zero value is stored into this metric using
pmstore(1), additional debug messages will be written to the PMDA log
file.
pmdaopenmetrics and libpcp internals impose some numerical
constraints about the number of sources (4095), metrics (1024) within
each source, and instances for each metric (4194304).
Install the OpenMetrics PMDA by using the Install script as root:
# cd $PCP_PMDAS_DIR/openmetrics
# ./Install
To uninstall, the following must be done as root:
# cd $PCP_PMDAS_DIR/openmetrics
# ./Remove
pmdaopenmetrics is launched by pmcd(1) and should never be executed
directly. The Install and Remove scripts notify pmcd when the agent
is installed or removed.
When scripts and .url files are added, removed or changed in the
configuration directory, it is usually not necessary to restart the
PMDA - the changes will be detected and managed on subsequent
requests to the PMDA.
$PCP_PMDAS_DIR/openmetrics/Install
installation script for the pmdaopenmetrics agent
$PCP_PMDAS_DIR/openmetrics/Remove
undo installation script for the pmdaopenmetrics agent
$PCP_PMDAS_DIR/openmetrics/config.d/
contains URLs and scripts used by the pmdaopenmetrics agent as
sources of openmetrics metric data.
$PCP_LOG_DIR/pmcd/openmetrics.log
default log file for error messages from pmdaopenmetrics
$PCP_VAR_DIR/config/144.*
files containing internal tables for metric and instance ID
number persistence (domain 144).
Environment variables with the prefix PCP_ are used to parameterize
the file and directory names used by PCP. On each installation, the
file /etc/pcp.conf contains the local values for these variables.
The $PCP_CONF variable may be used to specify an alternative
configuration file, as described in pcp.conf(5).
PCPIntro(1), pmcd(1), pminfo(1), pmlogger(1), pmstore(1),
PMWEBAPI(3), pmFetch(3) and
https://openmetrics.io/docs/instrumenting/exposition_formats .
This page is part of the PCP (Performance Co-Pilot) project.
Information about the project can be found at ⟨http://www.pcp.io/⟩.
If you have a bug report for this manual page, send it to
pcp@groups.io. This page was obtained from the project's upstream
Git repository ⟨https://github.com/performancecopilot/pcp.git⟩ on
2020-08-13. (At that time, the date of the most recent commit that
was found in the repository was 2020-08-11.) If you discover any
rendering problems in this HTML version of the page, or you believe
there is a better or more up-to-date source for the page, or you have
corrections or improvements to the information in this COLOPHON
(which is not part of the original manual page), send a mail to
man-pages@man7.org
Performance Co-Pilot PCP PMDAOPENMETRICS(1)
Pages that refer to this page: pmlogger(1) , pmdasenderror(3) , pmdaSendError(3) , pmwebapi(3) , PMWEBAPI(3)