1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
|
* Overview
Ping903 is designed to periodically monitor a very large number of
remote hosts using ICMP ECHO packets. The system is built using the
client-server architecture. The main component (*ping903*) is a daemon
that sits in memory and wakes up periodically to send certain number
of ICMP echo packets to a preconfigured number of hosts and to collect
replies. The resulting round-trip statistics is made available via
REST API.
The daemon reads its settings from a plain text configuration file.
Most settings have sensible defaults, so that the only thing that the
user needs to supply to get started is a list of IP addresses to
monitor. This list is referred to in this document as /ip-list/.
A simple command line client utility (*ping903q*) allows the user to
communicate with the daemon, obtaining the needed information about
each host in particular, or all monitored hosts at once. This utility
can operate in several modes. In particular, it can be used as
Nagios external check tool, instead of the standard *check_ping* utility.
The package and its companion packages are available for download from
https://download.gnu.org.ua/release/ping903.
For the recent news, visit the package development site at
https://puszcza.gnu.org.ua/projects/ping903.
The git repository is available at
http://git.gnu.org.ua/cgit/ping903.git.
* Installation
To build *ping903* you will need *GNU Libmicrohttpd library*. It is
available for download from http://ftp.gnu.org/gnu/libmicrohttpd.
When building from source package, usual incantations apply:
#+BEGIN_SRC shell-script
./configure
make
make install
#+END_SRC
This will install the package under */usr/local*. That is, the server
will be installed as */usr/local/sbin/ping903*, the client program as
*/usr/local/bin/ping903q*, etc. You can give a number of options to
*./configure* in order to customize your installation, in particular to
alter the default installation paths. For example, to install to the
*/usr* file hierarchy, use
#+BEGIN_SRC shell-script
./configure --prefix=/usr
#+END_SRC
Please refer to the =INSTALL= document in the source directory for a
discussion of available options to configure and their effect.
After installing the package, copy the file *src/ping903.conf* to
*/etc/ping903.conf* and edit it to your liking. This file contains
configuration settings that control the behavior of the server daemon
and, to a certain extent, that of a query tool. Short annotations
before each statement will help you navigate through it. You will
find a detailed discussion of the configuration file in the manpage
[[http://man.gnu.org.ua/manpage/?5+ping903.conf][ping903.conf]](5). What follows is a short outline, intended for quick
start.
At the very beginning you can leave most settings at their default
values. You might wish to supply a list of IP addresses to monitor,
although even that is not mandatory, since you can add them later,
when the program is already running. To specify them in configuration
file, use the =ip-list= statement. Its argument is either the name
of a file with the IP addresses, or a list of IP addresses as a
/here-document/:
#+BEGIN_SRC shell-script
ip-list FILENAME
#+END_SRC
or
#+BEGIN_SRC shell-script
ip-list <<EOF
...
EOF
#+END_SRC
(you can use arbitrary word in place of =EOF= in the latter example,
the only requirement being that the list end with exactly the same
word as the one that followed =<<= at its beginning).
In either case, IP addresses must be listed one per line of input.
Leading and trailing whitespace is ignored, as well as empty lines.
Comments are introduced by a hash sign (#) appearing as the first
non-whitespace character on a line.
You are not required to keep all your IP addresses in a single file.
If necessary, you can scatter them among several files and name each
of them in a separate =ip-list= statement.
IP addresses listed in ip-list files form the /immutable/ IP list,
called so because it cannot be altered while the program is running.
The REST API allows the user to add any number of IP addresses at
runtime as well as remove any of IP addresses added this way. These
addresses form the /mutable/ IP list. Mutable IP list is preserved
across program restarts.
This means that actually the immutable IP list is optional. You may
choose to keep all monitored addresses in an external storage (an SQL
database, for example) and load them dynamically after the daemon
has started. A working example program for adding IP addresses from
a MySQL database is shipped in the [[http://git.gnu.org.ua/cgit/ping903.git/tree/examples][examples]] directory. A full-fledged
client package able to add or delete keywords at runtime, both
individually or in batches and providing another features is available
from <http://git.gnu.org.ua/cgit/ping903/mangemanche.git>.
Normally, the ip-list file should contain IP addresses of the hosts to
monitor. It is OK, however, to use symbolic DNS names, too. If a
hostname resolves to a single A record, such usage is equivalent to
placing that IP in the ip-list. However, if the hostname resolves to
multiple IPs, only first one will be used.
By default, the server will wake up each minute and send 10 echo
requests within 1 second intervals to each registered IP. If the
number of collected replies is less than 7, the IP will be declared as
dead ("alive": false, in the returned JSON). Otherwise it is
considered alive ("alive": true).
The following settings control these parameters:
- probe-interval N ::
Interval between wake-ups in seconds.
Default *N=60*.
- ping-count N ::
Number of ICMP packets to send within each probe.
Default *N=10*.
- ping-interval N ::
Interval in seconds between two sequential echo requests.
Default *N=1*.
- tolerance N ::
Maximum number of lost requests after which the host is considered
dead.
Default *N=3*.
Another statement worth your attention is =listen=. It configures the
IP address and port on which the server will listen for incoming HTTP
requests. The default is *localhost:8080*. Change this setting if
this port is already occupied on your system or if you wish to make
ping903 accessible from outside.
The access to the HTTP interface is protected by the default access
control library (the files */etc/hosts.allow* and */etc/hosts.deny*).
Refer to [[http://man.gnu.org.ua/manpage/?3+hosts_access][hosts_access]](3) for details.
When you have finished with the configuration file, run *ping903* to
start the daemon. Check if there are no errors (on the standard
error and in the syslog channel *daemon*). To verify if the daemon is
operational, run
#+BEGIN_SRC shell-script
curl http://localhost:8080/config
#+END_SRC
This should return the running configuration.
Within the next =probe-interval= seconds the server will collect
enough statistics to answer your queries. You can request information
about any particular IP from your ip-list by running
#+BEGIN_SRC shell-script
ping903q IP
#+END_SRC
This will return the current status of the IP, e.g.
#+BEGIN_SRC shell-script
$ ping903q 203.0.113.1
203.0.113.1 is alive
#+END_SRC
To get the detailed statistics use the *-v* option. The result will be
formatted in a [[http://man.gnu.org.ua/manpage/?8+ping][ping]](8)-like manner:
#+BEGIN_SRC shell-script
$ ping903q -v 203.0.113.1
203.0.113.1 is alive
--- 203.0.113.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9414ms
rtt min/avg/max/mdev = 41.212/41.265/41.374/0.046 ms
#+END_SRC
In both cases, any number of IP addresses can be given. E.g. the
following command will returns statistics for two IPs:
#+BEGIN_SRC shell-script
$ ping903q -v 203.0.113.1 203.0.113.5
#+END_SRC
To check the current status of all hosts, run
#+BEGIN_SRC shell-script
$ ping903q -a
#+END_SRC
Note, that depending on your settings the output can be huge.
Please refer to [[http://man.gnu.org.ua/manpage/?1+ping903q][ping903q]](1), for a detailed discussion of the tool.
* System start-up sequence
To configure *ping903* to start automatically at the system start-up,
see the [[http://git.gnu.org.ua/cgit/ping903.git/tree/rc][rc]] subdirectory. It contains start up scripts for various
flavors of GNU/Linux distributions. Please refer to the *README* file
in this directory for detailed instructions.
* Nagios external check
The *ping903q* tool can be used as a Nagios external check program. The
following snippet illustrates the simple Nagios configuration that
makes use of it:
#+BEGIN_SRC nagios-script
# Define the check_ping903 command
define command {
command_name check_ping903
command_line /usr/bin/ping903q -r -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$
}
# Define the service using the new command
define service {
host_name server.example.net
address 203.0.113.1
service_description Server status
check_command check_ping903!200.0,20%!600.0,60%
check_interval 5
retry_interval 1
}
#+END_SRC
* Installation from a git clone
If you are building from a clone of the Git repository, you will need
GNU autotools to bootstrap the package first. Run
#+BEGIN_SRC nagios-script
./bootstrap
#+END_SRC
in the top level source directory. This will create the configure
script and populate the directory with the missing files. Then proceed
as described above.
* REST API
The default channel for communication with the *ping903* daemon is the
HTTP socket open on localhost port 8080. Only GET requests are
allowed. The following endpoints are provided:
** /id
Identifies the running instance. On success, a JSON object with the
following attributes is returned:
- "package": string ::
The package name.
- "version": string ::
Package version string.
- "pid": number ::
PID of the running instance.
** /id/ATTR
*ATTR* is one of the attributes discussed above. Returned is the value
of that attribute.
** /host/[NAME]?[select=HOSTLIST][attr=ATTRLIST]
*NAME* is the IP address or hostname and *HOSTLIST* is a comma-separated
list of such names. If *NAME* is supplied, it is added at the beginning
of *HOSTLIST* and the hosts in *HOSTLIST* are looked up in the list of
hosts being monitored. Note that *NAME* is treated as a character
string and must coincide exactly with the IP or hostname as it was
supplied in configuration. In particular, if a host was specified by
its symbolic DNS name in the configuration, exactly that name must be
used in URL to obtain statistics for that host. If you wish to use
IP, see the */match* endpoint, discussed below.
The return value is a JSON array whose elements correspond to the
entries in *HOSTLIST* (after addition of *NAME*, if given). Each element
is an object with the following attributes:
- "name": string ::
The IP or hostname of the host under which it was supplied in the
ip-list.
- "validity": boolean ::
Status of this record. If false, the data has not been collected yet
or the host is unreachable. A more detailed information is available in
the "status" member (see below). If "validity" is false, only the
following keys are warranted to be present in the object: "name",
"validity", "status", and "xmit-timestamp". If it is true, the full
statistics is available as described below.
- "status": string ::
Detailed status of the object. Following values are defined:
- "init" ::
Initial state: data are being collected ("validity":false).
- "valid" ::
The object is valid and its statistics is reliable ("validity": true).
- "pending" ::
The object is valid, it contains reliable statistics. The host
is being probed at the moment and the object will be updated
soon ("validity": true).
- "invalid" ::
Host is unreachable. No statistics available ("validity": false).
- "xmit-timestamp": number ::
Time (the number of seconds since the Epoch) when the last ICMP
ECHO request was transmitted.
- "start-timestamp": number ::
Time when the recent probe sequence was initiated.
- "stop-timestamp": number ::
Time when the recent probe sequence was finished.
- "xmit": number ::
Number of ICMP ECHO requests transmitted during the probe.
- "recv": number ::
Number of ICMP ECHO responses received during the probe.
- "loss": number ::
Percentage of lost packets.
- "tmin": number ::
Minimal round-trip time observed during the probe.
- "tmax": number ::
Maximal round-trip time observed during the probe.
- "avg": number ::
Average round-trip time.
- "stddev": number ::
Standard deviation of round-trip times.
- "alive": boolean ::
Host status computed as a result of the probe. It is true, if the
difference between "xmit" and "recv" parameters is less than the
"tolerance" configuration setting, and false otherwise.
Example of the returned JSON for a reachable host:
#+BEGIN_SRC json
{
"alive":true,
"avg":25.85150,
"dup":0.00000,
"loss":0.00000,
"name":"203.0.113.1",
"recv":10.00000,
"start-timestamp":1581666176.01285,
"status":true,
"stddev":0.03201,
"stop-timestamp":1581666185.27210,
"tmax":25.91400,
"tmin":25.81200,
"xmit":10.00000,
"xmit-timestamp":1581666185.24628
}
#+END_SRC
Example of the returned JSON for an unreachable host:
#+BEGIN_SRC json
{
"name":"203.0.113.2",
"status":false,
"xmit-timestamp":1581666176.01373
}
#+END_SRC
The "attr" request argument allows you to specify attributes in the
"stat" object that you are interested in. It is a comma-separated
list of attribute names. If given, each returned "stat" object will
contain only elements from that list.
** /host
Return statistics for all monitored hosts. The result is returned as
an array of JSON "stat" objects (described above).
This is an experimental endpoint. Be careful with it, as it may cause
considerable strain on the server.
** /match/[HOST]?[select=HOSTLIST]
Return monitored names that correspond to *HOST* or *HOSTLIST*. *HOSTLIST*
is a comma-separated list of host names or IPv4 addresses, *HOST* is a
single such address. Both *HOST* and *HOSTLIST* can be supplied:
=/match/HOST?select=HOSTLIST= is equivalent to
=/match?select=HOST,HOSTLIST=.
Each name in the resulting list is resolved and monitored hosts
with IPs matching any of its IPv4 addresses are returned as an
array of JSON objects. Each element in the array describes a single
host from the list and has the following attributes:
- "name": string ::
Name under which this host appears in the *HOSTLIST*.
- "hosts": array of strings ::
Array of monitored names corresponding to this "name". Each name
from the array can be used as argument in a GET request to the
*/host* endpoint.
This array is empty if none of the IP addresses of this "name" are
monitored by the server.
If any error occurred during processing, the following attribute is
added as well:
- "error": string ::
Textual description of the error.
** /config
Return current server configuration as a JSON object.
** /config/KEYWORD
Return the value of a particular configuration setting.
* Updating configuration on the fly.
The following requests allow administrator to update the IP list
without reloading the server. For the purpose of updating the
IP list is sectioned in two parts:
- 1. Immutable IP addresses ::
These are IP addresses specified in the configuration file via
the =ip-list= statement. These addresses cannot be modified using
the API described in this section. An attempt to do so will return
an error status.
- 2. Mutable IP addresses ::
These are additional IP addresses configured via this API.
The mutable IP addresses are saved in the file
*/var/lib/ping903/ip-list* before starting next ping probe.
This file is read upon start-up, after all files supplied
in the configuration have been read and processed. This ensures
that the mutable IP address list persists between restarts.
** POST /config/ip-list
Adds one or more IP addresses to the list. The request must have
the content type "application/json". The content can be either an
array of IP addresses in dotted-quad representation (or hostnames
that can be resolved to IPv4 addresses) or an object. The latter
must contain the attribute "ip-list" whose value is an array of
IP addresses formatted as described above, and the "mode" attribute.
If "mode" has the value "replace", the new addresses will replace
the current content of the ip-list. If its value is "append", the
new addresses will be appended to the ip-list.
On success, returns 200 (OK). On error, returns a meaningful error
status. If the error response has the content type
"application/json", the returned JSON object describes the error in
detail. It contains at least the "message" attribute with a
descriptive explanation of the error. If the error refers to an
element of the "ip-list" array, the "index" attribute contains the
1-based index of that element in the array.
** PUT /config/ip-list/IP
Adds IP to the current IP list. Returns HTTP status 201 (Created) on
success. On error, the following codes can be returned:
- 403 (Forbidden) ::
The entry for this IP address already exists or (if a hostname is
given) the argument cannot be resolved to a IPv4 address. If the
Content-Type of the response is "application/json", the "message"
attribute of the returned JSON object supplies an explanation of
the error.
- 500 (Internal server error) ::
Ping903 was unable to fulfill the request. See log files for a
detailed diagnostics.
** DELETE /config/ip-list/IP-OR-HOSTNAME
Deletes *IP-OR-HOSTNAME* from the IP list. Returns 200 (OK) on success.
If *IP-OR-HOSTNAME* was not found in the IP list or is immutable,
returns 404 (Not found).
All update requests are queued and take effect at the beginning of the
next ping probe.
* References
Ping903 is documented in the following manual pages:
- [[http://man.gnu.org.ua/manpage/?8+ping903][ping903]](8) :: Documentation for the ping903 server program.
- [[http://man.gnu.org.ua/manpage/?5+ping903.conf][ping903.conf]](5) :: Documents the ping903 configuration.
- [[http://man.gnu.org.ua/manpage/?1+ping903q][ping903q]](1) :: Documents the ping903 query program.
- [[http://man.gnu.org.ua/manpage/?5+ping903.cred][ping903.cred]](5) :: Documents format of the user's credential storage file.
* Bug reports
If you think you found a bug in *ping903* or its documentation, please
send a mail to [[mailto:gray@gnu.org][Sergey Poznyakoff]] or use the bug tracker at
https://puszcza.gnu.org.ua/bugs/?group=ping903 (requires authorization).
* Copyright
Copyright (C) 2020 Sergey Poznyakoff
Permission is granted to anyone to make or distribute verbatim copies
of this document as received, in any medium, provided that the
copyright notice and this permission notice are preserved,
thus giving the recipient permission to redistribute in turn.
Permission is granted to distribute modified versions
of this document, or of portions of it,
under the above conditions, provided also that they
carry prominent notices stating who last changed them.
* Document settings :noexport:
Please ignore this section. It supplies the variables necessary for
proper rendering of this document.
:PROPERTIES:
:VISIBILITY: folded
:END:
#+TITLE: ping903
#+STARTUP: showall
#+EXCLUDE_TAGS: noexport
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="style.css" />
#+OPTIONS: ^:nil
# Local Variables:
# mode: org
# paragraph-separate: "[ ^L]*$"
# version-control: never
# End:
|