User Tools

Site Tools


collectors:teemip-ip-discovery-collector

This is an old revision of the document!


IP Discovery Data collector

name:
IP Discovery Data colletor
description:
Data collector to discover IP addresses
version:
3.1.2
release:
2023-10-30
TeemIp:
3.x
iTop:
3.x
code:
teemip-ip-discovery-collector
localization:
-
state:
stable
provider:
TeemIp
keyword:
discovery, ping, fping, lookup, scan
download:
teemip-ip-discovery-collector-3.1.2-355.zip
github:
teemip-ip-discovery-collector
php-max:
8.1

TeemIp IP Discovery Data Collector is a standalone application that discovers active IPs on networks based on directives provided by the the TeemIp IP Discovery Extension installed on TeemIp.

Vocabulary: by “TeemIp solution” or “TeemIP”, it should be understood: TeemIp standalone or the iTop solution on top of which TeemIp as a module has been installed.

Revision History

Version Release Date Comments
3.1.2 2023-10-30 Implement GitHub PR #4, #5 and #6 - Correct SF Bug #140
3.1.1 2023-08-07 Use fping as an option
Make sure collection order is respected
Complete list of optional attributes
3.1.0 2023-07-25 Includes iTop Data Collector Base 1.3.0
Implement collection plan feature of that collector base
Insure compatibility with PHP 8.1
3.0.1 2022-09-21 Align data model to TeemIp 3.x
Report number of discovered IPs at subnet level
Allow DHCP ranges to be skipped during discovery
0.5.0 2020-10-20 Improves reporting - Consider “connection refused” for scans
0.4.0 2019-10-11 Improves IP lookup and handles DNS views
0.3.0 2019-02-09 Initial production release

Features

Reminder: TeemIp IP Discovery provides a solution to TeemIp administrators to discover or scan their IP networks and to document within TeemIp the results of these discoveries and scans.

The solution is made of 2 components:

  • a discovery engine, called the IP Discovery Data collector or the IP Discovery Application, that discovers IPs through pings, DNS reverse lookups and port scanning,
  • a TeemIp extension, called the IP Discovery Data model, that adapts TeemIp's standard data model to store the discovery outputs and enables communication between a TeemIp instance and the different discovery engines deployed on the network.

The IP Discovery Application is a standalone PHP software that extends iTop's collectors principles. It has been built on top of iTop data collector base extension. TeemIp administrators may deploy one or several IP Discovery Collectors on different servers, including the one that hosts TeemIp. Each of these remote applications connects to TeemIp to retrieve:

  • the list of subnets they have to discover,
  • the global discovery method they need to use (ping, IP lookup, port scan),
  • the restrictions that each subnet may impose (on ping, IP lookup, port scan).

At regular interval (defined by the administrator through a CRON command) the remote collector retrieves in TeemIp the list of networks that it has to discover and, for each of them, the list of IPs already registered. It, then, scans the network, computes the delta between what has been discovered and the situation before the scan, stores the results of its findings locally and pushes these results back to TeemIp through a standard synchronization mechanism:

  • New discovered IPs are created,
  • Ping / fping, IP lookup or scan statuses are updated,
  • A few statics on the discovery are documented within each subnet.
Should your network be compartmentalized by different firewalls, the different instances will allow you to bypass the restrictions imposed by the firewalls: limited pings or udp / tcp connections, DNS not available… Of course, you may install the collector on the server that hosts TeemIp but this is not necessary.

Licensing

TeemIp IP Discovery collector is licensed under the terms of the GNU Affero General Public License Version 3 as published by the Free Software Foundation. This gives you legal permission to copy, distribute and/or modify TeemIp IP Discovery collector under certain conditions. Read the ’license.txt’ file in the distribution. TeemIp IP Discovery collector is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE WARRANTY OF DESIGN, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE.

Limitations

Discovery of IPv6 networks is not handled at this stage.

Requirements

On the application's side, the server must be configured to execute PHP files (minimum PHP version 7.4) and it must have a web access to the TeemIp server through https.

Installation

Do not install this extension in your webserver directories (like apache). otherwise your configuration files (with URLs, credentials) may be accessible from outside

Expand the content of the zip archive in a folder of the server that will run the discovery.

When upgrading from the 3.1.1 version or older toward the 3.1.2 version, you must delete the 2 existing synchro data source TeemIp Discovery Application and TeemIp IPv4 Subnet Discovery that you can reach under the “Configuration / Synchronization Data Sources” menu.

Configuration

Like every iTop collector, the configuration of the application is based on the params.local.xml file located in the conf directory. It has the following XML content:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Parameters specific to the discovery instance. -->
<parameters>
    <!-- TeemIp Application -->
    <itop_url>https://localhost</itop_url>
    <itop_login>admin</itop_login>
    <itop_password>admin</itop_password>
 
    <!-- Class collection sequence -->
    <collectors_launch_sequence type="array">
	<!-- IPv4 addresses -->
	<collector>
	     <name>TeemIpDiscoveryIPv4Collector</name>
	     <enable>yes</enable>
	     <rank>1</rank>
	</collector>
	<!-- IPv4 subnets -->
	<collector>
	     <name>TeemIpDiscoveryIPv4SubnetCollector</name>
	     <enable>yes</enable>
	     <rank>2</rank>
	</collector>
	<!-- IP applications -->
	<collector>
	     <name>TeemIpDiscoveryIPApplicationCollector</name>
	     <enable>yes</enable>
	     <rank>3</rank>
	</collector>
    </collectors_launch_sequence>
 
    <!-- Synchronization parameters -->
    <contact_to_notify></contact_to_notify>
    <synchro_user>admin</synchro_user>
    <json_placeholders>
        <!-- IP Discovery Application UUID -->
	<discovery_application_uuid>1122335588</discovery_application_uuid>
        <!-- Name of synchronization data sources -->
	<ipv4_synchro_name>TeemIp IPv4 Discovery</ipv4_synchro_name>
        <subnetv4_synchro_name>TeemIp IPv4 Subnet Discovery</subnetv4_synchro_name>
    </json_placeholders>
 
    <!-- IP Addresses Parameters -->
    <ip_default_status>unassigned</ip_default_status>
    <ip_default_view>External</ip_default_view>
 
    <!-- Absolute path for ping command - "ping" will be executed with no path if not provided-->
    <ping_absolute_path>/bin/</ping_absolute_path>
    <!-- Absolute path for fping command - "fping" will be executed with no path if not provided-->
    <fping_absolute_path>/usr/bin/</fping_absolute_path>
    <!-- Absolute path for dig command - "dig" will be executed with no path if not provided-->
    <dig_absolute_path>/usr/bin/</dig_absolute_path>
 
    <!-- fping usage may be disabled here -->
    <fping_enable>yes</fping_enable>
 
</parameters>    
Parameter Meaning Sample value
itop_url URL to the TeemIp Application. https://localhost/teemip
itop_login Login (user account) for connecting to TeemIp. Must have admin rights for executing the data synchro. admin
itop_password Password for the iTop account. admin_pwd
collectors_launch_sequence List of collectors to run N/A
collector Details for a collector class N/A
name Name of the collector class TeemIpDiscoveryIPv4Collector
enable Enbale or disable the collect yes
rank Order of the collect amongst allother collectors 1
contact_to_notify The email address of an existing contact in TeemIp to be notified of the results of the synchronization. john.doe@demo.com
discovery_application_uuid String that uniquely identifies the remote the IP Discovery Application in TeemIp. The string is automatically created at creation time of the IP Discovery object FFF3_60D8_FEE6_520D
ipv4_synchro_name Name of the data IPv4 synchro that the discovery application will create in TeemIp. By default, the UUID will be appended to that string. TeemIp IPv4 Discovery
subnetv4_synchro_name Name of the IPv4 subnet data synchro that the discovery application will create in TeemIp. By default, the UUID will be appended to that string. TeemIp IPv4 Subnet Discovery
ip_default_status Status of IPs when create by the discovery application. Possible values: allocated, released, reserved, unassigned unassigned
ip_default_view String that defines the default view to be set on discovered IPs (if relevant) “”
ping_absolute_path Absolute path of the ping tool /bin/
fping_absolute_path Absolute path of the fping tool /usr/bin/
dig_absolute_path Absolute path of the dig tool /usr/bin/
fping_enable Enable usage of fping command yes
The Itop data collector documentation describes the configuration process in details
Since TeemIp 2.3 or iTop 2.5, access to the REST web services is now restricted to the users having the profile REST Services User.

Run the discovery

To launch the IP discovery and synchronization with TeemIp, run the following command (from the root directory where the IP Discovery application is installed):

php exec.php

The following (optional) command line options are available:

Option Meaning default value
--config_file Specify the full path to the configuration file. The file conf/params.local.xml is used by default if this parameter is omitted. empty
--console_log_level=<level> Level of ouput to the console. From -1 (none) to 9 (debug). 6 (info)
--eventissue_log_level=<level> Level threshold to create event issues. From -1 (none) to 9 (debug). 6 (info)
--collect_only Run only the data collection, but do not synchronize the data with iTop false
--synchro_only Synchronizes the data previously collected (stored in the data directory) with iTop. Do not run the collection. false
--configure_only Check (and update if necessary) the synchronization data sources in iTop and exit. Do NOT run the collection or the synchronization
--max_chunk_size=<size> Maximum number of items to process in one pass, for preserving the memory of the system. If there are more items to process, the application will iterate. 1000
--help Usage mode to display exec.php help.

Once invoked through the command line, the discovery application will:

  • Create its collection plan by connecting to TeemIp and retrieve:
    • The main discovery parameters to use,
    • The list of subnets to discover and their specific settings (with regard the ping, ip lookup and scan functions),
    • The list of IPs already configured in TeemIp and their attributes,
  • Connect to TeemIp to create the related Synchronization Data Sources (or check their definition if they already exist and update them if needed)
  • Ping or fping (default) and / or look up and / or scan all subnets' IPs according to plan,
  • Upload the collected data into TeemIp,
  • Synchronize the collected data with the existing TeemIp IPs,
  • Update the discovered subnets with their respective discovery durations,
  • Update statistics of discovery application.

Scheduling

Once you've run the IP Discovery interactively, the next step is to schedule its execution so that the discovery and import occurs automatically at regular intervals.

The IP Discovery application does not provide any specific scheduling mechanism, but the simple command line php exec.php can be scheduled with either cron (on Linux systems) or using the Task Scheduler on Windows.

For optimal results, don't forget to adjust the configuration parameter full_load_interval to make it consistent with the frequency of the scheduling.

Discovery functions

The IP Discovery application discovers networks through 4 standard network functions: fping, ping, IP lookup and port scanning, each of them being individually activated for the applications. Furthermore, a function that is activated at the discovery application level can be inhibited for a subnet.

FPing

If it has been installed on the server and if it has not been disabled through the configuration file, the fping command is used by default to ping IPs. Compared to ping, it provides enhanced performances and drastically reduces the time it takes to ping a network. it is called with the following parameters:

Parameter Defined in Default value
fping_absolute_path /conf/params.local.xml /usr/bin/
Ping timeout (s) IP Discovery Application in TeemIp 1 s

The command is invoked through the exec PHP function.

 <fping_absolute_path/>fping -r1 -t <timeout> -ga <subnet_first_ip> <subnet_broadcast_ip - 1>

For instance:

 /usr/bin/fping -r1 -t1000 -ga 10.128.0.0 10.128.7.254

Ping

This is the standard and well known command based on icmp request. It is called when fping is not installed on the server or when it has been disabled. The following parameters are used:

Parameter Defined in Default value
ping_absolute_path /conf/params.local.xml /bin/
Ping timeout (s) IP Discovery Application in TeemIp 1 s

The command is invoked through the exec PHP function.

 <ping_absolute_path/>ping -c 1 -W <timeout> <ip_to_ping>

For instance:

 /bin/ping -c 1 -W 2 10.11.12.13

IP Lookup

This function checks if an entry exists in the DNS space for the IPs. It relies on the dig command and sends reverse lookups to 1 or 2 DNS servers. It is called with the following parameters:

Parameter Defined in Default value
dig_absolute_path /conf/params.local.xml /usr/bin/
DNS server #1 IP Discovery Application in TeemIp - Can be a FQDN or an IP <empty_string>
DNS server #1 IP Discovery Application in TeemIp - Can be a FQDN or an IP <empty_string>

The command is invoked through the exec PHP function.

 <dig_absolute_path/>dig -x <ip_to_lookup> @<dns_server>

For instance:

 /usr/bin/dig -x 10.11.12.13 @8.8.8.8
 or
 dig -x 10.11.12.13 @mydnsserver.com
 

If no DNS server is provided, then the command is invoked without the @ and dig will rely on the DNS servers defined in the /etc/resolv.conf.

When 2 DNS servers are given then:

  • If the first one provides us with a response, no further lookup is done for the IP.
  • If the first one doesn't provide any response, the second one is tried.

Port scanning

The function is based on the PHP fsockopen function. The test is done according to the following parameters:

Parameter Defined in Default value
Port number IP Discovery Application in TeemIp none
Protocol IP Discovery Application in TeemIp none
Scan timeout (s) IP Discovery Application in TeemIp 1 s

For both UDP and TCP tests, the scan tries to open a socket on the given port.

  • For UDP, we must wait for an answer from the remote host for <scan timeout> seconds before declaring the test as failed.
  • For TCP test, answer may come faster if a remote host exists and listen on that port.

If the protocol is set to “both” then:

  • UDP is tried first.
  • If no answered is received, a test is done with TCP.

According to the parameter Consider “connection refused” as valid, a scan that fails with error 111 (connection is refused) may be considered as valid from a discovery standpoint.

Synchronize discovery applications, IPs and subnets

Discovery results of an IP Discovery Application are fed back to TeemIp through standard synchro data sources. Every IP discovery application will automatically create its own synchro data sources and will use them to push its finding and statistics into TeemIp, systematically, at the end of each discovery process.

These synchro data sources can be listed and managed from the Synchronization Data Sources menu under the Admin tools section. As usual with synchro data sources, the Status tab will provide information on every synchronization that occurred between the remote IP Discovery application and TeemIp.

TeemIp Discovery Application

Default parameters for a given data source are defined in the remote IP Discovery application configuration file.

  • The UUID of the application is appended to the name of the synchro data source for an easy identification,
  • Statistics on discovery durations are updated.

Reconciliation of the Application is made on the UUID. The following attributes may be affected by this synchro:

Name When ? Comment
Last discovery date Creation and update
Duration Creation and update

TeemIp IPv4 Discovery

Default parameters for a given data source are defined in the remote IP Discovery application configuration file.

  • The UUID of the application is appended to the name of the synchro data source for an easy identification,
  • Discovered IPs that don't already exist in TeemIp are created,
  • Already existing IPs are updated,
  • No automatic deletion is done.

Reconciliation of IP addresses is done on the Organization and Address attributes. The following attributes may be affected by this synchro:

Name When ? Comment
Organization Creation only
Status Creation only
Address Creation only
DNS View* Creation only If applicable
Last discovery date Creation and update
Responds to ping Creation and update
Responds to IP lookup Creation and update
FQDN from IP lookup Creation and update
Responds to scan Creation and update
The discovery collector automatically detects if the Zone Management extension is installed on the remote application:
  • If this is the case: the collector handles the DNS view attribute brought by the extension.
  • If this is NOT the case: the DNS view attribute is not managed.

TeemIp IPv4 Subnet Discovery

Default parameters for a given data source are defined in the remote IP Discovery application configuration file.

  • The UUID of the application is appended to the name of the synchro data source for an easy identification,
  • Statistics on discovery durations are updated.

Reconciliation of IP subnets is done on the Organization and Subnet IP attributes. The following attributes may be affected by this synchro:

Name When ?
Last discovery date Update
Ping duration Update
Ping # Update
IP lookup duration Update
IP Lookup # Update
Scan duration Update
Scan # Update
collectors/teemip-ip-discovery-collector.1699027525.txt.gz · Last modified: 2023/11/03 17:05 by cnaud