Deduplication > Adding Custom Data Sources

In addition to the built-in sources, custom data sources can be added to the deduplication pipeline. To include a custom source, create a new Layer 1 transformation task that reads from the source connector and writes its output to the appropriate L1 table.

The custom source task must strictly follow the base schema defined for the target L1 table. The Deduplicator connector validates incoming data against this schema, and rows that do not conform are rejected. Deviating from the schema produces incorrect or incomplete deduplication results.

Required Schema: Devices (L1_Devices-Result)

Each row in the L1_Devices-Result table must include the following fields. Required fields must not be empty.

The table uses the following color coding: rows highlighted in red are required; rows highlighted in yellow are optional but strongly recommended for accurate matching.

Column	Type	Description
DeviceId	String	The source system's unique identifier for the device. Required.
Source	String	A unique identifier for the data source. Required. Use a short, lowercase name with no spaces. The column name deviceEvidence is also accepted and maps to the same value; however, using deviceEvidence is not recommended, as this column name will be deprecated in a future version.
HostName	String	The device network name. Optional.
Fqdn	String	The fully qualified domain name of the device. Optional.
SerialNumber	String	The hardware serial number of the device. Optional but strongly recommended for accurate matching.
Uuid	GUID	The BIOS UUID of the device. Optional but strongly recommended for accurate matching.
Domain	String	The Active Directory domain of the device. Optional.
OperatingSystem	String	The operating system name and version. Optional.
Manufacturer	String	The hardware manufacturer of the device. Optional.
DeviceModel	String	The hardware model of the device. Optional.
Bios	String	The BIOS version or identification string of the device. Optional.
Cpu	String	The processor model name of the device. Optional.
CpuChipCount	Integer	The number of physical CPU chips installed in the device. Optional.
CpuCoreCount	Integer	The total number of physical CPU cores. Optional.
CpuCoreCountLimited	Integer	The effective CPU core count when a core limit is applied, for example due to licensing restrictions. Optional.
CpuLimitCoreCount	Integer	The configured maximum number of CPU cores allowed for this device. Optional.
CpuSpeed	Integer	The CPU clock speed in megahertz. Optional.
CpuThreadCount	Integer	The total number of logical CPU threads available on the device. Optional.
DeviceStatus	String	The current operational status of the device. Optional.
DeviceType	String	The category of the device, for example Workstation, Server, or Laptop. Optional.
Graphics	String	The graphics adapter model or description of the device. Optional.
HyperThreadingAct	String	Indicates whether hyper-threading is active on the device. Optional.
InstallationDate	Date (YYYY-MM-DD)	The date the operating system was installed on the device. Optional.
InventoryDate	Date (YYYY-MM-DD)	The date the inventory data was last collected for this device. Optional.
InventoryNumber	String	An asset inventory or tracking number assigned to the device. Optional.
IpAddress	String	The IP address or addresses associated with the device. Multiple addresses may be provided as a comma-separated list. Optional.
MacAddress	String	The MAC address or addresses of the device's network interfaces. Multiple addresses may be provided as a comma-separated list. Optional.
Network	String	Network information associated with the device. Optional.
Ram	Integer	The amount of installed RAM in bytes. Optional.
SourceTable	String	The name of the source table that provided this device record. Required. Its value can be set using the automatic value column function available in the map step of your transformation.
Storage	Integer	The total storage capacity of the device in bytes. Optional.
Tags	String	Custom tags or labels assigned to the device. Optional.
User	String	The username of the primary or last logged-in user of the device. Optional.
RowIdentifier	GUID	A unique identifier for the row, required for all tables. Use the automatic value column function available in the map step of your transformation to generate this value.

Required Schema: Software (L1_Software-Result)

Each row in the L1_Software-Result table must include the following fields:

The table uses the following color coding: rows highlighted in red are required; rows highlighted in yellow are optional but strongly recommended for accurate matching.

Column	Type	Description
SoftwareId	String	The source system's unique identifier for the software record. Required.
Source	String	A unique identifier for the data source. Required.
DeviceId	String	The device identifier on which the software is installed. Required.
Product	String	The software product name. Required.
Publisher	String	The software publisher name. Optional but strongly recommended for accurate matching.
ProductVersion	String	The product version string. Optional but strongly recommended for accurate matching.
InstallationDate	Date (YYYY-MM-DD)	The installation date of the software. Optional.
Description	String	A description or additional comment for the software entry. Optional.
Edition	String	The software edition, for example Professional or Enterprise. Optional.
FileName	String	The executable file name of the software application. Optional.
FilePath	String	The full installation path of the software on the device. Optional.
import_account_id	String	The account identifier from the import source associated with this software record. Optional.
LanguageCode	String	The language code of the software installation, for example en-US or de-DE. Optional.
last_usage	String	The date or timestamp of the last recorded usage of the software, stored as a string. Optional.
Os	String	The operating system associated with the software installation. Optional.
product_code	String	The product code of the software, for example a Windows Installer product GUID. Optional.
RowIdentifier	GUID	A unique identifier for the row, required for all tables. Use the automatic value column function available in the map step of your transformation to generate this value.
SourceTable	String	The name of the source table that provided this software record. Required. Its value can be set using the automatic value column function available in the map step of your transformation.
Type	String	The category of the software entry, for example ARP, MSI, or File. Optional.
usage_frequency_month	Integer	The number of times the software was used within a month. Optional.
usage_time	Integer	The total recorded usage time of the software in minutes. Optional.

Required Schema: Users (L1_Users-Result)

Each row in the L1_Users-Result table must include the following fields:

The table uses the following color coding: rows highlighted in red are required; rows highlighted in yellow are optional but strongly recommended for accurate matching.

Column	Type	Description
ImportId	String	The source system's unique identifier for the user record. Required.
Source	String	A unique identifier for the data source. Required.
Email	String	The user's email address. Optional but strongly recommended for accurate matching.
UserPrincipalName	String	The user principal name (UPN). Optional.
Login	String	The user login name. Optional.
FirstName	String	The user's first name. Optional.
LastName	String	The user's last name. Optional.
IsActive	Boolean	Indicates whether the user account is active. Optional.
Id	String	An additional identifier for the user record from the source system. Optional.
import_data_source_id	String	The identifier of the data source from the import configuration. Optional.
RowIdentifier	GUID	A unique identifier for the row, required for all tables. Use the automatic value column function available in the map step of your transformation to generate this value.
SourceTable	String	The name of the source table that provided this user record. Required. Its value can be set using the automatic value column function available in the map step of your transformation.
UserKey	String	A system-specific key for the user record in the source system. Optional.

Optional Schema: Device Relations (L1_Devices-Relations)

The L1_Devices-Relations table is fully optional. If the table does not exist or contains no values, the deduplication process continues without interruption. To include device relation data from a custom source, create a Layer 1 transformation that writes to the L1_Devices-Relations table following the schema below.

Column	Type	Description
from_device_key	String	The identifier of the source device in the relation. Required.
to_device_key	String	The identifier of the target device in the relation. Required.
relation_type	String	The type of relationship between the two devices, for example is_running_on_esx or SCCM_DETECTED_VM. Required.
source	String	A unique identifier for the data source. Required.
power_state	String	The power state of the source device in the context of this relation, for example Running or poweredOff.
source_table	String	The name of the source table that provided this relation record. Required. Its value can be set using the automatic value column function available in the map step of your transformation.
import_to_device_host_id	String	The host identifier of the target device as provided by the import source.

Optional Schema: Network Adapters (L1_NetworkAdapters)

The L1_NetworkAdapters table is fully optional. If the table does not exist or contains no values, the deduplication process continues without interruption. To include network adapter data from a custom source, create a Layer 1 transformation that writes to the L1_NetworkAdapters table following the schema below. Alternatively, you can create a MSSQL or MariaDB task that queries your data source for network adapter data and writes the results to this table.

Column	Type	Description
id	GUID	A unique identifier for the network adapter record.
devicerowIdentifier	GUID	The row identifier of the device this network adapter belongs to, referencing the RowIdentifier column in the L1_Devices-Result table.
deviceId	String	The source system's device identifier for the device this network adapter is associated with.
ipAddress	String	The IP address assigned to this network adapter.
macAddress	String	The MAC address of this network adapter.
source	String	A unique identifier for the data source.
rowIdentifier	GUID	A unique identifier for this network adapter row. Use the automatic value column function available in the map step of your transformation to generate this value.

After creating the custom L1 task, add it to the 1.1 Transform Data Collection to Unified Schema task group and ensure it runs before the corresponding Combine tasks.

In addition, custom data sources must be manually registered in the following Combine tasks so that the data is included in the deduplication output:

•1.2 Combine Network Adapters — Add the custom source here to include its network adapter data.

•2.2 Combine Unified Devices — Add the custom source here if the source includes DeviceRelations. If the source does not provide any DeviceRelations, the device data is picked up automatically and this step is not required.

•2.2.1 Combine Unified Software — Add the custom source here to include its software installation records.

•2.2.2 Combine Unified Users — Add the custom source here to include its user records.