Device Deduplication > Adding Custom Data Sources

In addition to the built-in sources, custom data sources can be added to the deduplication pipeline. To include a custom source, create a new Layer 1 transformation task that reads from the source connector and writes its output to the appropriate L1 table.

The custom source task must strictly follow the base schema defined for the target L1 table. The Deduplicator connector validates incoming data against this schema, and rows that do not conform are rejected. Deviating from the schema produces incorrect or incomplete deduplication results.

Required Schema: Devices (L1_Devices-Result)

Each row in the L1_Devices-Result table must include the following fields. Required fields must not be empty.

Column	Type	Description
DeviceId	String	The source system's unique identifier for the device. Required.
Source	String	A unique identifier for the data source. Required. Use a short, lowercase name with no spaces.
HostName	String	The device network name. Optional.
Fqdn	String	The fully qualified domain name of the device. Optional.
SerialNumber	String	The hardware serial number of the device. Optional but strongly recommended for accurate matching.
Uuid	GUID	The BIOS UUID of the device. Optional but strongly recommended for accurate matching.
Domain	String	The Active Directory domain of the device. Optional.
OperatingSystem	String	The operating system name and version. Optional.
Manufacturer	String	The hardware manufacturer of the device. Optional.
DeviceModel	String	The hardware model of the device. Optional.

Required Schema: Software (L1_Software-Result)

Each row in the L1_Software-Result table must include the following fields:

Column	Type	Description
SoftwareId	String	The source system's unique identifier for the software record. Required.
Source	String	A unique identifier for the data source. Required.
DeviceId	String	The device identifier on which the software is installed. Required.
Product	String	The software product name. Required.
Publisher	String	The software publisher name. Optional.
ProductVersion	String	The product version string. Optional.
InstallationDate	Date (YYYY-MM-DD)	The installation date of the software. Optional.

Required Schema: Users (L1_Users-Result)

Each row in the L1_Users-Result table must include the following fields:

Column	Type	Description
ImportId	String	The source system's unique identifier for the user record. Required.
Source	String	A unique identifier for the data source. Required.
Email	String	The user's email address. Optional but strongly recommended for accurate matching.
UserPrincipalName	String	The user principal name (UPN). Optional.
Login	String	The user login name. Optional.
FirstName	String	The user's first name. Optional.
LastName	String	The user's last name. Optional.
IsActive	Boolean	Indicates whether the user account is active. Optional.

After creating the custom L1 task, add it to the 1.1 Transform Data Collection to Unified Schema task group and ensure it runs before the corresponding Combine tasks.

In addition, custom data sources must be manually registered in the following Combine tasks so that the data is included in the deduplication output:

•1.2 Combine Network Adapters — Add the custom source here to include its network adapter data.

•2.2 Combine Unified Devices — Add the custom source here if the source includes DeviceRelations. If the source does not provide any DeviceRelations, the device data is picked up automatically and this step is not required.

•2.2.1 Combine Unified Software — Add the custom source here to include its software installation records.

•2.2.2 Combine Unified Users — Add the custom source here to include its user records.