<< Click to Display Table of Contents >> Raynet One Data Hub > 2026.4 > Administration and User Guide > Deduplication  Software Deduplication — Low Memory Mode

Low Memory Mode is an option available exclusively for Software Deduplication. When enabled, the deduplication process stores intermediate data on disk rather than holding the entire software dataset in RAM. This reduces peak memory consumption during the deduplication run, making it suitable for large software datasets or environments where available RAM is limited.

 

Important: Low Memory Mode is not available for Device Deduplication or User Deduplication.

 

When to Use Low Memory Mode

Consider enabling Low Memory Mode when:

 

·The software deduplication task fails with an out-of-memory error on the available hardware.

·The L1 software result table contains a very large number of records and RAM is constrained.

·Running the deduplication on a machine with limited available RAM is required.

 

If RAM is sufficient for your dataset size, the standard in-memory mode produces faster results and should be preferred.

 

Enabling Low Memory Mode

Low Memory Mode is configured in the Deduplicator connector task settings.

 

1.Open the Deduplicator connector task and navigate to the Configuration tab.

2.Under Type, select Software Deduplication.

3.Scroll to the Low Memory Mode checkbox and enable it.

4.Optionally, set a Cache Capacity value (see below).

5.Save the task configuration and run the task.

 

Cache Capacity

When Low Memory Mode is enabled, a Cache Capacity field appears. This setting controls how many software records are held in an in-memory cache while the remaining data is processed from disk. A larger cache reduces the number of disk reads and improves performance; a smaller cache conserves RAM at the cost of additional disk activity.

 

 

For most environments, leaving the Cache Capacity at its default value of 0 is recommended. The automatic calculation balances memory use and processing performance without requiring manual tuning.

 

Considerations and Limitations

 

·Performance: Low Memory Mode is slower than the standard in-memory mode because software records are read from disk during processing. Expect longer run times compared to a standard configuration with equivalent data.

·Result accuracy: The deduplication algorithm produces equivalent results to the standard mode for the same input data. Minor differences in the order of output records may occur due to the disk-based processing approach, but the detected duplicate pairs and confidence scores remain consistent.

·Availability: Low Memory Mode applies only to Software Deduplication. The option is hidden when Device Deduplication or User Deduplication is selected.

·Disk space: Additional temporary disk space is required during the run. The amount depends on the software record count and the content size of each record. Ensure sufficient free disk space is available on the server before running the task in Low Memory Mode.