> For the complete documentation index, see [llms.txt](https://docs.inogic.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.inogic.com/deduped/features/clean-up-history-duplicate-records-in-bulk.md).

# Clean Up History Duplicate Records in Bulk

{% embed url="<https://www.youtube.com/watch?v=7oerPERAS20>" %}

DeDupeD tool enables users to clean up historical duplicate records in bulk within Dynamics 365 CRM. Users can run a comprehensive duplicate merge process that identifies and merges multiple duplicate records based on set criteria. This ensures a clean and accurate data environment, and it is relatively more convenient to clean up a large number of duplicate records with the DeDupeD Tool, as it saves users the hassle of going to each record and identifying duplicates.

### DeDupeD Tool Installation?

* Download the DeDupeD tool from [here](https://inogicindia-my.sharepoint.com/:u:/g/personal/storage_inogic_com/EWwcK2oZ5gZPpww_JxvDKNYBI_DlnVOyuiFnUp33t08zfg?e=iwQweq\&download=1).
* After downloading the zip file, select and right-click on it, select properties, and if you got unblocked as shown in the image below, then check the checkbox. Extract the zip; the folder includes the setup and README.md file. Please follow the steps in the README.md file to install the tool.

<div align="left"><figure><img src="/files/7ak0IguQ3Er3YnMYtXMa" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure></div>

* Click on **DeDupeD.msi** to set up the tool.

  &#x20;

<figure><img src="/files/aImMDrZ3XC4Qk784iy5u" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* Click on **‘Next’.**

<div align="left"><figure><img src="/files/IPNMtK8hSoTXbGtjuY6p" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure></div>

* Click on **‘Next’.**

<div align="left"><figure><img src="/files/oCbXBFC6CazBliEfXeHf" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure></div>

* Click on **‘Next’** to install **DeDupeD Tool** on your computer.

<div align="left"><figure><img src="/files/vc5fNhM6vHvKXclPzy64" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure></div>

* Click on **‘Close’.**

<div align="left"><figure><img src="/files/QLKLeGcBmypxqDKaBg3n" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure></div>

* Click **‘Next’** to install **DeDupeDToolSetup** on your computer.

<div align="left"><figure><img src="/files/tVzBI8udzoEehWnBLuHi" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure></div>

• Click on **‘Close’.**

<div align="left"><figure><img src="/files/7hIOXzAT94nIjlw65Wdi" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure></div>

* Right click on **DeDupeD Tool** icon.

<figure><img src="/files/XDJaKeXMPrwjvrHMKNwV" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* You'll now see a screen providing details about the DeDupeD tool. Simply click on **'Continue'** to proceed.

<figure><img src="/files/JAE3dEV9bTnjRbYi8o4b" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* Select deployment type as **‘Office 365**’ and fill in the following details in the given fields-
  * **Display list of available organizations:** Enable the checkbox.
  * **Online Region:** Select the respective region.
  * **Username:** Add your CRM username.
  * **Password:** Add your CRM password.
* After filling out all the details, click on **‘Login’.**

<figure><img src="/files/32IdqZ622DIb5Emd9dez" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

For seamlessly Detecting and Merging (N) number of records in Bulk within Dynamics 365 CRM (On-Premises), choose the deployment type as **'On-Premises'** and provide the required details in the designated fields.

* **Server:** Add your Server IP Address.
* **Port:** Add your Port Number.
* **Authentication Source:** Select **‘Active Directory’.**
* **User Name:** Add your CRM username.
* **Password:** Add your CRM password.
* **Domain:** Add Server Domain Name.
* After filling out all the details, click on **‘Login’.**

<figure><img src="/files/x0gZcZf8UaeKwV032K7q" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* Select your desired environment and click on **‘Login’.**

<figure><img src="/files/wppbLcdE3QeHgYLFemMf" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

### DeDupeD Tool Functions

Once logged in, you will find details of all the configured entities. Select any one of the entities for which you want to find and clean up history duplicate records in bulk.

<figure><img src="/files/hMFFFHzRTeP20CydQ6v5" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* Now select the specific entity view for which you want to merge duplicate records.

<figure><img src="/files/A2nEYPCJAzf3Zauao0W8" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

### **Select Duplicate Detection Rules (Rule-Based Duplicate Detection)**

Previously, the DeDupeD Desktop Tool executed all configured published duplicate detection rules during every duplicate detection run. This limited user flexibility and often resulted in additional effort when users wanted to validate duplicates using only specific business rules.

You can now select one or more duplicate detection rules before initiating duplicate detection, providing greater control over the evaluation process:

·       Only the selected rules are evaluated during the duplicate detection process.

·       Unselected rules are excluded from execution.

·       If no duplicate detection rules are selected, all published rules will be considered during duplicate detection.

This provides greater flexibility in duplicate detection, reduces unnecessary processing by executing only the required rules, simplifies duplicate analysis for business-specific scenarios, and improves the overall user experience.

* Select the required Table for which you want to find duplicates.\
  ![](/files/tMLvtZEIpbhdxbjIGoEA)
* Select the View to filter the records to be scanned.\
  ![](/files/Qsf2ZzRLflvVuHX4UPvb)
* Under Duplicate Rules, select the specific duplicate detection rule(s) you want this run to use. If none are selected, all published rules for the entity are used.\
  ![](/files/86sQfPBZT4zpb45EGAYf)<br>
* Click on **‘Find Duplicates’.**

<figure><img src="/files/WAqHKtQGInFdIFWlD74l" alt=""><figcaption></figcaption></figure>

* DeDupeD tool will start scanning duplicates.

<figure><img src="/files/bombl47sfk57AATvPyA2" alt=""><figcaption></figcaption></figure>

* Now all the duplicate records for the selected entity will appear in a comprehensive view.

<figure><img src="/files/CAnBeAdPaP0UrhhCPr8o" alt=""><figcaption></figcaption></figure>

* You can select specific duplicate groups to merge records leaving the rest of the records unaffected.

<figure><img src="/files/xfkzvioBvvuX3rbUPQpp" alt=""><figcaption></figcaption></figure>

* You can also select all the duplicate records groups for bulk merge at once with just one tick. Here we are going to select all the records in **“All Leads”** view for bulk Merge.

<figure><img src="/files/uAgj4U8iW5lJFCCEjjRx" alt=""><figcaption></figcaption></figure>

* Click on the 'Merge' button to proceed with merging the selected duplicate groups.\
  ![](/files/8i1DKcuC1IHWZ99XCRIW)
* If the selected rules include only fuzzy matching conditions, DeDupeD displays a warning that some records may not be actual duplicates and recommends reviewing them carefully before proceeding. Click 'OK' to continue.\
  ![](/files/luVdS7dVajuC0QoMmvHz)
* Confirm the merge operation by clicking 'Yes' on the confirmation prompt. Note that this action cannot be undone once the merge is completed.\
  ![](/files/8QrrkGtV4ASNEEBloMlb)
* DeDupeD tool will merge the selected duplicate records.\
  ![](/files/5P0TAJXhJ1DFANZZVIYS)
* Once completed, the tool displays the number of records processed successfully, along with a download link for the results.\
  ![](/files/jj8eBdH45PqwEj2MqcEH)
* Click on the "Set Master" button to turn any subordinate record into a master record, depending on your data cleaning needs.

<figure><img src="/files/9PVb7qhtdKbfiyMotM8p" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* View a summary of all the duplicate records and duplicate record groups.

<figure><img src="/files/y2qVpmj9DF0WRJ9GsEIB" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* You can search for any duplicate record using the **‘Search Bar’.**

<figure><img src="/files/kjVKxtDCzhKILxQmAyIn" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* You can export all the duplicates by clicking on **‘Export Duplicates’** button.

<figure><img src="/files/JtZC4zmoi45EYClFedG6" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* You can view more duplicates by clicking on **‘Next’** button.

<figure><img src="/files/XYFUB2qanSBYORyuUg8c" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* To merge all the duplicate records, click on the **‘Merge’** button.

<figure><img src="/files/vx4ESXpfqhF3MgVx0trb" alt="Clean Up History Duplicate Records in Bulk"><figcaption></figcaption></figure>

* Click on **'Yes'.**

<figure><img src="/files/WGDUQS0XNjXq7R1umBFI" alt=""><figcaption></figcaption></figure>

* The tool will merge all the history duplicate records successfully!

<figure><img src="/files/J6EdGpJVAms2q9fRI2bc" alt=""><figcaption></figcaption></figure>

<figure><img src="/files/KkmUuEDbUCCglHDmB8ug" alt=""><figcaption></figcaption></figure>

### Fuzzy Indexing

The DeDupeD Desktop Tool provides a Fuzzy Indexing feature to help users maintain fuzzy indexes for existing records.

Fuzzy indexing is used to prepare records for fuzzy matching based on the fields configured in the entity's matching conditions.

#### How Fuzzy Indexing Works

When fuzzy matching is configured for an entity, the fields selected in the matching conditions are used for fuzzy indexing.

* For new records, fuzzy indexing is performed automatically in real time when the record is created. Therefore, no manual indexing is required for newly created records.
* For existing records, fuzzy indexing must be performed manually using the DeDupeD Desktop Tool.

#### Fuzzy Indexing for Existing Records

To index existing records:

* Open the DeDupeD desktop tool.
* Open the Fuzzy Indexing tab.
* Select the required Entity Name.

<figure><img src="/files/Eq6G80TBlnBOYAiK9PfH" alt=""><figcaption></figcaption></figure>

* Select the required View to filter the records for the selected entity.<br>

<figure><img src="/files/1lALNycwG6yOwQPx5QWg" alt=""><figcaption></figcaption></figure>

* Click Start to begin the fuzzy indexing process.

<figure><img src="/files/CojjBdnWRs7OeWuUHbCD" alt=""><figcaption></figcaption></figure>

* The system indexes the existing records based on the current fuzzy matching configuration.<br>

<figure><img src="/files/RS5E82bprBT0Xj2MvdhL" alt=""><figcaption></figcaption></figure>

* When the process is completed, the system displays a success message. The message also displays the number of records that were indexed.

<figure><img src="/files/nZMcrZ2YaDnJO92TKzr3" alt=""><figcaption></figcaption></figure>

#### Re-indexing After Configuration Changes

Fuzzy indexes are based on the current fuzzy matching configuration. Therefore, if the fuzzy configuration is changed, the existing indexes are removed.

For example, re-indexing is required when you:

* Add or remove a field from the fuzzy matching configuration.
* Change the fields used for fuzzy matching.
* Change fuzzy matching levels such as Low, Medium, or High.
* Make any other modification to the fuzzy matching configuration.

After making a fuzzy configuration change, you must run Fuzzy Indexing again from the DeDupeD desktop tool for the existing records.

This ensures that the existing records are indexed according to the latest fuzzy matching configuration and that the updated configuration works correctly for fuzzy matching.

Note: Whenever the fuzzy matching configuration is modified, always re-index the existing records before relying on the updated fuzzy matching results.

### DeDupeD Scheduled Jobs (Desktop Tool)

The Scheduled Jobs feature in the DeDupeD Desktop Tool allows users to view the results of deduplication jobs and review duplicate groups identified from different sources.

Job results can contain duplicate information generated through different interfaces, scheduled jobs, auto-merge processes, and records processed through the DeDupeD tool.

Viewing Job Runs

To view the scheduled job results:

* Open the DeDupeD desktop tool.
* Open the Scheduled Jobs section.
* Select the required Entity.

![](/files/6KJwh9uhiN4JRdbFirL0)<br>

* Click Show Job Runs.

<figure><img src="/files/0RyXbBjC8hMasXCxIVX1" alt=""><figcaption></figcaption></figure>

* The system displays the available job runs for the selected entity.
* The displayed results include the reviewed duplicate count for the job run.

You can select one job result at a time to view the duplicate groups associated with that job.

![](/files/2ncsScXHW5sVrbo3Fvt8)<br>

#### Viewing Duplicate Groups

After selecting a job result:

* Click Show Duplicates.
* The system displays the duplicate groups identified by the selected job.
* Only duplicate groups containing reviewed records are displayed.

Duplicate groups that have already been denied or merged are not displayed. This ensures that the duplicate list contains only records that are available for further action.

#### Merging Duplicate Records

Users can directly merge eligible duplicate groups from the Show Duplicates screen.

To merge records:

* Select the required duplicate group.
* Review the records in the group.
* Select the duplicate group that you want to merge.
* Click Merge.

![](/files/smNNVy7augPJhwANpX4V)<br>

* The system directly performs the merge operation.
* The resulting record is saved by the system.\
  ![](/files/zbXdjgkUizfC6IGYdlBM)

The Scheduled Jobs feature provides a simple workflow to select an entity, view its job runs, review eligible duplicate groups, and directly merge duplicate records.

{% hint style="success" %}
For further queries, reach out to us at <crm@inogic.com>
{% endhint %}