Solution
DCX worked with DLG to identify the high-level objectives and requirements for the new CDM, and agree the target infrastructure. We agreed on a phased solution.
- ‘Lift, Shift, and Integrate’ - initially, the new and improved database (CDM2) should match the old CDM, but be quicker to build, with better security, reduced overheads, better integration with other DLG systems, and future0proofed for ease of changes later on.
- ‘Improve and Enhance’ - introduce changes, such as amending the suppression hierarchy, adjusting the de-dupe logic (and handling the resulting changes to the unique customer view), incorporating new data feeds and source integration, and allowing for more frequent refreshes.
- 'Phase 3' - support DLG as they move to a more self-serve approach, enabling the organisation to handle simple CDM changes in-house and to have full control of their data destiny.
Keeping customer needs at heart of the solution, we recommended a technology solution with tight security, flexibility, and compatibility with the wider DLG technology stack, including:
- Snowflake - an industry-leader in cloud data warehousing giving DLG a robust, agile data platform purposefully built for the cloud.
- AWS Glue - a highly-regarded platform used for ELT (extract, load, and transform) and de-duplication, which integrates seamlessly with Snowflake to make full use of the key benefits of the data warehouse platform.
- Sagacity Connect - used for suppressions.
- DCX Azure hosted PAF processing - a cloud-based solution selected in order to future-proof CDM2.
Once the requirements and the solution design were agreed, we set about building CDM2, with the following key components:
- Load and validation
- Data cleaning
- PAF processing of new addresses
- De-duplication using machine learning
- Processing, logic, build of the consolidated CDM2 database
- Industry suppressions using Sagacity Connect
- Integrations with existing DLG tools (including Informatica Data Governance).
Results
Thanks to the new, improved CDM2, DLG has seen some fantastic results and they've radically improved their operational efficiency.
In the first stage of the solution, when both the old CDM solution and the new CDM2 were maintained in sync to allow for a reliable comparison, the data in CDM2 either matched CDM or was superior to it in terms of accuracy.
As a result of this, DLG could have confidence in the quality of their data and the solution.
On top of this:
- CDM2 saves DLG 95% of time compared to the old solution - the build now runs overnight allowing for daily builds, and CDM2 builds are completed in close to half the goal time (4 to 5 hours vs. 8 hours).
- DLG has seen a huge reduction in time to run its model scoring, from 7-8hrs to 6-8mins
- The full build runs end to end without any manual intervention – provided the input feeds pass the agreed tolerance checks
- Changes are easier to implement with CDM2 – new feeds of the same type as existing feeds can be integrated easily, and changes to de-dupe logic and suppression hierarchy have already been picked up
- Key security improvements have been realised due to new technology stack – and with reduced time overheads to maintain the infrastructure
- Snowflake has enabled DLG to seamlessly connect its marketing intelligence to enterprise tooling for data visualisation (Tableau) and data science (Domino)
