Large Scale Identity Data Migration

Issues and Considerations

Why This Matters

• Data migration is often more costly and time-consuming than expected.
• Poor planning leads to project failure.
• Without a proper audit:
  • System integrity is not assured.
  • Hidden issues may cause data loss or incorrect outcomes.

Data Audit Techniques

• Sampling
• Manual review
• Automated tools

Prominent Issues in Data Migration

• New biometric matches
• Data inconsistencies
• Technical implementation errors
• Data reshaping challenges
• Remediation gaps
• Parallel operations difficulties

New Biometric Matches

• New algorithms expose previously undetected issues:
  • Matches on previously unlinked individuals
  • False positives on prior candidates
  • Records with low biometric quality (“untemplate-able”)

Data Inconsistencies

• Example issues from legacy systems:
  • Special characters in names
  • Inconsistent date formats
  • Daylight savings effects
  • Legacy business case support
  • ...

Technical Implementation Problems

• Unique identifier mix-ups (biometric & biographic)
• Invalid XML/JSON in the database
• Error codes stored in place of actual data
• ...

Data Reshaping Issues

• Adapting old data to a new structure caused:
  • Data loss
  • Misrepresented relationships
  • Loss of visibility of important properties
  • ...

Data Remediation Challenges

• Fixes applied without resolving root cause
• Root cause resolved but no remediation applied
• Overlapping issues obscure diagnosis
• No clear “Data Migration Owner”:
  • Completion vs. Quality mindset conflict

Parallel Operations Risks

• Hard to keep both systems consistent
• Syncing processes often lag
• Business decisions may be made with outdated data

Conclusion & Recommendations

• Establish Migration team early
• Plan and audit all migration steps
• Use detailed data mapping specs
• Conduct data landscaping upfront
  • Biometric Quality important aspect
• Manage outputs across versions
• Prioritize parallel operations support
• Use automated reconciliation tools to ensure:
  • Traceability
  • Operational assurance
  • Targeted remediation
  • Confidence in switchover