Data cleaning – should this be done before or after a system migration?

by | Jun 23, 2016

Comedienne Roseanne Barr once stated that she wasn’t going to clean up until ride-on vacuum cleaners were invented.  While Roseanne circumvents this philosophical conundrum by employing housekeepers, most retail database managers operate on leaner resources, with data quality an ongoing challenge.  However the spectre of data quality typically remains in the background – until a major project such as ERP migration and Fulfilment EDI comes along.

The decision to migrate a key system is not taken lightly.  ERP projects, for example, typically cause business disruption, loss of revenue, and reduced productivity – and that is when they go according to plan!  Examples such as Hershey Foods’ botched SAP deployment, which lost $100 million in Halloween sales and 8% of their share price in 1 day, and Nike’s $400 million worldwide i2 deployment that lost $100 million in sales, reflect that even well-managed companies are not immune to project management failures.  Key findings are that planning and system suitability are key to robust outcomes.

Once the decision to migrate to a new ERP is made, various questions around scope and timelines must be addressed – including when and how data cleaning will be undertaken, which is the focus of this blog post (many other critical elements need to be addressed).  Traditional project management tenets require data to be cleansed prior to deployment of the new system, whereas current lean project management standards advocate an ongoing process to eliminate waste of resources.

Supporters of the traditional model believe that data cleaning should be conducted as a discrete project, generally asking employees to collaborate with suppliers over Excel lists of products, descriptions, and attributes, with the goal to cull and correct SKU lists.  Once this is completed, updated data is imported to the new ERP.  This method avoids import of dead SKUs and incorrect data, for a pristine commencement of the new ERP – at least in theory.  The broad criticism of this approach involves the amount of time the data cleanse process takes – which, in our experience, is 6+ months – and the practical reality that even after this huge investment of time and resources, quality problems persist.  In addition to data entry errors, which are virtually impossible to eliminate, a significant proportion of ‘corrected’ data will become invalid during the cleanse project cycle.  ‘Churn’ is impossible to eliminate.

Proponents of modern ‘lean’ project management, with which SPS Commerce agrees, find that any advantage of cleaning data prior to ERP migration is outweighed by the negatives.

 

TRADITIONAL ‘CLEAN FIRST’ APPROACH
Time: a discrete cleanse project delays the actual ERP migration by 6+ months
Productivity: team members in buyer, logistics & warehouse, and accounts teams see an overwhelming proportion of their work hours absorbed by the cleanse project
Inefficiency: every SKU must be checked individually for validity and compliance
Quality: team members focus on the cleanse task, to the neglect of core duties
Error: data entry errors occur as manual updates are transcribed to Excel spreadsheets, including data for new fields that don’t exist in the legacy ERP
Complacency: companies believe that the project will effectively eliminate database error, and plans for ongoing maintenance are neglected
Future needs: this method does not inherently lead to an automated system of attribute management

The alternative, correcting data after the system migration, leverages modern technologies and methods to avoid pitfalls associated with the traditional approach.  With SPS Commerce, for example, automated Fulfilment EDI solutions be used as trigger-points for data review, on an ad-hoc basis.

 

CONTEMPORARY ‘CLEAN AFTER’ APPROACH
Time: the ERP migration is not delayed by data cleanse
Productivity: time allocated to data cleaning is spread across a longer period
Efficiency: team members review data which has been found problematic, rather than reviewing every SKU
Quality: data maintenance is considered an ongoing core responsibility and conducted in parallel with other duties
Error: error rates are minimised through direct in-system ERP entry
Procedural rigour: companies implement a continuous process of data review and management
Future proof: retailers and consumer product companies are readily able to connect assortment (attribute automation) to their information system architecture

 

Modern lean project management practices support a clean-after approach, for better resource management, a higher quality outcome, and future-proofing.  However, if you are considering an ERP deployment, please talk to our consultants or partners to recommend a solution that fits your precise needs.

SPS Australia Blog Team

SPS Australia Blog Team

The Australian SPS blog team combines the experience and insights from dozens of colleagues to deliver news, how-to guides, reports, and more.
SPS Australia Blog Team

GET WEEKLY UPDATES DELIVERED TO YOUR INBOX.

Subscribe to our blog to receive retail and supply chain news, valuable resources, expert tips and more.

Want to learn what your peers are doing?

Almost 500 Australian retailers, suppliers, and distributors shared their winning strategies in our annual benchmark survey. Learn why some companies are pulling ahead while others are falling behind with our 2018 retail report.
Get Your Free Copy

1 Comment

  1. Helena

    Very good information, Thanks for sharing.

    Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

Did you like what you read? Get updates like this delivered directly to your inbox.

Subscribe to our blog to receive weekly retail and supply chain news, valuable resources, expert tips and more.