1. Data Migration in Oracle E-Business Suite
OAUG Paris, June 18, 2004
Trond Steensnaes
trond.steensnaes@accenture.com
Copyright 2003 Accenture. All rights reserved. Accenture, its logo, and Accenture Innovation Delivered are trademarks of Accenture.
2. Project Scenario
Challenges:
• Tight deadlines
• 5-15 Environments
• Squeezed time for migration activities
• Numerous legacy systems to integrate with
• 100% quality requirement
3. Agenda
• Introduction
• Migration to Oracle
– Migration to Oracle – using open interface / API
– Migration to Oracle – Oracle Utilities
– Migration to Oracle – Third party tools
• Migration from Oracle
4. Introduction: Migration scenarios
Data migration: moving data from one database to another.
Types of migration:
• Migrating in an upgrade process
• Migrating legacy data in a new implementation
• Migrating between CRP, Test and Prod instances
• Propagating master reference data
5. Introduction: Types of Data
• Setup Data (Application Object Library (AOL) data)
– Profiles
– Flexfield definitions
• Master Data
– Items
– Customers
• Transaction Data
– Purchase Orders
– Invoices
6. Agenda
• Introduction
• Migration to Oracle
– Migration to Oracle – using open interface / API
– Migration to Oracle – Oracle Utilities
– Migration to Oracle – Third party tools
• Migration from Oracle
7. Migration to Oracle
Supported:
- Entering data via the UI
- Using Open Interfaces / API’s
- Using Oracle data migration utilities
Not supported:
- Direct manipulation of the data in the database.
8. Agenda
• Introduction
• Migration to Oracle
– Migration to Oracle – using open interface / API
– Migration to Oracle – Oracle Utilities
– Migration to Oracle – Third party tools
• Migration from Oracle
9. Migration to Oracle – using open interface / API
Sample usage of open interfaces for conversion purpose:
3-step process:
• Load: Get the data from the flat file into the staging table
• Validate: Assess the quality of the data by calculating and
evaluating metrics.
• Transfer: Insert data into interface table / call APIs with
values in staging table
Run the conversion programs in the E-Business Suite for
security / accessability / traceability.
10. Migration to Oracle – using open interface / API
Load program: generic shell script taking name of
SQL*Loader control file as parameter.
Validate program: 1 per interface.
Transfer program: 1 per interface.
SQL*Loader API’s
Staging tables Core tables
11. Agenda
• Introduction
• Migration to Oracle
– Migration to Oracle – using open interface / API
– Migration to Oracle – Oracle Utilities
– Migration to Oracle – Third party tools
• Migration from Oracle
12. Migration to Oracle – Oracle Utilities
FNDLOAD
Usage: OA Instance OA Instance
Setup Master Transactional
Extract Import
• Moves OA data between database and text file representations.
• Can be modified with any editor
• Use to synchronize databases during new installations or maintenance.
• Good for moving data between test and prod instances, e.g. in upgrade
projects.
• Command-line based tool, requires technically skilled personell to execute
• No cost, well documented.
13. Migration to Oracle – Oracle Utilities
FNDLOAD
An example of downloading is:
FNDLOAD apps/apps@devdb 0 Y DOWNLOAD testcfg.lct out.ldt
FND_APPLICATION_TL APPSNAME=FND
An example of uploading is:
FNDLOAD apps/apps@custdb 0 Y UPLOAD fndapp.lct fnd1234.ldt –
Oracle Applications System Administrator’s Guide, Appendix C
14. Migration to Oracle – Oracle Utilities
iSetup
Usage: OA Instance OA Instance
Setup Master Transactional
Extract Import
iSetup.oracle.com Central instance
Target instance
iSetup Configurator iSetup Migrator
iSetup Reporter
• Load original configuration multiple times
• Load modified configuration
• Load extract
• Load subset of extract
15. Migration to Oracle – Oracle Utilities
iSetup
Typical scenario:
1. Configurator -> CRP
2. Migrator -> Systemtest
3. Migrator -> Production
- Complete or partial collection of data
- Copy data within the same instance (e.g. from one OU to
another)
http://isetup.oracle.com, Oracle iSetup User Guide
16. Migration to Oracle – Oracle Utilities
ADI
Usage: Anything OA Instance
Setup Master Transactional
Extract Import
• ADI is a spreadsheet-based extension of Oracle Applications.
• Limited to GL and FA data
• Import Text File feature to simplify work with Oracle Assets. (Assets and
physical inventory).
Applications Desktop Integrator User’s Guide
17. Agenda
• Introduction
• Migration to Oracle
– Migration to Oracle – using open interface / API
– Migration to Oracle – Oracle Utilities
– Migration to Oracle – Third party tools
• Migration from Oracle
18. Migration to Oracle – Third party tools
DataLoad Classic
Usage: Anything OA Instance
Setup Master Transactional
Extract Import
• Automated loading using copy & paste
• If there is a form for it, DataLoad can insert data into it
• Freeware, no support
• Quick and easy
• Not suitable for large volume loads, ‘out of sync’ problem.
http://www.dataload.net/
19. Migration to Oracle – Third party tools
DataLoad Professional
Usage: Anything OA Instance
Setup Master Transactional
Extract Import
• Automated secure loading
• Fast because load is processed on the middle tier
• Inexpensive
• Requires some technical skills
• Suitable for medium volumes
• Rerun load multiple times in test environment until 100% successful.
http://www.dataload.net/
20. Migration to Oracle – Third party tools
Mercury WinRunner
Usage: Anything OA Instance
Setup Master Transactional
Extract Import
• A testing tool, but has functionality that makes it suitable for automating data
load
• Work similarly to DataLoad Classic, but more reliable.
• Records keystrokes and creates playback script for you.
• Expensive
• If you have it for testing, might as well use it for migration too.
http://www.mercury.com/us/products/quality-center/functional-testing/winrunner/
21. Migration to Oracle – Third party tools
Mercury Object Migrator
Usage: OA Instance OA Instance
Setup Master Transactional
Extract Import
• Utilizes FNDLOAD
• Runs as a concurrent program within OA
• Version control, reporting capabilities, rollback functionality
• Useful if you have to synchronize lots of instances or require elaborate version
control.
http://www.mercury.com/us/products/it-governance-center/change-
management/extensions/object-migrator.html
22. Migration to Oracle – Third party tools
Various adapters / connectors
Usage: Anything Anything
Setup Master Transactional
Extract Import
Insevo (formerly Taviz)
Librados
iWay Software
• Bi-directional connectivity
• Any application, any platform, any data structure, any message type
• Reduced risk
http://wwws.sun.com/software/connectors/
23. Agenda
• Introduction
• Migration to Oracle
– Migration to Oracle – using open interface / API
– Migration to Oracle – Oracle Utilities
– Migration to Oracle – Third party tools
• Migration from Oracle
24. Migration from Oracle
Make your own customized outbound interfaces using
Materialized views
- Customizations must be non-invasive
- Complete control
- Several pitfalls:
- A change in any column in any table, even if it not selected in the
MV definition, will lead to refresh of the MV
- Cannot distinguish between inserts and updates of the MV
- Triggers work differently on MV’s than regular tables
25. Migration from Oracle
Business Event System
- Most significant events are predefined
- Define your own events
- Elegant way to trigger workflow activities / custom logic
- Non-invasive
Open interfaces are tables in which you put your data to be imported and then you run an concurrent job to do the actual import into the core tables. APIs (Application Program Interfaces) are code objects provided for you that you can call from a wrapper with the data to be imported as parameters. Do consider manual entry of the data as an option. Not necessary always elegant, but it might sometimes actually be the cheapest alternative. All product user guides warn you not to use SQL to manipulate data. This might compromise the integrity of the relational data and your data become corrupt.
Set up a document with the file structure definition describing the data requirements to be met by the legacy data. Data cleansing and enrichment should be done prior to extract of legacy data.
.lct file: configuration file (predefined by Oracle). .ldt file: data file. Sample usage: New installation: Concurrent programs from your DEV environment. Maintenance: updated lookup values Limited to migrating these datatypes: • Concurrent program definitions • Request groups • Lookup types and lookup values • Profile options and profile option values • Flexfields setup data • Attachments definitions • Messages • Security information
Consists of: iSetup Configurator: Setup a new Oracle E-business Suite instance iSetup Migrator: Migrate setup data from one Oracle E-Business Suite instance to another. iSetup Reporter: Generate reports on setup data Migrator: Create selection set. A Selection Set is a collection of setup objects for which you want to extract data. Add filters as required. Create a snapshot. This extract takes a source instance and a selection set as input. Key benefits: Reduced implementation risk Lower implementation cost Reduced implementation time Automatic validation and verification of setup dependencies Reduced time to migrate setup information to multiple instances Auditing capabilities to identify setup errors
The Import Text File feature allows you to import data from legacy systems, third party payables applications, barcode scanners, or any other flat data file sources, to your worksheet. Using the Import Text File feature, you can map fields in your source file to fields in your worksheet. Once you have created the map using the Import Text File feature, you can save it and apply it to future conversions.
Formerly Kintana Object Migrator
There are 2 issues to keep in mind when designing/building/testing outbound interfaces: 1. The ‘trigger on materialized views’ issue. Triggers work somewhat differently on materialized views than they do on tables. This is caused by the way the RDBMS maintains these views. Oracle maintains a materialized view in this way: first it DELETES the row (if it exists already) and then INSERTs a new (the updated) row. So for updates, it actually deletes the exisiting row and inserts a new one. This is the reason why an AFTER UPDATE trigger will never fire. The second anomaly is that if you use refresh on commit, it will perform one DELETE/INSERT for each table referenced in the definition of the materialized view that has changed since the last commit in the session. So for a MV that joins three base tables, an INSERT of a new row in the MV results in the following sequence of events: INSERT – DELETE – INSERT- DELETE – INSERT (if no commit has been performed until the three base tables were populated). An UPDATE of two of the three base tables will lead to the following sequence of events: DELETE-INSERT-DELETE-INSERT (if no commit was performed until both updates were executed). It is important that you find out if the interface shall export data upon creation only, or also after the information is updated. 2. When is the data eligible for export? If one joins three tables in the definition of the materialized view, the data will not appear in the view until the user has provided data in all tables, unless outer joins are used. When selecting values from several tables in the materialized view it is important to have an understanding of when data will be inserted into the materialized view. You should select from tables in either the materialized view or in the trigger: Select from the table in the materialized view if the row from that table is required before export of the information. (Or you can outer join to this table if the row is not required before export.) If changes to the row in the table shall trigger export it needs to be included in the materialized view, if not implement it as a lookup in the trigger.
The Business Event System is an application service that uses the Oracle Advanced Queuing (AQ) infrastructure to communicate business events between systems. The Business Event System consists of the Event Manager, which lets you register subscriptions to significant events, and event activities, which let you model business events within workflow processes. Business events represent integrations points. There are in excess of 500 predefined events in 11i. When a local event occurs, the subscribing code is executed in the same transaction as the code that raised the event. Subscription processing can include: executing custom code on the event information, sending event information to a workflow process sending event information to other queues or systems. Oracle Workflow is a component of Oracle9 i Integration, a comprehensive set of infrastructure and tools for integrating enterprise applications and enabling e-business marketplaces. Oracle Workflow leverages the Oracle technology stack, including the Oracle9 i database and Oracle Advanced Queuing, to enable application integration at the business process level. Business event messages from Oracle Workflow can be placed on or received from Oracle9 i Advanced Queues, providing support for Oracle Net as well as HTTP and HTTPS communication protocols. The Oracle9 i Messaging Gateway feature enables integration between Oracle Workflow and other messaging solutions such as IBM MQSeries and TIBCO Rendezvous. Oracle Workflow is also incorporated as part of Oracle9 i Integration in the Oracle E-Business Suite, an integrated suite of applications for the enterprise designed to transform your business into an e-business.
