Wednesday, March 1, 2017

Using ODI Procedures to load data from Source to Target

We came across a scenario where we used ODI procedure to load data from source to target. The scenario was that we had to implement a very complex logic using sub queries in an ODI Interface. given the short timeline, we went ahead with the approach of loading the same using query in an ODI Procedure. We will discuss the same below,

An ODI Procedure has two parts, Command on Source and Command on Target,



  • ODI Command on Source is where you write the source SQL query that you use to fetch data from source. 
  • ODI Command on Target is where you write the insert script to load data into the target. 
  • Load data from one source to another source (eg: Oracle to Teradata, MSSQL to Oracle, etc)
  • The source side query can be a plain query. The number of columns in select though should match with the number columns in the insert script and their datatypes also should match. 
 eg: SELECT ROLL_NO,STUD_NAME FROM STUDENT
  • The insert script should contain bind variables to select values from the source query, the insert script for the above source query would be, 
  INSERT INTO W_STUDENTS_D VALUES (:ROLL_NO,:STUD_NAME)
  • Extremely inefficient compared to interfaces.
  • Never preferred over interfaces. Interfaces are always faster as ODI follows all standards while generating queries. Preferred for loading of tables with less volumes of data.'
  • Row by row processing happens and hence it takes longer time for data load. 
Typically the errors you get are either SQL syntax errors, Data type mismatch, column size errors, column order mismatch errors. 

2 comments:

  1. Hello Experts,

    Please help us to resolve below issue.

    We are facing different issue while we try to delete data using procedure.

    Steps we followed :
    -Create a procedure where in source command we kept select query. "SELECT DOC_ID FROM SOURCE_TAB"
    -In target command we kept delete query which suppose to delete only unmatched data from "TARGET_TAB".
    "DELETE FROM TARGET_TAB WHERE TRAN_ID NOT IN (:DOC_ID)"
    -But our concern is here, procedure executes successfully and its delete all data from target table where source and target table data has same.

    Source table: SOURCE_TAB
    DOC_ID NUMBER(10);
    ====
    10
    20
    30
    40
    50

    Target table : TARGET_TAB
    TRAN_ID NUMBER(10);
    ====
    10
    20
    30
    40
    50

    NOTE: Source and target table data is sale. It should not delete any data.

    ReplyDelete
  2. For newcomers reading this: Deleting row by row for data coming from source, meaning DELETE WHERE DOC_ID NOT IN ( 10 ); then DELETE WHERE DOC_ID NOT IN ( 20 ) ; ...
    Thus, everything is deleted where DOC_ID is not null.

    ReplyDelete