Transferring huge amount of data - Page 2
DBAsupport.com Forums - Powered by vBulletin
Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 24

Thread: Transferring huge amount of data

  1. #11
    Join Date
    Dec 2002
    Location
    Bangalore ( India )
    Posts
    2,434
    Quote Originally Posted by PAVB
    Just to save 1) Space, 2) I/O and, 3) FTP bandwidth

    1- exp writes to a pipe and compress writes to the filesystem a already compressed file.

    2) because of #1, you do a lot less I/O

    3) because you get a compressed dump file you save network bandwidth during FTP

    Also, during uncompress/imp you will save a lot of I/O.
    Well point 1 & particularly 3 is all that is most important and techinically right too.

    Point 2 may be incorrect, as you will still need to do LIO + lots of CPU to compress/uncompress.

    Rgds
    Abhay
    funky...

    "I Dont Want To Follow A Path, I would Rather Go Where There Is No Path And Leave A Trail."

    "Ego is the worst thing many have, try to overcome it & you will be the best, if not good, person on this earth"

  2. #12
    Join Date
    Mar 2007
    Location
    Ft. Lauderdale, FL
    Posts
    3,554
    Quote Originally Posted by abhaysk
    Point 2 may be incorrect, as you will still need to do LIO + lots of CPU to compress/uncompress.
    I'm in agreement with your assesment of increased CPU utilization for compress/uncompress purposes.

    In regards to LIO -respectfully- I don't care about it, I care about physical I/O. Remember, we are doing exp/compress and uncompress/imp on-the-fly therefore in the first case exp is writting to an Unix pipe then compress reads from the Unix pipe and writes to disk; in the second case uncompress reads from the compressed file and writes to an Unix pipe from where imp reads.
    Pablo (Paul) Berzukov

    Author of Understanding Database Administration available at amazon and other bookstores.

    Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.

  3. #13
    Join Date
    Dec 2001
    Location
    Tel-Aviv,Israel
    Posts
    233
    PAVB - don't you think that running expdp/impdp from the target server is better than exp-compress-ftp-uncompress-imp?

  4. #14
    Join Date
    Mar 2007
    Location
    Ft. Lauderdale, FL
    Posts
    3,554
    Quote Originally Posted by nir_s
    PAVB - don't you think that running expdp/impdp from the target server is better than exp-compress-ftp-uncompress-imp?
    Just to be clear, I'm not suggesting exp-compress-ftp-uncompress-imp.

    What I've suggested is a three steps process, not a five steps process:

    Step #1: Export/Compress on-the-fly in a single step.
    Step #2: FTP compressed file
    Step #3: Uncompress/Import on-the-fly in a single step.

    The expdb over a NETWORK_LINK and them impdb will move uncompressed data over the network -which is probably your slower component- while proposed strategy will move compressed data.

    I will love to see the results of a test, would you mind in testing both alternatives for a medium size table and post results?
    Pablo (Paul) Berzukov

    Author of Understanding Database Administration available at amazon and other bookstores.

    Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.

  5. #15
    Join Date
    Dec 2001
    Location
    Tel-Aviv,Israel
    Posts
    233
    Thanks PAVB.
    I'll post the results when we finish testing.

    Regards,
    Nir

  6. #16
    Join Date
    Dec 2002
    Location
    Bangalore ( India )
    Posts
    2,434
    Quote Originally Posted by PAVB
    I'm in agreement with your assesment of increased CPU utilization for compress/uncompress purposes.

    In regards to LIO -respectfully- I don't care about it, I care about physical I/O. Remember, we are doing exp/compress and uncompress/imp on-the-fly therefore in the first case exp is writting to an Unix pipe then compress reads from the Unix pipe and writes to disk; in the second case uncompress reads from the compressed file and writes to an Unix pipe from where imp reads.
    I dont think you got the GIST.. i meant LIO + CPU Time for compression is nearly equal to time required for PIO to local file system.. all it matters is how much data your moving over network, that time vs data over netwrok is exponential.

    however you should also note that OS will be making lots of LIO+PIO while doing a compression/uncompression (plus a added over head of CPU due to extensive LIO + processing the compressing algorithm).. the point 2 of yours hardly matters.. all it matters is point 3, which is obivious and evident.

    Rgds
    Abhay.
    funky...

    "I Dont Want To Follow A Path, I would Rather Go Where There Is No Path And Leave A Trail."

    "Ego is the worst thing many have, try to overcome it & you will be the best, if not good, person on this earth"

  7. #17
    Join Date
    Mar 2007
    Location
    Ft. Lauderdale, FL
    Posts
    3,554
    Quote Originally Posted by abhaysk
    LIO + CPU Time for compression is nearly equal to time required for PIO to local file system
    That's not true, you are wrong.
    If you do not believe me, test it using a reasonable large table.
    Pablo (Paul) Berzukov

    Author of Understanding Database Administration available at amazon and other bookstores.

    Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.

  8. #18
    Join Date
    Dec 2002
    Location
    Bangalore ( India )
    Posts
    2,434
    u try to test your self with 2GB file..

    test 1: copy the file so that time elapsed will be representative of PIO.

    test 2: compress the file and capture the elapsed time.
    funky...

    "I Dont Want To Follow A Path, I would Rather Go Where There Is No Path And Leave A Trail."

    "Ego is the worst thing many have, try to overcome it & you will be the best, if not good, person on this earth"

  9. #19
    Join Date
    Mar 2007
    Location
    Ft. Lauderdale, FL
    Posts
    3,554
    Quote Originally Posted by abhaysk
    u try to test your self with 2GB file..

    test 1: copy the file so that time elapsed will be representative of PIO.

    test 2: compress the file and capture the elapsed time.
    So... you want to test in a cat to know how a dog will behave? LMAO

    We have extensively tested the real deal -test in a dog to know how a dog will behave- and I know what I'm talking about. This thread is getting boring, you are not listening.
    Pablo (Paul) Berzukov

    Author of Understanding Database Administration available at amazon and other bookstores.

    Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.

  10. #20
    Join Date
    Dec 2002
    Location
    Bangalore ( India )
    Posts
    2,434
    ok fine.. i will not talk more on this.. i dont want to get personal.. if i am wrong i would suerly change myself.. (i think you know the algorithm for compression)
    funky...

    "I Dont Want To Follow A Path, I would Rather Go Where There Is No Path And Leave A Trail."

    "Ego is the worst thing many have, try to overcome it & you will be the best, if not good, person on this earth"

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Click Here to Expand Forum to Full Width