Transferring huge amount of data

[abhaysk]

Just to save 1) Space, 2) I/O and, 3) FTP bandwidth

1- exp writes to a pipe and compress writes to the filesystem a already compressed file.

2) because of #1, you do a lot less I/O

3) because you get a compressed dump file you save network bandwidth during FTP

Also, during uncompress/imp you will save a lot of I/O.

Well point 1 & particularly 3 is all that is most important and techinically right too.

Point 2 may be incorrect, as you will still need to do LIO + lots of CPU to compress/uncompress.

Rgds
Abhay

[PAVB]

Point 2 may be incorrect, as you will still need to do LIO + lots of CPU to compress/uncompress.

I'm in agreement with your assesment of increased CPU utilization for compress/uncompress purposes.

In regards to LIO -respectfully- I don't care about it, I care about physical I/O. Remember, we are doing exp/compress and uncompress/imp on-the-fly therefore in the first case exp is writting to an Unix pipe then compress reads from the Unix pipe and writes to disk; in the second case uncompress reads from the compressed file and writes to an Unix pipe from where imp reads.

[nir_s]

PAVB - don't you think that running expdp/impdp from the target server is better than exp-compress-ftp-uncompress-imp?

[PAVB]

PAVB - don't you think that running expdp/impdp from the target server is better than exp-compress-ftp-uncompress-imp?

Just to be clear, I'm not suggesting exp-compress-ftp-uncompress-imp.

What I've suggested is a three steps process, not a five steps process:

Step #1: Export/Compress on-the-fly in a single step.
Step #2: FTP compressed file
Step #3: Uncompress/Import on-the-fly in a single step.

The expdb over a NETWORK_LINK and them impdb will move uncompressed data over the network -which is probably your slower component- while proposed strategy will move compressed data.

I will love to see the results of a test, would you mind in testing both alternatives for a medium size table and post results?

[nir_s]

Thanks PAVB.
I'll post the results when we finish testing.

Regards,
Nir

[abhaysk]

I'm in agreement with your assesment of increased CPU utilization for compress/uncompress purposes.

In regards to LIO -respectfully- I don't care about it, I care about physical I/O. Remember, we are doing exp/compress and uncompress/imp on-the-fly therefore in the first case exp is writting to an Unix pipe then compress reads from the Unix pipe and writes to disk; in the second case uncompress reads from the compressed file and writes to an Unix pipe from where imp reads.

I dont think you got the GIST.. i meant LIO + CPU Time for compression is nearly equal to time required for PIO to local file system.. all it matters is how much data your moving over network, that time vs data over netwrok is exponential.

however you should also note that OS will be making lots of LIO+PIO while doing a compression/uncompression (plus a added over head of CPU due to extensive LIO + processing the compressing algorithm).. the point 2 of yours hardly matters.. all it matters is point 3, which is obivious and evident.

Rgds
Abhay.

[PAVB]

LIO + CPU Time for compression is nearly equal to time required for PIO to local file system

That's not true, you are wrong.
If you do not believe me, test it using a reasonable large table.

[abhaysk]

u try to test your self with 2GB file..

test 1: copy the file so that time elapsed will be representative of PIO.

test 2: compress the file and capture the elapsed time.

[PAVB]

u try to test your self with 2GB file..

test 1: copy the file so that time elapsed will be representative of PIO.

test 2: compress the file and capture the elapsed time.

So... you want to test in a cat to know how a dog will behave? LMAO

We have extensively tested the real deal -test in a dog to know how a dog will behave- and I know what I'm talking about. This thread is getting boring, you are not listening.

[abhaysk]

ok fine.. i will not talk more on this.. i dont want to get personal.. if i am wrong i would suerly change myself.. (i think you know the algorithm for compression)