Fast Large File Copy
I work with some fairly large files, one of which is a 1TB+ database backup that I need to move weekly, which as you can imagine, takes a little time. In the past I’ve used ESEUTIL, the Exchange DB tool (/y was a move command) which did an unbuffered copy and used stream reader/writer rather than the windows copy commands. If you care about the inner workings the performance team gives a good overview of this.
They have a great blog that I need to spend more time reading! Anyways, I wanted to try this myself so I first started with writing a small C++ application and quickly learned my C is a tad out of practice. I then moved over to .NET thinking that most of this stuff is just a wrapper to what I’d be accessing in C and the overhead for .NET will be minimal when trying to transfer a large file.
What better way to use .NET than to do it with Powershell! As I got in to writing the function I figured I’d need a comparison to a normal copy so I use the normal PowerShell Copy-Item cmdlet and to my surprise its FAST. I typically check performance by just looking at the cards % utilization in task manager, figure its close enough. I just ran a move between two of my servers and was holding above 45% utilization with a fair deal of time spent around or at 50% of a 1gb connect which is a good deal higher than using ESEUTIL which would hold high 20’s. Obviously windows copy is FAR slower and other tools I’ve tried in the past were no better than ESEUTIL.
so the point of this post… use PowerShell more :)