Fast Large File Copy


I work with some fairly large files, one of which is a 1TB+ database backup that I need to move weekly, which as you can imagine, takes a little time. In the past I’ve used ESEUTIL, the Exchange DB tool (/y was a move command) which did an unbuffered copy and used stream reader/writer rather than the windows copy commands. If you care about the inner workings the performance team gives a good overview of this.

http://blogs.technet.com/b/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

They have a great blog that I need to spend more time reading! Anyways, I wanted to try this myself so I first started with writing a small C++ application and quickly learned my C is a tad out of practice. I then moved over to .NET thinking that most of this stuff is just a wrapper to what I’d be accessing in C and the overhead for .NET will be minimal when trying to transfer a large file.

What better way to use .NET than to do it with Powershell! As I got in to writing the function I figured I’d need a comparison to a normal copy so I use the normal PowerShell Copy-Item cmdlet and to my surprise its FAST. I typically check performance by just looking at the cards % utilization in task manager, figure its close enough. I just ran a move between two of my servers and was holding above 45% utilization with a fair deal of time spent around or at 50% of a 1gb connect which is a good deal higher than using ESEUTIL which would hold high 20’s. Obviously windows copy is FAR slower and other tools I’ve tried in the past were no better than ESEUTIL.

so the point of this post… use PowerShell more :)

Advertisements

About jrich

I am the Solutions Architect for Apex Learning in Seattle WA. I've been working with computers since I was 13. Started programming when I was 14. Had my first IT job as tech support at an ISP at the age of 15 and became a network admin at the age of 17. Since then I've worked at a variety of small to mid size companies supporting, maintaining and developing all aspects of IT. Mostly working with Windows based networks but have recently been working with Solaris system as well. I created this blog mostly as a place for me to take my own notes, but also share things that I had a hard time finding the info for.

Posted on September 7, 2010, in WMF (Powershell/WinRM) and tagged , . Bookmark the permalink. Leave a comment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: