Quantcast
Channel: Hacker News 50
Viewing all articles
Browse latest Browse all 9433

A faster way to delete millions of files in a directory

$
0
0

Comments:"A faster way to delete millions of files in a directory"

URL:http://linuxnote.net/jianingy/en/linux/a-fast-way-to-remove-huge-number-of-files.html


Yesterday, I saw a very interesting method for deleting huge number of files in a single directory. The method is provided by Zhenyu Lee at

http://www.quora.com/How-can-someone-rapidly-delete-400-000-files

Instead of using find and xargs, Lee ingeniously takes the advantage of rsync that he uses rsync –delete to sync the target directory with an empty directory. Later, I did a comparasion on various method that I've used. To my surprise, Lee's method is much faster than others. The following is my benchmark,

Environment:

  • CPU: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
  • MEM: 4G
  • HD: ST3250318AS: 250G/7200RPM
Method # Of Files Deletion Time rsync -a –delete empty/ s1/ 1000000 6m50.638s find s2/ -type f -delete 1000000 87m38.826s find s3/ -type f | xargs -L 100 rm 1000000 83m36.851s find s4/ -type f | xargs -L 100 -P 100 rm 1000000 78m4.658s rm -rf s5 1000000 80m33.434s

With –delete and –exclude, it enables to delete files according to some patterns. Moreover, it is useful when one have to keep the directory itself for some other purpose.

Now I am really curious about why Lee's method is much faster than others, even rm -rf. If anyone have any idea about it, comment here. Thanks very much.


Viewing all articles
Browse latest Browse all 9433

Trending Articles