Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
KaiserPro
on Sept 12, 2014
|
parent
|
context
|
favorite
| on:
My experience with using cp to copy 432 million fi...
WE use 10 rsyncs in parallel to copy .5pb in less than 3 days.
CPU is cheap.
spydum
on Sept 12, 2014
[–]
Have done the same, found that rsync seems to not natively parallelize itself, so spread across 20 cores, it really screamed.
rimantas
on Sept 12, 2014
|
parent
[–]
Damn, I could use some advice there. Right now I have a system which uses rsync to backup some files. It takes about 8 hours to go over 7+ million files (~3.5TB). I'd love to speed up this. I should mention that copy is done over curlftpfs :(.
Consider applying for YC's W25 batch! Applications are open till Nov 12.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
CPU is cheap.