[Wlug] large file performance on linux

[Andre Lehovich]

--- brad noyes <maitre at ccs.neu.edu> wrote:
> Do you have any suggestions on how to speed up performance
> on these writes, filesystem options, kernel options, other strategies, etc?

How are you doing the writes, using fwrite(3), write(2), or mmap(2)?  I've seen
 dramatic speedups reading large files using memory-mapped I/O.

>  - I have also tried spooling over several files (a la multiple volumes) 
>    but i see no difference in performance. In fact, i think this actually
>    hinders performance a bit.

Do you mean multiple physical spindles or drives here?  (I'd expect a slow down
using multiple partitions on the same physical drive.)

>  - I keep my own giant memory buffer where all the data is stored and 
>    then it is written to disk in a background thread. This helps, but
>    i run out of space in the buffer before i finish taking data.

With memory-mapped I/O you could just maintain the buffer and have the kernel
page it out to disk for you as needed; no need to shuffle stuff around with a
background thread.

--Andre




       
____________________________________________________________________________________Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online.
http://smallbusiness.yahoo.com/webhosting