[Wlug] large file performance on linux

[brad noyes]

All,

here's a summary of what i have tried so far. I have tried using the O_DIRECT
and O_SYNC flags with open(2) and i have been using a non-journalling
filesystem, ext2. None of these options have yielded any better performance at
the rates i'm looking for.

I have not tried AIO. I don't think i can use a non 32-bit kernel on the xeons.
Correct me if i'm wrong (it's happened before). 

As a temporary solution i created a memory filesystem, by 
  $> mount -t tmpfs tmpfs -o size=6G /data/memory
The writes are fast however i need another process to progressively move data
out of the memory filesystem since i don't have enough memory to hold all 22GB
of data i need.

For those interested i have attached a small program which will use different
options to open and profile the fwrite calls. To compile simply do 
 $> sh ./testwrite.c
 $> ./testwrite -h
 $> ./testwrite -l 16 file.out


I just wanted to keep you all updated. Thanks for all your help. I'll keep
trying various ideas. I wish i could make it to the meeting to buy you all
pizza for your help. Perhaps next month.

Thanks,
  -- Brad

-------------- next part --------------
A non-text attachment was scrubbed...
Name: testwrite.c
Type: text/x-csrc
Size: 4170 bytes
Desc: not available
Url : http://mail.wlug.org/pipermail/wlug/attachments/20070516/5ff568a5/attachment-0001.bin