[Wlug] large file performance on linux

[Jeff Moyer]

==> On Wed, 16 May 2007 16:26:53 +0000, brad noyes <maitre at ccs.neu.edu> said:

brad> On Wed, May 16, 2007 at 11:38:09AM -0400, Jeff Moyer wrote:
brad> > ==> On Wed, 16 May 2007 14:53:16 +0000, brad noyes <maitre at ccs.neu.edu> said:
brad> > 
brad> > brad> Hello All,
brad> > brad> I am seeing some really slow performance regarding large files on linux. I
brad> > brad> write a lot of data points from a light sensor. The stream is about 53 Mb/s and
brad> > brad> i need to keep this rate for 7 minutes, that's a total of about 22Gb. I
brad> > brad> can sustain 53Mb/s pretty well until the file grows to over 1Gb or so, then
brad> > brad> things hit the wall and the writes to the filesystem can't keep up. The writes
brad> > brad> go from 20ms in duration to 500ms. I assume the filesystem/operating system 
brad> > brad> is caching writes. Do you have any suggestions on how to speed up performance 
brad> > brad> on these writes, filesystem options, kernel options, other strategies, etc?
brad> > 
brad> > Of course.  Your data set is larger than the page cache, so when you
brad> > hit the low watermark, it starts write-back.  You can deal with this a
brad> > few different ways, and I'll throw out the easiest ways first:
brad> > 1) Get more memory
brad> > 2) Get a faster disk
brad> > 
brad> Ha :).  I have 12GB of memory. Which actually brings me to another question. 
brad> How do i alter the per-process memory limit? I can only allocate a memory 
brad> buffer that is 3GB. I'd like to make use of the other 8GB left in the machine.
brad> If i can double my buffer size i think i could sustain the 53MB/s for 7
brad> minutes that i need.

I'm guessing that you're using a 32 bit system, is that right?  32 bit
systems have the 3/1 memory split, unless you're using Ingo's 4/4
split patch (which Ubuntu doesn't carry, I think).

brad> i'm using the 2.6.20 kernel from the ubuntu source tree. I recompiled it to get
brad> the large memory support, up to 64GB.

OK, yes, 32 bit system.

brad> I was looking for some tunable vm options in sysctl, but i didn't see much that
brad> made sense to me. If nothing else helps perhaps i will ask about the vm
brad> options.

Look under /proc/sys/vm.  Documentation for these variables might be
in Documentation/filesystems/proc.txt (it's not always up-to-date).
But, as I said, I don't think this is the right avenue to explore.
You can get more predictable results by using AIO+O_DIRECT (or maybe
even O_SYNC as another mentioned).

-Jeff