[Wlug] large file performance on linux

[brad noyes]

On Wed, May 16, 2007 at 03:19:17PM -0400, John Stoffel wrote:
> >>>>> "brad" == brad noyes <maitre at ccs.neu.edu> writes:
> 
> brad> I am seeing some really slow performance regarding large files
> brad> on linux. I write a lot of data points from a light sensor. The
> brad> stream is about 53 Mb/s and i need to keep this rate for 7
> brad> minutes, that's a total of about 22Gb. I can sustain 53Mb/s
> brad> pretty well until the file grows to over 1Gb or so, then things
> brad> hit the wall and the writes to the filesystem can't keep up. The
> brad> writes go from 20ms in duration to 500ms. I assume the
> brad> filesystem/operating system is caching writes. Do you have any
> brad> suggestions on how to speed up performance on these writes,
> brad> filesystem options, kernel options, other strategies, etc?
> 
> You've already had a good bunch of suggestions, but I've got some
> questions on your hardware.  
> 
i don't mind answering more questions :).

>   - cpu? 
dual xeons (not sure if they are hyper threaded or dual core)
cpu MHz: 3067.044
cache size: 512 KB

>   - memory - 12gb I know 
>   - disk(s)
SCSI hard drives, i believe they are SCSI 320. I have tried in a raid1 as 
well as stand alone. 

>   - RAID setup at all?
tried using RAID1. I'm afraid to try RAID0 b/c the data is pretty vital, but i
may try it.

> 
> One way to get more performance would be to add another disk or two
> and to stripe your data between them.  This assumes you have enough
> PCI bus bandwidth available as well.  You don't say how you're
> capturing the light sensor data, but it's obviosly not over a serial
> port or some other slow device.  Network?  So if you've got 53
> Mbyte/second comming into the system, and another 53Mbytes/second
> writing out to disk, then you're starting to get close to the
> 132Mbytes/sec bandwidth of the PCI bus.
> 
Good point. Its not network, but it is still on the PCI bus. It doesn't seem
that bandwidth is really a problem b/c it works great for the first minute.
Once the file grows past 1GB the writes are extremely slow.

Like i said above i might try the RAID0 configuration.

> Finding a motherboard with two or more PCI busses would help.  Or
> something with PCI-E busses.  It all depends on your budget and the
> data acquisition tool you're using.  
> 
If i were to redesign this system i would really like to use PXI, which was
really meant for this sort of thing. I'm really retrofitting my design to to
something different than intended.

Thanks 
 -- Brad