data rate

[Brendan Kiburg <kiburg@npl.uiuc.edu>]

Hi Fred,
  I've noticed the lazy logger and the tape logger have been slipping
behind since we moved to this new higher data rate with kicker (~5-6
MB/s)  The archive seems to go at half that speed and the tape drive is
to be about 30 runs behind the data after 5 hours.  
  Currently there are 100 runs on cheetah that have been copied to both
places.  This is about 150 GB of data that could be removed by the lazy
logger right now, if it had to. Since we lose about 10 GB/hour (30 runs
* 1.6 GB/run / 5hours) to the tape drive, this means that in 15 hours, the
buffer on cheetah will be full of data that still needs to be written to
tape.  The archive falls behind even faster due to the 2- 3 GB/s rate that
is achievable. It seems to me there are ~100 runs between the file being
archive and the earliest file we have on disk.  This means that for every
run that is added to the disk, 2 old runs will be deleted by the lazy
logger, so over the next 100 runs written to disk, 200 runs will need to
be deleted to maintain space.  Since we write about 12 runs per hour, the
lag in the archive would cause the disk to run out of space in about 8
hours.

Should we consider trying to borrow the mulantb? if it has any space
left?  Do you already have plans to implement compression for the WFD
data?

I'm not sure the best route for solving this problem, but I'd like to try
to think about it before I get called at in the middle of the night
because the event builder has crashed because cheetah is out of space
:)  Perhaps you could double check my calculations and let me know what
you think.

Best regards,

Brendan