Reduce number of required file handles

[Phhere]

I have created a small hack to reduce the number of file handles.

It uses python stringio to to everything in memory and just write the files on close.

To enable it use USE_MEM=1 as environment var

It will use much more memory but sometimes this is easier to archive as to change ulimits

[MrTomRod]

@davidemms @Phhere Why open all files at the same time? Is it much slower to open the necessary files in append mode? Something like this:

def WriteOlogLinesToFile(file: str, text: str, lock: Lock):
    if len(text) == 0:
        return
    if debug:
        util.PrintTime("Waiting: %d" % os.getpid())
    lock.acquire()
    try:
        if debug:
            util.PrintTime("Acquired lock: %d" % os.getpid())
        with open(file, 'a') as f:
            f.write(text)
    finally:
        lock.release()
        if debug: 
            util.PrintTime("Released lock: %d" % os.getpid())

[alexpmagalhaes]

Can you tell us more about memory requirements? How much more should we expect?