High memory usage for long running commands with lots of output

[Karazum]

Describe the problem

Long running commands with lots of output consume too much memory.

All stdout and stderr data is aggregated for the duration of the command, but that is a problem for commands that should run as long as possible.

Even if I run tty-command as following:

TTY::Command.new(uuid: true, printer: :null, :out => "/dev/null", :err => "/dev/null").run(cmd) { |out, err| } 

the output data is still aggregated.

And I believe the problematic code is in this method:

   def read_streams(stdout, stderr)
        stdout_data = [] 
        stderr_data = Truncator.new

        out_handler = ->(data) {
          stdout_data << data
          @printer.print_command_out_data(cmd, data)
          @block.(data, nil) if @block
        }

        err_handler = ->(data) {
          stderr_data << data
          @printer.print_command_err_data(cmd, data)
          @block.(nil, data) if @block
        }

        stdout_thread = read_stream(stdout, out_handler)
        stderr_thread = read_stream(stderr, err_handler)

        stdout_thread.join
        stderr_thread.join

        encoding = @binmode ? Encoding::BINARY : Encoding::UTF_8

        [
          stdout_data.join.force_encoding(encoding),
          stderr_data.read.dup.force_encoding(encoding)
        ]
      end

stdout_data and stderr_data is appended without regard of using too much memory.

Actual behaviour

After few days of running a command that prints lots of data to stdout, the command is terminated by the system because it uses to much memory.

Expected behaviour

One of following:

• create an option to collect max N bytes of stdout/stderr data
• do not collect data if block is given with .run command
• aggregate the data in a fixed buffer size, so that only the last N bytes are stored in memory

Describe your environment

Docker container in kubernetes

• OS version: Debian bullseye
• Ruby version: 2.7.6
• TTY::Command version: 0.10.1