Crthuky

I have a python program which is, slowly, generating some output.

I want to capture that in a file, but I also thought I could watch it live with tail.

So in one terminal I'm doing :

python myprog.py > output.txt

and in another terminal :

tail -f output.txt

But it seems like the tail isn't showing me anything while the python program is running.

If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.

What am I doing wrong?

You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.

import sys
...
print('Some message')
sys.stdout.flush()

Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.

From man tee:

tee(1) - Linux man page

Name tee - read from standard input and write to standard output and files

Synopsis

tee [OPTION]... [FILE]...

Description

Copy standard input to each FILE, and also to standard output.

-a, --append 
 append to the given FILEs, do not overwrite 
-i, --ignore-interrupts 
 ignore interrupt signals 
--help 
 display this help and exit 
--version
 output version information and exit

If a FILE is -, copy again to standard output.

So in your case you'd run:

python myprog.py | tee output.txt

EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.

tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.

Run python with the unbuffered flag:

python -u myprog.py > output.txt

Output will then print in real time.

Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).

This is a redirect to a regular file.

C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.

You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.

On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.

Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.

Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.

Another option (if you don't care about the contents, just the progress) is pv:

NAME
 pv - monitor the progress of data through a pipe

SYNOPSIS
 pv [OPTION] [FILE]...
 pv [-h|-V]

Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.

If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.