Stderr and Stdout: Understanding Logs and Output

Standard output (stdout) and standard error (stderr) are simple concepts but play a central role in logging, error handling, and data stream management. This article explores the differences and applications of stdout and stderr, especially how to use them effectively in a Python environment.

1 Standard Output (stdout) and Standard Error (stderr)

In most operating systems, standard output and standard error are the two main output streams of a process. They provide a mechanism for a process to send information and error messages to a terminal or file. Although these two streams may be physically the same (e.g., both displayed on the same terminal interface), they are used for different logical purposes:

• Standard Output (stdout): Typically used for outputting the results of program execution or normal operational information.
• Standard Error (stderr): Specifically used for outputting error messages or warnings, which usually need to be seen or recorded even when standard output is redirected.

2 print and logging in Python

In Python, the print function sends information to stdout by default, while the logging module sends log messages to stderr by default. This distinction is made to separate normal program output from logs (including error and debug information), making it easier for developers to manage and filter output information.

2.1 Using print

print is the most basic output function in Python, used to send information to the standard output stream. It is simple and easy to use, suitable for quick debugging or displaying information to the user. For example:

1

print("Hello, world!")

2.2 Using logging

The logging module provides a flexible framework for adding log messages to an application. Unlike print, logging supports different log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL), allowing developers to adjust the detail level and output location of logs as needed. For example:

1
2
3

import logging

logging.error('This is an error message')

3 tqdm and stderr

In complex or long-running programs, using a progress bar is an effective way to show the progress of a process to the user. Python’s tqdm library is a widely used tool for adding progress bars to the command line. tqdm outputs progress information to stderr by default to avoid interfering with normal program output (stdout).

3.1 Splitting stdout and stderr

In some cases, it is useful to separate normal output from error or log messages, such as redirecting them to different files or terminals. On the command line, redirection operators > and 2> can be used to achieve this. In Python code, finer control can be achieved by configuring the logging module or using specific file objects.

1

python script.py > output.log 2> error.log

By using command line redirection, Python’s print function, or even the logging module, you can flexibly control and split these two types of output, making error handling, logging, and user interaction clearer and more orderly.

4 Managing stdout and stderr with nohup

When deploying long-running background processes, the nohup command becomes an important tool. nohup, or “no hang up”, allows commands to continue running after the user logs out, which is especially useful for remotely started tasks. A key feature of nohup is its ability to manage stdout and stderr.

By default, using nohup to run a command will merge stdout and stderr and redirect them to the nohup.out file unless otherwise specified. This means that both normal output and error messages will be captured in the same file for later review. However, in some cases, it may be more useful to separate these two outputs.

4.1 Separating stdout and stderr with nohup

To output stdout and stderr to different files when using nohup, you can use redirection operators in combination. For example:

1

nohup python script.py > output.log 2> error.log &

This command redirects stdout to output.log, stderr to error.log, and runs in the background using &. This way, even if the terminal or SSH session is closed, the program will continue to run, and its output will be properly recorded.

5 Buffering Behavior in Python

stdout and stderr behave differently when buffering data. By default, stdout is line-buffered when connected to a terminal, caching data until a newline character is received or the buffer is full; in non-interactive mode, stdout is block-buffered (like a file). stderr, however, is always line-buffered (before Python 3.9, it was block-buffered in non-interactive mode). The following content is from the official documentation sys — System-specific parameters and functions — Python 3.12.2 documentation

When interactive, the stdout stream is line-buffered. Otherwise, it is block-buffered like regular text files. The stderr stream is line-buffered in both cases. You can make both streams unbuffered by passing the [u](<https://docs.python.org/3.12/using/cmdline.html#cmdoption-u>) command-line option or setting the [PYTHONUNBUFFERED](<https://docs.python.org/3.12/using/cmdline.html#envvar-PYTHONUNBUFFERED>) environment variable.

Changed in version 3.9: Non-interactive stderr is now line-buffered instead of fully buffered.

The smaller the buffering granularity, the more timely the output, but the greater the IO cost. In Python 3.8 and earlier, stdout and stderr had the same buffering granularity, which was not very reasonable; after version 3.9, stderr has a smaller buffering granularity, meaning each write operation’s output is more timely than stdout. This difference makes stderr suitable for error and log information, ensuring that even if a program crashes or exits abnormally, this information has a higher priority than standard output.

In C++, standard error is unbuffered (see below), which is more aggressive, but I personally think this is more reasonable.

Fortunately, in Python, you can disable this buffering behavior using python -u or by setting the PYTHONUNBUFFERED environment variable, or directly control the output timing by operating sys.stdout.flush().

6 Behavior in Python Concurrent Environments

When using stdout and stderr in multithreaded or multiprocess environments, output may interleave or become chaotic because output from different threads or processes may interfere with each other when writing to a terminal or file. One way to solve this problem is to create separate output files for each thread or process, or use thread locks (thread locks) or process synchronization mechanisms (such as multiprocessing.Lock) to synchronize access to stdout or stderr.

7 Controlling stdout and stderr in Python

In complex applications, you may need more flexible control over the destination of output streams. Python provides several ways to achieve this:

• Redirecting stdout and stderr: You can redirect the standard output and error output of a Python program by changing the values of sys.stdout and sys.stderr. This is particularly useful for capturing and analyzing output, or redirecting output to non-standard output devices such as graphical interfaces.
• Using the subprocess module: When running external commands or scripts, the subprocess module allows you to control the stdout and stderr streams of the command, including redirecting them to variables within the Python program, or separating or merging them.
• Advanced applications of the logging module: Python’s logging module supports outputting logs to multiple destinations, including files, standard output, networks, etc. By configuring different log handlers, you can implement complex log management schemes, such as splitting logs to different outputs based on log level or message content.

7.1 Recommendations

• Manage output carefully: When designing software, clearly distinguish between output for user interaction (stdout) and output for error reporting or logging (stderr). This helps improve the usability and maintainability of the program.
• Optimize performance: Consider the performance impact of output operations, especially in scenarios with high-frequency logging or data output. Reasonable use of buffering and batch processing can reduce the impact on performance.
• Security considerations: Before outputting sensitive information, perform appropriate filtering and desensitization to avoid leaking sensitive data through logs.

By deeply understanding and flexibly applying stdout and stderr, you can build more robust and manageable Python applications, effectively handle logs and output, and improve user experience and application stability.

8 Buffering Behavior in C++

In C++, stdout (usually corresponding to std::cout) and stderr (corresponding to std::cerr) have different buffering strategies:

• std::cout is line-buffered by default, which means that when it is connected to a terminal, the output is flushed on each newline or when the buffer is full.
• std::cerr is unbuffered by default, so data written to std::cerr is immediately output, which is very useful for reporting error information because it reduces the risk of error information not being output due to program crashes.

9 Redirecting stdout and stderr

In a C++ program, stdout and stderr can be redirected in several ways. A common method is to use the freopen function to redirect standard output or error output to a file at runtime:

1
2

freopen("output.txt", "w", stdout);
freopen("error.log", "w", stderr);

This method can be used to redirect output to a file for later analysis and debugging.

10 Use in C++ Multithreaded Environments

When using std::cout and std::cerr in multithreaded C++ programs, race conditions may occur, leading to chaotic output. To avoid this, it is recommended to use mutex locks (such as std::mutex) to synchronize access to these streams:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#include <iostream>
#include <mutex>
#include <thread>

std::mutex cout_mutex;

void thread_function(int id) {
    std::lock_guard<std::mutex> lock(cout_mutex);
    std::cout << "Thread " << id << " is running\n";
}

int main() {
    std::thread t1(thread_function, 1);
    std::thread t2(thread_function, 2);

    t1.join();
    t2.join();

    return 0;
}

11 Controlling Output in C++

The C++ standard library provides std::streambuf, which can be used to implement finer control over std::cout and std::cerr, including redirection and custom buffering behavior. By inheriting from std::streambuf and overriding the corresponding member functions, you can create custom buffering strategies or redirect output to GUI components, network connections, etc.

11.1 Recommendations

• Use buffering wisely: Choose an appropriate buffering strategy based on the application scenario. For error information that requires immediate feedback, use std::cerr or manually flush std::cout.
• Avoid direct use of standard output in multithreading: Use mutex locks or other synchronization mechanisms to ensure the consistency and order of output.
• Use redirection and custom streambuf: To handle output more flexibly, consider using redirection or custom streambuf to meet special output needs, such as logging, network transmission, etc.

By mastering these advanced techniques, you can effectively manage and control program output while ensuring the robustness and flexibility of C++ programs.