Notes

Robust And Race-free Server Logging Using Named Pipes - The Art Of Machinery

Log files are a common tool in server administration. Log files can cause problems for servers that are not reliable. This is especially true when they need to be rotated or shipped (which is almost always). Race conditions can be caused by moving files around.



There are better ways. Named pipes allow for a simple and robust stack of logging, without any race conditions and without the need to patch your servers to support network logging protocols.



Problems with rotating log files



Let's begin by discussing the problems. Race conditions are a problem with file-based log logging setups. You can either rotate logs into archival storage or ship them to a remote stack of log processing. Logrotate is a popular tool, but I want to keep it simple.



Imagine you have an log file at /var/log/foo. It can get quite large, so you want to process it periodically and start over with a fresh, empty file. You (or your distro administrators) will set up logrotate, which has various rules about when to rotate each file.



Logrotate will by default rename the file to something similar to /var/log/foo.1. It will also create a new /var/log/foo where you can write to it. This works mostly for software that runs intermittently, such as a package manager that updates software. However, it won't work if the log file was generated by a long-running service. The filename is used only by the server to open the file. After that it writes to its open descriptor. It will continue to write to the filename /var/log/foo.1, while the new /var/log/foo.0 file will remain empty.



To handle this use-case, logrotate supports another mode: copytruncate. In this mode, instead of renaming, logrotate will copy the contents of /var/log/foo to an archival file, and then truncate the original file to zero length. As long as the server has the log file open in append mode, it will automatically write new logs to the start of the file, without needing to detect the truncation and do a file seek (the kernel handles that).



However, copytruncate mode creates an unacceptable race condition. Any log lines written after copy but before the Truncation will be destroyed. You will likely get the same race condition with default move and create mode. It's because it doesn't make sense to split logs into multiple files. Although most systems can compress an old log file, ultimately you will need to delete any uncompressed data. This race is not so common for occasional log writers like package managers. The delay flag to logrotate makes it less likely, but makes log handling a little more complicated.



Nginx and other servers support the modification of the default logrotate modes:



1. Rename the file 2. Create the new folder. (New step), inform the server that it is required to reopen the log file.



This works (as long the logs processor didn't delete an old file before the server had finished reopening), but you'll need special support from the server. There are many software programs out there and log file handling is not something that interests enough people to make it a priority. This approach also only works for long-running servers.



This is a good moment to stop and think about what it means to take a step back. It's inherently painful to have multiple processes juggle log file files around on disk. It causes bugs and makes logging stacks complicated (here's just one of many examples). Another option is to use a network protocol such as MQTT or networked Syslog. However, most servers won’t support this one. Log files are a great way to communicate with log writers.



It's okay because *nix, which means "everything can be a folder", allows us to quickly get a file interface for the writer and a streaming interface for the reader.



Named pipes 101



Perhaps you've seen pipes inside pipelines like this.



The pipe connecting sort to uniq is temporary, anonymous communication channel. Sort writes to it and uniq takes its readings. Named pipes are less common but they can also be used as communication channels. They are persistent on the filesystem like files, but that's the only difference.



Open a terminal and cd into a temporary directory. GAMING The following creates an anonymous pipe and uses cat for opening a writer.



Leave that cat command waiting, and open up another terminal in the same directory. Start your reader from this terminal



Now you can type in your thoughts into the writer's end and they will appear on the reader's end. Line buffering will be used by cat in interactive mode so that data is transferred each time you start a newline.



This works even if your cat doesn't know much about pipes. The pipe acts as a file if you just read or write to it. But if you check, you'll see the data isn't stored anywhere. You can pump gigabytes through a pipe without filling up any disk space. Once the data has been accessed once, it is lost. (Multiple readers are possible, but only one reader will have access to any buffer-load.



Another feature that makes pipes useful is their buffering or blocking. You can start writing before any readers open the pipe, and data gets temporarily buffered inside the kernel until a reader comes along. If the reader starts first, its read will block, waiting for data from the writer. The writer will also block if there is too much pipe buffer. If you run the two-terminal experiment again, this time with a regular document, you'll find that the reader cat will eagerly go through all the data and then exit.



A simple solution to an annoying problem



Maybe you're seeing how named pipes can help with logging: Servers can write to log "files" that are actually named pipes, and a logging stack can read log data directly from the named pipe without letting a single line fall onto the floor. You can do whatever with the logs. There is no need for any juggling of files on your disk.



One problem is that the writer doesn’t need a reader in order to start writing. However, if the reader opens the pipe and then closes the pipe, the writer will get a SIGPIPE ("broken pipes") which will kill it automatically. (Try killing the reader cat while typing things into the writer to see what I mean.) A reader can read the file without a writer. However, if a writer opens and closes the pipe, it will be treated as an end of file. The named pipe is persistent on disk but it is not a stable communication channel. Log writers and log readers can restart the pipe (as they would on a real server).



There's an alternative that is both strange and very simple. Multiple processes may open the pipe for reading or writing. The pipe will only close when all the readers or all the writers close it. A daemon is all that's needed to create a stable log pipe. It will hold the pipe open for reading and writing without actually reading or writing. I set this up on my personal server last year, and I wrote a tiny, zero-config program to act as my pipe-holding daemon. It opens all files in its current working directory. This is for both reading and writing. I run it from a directory which has symbolic links for every named pipe in the logging stack. The loop ends in a wait() and a SIGHUP. If I ever update the symlinks in the directory, I give the daemon a kill -HUP and it reopens them all. It could also monitor the directory, but the SIGHUP approach works reliably. The pipe buffer allows log writers and readers to be shut down and restarted independently at any time without interruption.



My server uses the supervision suite s6 to manage daemons. So I have s6–log reading from each log pipe. The bottom section of the s6 log documentation page provides valuable insights into the problems associated with popular logging systems and offers suggestions for better ways to go about it.



Imagine: a world without log rotation



Named pipes are not required for race-free logs processing. The s6 suite encourages writing logs to some file descriptor (like standard error), and letting the supervision suite make sure those file descriptors point to something useful. Named pipes offer a few additional benefits.



- It doesn't require any co-ordination between writer and reader - It integrates nicely with the software we have today - It gives things meaningful names (rather than /dev/fd/4)



I have seen companies spend as much on their logging stacks and infrastructure as they do on their serving infrastructure. Of course, log rotation and log shipping aren't the only problems to blame, but it feels so wrong that we've made logs so complicated. Consider if you really have to juggle log files around if your job involves any type of logging system. You could be making the world a better destination.


Here's my website: https://gametolia.com/