We were running an internal implementation of a syslog server in Go, receiving the logs of a bunch of machines in the same datacenter (i.e. internal network). That server was setup to run on boot, receive the logs on UDP port 514, do a bunch of things with them and eventually writing them down into /var/log/foo. Nothing complicated nor extraordinary.

We were running happily with that for a few years, until an OS upgrade came around the corner. Given that the service was doing nothing special, nobody believed it would cause any issues whatsoever. Building the package for the new OS was a no brainer, nor was building the machine’s image. Upgrade day comes and everything is running fine, except that /var/log/foo/foo.log is missing. Weird.

According to the service’s logs everything was running. Packages were flowing on that port, according to tcpdump and even the expected occasional CPU little spikes were there. Directory was still empty, though. Given that this service is written in Go, sending a mere ABRT signal caused it to dump useful stack trace to STDERR and terminate. Judging by the traces - the service was running fine, multiple goroutines, running fine and doing their job. Where are the logs, though?

Opening that foo.log file inside /var/log/foo is one of the first things that this program did, so one would even expect it to have that file opened as FD 3 and sure enough, judging by the opened file descriptors by this process, as reported by procfs , that’s precisely the case:

❯ ls -la /proc/1801/fd
lrwx------ 1 root root 64 Nov 20 09:07 0 -> /dev/null
l-wx------ 1 root root 64 Nov 20 09:07 1 -> 'pipe:[22087]'
l-wx------ 1 root root 64 Nov 20 09:07 2 -> 'pipe:[22087]'
l-wx------ 1 root root 64 Nov 20 09:05 3 -> /var/log/foo/foo.log
lrwx------ 1 root root 64 Nov 20 09:07 4 -> 'anon_inode:[eventpoll]'
lr-x------ 1 root root 64 Nov 20 09:07 5 -> 'pipe:[25179]'
l-wx------ 1 root root 64 Nov 20 09:07 6 -> 'pipe:[25179]'
lrwx------ 1 root root 64 Nov 20 09:07 7 -> 'socket:[24122]'
lrwx------ 1 root root 64 Nov 20 09:07 8 -> 'socket:[41476]'
lrwx------ 1 root root 64 Nov 20 09:07 9 -> 'socket:[41477]'
[..]

Okay, the file is opened properly, but no logs are ever written. Given that remote communication is involved here, it was easy to blame “The network”™. Our working hypothesis at point is just something getting stuck waiting for something to happen. Or probably writing down into the file failed? But where are the error messages about it? strace to the rescue:

❯ strace -p 1801 -f -s 4096 -e write -e fd=3
[pid  1802] write(3, "logline 1...\n", 232 <unfinished ...>
[pid  1801] write(3, "logline 2...\n", 322 <unfinished ...>
[pid  4902] write(3, "logline 3...\n", 581 <unfinished ...>
[pid  4902] write(3, "logline 4...\n", 273 <unfinished ...>
[pid  4902] write(3, "logline 5...\n", 149 <unfinished ...>
[pid  4903] write(3, "logline 6...\n", 485 <unfinished ...>
[..]

Let’s just break down what we’re seeing here, as strace(1) might not be that popular as I seem to believe, based on a few recent conversations outside of this scope:

• -p 1801 - intercept system calls performed by the process with PID 1801 (our server’s main process)
• -f - follow all threads of this process and child processes as well. This is a Go program after all, surely there are multiple threads and processes.
• -s 4096 - we are hunting for log messages, so dump strings up to 4096 bytes (instead of the default 32)
• -e write - trace only write(2) syscalls
• -e fd=3 - trace only syscalls that operate on file descriptor 3, which we know is our empty file, as seen above

I’ve truncated the output and log messages here, but there were pages of those. We are clearly observing log messages being written precisely into that file, from multiple goroutines without a failure, judging by the positive results of the syscalls. Where do they go? Why is the file missing? To make things even more annoying - restarting the service made the issue go away. It’s only observed on first boot. Once everybody’s tools for inspection are in place, things just worked.

Code works, service is available, packets fly, logs are being written. Just… files are missing. Assuming the code is fine. Let’s observe the environment deeper. procfs has way more to tell here.

Could the processes have another idea what is /var/log/foo from the rest of the world? This is what namespaces do in the linux world to begin with. Looking into /proc/{PID}/ns completely disproved that:

❯ ls -l /proc/1801/ns/mnt
lrwxrwxrwx 1 root root 0 Nov 20 09:16 /proc/1684/ns/mnt -> 'mnt:[4026531841]'
❯ ls -l /proc/$$/ns/mnt
lrwxrwxrwx 1 root root 0 Nov 20 09:16 /proc/6561/ns/mnt -> 'mnt:[4026531841]'

$$ is the PID of the current process of the shell. Clearly it sees the same mount namespace as our service. But how come does the shell see different contents? However, we know that there is disk mounted into /var/log/foo, let’s see the proc’s mountinfo :

❯ cat /proc/1801/mountinfo | grep /var/log
516 66 8:16 / /var/log/foo rw,relatime shared:192 - ext4 /dev/sdb rw
❯ cat /proc/$$/mountinfo | grep /var/log
516 66 8:16 / /var/log/foo rw,relatime shared:192 - ext4 /dev/sdb rw

Okay, we both see the same mount with ID 516, mounted over / which ID is apparently 66. How come only the process see that ghost file that nobody else is observing? Let’s look into that file a bit more. Apart from /proc/{PID}/fd which merely lists file descriptors and helps distinguish their type and location, there’s also /proc/{PID}/fdinfo which gives detailed information about each file descriptor:

❯ cat /proc/1801/fdinfo/3
pos:    218105
flags:  02102001
mnt_id: 66
ino:    132368

Eureka! mnt_id: 66. Remember that this was /, according to mountinfo ? How on earth is that file in the root filesystem, given that there is something else mounted in there, you might ask? Well, as I said - this only happens on first boot. If the service is restarted, everything runs smooth again. Apparently, this service starts way too soon, even before the mount is finished.

And sure enough. It indeed starts too soon… by a single damn second:

❯ journalctl -b | grep '/var/log/foo'
Nov 20 08:28:31 systemd[1]: var-log-foo.mount: Directory /var/log/foo to mount over is not empty, mounting anyway.
Nov 20 08:28:31 systemd[1]: Mounting /var/log/foo...
Nov 20 08:28:34 systemd[1]: Mounted /var/log/foo.

❯ ps -p 1801 -o lstart
Wed Nov 20 08:28:30 2024:    132368

What are the odds of this!?

… you might ask. Well, given that this machine boots from an image, it was happening every single time, we’ve tried. How come it never happened before - I don’t know. Different kernel, different systemd (where backwards compatibility does not mean a lot to begin with), different… who knows what.

The point is to know your tools and running environment well enough to be able to dig into this quick enough, regardless of what time it is.

Good programmers know what they’re doing. Great programmers understand one level deeper.

os.Stat returns fs.FileInfo which is an interface with six methods:

type FileInfo interface {
	Name() string       // base name of the file
	Size() int64        // length in bytes for regular files; [..]
	Mode() FileMode     // file mode bits
	ModTime() time.Time // modification time
	IsDir() bool        // abbreviation for Mode().IsDir()
	Sys() any           // underlying data source (can return nil)
}

Time of last modification (mtime) is the only time reported by this interface, though. There is no time of last access (atime), creation (btime), nor status change (ctime).

Sys()

Did you notice that Sys() method there? Yeah, me neither. This poorly documented method returns raw data for that file, as returned by the operating system. That means the underlying type of the result will be different per-OS. In our case, it’s a software intended to run exclusively on linux, so let’s see what happens there.

Well, in Linux, it returns a pointer to syscall.Stat_t :

type Stat_t struct {
	Dev       uint64
	Ino       uint64
	Nlink     uint64
	Mode      uint32
	Uid       uint32
	Gid       uint32
	X__pad0   int32
	Rdev      uint64
	Size      int64
	Blksize   int64
	Blocks    int64
	Atim      Timespec
	Mtim      Timespec
	Ctim      Timespec
	X__unused [3]int64
}

Great, we have atime, mtime and ctime. We’ve almost declared a win, until realizing that ctime is not creation time. It’s the time of last status change, which means it records any change of file’s metadata. What we seek is creation time a.k.a. birth time. Yep, I’ve just used almost 300 words to tell you that with os.Stat you can NOT get the creation time of a file.

statx(2)

To be fair, you would not be able to get the creation time using stat in C, either. Not to mention that most of the filesystems do not keep record of it. That was surprising to me - fat32, ext2, ext3 nor reiserfs store when a file has been created. Naturally, no such information is exposed by stat(2). Modern filesystems do, though.

After years of planning a new system call is finally merged into the Linux kernel and released with 4.11 in 2017. That’s years after the first released version of Go and five after 1.0. That explains why the standard library does not provide a straightforward way to obtain btime.

golang.org/x/sys/unix

The standard library might be a bit conservative in adding new things (although things are starting to move now with the new math/rand/v2 ), but that does not mean the Go team is sleeping. The golang.org/x/ packages are outside the main Go tree and are under way looser constraints for backwards compatibility. One of these packages is golang.org/x/sys/unix . The sys part is described as “packages for making system calls”, so do not expect the easiest to use API you’ve ever seen.

Sure enough, there’s a wrapper function of the statx syscall:

func Statx(dirfd int, path string, flags int, mask int, stat *Statx_t) (err error)

As expected the API is pretty much the same as with the syscall it wraps:

int statx(int dirfd, const char *restrict pathname, int flags,
          unsigned int mask, struct statx *restrict statxbuf);

• dirfd is a file descriptor identifying the directory to search from if provided path is relative (if is AT_FDCWD the call operates on the current working directory).
• path is obviously the filename we’re querying. Unless when empty and AT_EMPTY_PATH is in mask, in that case file/directory pointed to by dirfd is what we’re querying.
• flags can be used to influence a pathname-based lookup and what sort of synchronization the kernel will do when querying. When in doubt, use AT_STATX_SYNC_AS_STAT.
• mask is used to tell which fields we are interested in, so that the kernel does not bother populating the rest. The one we care about to is STATX_BTIME. If it’s there (or STATX_ALL which pretty much includes everything), then the creation time will be there.
• stat is being populated be the syscall when the returned error is nil.

All of those SCREAMING_SNAKE_CASE flags listed above are exposed as constants in the same package.

unix.Statx_t is a fairly big struct (matching the statx struct obviously) that has all of the timestamps we seek:

type Statx_t struct {
	// ...
	Atime            StatxTimestamp
	Btime            StatxTimestamp
	Ctime            StatxTimestamp
	Mtime            StatxTimestamp
    //..
}

StatxTimestamp has Sec and Nsec fields which is enough to create a time.Time value using time.Unix(). So, the whole thing might look something like this:

var statx unix.Statx_t
err := unix.Statx(unix.AT_FDCWD, name, unix.AT_STATX_SYNC_AS_STAT, unix.STATX_BTIME, &statx)
if err != nil {
    log.Fatal(err)
}

btime := time.Unix(statx.Btime.Sec, int64(statx.Btime.Nsec))
fmt.Printf("%s is created at %v\n", name, btime)

Easy, right?

Don’t agree? Yeah, me neither. A saner person would just npm the problem and look for a package exposing file times in a proper, easy-to-use way. I found this, which looks legit (only checked for GOOS=linux): github.com/djherbis/times .

I mean it. A third-party package with a higher-level portable API is the way to go in the real world. It’s good to know what’s happening underneath, but please beware of NIH syndrome.

Achieving the same result in Python isn’t much easier. The result from os.stat might have a st_birthtime on some operating systems (e.g., FreeBSD or Windows), but not on Linux. Sometimes seemingly trivial problems are tedious to solve because of weird historical reasons. And sometimes, a deeper dive is necessary, but practical solutions are often just a package away.

As I was writing my tip about inotifywait(1) , decided to show how different editors behave when saving a file. “I use vim btw”™, so it didn’t take long to try it in there.

Executed vim foo +wq and this is what inotifywait(1) printed:

/path/to/watched/dir/ CLOSE_WRITE,CLOSE 4913
/path/to/watched/dir/ DELETE 4913
/path/to/watched/dir/ MOVED_FROM foo
/path/to/watched/dir/ CLOSE_WRITE,CLOSE foo

That’s weird. It creates a file named 4913 and then moves it to the intended location. Why 4913, though? Tried it several times on multiple machines - always that same number. You know how when it comes to some vim mystery, everybody always tells you to use :help? Yeah, no luck there, this time.

git clone https://github.com/vim/vim.git && rg 4913 vim

Two meaningful results.

vim/runtime/doc/version7.txt: On MS-Windows sometimes files with number 4913 or higher are left behind.

😅

Fine. One meaningful result :

// Check if we can create a file and set the owner/group to
// the ones from the original file.
// First find a file name that doesn't exist yet (use some
// arbitrary numbers).
STRCPY(IObuff, fname);
fd = -1;
for (i = 4913; ; i += 123)
{
    sprintf((char *)gettail(IObuff), "%d", i);
    if (mch_lstat((char *)IObuff, &st) < 0)
    {
    fd = mch_open((char *)IObuff,
            O_CREAT|O_WRONLY|O_EXCL|O_NOFOLLOW, perm);
    if (fd < 0 && errno == EEXIST)
        // If the same file name is created by another
        // process between lstat() and open(), find another
        // name.
        continue;
    break;
    }
}
if (fd < 0)	// can't write in directory
    backup_copy = TRUE;
else [..]

This is part of a (haunting) function bigger than 6000 lines called int buf_write. This particular block is only accessible when :set backup and the user is trying to save changes in a file that already exists. So, the idea is fairly simple: make sure to write the changes in any file whatsoever inside the target directory in order to move it over the target file, so that the latter will be updated atomically. No one would ever observe partial change of state.

But why 4913? Reminded me of a meme about that random function which always returns 4, because it was a result of a perfectly fair dice throw. No idea, it appeared in the 7th commit since the migration to git for version 7. If anybody knows, please reach out.

The purpose of adding 123 until the end of time is clear, though. Pick “a random” non-existent file to the target directory. Choosing 123 in particular is there just because Bram had to write some number, probably. Doing this indefinitely is not really a bad idea in this case. I mean, who in their right mind will create thousands of files with names that are only numbers starting from 4913 and then adding 123?

$ time vim files/foo +wq

real    0m0.035s
user    0m0.015s
sys     0m0.012s

$ touch files/4913 && time vim files/foo +wq

real    0m0.034s
user    0m0.021s
sys     0m0.004s

CLOSE_WRITE,CLOSE 4913
CLOSE_WRITE,CLOSE 5036
DELETE 5036
MOVED_FROM foo
CLOSE_WRITE,CLOSE foo

$ n=4913
$ for i in {1..4096}; do n=$(($n+123)); touch files/$n; done
$ time vim files/foo +wq

real    0m0.041s
user    0m0.021s
sys     0m0.011s

$ for i in {1..50000}; do n=$(($n+123)); touch files/$n; done
$ time vim files/foo +wq

real    0m0.134s
user    0m0.020s
sys     0m0.098s

Having thousands files in a single directory is something you never want in the real world anyways. There are filesystems with a hard limit by design. While modern filesystems don’t, you will surely experience performance issues when working with such a directory.

Y tho?

Why am I telling you all this? Because I like digging deeper into such quirky details. The next time you stumble upon something peculiar, remember: there might just be an amusing or enlightening story waiting to be uncovered. Or maybe not, but you’ll learn something anyway.

Turns out you can have blocks of code inside a switch statement spilling between cases. Let me show you a simplified example:

void f(int a, int b)
{
	switch(a) {
	case 1:
		if (b > 0) {
	case 2:
			printf("a=%d b=%d\n", a, b);
		}
	}
}

No way!

Yep. That works. The if is taken into account only when a is 1, thus it prints only if b is greater than 0. However, if a is 2, the function always prints the values of a and b.

I’m not really sure whether this behavior has been allowed on purpose. It was more likely never thought of given that as stated in K&R (page 53):

[..] cases serve just as labels, after the code for one case is done, execution falls through to the next unless you take explicit action to escape

And really, they’re just goto labels. In C switch is just a syntax sugar for a branch table . No wonder that the condition should always be an integer. The execution should jump to the address of the given index (i.e. case condition) in the table. Once jumped, it always moves on and falls through cases.

Y tho?

When is it a good idea to do that, you ask? Never. Please, don’t. I cannot think of a single case (pun not intended) where this would make the code more readable or less error-prone. Using this approach would be objectively worse in every aspect. Besides, this doesn’t even work in “modern” C-style languages, where switch statements are not that simple, thankfully.

Why am I showing you this, then? Well, I find it hilarious and had a hard time explaining it to a friend with words. It’s nowhere trivial to find an example, either. But now I have it written and will be able to explain that insanity better while in a bar. I’m fun at parties.

❯ go doc context WithValue
package context // import "context"

func WithValue(parent Context, key, val any) Context
    WithValue returns a copy of parent in which the value associated with key is
    val.

    Use context Values only for request-scoped data that transits processes and
    APIs, not for passing optional parameters to functions.

    The provided key must be comparable and should not be of type string or any
    other built-in type to avoid collisions between packages using context.
    Users of WithValue should define their own types for keys. To avoid
    allocating when assigning to an interface{}, context keys often have
    concrete type struct{}. Alternatively, exported context key variables'
    static type should be a pointer or interface.

This whole article exists because of this:

Users of WithValue should define their own types for keys.

OK, but what would that type be? It should NOT be a string, but its underlying type could be one. In fact, over the years, I’ve used an internal string type with a value equal to the full package name:

// ctxt is a dummy type for storing context values.
//
// For more information see the documentation of context.WithValue.
type ctxt string

var ctxkey ctxt = "github.com/org/project/pkg"

Now &ctxkey is what we use as context key as suggested by the documentation in order to avoid useless allocations. Often I would introduce public functions to take or store their expected value, without dealing with keys, mostly for testing. Suppose all this package needs a ctx type for is to store values of type T:

// FromContext extracts a value of type T from the given context, returning its
// zero value if not present.
func FromContext(ctx context.Context) T {
	v, ok := ctx.Value(&ctxkey).(T)
	if !ok {
		v = NewT()
	}
	return v
}

// WithValue returns a copy of parent with stored T inside.
func WithValue(parent context.Context, t T) context.Context {
	return context.WithValue(parent, &ctxkey, t)
}

pkg.FromContext(ctx) takes a context and returns the pkg.T stored inside, if any, for instance.

The point to store the full package name is merely to ease debugging and/or tracing. This string is allocated only once, and we are using a pointer to it, in order not to allocate once per assigning that key to interface{}. If we want to get fanatical about that, the type of ctxt might be an empty struct:

// ctxkey is a dummy type for storing context values.
//
// For more information see the documentation of context.WithValue.
type ctxkey struct{}

Now we no longer need to use a pointer to ctxkey and just directly pass ctxkey{} instead, because a value of struct{} never allocates as its size is 0 anyways. When debugging you would still see the type of ctxkey, which (as with any type) contains the full path to it.

I have no memory which tool/use-case didn’t work well with struct{} so I’ve started using a string in the first place, but I’m back to using struct{} again these days.

What about generics?

Yes, of course. Go has generics now for quite some time, why not sprinkling some of it to solve that? We could. In fact, there is a proposal that is placed on hold (at the moment of writing this) in order to give the community time to play more with generics: https://github.com/golang/go/issues/49189

If you have tens of packages storing stuff in contexts and are pretty sure that this makes sense, might as well steal the proposed Key[T] from there.

This is part of the talk “UNIX archeology” I gave at OpenFest 2023.

If you, dear reader, are a vim user, you’ve probably seen this keyboard:

It explains a lot about our beloved text-editor. CTRL is just where it has to be in the first place, the Esc is so near and the : symbol we use for commands is not hidden behind a modifier like nowadays. But why the arrows?

So, one might say: “Well, because Bill Joy created vi using this ADM-3A terminal where the arrows just happened to be located there”. And that person would be absolutely right. But why were they located there?

To be fair, if you look at other terminals from these days, where to put the arrows was not really standardized. Not even the triangular shape, we’re so used with today. Most machines would put them in one line, but in different order. For instance, some would go with [↓] [↑] [←] [→]. So far, the answer that they just had to put them somewhere still holds.

ASCII

It was not random, though. You see, in ASCII characters are not just put at some point on random. There are fairly good decisions taken there that go mostly unnoticed today as people no longer work directly with ones and zeroes really. If you want to get the code for an upper-case letter you should add 64 to the position of that letter in the alphabet. If you want it to be lower-case, add 96 instead. That’s pretty clever, because it all boils down to flipping a bit:

This first bit is always 0, as ASCII is a 7-bit character set. If the second bit is up, we are dealing with a letter. If the third is up, then this letter is a lower-case. It’s that simple. But look what happens if we set both of those 2nd and 3rd bits to zero:

When pressing any of those four keys, the terminal has to take two decisions:

• If [Shift] is pressed, set the 3rd bit to 0
• If [CTRL] is pressed, set the 2nd and 3rd bits to 0

But wait, there’s more

Legacy we still observe from this terminal doesn’t end here, though.

~/

Take a look at the top right button. Yep. That’s why we are shortening the path to user’s home folder as ~/.

Regular expressions

If you ever wondered why ^ is the beginning of line or “home” position in regular expression dialects, well… now you know :)

This is part of the talk “UNIX archeology” I gave at OpenFest 2023.

In modern contexts, /usr is essential for housing the majority of user utilities and applications. This includes system-wide software and files like standard programs, libraries, documentation, and much more. Unlike the root filesystem, which contains the minimum necessary for booting and repairing the system, /usr provides the bulk of the operating system’s functionality.

However, have you ever thought why do we have seemingly duplicated with / directories in there? For instance, there is /bin and /usr/bin, /lib and /usr/lib, and lately we’ve started symlinking one to another. Surely, we could’ve just avoided the need to have them both in the first place. But why are they there?

The Filesystem Hierarchy Standard (FHS) is a reference describing the conventions used for the layout of Unix-like systems. Latest release defines its purpose as:

/usr is the second major section of the filesystem. /usr is shareable, read-only data. That means that /usr should be shareable between various FHS-compliant hosts and must not be written to. Any information that is host-specific or varies with time is stored elsewhere.

Large software packages must not use a direct subdirectory under the /usr hierarchy.

Actually, we only found out that we could mount /usr in read-only mode and even re-use it between a bunch of machines, decades after /usr was part of Unix.

no space left on device

Unix was created on a PDP-7 machine in 1969 and later migrated (which included rewriting a bunch of things, but that’s a story for another time) to the much more powerful PDP-11 , which had one RK05 disk pack (1.5MB):

Arnold Reinhold | CC-BY-SA

That was all. The entire / of Unix was on one of those drives, less than 1.5MB. Eventually it grew, with its userbase, and they needed another one. It was mounted into /usr in order to contain user home directories. There were /usr/ken, /usr/dmr, /usr/bwk, etc. However, multiple users were writing useful tools which they wanted to share and thus /usr/bin has been created. Eventually man pages started getting into shape, fonts started being interesting, people experimented with graphics and they all meant to be shared. The law was that nothing on that drive should be crucial for the machine to boot. For instance, you could not place the mount command in there, because something had to mount /usr in the first place. This limitation no longer exists thanks to initrd and initramfs.

Eventually, this drive grew bigger as well and they needed a 3rd one. It was mounted into /home and all of the home directories have been moved there, leaving /usr only for shareable files. That meant whole 3MB solely for the operating system. Hell, yeah!

And then we just kinda gone with it. Once Unix was made (not quite) generally available and the idea to split the OS into multiple drives was still legit. This was no longer the case only decades later. Why going through a revolution for that? Instead, people just made up rules and conventions on how to deal with /usr.

This is part of the talk “UNIX archeology” I gave at OpenFest 2023.

Hidden files are fairly popular these days, especially in our home folders. We store configuration, cache, logs and whatnot in there. Here’s what’s going on in the home folder on the machine I’m writting this right now:

❯ find ~ -name '.*' -type d | wc -l
5768
❯ find ~ -name '.*' -type f | wc -l
30573

But why?

Suppose you’re working on this new cool OS and receive it as a feature request: “We should introduce hidden files”. How would you do it? Knowing that files generally need attributes, why not create a new one for it? Exactly what Microsoft has one for their cool OS:

But that is not how we do that in Unix. Instead, soon after the beginning of time, in Unix hidden files are those which name starts with a dot. Looks odd when you think about it, trying to ignore the fact you’re so used with them. But why?

It’s not a bug, it’s a feature!

Rob Pike wrote an explanation on Google+. Even though Google killed that service long ago, The Wayback Machine has a record of that :

A lesson in shortcuts.

Long ago, as the design of the Unix file system was being worked out, the entries . and .. appeared, to make navigation easier. I’m not sure but I believe .. went in during the Version 2 rewrite, when the file system became hierarchical (it had a very different structure early on). When one typed ls, however, these files appeared, so either Ken or Dennis added a simple test to the program. It was in assembler then, but the code in question was equivalent to something like this:

if (name[0] == '.') continue;

This statement was a little shorter than what it should have been, which is

if (strcmp(name, ".") == 0 || strcmp(name, "..") == 0) continue;

but hey, it was easy.

Two things resulted.

First, a bad precedent was set. A lot of other lazy programmers introduced bugs by making the same simplification. Actual files beginning with periods are often skipped when they should be counted.

Second, and much worse, the idea of a “hidden” or “dot” file was created. As a consequence, more lazy programmers started dropping files into everyone’s home directory. I don’t have all that much stuff installed on the machine I’m using to type this, but my home directory has about a hundred dot files and I don’t even know what most of them are or whether they’re still needed. Every file name evaluation that goes through my home directory is slowed down by this accumulated sludge.

I’m pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.

How many bugs and wasted CPU cycles and instances of human frustration (not to mention bad design) have resulted from that one small shortcut about 40 years ago?

Keep that in mind next time you want to cut a corner in your code.

(For those who object that dot files serve a purpose, I don’t dispute that but counter that it’s the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.)

This is part of the talk “UNIX archeology” I gave at OpenFest 2023.

Have you ever thought why the 79 characters limit per line is so popular in programming and pretty much nowhere else? Python with their notable PEP-0008, the Linux kernel, PostgreSQL and the GNU project, just to name a few. Sounds like an arbitrary one. But why?

tty

People tend to believe that this is because terminals were able to print up to 80 symbols per line and this just got carried over. The most popular one being VT100:

Jason Scott | CC BY 2.0

To be fair, not all of them did, but VT100 is iconic enough to inspire such a tradition. It was able to operate in two modes - 80 columns by 24 rows or 132 columns by 14 rows. So, we might as well agree with that and call it a night. But why?

IBM 80-column punched card

Okay, so seems like VT100 and some of its predecessor were not the first ones sticking with 80 symbols. Here’s the IBM 12-row/80-column punched card format which came to dominate the industry:

Pete Birkinshaw | CC BY 2.0

This makes sense. The EOL is an actual characters and this is how we’ve ended up with 79 characters limit, depending on whether you’re counting the EOL as one. So, this makes sense to be the initial reason for that limitation - the most popular punched cards were limited to 80 symbols. But why?

Initially, punched cards allowed far less characters per-line in order not to tear the card with too many consecutive holes. In fact, way before that they weren’t used to store lines of text in the first place. The only thing that never changed was the size of those cards.

Hollerith tabulating machine

Adam Schuster | CC BY 2.0

The Hollerith Tabulating Machine, an ingenious invention of the late 19th century, marked a pivotal moment in the history of data processing. Conceived by Herman Hollerith, this revolutionary device was inspired by the challenges of efficiently processing vast amounts of data from the 1890 U.S. Census. The machine’s core innovation lay in its use of punched cards, a concept borrowed from the Jacquard loom, to store and sort data.

Each punched card represented a unique data point, with various patterns of holes encoding specific information. When fed into the Hollerith machine, a series of mechanical and electrical systems interpreted these holes, tabulating and summarizing data with unprecedented speed and accuracy.

Yep, U.S. Census. Each card did NOT encode text, but the details for each counted person. Here’s what it looked like:

If you wonder what any of this means, early computers have a great article about 1890 Census: The Meaning of the Hollerith Card Codes . Some of the entries here might not be approved by today’s standards. So, the cards for that machine were that big and every single machine dealing with punched cards, kind of just went with it. But why were the cards for that machines so big?

U.S. dollar bills. See those boxes next to the machine? They were used to transport bills which before 1920 were kind of huge. This way, you don’t get to manufacture new type of boxes and new type of paper cutters for those new special cards.

So, the next time somebody asks you why they should abide to that 79-characters per-line rule, you can tell them the old U.S. dollar bills are to blame. That’s why!

❯ git push -u origin baba
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 181 bytes | 181.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a pull request for 'baba' on GitHub by visiting:
remote:      https://github.com/user/project_name/pull/new/baba
remote:
To github.com:user/project_name.git
 * [new branch]      baba -> baba

A few years ago Rado Stankov wrote a great post about a few git aliases and scripts he uses to automate things: Git Tricks . I took literally all of the tricks he shows there in my dotfiles and strongly suggest for you to do the same.

One of tricks there is a git-pull-request scripts which allows automating that clicking on the provided link with a git pull-request command. Over time however, I’ve started using GitLab and this trick no longer works for that, as it assumes you’re pushing into a GitHub repository. So, I had to modify that script in order to be agnostic of the hosting you’re using and simply open whatever URL the upstream provdes on push (which to be fair, might be a bit risky, if you’re pushing to a shady git hosting, so use it with care):

#!/bin/sh
current_branch=$(git rev-parse --abbrev-ref HEAD)
git push -v --set-upstream origin "$current_branch" $2 2>&1 | while IFS= read -r line
do
	echo "$line"
	url=$(echo "$line" | grep "remote:" | grep -Eo '(http|https)://[^ ]+')
	if [ -n "$url" ]
	then
		xdg-open "$url" > /dev/null
	fi
done

Note that as a linux user, I’ve changed open to xdg-open. If you’re using macOS, might what to do s/xdg-open/open/ over this.

If this is saved in a file named git-pull-request with execute permissions (+x) at some of the directories listed in your $PATH, running git pull-request will push your current branch upstream and output exactly the same as above, but also open the provided link in your default web browser (presumably in a new tab).

Say we’re working on a system for publishing articles, allowing discussion below each article, with the following requirements:

• Moderators can edit all articles and comments
• Authors can create articles and edit only those created by themselves
• Users can read articles and post comments about them
• Any user-type can edit and delete their own profile

We could define different user-levels as a bit-mask and store that alongside user’s information (e.g. users table in the database):

const (
	user = 1 << iota    // 0001
	author              // 0010
	moderator           // 0100
	self                // 1000
)

Each of the actions listed above can now be expressed as a list of user types allowed to perform them, by using the bitwise OR operator, omitting actions allowed by everybody even without an account:

const (
	CreateArticle = author
	UpdateArticle = self | moderator
	DeleteArticle = self | moderator

	CreateComment = user | author | moderator
	UpdateComment = moderator
	DeleteComment = moderator

	UpdateUser = self
	DeleteUser = self
)

Each action has only the bits of every user type set to 1. For instance CreateArticle is 2 (0010), UpdateArticle is 12 (1100) and CreateComment is 7 (0111). We could now create a function which determines whether given action is allowed by simply checking if they share at least one bit set to 1, for which we’re going to utilize the bitwise AND operator.

self is a bit tricky here, because it must be populated differently per-entity. So we should make sure that both users, articles and comments have ID:

type User struct {
	ID   ID
	Role int
	Name string
	// ...
}

type Article struct {
	ID     ID
	Author ID
	Name   string
	// ...
}

type Comment struct {
	ID      ID
	Author  ID
	Article ID
	// ...
}

The ID type here can be anything comparable (e.g. int, string, UUID). Now the function in question is painfully obvious:

func Allowed(u User, action int, target ID) bool {
	if user.ID == target {
		u.Role |= self
	}

	return u.Role&action != 0
}

target is the ID of whatever is being acted upon or left with the zero value for that type when not applicable (e.g. on any Create… action). In order to check whether a user (u) can update given article (a) we call Allowed(u, UpdateArticle, a.Author) and handle (e.g. return error) when the result is false.

However…

You’ve probably already figured out that this mimics the way Unix operating systems encode filesystem access. I could think of a few pros to this approach:

• The check whether given action is allowed is trivial
• Adding new user type is a single line of code. Each new user adds exactly one bit to the mask, so until we go above 64, everything’s fine
• Permissions are obvious even to non-technical people
• Same is applicable to pretty much any programming language I can think of

However, this does not come without gotchas. Take that self which is not really a user type, but a made-up concept to allow everybody to edit some of their own creations. We had to add it, because it’s not a desired capability for all entities. Comments for instance can NOT be edited, once posted. Not only that, we had to add a new argument and logic to our little pure function.

One could think of a few new requirements that would seem simple, yet require more arguments, more checks and eventually a complete overhaul of this whole idea. Technically speaking, even the Unix permissions turned out to be not enough and modern Unix-like operating systems implement more complex ACLs stored in the extended file attributes on top of it.

It sounds strange, right? Why two? Well, I’ve started with a mandatory Friday deploy each week. That turned out to be unpractical, though. You can’t always plan for changes to be ready for merging precisely on Fridays.

Checklist

On a more serious note. Let’s go through what issues people tend to have with deploying on Friday:

Most developers are already in weekend mood, they’re not that focused thus it’s more likely to make a mistake while deploying. If developers are manually deploying instead of a CI/CD, you have a problem to fix. Developers shouldn’t have to do anything more than pushing a button to initiate a deploy (preferably the Enter button on a git push command).

There might be an ugly bug. Do you have tests, do you do code reviews or probably have a QA team? If not, think about it. If yes, then you know that there are still bugs that slip through. Just rollback.

But rolling back is hard, it takes time and people are already in that weekend mood, remember? That means your rollback process is more than a push of a button away. Go fix that.

Spotting bugs is harder on weekends. Why? Do you have people manually staring at logs and graphs? Setup a proper monitoring system. I guess your product doesn’t close doors on weekends, so you need it anyways. If it does, though… well… you got me.

But, but… many thing could go wrong you can’t even imagine! Yes, that’s most probably right. What difference does it make if you deploy it on Wednesday? It’s still good to be reachable a few hours after deploy especially if there are no people on-call 24/7, anyways.

TL;DR

Aim to deploy on Friday (preferably in the afternoon) every once in a while. That will force you to set up solid deployment and monitoring procedures (and habits in that matter) you really trust in.

Clicking on code in the fifth split left to write, actually got registered as a click in the third one. Started to experiment and realized that it registered properly on the 222nd column, but any other column after that got also registered there. Weird. Something happens on column 223?

This is what my vim configuration looked like, when it comes to mouse support:

" Mouse support
set mouse=a
if !has('nvim')
    set ttymouse=xterm2
endif

No idea how did I ever got to that ttymouse=xterm2. Probably copied it from somebody else’s vimfiles. Anyways, apparently mouse support gets activated with set mouse=a. Let’s go into mouse’s help and look for that magical number 223. Open vim, call :help mouse and search for 223.

… and voilà! First result is precisely about that xterm2 value of ttymouse:

     xterm        xterm-like mouse handling.  The mouse generates
                  "<Esc>[Mscr", where "scr" is three bytes:
                          "s"  = button state
                          "c"  = column plus 33
                          "r"  = row plus 33
                  This only works up to 223 columns!  See "dec",
                  "urxvt", and "sgr" for solutions.

So, I get to choose between dec, urxvt and sgr. After a bit of digging, it looks like sgr is the most modern one, supported since 7.3.632. Here’s the patch if you’re into this: http://ftp.vim.org/pub/vim/patches/7.3/7.3.632

So, this is how my “mouse setup” looks like now:

" Mouse support
set mouse=a
if !has('nvim')
    if has("mouse_sgr")
        set ttymouse=sgr
    else
        set ttymouse=xterm2
    endif
endif

The mouse support is now working great well enough, considering we’re talking about a terminal application here.

Y tho?

Why am I telling you all this? It’s a clear example of how one can debug and fix an issue in vim simply by browsing :help. Not to mention that I had two different ways to arrive at the same help page. Notice that if !has('nvim') there? I didn’t even bother to check if the same reproduces in neovim (spoiler alert: it does not). Instead of going through trial and error for that magical number 223, I could’ve just tried in neovim and then go directly into reading :help ttymouse.

One can learn a lot about vim, simply by wandering in there. Apparently having a good documentation helps your users. Modern software tend to overlook that. I tend to overlook it into the software I write and maintain. The new way is writing tutorials and endless README.md pages in the project’s root. We might want to reconsider this.

Later, pulled myself together, decided on a theme with solid card-like profile homepage and zero content in there and deployed it. Then, abandoned the idea again. A few months later pulled together again, bought another domain, started drafting a bunch of posts for the past few months and got into the state “as soon as I have 10 - we go live”.

That meant, getting to work on writing. So, naturally, I’ve spend some days in choosing a new theme, again and now I’m pretty sure this should go live before the end of 2023. If you’re reading this, looks like I’ve made it. The only thing I’ve been actually consistent enough is that I want to write on strictly tech-related topics and there should be a tips/ section, because I tend to forget a lot.

SQLite is a great database (although often overlooked) with a bit of annoying CLI defaults.

For instance:

sqlite> select * from Album limit 10;
1|For Those About To Rock We Salute You|1
2|Balls to the Wall|2
3|Restless and Wild|2
4|Let There Be Rock|1
5|Big Ones|3
6|Jagged Little Pill|4
7|Facelift|5
8|Warner 25 Anos|6
9|Plays Metallica By Four Cellos|7
10|Audioslave|8

Three columns, two of them are numbers, and yet things are hard to read. Write .mode box in your ~/.sqliterc and things change dramatically:

sqlite> select * from Album limit 10;
┌─────────┬───────────────────────────────────────┬──────────┐
│ AlbumId │                 Title                 │ ArtistId │
├─────────┼───────────────────────────────────────┼──────────┤
│ 1       │ For Those About To Rock We Salute You │ 1        │
│ 2       │ Balls to the Wall                     │ 2        │
│ 3       │ Restless and Wild                     │ 2        │
│ 4       │ Let There Be Rock                     │ 1        │
│ 5       │ Big Ones                              │ 3        │
│ 6       │ Jagged Little Pill                    │ 4        │
│ 7       │ Facelift                              │ 5        │
│ 8       │ Warner 25 Anos                        │ 6        │
│ 9       │ Plays Metallica By Four Cellos        │ 7        │
│ 10      │ Audioslave                            │ 8        │
└─────────┴───────────────────────────────────────┴──────────┘

My passion for technology ignited during my school years, when I wanted to learn how to “create a website”™. A bunch of tutorials, web forums (remember those?) and a few books later I began taking on freelance web development jobs.

I pursued my higher education at the Faculty of Mathematics and Informatics (FMI) at Sofia University St. Kliment Ohridski, specializing in Software Engineering. Although I embarked on a master’s degree in Artificial Intelligence, my interests shifted, leading me to focus more on practical software development rather than completing my master’s studies.

While in FMI teaching has also been a significant part of my journey, where I taught “Programming with Go” and “Programming with Python” . I also assisted in teaching courses “Operating Systems” and “Systems Programming”.

In my professional career, I’ve transitioned from web development to a more focused role in backend systems. Currently, I’m leading an infrastructure team at SiteGround , developing the platform that underpins most of our services.

The home page of this very site lists all the ways you could reach me. In case you prefer mails, just hit me on kiril＠vld.bg.

You can append .keys to the URL of your profile in order to receive the public keys you have on your account in a format to directly pipe it into ~/.ssh/authorized_keys:

curl https://github.com/<username>.keys | tee -a ~/.ssh/authorized_keys

Same goes for any GitLab instance.

Delete matched files

The most trivial one: find <path> [..] -delete (e.g. find . -name foo -delete).

However, depending on the OS -delete might not be available. In that case, see the next one.

Execute command with matched files

find <path> [..]  -exec <cmd> '{}' \;

is executed once for each matched file and the relative path to it is replaces '{}' and I always forget that \;, which might be omitted depending on a bunch of weird shell-isms (e.g. find . -name foo -exec rm -f '{}' \;).

If you want to call <cmd> exactly once (with caveats) with all the matched files passed as argument, add + at the end, but then '{}' should be right at the end of the command just before the +: find . -type f -name 'foo*' -exec rm -f '{}' +

Match files by modification time

find <path> [..] -mtime [+-]n

• n means modified precisely n days ago. This matches all files modified a week ago:
  find . -mtime 7

• +n means modified at least n days ago. This matches all files older than a week:
  find . -mtime +7

• -n means modified within the last n days. This matches all files older than a week:
  find . -mtime -7

By default it generates a bunch of passwords in columns. However with args it can be ustructed to generate one fairly long secret (64 characters):

pwgen 64 1

It has fairly good number of ways to configure what and how exactly to generate. The downside of this approach is that pwgen(1) is probably not installed by default on your machine.

/dev/urandom

Reading directly from /dev/urandom is the way to go on pretty much any UNIX these days (or fallback to /dev/random, although both are the same except on Linux) without having to install anything. The only question is how to make it output plain-text.

base64(1)

Arguably the easiest way if you don’t care that the only possible “special” characters besides letters and numbers are + and /:

base64 /dev/urandom | head -c64

tr(1)

If you need to define a strict alphabet to use for your secret:

cat /dev/urandom | tr -dc 'a-zA-Z0-9_!@#$%^&*()~+=-' | head -c64

openssl(1)

OpenSSL can also do the job with two possible outputs:

• hex: openssl rand -hex 42

• base64: openssl rand -base64 42 (uses = paddings)

The numeric argument is not characters, though.

gpg(1)

Because obviously GPG can do everything security related:

gpg --gen-random --armor 1 42

First number is “quality level” which is… complicated as with anything related to GPG anyways.

1. Prepare a single command that tries to reproduce the bug (e.g. script, test) off tree. Bisect is about to start jumping around, so it can’t be in the tree.

2. git bisect start: begin the bisect session.

3. git bisect bad: mark current HEAD as a bad commit (i.e. the bug is reproducible).

4. git bisect good <commit-ref>: some known commit in the past, where the bug does not exist.

5. You get dropped at a given commit. Try the command/test/script from step 1 and:

   • if the bug exists: git bisect bad
   • if the bug does NOT exist: git bisect good

6. Repeat until the initial bad commit is found.

7. git bisect reset.

git bisect run

Steps 5 and 6 can be automated with git bisect run if you can provide a script that exits with status code 0 when the chosen revision is good and something between 1 and 127 (except for 125) when the revision is bad.

The script should obviously be out of tree and executed with:

git bisect run <script_name> [<arg> ...].

nmap -sn 1.2.3.0/24

HOST DISCOVERY:
-sL: List Scan - simply list targets to scan
-sn: Ping Scan - disable port scan
-Pn: Treat all hosts as online -- skip host discovery
-PS/PA/PU/PY[portlist]: TCP SYN, TCP ACK, UDP or SCTP discovery to given ports
-PE/PP/PM: ICMP echo, timestamp, and netmask request discovery probes
-PO[protocol list]: IP Protocol Ping
-n/-R: Never do DNS resolution/Always resolve [default: sometimes]
--dns-servers <serv1[,serv2],...>: Specify custom DNS servers
--system-dns: Use OS's DNS resolver
--traceroute: Trace hop path to each host

Port Scanning

Quick TCP SYN scanning (-sS: requires root), skipping host discovery (-Pn), of most popular ports:

nmap -sS -Pn 1.2.3.4

Port ranges can be given with -p100-1000 (from :100 to :1000). All ports is -p-. Single port is also accepted: -p 22

UDP Scanning:

nmap -sU -Pn 1.2.3.4

SCAN TECHNIQUES:
 -sS/sT/sA/sW/sM: TCP SYN/Connect()/ACK/Window/Maimon scans
 -sU: UDP Scan
 -sN/sF/sX: TCP Null, FIN, and Xmas scans
 --scanflags <flags>: Customize TCP scan flags
 -sI <zombie host[:probeport]>: Idle scan
 -sY/sZ: SCTP INIT/COOKIE-ECHO scans
 -sO: IP protocol scan
 -b <FTP relay host>: FTP bounce scan

PORT SPECIFICATION AND SCAN ORDER:
 -p <port ranges>: Only scan specified ports
   Ex: -p22; -p1-65535; -p U:53,111,137,T:21-25,80,139,8080,S:9
 --exclude-ports <port ranges>: Exclude the specified ports from scanning
 -F: Fast mode - Scan fewer ports than the default scan
 -r: Scan ports sequentially - don't randomize
 --top-ports <number>: Scan <number> most common ports
 --port-ratio <ratio>: Scan ports more common than <ratio>

Misc:

• OS discover: -O (uppercase o, not a zero)
• Application discovery: -sV
• Everything: -A (traceroute, app discovery, OS detecetion, port scanning…)

Note the trailing slash in source path: this avoids creating [host:]/destination/path/path.

• -a: archive (i.e. ensures that symlinks, attributes, permissions, owner, etc. are preserved)
• -r: recursive
• -v: verbose (second v gives what files are being skipped and some stats at the end)
• -z: compress
• -h: human-readable format of numbers

• --del | --delete: delete files on the destination that are not sent from the source

• --stats: file-transfer statistics at the end

• -m: prune empty directories from file-list

• -t: preserve modification times

This is how I deploy changes of this site with rsync:

rsync -arvzh --delete public/ vld.bg:www/vld.bg/public_html/ 

Have you ever tried to SSH into an old machine, say a managed switch, UPS or router from the previous decade only to get declined with the following error:

Unable to negotiate with <host>: no matching host key type found.
Their offer: ssh-rsa,ssh-dss

It’s frustrating, isn’t it? OpenSSH implements all of the cryptographic algorithms needed for compatibility with standards-compliant SSH implementations, but since some of the older algorithms have been found to be weak, not all of them are enabled by default. If you want to learn more about it, definitely should read OpenSSH Legacy Options .

But this is a tip, so the short story is that you should use one of offered algorithms by the server, say -oHostKeyAlgorithms=+ssh-rsa. When it comes to ssh-rsa, however that won’t be enough and you would be challenged with another error:

Bad server host key: Invalid key length

The crypto-policy that comes with modern distributions does not allow RSA key sizes < 2048 bit. Some SSH servers are configured with 1024-bit key sizes that can lead to connection failures. -oRequiredRSASize=1024 overrides that default and Voilà.

Example output under different conditions:

touch foo && echo ... > foo

/path/to/watched/dir/ CLOSE_WRITE,CLOSE foo

mv foo bar

/path/to/watched/dir/ MOVED_FROM foo
/path/to/watched/dir/ MOVED_TO bar

rm bar

/path/to/watched/dir/ DELETE bar

Text editors

This is where things get interesting

vim

/path/to/watched/dir/ CLOSE_WRITE,CLOSE foo

Boring, right? However, let’s do :set backup and try again:

/path/to/watched/dir/ CLOSE_WRITE,CLOSE 4913
/path/to/watched/dir/ DELETE 4913
/path/to/watched/dir/ MOVED_FROM foo
/path/to/watched/dir/ CLOSE_WRITE,CLOSE foo

wut?

emacs

/path/to/watched/dir/ CLOSE_WRITE,CLOSE foo
/path/to/watched/dir/ DELETE .#foo

.#foo is never created, yet deleted.

vscode

/path/to/watched/dir/ CLOSE_WRITE,CLOSE asdf
/path/to/watched/dir/ CLOSE_WRITE,CLOSE asdf

Not sure why it does it twice.

Even have it as an alias in my dotfiles :

alias myip='curl -s https://zx2c4.com/ip | head -n1'