This is part of the talk “UNIX archeology” I gave at OpenFest 2023.

Hidden files are fairly popular these days, especially in our home folders. We store configuration, cache, logs and whatnot in there. Here’s what’s going on in the home folder on the machine I’m writting this right now:

❯ find ~ -name '.*' -type d | wc -l
5768
❯ find ~ -name '.*' -type f | wc -l
30573

But why?

Suppose you’re working on this new cool OS and receive it as a feature request: “We should introduce hidden files”. How would you do it? Knowing that files generally need attributes, why not create a new one for it? Exactly what Microsoft has one for their cool OS:

But that is not how we do that in Unix. Instead, soon after the beginning of time, in Unix hidden files are those which name starts with a dot. Looks odd when you think about it, trying to ignore the fact you’re so used with them. But why?

It’s not a bug, it’s a feature!

Rob Pike wrote an explanation on Google+. Even though Google killed that service long ago, The Wayback Machine has a record of that :

A lesson in shortcuts.

Long ago, as the design of the Unix file system was being worked out, the entries . and .. appeared, to make navigation easier. I’m not sure but I believe .. went in during the Version 2 rewrite, when the file system became hierarchical (it had a very different structure early on). When one typed ls, however, these files appeared, so either Ken or Dennis added a simple test to the program. It was in assembler then, but the code in question was equivalent to something like this:

if (name[0] == '.') continue;

This statement was a little shorter than what it should have been, which is

if (strcmp(name, ".") == 0 || strcmp(name, "..") == 0) continue;

but hey, it was easy.

Two things resulted.

First, a bad precedent was set. A lot of other lazy programmers introduced bugs by making the same simplification. Actual files beginning with periods are often skipped when they should be counted.

Second, and much worse, the idea of a “hidden” or “dot” file was created. As a consequence, more lazy programmers started dropping files into everyone’s home directory. I don’t have all that much stuff installed on the machine I’m using to type this, but my home directory has about a hundred dot files and I don’t even know what most of them are or whether they’re still needed. Every file name evaluation that goes through my home directory is slowed down by this accumulated sludge.

I’m pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.

How many bugs and wasted CPU cycles and instances of human frustration (not to mention bad design) have resulted from that one small shortcut about 40 years ago?

Keep that in mind next time you want to cut a corner in your code.

(For those who object that dot files serve a purpose, I don’t dispute that but counter that it’s the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.)