<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Linux on Kiril Vladimirov</title>
    <link>https://vld.bg/tags/linux/</link>
    <description>Recent content in Linux on Kiril Vladimirov</description>
    <generator>Hugo -- 0.168.0</generator>
    <language>en</language>
    <lastBuildDate>Sun, 19 May 2024 23:02:28 +0300</lastBuildDate>
    <atom:link href="https://vld.bg/tags/linux/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Retrieving file timestamps in Go</title>
      <link>https://vld.bg/articles/go-stat/</link>
      <pubDate>Sun, 19 May 2024 23:02:28 +0300</pubDate>
      <guid>https://vld.bg/articles/go-stat/</guid>
      <description>&lt;p&gt;Recently, a friend of mine needed to get the creation time of a file in a Go
project. There&amp;rsquo;s &lt;a href=&#34;https://pkg.go.dev/os#Stat&#34; target=&#34;_blank&#34;&gt;os.Stat&lt;/a&gt;
, so how hard can it be,
right?&lt;/p&gt;
&lt;p&gt;&lt;code&gt;os.Stat&lt;/code&gt; returns &lt;a href=&#34;https://pkg.go.dev/io/fs#FileInfo&#34; target=&#34;_blank&#34;&gt;fs.FileInfo&lt;/a&gt;
 which is an
interface with six methods:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#6ab825;font-weight:bold&#34;&gt;type&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;FileInfo&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6ab825;font-weight:bold&#34;&gt;interface&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;{&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#666&#34;&gt;	&lt;/span&gt;&lt;span style=&#34;color:#447fcf&#34;&gt;Name&lt;/span&gt;()&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6ab825;font-weight:bold&#34;&gt;string&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;       &lt;/span&gt;&lt;span style=&#34;color:#999;font-style:italic&#34;&gt;// base name of the file&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#666&#34;&gt;	&lt;/span&gt;&lt;span style=&#34;color:#447fcf&#34;&gt;Size&lt;/span&gt;()&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6ab825;font-weight:bold&#34;&gt;int64&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#999;font-style:italic&#34;&gt;// length in bytes for regular files; [..]&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#666&#34;&gt;	&lt;/span&gt;&lt;span style=&#34;color:#447fcf&#34;&gt;Mode&lt;/span&gt;()&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;FileMode&lt;span style=&#34;color:#666&#34;&gt;     &lt;/span&gt;&lt;span style=&#34;color:#999;font-style:italic&#34;&gt;// file mode bits&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#666&#34;&gt;	&lt;/span&gt;&lt;span style=&#34;color:#447fcf&#34;&gt;ModTime&lt;/span&gt;()&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;time.Time&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#999;font-style:italic&#34;&gt;// modification time&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#666&#34;&gt;	&lt;/span&gt;&lt;span style=&#34;color:#447fcf&#34;&gt;IsDir&lt;/span&gt;()&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6ab825;font-weight:bold&#34;&gt;bool&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#999;font-style:italic&#34;&gt;// abbreviation for Mode().IsDir()&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#666&#34;&gt;	&lt;/span&gt;&lt;span style=&#34;color:#447fcf&#34;&gt;Sys&lt;/span&gt;()&lt;span style=&#34;color:#666&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6ab825;font-weight:bold&#34;&gt;any&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;           &lt;/span&gt;&lt;span style=&#34;color:#999;font-style:italic&#34;&gt;// underlying data source (can return nil)&lt;/span&gt;&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&lt;span style=&#34;color:#666&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Time of last modification (mtime) is the only time reported by this interface,
though. There is no time of last access (atime), creation (btime), nor status
change (ctime).&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>Recently, a friend of mine needed to get the creation time of a file in a Go
project. There&rsquo;s <a href="https://pkg.go.dev/os#Stat" target="_blank">os.Stat</a>
, so how hard can it be,
right?</p>
<p><code>os.Stat</code> returns <a href="https://pkg.go.dev/io/fs#FileInfo" target="_blank">fs.FileInfo</a>
 which is an
interface with six methods:</p>
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">type</span><span style="color:#666"> </span>FileInfo<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">interface</span><span style="color:#666"> </span>{<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span><span style="color:#447fcf">Name</span>()<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">string</span><span style="color:#666">       </span><span style="color:#999;font-style:italic">// base name of the file</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span><span style="color:#447fcf">Size</span>()<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">int64</span><span style="color:#666">        </span><span style="color:#999;font-style:italic">// length in bytes for regular files; [..]</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span><span style="color:#447fcf">Mode</span>()<span style="color:#666"> </span>FileMode<span style="color:#666">     </span><span style="color:#999;font-style:italic">// file mode bits</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span><span style="color:#447fcf">ModTime</span>()<span style="color:#666"> </span>time.Time<span style="color:#666"> </span><span style="color:#999;font-style:italic">// modification time</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span><span style="color:#447fcf">IsDir</span>()<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">bool</span><span style="color:#666">        </span><span style="color:#999;font-style:italic">// abbreviation for Mode().IsDir()</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span><span style="color:#447fcf">Sys</span>()<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">any</span><span style="color:#666">           </span><span style="color:#999;font-style:italic">// underlying data source (can return nil)</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span>}<span style="color:#666">
</span></span></span></code></pre></div><p>Time of last modification (mtime) is the only time reported by this interface,
though. There is no time of last access (atime), creation (btime), nor status
change (ctime).</p>
<h2 id="sys">Sys()</h2>
<p>Did you notice that <code>Sys()</code> method there? Yeah, me neither. This poorly
documented method returns raw data for that file, as returned by the operating
system. That means the underlying type of the result will be different per-OS.
In our case, it&rsquo;s a software intended to run exclusively on linux, so let&rsquo;s see
what happens there.</p>
<p>Well, in Linux, it returns a pointer to <a href="https://pkg.go.dev/syscall#Stat_t" target="_blank">syscall.Stat_t</a>
:</p>
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">type</span><span style="color:#666"> </span>Stat_t<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">struct</span><span style="color:#666"> </span>{<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Dev<span style="color:#666">       </span><span style="color:#6ab825;font-weight:bold">uint64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Ino<span style="color:#666">       </span><span style="color:#6ab825;font-weight:bold">uint64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Nlink<span style="color:#666">     </span><span style="color:#6ab825;font-weight:bold">uint64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Mode<span style="color:#666">      </span><span style="color:#6ab825;font-weight:bold">uint32</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Uid<span style="color:#666">       </span><span style="color:#6ab825;font-weight:bold">uint32</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Gid<span style="color:#666">       </span><span style="color:#6ab825;font-weight:bold">uint32</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>X__pad0<span style="color:#666">   </span><span style="color:#6ab825;font-weight:bold">int32</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Rdev<span style="color:#666">      </span><span style="color:#6ab825;font-weight:bold">uint64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Size<span style="color:#666">      </span><span style="color:#6ab825;font-weight:bold">int64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Blksize<span style="color:#666">   </span><span style="color:#6ab825;font-weight:bold">int64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Blocks<span style="color:#666">    </span><span style="color:#6ab825;font-weight:bold">int64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Atim<span style="color:#666">      </span>Timespec<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Mtim<span style="color:#666">      </span>Timespec<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Ctim<span style="color:#666">      </span>Timespec<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>X__unused<span style="color:#666"> </span>[<span style="color:#3677a9">3</span>]<span style="color:#6ab825;font-weight:bold">int64</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span>}<span style="color:#666">
</span></span></span></code></pre></div><p>Great, we have atime, mtime and ctime. We&rsquo;ve almost declared a win, until
realizing that ctime is not creation time. It&rsquo;s the time of last status change,
which means it records any change of file&rsquo;s metadata. What we seek is creation
time a.k.a. birth time. Yep, I&rsquo;ve just used almost 300 words to tell you
that with <code>os.Stat</code> you can NOT get the creation time of a file.</p>
<h1 id="statx2">statx(2)</h1>
<p>To be fair, you would not be able to get the creation time using stat in C,
either. Not to mention that most of the filesystems do not keep record of it.
That was surprising to me - fat32, ext2, ext3 nor reiserfs store when a file
has been created. Naturally, no such information is exposed by stat(2).
Modern filesystems do, though.</p>
<p>After years of planning a new system call is finally merged into the Linux kernel
and released with 4.11 in 2017. That&rsquo;s years after the first released version
of Go and five after 1.0. That explains why the standard library does not
provide a straightforward way to obtain btime.</p>
<h1 id="golangorgxsysunix">golang.org/x/sys/unix</h1>
<p>The standard library might be a bit conservative in adding new things (although
things are starting to move now with the new
<a href="https://go.dev/blog/randv2" target="_blank">math/rand/v2</a>
), but that does not mean the Go team
is sleeping. The <code>golang.org/x/</code> packages are outside the main Go tree and are
under way looser constraints for backwards compatibility. One of these packages
is <a href="https://pkg.go.dev/golang.org/x/sys/unix" target="_blank">golang.org/x/sys/unix</a>
. The <code>sys</code>
part is described as &ldquo;packages for making system calls&rdquo;, so do not expect the
easiest to use API you&rsquo;ve ever seen.</p>
<p>Sure enough, there&rsquo;s a wrapper function of the statx syscall:</p>
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">func</span><span style="color:#666"> </span><span style="color:#447fcf">Statx</span>(dirfd<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">int</span>,<span style="color:#666"> </span>path<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">string</span>,<span style="color:#666"> </span>flags<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">int</span>,<span style="color:#666"> </span>mask<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">int</span>,<span style="color:#666"> </span>stat<span style="color:#666"> </span>*Statx_t)<span style="color:#666"> </span>(err<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">error</span>)<span style="color:#666">
</span></span></span></code></pre></div><p>As expected the API is pretty much the same as with the syscall it wraps:</p>
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">int</span> <span style="color:#447fcf">statx</span>(<span style="color:#6ab825;font-weight:bold">int</span> dirfd, <span style="color:#6ab825;font-weight:bold">const</span> <span style="color:#6ab825;font-weight:bold">char</span> *<span style="color:#6ab825;font-weight:bold">restrict</span> pathname, <span style="color:#6ab825;font-weight:bold">int</span> flags,
</span></span><span style="display:flex;"><span>          <span style="color:#6ab825;font-weight:bold">unsigned</span> <span style="color:#6ab825;font-weight:bold">int</span> mask, <span style="color:#6ab825;font-weight:bold">struct</span> statx *<span style="color:#6ab825;font-weight:bold">restrict</span> statxbuf);
</span></span></code></pre></div><ul>
<li>dirfd is a file descriptor identifying the directory to search from if
provided path is relative (if is <code>AT_FDCWD</code> the call operates on the
current working directory).</li>
<li>path is obviously the filename we&rsquo;re querying. Unless when empty and
<code>AT_EMPTY_PATH</code> is in mask, in that case file/directory pointed to by
dirfd is what we&rsquo;re querying.</li>
<li>flags can be used to influence a pathname-based lookup and what sort of
synchronization the kernel will do when querying. When in doubt, use
<code>AT_STATX_SYNC_AS_STAT</code>.</li>
<li>mask is used to tell which fields we are interested in, so that the kernel
does not bother populating the rest. The one we care about to is
<code>STATX_BTIME</code>. If it&rsquo;s there (or <code>STATX_ALL</code> which pretty much includes
everything), then the creation time will be there.</li>
<li>stat is being populated be the syscall when the returned error is nil.</li>
</ul>
<p>All of those <code>SCREAMING_SNAKE_CASE</code> flags listed above are exposed as constants in
the same package.</p>
<p><code>unix.Statx_t</code> is a fairly big struct (matching the statx struct obviously)
that has all of the timestamps we seek:</p>
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">type</span><span style="color:#666"> </span>Statx_t<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">struct</span><span style="color:#666"> </span>{<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span><span style="color:#999;font-style:italic">// ...</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Atime<span style="color:#666">            </span>StatxTimestamp<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Btime<span style="color:#666">            </span>StatxTimestamp<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Ctime<span style="color:#666">            </span>StatxTimestamp<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">	</span>Mtime<span style="color:#666">            </span>StatxTimestamp<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">    </span><span style="color:#cd2828;font-weight:bold">//..</span><span style="color:#666">
</span></span></span><span style="display:flex;"><span>}<span style="color:#666">
</span></span></span></code></pre></div><p><code>StatxTimestamp</code> has <code>Sec</code> and <code>Nsec</code> fields which is enough to create a
<code>time.Time</code> value using <code>time.Unix()</code>. So, the whole thing might look something
like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#d0d0d0;background-color:#202020;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">var</span><span style="color:#666"> </span>statx<span style="color:#666"> </span>unix.Statx_t<span style="color:#666">
</span></span></span><span style="display:flex;"><span>err<span style="color:#666"> </span>:=<span style="color:#666"> </span>unix.<span style="color:#447fcf">Statx</span>(unix.AT_FDCWD,<span style="color:#666"> </span>name,<span style="color:#666"> </span>unix.AT_STATX_SYNC_AS_STAT,<span style="color:#666"> </span>unix.STATX_BTIME,<span style="color:#666"> </span>&amp;statx)<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#6ab825;font-weight:bold">if</span><span style="color:#666"> </span>err<span style="color:#666"> </span>!=<span style="color:#666"> </span><span style="color:#6ab825;font-weight:bold">nil</span><span style="color:#666"> </span>{<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">    </span>log.<span style="color:#447fcf">Fatal</span>(err)<span style="color:#666">
</span></span></span><span style="display:flex;"><span>}<span style="color:#666">
</span></span></span><span style="display:flex;"><span><span style="color:#666">
</span></span></span><span style="display:flex;"><span>btime<span style="color:#666"> </span>:=<span style="color:#666"> </span>time.<span style="color:#447fcf">Unix</span>(statx.Btime.Sec,<span style="color:#666"> </span><span style="color:#24909d">int64</span>(statx.Btime.Nsec))<span style="color:#666">
</span></span></span><span style="display:flex;"><span>fmt.<span style="color:#447fcf">Printf</span>(<span style="color:#ed9d13">&#34;%s is created at %v\n&#34;</span>,<span style="color:#666"> </span>name,<span style="color:#666"> </span>btime)<span style="color:#666">
</span></span></span></code></pre></div><h2 id="easy-right">Easy, right?</h2>
<p>Don&rsquo;t agree? Yeah, me neither. A saner person would just npm the problem and
look for a package exposing file times in a proper, easy-to-use way. I found
this, which looks legit (only checked for GOOS=linux):
<a href="https://github.com/djherbis/times" target="_blank">github.com/djherbis/times</a>
.</p>
<p>I mean it. A third-party package with a higher-level <strong>portable</strong> API is the
way to go in the real world. It&rsquo;s good to know what&rsquo;s happening underneath, but
please beware of NIH syndrome.</p>
<p>Achieving the same result in Python isn&rsquo;t much easier. The result from <code>os.stat</code>
might have a <code>st_birthtime</code> on some operating systems (e.g., FreeBSD or
Windows), but not on Linux. Sometimes seemingly trivial problems are tedious to
solve because of weird historical reasons. And sometimes, a deeper dive is
necessary, but practical solutions are often just a package away.</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
