CFEngine: Issue when reading /proc or /sys files with readfile()

•  November 6, 2013
• cfengine  

Introduction

A first glance, reading files located in /proc or /sys directories on GNU/Linux systems is not very different from reading plain files.

But you may encounter surprises while doing so through CFEngine and its readfile() function. I will try to explain why, the current state with stable CFEngine versions, and a workaround.

TL;DR Workaround

Using CFEngine’s readfile()

Because of the “everything is a file” approach of all Unix derivatives, /proc or /sys files allow an usual I/O interface. In CFEngine, the readfile() function lets you reading an external file into a variable:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

body common control {
	bundlesequence => { "my_readfile" };
}

bundle agent my_readfile {

	vars:
		"content"  string  =>  readfile("/etc/resolv.conf", "10000");

	reports:
		cfengine::
			"$(const.endl)$(content)";
}

Looks sane:

1
2
3
4
5
6

$ cf-agent -f ./readfile.cf
2013-11-05T19:03:18+0100   notice: /my_readfile: R:
nameserver 212.27.40.241
nameserver 212.27.40.240
nameserver 2a01:e00::2
nameserver 2a01:e00::1

Fun starts here

So reading /proc/meminfo should be straighforward, isn’t it?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

body common control {
	bundlesequence => { "my_readfile" };
}

bundle agent my_readfile {

  vars:
    "content"  string  =>  readfile("/proc/meminfo", "10000");

  reports:
    cfengine::
      "$(const.endl)$(content)";
}

Surprise, nothing!

1
2
3
4
5

$ cf-agent --version
CFEngine Core 3.5.2

$ cf-agent -f readfile.cf
2013-11-05T19:24:05+0100   notice: R:

What happened? The answer lies on internal readfile() implementation, which is roughly:

• Getting the file size with stat()
• Preparing a buffer to read up to this size
• Retrieving the file’s content into the buffer, up to the determinated file size.

Nothing is read from /proc/meminfo. Try to make an educated guess:

1
2

$ ls -l /proc/meminfo
-r--r--r-- 1 root root 0  5 nov.  19:36 /proc/meminfo

/proc/meminfo appears to stat() as an empty file, therefore readfile() tries to read… 0 bytes.

However readfile() is not totally to blame. If the known file size is zero, nothing has to be read, which is a correct behavior for a plain file.

The discussion around the related bug report and the fix has led to some interesting findings.

I will use the stat(1) command to retrieve the file size given by its inode, and pv(1) to read the file and display how many bytes were really read.

Let’s test with a 4 bytes plain file:

1
2
3
4
5

$ echo "Meh" > foo.txt
$ stat -c "%s" foo.txt
4
$ pv -b < foo.txt > /dev/null
  4 O

Both stat(1) and v(1) agree that the file is 4 bytes long (the 4th byte is the invisible line feed 0x0a character)

Trying with /proc/meminfo:

1
2
3
4

$ stat -c "%s" /proc/meminfo
0
$ pv -b < /proc/meminfo > /dev/null
1.14kiB

Here, plain file I/O semantics are broken. /proc/meminfo which appears to be empty, holds data. Another way to check is a copy to a plain file:

1
2
3

$ cat /proc/meminfo > /tmp/meminfo
$ stat -c "%s" /tmp/meminfo
1170

But surprises do not stop here:

1
2
3
4

$ stat -c "%s" /sys/module/soundcore/coresize
4096
$ pv -b < /sys/module/soundcore/coresize > /dev/null
   5 B

After non-empty files that appears as empty ones, here is one which appears to have 4096 bytes, but only holds 5 bytes.

It becomes difficult to cope with plain files which exhibit a mix between plain and FIFOs I/O semantics !

Current situation in CFEngine

Upcoming CFEngine-3.6

This bug is fixed.

maxsize parameter of readfile() has to be set to 0 when reading /sys or /proc files to ensure a reliable read. (Documented here, will be part of CFEngine-3.6 reference manual)

Note there is a glitch: because of strings internal limits, readfile() can only read up to 4096 bytes. (any file, not just /proc or /sys ones)

CFEngine-3.5.2

1
2
3
4
5

$ cf-agent --version
CFEngine Core 3.5.2

$ cf-agent -f readfile.cf
2013-11-06T07:19:57+0100   notice: R:

=> Empty result, does not work.

CFEngine-3.4.5

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

$ cf-agent --version

   @@@
   @@@      cf-agent

 @ @@@ @    CFEngine Core 3.4.5
 @ @@@ @
 @ @@@ @
 @     @
   @@@
   @ @
   @ @
   @ @

$ cf-agent -f readfile.cf
R:

=> Does not work.

CFEngine-3.3.9

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

$ cf-agent --version

   @@@
   @@@      cf-agent

 @ @@@ @    CFEngine Core 3.3.9
 @ @@@ @
 @ @@@ @
 @     @
   @@@
   @ @
   @ @
   @ @

$ cf-agent -f readfile.cf
R:

=> Does not work.

Workaround

Using the execresult() function to catch the output of cat /proc/meminfo (first seen here) is a reliable way to read /proc or /sys files across CFEngine’s 3.x versions.

Tested with the following bundle, on 3.3.9, 3.4.5, 3.5.2 and upcoming 3.6 versions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

body common control {
	bundlesequence => { "my_readfile" };
}

bundle agent my_readfile {

  vars:
    cfengine_3_6::
      "content" string  => readfile("/proc/meminfo", "0");

    !cfengine_3_6::
      "content" string  => execresult("/bin/cat /proc/meminfo", "useshell");

  reports:
    cfengine::
      "CFEngine-$(sys.cf_version) $(const.endl)$(content)";
}

Note that execresult() trick will also work with CFEngine-3.6

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

$ cf-agent -f readfile.cf |head -n 5
2013-11-06T08:03:36+0100   notice: /my_readfile: R: CFEngine-3.6.0a1.4a6bd93
MemTotal:        8180216 kB
MemFree:         5947544 kB
Buffers:          145904 kB
Cached:           957696 kB

$ cf-agent -f readfile.cf |head -n 5
2013-11-06T08:01:31+0100   notice: R: CFEngine-3.5.2
MemTotal:        8180216 kB
MemFree:         5985668 kB
Buffers:          145776 kB
Cached:           956888 kB

$ cf-agent -f readfile.cf |head -n 5
R: CFEngine-3.4.5
MemTotal:        8180216 kB
MemFree:         5986488 kB
Buffers:          145768 kB
Cached:           956884 kB

$ cf-agent -f readfile.cf |head -n 5
R: CFEngine-3.3.9
MemTotal:        8180216 kB
MemFree:         5986024 kB
Buffers:          145776 kB
Cached:           956888 kB

If you plan to work with “large” strings variables (> 4KB), beware, you may hit an internal strings limit (bug report) and the file will not be fully read.

Strings limits will be discussed in a further blog post.