CFEngine: maximum strings length

2013-11-20

cfengine

Update 2013-12-04: If you would like to participate, there is a dedicated bug report and a thread on the mailing-list

As far as I know, the maximum length of strings variables is not outlined in the official documentation.

You will hit internal limitations through warnings and errors during cf-agent execution, some examples:

Buffer exceeded 8192 bytes (...)
Expansion overflow constructing string. Increase CF_EXPANDSIZE macro. Tried to add (...)
error: Fatal CFEngine error: Can't expand varstring

Because I prefer to know beforehand such limits, I have tried to empirically quantify them, by reading files of different sizes using 2 methods:

with readfile()
with execresult(), capturing the output of*cat /path/to/filename//

This way, I’m more confident that I will not encounter a bug specific to readfile() or execresult().

Here are my findings for each stable versions: 3.3.9, 3.4.5 and 3.5.2

Official documentation

3.3.9: Nothing ? The most accurate information I had found is

Each scalar may have one of three types: string, int or real

3.4.5: same as 3.3.9
3.5.2: unfortunately, the newest documentation is not more explicit:
```
Allowed input range: (arbitrary string)
```

Empirical tests

The following bundle will be used to find the limits. The protocol is:

Create plain files with a precise size: 1000, 2000, 10 000 bytes (teaser: going further is not necessary)
Read the file’s content into a string with both readfile() and execresult() functions.
Display the string with a reports: promise, and measure its length with awk.

body common control {
    bundlesequence => { "string_limits" };
}

bundle agent string_limits {

  vars:
    "sizes" ilist => { 1000, 5000, 10000};

    "size"  int   => "$(sizes)", ifvarclass => "s$(sizes)";

    "file_$(size)" string => "/tmp/$(size}";

    "content_cat_$(size)"      string => execresult("/bin/cat $(file_$(size))", "useshell");
    "content_readfile_$(size)" string => readfile("$(file_$(size))", "20000");

  classes:
    "file_$(size)_exists"  expression  =>  fileexists("$(file_$(size))");

  commands:
    "/bin/perl -e 'print \"A\"x$(size)' > $(file_$(size))"
      contain     => ushell,
      ifvarclass  => "!file_$(size)_exists";

  reports:
    cfengine::
      "cfe-$(sys.cf_version) - $(size) bytes - execresult()$(const.endl)$(content_cat_$(size))";
      "cfe-$(sys.cf_version) - $(size) bytes - readfile()$(const.endl)$(content_readfile_$(size))";
}

body contain ushell {
  useshell => "true";
}

CFEngine-3.3.9

1000 bytes: OK

$ cf-agent -f string_limits.cf -D s1000|awk '/R:/ {print} /AAA/ {print length($0)}'
R: cfe-3.3.9 - 1000 bytes - execresult()
1000
R: cfe-3.3.9 - 1000 bytes - readfile()
1000

5000 bytes: NOK

$ cf-agent -f string_limits.cf -D s5000|awk '/R:/ {print} /AAA/ {print length($0)}'
R: cfe-3.3.9 - 5000 bytes - execresult()
4053
R: cfe-3.3.9 - 5000 bytes - readfile()
4055

10 000 bytes: NOK

$ cf-agent -f string_limits.cf -D s10000|awk '/R:/ {print} /AAA/ {print length($0)}'
4094
$ cf-agent -f string_limits.cf -D s10000
Buffer exceeded 8192 bytes in exec /bin/cat /tmp/10000
Expansion overflow constructing string. Increase CF_EXPANDSIZE macro. Tried to add AAA(...)AAAA
Fatal CFEngine error: Can't expand varstring

The limit is between 1000 and 5000 characters

CFEngine-3.4.5

1000 bytes: OK

$ cf-agent -f string_limits.cf -D s1000|awk '/R:/ {print} /AAA/ {print length($0)}'
R: cfe-3.4.5 - 1000 bytes - execresult()
1000
R: cfe-3.4.5 - 1000 bytes - readfile()
1000

5000 bytes: NOK

$ cf-agent -f string_limits.cf -D s5000|awk '/R:/ {print} /AAA/ {print length($0)}'
$ cf-agent -f string_limits.cf -D s5000
Segmentation fault (core dumped)

10 000 bytes: NOK

$ cf-agent -f string_limits.cf -D s10000
Buffer exceeded 8192 bytes in exec /bin/cat /tmp/10000
Segmentation fault (core dumped)

The limit is also between 1000 and 5000 characters

CFEngine-3.5.2

1000 bytes: OK

$ cf-agent -f string_limits.cf -D s1000|awk '/R:/ {print} /AAA/ {print length($0)}'
2013-11-19T20:48:08+0100   notice: R: cfe-3.5.2 - 1000 bytes - execresult()
1000
2013-11-19T20:48:08+0100   notice: R: cfe-3.5.2 - 1000 bytes - readfile()
1000

5000 bytes: OK

$ cf-agent -f string_limits.cf -D s5000|awk '/R:/ {print} /AAA/ {print length($0)}'
2013-11-19T20:48:20+0100   notice: R: cfe-3.5.2 - 5000 bytes - execresult()
5000
2013-11-19T20:48:20+0100   notice: R: cfe-3.5.2 - 5000 bytes - readfile()
5000

10 000 bytes: NOK

$ cf-agent -f string_limits.cf -D s10000|awk '/R:/ {print} /AAA/ {print length($0)}'
8317
$ cf-agent -f string_limits.cf -D s10000
2013-11-19T20:48:39+0100    error: Buffer exceeded 8192 bytes in exec '/bin/cat /tmp/10000'
2013-11-19T20:48:39+0100    error: Expansion overflow constructing string. Increase CF_EXPANDSIZE macro. Tried to add 'AA(...)AA' to ''
2013-11-19T20:48:39+0100    error: Fatal CFEngine error: Can't expand varstring

The limit is between 5000 and 10 000 characters

Summary

More tests have been done to get the exact number of characters a string can hold before truncation/unexpected behavior results are on the following table:

	readfile()	execresult()
CFEngine-3.3.9	4055	4053
CFEngine-3.4.5	4055	4053
CFEngine-3.5.2	8028	8026

CFEngine-3.4.5 starts to segfault once the string length is 5000+

cf-agent -f string_limits.cf -D s5000
Segmentation fault (core dumped)

Questions:

Is something wrong regarding my “protocol” to find limits ?
Does it worth adding these limits to the documentation ? What do you think ?