• 2014-01-19 update: A memory leak has been corrected in the development version
• 2013-12-22 update: Added a graph on results with classes defined inside classes: promise type
• 2013-12-16 update: Quick reaction from the CFEngine developers: a bug has been opened to investigate the Git memory consumption

This article is based on the same principle as my latest post CFEngine: maximum strings length : using CFEngine under extreme situations to reveal where some limits are, or getting confident on CFEngine’s scalability.

Tests will be lead on “soft classes” (the ones defined by the user), on each stable version of CFEngine: 3.3.9, 3.4.5 and 3.5.3 (also git version 3.6.0a1.7697d78)

Official documentation

• 3.3.9: no limits specified ?
• 3.4.5: same as 3.3.9 (shared documentation)
• 3.5.3: no limits specified on the well-written language concepts section

Does that mean no limits?

Empirical tests

The module feature of commands: promises is used to mass-define user classes:

body common control {
	bundlesequence => { "classes_limits" };
}

bundle agent classes_limits {

  vars:
    "classes_count"   int => "250000";

  commands:
    "/bin/bash -c 'for i in {1..$(classes_count)};do printf \"+myclass_%i\n\" $i;done' "
      module  => "true";
}

The “classes_count” integer is the upper limit of the for loop, embedded in the command promise. For “classes_count” = 5, output will be:

$ for i in {1..5};do printf "+myclass_%i\n" $i;done
+myclass_1
+myclass_2
+myclass_3
+myclass_4
+myclass_5

For more details, have a look at the official documentation (Scroll down to the module section)

Various number of classes will be tested, and cf-agent watched for:

• memory consumption (thanks to Valgrind massif plugin)
• time taken to complete (wall clock time)

This is rather artificial, but I think this is a good start to spot something obviously wrong.

Setup

To give a rough idea of the hardware where tests took place:

• Intel(R) Core(TM)2 Quad CPU Q8400 @ 2.66GHz
• 8GB ram
• SSD
• Idle ArchLinux system

It was also the opportunity to learn Pyplot for plotting nice graphs (powerful and well-documented, by the way)

• Memory test is evaluated once, peak memory usage is kept.

• Timing test is evaluated 5 times, because of possible kernel interferences (kernel scheduler), and average time kept.

• Cf-agent command line:

  cf-agent -Kf classes_limits.cf

• Valgrind command line:

  valgrind -q --tool=massif --massif-out-file=<foo> cf-agent -Kf classes_limits.cf

Results

• x-axis: number of defined classes (“classes_count” variable on the above bundle)
• y-axis: 1st graph - peak heap memory usage (MB)
• y-axis: 2nd graph - time to complete (seconds)

All the massif output files are available for download here

Up to 10 000 classes

Analysis:

Memory usage:

• Overlap of 3.3.9 and 3.5.3 curves
• 2 trends: 3.3.9, 3.4.5 and 3.5.3 consume memory very sparingly, while the git version is more greedy

Keep in mind the y-axis scale: for 10 000 classes, worst case is a tremendous 4.8MB memory usage :-)

Timings:

• Again, 2 trends: 3.5.3 and git versions, versus 3.3.9 and 3.4.5 versions
• Worst case: around 0.6s to define 10 000 classes, hardly noticeable !

Up to 500 000 classes

For the fun!

Here, we have confirmation of the previous trends.

To define 500 000 classes, 3.5.3 and git versions are very fast (~5 seconds), while 3.3.9 and 3.4.5 versions are very slow (~ 2500 seconds) Regarding memory consumption, git version uses way more memory (~ 200MB) than the other versions, which stay at a very reasonable 30MB peak memory usage.

Conclusion

Theoretically, CFEngine does not limit how many classes you can define, through the module feature. The limiting factors will be your imagination, patience or memory usage :-)

For what I consider as an “ordinary” use (I mean < 10k classes), latest CFEngine versions scale pretty well. Git version, which means “the one in development”, is not to be taken into account, but is an interesting peek into the future.

Among latest stable versions, worst case to define 10 000 classes is a 4.8MB peak memory consumption and 0.6s to complete.

Comments and insights are welcome ! If you find my protocol absurd or want more details…

Update: what about classes: ?

As requested on the CFEngine mailing-list, I’ve ran the same tests, but this time the classes were defined using classes: promise type, directly on the police file:

body common control {
        bundlesequence => { "classes_limits" };
}

bundle agent classes_limits {

  classes:
    "my_class0" expression  =>  "any";
    "my_class1" expression  =>  "any";
    "my_class2" expression  =>  "any";
    (...)
}

You will find below an animation to show the differences between the 2 methods (classes defined through commands:/module vs classes:). Same trends, but the overhead induced by the parser clearly appears. Not a surprise, because the generated policy file is huge, hence a longer work for the parser. As an example, for 100k classes, the resulting policy file’s size is 3.8MB

Massif files are available for download

Explanation of file names inside the archive:

• 3.3.9_100000_embedded.massif CFEngine-3.3.9, 100 000 classes defined using classes:, massif output file
• 3.3.9_100000_embedded.time the same, but the time taken by cf-agent to complete (time cf-agent…)
• git_5000_module.massif CFEngine 3.6.0a1.7697d78 (master), 5 000 classes defined through command: promise, massif output file