I finally played with libcrm114, a C library that implements several text classification algorithms. It is a potential replacement for the mailreaver.crm tool, which is the basis for my SpamAssassin plugin.

Having a C library removes the need to fork a crm interpreter process (and in most cases also the need to read the learned feature data) for every single classification; it also enables the inclusion into other languages (so far I know modules for PHP and Python – now my module seems to be the first one for Perl).

A first and unprofessional benchmark against my mailbox confirms the expected performance improvement. I use Text::AI for the old fork-interpret-model (source) and Text::AI::CRM114 for the new library in-memory model (source).

~> time perl test_ai_crm114.pl
classified 9111 texts in 100.86 seconds (11.070 millisec per text)
Spam Texts: 68
 Ham Texts: 9043
30.717u 64.212s 1:41.50 93.5%   200+8161k 9361+0io 0pf+0w

~> time perl test_text_ai_crm114.pl
classified 9111 texts in 7.89 seconds (0.866 millisec per text)
Spam Texts: 68
 Ham Texts: 9043
6.719u 0.485s 0:08.41 85.4%     9+2691k 0+0io 0pf+0w

• syslog-sign library: take the syslog-sign code from syslogd and turn it into a general library to sign arbitrary streams of text or packets.
• timestamping: extend syslogd or the syslog-sign library to include qualified timestamps.
• PEAR::Payment_SEPA: write a free library to read and generate SEPA credit transfer messages, basically the next generation of Payment_DTA. Problem is it’s a big XML base specification and customer-to-bank messages may still use different formats; leading to the secondary problem that validation services to test such an implementation are expensive.
• Puppet package plugin for FreeBSD to use portmaster instead of portupgrade.

Today both memory and disks are cheap enough to make this question insignificant in many cases. One can usually buy enough memory to prevent swapping, and with terrabyte disks it makes no real difference whether the swap partition is 1 Gb or 4 Gb in size.

I still use the old rule of thumb and allocate twice the size of RAM for a swap partition. But the intention is not so much to swap memory pages but rather to always have enough space for kernel core dumps. My basic reasoning is:

1. In case of kernel errors I want to get a core dump.
2. A later RAM upgrade is relatively cheap and easy, thus probable over a system’s lifetime.
3. A later disk re-partitioning is more difficult, thus to avoid.
4. Disk space is cheap, allocating ≤ 5% for a mostly unused swap partition has no significant cost.

2011-11-19T01:31:21 frodo kernel: pid 47812 (try), uid 0: exited on signal 10 (core dumped)

Source of the program is Perl’s Configure script, which compiles its C library test cases into programs named try. And apparently amd64 systems need to use the function va_copy() because the test case without this function leads to a segfault.

[Short english explanation:] I finished my diploma thesis: Design and Implementation of an IPv6 Plugin for the Snort Intrusion Detection System.

Both techniques give the OS more control over processes and the ability to kill running applications. At first sight this looks scary and a loss of control over your computer. But it is viable for the OS to provide better resource management. Think of it as Swapping 2.0 — only instead of moving virtual memory pages to disk and back, the application is simply killed to free its memory pages.

It might get even more interesting if developers accept this new model and design their application accordingly. Then there could be quite an incentive to reduce startup times (maybe also to reduce the overall size of a process’s state and use transaction safe I/O). Finally: Recovery-Oriented Computing on the desktop :-)

Once upon a time I had to extract some application data using the antiqe DDE API and send it to a webservice. Once I found the Win32::DDE::Client it was quite easy to write 200 lines of Perl for the task. Because the script should not require a Perl installation on every machine it is used on, I used the Cava Packager to put everything into an .exe file. (Now I have to extend that old script and find that Cava is no longer free, but requires a subscription for commercial use.)

In order to know my alternatives I tried the Python Win32 extensions and py2exe and these seem to be just as good (or in case of the undocumented DDE module: just as bad) as their Perl counterparts. (Py2exe is not as easy to use as Cava; but not too difficult either.)

I found three ways to get complexity measures for PHP code.

The most advanced technique is to use a continuous integration server. One good example is the server at test.pear.php.net which uses CruiseControl (a Java CI server) with phpUnderControl for the PHP specific parts. Certainly a nice solution for big projects, but nothing to use on your own laptop to improve a single package.

Often the easiest way is to use PHP_CodeSniffer, which includes checks for cyclomatic complexity and nesting level (causing warnings and errors above some thresholds). If you already use it to maintain a uniform coding style (e.g. PEAR, Zend) then you just have to add these checks to your standard definition. These commands add metrics to the PEAR standard:

cd /usr/local/share/pear/PHP/CodeSniffer/Standards
sudo cp -R Generic/Sniffs/Metrics PEAR/Sniffs
sudo sed -I .bck -e 's/Squiz_/PEAR_/' PEAR/Sniffs/Metrics/*

The program PDepend creates a more detailed analysis of methods, classes, and projects (see this release note for details). Because it is intended as a component within a CI server its primary output is in XML. So I wrote myself a small wrapper (pdepend_summary.php) to format the data as an ASCII table. This allows for quick edit-check-cycles and now I run it along with phpcs and phpunit before I commit PEAR code.

Example output:

Package              Class                                  LoC %Comment Variables Methods Class_SiZe   WMC
-------              -----                                  --- -------- --------- ------- ----------   ---
Payment_DTA          DTA                                   1084     36.3         3      15        114   111
Payment_DTA          DTAZV                                  757     37.1         1      12         72    71
Payment_DTA          DTABase                                563     33.0         6      19         33    27
Payment_DTA          Payment_DTA_ChecksumException            3      0.0         0       0          0     0
Payment_DTA          Payment_DTA_FatalParseException          3      0.0         0       0          0     0
Payment_DTA          Payment_DTA_Exception                    3      0.0         0       0          0     0
Payment_DTA          Payment_DTA_ParseException               3      0.0         0       0          0     0
 
Package/Class             Method                                   LoC %Comment    CCN    NPath
-------------             ------                                   --- --------    ---    -----
Payment_DTA/DTA           _generateCrecord                         158     25.9     13     1600
Payment_DTA/DTA           _parseCextension                          81     30.9     13       93
Payment_DTA/DTA           _processCextension                        47      2.1      8        8
Payment_DTA/DTA           parse                                     59     35.6      7       36
Payment_DTA/DTAZV         _exchangeFillSender                       33      0.0      7       64
Payment_DTA/DTAZV         parse                                     41     17.1      7       36
Payment_DTA/DTAZV         _exchangeFillReceiver                     26      7.7      6       24
Payment_DTA/DTA           _exchangeFillArrays                       31      0.0      6       32
Payment_DTA/DTA           _parseCrecord                            109     27.5      6       72
Payment_DTA/DTAZV         setAccountFileSender                      41      0.0      5       15
Payment_DTA/DTA           _parseErecord                             49     30.6      5       16
Payment_DTA/DTABase       getStr                                    18      0.0      4        8
Payment_DTA/DTAZV         __construct                               19     10.5      4        4
...

Probably every TeX-User knows the advocacy about how powerful the language is… but do (or can) you proofread everything before you compile it?