Improved Character Encoding Support

I’ve had a number of bug reports regarding issues with character encodings. Particularly for folks working in Europe and Asia, but really for anyone using multibyte characters in their source code and log messages (and we all do nowadays, don’t we?), it has been difficult to find the proper incantation to get SVN::Notify to convert data from and to their proper encodings. Using a patch from Toshikazu Kinkoh as a starting-point, and with a lot of reading and experimentation, as well as regular and patient tests on Toshikazu’s and Martin Lindhe’s production systems, I think I’ve finally got it nailed down.

Now you can use the --encoding (formerly --charset), --svn-encoding, and --diff-encoding options—as well as --language—to get SVN::Notify to do the right thing. As long as your Subversion server’s OS supports an appropriate locale, you should be golden (mine is old, with no UTF-8 locales :\). And if all else fails, you can still set the $LANG environment variable before executing svnnotify.

There is actually a fair bit to know about encodings to get it to work properly, but if you use UTF-8 throughout and your OS supports UTF-8 locales, you shouldn’t have to do anything. You might have to set --language in order to get it to use the proper locale. See the new documentation of the encoding support for all the details. And if you still have problems, please do let me know.

Output Filtering

Much sexier is the addition of output filtering in SVN::Notify 2.70. I got pretty tired of getting feature requests for what are essentially formatting modifications, such as this one requesting support for KDE-style keyword support. I myself was using Trac wiki syntax in commit messages on a recent project and wanted to see them converted to HTML for messages output by SVN::Notify::HTML::ColorDiff.

So I finally sat down and gave some though on how to implement a simple plugin architecture for SVN::Notify. When I realized that it was generally just formatting that people wanted, it became simpler: I just needed a way to allow folks to write simple output filters. The solution I came up with was to just use Perl. Output filters are simply subroutines named for the kind of output they filter. They live in perl packages. That’s it.

For example, say that your developers write their commit log messages in Textile, and rather than receive them stuck inside <pre> tags, you’d like them converted to HTML. It’s simple. Just put this code in a Perl module file:

package SVN::Notify::Filter::Textile;
use Text::Textile ();

sub log_message {
    my ($notifier, $lines) = @_;
    return $lines unless $notify->content_type eq 'text/html';
    return [ Text::Textile->new->process( join $/, @$lines ) ];
}

Put the file, SVN/Notify/Filter/Textile.pm somewhere in a Perl library directory. Then use the new --filter option to svnnotify to put it to work:

svnnotify -p "$1" -r "$2" --handler HTML::ColorDiff --filter Textile

Yep, that’s it! SVN::Notify will find the filter module, load it, register its filtering subroutine, and then call it at the appropriate time. Of course, there are a lot of things you can filter; consult the complete documentation for all of the details. But hopefully this gives you a flavor for how easy it is to write new filters for SVN::Notify. I’m hoping that all those folks who want featurs can now stop bugging me and writing their own filters to do the job, and uploading them to CPAN for all to share!

To get things started, I scratched my own itch, writing a Trac filter myself. The filter is almost as simple as the Textile example above, but I also spent quite a bit of time tweaking the CSS so that most of the Trac-generated HTML looks good. You can see an example right here. Thanks to a number of bug fixes in Text::Trac, as well as Trac-specific CSS added via a filter on CSS output, it works beautifully. If I’m feeling motivated in the next week or so, I’ll create a separate CPAN distribution with just a Markdown filter and upload it. That will create a nice distriution example for folks to copy to creat their own. Or maybe someone on the Lazy Web Will do it for me! Maybe you?

I wish I’d thought to do this from the beginning; it would have saved me from having to add so many features/cruft to SVN::Notify over the years. Here’s a quick list of the features that likely could have been implemented via filters instead of added to the core:

• --user-domain: Combine the SVN username with a domain for the From header.
• --add-header: Add a header to the message.
• --reply-to: Add a specific header to the message.
• SVN::Notify::HTML::ColorDiff: Frankly, looking back on it, I don’t know why I didn’t just put this support right into SVN::Notify::HTML. But even if I hadn’t, it could have been implemented via filters.
• --subject-prefix:: Modify the message subject.
• --subject-cx: Add the commit context to the subject.
• --strip-cx-regex: More subject context modification.
• --no-first-line: Another subject filter.
• --max-sub-length: Yet another!
• --max-diff-length: A filter could truncate the diff, although this might be tricky with the HTML formatting.
• --author-url: Modify the metadata section to add a link to the author URL.
• --revision-url: Ditto for the revision URL.
• --ticket-map: Filter the log message for various ticketing system strings to convert to URLs. This also encompasses the old --rt-url, --bugzilla-url, --gnats-url, and --jira-url options.
• --header: Filter the beginning of the message.
• --footer: Filter the end of the message.
• --linkize: Filter the log message to convert URLs to links for HTML messages.
• --css-url: Filter the CSS to modify it, or filter the start of the HTML to add a link to an external CSS URL.
• --wrap-log: Reformat the log message for HTML.

Yes, really! That’s about half the functionality right there. I’m glad that I won’t have to add any more like that; filters are a much better way to go.

So download it, install it, write some filters, get your multibyte characters output properly, and enjoy! And as usual, send me your bug reports, but implement your own improvements using filters!

• (?{ }), a zero-width, non-capturing assertion that executes arbitrary Perl code.
• $^N, a variable for getting the contents of the most recent capture in a regular expresion.

The cool thing is that, used in combination, these two features can be used to hack named captures into Perl regular expressions. Here’s an example:

use warnings;
use strict;
use Data::Dumper;

my $string = 'The quick brown fox jumps over the lazy dog';

my %found;

my @captures = $string =~ /
    (?: (quick|slow) \s+    (?{ $found{speed}  = $^N  }) )
    (?: (brown|blue) \s+    (?{ $found{color}  = $^N  }) )
    (?: (sloth|fox)  \s+    (?{ $found{animal} = $^N  }) )
    (?: (eats|jumps)        (?{ $found{action} = $^N  }) )
/xms;

print Dumper \@captures;
print Dumper \%found;

The output of running this program is:

$VAR1 = [
          'quick',
          'brown',
          'fox',
          'jumps'
        ];
$VAR1 = {
          'color' => 'brown',
          'speed' => 'quick',
          'action' => 'jumps',
          'animal' => 'fox'
        };

So the positional captures are still returned, and we’ve assigned them to keys in a hash. This can be very convenient for complex regular expressions.

This is a cool feature, but there are a few caveats. First, according to the Perl regular expression documentation, (?{ }) is a highly experimental feature that could go away at any time. But more importantly, if you’re relying on this feature you should be aware of the side effects. What I mean by that is that, if a regular expression match fails, but there are some successful matches during execution, then the code in the (?{ }) assertions could still execute. For example, if you changed the word jumps to poops in the above example, the output becomes:

$VAR1 = [];
$VAR1 = {
          'color' => 'brown',
          'speed' => 'quick',
          'animal' => 'fox'
        };

Which means that the match failed, but there were still assignments to our hash, because some of the captures succeeded before the overall match failed. The upshot is that you should always check the return value from the match before relying on whatever the code inside the (?{ }) assertions did.

The problem becomes even more subtle if your regular expressions trigger backgracking. In that case, you might have an optional group match and its value assigned to the hash, and then the next required group fail. Perl will then backtrack to throw out the successfull group match and then see if the next required match succeeds. If so, you can have a successful match and potentially invalid data in your hash. Here’s an example:

my @captures = $string =~ /
    (?: (quick|slow) \s+    (?{ $found{speed}  = $^N  }) )
    (?: (brown|blue) \s+    (?{ $found{color}  = $^N  }) )?
    (?: (brown\s+fox)       (?{ $found{animal} = $^N  }) )
/xms;

print Dumper \@captures;
print Dumper \%found;

And the output is:

$VAR1 = [
          'quick',
          undef,
          'brown fox'
        ];
$VAR1 = {
          'color' => 'brown',
          'speed' => 'quick',
          'animal' => 'brown fox'
        };

So while the second group returned undef for the color capture, the %foundhash still had the color key in it. This may or may not be what you want.

Of course, all this seems cool, but since it’s a truly evil hack, you have to be careful. If you can wait, though, perhaps we’ll see named captures in Perl 5.10.

Or take this failure. It says, [CP_ERROR] [Mon Sep 5 09:32:08 2005] No such module ‘mod_perl’ found on CPAN. Yet here it is. Maybe the CPANPLUS indexer has a bug? Or are people’s configurations just horked? Or am I just doing something braindead?

Opinions welcomed.

I just released FSA::Rules 0.25. This version came about as I returned to the module to handle setting up a PostgreSQL database and found the graphics that it churned out, well, wanting. I wanted a decision tree, but the graphics just had the names of the states for the nodes, and then long question-like labels on the edges. What I wanted instead was for each node to be a question (or a statement about what the node was doing), and for the edges to be simple answers to those questions (or indicators as to the success of the code run in a state).

So I added a new attribute to the state class, label. You can use this attribute to say something more about the state. In my case, I used it to store the question the state asks, or the description of the state’s activities. I then changed the code that creates the graph to use this attribute in preference to the state name when creating node labels. The result is a much more natural decision graph, as you see here

The release features a number of other goodies, including the elimination of a dependence on the Clone module, and thus also a big memory savings. There is now a lot more control over the format of graphs, too. Enjoy!

sub isa_gtin {
    my @nums = reverse split q{}, shift;
    (
        sum( @nums[ grep {   $_ % 2  } 0..$#nums ] ) * 3
      + sum( @nums[ grep { !($_ % 2) } 0..$#nums ] )
    ) % 10 == 0;
}

But it seems wasteful to generate the series of numbers twice and to calculate whether they’re odd or even twice. Surely there’s a more efficient way to do this in Perl, perhaps even more expressive? Python seems to have a useful syntax for creating array slices that step. In Python, I’d do something like this:

  sum( nums[1:10:2] ) * 3 + sum( nums[2:10:2])

But barring such a slice feature in Perl is there some cleaner way than the ugly grep approach I created to generate a stepped series in Perl?

use My::DBI;
BEGIN {
    sub DBI::Shell::Base::DBI () { 'My::DBI' };
}
use DBI::Shell;

Yes, it’s extremely sneaky. DBI::Shell::Base uses the string constant DBI, as in DBI->connect(...), so by shoving a constant into DBI::Shell::Base before loading DBI::Shell, I convince it to use my subclass, instead.

’Course the other thing I realized, after I made this change and all the tests pass, was that there is no equivalent of sendmail on Windows. So I added the —smtp option, so that now email can be sent to an SMTP server rather than to a local sendmail. I tested it out, and it seems to work, but I’d be especially interested to hear from folks using wide characters in their repositories: do they get printed properly to Net::SMTP’s connection?

The whole list of changes in 2.57 (the output remains the same as in 2.56):

• Finally ported to Win32. It was actually a simple matter of changing how command pipes are created.
• Added —smtp option to enable sending messages to an SMTP server rather than to the local sendmail application. This is essential for Windows support.
• Added —io-layer to the usage statement in svnnotify.
• Fixed single-dash arguments in documentation so that they’re all documented with a single dash in SVN::Notify.

Enjoy!

The change I’m most pleased with in 2.56 is the addition of SVN::Notify::Alternative, based on a submission from Jukka Zitting. This new subclass allows you to actually combine a number of other subclasses into a single activity notification message. Why? Well, mainly because, though you might like to get HTML messages with colorized diffs, some mail clients might not care for the HTML. They would much prefer the plain text version.

SVN::Notify::Alternative allows you to have your cake and eat it too: send a single message with multipart/alternative sections for both HTML output and plain text. Plain text will always be used; to use HTML::ColorDiff with it, just do this:

svnnotify —repos-path "$1" —revision "$2" \
  —to developers@example.com —handler Alternative \
  —alternative HTML::ColorDiff —with-diff

This incantation will send an email with both the plain text and HTML::ColorDiff formats. If you look at it in Mail.app, you’ll see the nice colorized format, and if you look at it in pine, you’ll see the plain text.

For the curious, here are all of the changes since 2.50:

2.56 2006-04-04T23:16:37

• Abstracted creation of the diff file handle into the new diff_handle() method.
• Documented use of diff_handle() in the output() method.
• Added optional second argument to output() to optionally suppress the output of the email headers. This argument is used by the new Alternative subclass.
• Added SVN::Notify::Alternative, which allows multiple versions of a commit email to be sent, such as text/plain plus HTML. The multiple versions are assembled into a single email message using the multipart/alternative media type. For those who want HTML messages but must support users that can only read plain text or rely on archives that ignore HTML messages, this can be very useful. Based on an implementation by Jukka Zitting.
• Fixed use_ok() tests that weren’t running at all.
• Added an extra newline to separate the file list from an inline diff in the plain text format where —with-diff has been specified.
• Moved the multipart/mixed content-type header generation from output_headers() to output_content_type(), not only because this makes more sense, but also because it makes attachments behave better when using SVN::Notify::Alternative.
• Documented accessors in SVN::Notify::HTML.

2.55 2006-04-03T23:11:11

• Added the io-layer option to specify an alternate IO layer. Will be most useful for those with repositories containing text in multiple encodings, where it should be set to raw.
• Fixed the context output in the subject for the —subject-cx option so that it’s smarter about determining the longest common path. Reported by Max Horn.
• No longer modifying the values of the to_regex_map hash, so as not to mess with folks who might be passing it as a hash to more than one call to new(). Reported by Darby Felton.
• Added a meta http-equiv="content-type" tag to HTML output that includes the character set to help some clients in the proper display of the characters in an HTML email. I’m not sure if any clients actually need this help, but it certainly can’t hurt!
• Added the —css-url option to specify an alternate style sheet for HTML emails. SVN::Notify::HTML’s own CSS is left in the email, as well, so the specified style sheet can just override the default, rather than have to style everything itself. Yes, it takes advantage of the cascading feature of cascading style sheets! Based on a suggestion by Steve James.

2.54 2006-03-06T00:33:42

• Added /usr/bin to the list of paths searched for executables. Suggested by Nacho Barrientos.
• Added —max-diff-length option. Patch from David Burley/SourceForge.

2.53 2006-02-24T21:30:48

• Added header and footer attributes and command-line options to specify text to be put at the head and foot of each message. For HTML messages, the text will be escaped, unless it starts with <, in which case it will be assumed to be valid HTML and will therefore not be escaped. Either way, it will be output between <div> tags with the IDs header or footer as appropriate. Based on a patch from David Burley/SourceForge.
• Fixed the executable-searching algorithm added in 2.52 to add .exe to the name of the executable being searched for if $^O eq 'MSWin32'.
• Fixed encoding issues so that, under Perl 5.8 and later, the IO layer is set on file handles so as to encode input and decode output in the character set specified by the charset attribute. CPAN # 16050, reported by Michael Zehrer.
• Added a second argument to all calls to encode_entities() in SVN::Notify::HTML and SVN::Notify::HTML::ColorDiff so that only '>'. '<', '&', and '"' are escaped.
• Fixed a bug in the _find_exe() function that was attempting to modify a constant variable. Patch from John Peacock.
• Turned the _find_exe() function into the find_exe() class method, since subclasses (such as SVN::Notify::Mirror) might want to use it.

2.52 2006-02-19T18:50:24

• Now uses File::Spec->path to search for a validate sendmail or svnlook when they’re not specified via their respective command-line options or environment variables. Suggested by Andreas Koenig. Not that they should probably be explicitly set anyway, as the $PATH environment variable tends to be non-existent when running under Apache.

2.51 2006-01-02T23:28:11

• Fixed ColorDiff HTML to once again be valid XHTML 1.1.

Enjoy!

sub _pipe {
    my ($self, $mode) = (shift, shift);
    # Safer version of backtick (see perlipc(1)).
    local *PIPE;
    my $pid = open(PIPE, $mode);
    die "Cannot fork: $!\n" unless defined $pid;

    if ($pid) {
        # Parent process. Return the file handle.
        return *PIPE;
    } else {
        # Child process. Execute the commands.
        exec(@_) or die "Cannot exec $_[0]: $!\n";
        # Not reached.
    }
}

The problem is that it doesn’t work on Windows. perlipc says:

Note that these operations are full Unix forks, which means they may not be correctly implemented on alien systems. Additionally, these are not true multithreading. If you’d like to learn more about threading, see the modules file mentioned below in the SEE ALSO section.

’Course, the SEE ALSO section doesn’t have much of for alien systems, but I have a comment in my code that suggests that Win32::Process might do for Windows compatibility. But I honestly don’t know.

So what’s the best approach for me to port SVN::Notify to Windows while keeping file handle pipes around for efficiency? Anyone care to take a stab at it, with tests for Winows, and send me a patch?

Perltidy witll turn it into this:

my $process = Background->new( $^X, "-I$lib", "-MMyLong:Namespace::Bar::Bat",
    "-e 1", "other", "arguments", "here" );

That’s a little better, but I’d much rather that it made it look like this:

my $process = Background->new(
    $^X,    "-I$lib", "-MMyLong:Namespace::Bar::Bat",
    "-e 1", "other",  "arguments", "here",
);

Or even this:

my $process = Background->new(
    $^X,
    "-I$lib",
    "-MMyLong:Namespace::Bar::Bat",
    "-e 1",
    "other",
    "arguments",
    "here",
);

Anyone know how to get it to do that? If so, please leave a comment!

With Perltidy installed and this function thrown into your ~/.emacs file, you can run perltidy on a region by just hitting C-C t. If no region is selected, it’ll run perltidy on the whole buffer.

2.50 2005-11-10T23:27:22

• Added —ticket-url and —ticket-regex options to be used by those who want to match ticket identifers for systems other than RT, Bugzilla, GNATS, and JIRA. Based on a patch from Andrew O’Brien.
• Removed bogus use lib line put into Makefile.PL by a prerelease version of Module::Build.
• Fixed HTML tests to match either ' or &#39;, since HTML::Entities can be configured differently on different systems.

2.49 2005-09-29T17:26:14

• Now require Getopt::Long 2.34 so that the —to-regex-map option works correctly when it is used only once on the command-line.

2.48 2005-09-06T19:14:35

• Swiched from <span class="add"> and <span class="rem"> to <ins> and <del> elements in SVN::Notify::HTML::ColorDiff in order to make the markup more semantic.

2.47 2005-09-03T18:54:43

• Fixed options tests to work correctly with older versions of Getopt::Long. Reported by Craig McElroy.
• Slick new CSS treatment used for the HTML and HTML::ColorDiff emails. Based on a patch from Bill Lynch.
• Added —svnweb-url option. Based on a patch from Ricardo Signes.

2.46 2005-05-05T05:22:54

• Added support for Copied files to HTML::ColorDiff so that they display properly.

2.45 2005-05-04T20:38:18

• Added support for links to the GNATS bug tracking system. Patch from Nathan Walp.

2.44 2005-03-18T06:10:01

• Fixed Name in POD so that SVN::Notify’s POD gets indexed by search.cpan.org. Reported by Ricardo Signes.

2.43 2004-11-24T18:49:40

• Added —strip-cx-regex option to strip out parts of the context from the subject. Useful for removing parts of the file names you might not be interested in seeing in every commit message.
• Added —no-first-line option to omit the first sentence or line of the log message from the subject. Useful in combination with the —subject-cx option.

2.42 2004-11-19T18:47:20

• Changed Files to Paths in hash returned by file_label_map() since directories can be listed as well as files.
• Fixed SVN::Notify::HTML so that directories listed among the changed paths are not links.
• Requiring Module::Build 0.26 to make sure that the installation works properly. Reported by Robert Spier.

Enjoy!

[
    q{this },
    q{is },
    q{O'},
    q{Reilly'},
    q{s },
    q{string},
];

So it’s imperfect, but it works well enough for me. I’m thinking of using the Unicode character class for words, instead, at least for more recent versions of Perl that understand them (5.8.0 and later?). That would be /(?<!\p{IsWord})(?=\p{IsWord})/msx. The results using that regular expression are the same.

But otherwise, I’m not sure whether or not this is the best approach. I think that it’s good enough for the general cases I have, and the matching of words in and of themselves is not that important. What I mean is that, as long as most tokens are words, it’s okay with me if some, such as O', Reilly', and s in the above example, are not words. What I don’t know is how well it’ll work for non-Roman glyphs, such as in Japanese or Korean text. I tried a test on a Korean string I have lying around (borrowed from the Encode.pm test suite), but it didn’t split it up at all (with use utf8;).

So what do you think? Does Text::WordDiff work for your text? Is there a better and more general solution for tokenizing the words in a string?