PS3 Media Server is a DLNA server capable of, among other things, streaming and transcoding local media files to Digital Media Players. While this includes the PS3, it also includes the Xbox 360, which makes a pretty decent media player, and the server handles things like subtitles rather neatly.
Transcoding means the server can handle converting, for example, mkv video files on-the-fly, obviating any need to convert them before viewing. However, my initial experiments with mkv files showed the following error in the debug log:
[wmav2 @ 0x33e4946240] output buffer size is too small
However, looking at the command issued to mencoder by PS3 Media Server, there were no knobs to increase the buffer size beyond what was already specified… And only the Xbox 360 had this issue, as other devices did not require wmav2/asf.
Counterintuitively, the solution was to upgrade ffmpeg to 1.0.1. While ffmpeg transcoding worked perfectly well, its libavformat and libavcodec libraries are used by mencoder to transcode to wmav2, and something between mplayer/mencoder and the older version of ffmpeg led to the error.
On a Gentoo box, this stack works well:
PS3 Media Server 1.72 MPlayer 1.1 ffmpeg 1.0.1