Outlook, Mail Archives, and Duplicates

Exchange and Outlook are dismal examples of code, but the fact remains that they are ubiquitous. Nobody has managed to create a mail/calendar/contacts/task application with wider adoption, and it has enough inertia that well designed applications have little chance to make inroads, which means a lot of people are stuck with it. For those of us who prefer elegant, well designed applications, putting up with their quirks is maddening.

Outlook, for example, has a hard-to-explain 2 gigabyte limit on mail archives — and mail archives are arguably one of the niftier features that Outlook offers. Early versions of Outlook don’t know any better, and simply corrupt your mail archives. Later versions of Outlook know better, and warn you not to exceed the limit. While some noise has been made about Outlook finally removing the 2 gigabye limit, it’s actually not quite true, it’s only been removed for Exchange style mailboxes, and is still there, for example, for imap mail boxes.

For those of us with lots of mail and the need to archive it (I receive a lot of technical documents, some very large, via email) using Outlook’s built-in “archives” isn’t really an option, so I used the simple expedient of setting up an archive IMAP server, where the size wouldn’t be an issue. While this works reasonably well going forward, Outlook puked enough while trying to move messages from its proprietary formats to imap, that I was left with a vast number of duplicates.

On a significantly large mailbox, this is a bigger problem than it sounds like — especially since the duplicates were created with different mail id’s, and in many cases the white space or envelopes are different, while the messages are clearly identical. Maddening, but it largely means that any automated duplicate removal will have to happen through IMAP, not through the filesystem.

While it seems that a tool to locate and eliminate duplicate IMAP emails would be simple to find, it appears that such a beast simply does not exist, except for the trivial case in which the message id’s are identical. At the imap level, there are a decent number of tools here:

http://www.athensfbc.com/imap_tools/

Which work admirably, for the most part. For the remainder, I used this Thunderbird Add-on, which took care of the remaining fringe cases. The only problem, of course, is that on a really large email folder, Thunderbird starts to complain endlessly about script timeouts. However, you shouldn’t really need to do this regularly.

Share

Mail, DNS entries, and domains

I recently overhauled bits of the mail system here to take care of a few lingering quirks that I’d never had the time nor inclination to track down. All of my various email addresses and aliases go to the exact same mailbox, through the multiple expedients of fetchmail, which picks up my mail from gmail and AOL, and DNS MX records that point everything to the same place.

Until recently, if you sent mail to “user@goodjobsucking.com” it would be transformed by the server into “user@baddomain.com” unceremoniously. It would show up that way in the mailbox, and only by delving into the mail headers was it obvious that the mail was originally destined for a different domain. For addresses I didn’t make use of much, this was fine, though it leads to the curious circumstance where somebody sending mail to goodjobsucking.com would get replies from baddomain.com, which deviates from the principles of separating domains in the first place.

It turns out the root cause was that goodjobsucking.com, rather than having its own A record in the DNS tables, used a CNAME to baddomain.com. Apparently this implies that mail sent to goodjobsucking.com is actually for baddomain.com. I imagine this would be particularly useful for adjunct or typo domains, where you want to correct the original destination or transition from one domain to another. It’s also useful in that the mailer only needs to internally relay for, and listen to, mail destined for baddomain.com; any mail sent to a CNAME from another domain pointing to it works perfectly well.

Moving the domain from a CNAME to an A record effectively separates things out again, though now the mailer must also be aware that it’s listening for mail for yet another domain.

Share

LaCie 1TB NAS

LaCie has a wonderful little NAS box called the “Ethernet Big Disk” that comes in a 1TB size, supports USB and gigabyte ethernet, and is reasonably priced. The NAS part supports smb, afs, ftp, and http, which makes it pretty darned useful out of the box.

Even better, it sports a version of embedded Linux on an ARM CPU:

Linux version 2.6.12.6-arm1-lacie5a (root@lacie) (gcc version 3.4.4 (release) (CodeSourcery ARM 2005q3-2)) #1 Tue Oct 31 11:26:21 CET 2006CPU: ARM926EJ-Sid(wb) [41069260] revision 0 (ARMv5TEJ)

Naturally, it’s a prime target for hacking in some useful things. This can be accomplished in several easy steps:

  1. Open it up and pull out the drives
  2. Mount the SATA drives on another Linux box
  3. Add an exploit to the web server (which runs as root, making the rest easy)
  4. Put it back together
  5. Make yourself a root account
  6. Cross compile some binaries and copy them useful places
  7. Give yourself a command line

Okay, so up until step 5, it’s pretty straightforward, and documented here and here. Actually, even making a root account is pretty straightforward; using the web exploit, you can copy passwd and shadow to the NAS share, edit them, and copy them back. I chose to add an account called “myroot” and leave the default root account’s password alone, just in case it’s needed for … something.

Building a cross-compiler is, frankly, a major pain. However, once I got it built properly (in my case, using a handy port for the MacIntosh) it was fairly straightforward to cross-compile utelnetd and rsync. utelnetd can actually be launched from the web exploit, and it’s possible then to log in directly as root.

For those who appreciate a kick start, here are the compiled binaries that run on the LaCie. (Naturally, these are without any warranty of any kind, and whatever happens to your NAS is your own fault, etc.)

lacie.tar.gz

I can’t think of much more one could want than telnet, rsync, and nfs. Well, maybe ssh, but it’s been done, and I don’t really need it.

Share

Rsync over SMB

This is a short one — I was using rsync to back up about a terabyte of data, which suffered a number of interruptions. Each time it restarted, I noticed it was copying over the same files over and over again. As it turns out, SMB rounds off the time stamps to the nearest two-second interval, so when rsync compares timestamps, it believes the file is different.

There’s a simple solution:

rsync -rlptDv –modify-window=1 [source] [destination]

The “–modify-window” switch tells rsync to relax its timestamp comparison just enough for it to behave.

The remaining switches are useful for backing up to a NAS box, it’s essentially -a (archive mode) expanded, but without preserving ownership or groups (which doesn’t work well on the NAS.)

Share

The Hard Sell

I’m sure I’m not alone in my buying habits of researching a product that I want, then searching for the lowest price on the Internet. In general, this practice has served me well, and I’ve had trouble-free dealings with a number of storefronts without any trouble whatsoever.

I occasionally run into web sites with “teasers” — such as are designed to give good prices in search engines like Pricewatch or Google Product Search — where the website will suggest add-ons and accessories like extended warranties and overpriced doodads. The worst websites will automatically select these or add them to your cart. While irritating, the wary can still get a good deal with a lot of checking and cross checking to ensure that what you want is exactly what you get when you go to confirm your order.

A whole new level of dickery was achieved recently when following this basic procedure at Fotoconnection, where the tactic employed is to get some high pressure salesman to call you after your order has been placed. After placing my order for a relatively inexpensive, point-and-shoot camera, I got two emails about an hour apart asking me to confirm my order.

Perhaps unlike most people, my immediate response is, “up yours, jerks, I typed everything correctly, if you can’t handle it, it’s your loss.” Among many reasons I like online ordering is that I don’t actually have to talk to anybody. I just don’t like doing it.

So they call me.

“Sir? We’ve had a lot of problems with stolen credit cards and so we’d like to confirm your billing and shipping address.”

On the surface, this doesn’t make a hell of a lot of sense, since unless I managed to type my address incorrectly, this wouldn’t do them a lot of good. However, maybe they’re actually verifying my phone number, so I play along, figuring an address is at least innocuous. If they had asked for my credit card number, I would not have given it out. (It’s a not-uncommon scam to call somebody in the phone book and ask for their credit card number under some pretense. If they needed the number again, I’d have insisted upon calling them. But more likely, I’d have told them to pound sand.)

“I also noticed that you didn’t order a battery or charger with this camera. Would you like us to add the 5 hour or the 2 hour battery to your order?”

Suddenly, it’s obvious why they called. But, having researched the camera first, I knew better.

“I expect there to be a battery in the box,” I replied. “It’s a sealed box, right?”

“Well, yeah,” the salesman went on, “but that battery is only good for fifteen minutes.”

I’m honestly at a loss for words at this point. First of all, who measures camera batteries in “minutes?” In the context of a camera, what does this even mean? Fifteen minutes of continuous shooting? That actually doesn’t sound too bad. Fifteen minutes of shelf life? That would make even the “5 hour” battery pretty retarded. I can only conclude that the sales guy is an idiot.

“Well, 15 minutes sounds fine for now,” I said, “once I get my original order, I’ll decided if I need anything else.”

“We’ll ship it out today,” he replied, seeming either annoyed or dismayed, but not quite discourteously, and hung up.

I assume that I’ll get what I ordered; if not, I’ll sort it out with my credit card company, but in the meantime, I feel like I need to take a shower. Just. Ewwwww.

So I’m now amending my procedure above, and checking the ratings of online vendors I haven’t used before on a site like Resellerratings. Thank god I didn’t order from somewhere this bad. That being said, fotoconnection has managed to lose a customer, and next time I get a call like that, the only thing I’m going to say is “cancel. my. order.

Share

Oh how I loathe thee, Billing Dispute Department

It’s unusual for me to see charges I don’t recognize on my credit card bill, and even then, it’s usually something my wife bought — or that I bought and the vendor name shows up as something unusual. Therefore, when I saw a charge I didn’t recognize from “Silicon Solar,” I took the step of actually calling Silicon Solar, to see if, in fact, I’d ordered anything or if they did business under another name.

Silicon Solar had no orders or history under my name, or anything remotely like it, so I filled out the paperwork to register a billing dispute with Washington Mutual. You can’t just tell them about it, you have to fill out a form — one that you have to call and have mailed to you. A pain in the butt, but it’s an acceptable level of bureaucracy. Washington Mutual issued a “temporary credit” that appeared on my next bill.

While I was in Canada, a letter came in that the merchant had responded, and they wanted “more information” from me. I called them — in an of itself a feat, because they provided an 800- number only reachable from the U.S., and declined to give any other way of reaching them — like a regular phone number, fax, etc. The letter didn’t say much else other than “call us!”

The representative on the phone said the merchant had responded, and that they needed more information from me. “What did they respond?” I asked.

“We don’t have access to that, we’ll mail it to you,” said the representative.

“Well, what exactly do you want me to do?” I asked. “Nothing has changed, and I have no new information.”

“Nothing,” said the representative. “You’ll receive the merchant’s response from us, along with a questionnaire to fill out. Send that back.”

“Fine,” I said.

The next thing I receive is a letter from Washington Mutual with the Silicon Solars’ response. This is alone galling: it’s a printout from DHL of a box being delivered to Venezuela, and a print out of an order by some guy named Javier Toyo, along with his address in Venezuela. A cover page by Silicon Solar is attached, that says “the cardholder ordered, paid for, and received all merchandise as they’ve requested without any indication of a problem.”

Obviously I take issue with the word “cardholder” in the above.

Worse, the cover letter from Washington Mutual says “We conducted an investigation of this charge based on the information available to us and concluded that we are unable to pursue this dispute further on your behalf.” What?

It goes on, “We have not received a response from you as previously requested.”

What? Apparently calling them wasn’t good enough. Which is strange, because that’s all the previous letter asked me to do — and I did. And, when I asked specifically if there’s anything else I should do or send, I was told, no, wait for the merchant’s response.

I called right away, but since it’s a Saturday, the billing dispute department wasn’t open, so I talked to the fraud department. Fraud fits the bill anyway, since I assume that somebody making unauthorized charges to my account is, in fact, fraud. They can’t help, since “it was originally entered as a billing dispute.” Huh? This implies that when you see an unauthorized charge on your account, you should somehow know that it’s fraudulent — or that you shouldn’t dispute it? Beats me, it makes no damned sense, but I also make no headway. The fraud department refuses to reverse the charge — and, strangely, also refuses to cancel my card. Yes, that seems utterly bizarre; I suspect whoever was on the phone just gave up and started lying to me.

Monday rolls around, and I talk to somebody from the actual Billing Dispute Department. As they are trained to be, they’re pleasant, but completely powerless to actually do anything. Apparently there’s some kind of rule written in stone that says that a customer may not, under and circumstances, talk to anybody actually capable of dealing with anything. That power is in the hands of a select few, who are not directly reachable under any circumstances. The best I can get is a voice mail box.

I try a couple more times — in retrospect, I probably shouldn’t have bothered, but I’m not happy at all with the response — or, more accurately, lack thereof. None of the first line wankages can do anything but tell me they understand my frustration, and reiterate the point that they’re not capable of doing anything.

On the plus side, calling them up and being a major pain in the ass does seem to have finally paid off in that the manager called, probably much, much sooner than the 24-48 hours I was told I would have to wait. It’s worth pointing out that there’s a huge difference between being a pain in the ass, where you calmly restate your case and refuse to take “no” for an answer, and being an abusive jerk, where you yell or insult the poor people answering the phone. For the most part, it’s not their fault, even though most need a great deal of persuasion to walk off their scripts and actually try to do something. I know enough about support and customer service organizations to know that the front lines is a dismal place to be, and that manager’s aren’t available at a moment’s notice to take the call of every whining bastard who doesn’t like the answer he’s given — and if your case truly does require a higher standard of care and attention, it’s hard to make the point.

At any rate, the manager finally did call back, and upon her review, she decided to issue a “courtesy credit.” Essentially, they can’t charge the merchant back (which would have been the right thing to do) because they let too much time lapse. Washington Mutual clearly screwed up by not providing me a form or some kind of response, since they fall under Visa’s chargeback rules, and without a response had to take Silicon Solar’s response at its ludicrous face value. So, Washington Mutual can’t get their money back from Silicon Solar, and they’re certainly not getting any money from me for this debacle.

Today’s cost of bureaucracy and incompetence: $99.30. But let’s look at the scorecard:

Washington Mutual: LOSS — Out $99.30 and a customer. I don’t trust a bank who lets merchants get away without, oh, the card number or name or anything matching the actual transaction. It’s just begging for something worse to happen.

Silicon Solar: WIN — Although I may never order anything from them, they got a paid order and a happy customer in Venezuela, despite apparently sloppy verification procedures.

Javier Toyo: WIN — Free stuff shipped to Venezuela? Oh hell yes, it’s a good day. God knows where he got my credit card number, his obviously false name isn’t as obvious outside of Venezeula. Due to idiocy all around, his chances of getting caught are approximately zero. It sure explains a few things about losses in the credit industry.

Me: DRAW — At the end of the day, I’m not out money for stuff I didn’t order, which is neutral at best. I don’t have to adhere to Visa’s rules, and I retain the option to just… not pay Washington Mutual, which I would certainly do if not for the “courtesy credit.” This courtesy keeps them out of the courts, I suppose, but I’m not under any illusion that they’re doing me a favor, and I spent several hours on this crap that I never should have had to.

Share

HDD LED for add-on cards

Among the many little things that irritate me, it’s when you have a computer case with multiple drives in it, attached to multiple add-on cards as well as the motherboard, but (of course) there’s only one hard drive LED on the case — so you get to pick a controller card to drive the lights on the case, and … well, I suppose you can be happy with that, especially if you don’t pay it much attention.

When looking for a simple cable to combine the inputs, I realized that I may be part of a subset of the larger population, the subset who wants their leds to behave themselves whether it really matters or not. Either my skills with Google are not up to snuff, or there is no such animal commercially available.

Partly it’s because if you’re hapless enough to simply connect them all in parallel, you’re likely to burn out your case led the moment there’s activity on more than one controller card. So slightly more complex circuitry is necessary to make the lights work.

Lying around, I had a tiny case, optocouplers, and resistors, so I had to get power connectors, header connectors, and a breadboard — but there’s really not much more to it than that.

Case open

To the left, the circuit assembled on a breadboard in a little case — to the right, is a sketch of one of the optocoupler circuits. And here’s what it looks like with the case closed:

Outside

Share

Skipping the FreeBSD vulnerability check during portupgrade

FreeBSD has an excellent facility for checking its ports for vulnerabilities provided by security/portaudit. This is very handy when installing an unknown package. However, it can be quite a hindrance when upgrading from one very vulnerable version of a port to one with fewer vulnerabilities, since portupgrade will flatly refuse to upgrade the port, with this kind of error:

===> wordpress-2.2.1,1 has known vulnerabilities:
=> wordpress — unmoderated comments disclosure.
Reference: <http://www.FreeBSD.org/ports/portaudit/6a31cbe3-1695-11dc-a197-0011098b2f36.html>
=> Please update your ports tree and try again.

It’s best, of course, to confirm that the vulnerability is something you can live with. If so, you can pass a flag to “make” to have it skip the vulnerability check:

portupgrade -m -DDISABLE_VULNERABILITIES wordpress

Naturally, I wouldn’t recommend doing this in conjunction with “portupgrade -all” since it would defeat the purpose of having the vulnerability check at all.

Share

A Tale of Audio, Firefox, and X-Windows

The venerable X-Windows has network support with grace and elegance that other window systems (I’m looking at you, Windows) have yet to come anywhere near. Point your applications at any xserver on the network, and there they run — what could be easier? On the down side, this seems to have been developed before sound was particularly important, so without doing anything fancy, your application runs anywhere but (if you’re lucky) any sound it produces emanates from the machine running the actual application. This can be somewhat disorienting, at best.

The dubious goal: to get a flash game working on an x-server — one that doesn’t even have a browser installed.

The first step was to get Firefox/Flash sound working on the gentoo x-client. (It’s been gone over many times, but X appears to use these terms backwards, a tradition which I will continue.) Simple enough:

export DISPLAY=xserver:0.0
firefox

There’s firefox, but, no sound on either the client or the server. A review of what the terminal is spewing out shows:

ALSA lib confmisc.c:848:(snd_func_card_driver) cannot find card '0'
ALSA lib conf.c:3500:(_snd_config_evaluate) function snd_func_card_driver returned error: No such device
ALSA lib confmisc.c:397:(snd_func_concat) error evaluating strings
ALSA lib conf.c:3500:(_snd_config_evaluate) function snd_func_concat returned error: No such device
ALSA lib confmisc.c:1248:(snd_func_refer) error evaluating name
ALSA lib conf.c:3500:(_snd_config_evaluate) function snd_func_refer returned error: No such device

Weirdly, other sound applications appeared to work perfectly well, albeit from the wrong box. Also weirdly, the card is clearly there and enabled, and ALSA is configured in the kernel (not as a module) with the correct sound card.

After a couple of useless dead ends, running Firefox as root revealed that it’s a permission problem. I can think of more useful ways in conveying this, but placing the user into the audio group takes care of that problem. Now I’ve got local audio, at least.

The next trick is to get the client (Gentoo) to send its audio to the server (FreeBSD) .

Share

EVMS and degraded RAID

I’ve been very happy with EVMS under Linux; it’s an excellent way to coherently tie together lvm and md and other mirroring, striping, and logical volume technologies, and has a nifty GUI for managing drive resources. I move drives around much more than I like, often due to the grim reality of bad sectors or filesystem crashes.

After building a RAID volume, a loose power cable on one of the drives caused it to immediate degrade. This shouldn’t be a big deal, since EVMS has a “remfaulty” function for its RAID-5 plug-in. Except that it doesn’t work. I’m not sure whether it’s missing or broken in my version 2.5.5, but it doesn’t work as documented.

Looks like the way to fix this is through md directly, using mdadm. The tricky part about this is that one needs to refer to the EVMS controlled devices, or it doesn’t work. So, to re-add the drive and invoke the synchronization process, one needs to run:

mdadm --add /dev/md0 /dev/evms/.nodes/sda5

Note the “.nodes” part of the command

Share