tychoish, now without a net

“So basically, what I’m doing is, sorting the email on the server in a git repository, and then pushing and pulling to that as needed, or just working their over ssh.” I explained

“So what you’re saying, is that you’ve basically reinvented IMAP,” Chris said.

“Yeah, pretty much, except that this works,” I said.

“If you say so.”

I do say so. So this is what it comes down to:

IMAP got one thing right: we need a way of accessing our email that works on public computers, is machine independent, that works offline and online, and that keeps all these mail reading environments synchronized.

The problem is that IMAP is incredibly flakey and inconsistent. Messages that you’ve read suddenly become unread, messages that you’ve moved suddenly pop back into your inbox, it’s slow, and if you don’t have the right mail client and a server that’s tweaked in the right way, it might not really work at all.

If IMAP worked as well as it could, we’d all use it because, ideally it’s the best way to manage email. Everything is stored remotely but cached locally for offline use and backup, you can use multiple machines without worry. Instead I suspect most people who need these features use webmail if they need multi-machine email accounts,[^webmail] and that’s great, if it works for you. Gmail and the like are great pieces of software, I don’t mean to begrudge webmail, I just don’t enjoy the experience of the browser, and seek to avoid it except for browsing.

So while I’d given up on email that synced really well in the last couple of months (preferring to use mutt and procmail locally,) I still longed for this kind of email set up.

So I thought a bit and figured some things out, and here’s what I came up with. (A step by step explanation of how I get email):

1. A bunch of email addresses (for different contexts) are forwarded to a gmail account, which despamifies my email, and uses it’s own filters to sort and forward mail (using +address aliases) to a secret email account on the web server.
2. The webserver uses procmail to sort and deliver the email into a non-public (obviously) folder/git repository.
3. I have a series of scripts to manage the git/mail interaction depending on if I’m on an SSH connection or not, mostly this is straightforward, but here’s how the sync works (commands from the local perspective in parenthesis):
   1. via ssh I add and commit any new mail that may have arrived since the last sync to the repository on the server (ssh git add * && git commit).
   2. Locally I commit any change that’s been made to my mail directory since the last sync, and pull down the new mail from the server (git add * && git commit && git pull)
   3. Push my changes up, and everything merges and the repositories look the same now. and I reset the index of the remote server so to reflect the changes. (git push && ssh git reset --hard)¹
   4. Rejoice.²
   5. Set up a cronjob or a launchd daemon to do this automatically every, say 15 minutes, and then you get the illusion of having “push” email.
      • If you get growl to notify you of the results of the pull (end of step 2) you’ll be able to see weather there’s new mail or not.

I think you could probably set much of this up in post-commit/post-push hooks. And I think it goes without saying that if there are more than one “client” repository in play, you’ll want to pull changes more often than you sync.

So why would I (and you?) want to do this? Easy:

• It’s quick. Git is fast, by design, and since email files have a lot of redundancy in the headers and what not, git can save a lot of space in the “tube” and speed up your email downloads.
• It’s robust and flexible. If my server stooped working for some reason, I could set up another, or simply start using fetch-mail again. If the server went back up, I just push the changes up, and I’m back in business.
• Git handles renames implicitly, and this is a really cool feature that I don’t think gets quite enough mention. Basically if I move or rename a file, and add the new file to git, (git add * does this) git realizes that I’ve moved the file. so as files move and get named other things, git realizes that it’s happened, deals with it and moves on.
  • Understandably, if you delete a file locally git won’t delete it from the repository “head” unless you tell it to. This bash script takes care of that: for i ingit status | grep deleted | awk ‘{print $3}’; do git rm --quiet $i; done
• It’s secure, or at least is if you do it right. All the email gets downloaded over SSH, so no clear text passwords (if you’re using public key authentication) and encrypted data transmission.
• Oh yeah, and it’s not flakey like IMAP. Totally worthwhile.
  • If there is some sort of syncing problem, you know where it is, as opposed to IMAP making an executive decision without telling you and un-sorting all your email.

Any questions?

Onward and Upward!

permalink • • coments
tagged: technology

Sorry about the posting confusion this week. I’ll be out on Monday here and on Critical Futures, but everything will be back to normal on Tuesday.

This is a post that I’ve been working on for a while, and I’m not sure that it’s done but I’m fresh out of ideas for more book recommendations. But then, this is a blog, and there’s a comment form for a reason, so if I’ve been remiss and forgotten something important, please do remind me. Enjoy!

I have perpetual fears about not being well read enough. I think this is mostly an existential problem, as I read a bunch, and I’ve read a lot of stuff in my day. But there’s always more, particularly as I think about shifting my academic/intellectual specialty. In any case, I found myself a few weeks ago recommending a science fiction novel that I had read,¹ and I thought it might be good to post a list of recommendations. Not just “books I like,” but “books I’d tell you to read if you were looking for something specific.”

For people who like military SF and some people who like Heinlein, John Scalzi’s Old Man’s War is a great deal of fun.

For a great novella/tightly wound plot, Samuel R. Delany’s Empire Star and for lingustic sf, the other half of the book, Babel-17. I’m a huge fan. For anyone interested in urban systems and community, Times Square Red/Times Square Blue Amazon is a great read, and is incidentally the most cited source of my college career.

For a very smart, but also very comforting and enjoyable space opera trilogy, Melissa Scott’s Five-Twelfths of Heaven Trilogy can’t be beat. Her Trouble and her Friends is also a great example of what cyberpunk can be when it’s not trying too hard to be the next New Thing.

I don’t find myself in a position to recommend mainstream fiction very much, but I am a big fan of Anne Lamott’s All New People, and Barbara Kingsolver’s Prodigal Summer, which are both clever and fun, and are unabashedly delightful.

Books about writing? Five or more years ago, I would have had a list of books about writing, but as a genre I’m not particularly convinced of their worth/utility or frankly interesting-ness. Anne Lamott’s Bird by Bird, is simply amazing, though not particularly for the specific writing related hints. I’ve always thought that Stephen King’s writing book is pompous and sort of unfocused, but admittedly I couldn’t even force myself to finish it.

That’s what come to mind, do you all have any good recommendations?

permalink • • coments
tagged: reading • science fiction • writing

Ed: I so I totally meant to post this on Monday instead of this post, but the wires got crossed, and Monday’s post got out early. Sorry. –ty

The one thing that really undermines my argument against preemptive rewriting is the fact that I’m a compulsive outliner, at least for some things.

Since I’m currently in the “planning a new story/project” phase¹ I’ve been thinking about the outlining process a bit.

The difference between blogging and writing is that in a blog post I maybe scribble something on a paper ahead of time, but the rhetorical thread comes out of my fingers and on to your screens pretty much straight through.² Blogging is very much like the research process of fiction writing, in that it’s a process of taking an idea and figuring out what it means, and a lot of the possibilities in whatever thing or situation (blog and fiction respectively). In blogging, you pray someone’s interested or you’ve written something that gets folks irked enough to respond, in fiction you take all this thought work and then write about only the things that really need writing about.

And then I realized the other day, that I write the outlines of my stories–indeed my entire personal wiki notepad–to you, dear blog readers. At some point years ago I got a pretty good idea of who “you” are, and as a result generally my blog posts are all addressed to the same positional reader (even though that reader/group has changed a bit, and my conception of “the reader” has always been pure fantasy on my part.) Even though I’m the only one that reads this document, all the parts are written to the same person. It’s all very strange.

But I digress. Research.

I think the sort of stock advice to the new writer doing research is “don’t get too caught up in planning, writing is about writing, not planning to write,” and while I do think that the reminder to always keep the writing in mind, at the same time, there’s something really important and intense about the planning process that shouldn’t be ignored. It’s a lot of work, there are a lot of words that go into getting a story to the point where you can even think about starting to write in ernest. In the end I think it’s not so much about what you outline or what you read up on, as the act of doing it. The time spent in your head thinking about this world, and this idea.

Speaking of thought work, I have some to do.

Onward and Upward!

permalink • • zero comments
tagged: productivity • writing

10 pearls of collected wisdom for the aspiring terminal/console/command line user. This assumes *NIX but little actual command line experience.

1. Customize your your .bash_profile or .bashrc files and continue to tweak this file as you learn more about how you use the shell. I even have a line in my profile that makes it easier for me to edit this page.
2. Never open files unless you have too. tools like less, more, cat, and grep should be enough to keep you going for most routine checks.
3. Having said that, getting to know your text editor really well should be on the top of your list of things to do. There’s something to be said for learning how to use vim, though I can understand if you might want to use something a little less sharp around the edges.
4. It’s good to be able to hack your way through bash/shell scripting and at least one other general purpose scripting/programing language. Like Perl or Python, but Ruby and PHP would work. Power users don’t necessarily need to be able to write brilliant programs, they just need to figure out how to glue other programs together.
5. Familiarize yourself with your operating system’s package manager, or get macports if you’re on a mac. Or get an operating system with a better package manager. To my tastes this means means getting a debian-based linux distribution, but there are others if this won’t work for you. These package installers make it so much easier to install software and have it work because other people do the testing. Compiling things on your own is ok, but package mangers are better. Learning how to use the cpan shell and ruby gems falls under this imperative.
6. Do first and script second. While you may be tempted to write nifty little scripts for all the things you think you’re going to do, don’t. Work first and figure out what your habits are and then write the scripts/macros/short cuts that will best serve you. That way you’ll use them.
7. Figure out how to schedule tasks/automate background tasks. If there’s an internet connection, and probably even if there isn’t, my computer checks my email every 8 minutes or so. Because I have a little “check email and tell tycho if there’s anything new” script set to run in a launchd deamon. You could use cron if you’re not using a mac, but the general idea is that if there are things that you know need to be done, regularly, you just tell the computer to do them, and then you don’t have to worry about it.
8. Read the manuals and google for help, but also relax. The terminal lets you do a lot of new things, and it saves you a lot of time. It’s also hard to learn and a lot of die-hard terminal users are also, to be blunt, assholes. I don’t want to recount the number of times that I’ve seen people rant on about proper forum, and how new folk ask too many questions. This is dumb, new people always ask questions, and the truth is that some things aren’t well documented. Also a lot of terminal assholeishness comes from a period of time, when certain operations took a lot of CPU power, and CPU power was more at a premium than it is today. Most contemporary computers, even ones that are a few years old, run so fast that even inefficient terminal applications still run incredibly fast on modern hardware, and can outperform the best GUIs. You’ll learn later.
9. Remember the Unix Philosophy. Basically that programs should do one thing well, and not complicate themselves with doing more than one thing. If you know that this is how things can and should work and you can learn how to work with this, then you’re in good shape. (It’s also ok to bend it a little bit, from time to time.) Also, if you’re an oddball like me–and using the command line for something, like writing fiction–slowly get a sense of what existing tools do, and figure out if its useful to you. Also know that just because something’s cool it doesn’t mean that it’s going to be useful for you.
10. Customize the appearance of your console window. Apple stocks everything with an ugly black text on white background thing, which makes my head hurt from the squinting. Readably sized fonts, good coding fonts, anti-assailing, colorizing your prompt, light text on dark backgrounds, and some transparency all make the terminal more functional and elegant.

permalink • • zero comments
tagged: technology

I just wanted to pop in out of lecture mode (sigh) and give a brief sort of “meta report,” of what I’m working on, where I am in life, and what my plans are for the blog are in the next little bit.

First off, as next Monday is a national holiday (whew) and I’m going to be out of town doing family stuff for the long weekend, I’m not going to have a Critical Futures post on monday. The current story, “Visa Riots” (in Trailing Edge) will run through Wednesday (6 parts) and then I’ll have a little two part Station Keeping story so we’ll be able to start September with a new section of something fun.

I’ve been hard at work planning a new novel, writing more Station Keeping. I’ve also been working a lot, but my current employment situation is going to be changing significantly in September (I’m in a “hurry up and wait,” situation, but I remain optimistic.) In any case, the next job will be more flexible, whatever it is, and I’m pretty confident that the impending changes in my life that happen this fall will be exciting and productive.

In terms of the blog, I’ve decided that unless something seems really important I’m not going to talk about things I’m writing and the writing process. I might still post book reviews and thoughts on reading, but I’ll probably tone that down as well. I’m not sure that it’s been helpful to my process and workflow to spend time reflecting on these sorts of things, and I’d rather spend my blogging time talking about other things, like…

Knitting and technology. I’m starting to knit more again, and I’m pretty aware that you all in blogland are–at least for the moment–primarily knitters. So, I think blogging about knitting will be helpful and fun.

In terms of technology, I’ve been through a couple of phases about my writing about technology for this blog, and I’ve not been incredibly happy with my previous modes. I think writing about material technology gadgets/hardware, endorses consumerism in a way that I don’t thinks interesting or fun. And while I remain interested in productivity and technology, writing about being productive isn’t productive (ie. productivity porn) and in my experience always gets hung up on fadish systems or tools, and that’s not interesting. So I’ll probably continue to blog about producivity, but much less than I have been.

I’m interested in writing more about open source stuff. My post about drupal was a lot of fun, and I’ve had some other thoughts which will (hopefully) germinate into full grown posts in the next couple of weeks.

I understand that the geeks might not get the knitting and the knitters might not get the tech stuff but stay tuned in any case, I’m a generalist in both camps, and I promise to be enthusiastic.

Also I have a little script to ease in the posting of links and other miscellany that I find along my travels through cyberspace, I just have to get better at using it. That should help even out any over specialization that I might be prone to.

So that’s what’s on my plate and mind. What’s on yours?

On/up!

permalink • • one comment
tagged: journal • site