import urllib2, hashlib

TWITTER = 'http://twitter.com/statuses/public_timeline.json'

def hashed(seq):
    sha1 = hashlib.sha1()
    for x in seq:
        sha1.update(x)
        yield sha1.digest()

def words(wordsize, seq):
    for x in seq:
        for i in xrange(0, len(x), wordsize):
            yield int(x[i:(wordsize + i)].encode('hex'), 16)

def twitrandoms(wordsize = 4, chunksize = 37):
    def infinite_twitrandom():
        while True:
            data = urllib2.urlopen(TWITTER).read()
            for i in range(0, len(data), chunksize):
                yield data[i:(chunksize + i)]
    return words(wordsize, hashed(infinite_twitrandom()))

Anyway, I want code that does this:

import Prelude hiding (Left, Right)

data EitherBoth a = Left a | Right a | Both a a deriving Show

cmpseq :: (Ord a) => (a -> a -> Ordering) -> [a] -> [a] -> [EitherBoth a]
cmpseq _ [] rs = map Right rs
cmpseq _ ls [] = map Left  ls
cmpseq cmp (l:ls) (r:rs) = case (cmp l r) of
                             LT -> Left   l : cmpseq cmp ls  (r:rs)
                             GT -> Right  r : cmpseq cmp (l:ls) rs
                             EQ -> Both l r : cmpseq cmp ls     rs

So given two ordered lists and a comparison function, I want to know (efficiently, and preserving order) which elements are just in one list and which are in both. The Haskell version is pretty clear and pretty concise. (Bonus points: if you have a better way to do this in Haskell — and I’m sure there is one — I’d love to hear about it.)

So far, the best I can do in Python is this rather ugly mess:

def cmpseq(f, xs, ys):
    nx, ny = len(xs), len(ys)
    ix, iy = 0, 0
    while ix < nx and iy < ny:
        r = f(xs[ix], ys[iy])
        if -1 == r:
            yield (xs[ix], None)
            ix = 1 + ix
        elif 1 == r:
            yield (None, ys[iy])
            iy = 1 + iy
        else:
            yield (xs[ix], ys[iy])
            ix, iy = 1 + ix, 1 + iy
    while ix < nx:
        yield (xs[ix], None)
        ix = 1 + ix
    while iy < ny:
        yield (None, ys[iy])
        iy = 1 + iy

Yuck. I’m going to go out on a limb and say that five yields is too many. What’s the idiomatically correct way to do this? Perhaps something to do with feeding values back in to generators? Python language lawyers: please show me the error of my ways!

As it turns out, the easiest way to implement a shuffle is by sorting. It’s not exactly faster, as the typical sort is O(n log n) compared to the O(n) of the Knuth Fisher-Yates shuffle algorithm. We’ll just sort by a random number– in this case, a GUID.

   var cards = Enumerable.Range(0, 51);
   var shuffledcards = cards.OrderBy(a => Guid.NewGuid());

So we can ultimately implement a secure, unbiased shuffle as a one-liner in a modern programming language.

The obligatory nerd response here is to bang out the one-liner in your language of choice. Here’s one for Python:

>>> from random import random
>>> shuffled = [ b for (a,b) in sorted((random(), x) for x in cards) ]

 function get(url) {
  var ajax = new XMLHttpRequest();
  ajax.open('GET', url, false);
  ajax.send(null);
  return ajax.responseText;
}

Of course this will be different for IE….

import re

ROMAN_RE = re.compile("""
    ^                   # beginning of string
    (M{0,4})            # thousands - 0 to 4 M's
    (CM|CD|D?C{0,3})    # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 C's),
                        #            or 500-800 (D, followed by 0 to 3 C's)
    (XC|XL|L?X{0,3})    # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 X's),
                        #        or 50-80 (L, followed by 0 to 3 X's)
    (IX|IV|V?I{0,3})    # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 I's),
                        #        or 5-8 (V, followed by 0 to 3 I's)
    $                   # end of string
""", re.VERBOSE)

ROMAN_DIGITS  = {
    'M'    : 1000, 'MM'   : 2000, 'MMM'  : 3000, 'MMMM' : 4000,
    'CM'   : 900,  'CD'   : 400,  'D'    : 500,  'DC'   : 600,
    'DCC'  : 700,  'DCCC' : 800,  'C'    : 100,  'CC'   : 200,
    'CCC'  : 300,  'XC'   : 90,   'XL'   : 40,   'L'    : 50,
    'LX'   : 60,   'LXX'  : 70,   'LXXX' : 80,   'X'    : 10,
    'XX'   : 20,   'XXX'  : 30,   'IX'   : 9,    'IV'   : 4,
    'V'    : 5,    'VI'   : 6,    'VII'  : 7,    'VIII' : 8,
    'I'    : 1,    'II'   : 2,    'III'  : 3
}

def rtoi(roman):
    match = ROMAN_RE.match(roman.upper())
    if match and 0 < sum(1 for x in match.groups() if 0 < len(x)):
        return sum(ROMAN_DIGITS[x] for x in match.groups() if 0 < len(x))
    return None

One of the problems deals with Collatz sequences: For any number, divide by two if the number is evenly divisible by two, multiply by three and add one otherwise. Stop when you reach the number 1. The task at hand was to find the number between 1 and 1,000,000 with the longest Collatz sequence.

My first attempt at counting this sequence, rather predictably, failed on large input:

 collatzsz 1 = 1
 collatzsz n = 1 + collatzsz nxt where
     nxt = if 0 == mod n 2 then div n 2 else 3 * n + 1

Duh. Stack overflows everywhere. (Note: there’s probably a flag for GHC that will make this Just Work.) I figured that Haskell would probably go ahead and optimize tail calls for me, so I gave it a go:

 collatzsz 1 accum = 1 + accum
 collatzsz n accum = collatzsz nxt (1+accum) where
     nxt = if 0 == mod n 2 then div n 2 else 3 * n + 1

Sure enough, it works for even the most obnoxiously large values of n.

Yes, I know that this is exactly what the language is supposed to do, but sometimes it seems like a small victory when a piece of software does even that. So thanks to the good folks who make GHC. You made me smile today.

 def reduce(f, seq, val = None):
     if val is None: val = next(seq)        
     for x in seq:
         val = f(val, x)
     return val

Sun Microsystems Demands University Study Retraction

The University of Washington, apparently hoping to capitalize on the recent hype around their controversial study on Baby Einsteinâ„¢-style videos, followed up yesterday with another, similar study. In the new study, researchers found that Java programmers understand an average of seven fewer Computer Science concepts per hour spent with Java each day compared to similar programmers using other languages. Sun calls the study “seriously flawed”, citing the fact that you can combine the names of Gang of Four Design Patterns to form new Computer Science concepts that all Java programmers understand, such as the ObserverFactoryBridge, the BridgeFactoryObserver, and the well-known FactoryObserverBridgeChainOfCommandSingletonProxy, beloved of Java programmers everywhere. Java experts at Sun say they’re not sure how many combinations there are of the twenty-three pattern names, but there are “definitely a lot of them.”

import Test.HUnit
import Regex

tests = TestList [ TestLabel (makeLabel x) (makeTest x) | x <- test_data ]
    where makeTest  tup@(reg, str, bool) = 
              TestCase $ assertEqual (makeLabel tup) bool (match reg str)
          makeLabel (reg, str, bool) | True == bool  = reg ++ " =~ " ++ str
                                     | False == bool = reg ++ " !~ " ++ str
          test_data = [ ("foo",  "foo",     True),
                        ("foo",  "bar",     False),
                        ("",     "foo",     True),
                        (".",    "Foo",     True),
                        (".*",   "Foo",     True),
                        ("F.",   "Foo",     True),
                        ("F.*",  "Foo",     True),
                        ("Fo*",  "Foo",     True),
                        ("Fo*d", "foo",     False),
                        ("Fo*d", "Foooood", True) ]

main = runTestTT tests

As mentioned in the article’s comments, the case where “.*” (“match anything”) is supposed to match “Foo” fails. Here’s the test output for the original code:

% runhaskell TestRegex.hs
### Error in:   4:.* =~ Foo                
user error (HUnit:.* =~ Foo
expected: True
 but got: False)
### Error in:   6:F.* =~ Foo               
user error (HUnit:F.* =~ Foo
expected: True
 but got: False)
 Cases: 10  Tried: 10  Errors: 2  Failures: 0
Counts {cases = 10, tried = 10, errors = 2, failures = 0}
% 

Easy enough. Finding out which cases fail makes tracking the problem down relatively straightforward. Turns out that the “matchstar” method doesn’t deal properly with periods at the end of the match. The original code:

matchstar c [] (y:ys) = c == y

Replace it with the following:

matchstar c [] (y:ys) = if c == '.' then True else c == y

And you get the following out of the unit tests:

% runhaskell TestRegex.hs
Cases: 10  Tried: 10  Errors: 0  Failures: 0
Counts {cases = 10, tried = 10, errors = 0, failures = 0}
% 

Horray!

A new data type in the collections module: NamedTuple(typename, fieldnames) is a factory function that creates subclasses of the standard tuple whose fields are accessible by name as well as index. For example:

var_type = collections.NamedTuple(‘variable’, ‘id name type size’)

var = var_type(1, ‘frequency’, ‘int’, 4)

print var[0], var.id # Equivalent

print var[2], var.type # Equivalent

Neat! It all exists in libraryspace, so it’s not a groundbreaking change to the language. Still, this has to save time. Without this creating the same sort of class would have taken something like:

class variable(object):
    def __init__(self, id, name, typ, size):
      self.id, self.name, self.type, self.size = id, name, typ, size

Not too much typing, but certainly more, and you can’t even access it as a tuple without additional work. I wonder if the various Python ORM libraries use techniques like this?

Inspired by Erich’s dice-by-cards Catan variant (and also by not having a spare deck of playing cards, and also by having recently taken pictures of a six-sided die) I put together Fair Dice. It’s a goofy little JavaScript application that will roll all thirty-six combinations of two dice once in random order. When it’s done, it will do it again.

Now, is this sort of thing useful? Who knows. Probably half the fun of a game like Settlers is watching people fume when threes and elevens come up over and over and over again while those sixes and eights that they so wisely choose at the beginning of the game go fallow. So you know me: I like to destroy fun.

A random non-gaming aside: How do people write web pages without FireBug? I looked briefly at making sure that this works correctly on IE and Safari, and I couldn’t tell what in the world was going on. For the longest time the page didn’t do anything because apparently Safari and IE (IE 6, at least) don’t understand the “const” keyword in JavaScript. Infuriating! Anyway, if you’re having to contend with JavaScript, I can’t recommend FireBug quite enough.

(Fair Dice)

Can you see the disaster in progress? Most users can’t either.

This dialog has buried the lede. It focuses the user’s attention on the fact that there is another file with the same name in the destination folder. It fails to point out a much, much more interesting condition: The user is about to overwrite a newer file with an older file.

[...]

The above dialog’s text fails at this, as does its layout and typography. There are numerous pieces of text competing for attention, but among the most prominent are the bolded file names. That’s a bit odd, since the entire premise of the dialog is that these two file names will be the same. The dialog has carefully drawn the user’s attention to information which is guaranteed to be redundant. (If the user is moving multiple files, only some of which have conflicts, the file names are relevant—but that case can and should be handled specially.)

I’ve never thought consciously of following the inverted pyramid while laying out a UI, but it makes a lot of sense. I’m curious to look in on some of the screens I’m working on currently to see how well I stick to this principle.

Thank you for releasing your CLARAty software to the public. That is a wonderful move that we can all appreciate. However, I have one comment regarding your installation procedures: Csh? Really?

Good Day.

Here’s a brief example

DoublerApplet.java

package org.obfuscated.example;

import java.applet.Applet;

public class DoublerApplet extends Applet {
    public double f(double x) {
        return x + x;
    }
}

Nothing special there. Note that I’ve not yet tried passing classes or the like to an applet. I imagine that they might be trouble.

index.html

<html>
  <head>
    <title>Applet Test</title>
    <script src="prototype.js"></script>
    <script src="doubleit.js"></script>
  </head>
  <body>
    <object id="doubler"
       classid="java:org.obfuscated.example.DoublerApplet.class" 
       type="application/x-java-applet"
       archive="DoublerApplet.jar" 
       height="1" width="1" >
      <param name="scriptable" value="true"/>
    </object>

    <b>Double me!</b>
    <br/>
    2 * <input size="5" type="text" id="x"/> = <span id="answer">???</span>
    <br/>
    <input type="submit" id="go"/>
  </body>
</html>

The important part is the “scriptable” parameter. It tells the applet to listen to the JavaScript.

doubleit.js

var init = function() {
    Event.observe('go', 'click', function() {
        var x = Number($('x').value);
        var applet = $('doubler');
        $('answer').innerHTML = applet.f(x);
        return false;
    });
};

Event.observe(window, 'load', init);

The “$” magic that makes the JavaScript code quite so terse is Prototype, which I can’t quite recommend enough.

Anyway, I’m quite pleased that I’m able to use this technique, as dragging out Tomcat, keeping it running, etc., would be a whole lot of work just to be able to call one function. (Which is all I need to do in this particular case.) I’m sure the server-based solution would be more correct if I needed to do more in Java-land, but for quick and dirty things or one-off’s, JavaScript to Applet communication is nice.

Each of these articles are incredibly useful. You have to know how to get data in and out of your program, and there aren’t a lot of tasks that don’t require parsing data. The two techniques together, as you might expect, are crazy useful.

To summarize perhaps too quickly, you can do simple stdin/stdout IO using the “interact” function. Here’s the Haskell implementation of the popular “tac” utility:

main = interact $ unlines . (map reverse) . lines

The “lines” method turns a large string (stdin) in to a list of strings, “unlines” turns a list of strings in to one large string (stdout), “interact” ties the whole process together, and “(map reverse)” is where you would put your code.

In place of “lines” you can usefully drop Parsec in such that you can work directly on data structures instead of fussing with lines of text. (Haskell isn’t Perl, after all.) Here’s a quick example that reads pairs of integers:

module Main where
import Text.Printf
import Text.ParserCombinators.Parsec

atoi :: String -> Int
atoi = read

intsList :: Parser [(Int,Int)]
intsList = do
  intsList' <|> (eof >> return [])
      where intsList' = do
              a <- many1 digit
              many1 space
              b <- many1 digit
              newline
              r <- ( many space >> (intsList' <|> return []))
              return ((atoi a, atoi b) : r)

parseInput :: String -> [(Int,Int)]
parseInput s = case parse intsList "stdin" s of
                 Left err -> []
                 Right cs -> cs

rpt :: (Int,Int) -> String
rpt (a,b) = let c = maximum $ map ( length . cyc ) [a..b]
            in printf "%d %d %d" a b c
    where cyc n | 1 == n       = [1]
                | 1 == mod n 2 = n : cyc (1 + 3 * n)
                | otherwise    = n : cyc  (div n 2)

main = interact $ unlines . (map rpt) . parseInput

And just like that you can move input parsing in to a parser where it belongs and not worry about it in the rest of your system. Certainly a lot more work than the same thing in, for instance, Ruby:

STDIN.each { |l| a,b = l.split.map {|x| x.to_i} ; puts whatever(a, b) }

However, dropping Parsec works the same way for arbitrarily complex data structures. This same code structure workspairs of integers, HTTP, JSON, whatever. Needless to say, I find this very pleasing.

Just imagine the difference we could make if, as an industry, we turned each of these conferences into a chance to raise much needed money for worthy charities. Imagine if, rather than getting yet one more burlap bag with a sponsor’s name on it, you instead got a slip of paper saying that the price of that bag was being used to buy vaccine for 5 kids, or a book for a literacy project. Imagine what could happen if a conference with 5,000 attendees raised just $20 per attendee. Then imagine $50, or $100. It starts to get serious.

As somebody who falls squarely in to the “I’m not going to use your Complete Enterprise Solution even if you give me a tote bag and a glow-in-the-dark yo-yo” camp, I think this is a great idea. Nobody needs another Sun-branded pen or Google Frisbee; there are lots of people out there who could make good use of a nerd’s pocket change.

My original plan for my convalescence was to go charge a book on JavaScript to the company account and see if I couldn’t learn a bit about a language that I’m likely going to use for an upcoming project. However I don’t know what the good JavaScript books are, and walking to the book store is a lot more work than surfing the internet, so I checked for good online resources.

It turns out that there are many. In particular, Yahoo!’s YUI team has put together large collection of videos about JavaScript. I watched videos of three of Douglas Crockford’s lectures: “The JavaScript Programming Language,” “An Inconvenient API: The Theory of the DOM” and “Advanced JavaScript.” All three were good lectures. They were, for shots of a lecture about a programming language, well produced, and Crockford’s talks were well structured. I got a lot more out of the first talk than the other two, but that could arguably be couched more as a compliment for the introduction video than a knock on the other two, as I felt like I knew the language pretty well after it. (I’d never used the language seriously before.)

I didn’t get too far in to any of the other videos, but there are a ton of them there, so there’s got to be at least a few more good ones. If you’re looking to pick up JavaScript, I can recommend the YUI Theater as a good place to start.

import Control.Monad

powerset :: [a] -> [[a]]
powerset = filterM (const [True, False])

I saw this over on a rather good introduction to dynamic programming. It does exactly what you’d expect it to do. (i.e., powerset [1,2] = [[1,2],[1],[2],[]].)

Outstanding.

From Ruby on Grails? Why the hell not?