Sorry for the delay in responding to this – the hazards of family holidays :-).

I’m not sure why the old link doesn’t work anymore. I found the new link, and have edited the link in the article above. Thanks for pointing out the problem!

Oy! Link to preprint/paper is dead. Have we another recourse, or has the evil Springer empire eaten the thing completely and hidden it from public eyes?

OK, I’m not a scientist (nor do I play one on TV) and though I’m not sure what you might use this N-gram stuff for, I think it is interesting to read about.

A long, long time ago (1991 SFI Summer School, I think) we played with masked long-range N-grams as a way of capturing multi-scale linguistic structure. Admittedly, we were considering texts and CA patterns character-wise, not word-wise, but the idea still seems applicable and interesting.

Most N-gram datasets consider only contiguous atoms, as in strings of three characters, or runs of three words. Back in the Dark Ages we looked at “masked” runs with “don’t care” spaces in them. For example, as I recall we looked at how well “English-like” texts were reproduced by standard 4-grams [1234], compared to masked schema like [12#3##4] and [1##23##4], where “#” indicates a “don’t care” space-holder.

There was some math, and some fancy statistics we did on which masks were “better” for particular tasks… but it was just Summer School play. We looked at a limited subset of possible masks, and spent most of our time considering how to determine “Englishness” automatically. And on Mac IIs and first-generation NeXT boxen, so very very slowly.

I’m not sure if anybody ever followed up on it.

As I see it, you and Riccardo are essentially “lossily compressing” the kind of subtree archives that are useful in Conor Ryan’s and Guido Smits’s and others’ work in GP. That seems like a great idea, and a challenge we’re tackling right now in practice here in Ann Arbor. We’re building a Push-GP system, and wanted to use “subtree” archives (Push programs aren’t really trees) to accelerate and do pattern mining. I think we’ll look at your approach instead.

And I can see how this would really boost the exploration/exploitation angle in GP: I can use your predictive EDA rules in an ongoing run of GP to bias search towards successful patterns (as you have), but can use the same infoto hare off into unexplored territory by invoking some kind of inverse. By avoiding what’s been shown to be useful before.

Maybe somebody should do that right away. (He said, pointedly.)

Very cool. I’d thought about turning my bit of Ruby code into a web app so people could play, but hadn’t actually done anything about it, in part because I assumed that there were probably similar things out there on the web already.

One important difference between the tool you found and the program I used is that he works at the character level, and I worked at the level of words. Thus his program can generate “words” that aren’t (like “linguish” and “documential”), whereas the words from my program will always be real, and the breakdown will be more at the sentence level.

I hadn’t thought about the relationship to spam text, but it does look a lot like the stuff they tend to include. I suspect this is not a coincidence!

I found an online tool that creates N-grams (http://jonathanwellons.com/n-grams/index.cgi) and ran your “intro to N-grams” through it a couple of times to see what I would get:

N=1, generate 20 characters:
TTTTTTTTTTTTTTTTTTTT

N=5, generate 600 characters:
t (a book, say you can capture, but we table the beginning, as well.A quick is to ment; we could be very useful computational linguish between two texts (although if therefore, are lots of then N=1), while the butler whenever, does the but we table following sequently the otherefore, are words within a technical paper once at texts (does the frequency of triplets overlap, so we’re us and however, doesn’t really words, and we’re simply counting can capture, are like when words) that triplets), once in the documential idea is to looking up the beginning, as well.A quick is meaningful compromise

Looks interesting. The second example looks like the “classic prose” that spammers use to get their junk mail past spam filters.

I like the term “computational linguish” to describe it. Maybe we can evolve that term and find a more successful one. :)