A momentary interruption in the (slow) posting on the road trip (which has been done for nearly two weeks now!) to provide a time sensitive plug for those of you interested in genetic programming, evolutionary computation, and the like.

The Springer journal Genetic Programming and Evolvable Machines is celebrating its first 10 years with a special anniversary issue of articles reviewing the state of the field and considering some of its possible futures. For the month of July (which ends in two days!) the entire issue is available for free download.

Included in the issue are:

• Human-competitive results produced by genetic programming
• Theoretical results in genetic programming: the next ten years?
• Genetic Programming and Evolvable Machines: ten years of reviews
• Open issues in genetic programming
• Grammar-based genetic programming: a survey
• Developments in Cartesian Genetic Programming: self-modifying CGP
• Bio-inspired artificial intelligence: theory, methods, and technologies

Once the month ends these will all start costing money again with two exceptions: the article on human-competitive results and the survey of 10 years of reviews will remain free in perpetuity.

In the spirit of full disclosure, I’m on the editorial board of the journal and contributed to one of the articles. Still, it’s a cool resource marking an interesting time in the development of the field, so take advantage of it while you can!

photo credit: Unhindered by Talent

When I edit, I tend to scribble a lot, even when it’s my own stuff (or the writing of people I really like). Last January, for example, I took a set of photos after scribbling all over a paper that Riccardo and I were working on for GECCO. This paper went on to win the Best Paper award in the genetic programming track at GECCO last month, so I’d like to think that all this editing had some value :-).

I posted the full set over in my events account, and I plan on using some of them to show my students that I’m not just being mean to them — I’m mean to everyone, myself included!

This showed up here now because a publisher contacted me about using it in a college writing textbook. I figured I’d clean it up and post the full size version.

I’ll try to post at least sporadically in the upcoming weeks, but I suspect things are likely to be light here well into the summer.

We won!

That a paper co-authored with two undergraduates from a small, public, undergraduate liberal arts institution like the University of Minnesota, Morris, could win an award like this at an international science conference is just too damn cool. Well done to both Brian and Tyler!

In the hectic melee of the conference, most folks don’t have time to do anything more than skim the nominated papers, and usually not even that. This makes the talks a crucial part of an award like this, as much of the voting is based on them. Tyler (pictured above at Castel dell’Ovo in Naples) was a huge help in that regard. He flew over to the UK several days early so we could work on our talk, giving us the time we needed to revise and practice. He also produced a super cool little six page comic with a nifty introduction to our work that the audience could follow along with. We did a joint presentation, each covering about half the paper. Our talk was well received, and Tyler’s comic was incredibly (and deservedly) popular, and there’s no doubt that his participation was a huge help.

(And all this is on top of Brian and Tyler’s hard work and contributions on the paper itself. Obviously without that content we never would have had the paper accepted or nominated in the first place. So they both deserve huge kudos for that as well.)

Friday morning our paper was voted Best Paper by the conference attendees, and we were presented with a certificate, a box of Italian lemon cookies, and a box of Irish chocolates. All the Best Paper winners from the various EvoStar conferences and workshops also got to choose a free book from the Springer table. Tyler got a really cool book enitled Leonardo’s Lost Robots, and I got The forgotten revolution: How science was born in 300 BC and why it had to be reborn. (It was all terribly liberal arts of us – pretty much everyone else took evolutionary computation/artificial intelligence books of one form or another.)

Tyler upheld a fine tradition of our students making UMM look really good at conferences like this. From his deportment and grasp of the material, most people assumed he was a graduate student, despite the fact that the looks like he’s about 16 :-). He’s currently doing contract work as a web developer and designer, but is seriously interested in going to graduate school in the near future, and he definitely impressed the folks at the conference. I’ve been really lucky to work (and co-publish) with a string of great UMM students, and am looking forward to continue that with a very sharp student named Sara Lahr when we get back.

The trick for me (sometimes) is remembering just how good our students can be. The room we spoke in was this grand space of inlaid wood and marble that was quite a surprise in several ways. This was made worse by the fact that we were in the first session, so we had very little time to adjust and adapt. I was worried about running long (we had a lot of material to cover), and started to lose my nerve about having Tyler wandering around the room at the beginning handing out the comic. Tyler was really calm and collected about it, though, talked me down, and everything did in fact go really smoothly. The moral? Handouts are Good, really cool comics handous are Even Better, and I need to remember to listen to my students :-).

Thanks a ton to Brian and Tyler and all the people and offices at UMM that supported our work, and everyone who voted for our paper at EuroGP! Special thanks also to Riccardo Poli for hosting me on this sabbatical at the University of Essex. I’ve gotten a ton of cool work done here with Riccardo, including “A linear estimation of distribution GP system” at EuroGP, which was also nominated for Best Paper (and which I suspect was also strongly in the running).

I’ve dumped all the photos Tyler and I took in Naples onto my events account on Flickr. I’ll try to clean up a few to post to my main Flickr account in the next week or so.

The book was a big hit with the Evo* crowd, and we sold out the full set of 50 pretty quickly. Lots of folks had us autograph their copies, many of which are now destined to be collector’s items with the signatures of all three of the authors. Some even have the signature of Tyler Hutchison, who did the nifty cover art for us and helped a lot with the roll-out.

There were tons of photos taken at our booth, including candids of us signing and working the crowd, and posed shots with our cool Field Guide shirts. People have promised to send us photos and links, so check back in the next week or two for some of the finest in EC book release amateur photojournalism! (And if you’ve got a photo from the event, or a nifty shot of your copy in its place of pride on your bookshelves, please pass it along.)

As mentioned before, the book is now officially released and available to any and all via lulu.com, both in an inexpensive printed form (what we were selling last night) and as a free downloadable PDF.

So go check it out – 50 whole Field Guide fans can’t be wrong!

I’ve been sitting on this for a while I waited for EuroGP to get their web site updated; they have, freeing me up to do a little unabashed chest-thumping, leavened with some praise for UMM‘s excellent students.

A few weeks from now the Eleventh European conference on Genetic Programming (EuroGP) will happen in Naples Italy. The only other time I’ve been to EuroGP was in 2001 when we were here on sabbatical the first time. I really love the conference, and it’s small and intimate and tends to have a really high signal-to-noise ratio. Unfortunately, it’s also an expensive flight from Minnesota for a three day event, and the timing tends to be really awkward in my teaching schedule, so I’ve never made it back. One of my many fond memories of that conference was winning the best paper award with Riccardo Poli for a pair of papers we’d written together as part of that sabbatical visit. (Oddly, both times I’ve attended have happened to be the only two times it’s been in Italy. The little photo is from the 2001 event at Lake Como.)

The best paper nominations for this year’s event have been released, and I’m quite excited that both of the papers that I submitted this year are on the list. One is another join project with Riccardo, and the other is a paper with two UMM undergrads: Brian Ohs and Tyler Hutchison.

That’s Tyler in the photo up top, presenting some work he did with Andy Korth and I that won the best student paper award at MICS a year ago; Tyler also did the cover illustration for the forthcoming book Riccardo, Bill Langdon, and I are just wrapping up. In a big happy, Tyler was able to pull together the funds to fly out for the conference, so we’ll be able to do a joint presentation enlivened by his presence and cool drawings. Unfortunately Brian can’t make it, but it’s cool that Tyler can; this will be the first of my students co-authors that’s made it to a European conference with me.

The competition is gonna be tough for the best paper award, including a very nice paper by one of Riccardo’s students (Stephen Dignum). Fingers crossed!

The full program is also now on-line (as a PDF) — it looks like some cool material. I’m quite looking forward to the conference, although I must say I’m a bit nervous about the ongoing trash crisis in Naples (here and there).

Sometimes you’re just minding your own business, trying to get a little science done, and a little art pops out at you all unexpected.

What?!? You want to know where this comes from? All is revealed beneath the fold…

The dotted line is, not surprisingly, the sine function plotted from about -10 to +15. The dashed line is a crazy rational function evolved using genetic programming. The bold line shows where the test data came from that was used to evaluate the quality of the evolved solutions, and it’s clear that in that range the evolved function matches very closely.

It’s also clear that outside of that range (which was never part of the evolutionary training), the evolved function rapidly wanders off into its own happy place (including a nifty discontinuity at around x=10). This is in fact to be expected. The GP system was only allowed to use +, *, -, and /, i.e., all it could construct were rational functions. There are no rational functions that match sine across its entire range, so this is really about as good as we could do given those tools. If we’d given the GP system a periodic function (like cosine) in its toolkit, then it could potentially evolve something that would match sine everywhere. With rational functions, though, it just ain’t gonna happen.

But this graph looks a lot cooler than it would if they matched everywhere :-).

Oh?

The function that it actually evolved?

You sure you want to see that?

Well, OK…

where

• a = 2.76609789995
• b = 10.240744822
• c = 3.9532436939
• d = 3.20011637632
• e = 12.6508398844

I never promised it would make a lot of sense, did I? Evolution often doesn’t produce pretty. So much for intelligent design…

It’s pretty cool when you update your repository and see

At revision 400

We just hit that on the genetic programming book that Riccardo and Bill and I are working on; we’re currently averaging close to 10 commits a day here in the final stages. We hope to wrap it up in the next 1.5 weeks and then off to the printers for fun (and no profit in the traditional sense)!

P.S. Anyone want to proof read a few pages? Get in touch and we can work something out.

Just finished my GECCO reviewing, and I must say that is seriously sucks when people don’t attend to even the most basic of issues. Two things almost guaranteed to majorly annoy a reviewer:

• Weird random floating fragments of text that are obviously the disemboweled remnants of some cut and paste action.
• Only 8 entries in the bibliography on a subject that has been heavily researched for over a decade.

And just guess the average publication date of 8 fine references.

1985.

Yeah, over 20 years ago.

3 entries were books (two of which were over 10 years old), and the only 2 journal articles were from 1938 and 1964 respectively.

Strangely enough, I didn’t encourage acceptance of that paper.

The really depressing thing is that most of our (undergraduate) students at UMM would do better than this.

No, maybe that’s the uplifting thing.

Maybe the really depressing thing is that I see papers with this kind of bibliography fairly often. I’d almost recommend some sort of automatic rejection system for obviously stupid bibliographies, but then people would just start padding their bibliographies with random citations to get past that blockade.

Sigh.

photo credit: markopoulos

“I would have sent to Rome that’s worthy death?”

“The naturalist looking at species as he might succeed from a fork low down in the separation of the species of any species in a more or less from their sap this is unimportant for the instincts already possessed by certain plants so that natural selection of mere piles of superimposed strata and watch the sea separating an island even if we believe that pure water can effect little or no offspring.”

“The troubling aspects of a building block semantics in a given tree in the context and false.”

The answer and (much) more is below the fold.

That probably wasn’t too hard.

Except, of course, I’m a mean person and it was a trick question. In fact none of those sentences was actually written by a human being. All three of these pseudo-quotes were generated using simple statistics and the notion of N-grams (which I will describe below) to capture and reproduce the sense of a text, without really understanding it at a larger scale. I wrote a program to create these examples as an opening illustration for my Dagstuhl talk, and since they’re a lot of fun I thought I’d summarize the key concepts here.

A quick intro to N-grams

The essential idea is to look at the frequency of the occurrences of groups of adjacent symbols (called N-grams). In this example the symbols are words, and we’re using N=3, so we’re looking at triplets of adjacent words. The preceding sentence, for example starts with the following sequence of triplets:

• (“In”, “this”, “example”)
• (“this”, “example”, “the”)
• (“example”, “the”, “symbols”)
• Etc.

Note that the triplets overlap, so most words will in fact appear in three N-grams (triplets), once at the end, once in the middle, and once at the beginning, as happens for “example” above.

Let’s say you go through a text (a book, say), and make a great big list of all the N-grams and how often they each occur. What would that tell us about that text? That depends pretty crucially on the value of N. If N=1, for example, then we’re simply counting up the frequency of individual words. This could be very useful in certain circumstances; we could, for example, probably use that to decide whether the text is a crime novel or a technical paper on econometrics. N=1, however, doesn’t give us any sense of the larger structure of the document; we may find it difficult to distinguish between two particular novels, and we’d be unable to recognize when words tended to appear in related contexts (does the author tend to mention the butler whenever missing candlestick is discussed?).

If, on the other hand, we take N to be much larger (several 100s or 1,000s of words) then we capture lots of long range structure, but we lose any kind of statistics. Almost all sequences of 100 consecutive words, for example, are likely to be unique. Consequently they will tend to only appear once in our table for a given text, and are unlikely to appear in the tables for any two texts (although if they did that would be a useful plagiarism flag). Very large sequences don’t really work, therefore, as a tool for looking for relationships between documents, or even patterns within a text.

Experience in fields such as computational linguistics suggest that taking N=3 is a useful compromise position. [1] Different triples occur often enough (assuming the text has some length) that their distribution is meaningful (like when N=1), while there’s enough structure and overlap with N=3 that you can capture some long-range regularities as well.

Generating my “pseudo-quotes”

All of which is illustrated by my pseudo-quotes above. Each was generated by first computing the frequency of triples in three texts: Coriolanus, The origin of species, and a paper I wrote last fall with a couple of UMM students. [2] I also kept track of the frequency with which words occurred as the first word of a sentence, and the frequency of pairs containing the first two words in every sentence.

The program starts generating a sentence by picking a first word based on the frequency of first words (so if 20% of the sentences started with “The”, there would be a 20% chance of starting our new sentence with “The”). Given that choice, the program would take the table of frequencies of all the pairs of first two words, and pull out just those pairs that used our chosen first word. Once we have the first two words, we can repeatedly generate words based on the frequencies of 3-grams in our big table. If we’re using Coriolanus, and the last two words we’ve generated were (in order) “I” and “would”, then the relevant triples that occur in that play more than once are:

with another 11 “I would” triples that occurred a single time. Thus “I would have” (the prefix we chose in the “Shakespearean” sentence above) was more likely to be chosen than “I would he” by a ratio of 3:2, and three times more likely to be chosen than any of the triples that occurred just once. It was, on the other hand, less likely to be chosen than either “I would they” or “I would not”, but the (digital) dice rolled in its favor on this particular run.

Now that we have that start, we continue choosing words based on the last two that we’ve added to the sentence. So we look in our table at what triples start “would have”, and chose (this time) “sent” as the next word. We then look up “have sent” and chose “to”. The process continues (in my program at least) until we choose a “word” that’s in fact a terminal punctuation mark (a period, a question mark, or an exclamation point), thus ending the sentence.

A few observations

As we can see from the pseudo-quotes that I opened with, this process can generate sentences whose style is clearly recognizable, and which can make a great deal of sense at the local level (e.g., at the level of phrases). This process, however, doesn’t “understand” or respect larger structural connections or semantics. The third pseudo-quote, for example,

“The troubling aspects of a building block semantics in a given tree in the context and false.”

lacks a verb. The large Darwin pseudo-quote

“The naturalist looking at species as he might succeed from a fork low down in the separation of the species of any species in a more or less from their sap this is unimportant for the instincts already possessed by certain plants so that natural selection of mere piles of superimposed strata and watch the sea separating an island even if we believe that pure water can effect little or no offspring.”

rambles all over the shop, containing very sensible phrases like “The naturalist looking at species” as well as nonsense such as “pure water can effect little or no offspring”.

What we did with it

In the research I was reporting on, Riccardo Poli and I applied this idea of N-grams as a tool for capturing regularities in a language to computer programs that were represented as sequences of simple “machine” instructions in a highly simplified programming language that was designed for a specific set of test problems. We used a type of Estimation of distribution algorithm (EDA) to essentially evolve the triplet frequencies, instead of taking them from a text like I did with Coriolanus, and then used those to generate programs much like I did with the text examples. We would generate a set of, say, 100 programs this way, and try each of them on our test problem. Some were better than others, so we’d take the better ones (say the top half) and use those to update the frequencies; triplets that appeared in those better programs would have their frequencies increased somewhat, while those that didn’t would have their frequencies reduced.

When repeated over several generations, this process would evolve/learn/find/discover a set of frequencies that allowed it to generate successful programs with a reasonably high probability, at least on problems that had solutions that could be formed from repeated sequences of instructions. Even though it was limited to only 3-grams, the system was able to “learn” some fairly long sequences of instructions. In one case, for example, the evolved set of probabilities generated, given a particular starting pair, a particular sequence of 9 instructions with probability of over 60%, which is some 500,000 times more likely than generating a sequence of that length by randomly drawing instructions from a hat. The solutions that were generated tended to be composed of numerous copies of a small number of basic patterns, and it seems likely that this approach will do better in problem spaces where there are solutions that exhibit that kind of regularity.

If you’re interested in learning more, check out the paper: “A Linear Estimation-of-Distribution GP System”.

[1] Does anyone know of a theoretical justification for this?

[2] ”Semantic building blocks in Genetic Programming”, which will be appearing next month in the Proceedings of the European Conference on Genetic Programming (provide link).