The cool part is that most of the photos are mine! This set on Flickr shows all the photos they used, although many of them actually look much better in the calendar. Their staff did a really great job of straightening, cropping, and adjusting the lighting on the shots that they used, and it really made the photos look really nice. Thanks to whoever did the excellent work! Christian Lindig informs me (see the comments) that the design work was done by Margot Behr. Thanks for the great work, Margot!

It was really weird when I first looked at the calendar, because I really wasn’t sure how many of the photos were mine. There were two or three that I immediately recognized as mine (like the image at the top), but there were quite a few indoor detail shots that seemed like the kind of thing that I’d take (like the dragon below), but which I didn’t really recognize. There were also several of buildings that could have been mine, but could have been taken by most anyone. Going through, them, though, I was able to determine that all but two were in fact mine. The cropping (and to a lesser degree the cleaning) that the Dagstuhl folks did often threw me as it sharpened the focus in cool ways that I hadn’t seen or thought of.

There are two photos in the calendar that aren’t mine, both taken in specific conditions that I’ve never been there for. One is a really cool panorama with a beautiful fresh coating of snow and the other is the grounds around the chapel in the summer.

Regular readers here will likely remember various past posts extolling the virtues of Dagstuhl, this really wonderful computing research facility in Germany. I’ve been lucky enough to attend several seminars on the Theory of Evolutionary Algorithms in the past, and have taken (and posted) quite a few photos from those trips.

A few months ago I had the (totally) unexpected pleasure of being contacted by Christian Lindig, a member of Dagstuhl’s scientific staff, and asked if I would be willing to return to Dagstuhl for the specific purpose of taking photographs for them. They’re apparently in the process of re-doing all their brochures and such, and liked what they’d seen on Flickr.

As a result I’ve spent a very pleasant few days back at Dagstuhl in some gorgeous spring weather (I’ve only every been here in the winter before), enjoying the company of a fine group studying the Design and Analysis of Randomized and Approximation Algorithms. (They’ve been very patient with an intruder who always seemed to be waving a camera about, and for this I am grateful.)

I’ve taken something over 1,000 photos, with a few more planned for tomorrow morning before I leave. Some of the participants have asked if the photos will be made available. I have uploaded everything I’ve taken this week to my “events” account on Flickr, all under a Creative Commons license, which means that participants are welcome to download and use any that they wish, as long as they provide attribution. (Note that I am not, and can not, provide any sort of permissions from individuals photographed – that’s their right and prerogative. If you want to use someone here in a beer ad, you need to get their permission first.)

Where are they all?

• Everything I’ve taken this week is on my “events” account.
• I also have a Dagstuhl set on my “real” Flickr account that has some of my favorites (with cleaning, cropping, etc.) from my various visits to Dagstuhl. At the moment this set is heavy on shots from previous trips, but I hope to add more from this visit as time allows.
• Earlier this year I posted everything (without cleaning or editing) from the Jan/Feb seminar on Evolutionary Algorithms to my “events” account.

It’s not yet clear what all Dagstuhl may do with these, but they will contact anyone who is recognizable in a photograph for permission before using your image in any of their materials. If anyone objects to being included in these vast oceans of photos, please let me know and I’d be happy to remove the photo in question.

Thanks again to Christian for inviting me out to Dagstuhl, and to the Randomized and Approximate Algorithms group for being patient “hosts”. Enjoy the photos!

Almost had an alliteration in the title, but then lost it at the end. Sigh.

I’ve finishing dumping all my Dagstuhl photos (uncleaned and unedited) to my event account on Flickr, so those with more time than sense can rush over and gaze upon them all. Over the next week or two I’ll work on cleaning some of my favorites and posting them to my “real” Flickr account, but who knows how long that will take.

“I would have sent to Rome that’s worthy death?”

“The naturalist looking at species as he might succeed from a fork low down in the separation of the species of any species in a more or less from their sap this is unimportant for the instincts already possessed by certain plants so that natural selection of mere piles of superimposed strata and watch the sea separating an island even if we believe that pure water can effect little or no offspring.”

“The troubling aspects of a building block semantics in a given tree in the context and false.”

The answer and (much) more is below the fold.

That probably wasn’t too hard.

Except, of course, I’m a mean person and it was a trick question. In fact none of those sentences was actually written by a human being. All three of these pseudo-quotes were generated using simple statistics and the notion of N-grams (which I will describe below) to capture and reproduce the sense of a text, without really understanding it at a larger scale. I wrote a program to create these examples as an opening illustration for my Dagstuhl talk, and since they’re a lot of fun I thought I’d summarize the key concepts here.

A quick intro to N-grams

The essential idea is to look at the frequency of the occurrences of groups of adjacent symbols (called N-grams). In this example the symbols are words, and we’re using N=3, so we’re looking at triplets of adjacent words. The preceding sentence, for example starts with the following sequence of triplets:

• (“In”, “this”, “example”)
• (“this”, “example”, “the”)
• (“example”, “the”, “symbols”)
• Etc.

Note that the triplets overlap, so most words will in fact appear in three N-grams (triplets), once at the end, once in the middle, and once at the beginning, as happens for “example” above.

Let’s say you go through a text (a book, say), and make a great big list of all the N-grams and how often they each occur. What would that tell us about that text? That depends pretty crucially on the value of N. If N=1, for example, then we’re simply counting up the frequency of individual words. This could be very useful in certain circumstances; we could, for example, probably use that to decide whether the text is a crime novel or a technical paper on econometrics. N=1, however, doesn’t give us any sense of the larger structure of the document; we may find it difficult to distinguish between two particular novels, and we’d be unable to recognize when words tended to appear in related contexts (does the author tend to mention the butler whenever missing candlestick is discussed?).

If, on the other hand, we take N to be much larger (several 100s or 1,000s of words) then we capture lots of long range structure, but we lose any kind of statistics. Almost all sequences of 100 consecutive words, for example, are likely to be unique. Consequently they will tend to only appear once in our table for a given text, and are unlikely to appear in the tables for any two texts (although if they did that would be a useful plagiarism flag). Very large sequences don’t really work, therefore, as a tool for looking for relationships between documents, or even patterns within a text.

Experience in fields such as computational linguistics suggest that taking N=3 is a useful compromise position. [1] Different triples occur often enough (assuming the text has some length) that their distribution is meaningful (like when N=1), while there’s enough structure and overlap with N=3 that you can capture some long-range regularities as well.

Generating my “pseudo-quotes”

All of which is illustrated by my pseudo-quotes above. Each was generated by first computing the frequency of triples in three texts: Coriolanus, The origin of species, and a paper I wrote last fall with a couple of UMM students. [2] I also kept track of the frequency with which words occurred as the first word of a sentence, and the frequency of pairs containing the first two words in every sentence.

The program starts generating a sentence by picking a first word based on the frequency of first words (so if 20% of the sentences started with “The”, there would be a 20% chance of starting our new sentence with “The”). Given that choice, the program would take the table of frequencies of all the pairs of first two words, and pull out just those pairs that used our chosen first word. Once we have the first two words, we can repeatedly generate words based on the frequencies of 3-grams in our big table. If we’re using Coriolanus, and the last two words we’ve generated were (in order) “I” and “would”, then the relevant triples that occur in that play more than once are:

with another 11 “I would” triples that occurred a single time. Thus “I would have” (the prefix we chose in the “Shakespearean” sentence above) was more likely to be chosen than “I would he” by a ratio of 3:2, and three times more likely to be chosen than any of the triples that occurred just once. It was, on the other hand, less likely to be chosen than either “I would they” or “I would not”, but the (digital) dice rolled in its favor on this particular run.

Now that we have that start, we continue choosing words based on the last two that we’ve added to the sentence. So we look in our table at what triples start “would have”, and chose (this time) “sent” as the next word. We then look up “have sent” and chose “to”. The process continues (in my program at least) until we choose a “word” that’s in fact a terminal punctuation mark (a period, a question mark, or an exclamation point), thus ending the sentence.

A few observations

As we can see from the pseudo-quotes that I opened with, this process can generate sentences whose style is clearly recognizable, and which can make a great deal of sense at the local level (e.g., at the level of phrases). This process, however, doesn’t “understand” or respect larger structural connections or semantics. The third pseudo-quote, for example,

“The troubling aspects of a building block semantics in a given tree in the context and false.”

lacks a verb. The large Darwin pseudo-quote

“The naturalist looking at species as he might succeed from a fork low down in the separation of the species of any species in a more or less from their sap this is unimportant for the instincts already possessed by certain plants so that natural selection of mere piles of superimposed strata and watch the sea separating an island even if we believe that pure water can effect little or no offspring.”

rambles all over the shop, containing very sensible phrases like “The naturalist looking at species” as well as nonsense such as “pure water can effect little or no offspring”.

What we did with it

In the research I was reporting on, Riccardo Poli and I applied this idea of N-grams as a tool for capturing regularities in a language to computer programs that were represented as sequences of simple “machine” instructions in a highly simplified programming language that was designed for a specific set of test problems. We used a type of Estimation of distribution algorithm (EDA) to essentially evolve the triplet frequencies, instead of taking them from a text like I did with Coriolanus, and then used those to generate programs much like I did with the text examples. We would generate a set of, say, 100 programs this way, and try each of them on our test problem. Some were better than others, so we’d take the better ones (say the top half) and use those to update the frequencies; triplets that appeared in those better programs would have their frequencies increased somewhat, while those that didn’t would have their frequencies reduced.

When repeated over several generations, this process would evolve/learn/find/discover a set of frequencies that allowed it to generate successful programs with a reasonably high probability, at least on problems that had solutions that could be formed from repeated sequences of instructions. Even though it was limited to only 3-grams, the system was able to “learn” some fairly long sequences of instructions. In one case, for example, the evolved set of probabilities generated, given a particular starting pair, a particular sequence of 9 instructions with probability of over 60%, which is some 500,000 times more likely than generating a sequence of that length by randomly drawing instructions from a hat. The solutions that were generated tended to be composed of numerous copies of a small number of basic patterns, and it seems likely that this approach will do better in problem spaces where there are solutions that exhibit that kind of regularity.

If you’re interested in learning more, check out the paper: “A Linear Estimation-of-Distribution GP System”.

[1] Does anyone know of a theoretical justification for this?

[2] ”Semantic building blocks in Genetic Programming”, which will be appearing next month in the Proceedings of the European Conference on Genetic Programming (provide link).

All good things must come to an end, and our week at Dagstuhl ends tomorrow after lunch. Above is yesterday’s group picture (I’m in yellow near the front) just before the traditional Wednesday hike (below), which was wet and misty but still an enjoyable few hours out in the world.

I gave a talk this morning which (I think) went well, especially since I didn’t know I was going to be giving a talk until Tuesday afternoon! There was certainly a lot of good discussion and people came up with tons of suggestions and ideas, which is what I really love about presenting at Dagstuhl. I started with something quite fun, which I’ll post here later. Most of the talks have used computer slides (PowerPoint or some more sensible alternative like LaTeX/Beamer), but Jon Rowe did a great blackboard talk on Tuesday (pictured below).

I was greatly inspired and did almost all of mine on the boards as well. I had four slides at the beginning that really needed to be slides, and then I did the bulk on the boards, and came back to a fifth slide at the end.

Tomorrow there are talks before lunch, including a talk/discussion thing that Riccardo is doing that I’m sure I’ll be roped into in some mysterious way. Then we eat, and it’s a taxi out to the Frankfurt Hahn airport for our flight back to the UK!

This is from the morning break here on our first full day at Dagstuhl. I love the look on his face.

Dagstuhl isn’t all heavy thinking and hard work, though. I had a beer with dinner (pictured below); unfortunately it made me very sleepy for a while. I’m such a lightweight…

We all arrive at Dagstuhl on Sunday night for the week’s workshop. After a long day of traveling we enjoy some of Dagstuhl’s exceptional food, meet with folks, and catch up.

Above is the conversation during dinner tonight. It’s not the sharpest photo I’ve ever taken, but I think it captures the spirit of the room quite nicely.

And then we (at least quite a few of us) get to work. The photo below was taken at 9:30pm (probably 2.5 hours after the previous one), and there were quite a few people in this lab at the time. And there’s another lab elsewhere in the facility, and the library, and people’s rooms. I’m sure that lots of folks were also hanging out in the coffee room or playing pool, but there were a lot of people working on a Sunday night as well.

It helps if you enjoy your work, and most of these folks are extraordinarily interested in what they’re studying.

While I’m at Dagstuhl this year I’m going to try (amidst all the “real” work) to capture something of what the workshop is like and, more generally, what it is to do (computer) science. This is hard because it’s not flashy high-action bull-riding kind of work, but it’s important, significant work and deserves to be documented. I’m just going to have work harder at it.

I’m also probably going to take more people pictures than I would be naturally inclined to. If anyone finds them self in a photo here and objects, let me know and I’d be happy to remove it.

It’s late and I’m way tired, but I think the packing is under control (more or less) (sorta kinda) (as well as can be expected) for tomorrow’s grand journey to Dagstuhl. Not sure how much posting will happen from there; depends a lot on how good the wireless is in my room this year.

In the meantime I leave you with this travel (and research) related shot from my excellent visit to Dublin last month: a group of vehicles driving onto O’Connell Bridge. It looks like it’s the middle of the night, but it was actually only a bit after 6pm as everyone was hurrying home from work. Short days in Dublin in December.

Ciao!

JOCP! It’s been almost five days since I’ve posted anything here, and I have so much backed up in the queue…

There’s a major conference deadline (GECCO 2008) in a few days, and I’m struggling to finish up four different (and only loosely related) papers for submission. On top of that I leave Sunday morning for an excellent week in Germany at a research seminar at the wondrous Schloss Dagstuhl. (Feel free to visit some of my photos from my last visit to Dagstuhl.)

So sleep is short and fun on the blog is shorter still. In two weeks, though, I should be able to get back in the game a bit.

The top photo is of a student (Tyler – now graduated) during a talk he was giving with another student (Andy) at a regional computer science conference (MICS) last April. On the next day the two of them received the best student paper award for this work :-).

I’ve spent numerous hours this week drawing and redrawing graphs and tables, so this is all too reminiscent of my life at the moment.

The photo below is from a beautiful snowfall we had during the Dagstuhl workshop two years ago.