Dirt Pile Perspectives

There are two undeveloped lots across the street from our house. The builders have promised to build houses on the lots next summer or winter for the last 4 years. The neighborhood kids use the space as a playground. Rocks and dirt have a way of attracting kids.

From time to time the builder sends trucks to take dirt from the lot presumably to build a giant mud pile somewhere else in the world. Each time they show up Christine and I get a little tinge of excitement at the thought that they might actually start construction on the houses they have been promising to build “Next summer” or “Next Winter” for the last 4 years.

The other day a yellow front loader and a dump truck showed up and they started removing some dirt.

Alex saw this and marched across the street and right up to the driver of the front loader.

“Excuse me," he said. "Why are you taking my dirt!?” he asked in a stern accusatory tone.

The 20 year old driver was at a loss for words. He mumbled something about his boss telling him he had to. Unimpressed, Alex marched over to the dump truck driver and remaining polite but indignant.

So that dirt pile I think of as an eye sore that hurts my property value is instead, a big pile of fun to Alex. The kid's got a great perspective.

A Smile

Allison is all smiles... In this picture anyway. You should have seen all the faces and noises I had to make to get this picture. Totally worth it though!

Happy Birthday Julian!

Julian was born July 30th at 11:33 pm to eccentric yet lovable parents Reza and Teresa. Actually Teresa's totally normal, it's Reza who's a bit out there...

Like his Russell Brothers and Sisters, Julian's birth was marked by unicorn sightings, the appearance of a single white dove and a moment of revered silence immediately followed the sound of digital cameras trying desperately to keep up with the tidal wave of cuteness that he brings wherever he goes.

My Water Bottle

This is one post in a series, describing what I've learned while attempting to understand my ecological footprint.

Bottled water consumption in the United states is a unnecessary scourge on our environment. I usually try to stay away from inflammatory statements, but a little reading and its hard to come to any other conclusion.

Articles like Charles Fishman's Message in a bottle (also see NPR's audio coverage [6 minutes], The Pacific Institutes Bottled Water and Energy A Fact Sheet, The Earth Policy Institute's BOTTLED WATER: Pouring Resources Down the Drain and several others explain the life-cycle and costs of bottled water as well as the psychology behind why we spend $15 billion a year and pollute our environment for something we can already get out of the tap with no added pollution or cost.

Those articles do a great job of ruling out taste, convenience, safety, and quality. So why do we drink bottled water? Great Marketing... Fortunately public opinion is beginning to shift.

How I Got Past Bottled Water Marketing

From 1999 to December 2006 I consumed an average of four 16.9 oz. bottles of water a day. Most of the time I walked by a kitchen sink to pull it out of the fridge, and many of those ended up half finished on the floor of my car.

In December I got to thinking that there was probably a substantial ecological impact as a result of getting that water, making the bottles, shipping them to me and then keeping them in giant refrigerators until I drank them. I didn't bother with the math. Water is heavy and Giant smelly diesel trucks driving it around were enough to get me thinking about the impact.

When I asked myself why I was drinking Bottled instead of Tap I experienced a strong emotional answer from my gut "Because tap water is bad and bottled is clean." To which my brain responded "That's silly! 'tap water is bad?', what does bad mean?" So I thought about it. I grew up on tap water and never got sick from it. It's common knowledge that the US has the best tap water in the world. And pretty soon I heard a voice in my head say "I do believe I've been brainwashed. All that bottled water is pure excess."

So I decided to switch to tap water. I'd keep the last water bottle I had, put my name on it and reuse it. I saved more than one just in case I lost a few.

The first time I refilled the bottle I found myself feeling concerned. Somewhere in the past 10 years I'd gone from drinking only tap water to having internalized the idea that tap water was going to hurt me. But I stuck with the tap and each time I refilled my bottle it got easier.

I've been reusing two bottles since late December 2006. I rinse a bottle out once a day and fill it up when I'm thirsty. The bottles look the same as the day I started. I haven't gotten sick, the water tastes fine, and by my estimate I've avoided consuming 888 since starting. By the end of the year I'll be at 1460 bottles!

Going by the Pacific Institutes estimate that "the total amount of energy required for every bottle is equivalent, on average, to filling a plastic bottle one quarter full with oil" in one year I figure I've avoided using 48 gallons of oil (~1.15 barrels of oil).

Oil and Water Math

((16.9 oz per bottle*(1/4 pacific institute estimate of oil per bottle)*4 bottles a day * 365 days a year )/128 oz per gallon = 48.19 gallons = 1.15 barrels of oil

What gets me about this bottled water thing is how impressive the marketing has been. Even after I thought through it and convinced myself of the silliness of drinking bottled water over tap I still felt weird about drinking tap water. It makes me wonder if bottled water was the first cult I was unknowingly part of.

There's a tun of bottled water literature out there. But if all that reading is just too much, Penn and Teller do a pretty good job describing the psychology of bottled water in this video...

How I Think I Caused Yahoo to block Google’s IP addresses: A Cautionary Tale

Warning: This post contains extremely geeky content.

This is a tale of how one small-time , hobby-level, mash-up widget developer (that would be me) caused Google to appear to DOS attack Yahoo, and how Yahoo protected itself by blocking Google from accessing its content. This is a tale that almost has the pieces you'd want in a good story... It's got a hint of mystery... It almost has suspense... It's chock full of buzz words... And it even has two valuable lesson for developers of mash-up frameworks and Web Service APIs.

It all happened, unintentionally, through the seemingly innocent combination of yahoo’s del.icio.us social bookmarking API, the iGoogle mash-up homepage, Google’s proxy server and a widget I created to view my bookmarks.

Extremely High Level Summary
Uber Geeks Only

For a slightly less geeky explanation, some background info, and pictures skip to "The Full Details" section below.

I created a module for iGoogle (module home page) that displays del.icio.us bookmarks. Every time a user that had installed my del.icio.us widget reloaded their iGoogle page the widget made a lot of requests to the del.icio.us web API (owned by Yahoo). Apparently the volume of requests from each page reload combined with a number of my friends using the widget was enough to be considered on par with a small un-sustained DOS attack.

Because all requests from an iGoogle widget need to go through the Google proxy servers to avoid cross domain scripting security in the web browser, Yahoo saw the requests from my widget as coming from Google. Yahoo determined that Google was launching a DOS attack (they called it a swarm) and blocked some of their IP addresses from being able to access the del.icio.us web API.

The effect seems to be localized to IP addresses from a Google Data center near me because if I proxy my requests to Google through another part of the world the problem goes away. I emailed Yahoo (thread here) and they basically said they would keep the IP addresses blocked unless someone from Google contacted them and promised to throttle all future traffic.

Keep reading to find out how I got the widget working while Yahoo and Google remain in a standoff.

The Full Details (With Pictures!)

A little History

Del.icio.us is a social bookmarking site that I use to keep track of bookmarks without having to lug a computer around with me everywhere. (Here's a 3 minute video description of social bookmarking if you're not familiar with the concept.) iGoogle is a mash-up that allows you to create a personalized home page made up of publicly available modules.

There are a lot of modules out there for putting del.icio.us bookmarks on your iGoogle page, but none of them did exactly what I wanted.

Basically I wanted to see a list of my del.icio.us tags, and then when I clicked on a tag, the bookmarks with that tag should be immediately listed. All the modules I found brought me to the del.icio.us page for that tag, but that was way too much page reloading for me. So I created a module that has these features:

  • Displays your del.icio.us tags in a tag cloud
  • When you click on a tag, the bookmarks with that tag appear below the tag cloud. Click on that tag again to clear the list of bookmarks from the screen
  • Clicking on the title brings you to your del.icio.us home page
  • Supports up to 100 tags with 100 bookmarks per tag

How it works: Mash-ups & this Situation

iGoogle (a mash-up framework) loads the widget I wrote which in turn gets my list of del.icio.us tags from del.icio.us using a JavaScript >script< tag. Thus when the iGoogle page loads the tags are there. Unfortunately due to a limitation in the del.icio.us API you can’t load all the bookmarks at once (del.icio.us limits you too 100). So I resorted to dynamically fetching them based on their tags. First get the list of tags, and then get the bookmarks for each tag...

Cross Site Scripting and mash-up Proxies

Now there is a catch with dynamically fetching content in a web browser called the Cross Site Scripting Security. Basically this means that if you get a page from google.com, the only place you’re allowed to make an AJAX request to, while on that page, is google.com. This is for your protection and is a good thing because it eliminates one more way people can steal your bank account PIN. Like most things for your protection the world has found a way around it and we are better off because the workaround has enabled mash-ups like iGoogle.

The solution (shown in the picture above) for getting around cross domain scripting security in a mash-up is for the owner of the original page (in this case google.com) to host a proxy server. All AJAX requests destined for other domains (like del.icio.us.com) are sent to that proxy at google.com and the proxy marshals the traffic between external sites and the web browser. The browser still protects against cross site scripting, but since all the traffic is going through the google.com proxy, the browser is none the wiser to the fact that data is coming from del.icio.us.com.

What Went Wrong

I made a bad assumption. I assumed that del.icio.us had a big data center and couldn't possibly care if I made 50 rapid requests of their web API. I actually remember asking myself the question "Should I throttle these requests?" It was a trade off between rapid UI response time for the widget users and load on the del.icio.us servers. I decided that since del.icio.us was owned by Yahoo and Yahoo serves up billions of pages a day, that they couldn't possibly care about a flutter of requests from my widget. I was wrong.

After almost 9 months of the widget growing in popularity Yahoo noticed and started returning the following page when the widget fetched bookmarks through the google proxy:

I originally figured this was just a del.icio.us API glitch and would go away. A week later through my emails with the del.icio.us support team I learned otherwise. The short version of the email thread is that Yahoo was actively blocking Google's IP addresses as a result of the traffic from my widget.

In an act of belated responsible development, I changed my module to only request the tags at start-up and to make an individual bookmark request only when the user clicks on a specific tag. Since the widget is dynamically served each time someone logs into iGoogle it was easy to replace the version everyone was using.

Yahoo continued the email discussion off the forum because as they stated "it's probably not as interesting to everyone at this point". I'd argue that it was never interesting to everyone and there are only a handful of people in the world that would care. But to me, the value of a web forum is to be able to find those obscure bits of information about a specific topic. So why take things off a forum? But I digress...

Through those private email I learned that the "swarms" subsided after my widget change but then picked back up a little. They were a bit vague on the numbers. Because del.icio.us doesn't have a way to track the application making the requests at a more granular level than IP address, they are basically blind as to the real cause of swarms.

I did some testing and discovered that the issue appears to be localized to my geographic region. (North East US) This makes sense since I probably hit the same google data center every time I use iGoogle and many of my friends using the widget are in this area. So the problem is limited to that small community of iGoogle / del.icio.us users, living near me, that want to access their bookmarks using their iGoogle home page through the delicious JSON API.

I have no definitive proof that it was only my module causing the problem, but circumstantial evidence suggests that the module was at least part of the problem.

In the end Yahoo refused to unblock the Google IP addresses and my posts to the iGoogle Google Group asking them to contact Yahoo went unanswered. I can't say I'm surprised, this is hardly a major issue in either of their world views. And rightly so...

Lessons Learned

This experience has taught me two things.
  1. If you are a mash-up framework providing an AJAX proxy, you should monitor the HTTP response codes coming back to assure that a widget developer has not damaged your relationship with an external service API.
  2. Whenever you provide developers an API for accessing your web content, be sure to provide a developer API Key and require its use with every request. This allows you to filter offending behavior by the developer or application instead of IP address. If del.icio.us had done this they could have simply blocked my widget and left their relationship with Google intact.

Epilogue - The End Run...

One night it occurred to me that I could circumvent the Yahoo IP block by putting a second proxy server between Google and Yahoo. Five lines of CGI and 10 minutes later and my second proxy was working swimmingly. Then I changed my module to tell Google to make a request to my proxy server which in turn forwards the request to del.icio.us. and once again del.icio.us is none the wiser as to the source of the traffic.

I decided not to publish the updated version using my proxy server because I didn't implement either of the lessons I learned when I wrote it.

Harry Potter and the Deathly Hallows

The last book in the Harry Potter Series arrived by Amazon Owl Post the day after it was released. I went to great lengths to avoid finding out how it ended until I was able to read it for myself. Despite efforts of people scanning the book and putting it on the internet, and Newspapers reviewing the book two days before it was released I'm glad to say I succeeded in learning Harry's fate from the book.


The text below this paragraph may appear as a spoiler to anyone that knows my taste in stories. To put this in perspective, I think Dodgball: A True Underdog story may be the funniest movie ever made.

Overall, I was thrilled with the way the series wrapped up. My frustrations with the morbid ending to the 6th book was relieved by a brilliant storyline in 7th. Beginning to end the Harry Potter series is the most entertaining and engaging fiction I've read.

Even so, its amazing to see how the story crosses geographic, gender, and age differences turning masses of people into little kids that can't wait to find out what happens to Harry and his friends. It was fun to be one of those people kids again.

For a real spoiler and a good laugh check out the Harry Potter Public Enlightenment Project. They are a "...privately funded foundation, helping alleviate anxiety by presenting useful plot information in a readily accessible format."