Yahoo! Developer Network Blog

YQL Open Table for Google Buzz now live

Tue, 09 Feb 2010 15:19:26 -0800

We were seeing a lot of hype around Google Buzz today and we thought YQL would love to chew on the data. Fifteen or so minutes later we had an Open Table (the blog post took a bit longer :P ) wrapping the Google Buzz API. It's now live on GitHub: github.com/yql/google/google.buzz.updates.xml.

You can use it in your queries like this: USE "http://github.com/yql/yql-tables/raw/master/google/google.buzz.updates.xml" AS google.buzz.updates; SELECT * FROM google.buzz.updates WHERE user="sydlawrence".

---
Erik Eldridge, Christian Heilmann, Jon LeBlanc
YDN Team

Follow @YDN on Twitter for more from the Yahoo Developer Network.

INSERT INTO twitter.status ...

Mon, 08 Feb 2010 07:21:28 -0800

Want to start building a Twitter application using YQL? We've recently added a number of new tables to the community table list that provide full read/write access to Twitter's APIs using OAuth. These tables allow you as a developer to interact with Twitter as if it were any other YQL table: Not only can you read from and write to Twitter, but you can easily merge data with other Yahoo! and 3rd-party APIs and web services.

For API endpoints requiring authorization, OAuth signing is done automatically via YQL Execute and HTTPS. All you need to pass through is your consumer key / secret and access token key / secret as keys in your YQL query. That means you no longer need to ask your users for their passwords and there's no reason not to start using OAuth in your Twitter applications. Why should I use OAuth for my applications?

To get started with these new tables, we've got some simple examples:

1. First, set up a new Twitter application. (You'll need your own Consumer Key and Secret for some read/write actions from Twitter)

2. Load the YQL Console with the new tables:
https://developer.yahoo.com/yql/console/?env=store://datatables.org/alltableswithkeys

3. Experiment with the following YQL queries.

a. Get single status:
SELECT * FROM twitter.status WHERE id='8036408424';

b. Get a user:
SELECT * FROM twitter.users WHERE id='mlaaker';

c. Tweet something:
INSERT INTO twitter.status (status, oauth_consumer_key, oauth_consumer_secret, oauth_token, oauth_token_secret)
VALUES ('tweeting from yql!', '@your_consumer_key', '@your_consumer_secret', '@your_access_token', '@your_access_secret');

d. Current weather at a tweet's geo-location:
SELECT * from weather.woeid WHERE w in (select woeid from geo.places WHERE text in (select Placemark.address from google.geocoding WHERE q in (SELECT geo.point FROM twitter.status WHERE id="8098940289") LIMIT 1));

With these new tables, we hope to see more interesting Twitter applications built on top of YQL... especially those that integrate other best-of-breed web services. Don't see an API or web service you want? Check out the existing open tables, build your own, and contribute for others to use in their projects.

Zach Graves
Product Design Developer

Announcing the Yahoo! Brasil Open Hack Day 2010, 20-21 March

Mon, 08 Feb 2010 02:45:54 -0800

Brasilian developers, start your engines! It’s time again to show off your developer skills at our second Yahoo! Open Hack Day held in São Paulo, Brasil on 20 and 21 March 2010.

This event is the Brasilian version of the already traditional Yahoo! Open Hack Day held in Sunnyvale Campus and other editions in London, Bangalore, New York, and Djakarta. The last Open Hack Brasil in November 2008 was a huge success, with approximately 200 attendees and lots of very cool hacks!

It will be 24 hours non-stop of hacking, learning, fun and the chance to meet fellow hackers from around the region. Yes, 24 hours. But don't worry, we will have food, beverages, and everything you will need to be comfortable, happy and unleash your creativity! There will also be sleeping areas in case you want to take a nap.

In our last Hack Day in Sao Paolo, we had amazing hacks, from mobile applications to hardware hacks. For instance, the winners of the "What the Hack" category used Python, Flickr, Twitter, and open source hardware to build a circuit board with LEDs that flickered more or less intensely depending on how many people tagged photos on Flickr with the official hack day tag.

We also had a very creative non-code hack that made us create a new category - the "Using the environment" hack:

Puff Hacking from fczuardi on Vimeo.

This is the Hack Day spirit!

Join us in creating new applications on top of Yahoo!'s open platforms like YQL, YAP, Meme, YUI3, Pipes, and Flickr. Of course, feel free to use any other open developer tools you like. The best hacks will be recognized with awards, and hackers will win bragging rights until the end of eternity or the next Hack Day, whichever comes first.

While we’d love to have everyone there for the festivities, we do have space limitations at the venue, so book your place now at http://hackday.com.br (in Portuguese)! If you have any questions, please contact us at openhackbrazil@yahoo-inc.com.

The hacking season is now open in Brasil!

Guilherme Chapiewski & Anil Patel
Yahoo! Brasil Open Hack Day Team

Marketing hacks, linchpins, and tech women of valor

Sun, 07 Feb 2010 15:38:31 -0800

I've been unusually busy the last few weeks, so I haven't had much of a chance to sit quietly, read deeply, or sift singlemindedly through the online conversations I care about. Here's a breathless sort of wrap-up.

Jailbreaking the Status Quo

While I've been chipping away at a backlog of work, people I attend to have been reacting to disaster in Haiti; the mystery and mystique of the imminent iPad; the form and future of reading; closed devices and the threat to tinkerers; the form and future of work; and issues of gender, self-assertion, and corporate advancement.

Back on January 15, Internet pundit and author Clay Shirky posted A Rant About Women, which encouraged women to (depending on your point of view and interpretation): adopt the most callow attributes of male braggadocio, promote themselves more boldly in the workplace, use "male" tactics of hyper-confidence and self-inflation to open doors. Not surprisingly, the post unleashed a firestorm of response, ranging from sweaty high-fives to thoughtful rejections.

This topic and recurring experiences in my own professional life always remind me of the Jodie Foster movie Contact, which does such a wonderful job of showing how even the most tough-minded and successful women can be undermined by lying and unscrupulous corporate climbers. It also reminds me to mention one antidote: the Grace Hopper Celebration, a conference for women in computing, where the call for participation remains open till March 16.

Linchpins

January 26 marked the publication of "Linchpin: Are You Indispensable?," a new title from unstoppable marketeer and prolific author Seth Godin. In a genre pioneered by Daniel Pink and Timothy Ferris, Godin dispenses advise in an appealing new-think business book about releasing your inner artist and letting your creativity ship. Godin describes a familiar landscape, where being a well-greased cog in the machine and following the rules no longer buys you a job for life. It's a world where creativity and daring and getting your lizard brain out of the way trump corporate entropy and make you indispensable as an artist and employee. To me, one of the most interesting things about Linchpin was the highly original marketing strategy Godin is using to pre-launch the book and extend its buzz. His tactics form a brilliant live demo for some of his key principles.

Godin shipped early review copies to anyone who sent a minimal donation to the Acumen Fund, "a non-profit global venture fund that uses entrepreneurial approaches to solve the problems of global poverty." The methods and mantras of Acumen are perfectly congruent with the messages of "Linchpin." By aligning a new, entrepreneurial model of philanthropy with a new model of self-marketing, Acumen gained the attention of a potential community of engaged advisors and contributors. Through a short series of incredibly well-crafted "direct marketing" emails about Linchpin and Acumen, Godin, was able to assemble a niche of people who probably self-identify as linchpins, influencers, and communicators, and would jump at the chance to participate and contribute a positive review. Twelve years after selling his pioneering Internet marketing firm Yoyodyne to Yahoo! for millions, Seth Godin remains the kingpin of high-order and successful marketing hacks.

For Godin, it's not about the size of your swagger, it's about finding artful workarounds that cut through the clutter. With Linchpin, Godin cuts a clear trail straight to an ideal audience of social media consultants, corporate innovators, and an assortment of dreamers who have everything to gain by aligning themselves with his message and his brand.

Uppity Like Me

Which brings me, in a roundabout way, to She's Geeky, an unconference for "connecting women in tech," which took place last weekend at the Computer History Museum in Moutain View. The event was organized by Kaliya Hamlin, a larger-than-life woman and an artist of the sort that Godin describes -- entrepreneurial, altruistic, courageous, and committed. Kaliya is also an interesting proof of concept and/or counterpoint for some of Clay Shirky's allegations.

As a hired facilitator and volunteer Open Space practitioner, Kaliya has been reinventing tech conferences, meetups, and camps for as long as I've known her. As a speaker and pioneering advocate for OpenID (way ahead of many of the guys!), the Open Web, and gender balance at tech conferences, she's been banging on "men-only" doors, sparking conversation, and igniting opportunity for the rest of us too.

Havi Hoffman
YDN Blog Editor

Follow @ydn on Twitter.
Become a Yahoo Developer Network fan on Facebook.

Yahoo! India invites you to join the first India Hadoop Summit

Thu, 04 Feb 2010 19:09:06 -0800

On Feb 28, 2010, Yahoo! India will taking part in the country's first Hadoop Summit, in Bangalore.

This day-long event will be co-hosted with CloudCamp Bangalore 2010. Hadoop will be a dedicated track in this session.

At this event, you'll hear participants from the Yahoo! India Hadoop team, industry experts, and major universities. The event brings together leaders from the Hadoop developer and user communities.

Speakers will cover a rich variety of topics including the current state of Hadoop development and deployment, Pig, performance optimization of Hadoop Cluster, testing in Hadoop, real-world case studies and Hadoop in academic research.

We'd love it if you could join us. You can find more information, see the schedule, and register at: cloudcamp.org/Bangalore.

Preeti Priyadarshini
Hadoop Team, Yahoo! India

Tech Thursday - Python evolution, HTML5 image editor and the iPhone as a disk drive

Thu, 04 Feb 2010 12:04:14 -0800

Every Thursday is Tech Thursday where we share a random assortment of technical links we found and liked.

Eugene Tsu shows the world how to write a CV
Shutup.css is a custom CSS to hide comments from famous blogs - in case you like to concentrate on the content.
A nifty trick using CSS3 allows you to check links on the page for missing hrefs.
Simon Whatley has a list of free user experience books to read online.
Want to use your iPhone as a disk drive? iPhone Explorer lets you do that.
TTF to EOT is a small script to convert font files. Good for @font-face.
Mozilla has an HTML 5 offline image uploader and editor application hack.
WideImage is a simple PHP image library.
Nine things developers want more than money is something to show to your bosses.
Evolution of a Python programmer - as a python source code.

You can propose links to us on Twitter (@YDN) or try bookmarking them on delicious with the tag "forydntt".

2010 starts off with University Hack Day in Hyderabad

Thu, 04 Feb 2010 11:45:22 -0800

Happy new year to all Hackers! Yahoo!'s University Hack Day program kickstarted the new year in India, with a grand event at the International Institute of Information Technology, Hyderabad (IIIT H). This institute is well known for its strong industry focus. Yahoo! Bangalore has been in touch with the professors and students from IIIT H for various research projects. Rajeev Rastogi and Prabhakar Raghavan had visited the campus and were impressed with the faculty and students alike. Muthusamy Chelliah, head of Academic relations at Yahoo! Bangalore, emphasized the interest levels in the college and how eager the students were to learn more about hacking. The nine-member Yahoo! hack day crew landed up in Hyderabad on 22nd January morning. The Yahoo! team was greeted with a huge cheer--we were overwhelmed by the student turnout.

It was an honor to have Maj.Gen.Dr.R.K.Bagga for the event kick off. He spoke about how today's society, leadership and the media are becoming more internet savvy, fueling an insatiable need for innovation and creativity. He challenged his students to participate and make their Hack U the most successful Yahoo! Hack U ever. The students responded with roaring applause and cheers. He wished success to all of us.

Yahoo!'s Jamie Lockwood opened with introductions and showed the students video news coverage from the 2008 Open Hack event in Yahoo! Sunnyvale and more videos about Open Hack. During these talks, students kept pouring in, and the environment became more and more electric. Jeremy Hubert, prototyper extraordinaire from Yahoo! Search, spoke about his hack experiences and introduced students to some cool hacks from earlier hack events. To pep up the students even more, question and answer sessions were held and t-shirts were tossed out into the crowd. Rohan Monga, an IIIT H alum, demoed a couple of his winning hacks to the students. He had a Flickr game hack that showed how something fun can be made useful. Me and Jeremy later shared the stage to encourage students to focus their hack ideas on problems they faced when using technology and internet, things they wanted to see solved, or features they would love to have in products they use.

Of the four hack days we'd conducted before IIIT H, this event was the hands-down winner as far as crowd response and student turnout. The enthusiasm made us certain that the hack day was going to be a success. After the Q & A session and a short break, I got back on stage for my talk "Get me my data". This is the 3rd time I've done this talk: it's part of the introduction focused on how to use the internet as a data exchange medium and how every resource is a data source. This talk primarily focused on two of my favorite services in the Yahoo! stack, YQL and Pipes. I showed examples of how to fetch data from various sources like XML, JSON, and even HTML. I demoed a couple of hacks and showed the audience how easy and fast it is to put something together. Some examples showed how Yahoo! Pipes can replace code for processing data and how YQL can be used to normalize the way in which we seek data on the internet. The next day morning (23 Jan), we had couple of deep dive talks before the actual hack event kicked off. Saurabh Sahni gave his BOSS talk, where he demoed how easy it is to create a search experience and how the BOSS API can be used for hacks. Rajagoal, from the Yahoo! India Maps team gave a talk on Yahoo! Geo Technologies. He focused on the Geoplanet APIs, AJAX Maps APIs, YQL interface for GeoPlanet and Fire Eagle service. Geo APIs are usually very popular among hackers. It was great to see that it was almost the same number of students turn up in the morning for the deep dive talks.

Officially, the hack event started at 11:30 am on 23rd Jan, though students were thinking about their ideas from previous night itself. Students were encouraged to form teams of 3 or 4. As the day progressed, there were as many as 65 hack ideas registered, highest among Hack Us in India. Many in the hack technical crew and agreed on doing various BOF (Birds of a Feather) sessions to help students with Yahoo! services. I did a couple of YUI talks. There was a real buzz around college with almost all hackers sitting in the allotted hack rooms in college. This made it easy to have various discussions and solve any road blocks students hit during their hack implementations. I must compliment the enthusiasm and commitment of the students who were awake right through the night focussed on their ideas. I spoke with the Hack Tech crew, and all of them loved the interactions with the students and were impressed with the questions and thinking. Geo APIs, YQL, Pipes, YUI, Answers, Local, Mobile, Speech, text processing, GreaseMonkey and Indian languages were definitely the hot threading topics. There were even couple of student groups who did electronic device-based hacks.

The next day, the Demos were suppose to start by 11:30 am, exactly 24 hours after the kick off. There were 46 hacks ready to be demoed!. This again is the highest we have had in all India Hack U events. We had 4 judges: Jeremy from Yahoo! in Sunnyvale, Professor Vasudeva Varma, Professor Vikram Pudi from IIIT H faculty, and me. Professor Vikram Pudi was actually an ex-Yahoo.

All of us were blown away by the ideas and energy of the students demoing their hacks. The IIIT H students were a smart, creative and committed bunch. Many students used mobile as a platform for innovation. One group aimed to provide best deal business quotes and summaries on the mobile. Another group was working on voice search and SMS-based search. A couple of groups used the Google Android platform to implement their take on front-ends to services like Twitter and Yahoo! finance. Many groups used grease monkey to improve the relevance and usefulness of sites by adding on features on popular pages. One group focussed on collecting and displaying collaborative summaries of web pages to a social network. Another hack tried to explore the idea of converting speech to text and again back to speech after translation so that 2 people speaking different language can communicate. It was really hard to judge the best hacks in the demos. Every idea was unique in its own way, and to accomplish what each team had done in 24 hours is truly amazing. I think the college must feel really proud of its students.

Among the 45 odd hacks that were demoed, we shortlisted 12 hacks as the finalists. Out of the 12, we choose 4 winnings hack teams and 2 Yahoo! tech crew favorites. The 12 hacks that were finalized are. The top 12 hacks and winners are as follows

RAT - Rural Area Twitter (Winner) A wireless hardware device aimed to be fit on every cycle in a rural area to facilitate viral message broadcast and distribution
Fun Search Hacks (Winner) A collection of 3 fun hacks, roughly based on Search. The best one was filtering of news results based on Good and Bad news. We all thought the demo was really amazing
Clickless (Winner) A grease monkey script to help specially abled users to convert clicks to mouse over actions so that user could just point to a link and perform action
Real Time Form Collaboration (Winner) A grease monkey script to create real time form editing by multiple people at the same time across the internet. Social networks + Content collaboration
Twitter Remind Me (Honorable mention) Using Twitter as a reminder system were one could set reminders to their friends
Party Rocker (Honorable mention) Based on listening interests, playlists and song choices of members in last.fm, a network of friends can create a customized intelligent song playlist that would have mixes that everyone liked.
BoratTool to enable writing code in any language and compile in any language.
Smart Wiki Grease monkey script to make the Wiki experience little more rich by adding image search, weather and other modules live on any Wiki page
Chez allows users to add places on a common map utility and search for places around a radius of 15km
NetPool A smart really useful script to pool in multiple internet connects to boost up downloading speed. 1 Mbps Wifi connection + 1 Mbps Data card connection = 2 Mbps line for computer
Twtr Automatic smart shortening of any message to fit the Twitter 140 character limit. 'Talk to you later' becomes 'TTL' automatically
Cloudy Transfer Using SMS to set up a send mail option along with options to attach a file from a remote machine
Fast News
Multi Language Search

Standing up on that stage and looking at huge crowd all fired up and enthusiastic was such a rush. I hope other colleges we go to in the future have similar response, boosting our energy to make it a fun and useful event. I must also mention that every one in the Yahoo! team loved the healthy food on campus, especially our visitors from Sunnyvale.

We had really smart engineers part of the Technical Support Crew, Rohan Monga, N Rajagopal, Rajesha and Saurabh on campus. All of these guys are exceptional technical brains and true geeks who love hacking. We had even more support on IRC, a special call out to Arnab Nandi who is the ever enthusiastic go to guy for ideas and solutions. A big shout out to Jeremy who managed to stay up all night and help the students despite a long flight and horrible jetlag.

A huge shout out to Chelliah, Teenu, and Jamie Lockwood who were the main people responsible for the event and all its success. These guys have amazing energy levels and ensured everything went off smoothly and as planned.

IIIT H has set a new benchmark for Hack U in 2010. Looking forward to the events to come. Maybe it's time to think of the ultimate Hack U challenge where winners from different colleges come together for one hack day.

Subramanyan Murali
YDN Evangelist, Hacker, Web Developer

Yahoo! Bangalore

YDN at She's Geeky

Wed, 03 Feb 2010 19:07:03 -0800

Last week, Havi and I attended She's Geeky along with Erin Malone, and my daughters, Cady and Grace Tippins. She's Geeky is styled along the lines of a typical unconference, with participants proposing a schedule of sessions at the start of the day.

At our session, we showed participants how to play the Social Mania Game, a social patterns card game that allows you to build products, gain points for great features, and eventually 'pitch' your completed product to the player representing a VC. The game was fun, of course, but the feedback we got was the great benefit. Erin and Christian Crumlish created Social Mani to teach people about social patterns and as a complement to their book, Designing Social Interfaces. Since the game is still in beta, some of the suggestions made at She's Geeky may make it into future revisions.

Lynn Langit, Cady Tippins, Grace Tippins
Photo credit: Duzins

My favorite session, "How to teach kids to program," was taught by Lynn Langit, Developer Evangelist at Microsoft Developer Network. After that session, Cady went home and spent all weekend, head-down, running through recipes in Small Basic.

Since it was at the Computer History Museum in Mountain View, several attendees snuck downstairs to partake of the other offerings as well. There were more than a few who joined us to watch the Babbage Difference Engine demo at 2pm, and me and my children spent half an hour looking at the examples of computers and game consoles of old (i.e., pre-1995).

She's Geeky was a large success, and I give props to the ladies that put the whole thing together. The five of us were only there on Friday, but watching the tweets from Saturday and Sunday told me that the event continued to generate attention and discussion on those days as well.

Robyn Tippins
Community Manager, YDN

More About Cloud Computing Research on the Open Cirrus Testbed: The Videos

Wed, 03 Feb 2010 12:30:22 -0800

Last week, we hosted cloud computing technologists and researchers from around the world at Yahoo! for the second Open Cirrus Summit. Here's a collection of short clips from attendees who are using the Open Cirrus Testbed for a unique approach to cloud research. We chatted with researchers from Carnegie Mellon University, Cornell, and UC Berkeley to learn about some of the scientific research being conducted.

Here's Randy Bryant, University Professor and Dean, School of Computer Science, Carnegie Mellon University, speaking about CMU's involvement:

Additional links

View the full set of videos and photos from the Open Cirrus Summit on Flickr.

Find slideware and other details about the Open Cirrus Summit.

To learn more about Yahoo!’s research programs with the academic community, here's a blog post from Yodel Anecdotal.

If you are interested in Hadoop and other Cloud Computing technologies, check out our Hadoop blog and we’ll keep you posted about the third annual Hadoop Summit, happening later this year.

Thomas Kwan
Director of Yahoo! Labs and Research Operations, and organizer of the Open Cirrus Summit

Follow @ydn on Twitter for the latest updates from the Yahoo! Developer Network and other Yahoo! technology initiatives.

Sneak Peek at the Create Consumer Key API

Tue, 02 Feb 2010 07:17:54 -0800

In about a month we'll be launching a new feature on the Yahoo! Developer Network (YDN) called the Create Consumer Key API (CCK API). This blog post is a sneak peek-- describing what it is, who should use it, and how interested 3rd parties can start integrating with it early.

What is it?

Many developers who use Yahoo! Web services and APIs distribute applications that enable their users to tap into Yahoo! user data (with user permission). In these cases, both the developer and each of their end-users signs up for a distinct OAuth Consumer Key. The Consumer Key ensures that the right data, for the right user, is going to the right developer.

YDN aims to make it easier to broker out or sub-syndicate access to Yahoo! data, so we have been working on a programmatic way to trigger the creation of Consumer Keys.

The CCK API offers the following benefits to developers:

Streamlines the process of using your application: Your users do not separately request an OAuth Consumer Key from YDN.
One call: One form provides all the necessary information for a Consumer Key, such as access scopes and redirect URLs.
One-click authorization: Your users do not need to know about access scopes, and you have the option to automatically send the Consumer Key and Secret back to your application with no copy-and-paste necessary.

Who should use it?

Developers building applications they plan to distribute (and which require access to private Yahoo! user data) should consider integrating with the CCK API. Examples of distributed applications might include a JavaScript sharing widget that interacts with Yahoo! Contacts or a Wordpress plugin that blasts new blog posts into Yahoo! Updates, etc.

Where to find more details

We'll be posting additional details on the YDN blog when we are closer to launch, including screenshots of the user flow, links to the API, and documentation for how to integrate with it.

In the meantime, if this sounds like something that interests you and you want to get started early, please send an email to cck-api@yahoo-inc.com with your name/role, a description of your product, and a URL we can check out. We'll be happy to share additional details so that you can start exploring and/or planning your integration early.

The Yahoo! Developer Network offers services that make it easy to build applications that make the web a more useful and interesting place for everyone. This API should made it even easier.

Daniel Raffel
Senior Product Manager

Yahoo! Open Strategy

Yahoo!'s Accessibility Evangelist shares his expertise at a joint WIPO-ITU workshop in Geneva, Switzerland

Mon, 01 Feb 2010 13:00:00 -0800

London-based accessibility evangelist Artur Ortega will speak at the Joint WIPO-ITU accessibility Workshop in Geneva on Tuesday, 2nd February to promote awareness of disability and accessibility. He will be available throughout the workshop to share his expertise with web developers within the United Nations system and other international organizations in Geneva to implement accessibility in their daily work.

Innovation By Accessibility V1, Wipo 2010

View more presentations from Yahoo Developer Network.

Artur Ortega was invited by the World Intellectual Property Organization (WIPO) workshop organizers. The workshop is a joint effort with the International Telecommunications Union (ITU) at the WIPO Headquarters for staff of the United Nations system and other International Organizations. You can view the program for the four days of the accessibility workshop here.

Background

The background for this workshop is the United Nations Convention on the Rights of People with Disabilities, which entered into force on May 3rd, 2008. Since then, many countries have ratified the convention and made it part of their national legislation.

In particular, Article 9 (g) and (h) of this Convention require that measures be taken to ensure that accessibility is taken into account in the design of new information technologies and systems.

WIPO

The WIPO is quite active in promoting accessibility. For example, they explored in detail how the needs of persons with print disabilities can be better addressed through trusted intermediaries, new technologies, better formats, and improved "best practices" in the publishing industry.

Just recently, the members of the Standing Committee of the WIPO from Brazil, Ecuador, and Paraguay submitted a draft treaty as a formal proposal to facilitate access to copyrighted works for persons who are blind, have visual impairments, and other reading disabilities. Such a treaty attempts to solve the problems of both production and distribution of accessible materials for print disabled persons and makes this type of access an integral part of the international copyright system.

Unfortunately accessibility doesn't happen overnight. Such a possible international instrument was already described and recommended as a solution by WIPO and UNESCO almost 25 years ago - but only gained real momentum in the last few months.

Presentation

Artur will raise awareness in his presentation of how many of the day-to-day items in our office are based on innovations by and for people with disabilities. The presentation shows how important accessibility and people with disabilities were, are, and will be for innovations for everyone. It's important to remember that without these essential innovations, today's publishing industry wouldn't be possible.

Sophie Davies-Patrick
Head of YDN International

Securing Web Extensibility

Sun, 31 Jan 2010 19:35:53 -0800

In recent years, we've seen increased energy put into web extensibility platforms. These platforms let distributed developers collaborate to produce new kinds of interactive features on websites and in the web browser itself. Because these platforms frequently enable data-sharing between multiple distinct organizations, and often sit between two completely different security domains (desktop vs. web), the security and privacy issues that arise are complex and interesting. This post explores some of that complexity: both the current state of platforms that extend the web and their associated security challenges.

Plugins, Extensions, and Mashups: A Primer

The extension platforms of the web today exist at every level of the technology stack. First there are mashup platforms, which include Facebook Apps, Google Gadgets, and YAP Apps, built on Yahoo!'s Application Platform. All these platforms are abstractions that allow developers to embed content within a host site. Often this content has a level of dynamism and interactivity that exceeds what is expected of "content" — so they're called "Applications" or "Gadgets."

A deeper type of extensibility occurs in web replacement technologies. Examples include JavaFX, Adobe Flash, Microsoft Silverlight, and Google Chrome Frame¹. These technologies leverage exisiting hooks in browsers to replace native rendering and scripting technologies from the browser vendors with new environments that claim to offer a variety of benefits.

In addition to web replacement technologies, there's the related area of web augmentation technologies. These use hooks exposed by browser vendors to bolt on native code, however, unlike replacement technologies, these tools attempt to expose new scriptable facilities to browser-based JavaScript. (Google) Gears, (Yahoo!) BrowserPlus, Mozilla Geode², and Loki are some prominent recent examples of such technologies. At the fringe of this category, there are several projects that abstract different replacement and augmentation technologies to provide feature focused abstractions. Prominent examples include include SVG Web for cross browser SVG support and the YUI Uploader for in-browser content upload with in-page feedback.

The existence of both replacement technologies and augmentation technologies for the web is made possible by the browser vendors themselves, who generally expose two different ways of extending the browser.

Browser plugins were originally intended to allow 3rd parties to add support to the browser so as to handle new types of content. They have since evolved into a rich means of embedding scriptable code into browsers, code that may (optionally) render in the browser's content area. (Microsoft calls these plugins Content Extensions, while other browser vendors have converged on the NPAPI architecture). A second, much more fragmented means of extending modern browsers is to write a browser extension (referred to as a "Browser Helper Object" by our friends in Redmond). Extensions typically have browser lifetime, as opposed to page lifetime. They also have programmatic access to manipulate the user interface of the browser.

In terms of Browser Extension environments, the most recent developments include Mozilla JetPack and Google Chrome Extensions. Both JetPack and Chrome employ web technologies to broaden their target developer audience. In Chrome, an extension is "essentially [a] web page", while with JetPack, "anyone who knows HTML, CSS, and JavaScript" can create an extension. Because of the large numbers of distributed developers authoring software which teeters between the world of unrestricted code running on your desktop and the sandboxed code running in your browser, these projects raise unique security issues. Our own work with BrowserPlus addresses a closely related set of challenges and will be used throughout the article as a point of comparison.

Sandboxing vs. Code Review

In designing an extension system where developers contribute code, there's a key upfront decision that has a very deep effect on the platform — To what extent should plugins be trusted? Should plugin authors be forced to attain the approval of some body of reviewers, or do we instead rely on a software sandbox to mitigate the potential harm that could be done by untrusted code from unknown authors (and hence relax the review requirement)? The former approach can minimize upfront investment in platform development, while the latter can reduce delays in publishing 3rd party code.

In addition to increasing complexity, the sandbox-based approach constrains the creativity of developers who use the platform. The universe of what is possible is pre-defined by the platform authors, which is contrary to the idea of an extension system. Thanks to Atul Varma's recent overview of JetPack development, I now can identify this tension as Jonathan Zittrain's "Generative Dilemma".

At first glance, the platform developer might feel stuck with a binary decision between generativity vs. self service (the ability of plugin authors to publish immediately without oversight or review). A little more thought, however, reveals that in reality there's a spectrum with several interesting intermediate choices. The first hybrid to consider: a sandbox with a set of additional capabilities or permissions that may be requested by code running therein.

Building a Capable Sandbox

In many ways, browser extensions can be considered conceptual peers of the browser: They share direct access to the end user and to the resources of the machine within which they run. Additionally, given the wide target audience of modern extension systems, developers with widely varied levels of experience and grasp of security issues are empowered to author and quickly publish extensions. Providing extensions with system access to permit generativity creates a tension with the requirement that extensions run in a locked-down environment to preserve reviewless publishing and prevent accidental or malicious end-user compromise. We seem to be converging on a capability-based security model, where sandboxed code may explicitly or implicitly³ request enhanced "capabilities" or "permissions." Several of the projects mentioned here have mechanisms which allow an extension to express its required permissions:

Chrome extensions require explicit expression of capabilities (or "permissions") in a manifest.
In JetPack, it may eventually be possible to automatically determine the capabilities required by a given JetPack, based on the dependency graph of "superpowers" it uses (see below)
In BrowserPlus, as in Chrome, permissions are explicitly stated in a manifest.

As different projects attempt to build capabilities-based systems, an interesting opportunity for collaboration arises. Users are going to learn to some extent what "writing to a temporary location on your disk" means. They will also learn more about "using location," and maybe also about "capturing video using a webcam." As new platforms emerge that ask the user more and more questions, it would be beneficial if these platforms shared similar capability lists, perhaps even a similar language or visual vocabulary.

From Capabilities to Meaningful Decisions

While we all seem to agree that an enumerable list of capabilities is a prerequisite to intelligent interface design, it's not clear that we've yet determined how to consolidate this information in a user interface that allows meaningful decisions. This is not for a lack of thought: Over a year ago, Dion Almaer suggested a "nuanced question" combined with good post-decision feedback. Since then, he's summarized ideas from Mozilla folks regarding a "layer [of] social trust on top of the technical security." More recently, there's been some initial discussion of a stop-light style representation of risk, which if correctly applied could distill a complex decision into a series of simple questions, (for example, Do you trust yahoo.com?) which iteratively becomes more ominous as the stakes (and the implied risk) get higher.

Better Responses Through Fewer Questions

One approach employed by BrowserPlus attempts to minimize explicit prompting and leverages mechanisms of implicit consent instead ⁴ by carefully crafting the API between trusted and untrusted code. One such example is file selection, where the act of navigating an operating system-supplied file picker dialog indicates implicitly that the site should be allowed to read the contents of selected file(s). Another example where this technique might be applied involves webcam access. Rather than asking the user if a site may use the webcam, pop up a window displaying the view from the webcam, allowing the user to capture a picture inside that frame (outside the control of the page), and finally after capturing "share" the capture with the page.

Architecting APIs and UI in this way can somewhat alleviate the need for explicit up-front questions and allow in-context kinesthetic decision making to occur. This approach is not without fault however, as it can also confine the freedom of UI designers. Eliminating needless questions assumes that only by asking fewer questions of higher quality, will we truly empower the user to make meaningful decisions.

Permission Prompts in Practice

In addition to ongoing thought experiments in meaningful user prompting, there are plenty of instances in practice where user questions have been built to varying degrees of success.

Facebook combines "social trust" (the rating) with a generic bit of language about how an app can poke at your stuff:

My Yahoo! takes a language-laden approach to the problem:

BrowserPlus uses a lot of screen real estate to convey a fairly literal translation of capabilities into human language:

Google Gears adds a visual representation of a capability and a prominent display of who is asking:

Language Choice and the Generativity Continuum

The choice of implementation language adds another dimension to the problem of security in web extension systems. As might be expected, native code (C, C++, or Objective C) will always afford maximal access to the system, yielding the ultimate in generativity. Certain features cannot be accessed from a higher level language — some good examples include access to physical position information from a motion sensor⁵ or access to a wifi card to extract nearby wifi base stations with associated signal strength. The benefit to working at such a low level is, in turn, this freedom. Ultimately, attempts to introduce sandboxing into an extension system that supports native code run the risk of introducing extra developer facing complexity without giving much in return⁶. In general, the choice of implementation language can bound where an extension system falls on the generativity continuum:

Languages like Lua or JavaScript land on the other side of the extreme and apply well to a system where automatic publishing is important. In the case of JavaScript, there is no common defined standard library⁷, so this makes it possible to introduce new protected file system APIs (for instance) without surprising a developer. This same reasoning is what places Ruby and Python in the middle of the continuum. While it would be possible to embed Ruby and shoehorn in a capability-enforced access system (or perhaps use a modified version of the "Safe" mechanism that already exists in Ruby), it would yield a system sufficiently different from expectations that it might feel clumsy or confusing to developers.

While none of these claims are without exception, it is true that the two newest browser extension platforms today are built upon JavaScript and are sandboxed and capabilities-based. Older plugin and extension systems tend to leverage native code and either use code review or don't even attempt to address the issue of security in any granular way.

Case Studies in Extension Security

So far we've enumerated several pertinent decisions that extension platform authors must make:

Are submissions sandboxed or reviewed?
What language are extensions written in?
Is the system capability based? If so, how is this implemented?

As it turns out, the two newest browser extension platforms avoid straight answers to these questions. Given the tradeoffs with each decision, both systems are hybrids that attempt to capture the benefits of reviewless publishing, yet still preserve the generativity of native code. They leverage JavaScript to provide an authoring environment that's accessible to a majority of the world's developers, but allow the sprinkling of native code in controlled ways to preserve generativity. It's deeply interesting to look at how these platforms straddle some of the trickiest questions.

Google Chrome's Extensions

Aaron Boodman, working on extensions for Google Chrome, recently spoke about where Chrome can be found on the generativity continuum. Chrome uses a hybrid approach where "webby" extensions may optionally include native code bundled as scriptable NPAPI plugins. Extensions which are based solely on the Chrome extension environment's JavaScript APIs need no review and are "enabled automatically." Extensions that include unsandboxed native code are subject to additional review. In Aaron's words:

Aaron's comments are at 21:30

We're really proud of the fact that in the Google Chrome extension gallery we enable extensions automatically, there's no review period. But we make an exception for NPAPI because when you get to native code a lot of the security mechanisms built into the extension system can no longer apply. Once you have native code running it can modify the registry or make permanent modifications to your system... So we have additional review for NPAPI extensions.

This strategy seems reasonable. One would hope that a lion's share of extensions are developed without reliance on native code. The set of extensions that bundle native code can then be periodically examined. Those which leverage native code to do things that may be interesting to a large number of extensions developers and can be considered for uplifting into the core platform.

Mozilla's JetPack

One interesting thing that the Chrome model lacks is a means for sharing privileged functionality between extensions without actually merging it into the core extension platform. JetPack, on the other hand, is taking a much more ambitious initial approach to building a platform that can more rapidly evolve. Atul recently summarized the design goals of JetPack's capability-based security model. They aim to "allow anyone to create capabilities that securely expose privileged functionality." The JetPack world might end up looking something like this:

The key difference from Chrome extensions is the existence of a middle tier consisting of community contributed bundles. These provide JavaScript APIs to access system resources not otherwise exposed by the core extension system. By breaking out potential platform features into a separate layer you empower the community to not only imply the features that they'd like to see in the plugin platform, but to go and build them. This is an exciting idea. It will be interesting to see how this middle tier of superpowers is embraced by the community of extension developers, as well as to learn the details of how the review process is structured.

Today's Extensions as Tomorrow's Web

Web extensions today are a dynamic and fascinating area at the intersection of several disciplines. This post attempts to give a taste of some of the considerations and tradeoffs involved in the architecture of such systems. Hopefully I've sparked your interest to learn more or perhaps even join the fray. A renewed interest in more open software platforms is no surprise, and the potential to harness the creativity of developers of the world is inspiring.

One area, however, that dampens the joy of web extension platforms covered here is that they're limited in scope — vendor locked and browser specific. While we'll hopefully see increased (and appropriate) cross-vendor collaboration, it's reasonable to conclude that because "browser extensions" are about extending web browsers — and because different web browsers can have vastly different architectures — browser extension APIs will be slow to converge.

Imagine, however, an extension system which struck a balance between simplicity, generativity, and security, and was pertinent to the whole darn web. This is the promise of Web Augmentation technologies. Distributed Extensibility has been a topic of discussion in terms of HTML5 for a while now, but the conversation thus far has failed to cover a distributed way to explore new standard JavaScript APIs (the stuff that so many emergent HTML5 standards are made of). A significant obstacle to this conversation is the problem of helping users understand and securly navigate a dynamically evolving web.

The current work being done to secure extension platforms logically extends to web augmentation platforms. The degree to which we can collectively refine these permissioning models may become the key limiting factor in how far we can migrate our current desktop experience to the web.

Lloyd Hilaiel
BrowserPlus Hacker

1. Chrome Frame is certainly a fringe case where the rendering engine of one browser is replaced with that of another. Discuss.

2. Geolocation was ultimately built into Firefox 3.5, however Geode still serves as a great example of web augmentation — in this case using web extensions as a way to experiment with new potential core features for the web.

3. Whether the request of capabilities is implicit or explicit is an interesting related question. Having a platform that can discover the required capabilities of an extension without the author having to explicitly enumerate them would yield something that's easier to develop for, at the cost of complexity for the platform implementor.

4. The notion of implicit consent isn't new — there's precedent, for instance, in Flash where certain API functions can only be performed from an event handler generated by a user action (mouse or keyboard). The simple idea is that user actions can be a meaningful part of the security system.

5. What is possible without native code is obviously a moving target. Browser-based access to the motion sensor was introduced using BrowserPlus a couple years ago, and subsequently has made it into Firefox 3.6. What's next?

6. The potential efficiency of native code is an exception, and Google Native Client is an example of a sandbox system around native code. The project focuses on enabling the efficiency of native code with the saftey of the web, and additionally wraps and exposes some system resources, such as audio. Another glaring exception might be Apple's sandboxed environment for authoring iPhone Apps, a case where Objective C affords a slightly higher level of abstraction for developers yet still allows native code efficiency — important on a mobile device.

7. The CommonJS movement is an attempt to bring a standard library and means of including (or "requiring") code to JavaScript. While this could standardize APIs for JavaScript, developers will still generally be forced to understand the difference between core JavaScript and libraries provided by the execution environment.

First we take London: The Social Pattern Detective in Europe

Sat, 30 Jan 2010 16:46:04 -0800

A week or so ago I undertook a whirlwind visit to the UK and the Continent, giving two presentations about design patterns and social design, one in London on Tuesday, and another in Berlin on Thursday, each event sponsored by YDN (and the one in Germany co-sponsored by the local IxDA group).

The London event was in a wonderful gallery/cafe venue called Wallacespace filled with a standing-room only crowd. I was pleased to see a couple of friends from the international UX community there and the audience as a whole was wonderful, attentive, and ready with interesting, challenging questions for me when I was done.

Afterward we ate some snacks and drank some beers courtesy of YDN, before heading over to a nearby pub for more beers and conversation. This was my first time back in London in fourteen years and I was impressed by the vibrancy of the web-design community in what may be the "capital" of the Web in Europe.

The next day I headed to Berlin, where a pal picked me up at the airport and helped me get settled in my hotel in Alexanderplatz. It's actually been 20 years since I was in Berlin! Back then, the Wall had only recently been dismantled and the east was frozen in a sort of time capsule due to economic stagnation. A lot has happened since Berlin reunited and resumed its role as the capital of Germany and arts mecca of Mitteleuropa. In fact, there was a fashion convention going on during my visit, so the airport and hotel were full of people who made me feel, in comparison, more like a geek than a designer.

East Berlin is now full of trendy gentrified neighborhoods. I had lunch at a burrito place (!) called Dolores that's decorated with maps of the Mission in San Francisco. Clearly the internet-savvy crowd in Berlin feels a kinship with our own community in the Bay Area.

Berlin is also the home of a thriving local Interaction Design Association (IxDA) group, which helped secure the venue for my talk--(Newthinking Store) and helped promote and publicize my talk. I had a chance to meet some longtime virtual acquaintances from the IxD and IA communities in Berlin, such as Jan Jursa, of IATV and the Berlin IA Cocktail Hour.

The Berlin talk was also full, and again I was blessed with a generous and attentive crowd. More great questions. (We did the whole evening in English. Try as I might to speak slowly, I still probably spoke a bit too fast at times but just about all the German I know is noch ein Bier, bitte so it's just as well.)

One interesting difference between the two groups is that the folks in Berlin asked me more process questions: How was the social design project organized? How did the wiki figure into the writing of the book? What's an unbook? and so on. The questions in London tended to be more about the efficacy of design patterns in general and the application of social design patterns.

At both sessions, certain attendees had reached out to me in advance over Twitter and proposed questions that they had a chance to ask at the events. In London and again in Berlin I was asked the perennial question about whether the use of design patterns stifles innovation. My traditional answer, "No. Now shut up and do your wireframes!" got a laugh in both settings as well. (My real answer: "Not if they are applied as guidelines and with sensitivity to context.")

One other curious difference between the two events was that the audience in London had nearly perfect gender balance, whereas the one in Berlin was, by my estimate, about 90% male. I'd like to learn more about what the differences are between the web design and development communities in the two cities that might account for that variance.

I'd like to thank YDN for sponsoring the trip, and O'Reilly Media for providing logistical support (and some copies of the book to give away as rewards for great questions). Interested folks can see my slides on Slideshare:

Yahoo! Pattern Library & Social Design Patterns

View more presentations from Christian Crumlish.

Several attendees in London took great notes of my talk and published them on their blogs or personal websites, including Jeff Van Campen, Suw Charman-Anderson, Michael Mahemoff, and O'Reilly's Craig Smith.

Also, this may be as good a place as any to share my upcoming speaking schedule over the next few months:

Next week Erin Malone and I will be playing our Social Mania game at Interaction 10 in Savannah, Georgia.
February 23 I'll be speaking at the Bay Area Application Developers Adobe User Group's monthly meeting.
At Ignite SF I'll do a five-minute 20-slide version of "Designing for Play."
I'll be signing books (and talking about hacking the ukulele fretboard) at South by Southwest in Austin in mid-March.
Mar 23 I'll be speaking in San Diego on the Web App Masters Tour.
May 3 I'll be speaking on the topic of "Designing for Play" at the Web 2.0 Expo in San Francisco.
May 19-21 I'll be up in Portland for Web Visions with my colleague, YDN evangelist Tom Hughes-Croucher. He'll be presenting Let's Run Javascript Everywhere on the phenomenom of server-side javascript. I'm presenting ideas on Designing for Play. Also, Erin Malone and I will be running our Designing Social Interfaces workshop, and Erin will be presenting her Go with the Flow talk.
And on June 9-11 I'll be back in London, presenting on Designing for Play and running a Design Social Interfaces workshop at @media, along with Tom H-C, who'll be presenting an Intro to Server-side Javascript.

Image credit: Jeff Van Campen

Christian Crumlish
Pattern Detective
design.yahoo.com

Cloud Computing Brainiacs Converge on Yahoo! for Open Cirrus Summit

Thu, 28 Jan 2010 19:01:33 -0800

Today and tomorrow, Yahoo! is hosting the second Open Cirrus Summit, attended by cloud computing thought leaders from around the world. Computer scientists from leading technology corporations, world-class universities, and public sector organizations have gathered in Sunnyvale to discuss the future of computer science research in the cloud. The breadth of the research talent is expanding this week, as the School of Computer Science at Carnegie Mellon University officially joins the Open Cirrus Testbed.

Open Cirrus Summit attendees
Image credit: Yahoo Developer Network
View more photos from the Open Cirrus Summit Cloud Research Event @ Yahoo!

The event will feature technical presentations from developers and researchers at Yahoo!, HP, and Intel, along with updates on research conducted on the Testbed by leading universities. Specifically, the Yahoo! M45 cluster, a part of the Open Cirrus Testbed, is being used by researchers from Carnegie Mellon, the University of California at Berkeley, Cornell, and the University of Massachusetts for a variety of system-level and application-level research projects. Researchers from these universities have published more than 40 papers and technical reports based on studies using the M45 cluster in many areas of computer science, with several studies related to Hadoop.

Yahoo! is the world’s largest contributor to Apache Hadoop, and last year we open-sourced a production-ready version of Hadoop, the Yahoo! Distribution of Hadoop. We’re especially excited that the global community of researchers using Hadoop has expanded because of the Open Cirrus Testbed.

pipelining data between map and reduce stages of Hadoop jobs to improve user interaction
deploying log-analysis techniques to improve performance of Hadoop clusters
applying RAID techniques to improve the Hadoop distributed file system

using the cluster to continuously extract knowledge from Web pages, the Read the Web Project
experimenting with new natural language processing models
exploring large-scale graph algorithms
analyzing Wikipedia group dynamics benchmarking statistical machine translation techniques
performing large-scale document analysis
prototyping statistical machine learning algorithms and studying computational sustainability

The Open Cirrus Cloud Computing Testbed is a unique initiative because it enables research beyond the application-level, allowing experimentation with the system software itself. A complete, open-source cloud computing software stack is also emerging from work at Open Cirrus. This stack consists of four layers:
1) Pig, the top layer, is a parallel programming language for expressing large-scale data analysis programs. Pig was designed and developed by researchers at Yahoo! Labs;
2) Hadoop, the layer below Pig, is a distributed file system and parallel execution environment that can run Pig/Map-Reduce programs.
3) Tashi, an Apache Incubator project and the layer below Hadoop, is a cluster management system for managing virtual machines;
4) Zoni, the bottom layer, available in the Apache Incubator within Tashi, is a service that manages VLAN-isolated computer, storage, and networking resources.

Pig, Hadoop, Tashi, and Zoni are all open-source projects available for worldwide cloud computing research and experimentation.

Yahoo! has built one of the world’s largest private clouds and uses the cloud to accelerate innovation and improve its consumer and advertiser experiences. Almost every part of Yahoo! now touches the cloud. All of us involved with Open Cirrus are excited to be at the forefront of cloud computing research and proud of the unique contribution of our testbed and our software stack. We look forward with great anticipation to future research from the Open Cirrus team.

Ron Brachman
Vice President, Yahoo! Labs and Research Operations, and Head, Yahoo! Academic Relations

Tech Thursday: Browsers looking like Firefox, SVG, Windows 3.11 and the internet 2009

Thu, 28 Jan 2010 07:46:45 -0800

Every Thursday is Tech Thursday where we share a random assortment of technical links we found and liked.

Inspectelement shows how to how to code a backwards-compatible one-page portfolio with html5 and css3
Michael van Daniker has a beautiful visualization of browser statistics from 2002 until now.
SVG has been hot lately. Sitepoint has a what everybody ought to know about SVG right now article and A List apart has a double feature on using SVG for backgrounds.
Ever wondered if you can simulate Windows 3.11 in HTML, CSS and JavaScript? Well, you can.
Webaim has a long treaty how to construct an accessible web site by putting people in the center of the process.
The Internet in numbers

You can propose links to us on Twitter (@YDN) or try bookmarking them on delicious with the tag "forydntt".