Transcoding

David Harper

Rules for Responsible Reformatting: A Developer Manifesto by Luca Passani

Rules for Ethical Reformatting: A Developer Manifesto by Luca Passani.

A ghost is haunting the mobile web, the ghost of content reformatting. The mobile web has never been a simple platform to develop for, given the fragmentation of the underlying technology: different devices, different browsers and different networks have made mobile web development a challenge for programmers and content authors from the beginning.

Yet, in this hard environment, one thing has stayed sacred throughout the years: the HTTP Protocol.

Until today, developers of mobile websites could count on the fact that the HTTP headers from a mobile device would reach the intended webserver with their integrity respected.

Alas, this basic assumption on which thousands of developers have built their applications is put to the test by a recent trend among Mobile Network Operators (MNOs).

Tools that were originally devised as utilities to split up web pages and deliver a best-effort user-experience on a mobile device are today being marketed as solutions to bring the full web to mobile by reformatting vendors. Some MNOs are buying into this vision and are implementing reformatting proxies into their networks that will intercept and reformat all HTTP traffic to any HTTP server without safeguarding those sites which strive to deliver a mobile-optimised user-experience, or, at least, not safeguarding them sufficiently.

This situation is a threat to the neutrality of the web and one that can jeopardize the mobile web as a platform in the years to come.

To this end, mobile developers of various nationalities and background, assembled on the WMLProgramming list at Yahoo Groups create this manifesto to make their position known in the face of those who try to misrepresent the needs and the wishes of the mobile ecosystem for their own petty monetization needs.

Learn more and consider lending your support: Rules for Ethical Reformatting: A Developer Manifesto by Luca Passani.

–David Harper, Founder & CEO, Winksite

David Harper

The New Walled Garden?

It’s getting even worse for mobile developers and content providers.

Now it’s Yahoo!/Novarra. Who else?

Novarra to Provide Mobile Web Tranformation Services to Yahoo!

Chicago, IL USA – July 24, 2007 – Novarra, the leading provider of next-generation mobile Internet platforms and services, announced today that Yahoo! has selected Novarra’s Vision™ server version 6.5 platform to provide web transformation for Yahoo!’s oneSearch service. When a user clicks on a link to a Web site in a Yahoo! oneSearch result, Novarra’s Vision server platform transforms the Web site for the mobile phone.

We decided to check up on what was happening to Winksite.

This is the result…

Yahoo!’s oneSearch/Novarra is providing mobile device search results that link to transcoded versions of our broadband web site rather then linking to our mobile version.

This is after requesting that our domain be placed on the oneSearch White List MONTHS AGO and submitting three of our sitemaps for indexing – broadband site, WML, & XHTML.

WTF.

I have a responsibility to 25K plus publishers who trust their mobile sites to our care to do everything I can to see to it that their content makes it through to their audience.

Sorting out an off-portal publishing system to deliver mobile device-specific 100% W3C mobileOK and .mobi standards-compliant templates was apparently not enough. Now it seems I have to take on everyone in the world who is bastardizing us with their freaking Novarra “web transformation” engines and poor customer service (i.e. whitelisting.)

Is there a reason why mobile developers and content providers have to jump through so many hoops and fight so many battles in order to reach their viewers in a manner equal to how the broadband Internet operates everyday?

Is anyone listening?

CLARIFICATION:
Yahoo!’s oneSearch service clearly distinguishes mobile and web results. They are returning both PC and desktop links to Winksite. Links in the PC section goes to the transcoder, while links in the Mobile Web section goes direct.

So why do I have a problem with this? Let me explain…

When someone visits Winksite or a mobile site published at Winksite we send that visitor to a version appropriate to their device. We asked to be White Listed because a mobile phone should never end up at a transcoded version of our broadband site. If Yahoo! respected that ALL the links in the results would pass to us directly where we could then do the job that people count on us to do.

Instead our routing is intercepted and adapted content NOT our content or the content of our publishers as it was intended to be distributed is sent. This cripples our site and renders our community features unusable.

I question Yahoo!’s (and anyone else’s) right to do that…

…and without permission to create a derivative work that violates our copyright AND hinders our ability to provide a service that took years of hard work to build.

David Harper

Vodafone UK is Clearly Wrong – The User-Agent String Issue

UPDATE:
Nigel Choi has kindly invested his last weekend to provide the community with evidence of how the Novarra/Vodafone transcoding service disrupts the mobile experience:
http://wurfl.sourceforge.net/vodafonerant/vodawsj/nigel.html
————

For several weeks a firestorm has been taking place at Vodafone’s Developer Forum (Betavine, Mobile Internet Content Adaptation forum)

Simply put…

Vodafone UK started striping out essential device identification information that mobile phones send to content providers (i.e. the User-Agent String.)

The User-Agent String is used by developers and content providers to deliver mobile-optimized sites OR device-optimized content/services.

Vodafone’s actions thwart the efforts of companies in the mobile ecosystem who set out to provide a customized mobile presentation of their services, hurt these companies financially, and is counter to the advancements facilitated by groups such as the W3C and dotMobi.

For a distilled explanation of the Vodafone User-Agent string issue and how it affects the mobile Web please visit “Vodafone UK is abusing its position” by Luca Passani.

Mike Butcher over at Techcrunch UK has picked up on the story.

Luca Passani:

I am irritated with Vodafone. More that that. I am furious. I see an abuse and I am not sure what to do about it. But it’s an abuse. A Big one. Perpetrated by a large company in a dominant position against a myriad of small companies and against its own customers. An abuse that is damaging a whole industry in its infancy. I am talking about the industry of the mobile internet. I am talking about the possibilities for existing and new companies to have a new channel for selling content and services to consumers, and about a company which, from one day to the next, decides to pull the plug on the infrastructure that made this possible. The plug is pulled because this decision makes some tiny extra business sense for the big abusive company here and now, but it has no legitimacy whatsever, and the reason why the big abusive company can do it is merely technological: they manage the pipe the brings the data from the service provider to the consumer, and they decided to exploit this possibility to cut everyone else out.

Is this legal? I don’t know. Probably not. The problem is that, being this a relatively new field, there are no specific established regulations which spell out clearly what companies can or cannot do. By exploiting this uncertainty, the big abusive company is applying its dirty tricks and hoping to get away with it.

The abusive company I am talking about is Vodafone UK and the abuse is their decision to strip out essential device identification information that mobile phones send to content providers in order to let them serve customized content for each user’s device.

I want to bring the problem to public attention, make people aware of the issue and get everyone involved to do something about it. Read on…

David Harper

How to Free Yourself From Google's Mobile Transcoding Services in 5 Days or More.

The other day I wrote about Google’s mobile transcoding services and the reasons for our issues with it in a post titled, “An Open Letter to Google: “Page adapted for mobile phone?” Please stop now, you are crippling sites, not adapting pages.”

The lack of clarity as to how the transcoding services worked and how you could opt-out was frustrating. In our case, Google’s transcoding service was doing more harm then good. Naturally we “Googled” Google and eventually found a mobile FAQ that stated:

Our system automatically translates regular pages into wap-compatible [i.e. mobile] pages. In cases where a wap-compatible site already exists, we redirect the user to the wap-compatible site instead of translating the page ourselves.

We found that not to be the case. In Scott Rafer’s words:

To use WINKsite as a specific example, Google Mobile Search ignores the mobile site that took our founders years of hard work and which is now a global mobile community used by tens of thousands of people every day. Instead, Google Mobile Search uses our standard web homepage, rips it apart, and sends it to people’s mobile phones as 15 crappy pages of unintelligible garbage.

Reading back through the FAQ later we discovered that Google did in fact provide a method for web site publishers to be excluded from Google Mobile’s re-rendering by sending a “removal request” by email to mobile-support@google.com. Accordingly we sent this email to Google on 3/12/2006 11:16 PM:

Please remove transcoded pages for http://winksite.com (re: http://www.google.com/webmasters/remove.html#transcoded)

Thank you.
Dave Harper
Founder, Wireless Ink/WINKsite

On 3/12/2006 11:16 PM we received the following response:

Thank you for your note. This is just an automated reply to let you know that we received your email. We’re currently putting most of our energy into improving Google Mobile, so we can’t promise a personal reply to every question. Please be assured that your feedback will be used to improve Google Mobile. For a quick answer, you may want to visit our FAQ at http://www.google.com/mobile/faq.html

Regards,
The Google Team

Followed the next morning (3/13/2006 11:46 AM) with:

Hi Dave,

Thank you for your note. As requested, your page will no longer be transcoded to a mobile-ready format.

We’re constantly working to improve the way pages are displayed on mobile web browsers. If you have any suggestions, please let us know.

Regards,
The Google Team

Well nothing changed, our audience was still blocked from our mobile site, and a debate which originally started at MobHappy continued here, here, here, and here.

Finally on the morning of 3/17/2006 I awoke to discover that after 5 days of jumping up and down on blogs all over the blogosphere — WINKsite had been set free. “Being Free” is good.

…but what about others in the same fix.

The process and exclusion list doesn’t seem like a good permanent solution (and it’s not just Google that’s handling mobile wrong.) What’s needed is a way to respect the rights of mobile publishers and their audience without making them jump through hoops. The conversation has started. Hopefully the dialogue leads towards a better understanding of the mobile web.

Count us in.

Update: Received 3/17/2006 8:19 PM

Hi Dave,

We just wanted to let you know that your site is no longer being transcoded. We apologize for any inconvenience this may have caused.

If we can assist you further, please let us know.

Regards,
The Google Team

David Harper

An Open Letter to Google: “Page adapted for mobile phone?” Please stop now, you are crippling sites, not adapting pages.

Dear Google,

I just used Google on various mobile devices (i.e. a Nokia 6600, Sony w800, and Motorola RAZR V3) to search for my service Winksite.com. Rather than link to Winksite directly the results instead link to a reformatted version that in Google’s words is “adapted for mobile phone”.

You’re assuming way too much power in hijacking Winksite and replacing it with your own version. The 15-screen, bastardized version of the site that you deliver is not even close to what WE want to deliver or what our audience expects to receive. Let me explain.

From my perspective, the issue is not that Google unilaterally strips away eye candy only to deliver a hodgepodge of text on mobile devices. It is that you remove user access to mobile-specific services on which ours and many other businesses are based. By default, your actions censor those of us who provide a unique and/or useful mobile experience.

The mobile web and its proponents were in place for years before Google “discovered” the mobile web and started to hijack it. Individuals, small development teams, and companies that respect and value the mobile audience provide mobile sites and services designed for that audience. With a bit of browser detection, Winksite and others send these visitors to either mobile optimized versions of their sites or even mobile phone-specific services. Google mobile web search intercepts and overrides that detection, context, and delivery. I question your right to do that and without permission to create a derivative work. (At Mobhappy Google’s tactics are questioned for other reasons – Who Gave Google Permission to be the Judge and Jury of Mobile Content?.)

Google’s actions cripple every truly mobile web site that its search uncovers, violating the copyright of each and reducing all of them to a lowest common denominator that sets the mobile web back ten years. “Do No Evil” requires that you stop now.

Dave Harper
Founder, Wireless Ink

P.S.
Scott continues the conversation with “Garbage 2.0 In, Garbage 2.0 Out“.