Posts Tagged ‘government’

UK Government says no to upgrading IE6 – who is to blame?

Thursday, August 5th, 2010

Back in June Dan Frydman of Inigo Media Ltd submitted a petition to the UK government to encourage government departments to upgrade from IE6 and 6223 people signed it.

A short time ago we got an answer by her Majesty’s government which was a no – of course.

Government says no

Disregarding the horrible PR mumbo-jumbo re-assuring us that the government takes security serious (when they are not leaving personal data files on trains) it gets actually interesting:

Complex software will always have vulnerabilities and motivated adversaries will always work to discover and take advantage of them. There is no evidence that upgrading away from the latest fully patched versions of Internet Explorer to other browsers will make users more secure. Regular software patching and updating will help defend against the latest threats.

This of course is a wonderful example of stating the bleeding obvious, but it is interesting that there is “no evidence that upgrading IE6 makes computers more secure”. I wonder why Microsoft then keeps advertising that IE8 is more secure? True, IE6 can get all the patches for massive attacks but phishing warnings and other interface changes in latest browsers do not get added. So we protect users under the hood but we still leave the barn door wide open for social engineering attacks. A malware warning like Firefox, Chrome or more modern IEs have would help there (unless it gets removed when it affects advertising). If there actually is no proof it would be a good opportunity for Apple, Google and Mozilla to collect some numbers and publish them – not on blogs or other “in crowd” media but in the magazines read by the people who make IT decisions for governments and large corporates.

Security patching not an issue?

The government statement then continues to stress the great relationship they have with Redmond for security related matters:

The Government continues to work with Microsoft and other internet browser suppliers to understand the security of the products used by HMG, including Internet Explorer and we welcome the work that Microsoft are continuing do on delivering security solutions which are deployed as quickly as possible to all Internet Explorer users.

There is a distinct lack of information about what they are – both the other suppliers or the measures. My guess is that Google is starting to approach governments with Chrome and the online office suite. Let’s note down though one thing here: that there is no problem to deploy fixes very quickly to all IE users – we will go back to that.

No centralised security mandate?

Each Department is responsible for managing the risks to its IT systems based on Government Information Assurance policy and technical advice from CESG, the National Technical Authority for Information Assurance. Part of this advice is that regular software patching and updating will help defend against the latest threats. It is for individual departments to make the decision on how best to manage the risk based on this clear guidance.

So, wait – beforehand we were told that there is continuous patching with ease as Microsoft helps a lot and now we learn that it is up to the department to really follow that advice. It is not a mandate, but only a guidance. This means actually that there are probably terribly outdated IE6 in use as changing the IT infrastructure is quite low on the list of priorities for a lot of departments when there are people in the waiting rooms complaining. Which means that if upgrading and patching is not centrally mandated there is no chance we’ll ever have a secure and homogenous IT environment in government bodies.

A departmental decision?

Public sector organisations are free to identify software that supports their business needs as long as it adheres to appropriate standards. Also, the cost-effectiveness of system upgrade depends on the circumstances of the individual department’s requirements.

Which means that a department could switch to other software – especially when they could save money? The catch here is “appropriate standards” which probably means a EULA. Or what, exactly? The other big “oh well, we really can’t do that, can we” here is the cost-effectiveness of a system upgrade. In many cases of Microsoft systems this probably means that the hardware in use is not up to scratch to support other OSes than Windows 2000 or XP1.

Upgrading is an issue?

It is not straightforward for HMG departments to upgrade IE versions on their systems. Upgrading these systems to IE8 can be a very large operation, taking weeks to test and roll out to all users.

How so? Earlier we heard that patching IE is not an issue, so how is replacing IE an issue? Unless of course we’d own up here and admit that it is the infrastructure and the hardware that was defined and set in stone around the millennium when all were scared about Y2K and believed that the IE6/XP Suite will never have to be upgraded.

No time for testing?

The other issue seems to be that testing our systems is hard:

To test all the web applications currently used by HMG departments can take months at significant potential cost to the taxpayer. It is therefore more cost effective in many cases to continue to use IE6 and rely on other measures, such as firewalls and malware scanning software, to further protect public sector internet users.

This to me says that there are systems that were built in a short-sighted manner a long time ago – for IE6 and windows 2000 when they were the new black and every consultant got his Microsoft certification training and out of a sudden was a real expert who can predict the future of the next 10 years. So instead of fixing and replacing the rotten core of the system we add new doors with shiny hinges and a security guard before it and it will be fine. This is like hiring a bouncer for a club where people fight on the dance floor.

The fascinating part of the firewall and malware scanning software is that it makes the life of the end users even more hell than surfing with IE6 already is. One of my favourite things when I switched to Mac/Linux is that my processor can now deal with stuff I want to do rather than analysing my traffic and incoming requests and that I can work without being interrupted by a “scanning all your files, come back in 2 hours” message.

Who is to blame?

The answer of the government was not only predictable, but (in a very shortsighted and limited view) also understandable. Nobody wants to own up having been cheated. And consultants telling people that a network will never have to change do cheat people – no software is 100% future-proof and you cannot run an office on 10 year old hardware without upgrading. The speed of innovation and wealth of information we encounter these days can not be easily consumed on systems that were meant to be used when having a 100kb JPG on the homepage was a huge decision and meant you lost 1/3 of your visitors.

Funnily enough the easiest and favourite target of web geeks in this issue – Microsoft – is not to blame. They do offer a simple way to make their new software support IE6 with a meta tag or – much more appropriate – with a header send by the server (IIS in this case). So the argument that software built for IE6 has to be tested by every department on IE8 is moot as Microsoft solved that issue for us. That the government probably didn’t even know about that option is where it gets interesting:

Reactions like this to an obvious upgrade are our fault

To a degree I have to say after all my years on the web and as a developer, writer, blogger and editor we are the first to blame for no movement in large corporations and the government.

When luminaries of the web design and web development world only showcase things made up to use a certain new technique instead of real world examples it is not surprising that developers working for government agencies don’t get sent to conferences or get their books.

When famous designers say that working for a large company or government is “boring work” and “that there is no point for a creative person to deal with politics in companies” then I really wonder if we have become self-sustaining and complacent. We moved on from shaking the foundations of web development and making people understand the massive opportunity the web as a media and the open web technologies as tools represent to inventing for ourselves rather than for the end user. What will have more users who are much more frustrated when something doesn’t work? The readers of a famous design blog or people who have to pay their council tax online?

When industrial grade research information and tools from companies like Yahoo, Google and Microsoft are never read or – even worse – reproduced in a shinier but less consistent manner by one man army companies and considered to be better (until the one man army is bored of it a month later and never updates) then there is no wonder that other companies don’t believe in these solutions either. Furthermore it means that these companies – who really formed and run the internet as we know it now – will stop sharing their tricks or spending time and money writing them down in a manner that makes sense for people not on the inside.

Shifting our focus

The only way that I can see how responses like the one from the UK government can be prevented in the future is by shifting our focus:

  • Instead of design prototypes and made-up web sites to show a certain technique let’s demand real production case studies and their effects (I remember one @media where the redesign of blogger was shown and how much traffic shifting to CSS saved the company – more of that, please).
  • Ask Microsoft to invite experts, host videos and tutorials of experts with modern solutions and distribute them on their network of clients
  • Make a massive comparison of government web sites and praise what some have done well (nothing works better than competitiveness)
  • Collect success stories of switching to open source solutions and how it saved money and time
  • Take a horrible IE6 only solution and show what it could look and work like if HTML5 and CSS3 were supported
  • Stop plotting shiny pixels on canvas elements and call it a cool HTML5 solution and instead build a complex online form or spreadsheet system using all of the goodies of HTML5
  • Stop applauding people for redesigns of their blog and instead shift people into the limelight who made a difference in an environment like large financial systems or local government

I’ve had these and other points in 1:1 discussions for years now and I yet have to see movement in these areas. Right now, we are happily thinking we innovate and push the envelope where in reality we are making each other go “Oooohhhh” while a large chunk of the audience that could benefit from our knowledge is stuck with really poor experiences on the web. I’d like to pay my council tax on my mobile phone’s browser and get notified when I need to do it – right now there is no way to do that.

How I build my data.gov.uk mashup – UK-House-Prices.com

Thursday, January 21st, 2010

UK-House-Prices.com is a web site to see how the prices in a certain area changed over the years using a data set released by the UK government as part of the data.gov.uk initiative.

Here’s a screencast showing the app:

The first step was to get the right data. I was lucky enough to be invited to the initial “hack day” and pre-release of the data and looked around for something to mash up. Initially I wanted to do something with environmental data but I found a lot of it to be very old. Therefore I just did a search for “2009” at data.gov.uk and found that the house prices data from 1996 to now in England and Wales is available. The plan was set. This was it:

  • I wanted to build an interface to show this information that was very fast, very portable and show a nice map of the area next to the numbers.
  • I wanted to build this as a web app and as an application for the Yahoo homepage (as I needed to build one as a demo anyways)
  • Traffic and speed was the most important issue – as this might get huge.

Cleaning and converting data

I got the spreadsheet and was confronted with my old nemesis: Excel. After saving the sheet as CSV and spending some fun time regular expressions and split() I had the data in a cleaner, and more usable version (JSON, specifically). One fun part is that when there was no data available for a certain area the field was either “..”, “n/a” or just empty. Something to work around. The numbers were also formatted like 100,312 which is nice on the eye but needs un-doing when you want to sort them outside Excel.

Once I had the list of locations and their numbers I wanted to turn them into geographical locations to display maps of the area. For this I used Yahoo Placemaker, especially the YQL table (see an example for Rugby in the YQL console). This is the script I ran over the list of locations:


$out = '';
for($i=0;$i<sizeof($lines);$i++){
$select = preg_replace('/,.*/','',$lines[$i]);
$select = preg_replace('/ UA/','',$select);
$url = 'http://query.yahooapis.com/v1/public/yql?q=select%20match.place.woeId%2Cmatch.place.centroid%20from%20geo.placemaker%20where%20documentContent%20%3D%20%22'.urlencode($select.',uk').'%22%20AND%20documentType%3D%22text%2Fplain%22%20and%20appid%20%3D%20%22%22%20limit%201&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
$data = json_decode($output);
echo '{"place":"'.$select.'",';
echo '"w":"'.$data->query->results->matches->match->place->woeId.'",';
echo '"lat":"'.$data->query->results->matches->match->place->centroid->latitude.'",';
echo '"lon":"'.$data->query->results->matches->match->place->centroid->longitude.'"'."},n";
;
}

That was that – I had a data set I can work with.

Adding more information

The next thing I wanted to add was some more information about the area which meant using maps. As both Yahoo and Google maps have static map versions but are rate limited I wondered if there is a free version of that. And there is. Openstreetmap was the answer, especially the somewhat unofficial API I found with Google. To play safe, I wrote a script that gets the images and I cache it on my server to avoid killing this API.

I also wanted to show currently available houses in the area in case you are looking to buy. For this the natural choice for me was to use Nestoria as they also have an open YQL table (see the Nestoria table in the YQL console). So I used YQL and sorted the results by date:

select * from nestoria.search where place_name="Rugby" | sort(field='updated_in_days')

Using this I can get offers in the area live:

$url = 'http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20nestoria.search%20where%20place_name%3D%22'.urlencode($city).'%22%20|%20sort%28field%3D%27updated_in_days%27%29&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&diagnostics=false';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
$data = json_decode($output);
if($data->query->results){
$i=0;
$results = array_slice($data->query->results->listings,0,5);
if(sizeof($results)>0){
echo '<div class="listings"><h3>Current property listings (powered by <a href="http://www.nestoria.co.uk/">Nestoria</a>) <input type="button" class="hide" value="hide"></h3><ul>';
foreach($results as $r){
echo '<li><img src="'.($r->thumb_url).'" alt="">';
echo '<h4><a href="'.($r->lister_url).'">'.($r->title).'</a></h4>';
echo '<p> Price: '.($r->price_formatted).', Type of property: '.ucfirst($r->property_type).', Updated: '.($r->updated_in_days_formatted).' ('.($r->updated_in_days).' days)</p>';
echo '<p>Listed at: '.($r->datasource_name).' by '.($r->lister_name).'.</p>';
echo '</li>';
}
echo '</ul></div>';
}
}

Finding a charting solution

Adding interactive charts was the next step. I had a few issues with that:

  • While Google charts are full of win, they are rate-limited and I didn’t want to pull images. As the app was also meant to become a Yahoo application every image would have to be run through Caja for safety reasons which slowed it down.
  • Canvas and Flash solutions like YUI charts or Raphael were also not possible because of the performance of the YAP app.

So I wrote my own pure CSS bar charts to work around that issue.

Building the API

I put all these solutions together and built a small API that will give me the search results with three parameters: the location as an id and the start and end of the time range.

http://uk-house-prices.com/graphs.php?loc=1&start=10&end=20

Building the interface

To build the interface, I went all-out YUI. I took the YUI grids builder to create the main layout, the AutoComplete demo, the dual slider demo and the button and put them all together. Add an Ajax call to the form, and you are done. OK, I admit, there was quite a bit of cleaning up to be done :)

Notice that I am using progressive enhancement all the way. Without JavaScript you get dropdowns:

UK House Prices - without JavaScript by  you.

That’s it

The next thing I had to do is move the app over to the Yahoo Application Platform which was easy as I based it on an API - but this is another blog post :)

UK government browser guidance in dire need of upgrading

Monday, September 8th, 2008

One thing web developers who do not work in large corporations or with the public sector or education often forget is that there’s a lot of red-tape and checkbox ticking to be done before you even start a line of code. This get worse once there has been a decision made or a guideline in place, as replacing or upgrading those slips far down the list of need-to-do’s.

The web is a large and confusing place and the fact that you just cannot control or demand the setup your visitors use to come to your site and consume what is there can be frustrating. To me, it is what the web is about and I love the challenge of the unknown. Official sites, however, do not revel in unknowns and challenges and try to help webmasters to release quickly by cutting down on things to support.

Last friday, the UK government’s Central Office of Information (COI) published a public consultation on browser standards for public sector websites which misses the mark of good advice by quite a bit.

Bruce Lawson checked the guidelines in detail and responded to them on the WaSP blog

I agree with all that is said there, and humbly point the COI to the graded browser support my employer applies to steer the wild web into easier supportable channels.

There’s a comment form on the bottom of the page on the guidance site that gives you a chance to react to this. It might not mean much, but let’s not forget that if we can have an impact on the public service, it’ll mean a lot more web sites out there that do the right thing. These are the areas we should concentrate on – if your blog doesn’t render properly that is much less of an issue than you not being able to pay a parking ticket or sign up your kids for school.

UK Government initiative calls for hackers to mash-up public data

Friday, July 4th, 2008

It is pretty cool to see what is happening right now in the UK when it comes to mashups and data. Show us a better way is a web site and competition that asks ethical hackers to come up with ideas to use a wide range of public data for the good of the public. Straight from the horse’s mouth this sounds like this:

The UK Government wants to hear your ideas for new products that could improve the way public information is communicated. The Power of Information Taskforce is running a competition on the Government’s behalf, and we have a 20,000 pound prize fund to develop the best ideas to the next level. You can see the type of thing we are are looking for here

To show they are serious, the Government is making available gigabytes of new or previously invisible public information especially for people to use in this competition.  Rest assured, this competition does not include personal information about people.

We’re confident that you’ll have more and better ideas than we ever will. You don’t have to have any technical knowledge, nor any money, just a good idea, and 5 minutes spare to enter the competition.

There is a vast amount of APIs available to play with so what stops you from giving it a whirl? My own idea, cabsharing is something I was actually planning to do for quite a time, maybe even as a start-up, but why not here?