Having moved to Canada late in life I never really got into the hockey spirit. It might have been that hockey looked hard or it might have been a strategic campaign of hockey disinformation perpetrated by my parents in order to avoid driving to Tiny Town in the Middle of Nowhere, Alberta at 6am on a Saturday. If the latter then it is a brilliant plan I intend to duplicate with my children. Ideally they will play a sport which enables me to travel to warm location and sit on a beach while they do whatever they do. Competitive tiddlywinks, perhaps.

Despite this lack of interest in hockey I do find myself on the NHL website several times a month. Partially my visits are due to a well developed sense of schadenfreude around the pathetic Calgary Flames. Partially they are due to a love of statistics. I like watching the numbers of points move and the teams go up and down and calculating just how hopeless the Flames are.

I don’t think I’m the only one who likes sports statistics but the presentation of these statistics is really dry. A table of numbers is, generally, not very accessible to people. I thought I might try building a few simple visualizations to try to improve upon that. If you happen to be a representative of some professional sporting organization which pays their players millions of dollars and you want to steal these visualizations then they are ultra-mega-super-purple-ninja copyrighted(we can work out a massive license agreement). If you’re anybody else then they’re open source.

Today I’m going to look at a best of series. These are very common and are typically in the form of the first team to 3 wins wins the series. You know, unless you’re the NHL and stretch the season out into July by making your series best of 7. On nhl.com tonight there is a graphic showing how the teams are doing. We’re down to 4 teams so these are the semi-finals.

The key information for which people are looking on an image like this is:

1. which teams are playing

2. who is winning the series

3. the next game time

The graphic they use is pretty good at giving the teams playing. It is clear which teams are playing against each other and, if you’re fan enough to go to nhl.com, the logos are obvious. The next game in the series is also obvious. However the number of games won by each team in the series isn’t. My eye is drawn to the number under the team abbreviation. This is actually the standing of the team in the regular season. For the series on the left it is relatively apparent this isn’t the information for which I’m looking: there can be no one team with 5 wins. The series on right is less obvious. Boston could well be leading 4-1.

Instead of putting in “BOS LEADS 3 ““ 0”³ we could put in an easy visualization.

There are some improvements possible here too:

• use the team colours to fill the boxes
• use a gradient to show how each game gets closer to hitting the magic 4
• make it more obvious that they only have to get to half way
• change the shade of the fill depending on the score differential

You can check out my d3.js version of this over athttp://bl.ocks.org/stimms/5727304

In an ideal world every website would have a friendly JSON interface which would allow you to, at the very least, read data. We sure don’t live in that world. For a couple of the windows 8 and windows phone apps with which I’ve been playing I’ve needed to go out to a website and parse some content. I started off in python using a library called BeautifulSoup. I got that all working fine but I ran into a problem inserting the records I had retrieved into an SQL Azure database. As it turns out there are some silly inconsistencies with using FreeTDS on OSX which would have required me to recompile to get it to work against Azure SQL.

Nope.

There was a time when I was all over building my own kernel and tweeking things. I’m past that now and I just want stuff to work. God, just work (I have a whole rant about printers and printing not working but it descends into language which would melt your face so quickly that I dare not post it). So that went out the Window and I switched to C#. Actually, first I switched to using a JavaScript scheduled task in Azure mobile services. However there were limited node modules available and I didn’t feel like writing or including a whole HTTP library.

Right. I chose to use the HTML agility pack which is a pretty good parser for HTML. Frequently sites don’t have well formed HTML which makes parsing them with a full featured XML parser impossible. You don’t want to get involved in XML anyway - it is a gateway drug to proprietary file formats.

The agility pack has a built in HTTP client but I wasn’t too impressed with it as it just returned the document and not the status code. Without knowing if the page was a 404 my particular parsing job was difficult. Instead I made use of the HTTP client in System.Net.Http to pull the page. In this case I’m pulling down a page from the web comic XKCD.

All this stuff is asynchronous in .net 4.5 which is great. For all too long we’ve laboured in code bases which assume that network operations run at local speeds. I’m looking at you windows file system.

Parsing the returned document is also very simple

Now we have the document loaded into the agility pack we can perform simple XPath queries against it.

The first query there selects the first image in the first div with an id attribute with value comic. Using a double // means that I’m not concerned about how deep in the tree the node appears. The second query just grabs the first div with an id of ctitle. You can also use a SelectNodes function to select multiple matching nodes.

I’m not crazy about using xpath for this sort of thing. It would be great to have a CSS selector based library in C#. As it turns out there is a project called Fizzler. I guess it is still in beta, but the few things I read about it suggest it works quite well. I’ll have to play with it for another post. It certainly would be nice to only have to know one HTML query language.

I’ve been playing around a bit with angularJS and Windows 8 JavaScript. For the most part it has been a pleasant sort of endeavour. I did run into one thing for which I had to find a solution and I thought I would post it here.

The JavaScript context of WinJS applications is slightly more restrictive in some ways than IE10 JavaScript on which it is based. One of the limits is around adding content to a page. Because the Windows 8 apps have more power over the local file system than the sandboxed IE10 implementation there are some increased risks to allowing arbitrary content to be added to the page.

The functions

• innerHTML
• outerHTML
• document.write

Are all illegal functions. Instead one should be making use of

• innerText
• document.createElement
• setAttribute

There are two posts on the topic

• http://onehungrymind.com/windows-8-and-angularjs/
• http://blog.jonathanchannon.com/2013/01/24/using-angularjsbackbonejs-in-windows-8-javascript-app/

However both of them are out of date as they reference custom builds of jQuery. As of jQuery 2.0 you don’t need to use these tricks to get jQuery to work. However I did find that the latest angular still had an issue. In the final line of angular some styling information is injected onto the page.

angular.element(document).find(‘head’).append(‘‘);

As you can see this just uses an append. Now append typically just calls appendChild which is permitted by WinJS. I still found it to be an issue, perhaps because we’re appending a child with properties instead of a child with properties set with setAttribute. I solved it by using the execUnsafeLocalFunction method in WinJS. This overrides the security restrictions for the function passed in as an argument. I was confident the angular code was not going do harm so I felt okay about using this. In general I would take the time to rewrite the code to be more secure.

MSApp.execUnsafeLocalFunction(function () { angular.element(document).find(‘head’).append(‘‘); }); You just need to make sure to include angular after the Microsoft JavaScript libraries.

I took a trip down to the YYCJS meetup group the other night. This is one of those rambling, incoherent posts which serves as the notes I took while there.

I hadn’t been before so that was pretty exciting. I felt like a bit of a fraud because

a) I only pretend to know JavaScript

b) I work for The Man

Maybe I’m assuming too much about the thick framed fellows I met at YYCJS but it seemed like a good place to pretend I knew about made up bands and worked for a small socially aware company instead of a huge people-hatingconglomerate. Everybody had a macbook and an android phone. I half fit in.

I guess this was the second half of a series of NodeJS talks ““ I missed the first half so I could be in big trouble.

• There is a constant debate between native and web applications
• Web is still much slower but is catching up at an alarming rate. (Good! -ed.)
• There is apparently an LLVM to JavaScript compiler which means that you can write your JavaScript in pretty much any language.
• There was a demo of an amazing idea to use a smart phone as a front end to remote control a web browser using web sockets. You associated the device and the browser by having a QR code shown on the browser which contains a url. That associates the phone with the browser.QRControl
• Google RollIt extends the idea into a full game using the accelerometer interface for chrome. I didn’t even know that there were hooks for accelerometers in chrome.
• I should hang out here more, Jesus my job is boring.
• I’m so appreciative that the people demoing make use of the command line.
• Using the same QR code linking with websockets Oliver showed using a device to draw on a canvas from the phone. Some great applications for that.
• Oh god, they’re talking about nodejs ““ node your head knowingly. They’ll beat you to death with designer beer bottles if you let them know you’re hosting on IIS
• Oliver has created a framework with another guy called Oliver. The framework is called Olives.
• Olives is a DOM agnostic MVVM framework. That means that it can be used in Node which has no DOM.
• Oliver isn’t use to a QWERTY keyboard. He is pretty cool so I bet he has his own keyboard mapping. Hardcore
• It seems like require.js is pretty much a given in the JS community, I really need to start pulling that in to my JavaScript stuff. Require is a dependency resolving library which live loads libraries as they become needed.
• Socket.io also seems to be pretty much a given. Looking at the socket.io site it looks super easy, far easier than SignalR but I don’t know if it has a good scalability story.
• Live binding in MVVM frameworks rerender only segments of the page instead of the whole thing. That makes it much faster. So I guess the morale is keep your templates small and bind at the lowest possible level.
• Apparently AngularJS is slow. Oh. Damn, I was going to make use of that framework.
• I have to say live binding is pretty impressive. I guess it is just implemented as an object wrapper which keeps track of what it binds and updates it.
• Olives support live binding but it is provided through a plugin so you don’t need to use it or you could swap out the implementation.
• The binding for Olives is in the HTML through data-* attributes. Acutally it makes use of data-model and then you pass descriptive strings into that. I’m not crazy about that, seems like tooling around richer data attributes would be easier.
• Olives has various other plugins including a nifty socketIO transport one which can live update from changes in a database such as Couch
• JavaScript can be used at every level, on the browser, on the server with NodeJS and in the database with CouchDB
• The demo for CouchDB is very interesting. It looks like what happens is that the NodeJS server subscribes to a document collection on the CouchDB server. When the document collection changes CouchDB publishes a change notification and sends the entire collection back down to the NodeJS server. This then passes that updated collection onto the browser.
• There are some serious scalability concerns in my mind there

Authentication

• Express seems to be THE framework to put on top of NodeJS
• There are some authentication libraries for Node. Passport.js seems to be a very serious contender
• Defining modules in Node is easy. You basically just put the functionality inside a file and require it. Inside that file it seems sensible to add your functions to an object and then set module.export to that object.
• By defining an index.js and shoving it in your module’s root it will automatically be loaded by just requiring the directory. Inside that file you can require additional things which will bring in all the rest of your files.

I had the need to emit an HTML attribute today which contained a hyphen. Aside: a hyphen is the thing which joins two words while a dash is the thing which is used to denote a sentence interruption

Dash

The zombies are coming “” I’m not afraid.

Hyphen

Zombies are flesh-eaters.

Both

The flesh-eating zombies are coming “” I’m not afraid.

There are two different dashes too, but we’re already pretty far off track here “” let’s reel it in.

There are actually an infinite number of valid HTML5 attributes which contain hyphens. You can write data-[anything] to add data to an arbitrary data to your elements. I find it to be very useful to have this standard for data attributes. If you’re using AngularJS then you’ll probably be making use of a lot of ng-* attributes. I came across some trouble when I was trying to emit this data from the HTML helper methods in ASP.net MVC. You can’t just do

@Html.TextAreaFor(x=>x.Name, new { data-age=”fifteen”})

Because the hyphen is interpreted as a minus. Solving this is actually very easy. Razor will automatically replace underscores with hyphens so

@Html.TextAreaFor(x=>x.Name, new { data_age=”fifteen”}) gets you what you want.

As I posted a few days ago I put my first app into the approval process for Windows 8. I actually put it into the approval process on the 21st of May, 9 days ago. It has been rejected 3 times and I’m still waiting for it to be approved. Honestly if it doesn’t go this time I’m going to give up and move onto other things. It isn’t worth my effort. I feel that most of the blame for this lies with the approval process. This is a rant about what’s wrong with the process.

REJECTED

My first complaint was that there sure were a lot of fields to be filled in when setting up the app. Somewhere in that mess of fields I forgot to fill in a privacy policy. I guess privacy policies are needed for any app which communicates over the network. Fine. The thing is you need to put it in two places, one in the app and one in the approval metadata. I had forgotten the one in the approval metadata. Okay, okay I added that and tried again.

REJECTED

WTF, I got that privacy policy in there. Oh there is a password requirementburiedin the app. I used a template for my first app and didn’t realize that there was a password requirement. Mea culpa. I could have supplied a test account but I just disabled that functionality. At 6am when I’m trying to get out to work I didn’t care. I’ll add them back in version 1.1.

REJECTED

Oh come on, now what? Oh it is in the wrong category. What? It is a blog and I put it in books. A blog is sort of like a book. Well I’m sure they gave me guidance as to which category it should be in. Oh, nope they didn’t bother with that. Jerks. I suppose I’ll guess again since there is zero guidance out there on which category to put things in.

What is pissing me off is that each one of these rejections costs me 3 days of waiting. If they had bothered to test the entire application the first time and written down all the problems then we wouldn’t be in this rejection cycle.There is a concept in computing called “fail fast” the theory being that if you fail quickly instead of trying to compensate you can get problems solved while they are still small. This isn’t a place where that theory should be applied. The long approval cycle time removes allbenefitfrom failing fast. TELL ME EVERYTHING UP FRONT.

That the app was in the wrong category should have been instantly apparent to whatever people they have approving apps. That could have been easily fixed if it had been mentioned in the first rejection, same with the login.

While I’m bitching let me say that I’m sick of signing into the store. Every fricking action needs me to sign in. Run the app through the validation process? Sign in. Create a new app? Sign in. Check on the progress of your app? Sign in. Go to the store? Sign in. Can we maybe come up with a shared key store or at least remember the god damn password within Visual Studio? After developing an app I had to throw out my keyboard as a security risk “” all the keys used for my password were worn down.

What can be done to fix it

1. Test the app from end to end. Every. Time. You’re putting a bad taste in people’s mouth by failing them again and again. Just fail them once and help them out, Jesus.

2. Be more helpful when you fail the app. I’m just guessing that my app is in the correct category now. Who knows. If the tester had told me which category it should be in I wouldn’t have had to guess. It would have been a dozen key strokes, help a brother out.

3. Tighten up the release cycle times. 3 days is too long. This isn’t 1995 when I had to ship a DVD to some magic DVD mastering center to be put into a package and mailed out to my customers.Sorry I live in a world of instant gratification. I’m sure somebody will comment and tell me how much better the windows store process is than Apple’s. I don’t care, that’s not the point. Apple should fix their stuff too.

4. The sign in stuff. Please, it is a minor annoyance but it sure as hell stacks up on top of the other stuff with which I’m dealing.

5. During the automated testing of the app it should be apparent that the app is talking to a network. You could flag the lack of privacy policy there. It is an easy fix and would have saved me a whole rejection.

6. If you’re using approvals/day as metric for the testers stop it. Crossing well into the world of supposition here I bet that some middle manager at Microsoft has looked at the approval process and said “we need to instrument this”. The obvious metric is how many apps a tester looks at in a day. By instrumenting this they are actually encouraging testers to quickly fail things and not test themthoroughly. That isn’t thebehavioryou want. Instrument something else.

This is stuff which needs to be done now. Windows 8 is shaping up to be a bigger disaster than Vista and improving theexperiencefor developers should be high on the list of things to do. I don’t have to put up with this bullshit when I deploy my app through apt-get or chocolatey nuget. The advantage of having my app in the windows store is pretty small from what I can see. I would rather distribute a classic app through traditional channels than deal with the windows store again.

Today somebody (it might have been an xkcd comic) pointed me to the addictive game GeoGuessr. I didn’t link it because if you click on it you’ll never come back to finish reading this blog post. The idea of the game is that they drop you into street view somewhere on the planet and you have to guess where you are. The closer you are to being right the more points you get. It is a very simple concept for a game but at the same time real good fun. I was playing it when somebody came into my office and asked about it. We spent 5 minutesdissectinga series of photos using logic and best guesses to figure out where we were. The whole exercise got me thinking about how I’m going to use this game as an interview tool the next time I interview somebody.

What?

Stay with me on this one. One of the more popular interview techniques for tech companies is to present a puzzle and have people solve it. A lot of people look down on puzzle interviews because it is such a gotcha style of interview. If you’ve seen the puzzle before or you guess the trick then you can get it quickly and you win the interview. I don’t mind puzzle interview questions. It isn’t because I think being able to figure out the puzzle is important it is because it gives me some insight into how well the interviewee will approach a problem. We’ve all worked with that developer who gives up in the face of a difficult problem. Throwing ideas against the wall and seeing what sticks is a hallmark of good programming problem solving. I don’t expect people to have the answers to difficult problems right away but I do expect them to puzzle out the problem.

I think that playing a round or two of GeoGuessr(I linked it there because you got this far and deserve a reward) during the interview would allow for a quick evaluation of how the programmer thinks. Just have them puzzle out loud or play with their potential team mates. In the brief time I played with somebody we said things like

“Well look, they’re driving on the left side of the road so that eliminates a lot of places. The language on the signs seems to be English. So UK, South Africa or Australia.Seems prettyarid, perhaps Australia?”

and

“Right, there is a truck. A dodge truck so North America is pretty much for sure. There is a highway marker for an in-state highway 45. So we know US for sure. I bet most states have a highway 45, it is a good number 45. No mountains so that cuts out a few places. There is a train, looks like it has grain cars ““ a lot of grain cars. Bread basket somewhere? Hey, hey look! A sign for interstate 69. Google it! It goes North/South through Arkansas and Kentucky. Let’s try Kentucky.”

This demonstrates that we’re working though a difficult problem and that we’re doing it as a team. Also this sort of thing is a great ice breaker during an interview. If you try this out before me drop me a line and let me know how it works.

I have been putting off building a Windows 8 app for ages. I actually don’t have a a computer in the house which runs Windows 8 on bare metal but I do have a couple of VMs. Well the impending end of the developer movementfinally got me going(it ends on the 15th of June, BTW). I had read something a few months back about an open source Windows 8 app which republished your WordPress blog as an app. Perfect!

Ideapressis the project and they have a pretty simple HTML/JavaScript app which pulls down your posts and republishes them. On their site is even a wizard for creating the app. I ran through the wizard and gave it various images and an API key for wordpress.com. At the end of the process(which finished neither late, nor early butpreciselywhen it meant to) I had a zip file with a VS2012 project in it.

I threw it on a Windows 8 box (as you can only develop for Windows 8 on Windows 8) and hit F5 to run it. I ended up with a pretty darn good app. I was just about to call it good enough and publish it when I realized that none of the source code on my blog was showing up.

I use embedded gists to show source code an provide syntax highlighting. WordPress has built in support for gists. It isn’t actually difficult to do as if you append .js to the end of a gist you can get a javascript version of the gist which injects styling and a div onto your page. Pretty smart. This wasn’t working on the app for some reason, likely security related. The security model for injecting content into the page in Windows 8 is more restrictive than on a browser.

In order to fix that I pulled in a handy library called gist-embedded. This library made us of jQuery which is where I ran into my first problem. I guess that before jQuery 2.0 there were somecompatibilityissues between jQuery and the Windows 8 runtime. These have been resolved in jQuery 2.0. However, my application failed validation due to the jQuery file which nuget added. I was confused because I failed in two sections. The first was that the code didn’t run and the second was that the encoding was wrong. The encoding was easy enough to fix. You just resave the jquery file with UTF-8 encoding.

But I was totally flumoxed by thecompilationerrors. I could not manage to duplicate them in the app or in IE10. Eventually I ran the validation again and found everything now worked. I guess the incorrect encoding was causing more than one issue.

Now that I had the application working the way I wanted and passing the validation I signed up for the windows store. I had to give Microsoft $49 of my hard earned dollars for the privilege of giving away a freeapplication. It is so funny that companies likeMicrosoftand Apple have tricked us into paying for the right to release software on their platform. If he wasn’t a mildly insane toe nail eater I would beinterestedto hear what RMS would say about it.

Weirdly Microsoft validate your credit card by charging a small amount then refund it and use it as a method of validating your account. They require that you enter the amount charged to validate your account. This means that if, like me, your bank sucks at posting charges with any rapidity you’ll be stuck waiting.

When setting up the application you have to enter a stunning amount of meta-data about where you want your app advertised, which age groups, what category, whether it contains cryptography”¦ it wasoverwhelming and at least half the stuff could have been replaced by setting some reasonable defaults.

At last I got to the point where I could submit my app for publishing.Yay! Now I can sit here and wait. And wait.

The phase which was suppose to take an hour took a day. I’m now into hour 20 of the 3 hour security testing. I’m obviously more excited about getting this app out than I should be. No amount of refreshing seems to speed the process along. It is superunfortunatethat this process is so time consuming.

I can’t comment on the rest of the process because I’m still waiting.

Overall I think that building and distributingWindows 8 apps is more painful than it needs to be. I suppose you do get access to a global distribution channel and a place in the Windows store for your $49 but if uptake on Metro apps has been as bad as reportedthen cleaning up app submission should be high on a priority list at Microsoft.

The submission process needs to get faster. I know that the process is far better than Apple’s but it still sucks. I’m use to hitting publish in my CI system and 5 minutes later everything is up and running on my site. I don’t expect updates to be pushed out to clients with that rapidity but I do expect the app to at least be published in less than a day. App stores are the antithesisof the continuous deployment model.

Actually programming an app using JavaScript is great. I really like the JavaScript and HTML model so, so much more than dealing with XAML and C#. I’ll write more about then when I talk about writing my first Windows Phone 8 app. The security model take some getting use to and I haven’t tried a framework like AngularJS out yet so I can’t comment much on that.

Is it worth making Windows 8 specific apps? That’s a firm maybe. I don’t think it should be the default for any company but if there is a customer base asking for it then you might as well. It is no more difficult than building a normal application.

Choosing keys in a database can be a pretty difficult process. In university it was very simple, we came up with superkeys and compound keys and made sure things were in BCNF. In the real world that doesn’t work. Dealing with normalized databases is a huge pain. The other thing which is painful is dealing with compound keys where a record is referenced by two or more fields. I always recommend adding a unique identifier for each row of data. That key can be either an integer or a GUID, I know of arguments both ways and that isn’t what this post is about.

Iwas recently sitting in on the evaluation of a piece of software. It was the standard CRUD sort of application with a few additional features. However one of the things which was permitted was allowing users to set the value of the primary key field. As soon as the presenter mentioned it hand shot up all over the room.

“What if we have the same object ID for multiple things?” “Err, well that would never happen.” “Not true, we have 8 examples of it right here off the top of our head” “Err, well I guess we don’t support that situation”

We talked it around for half an hour and the consensus was that this was a serious design flaw in the application. The developers had assumed that a certain piece of user generated data was certain to be unique but, as it turns out, it wasn’t. Things got worse the further we pushed into the problem. There were other restrictions which further limited users’ ability to enter the data they wanted in the primary key field.Unfortunately,the work needed to change the constraints was considerable as the primary key was, as primary keys are, heavily used throughout the application.

As a general rule I never allow users to enter key information because they may make mistakes. Mistakes are made all over the place by users but these tend to have more direconsequencesas they cross from user data over into system data. Think about what would happen if the user wants to change the value they entered in that field. You now need to crawl through countless other tables and data structures looking for places where that key was used and change it.

I consider there to be a division in the application between data used by the system and data used by users. You wouldn’t want users seeing your application’s routing tables because it isn’t an abstraction about which they should have to care. Equally the IDs of records should not be visible to users. Just as you keep this data away from users you should keep the user data away from the application. I can’t count the number of problems I’ve seen which, when you drill down, turn out to be a result of the system relying on user data which becomes inconsistent.

Note: I’m totally guessing at how a lot of this stuff works so maybe don’t use it as the basis of building a space ship or anything important.

Here in Canada we get Netflix but it isn’t the same as the US Netflix. If it wasn’t for the fact that Canadian Netflix has Startgate SG-1 and US doesn’t I would be much angrier about the whole thing. None the less there are times when I want to watch something which isn’t available in Canada. I was always aware that there were services and ways of watching US Netflix in Canada but I hadn’t really bothered looking into until a friend of mine mentioned Blockless. Their service is really simple, you just need to change the DNS settings on your router and away you go. This was way easier than the SSH backed VPN tunnel to an Amazon virtual machine I had envisioned.

Well that got me thinking, how the heck does this work? Well if I was going to set up a geographically specific service I would check theincomingIP addresses of connections and then block or allow them as I saw fit or serve out different content. This doesn’t, however, seem to be the way that Netflix works. Instead they seem to use a technology called GeoDNS. This is the same technology which is behind CDNs which deliver content from the nearest node.

When you ask the DNS server to give you an IP address for a name it usually just blindly hands you back an address. With a GeoDNS service the server will first look up your location and then return the IP address of the node which is closest. Netflix uses this to ensure that if you’re in Canada you get Canadian content.

When you change your DNS server with Blockless or Unblock Us they intercept the DNS request and substitue the US location in for the Canadian IP address. With that you now get US Netflix. The same thing can be done to simulate being in any country you want.

This is a really great solution to the pain of geolocking, but it only works if the service you’re talking to take a naive approach to their filtering. IF Netflix also checked the incomming IP address when they were serving the content then you would need to go with a fully fledged VPN. With a VPN the actual end point accessing Netflix would be in the US and there would be no real way for Netflix to filter it.