I talk a lot about alternatives to writing JavaScript(typescript, coffeescript, dart,”¦) and that JavaScript is really just an assembly language for the web. That got me thinking about SQL and whether we should be considering it an assembly language for databases.

SQL has a lot of problems which make it more difficult to use. Just look at the list of key words for it.aspx): the list is huge and the list of future reserved words bring the list to crazy levels. It would be nicer if the keywords were moved from being keywords to being library functions. The syntax is also not conducive to providing autocompletes. Consider the simple SQL

select Id, Name, Number from tblThingies

The autocomplete domain is the set of columns in tblThingies, because this is naturally typed from left to right (at least in English, how does this work in Arabic?) the domain isn’t defined until after the terms Id, Name and Number are written. This is not at all conducive to autocomplete.

The concepts in SQL are sound. Largely functions in SQL are set operations and if you treat the language as a set manipulation language instead of a procedural language then it is really good at it. Modern languages have a different approach to syntax and formatting from which SQL couldbenefit.

There are actually a number of languages which do compile to SQL. LINQ is one example and the HQL language from Hibernate and NHibernate is another. I think HQL gives a good example of what a more modern query language looks like. From the HQL community documentation I took this example:

You can see the select ordering still doesn’t provide us autocomplete capabilities and the key words are still very SQLly. I don’t think that HQL goes far enough in fixing the problems of SQL LINQ does a better job. That same query in LINQ would look something like

(although I haven’t tested this).

You can see the select is moved to the end and at the same time the select is transfigured into a more object oriented syntax. I like the idea of returning objects by default. It makes the language more powerful and it reduces the friction in talking to OO languages which are going to be the majority of languages now.

I don’t think we’ve explored the idea of SQL as an assembly language enough yet. LINQ is available through LINQ pad but I’m not aware of any standalone system which allows for HQL queries to be run against a server. I like the idea of compiling languages to SQL. The syntax is going to take some work but it would be great to see something adopted by the major vendors.

Some years back Google decided to build a browser. As I understand it Chrome cam about not as a 20% project like so many other google products but as an effort by the foundersSergey Brin and Larry Page. It was a brilliant move which has changed the face not just of browsers but of how we use computers in general.

Is it that Chrome ushered in the era of fast JavaScript? Is it that Chrome wasn’t afraid to innovate by including new features? Is it that Chrome introduced process per tab to eliminate the crashing hanging monstrosity that was Macromedia Flash? No the big innovation in Chrome was that it updated itself.

I know what you’re saying: lots of applications update themselves. Well that’s true. I remember building an automatic update system for a desktop application I worked on back in 2005. It was a great system but it had a flaw which I didn’t even realize until Chrome solved the problem. Myapplicationasked users if they wanted to update before the update ran. The update was not silent. Chrome just updates in the background. This results in adoption curves which look like this:

Within a few weeks almost everybody is using the latest and greatest. For years we’be dealing with the legacy of slow adoption of internet explorer versions. That should not be anywhere near as bad a problem with Chrome.

While it is impressive how easily Chome is updated what is moreimpressiveto me is that there are so few problems with this continual upgrade cycle. As was pointed out to me in a meeting today IT departments tend to be very resistant to this sort of update model. They are terrified that new versions of software are going to break things for their users. This has proven to be false. It was always amusing to me that IT departments would claim that they need to do some sort of comprehensive testing on all their software when a new version of a browser came out. There is so much to test I really don’t know how you would ever do it.

Instead it is easier to just be spry and update your apps to account for new versions of browsers. The continual updates are likely to be small and not break much. If the entire internet can handle chrome updates it seems like a single IT department could do it too.

Chrome’s big revolution has been to prove the viability of continual updates.

I hear that the way github as a company works is that if you work there and think that github needs a feature you just build it. One of the features I’ve just recently discovered are github pages. Pages allow projects and people to put up simple static pages to communicate about the project.

I happen to do some development on a little project called Angela Smithand it needs a website. At the moment all the documentation is scatted over a series of blog posts by the various contributors. It would be great to bring that all together.

It was pretty simple to set up the most basic of pages. I checked out the latest version of AngelaSmith and created a new branch.

git branch gh-pages git checkout gh-pages

The gh-pages branch is a special branch which is monitored by a build process at github. When it detects a new checkin it will build the site and deploy it to .github.io/. In my case that meanshttp://stimms.github.io/AngelaSmith/.

That’s not a very friendly name so it is possible to set up a custom url to point at that.

You can also use the Jekyll static page generator to build pages from templates and using includes. There is a bunch of documentation out there on how to use Jekyll. I have to say that while the documentation looks good it isn’t super informative. It took me quite some time to figure out that the character encoding was very important. I ended up backing my content down to code page 437 and not UTF-8.

Once I get Jekyll figured out I’ll put up more posts on it.

If you read the comments on yesterday’s post there was a bit of a discussion about some alternatives to the number of boxes in the best of visualization I created. When I built that visualization I based it on the number of games + 1. This would prevent a situation where a full series was required and the final score was ambiguous

By using colours it is possible to make this look a bit better. This also offers us an opportunity to generalize some of the code written yesterday.

The most obvious improvement is around the function shouldBeFilled

First thing is that the function name is lying. It isn’t telling if the box should be filled or not. It is setting the fill color. So let’s rename it. Next up the use of hard coded values for the colours bugs me. If we’re going to be using a different colour for each team then we have to change this.

Here I’ve gone ahead and assumed that I have an options object in my class. I don’t so one will need to be added. I’d like to maintain backwards compatibility so I’ll set up some default values for options.

Nobody calling my needs to change their signature but they can still override the options if they wish. $.extend is a jQuery helper which merges the properties on two objects. One could even say that it composes a new object.

To set the options a caller can just do

There was also talk of adding some emphasis to the center block. I do have some ideas for that which we’ll explore in a future post. You can see the results of today’s post athttp://bl.ocks.org/stimms/5734160

Having moved to Canada late in life I never really got into the hockey spirit. It might have been that hockey looked hard or it might have been a strategic campaign of hockey disinformation perpetrated by my parents in order to avoid driving to Tiny Town in the Middle of Nowhere, Alberta at 6am on a Saturday. If the latter then it is a brilliant plan I intend to duplicate with my children. Ideally they will play a sport which enables me to travel to warm location and sit on a beach while they do whatever they do. Competitive tiddlywinks, perhaps.

Despite this lack of interest in hockey I do find myself on the NHL website several times a month. Partially my visits are due to a well developed sense of schadenfreude around the pathetic Calgary Flames. Partially they are due to a love of statistics. I like watching the numbers of points move and the teams go up and down and calculating just how hopeless the Flames are.

I don’t think I’m the only one who likes sports statistics but the presentation of these statistics is really dry. A table of numbers is, generally, not very accessible to people. I thought I might try building a few simple visualizations to try to improve upon that. If you happen to be a representative of some professional sporting organization which pays their players millions of dollars and you want to steal these visualizations then they are ultra-mega-super-purple-ninja copyrighted(we can work out a massive license agreement). If you’re anybody else then they’re open source.

Today I’m going to look at a best of series. These are very common and are typically in the form of the first team to 3 wins wins the series. You know, unless you’re the NHL and stretch the season out into July by making your series best of 7. On nhl.com tonight there is a graphic showing how the teams are doing. We’re down to 4 teams so these are the semi-finals.

The key information for which people are looking on an image like this is:

1. which teams are playing

2. who is winning the series

3. the next game time

The graphic they use is pretty good at giving the teams playing. It is clear which teams are playing against each other and, if you’re fan enough to go to nhl.com, the logos are obvious. The next game in the series is also obvious. However the number of games won by each team in the series isn’t. My eye is drawn to the number under the team abbreviation. This is actually the standing of the team in the regular season. For the series on the left it is relatively apparent this isn’t the information for which I’m looking: there can be no one team with 5 wins. The series on right is less obvious. Boston could well be leading 4-1.

Instead of putting in “BOS LEADS 3 ““ 0”³ we could put in an easy visualization.

There are some improvements possible here too:

• use the team colours to fill the boxes
• use a gradient to show how each game gets closer to hitting the magic 4
• make it more obvious that they only have to get to half way
• change the shade of the fill depending on the score differential

You can check out my d3.js version of this over athttp://bl.ocks.org/stimms/5727304

In an ideal world every website would have a friendly JSON interface which would allow you to, at the very least, read data. We sure don’t live in that world. For a couple of the windows 8 and windows phone apps with which I’ve been playing I’ve needed to go out to a website and parse some content. I started off in python using a library called BeautifulSoup. I got that all working fine but I ran into a problem inserting the records I had retrieved into an SQL Azure database. As it turns out there are some silly inconsistencies with using FreeTDS on OSX which would have required me to recompile to get it to work against Azure SQL.

Nope.

There was a time when I was all over building my own kernel and tweeking things. I’m past that now and I just want stuff to work. God, just work (I have a whole rant about printers and printing not working but it descends into language which would melt your face so quickly that I dare not post it). So that went out the Window and I switched to C#. Actually, first I switched to using a JavaScript scheduled task in Azure mobile services. However there were limited node modules available and I didn’t feel like writing or including a whole HTTP library.

Right. I chose to use the HTML agility pack which is a pretty good parser for HTML. Frequently sites don’t have well formed HTML which makes parsing them with a full featured XML parser impossible. You don’t want to get involved in XML anyway - it is a gateway drug to proprietary file formats.

The agility pack has a built in HTTP client but I wasn’t too impressed with it as it just returned the document and not the status code. Without knowing if the page was a 404 my particular parsing job was difficult. Instead I made use of the HTTP client in System.Net.Http to pull the page. In this case I’m pulling down a page from the web comic XKCD.

All this stuff is asynchronous in .net 4.5 which is great. For all too long we’ve laboured in code bases which assume that network operations run at local speeds. I’m looking at you windows file system.

Parsing the returned document is also very simple

Now we have the document loaded into the agility pack we can perform simple XPath queries against it.

The first query there selects the first image in the first div with an id attribute with value comic. Using a double // means that I’m not concerned about how deep in the tree the node appears. The second query just grabs the first div with an id of ctitle. You can also use a SelectNodes function to select multiple matching nodes.

I’m not crazy about using xpath for this sort of thing. It would be great to have a CSS selector based library in C#. As it turns out there is a project called Fizzler. I guess it is still in beta, but the few things I read about it suggest it works quite well. I’ll have to play with it for another post. It certainly would be nice to only have to know one HTML query language.

I’ve been playing around a bit with angularJS and Windows 8 JavaScript. For the most part it has been a pleasant sort of endeavour. I did run into one thing for which I had to find a solution and I thought I would post it here.

The JavaScript context of WinJS applications is slightly more restrictive in some ways than IE10 JavaScript on which it is based. One of the limits is around adding content to a page. Because the Windows 8 apps have more power over the local file system than the sandboxed IE10 implementation there are some increased risks to allowing arbitrary content to be added to the page.

The functions

• innerHTML
• outerHTML
• document.write

Are all illegal functions. Instead one should be making use of

• innerText
• document.createElement
• setAttribute

There are two posts on the topic

• http://onehungrymind.com/windows-8-and-angularjs/
• http://blog.jonathanchannon.com/2013/01/24/using-angularjsbackbonejs-in-windows-8-javascript-app/

However both of them are out of date as they reference custom builds of jQuery. As of jQuery 2.0 you don’t need to use these tricks to get jQuery to work. However I did find that the latest angular still had an issue. In the final line of angular some styling information is injected onto the page.

angular.element(document).find(‘head’).append(‘‘);

As you can see this just uses an append. Now append typically just calls appendChild which is permitted by WinJS. I still found it to be an issue, perhaps because we’re appending a child with properties instead of a child with properties set with setAttribute. I solved it by using the execUnsafeLocalFunction method in WinJS. This overrides the security restrictions for the function passed in as an argument. I was confident the angular code was not going do harm so I felt okay about using this. In general I would take the time to rewrite the code to be more secure.

MSApp.execUnsafeLocalFunction(function () { angular.element(document).find(‘head’).append(‘‘); }); You just need to make sure to include angular after the Microsoft JavaScript libraries.

I took a trip down to the YYCJS meetup group the other night. This is one of those rambling, incoherent posts which serves as the notes I took while there.

I hadn’t been before so that was pretty exciting. I felt like a bit of a fraud because

a) I only pretend to know JavaScript

b) I work for The Man

Maybe I’m assuming too much about the thick framed fellows I met at YYCJS but it seemed like a good place to pretend I knew about made up bands and worked for a small socially aware company instead of a huge people-hatingconglomerate. Everybody had a macbook and an android phone. I half fit in.

I guess this was the second half of a series of NodeJS talks ““ I missed the first half so I could be in big trouble.

• There is a constant debate between native and web applications
• Web is still much slower but is catching up at an alarming rate. (Good! -ed.)
• There is apparently an LLVM to JavaScript compiler which means that you can write your JavaScript in pretty much any language.
• There was a demo of an amazing idea to use a smart phone as a front end to remote control a web browser using web sockets. You associated the device and the browser by having a QR code shown on the browser which contains a url. That associates the phone with the browser.QRControl
• Google RollIt extends the idea into a full game using the accelerometer interface for chrome. I didn’t even know that there were hooks for accelerometers in chrome.
• I should hang out here more, Jesus my job is boring.
• I’m so appreciative that the people demoing make use of the command line.
• Using the same QR code linking with websockets Oliver showed using a device to draw on a canvas from the phone. Some great applications for that.
• Oh god, they’re talking about nodejs ““ node your head knowingly. They’ll beat you to death with designer beer bottles if you let them know you’re hosting on IIS
• Oliver has created a framework with another guy called Oliver. The framework is called Olives.
• Olives is a DOM agnostic MVVM framework. That means that it can be used in Node which has no DOM.
• Oliver isn’t use to a QWERTY keyboard. He is pretty cool so I bet he has his own keyboard mapping. Hardcore
• It seems like require.js is pretty much a given in the JS community, I really need to start pulling that in to my JavaScript stuff. Require is a dependency resolving library which live loads libraries as they become needed.
• Socket.io also seems to be pretty much a given. Looking at the socket.io site it looks super easy, far easier than SignalR but I don’t know if it has a good scalability story.
• Live binding in MVVM frameworks rerender only segments of the page instead of the whole thing. That makes it much faster. So I guess the morale is keep your templates small and bind at the lowest possible level.
• Apparently AngularJS is slow. Oh. Damn, I was going to make use of that framework.
• I have to say live binding is pretty impressive. I guess it is just implemented as an object wrapper which keeps track of what it binds and updates it.
• Olives support live binding but it is provided through a plugin so you don’t need to use it or you could swap out the implementation.
• The binding for Olives is in the HTML through data-* attributes. Acutally it makes use of data-model and then you pass descriptive strings into that. I’m not crazy about that, seems like tooling around richer data attributes would be easier.
• Olives has various other plugins including a nifty socketIO transport one which can live update from changes in a database such as Couch
• JavaScript can be used at every level, on the browser, on the server with NodeJS and in the database with CouchDB
• The demo for CouchDB is very interesting. It looks like what happens is that the NodeJS server subscribes to a document collection on the CouchDB server. When the document collection changes CouchDB publishes a change notification and sends the entire collection back down to the NodeJS server. This then passes that updated collection onto the browser.
• There are some serious scalability concerns in my mind there

Authentication

• Express seems to be THE framework to put on top of NodeJS
• There are some authentication libraries for Node. Passport.js seems to be a very serious contender
• Defining modules in Node is easy. You basically just put the functionality inside a file and require it. Inside that file it seems sensible to add your functions to an object and then set module.export to that object.
• By defining an index.js and shoving it in your module’s root it will automatically be loaded by just requiring the directory. Inside that file you can require additional things which will bring in all the rest of your files.

I had the need to emit an HTML attribute today which contained a hyphen. Aside: a hyphen is the thing which joins two words while a dash is the thing which is used to denote a sentence interruption

Dash

The zombies are coming “” I’m not afraid.

Hyphen

Zombies are flesh-eaters.

Both

The flesh-eating zombies are coming “” I’m not afraid.

There are two different dashes too, but we’re already pretty far off track here “” let’s reel it in.

There are actually an infinite number of valid HTML5 attributes which contain hyphens. You can write data-[anything] to add data to an arbitrary data to your elements. I find it to be very useful to have this standard for data attributes. If you’re using AngularJS then you’ll probably be making use of a lot of ng-* attributes. I came across some trouble when I was trying to emit this data from the HTML helper methods in ASP.net MVC. You can’t just do

@Html.TextAreaFor(x=>x.Name, new { data-age=”fifteen”})

Because the hyphen is interpreted as a minus. Solving this is actually very easy. Razor will automatically replace underscores with hyphens so

@Html.TextAreaFor(x=>x.Name, new { data_age=”fifteen”}) gets you what you want.

As I posted a few days ago I put my first app into the approval process for Windows 8. I actually put it into the approval process on the 21st of May, 9 days ago. It has been rejected 3 times and I’m still waiting for it to be approved. Honestly if it doesn’t go this time I’m going to give up and move onto other things. It isn’t worth my effort. I feel that most of the blame for this lies with the approval process. This is a rant about what’s wrong with the process.

REJECTED

My first complaint was that there sure were a lot of fields to be filled in when setting up the app. Somewhere in that mess of fields I forgot to fill in a privacy policy. I guess privacy policies are needed for any app which communicates over the network. Fine. The thing is you need to put it in two places, one in the app and one in the approval metadata. I had forgotten the one in the approval metadata. Okay, okay I added that and tried again.

REJECTED

WTF, I got that privacy policy in there. Oh there is a password requirementburiedin the app. I used a template for my first app and didn’t realize that there was a password requirement. Mea culpa. I could have supplied a test account but I just disabled that functionality. At 6am when I’m trying to get out to work I didn’t care. I’ll add them back in version 1.1.

REJECTED

Oh come on, now what? Oh it is in the wrong category. What? It is a blog and I put it in books. A blog is sort of like a book. Well I’m sure they gave me guidance as to which category it should be in. Oh, nope they didn’t bother with that. Jerks. I suppose I’ll guess again since there is zero guidance out there on which category to put things in.

What is pissing me off is that each one of these rejections costs me 3 days of waiting. If they had bothered to test the entire application the first time and written down all the problems then we wouldn’t be in this rejection cycle.There is a concept in computing called “fail fast” the theory being that if you fail quickly instead of trying to compensate you can get problems solved while they are still small. This isn’t a place where that theory should be applied. The long approval cycle time removes allbenefitfrom failing fast. TELL ME EVERYTHING UP FRONT.

That the app was in the wrong category should have been instantly apparent to whatever people they have approving apps. That could have been easily fixed if it had been mentioned in the first rejection, same with the login.

While I’m bitching let me say that I’m sick of signing into the store. Every fricking action needs me to sign in. Run the app through the validation process? Sign in. Create a new app? Sign in. Check on the progress of your app? Sign in. Go to the store? Sign in. Can we maybe come up with a shared key store or at least remember the god damn password within Visual Studio? After developing an app I had to throw out my keyboard as a security risk “” all the keys used for my password were worn down.

What can be done to fix it

1. Test the app from end to end. Every. Time. You’re putting a bad taste in people’s mouth by failing them again and again. Just fail them once and help them out, Jesus.

2. Be more helpful when you fail the app. I’m just guessing that my app is in the correct category now. Who knows. If the tester had told me which category it should be in I wouldn’t have had to guess. It would have been a dozen key strokes, help a brother out.

3. Tighten up the release cycle times. 3 days is too long. This isn’t 1995 when I had to ship a DVD to some magic DVD mastering center to be put into a package and mailed out to my customers.Sorry I live in a world of instant gratification. I’m sure somebody will comment and tell me how much better the windows store process is than Apple’s. I don’t care, that’s not the point. Apple should fix their stuff too.

4. The sign in stuff. Please, it is a minor annoyance but it sure as hell stacks up on top of the other stuff with which I’m dealing.

5. During the automated testing of the app it should be apparent that the app is talking to a network. You could flag the lack of privacy policy there. It is an easy fix and would have saved me a whole rejection.

6. If you’re using approvals/day as metric for the testers stop it. Crossing well into the world of supposition here I bet that some middle manager at Microsoft has looked at the approval process and said “we need to instrument this”. The obvious metric is how many apps a tester looks at in a day. By instrumenting this they are actually encouraging testers to quickly fail things and not test themthoroughly. That isn’t thebehavioryou want. Instrument something else.

This is stuff which needs to be done now. Windows 8 is shaping up to be a bigger disaster than Vista and improving theexperiencefor developers should be high on the list of things to do. I don’t have to put up with this bullshit when I deploy my app through apt-get or chocolatey nuget. The advantage of having my app in the windows store is pretty small from what I can see. I would rather distribute a classic app through traditional channels than deal with the windows store again.