Simon Online

Configuring the Solr Virtual Appliance

2011-08-18T11:25:00.000-07:00

I ran into a situation today where I needed a search engine in an application I was writing. The data I was searching was pretty basic and I could have easily created an SQL query to do the lookups I wanted. I was, however, suspicious that if I implemented this correctly there would be a desire for applying the search in other places in the application. I knew a little bit about Lucene so I went and did some research on it reading a number of blogs and the project documentation. It quickly became obvious that keeping the index in sync when I was writing new documents from various nodes in our web cluster would be difficult. Being a good service bus and message queue zealot that seemed like an obvious choice: throw up some message queues and distribute updates. Done.

I then came across Solr which is a web version of Lucene. Having one central search server would certainly help me out. There might be scalability issues in some imaginary future world where users made heavy use of search but in that future world I am rich and don't care about such thing being far too busy plotting to release dinosaurs onto the floor of the New York Stock Exchange.

I was delighted to find that there exists a virtual appliance with Solr installed on it already. If you haven't seen a virtual appliance before it is an image of an entire computer which is dedicated to one task and comes preconfigured for it. I am pretty sure this is where a lot of server hosting is going to end up in the next few years.

Once I had the image downloaded and running in Virtual Box I had to configure it to comply with my document schema. The installed which is installed points at the example configuration files. This can be changed in /usr/share/tomcat6/conf/Catalina/localhost/solr.xml, I pointed it at /usr/local/etc/solr. The example configuration files were copied into that directory for hackification.

cp -r /usr/share/apache-solr-1.4.1/example/solr/* /usr/local/etc/solr/

Once you've got these files in place you can crack open the schema.xml and remove all the fields in there which are extraneous to your needs. You'll also want to remove them from the copyField section. This section builds up a super field which contains a bunch of other fields to make searching multiple fields easier. I prefer using this DisMax query to list the fields I want to search explicitly.

The ONE API

2011-01-25T06:43:00.000-08:00

My boss was pretty excited when he came back from some sort of mobile technology symposium offered in Calgary last week. I like it when he goes to these things because he comes back full of wild ideas and I get to implement them or at least I get to think about implementing them which is almost as good. In this case it was THE ONE API. I kid you not, this is actually the name of the thing. It is an API for sending text messages to mobile phone users, getting their location and charging them money. We were interested in sending SMS messages, not because we're a vertical marketing firm or have some fantastic news for you about your recent win of 2000 travel dollars, we have legitimate reasons. Mostly.

Anyway I signed up for an account over at https://canada.oneapi.gsmworld.com/ and waited for them to authorize my account. I guess the company is out of the UK so it took them until office hours in GMT to get me the account. No big deal. In I logged and headed over to the API documentation. They offer a SOAP and a REST version of the API so obviously I fired up curl and headed over to the sandbox url documentation in hand. It didn't work. Not at all.

curl https://canada.oneapi.gsmworld.com/SendSmsService/rest/sandbox/ -d version=0.91 -d address=tel:+14034111111 -d message=APITEST

In theory this command should have sent me a message(I changed the phone number so you internet jerks can't actually call me) or at worst return a helpful error message, so said the API documentation.

What actually happened was that it failed with a general 501 error, wrapped in XML. Not good, the error should be in JSON, so says the API docs. It also shouldn't fail and if it does the error should be specific enough for me to fix. I traced the request and I was sending exactly what I should have been.

No big deal, I'll try the SOAP API and then take my dinosaur for a walk. The WSDL they provided contained links to other WSDLs, a pretty common practice. However the URLs in the WSDL were pointing to some machine which was clearly behind their firewall making it impossible for me to use it.

I gave up at that point. These guys are not the only people who are competing in the SMS space and if they can't get the simplest part of their service, the API, right I think we're done here. Add to this that they only support the three major telcos in Canada(Telus, Bell and Rogers) and there are much better options available. Twilio supports all carriers in Canada and the US and they charge, at current exchange rates, 1.5 cents a message less than these guys. Sorry THE ONE API you've been replaced by a better API, cheaper messaging and better compatibility.

Test Categories for MSTest

2011-01-12T11:04:00.000-08:00

The version of MSTest which comes with Visual Studio 2010 has a new feature in it: test categories. These allow you to put your tests into different groups which can be configured to run or not run depending on your settings. In my case this was very handy for a specific test. Most of my database layer is mocked out and I run the tests against an in memory instance of SQLite. In the majority of cases this gives the correct results, however I had one test which required checking that values were persisted properly across database connections. This is problematic as the in memory SQLite database is destroyed on the close of a connection. There were other possible workarounds for this but I chose to just have that one test run against the actual MSSQL database. Normally you wouldn't want to do this but it is just one very small test and I'm prepared to hit the disk for it. I don't want this particular test to run on the CI server as it doesn't have the correct database configured.

In order to make use of a test category start by assigning one with the TestCategory attribute.


[TestMethod]
[TestCategory("MSSQLTests")]
public void ShouldPersistSagaDataOverReinit()
{
     FluentNhibernateDBSagaPersister sagaPersister = new FluentNhibernateDBSagaPersister(new Assembly[] { this.GetType().Assembly }, MsSqlConfiguration.MsSql2008.ConnectionString(ConfigurationManager.ConnectionStrings["sagaData"].ConnectionString), true);
     ...buncha stuff...
     Assert.AreEqual(data, newData);
}

Next in your TFS build definition add a rule in the Category Filter box to exclude this category of tests.

The category field has a few options and supports some simple logic.

That is pretty much it, good work!

Updating TCPIPListener

2011-01-07T09:08:00.000-08:00

I still have a post on encryption for S3 backups coming but I ran into this little problem today and couldn't find a solution listed on the net so into the blog it goes. I have some code which is using an obsolete constructor on System.Net.Sockets.TcpListener. This constructor allows you to have the underlying system figure out the address for you. It became obsolete in .net 1.1 so this is way out of date. In order to use one of the new constructors and still keep the same behavior just use IPAdress.Any.

Old:


    new System.Net.Sockets.TcpListner(port);//warning: obsolete

New:


    new System.Net.Sockets.TcpListner(IPAddress.Any, port);

S3 backup - Part II - Bucket Policy

2011-01-04T17:11:00.001-08:00

This wasn't going to become a series of posts but it is kind of looking like it is going to be that way. I was a bit concerned about the access to my S3 bucket in which I was backup up my files. By default only I have access to the bin but I do tend to be an idiot and might later change the permissions on this bucket indadvertantly. Policy to the rescue! You can set some pretty complex access policies for S3 buckets but really all I wanted was to add a layer of IP address protection to it. You can set policies by right clicking on the bucket in the AWS Manager and selecting properties. In the thing that shows up at the bottom of your screen select "Edit bucket policy". I set up this policy


{
    "Version": "2008-10-17",
    "Id": "S3PolicyId1",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*" 
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::bucket/*",
            "Condition" : {
                "IpAddress" : {
                    "aws:SourceIp": "255.256.256.256" 
                }
            } 
        } 
    ]
}

Yep policies are specified in JSON, it is the new XML to be sure. Replace the obviously fake IP address with your IP address.

This will protect anybody other than me or somebody at my IP from getting at the bucket. I am keeping a watch on the cat just in case he is trying to get into my bucket from my IP.

Next part is going to be about SSL and encryption.

On Backing up Files

2011-01-02T16:05:00.000-08:00

I've been meaning to set up some backups for ages and at last today I got around to it. I have a bunch of computers in the house and they kind of back each other up but if my house goes up in smoke or if some sort of subterranean mole creates invade Alberta in revenge for the oil sands I'm out of luck.

I really like this company BackBlaze who offer infinite backups for $5 a month. I have some issues with them which I've outlined in another blog post which is, perhaps, a bit too vitriolic to post. Anyway they have little client which runs on OSX or Windows and steams files up to their data center. What they lack is support for UNIX operating systems and I wanted a consistent backup solution for all my computers so something platform portable was required. They also had some pretty serious downtime a while ago which I felt highlighted some architectural immaturity.

My next stop was this rather nifty S3 BackupSystem. Again this was windows only and couldn't backup network drives so it was out.

Finally I came acorss s3sync.rb a ruby script which was designed to work like rsync. It is simple to use and can be shoved into cron with narry a thought. At its most basic you need to do:

Set up Keys

export AWS_ACCESS_KEY_ID=<YOUR KEY>
export AWS_SECRET_ACCESS_KEY_ID=<YOUR SECRET KEY>

or set the variables in

$HOME/.s3conf/s3config.yml

Backup /etc to S3

s3sync.rb -r /etc <BUCKET_NAME>:<TOP_LEVEL_FOLDER_NAME>

Restore /etc from S3

s3sync.rb -r <BUCKET_NAME>:<TOP_LEVEL_FOLDER_NAME> /etc

This assumes you have set up S3 already. If not head over to Amazon and set one up. When you sign up take note of the keys they give out during the sign up. If you fail to do so they can still be accessed once the account is set up from Security Credentials under your account settings. S3 is pretty cheap and I think you even get 15gig for free now. If that ins't enough then it is still something like 15 cents a gig/month in storage and 10 cents a gig in transfer. Trivial to say the least.

Setting Version Number in TFS

2010-12-23T08:49:00.000-08:00

Yet another TFS post! Is it because I believe that TFS is the cat's meow and that you should all run out an buy it right now? Not so much but it is starting to grow on me and I can admit that building .net projects building it is okay. There is really a need for a collection of workflow tasks and some better generic templates. I'm going to see if I can convince my boss to let me open source some of the stuff I'm building for running NUnit tests and the such.

One thing which is missing from TFS is the ability to embed build information in the binaries which are built. I always like this because it is easy to get clients to right click on a file and tell you the version number which will give you an idea as to where to start looking at their problem. The version numbers in .net builds are set using attributes which are commonly defined in Properties/AssemblyInfo.cs. The quickest solution is to update the .cs files before running the compilation step of the build. To do this I created a new task to plug into the windows workflow process that is run during a build in TFS2010. Funnily enough Jim Lamb used setting the build numbers as his example for creating a custom TFS task.

I started with the project he provided and made changes to it because my build numbers are slightly different from his. Jim uses the date as the build number, that's reasonable but I want to know the exact changeset. I also changed what was passed into and out of the task. It made sense to me to pass in the IBuildDetails object rather than trying to couple task together to get out of the object what I wanted.

First change the in argument to BuildDetail


[RequiredArgument]
   public InArgument<IBuildDetail> BuildDetail
   {
       get;
       set;
   }

Then the execute to set up all the fields I want


protected override void Execute(CodeActivityContext context)
   {
       String filePath = FilePath.Get(context);

       if (String.IsNullOrEmpty(filePath))
       {
           throw new ArgumentException( "Specify a path to replace text in", "FilePath");
       }

       if (!File.Exists(filePath))
       {
           throw new FileNotFoundException("File not found", filePath);
       }

       IBuildDetail buildDetail = BuildDetail.Get(context);
       if (buildDetail == null)
       {
           throw new ArgumentException("Specify the build detail", "BuildDetail");
       }

       // ensure that the file is writeable
       FileAttributes fileAttributes = File.GetAttributes(filePath);
       File.SetAttributes(filePath, fileAttributes & ~FileAttributes.ReadOnly);

       string contents = File.ReadAllText(filePath);

       contents = SetAssemblyConfigurationString(contents, buildDetail.BuildNumber);
       contents = SetVersionNumberString(contents, buildDetail.SourceGetVersion);
       contents = SetVersionFileNumberString(contents, buildDetail.SourceGetVersion);
       contents = SetAssemblyCopyrightString(contents);

       File.WriteAllText(filePath, contents);

       // restore the file's original attributes
       File.SetAttributes(filePath, fileAttributes);
   }

Slap it into the workflow, set the properties and you're good to go. I put my task after Get Workspace in a branched version of the default workflow.

One additional thing to remember is that you need to check a compiled version of your task into TFS somewhere and point the build controller at the location. Thanks to this all the binaries which come out of my build are stamped properly.

Web Deployment in TFS Build

2010-12-21T09:36:00.000-08:00

I have been working the last couple of days on doing web deployment as part of one of my TFS builds. There are quite a few posts out there on how to do it and so far every last one of them is wrong. Technically they are fine, doing what they say will for sure deploy a package to your webserver*. However they fail to grasp the deploy only on successful build model. Why would I want to deploy something to a webserver which I had tests proving it didn't work? All that is gonna get you is trouble, and I have enough trouble around these parts. In order to get the behaviour I wanted I needed to edit the build workflow.

First step is get your build to produce the zip file and other files associated with deployment. To do that you need only add

/p:CreatePackageOnPublish=true /p:DeployOnBuild=true

to the MSBuildArgumetns in your build definition. Here is a picture because it makes this post look longer and easier to follow:
Woo, picture!

Next I branched an existing build template. These templates are located, by default, in BuildProcessTemplates folder at the root of your TFS project. You can put them anywhere you want, however, and you might want to put them in the same directory as your solution files, just to keep them spatially proximal. You want to branch rather than creating a new one because the default template is amazingly complex and recreating it would be painful. The default workflow is actually pretty reasonable and I have only minor quibbles with how it is put together.

Next we're going to to add some new parameters to the build so that people can easily configure the deployment from the build definition. To do this open up your newly branched template and click on arguments. I have yet to figure out how you can put these arguments into categories from the UI but from the XML it is pretty easy. I don't care enough to put things into categories at this time, although it probably took me longer to write this sentence about not caring than it would have taken to just do it.

As you can see I added 6 new properties

PerformDeploy - A boolean which denotes if we should attempt a deploy at all. This keeps our project generic so it can be used on anything.
DeploymentURL - The URL to which the deployment should run. I'm stuck on IIS6 so this is the MSDeployAgentService URL
DeploymentUserName - The username to use during the deploy
DeploymentPassword - The password to use during the deploy. You might want to be more creative about how you get your password to comply with whatever rules exist at your site.
DeploymentApplicationPath - The path in IIS to which deployment should be run
DeploymentCommand - The name of the *deploy.cmd file to run. This should

Now for the fun part: adding the new workflow item. If you have a slow computer, say anything slower than the computer from Voyager then you're going to be sorry you ever opened this workflow. Scroll to the very bottom where there is a task for dealing with gated checkins. Our new task is going to go after this. Drag and If from the toolbox. In the conditional we're going to check that we had no build failures and that we should run a deploy:

BuildDetail.TestStatus = Microsoft.TeamFoundation.Build.Client.BuildPhaseStatus.Succeeded And PerformDeploy

Now drop in an invoke process to the Then. In the arguments put

String.Format("/M:{0} /U:{1} /P:{2} ""-setParam:name='IIS Web Application Name',value='{3}'"" /Y ", DeploymentURL, DeploymentUserName, DeploymentPassword, DeploymentApplicationPath)

File name:

Path.Combine(BuildDetail.DropLocation, DeploymentCommand)

I also dropped a couple of WriteBuildMessage actions on there because that is how I roll. Remember this: the longer the build log the more important your job. Let's see that junior developer guy who only gets paid half what you do figure this 29Meg log file out.

That is pretty much it, save it, commit it and create a new build definition which uses it. You can either run it on commit or manually. I am running mine manually because I don't trust our unit tests to prove correctness yet.

*Assuming you have everything configured properly and you're not relying on black magic or even mauve magic.

Selenium - UI Testing - Part 1

2010-12-15T20:21:00.000-08:00

I had a bug to fix earlier this week which was really just a UI bug. The details aren't all that important but it was something like a field in a popup div wasn't being properly updated. This felt a lot like something which could use not just a test in the back end to ensure the correct data was being passed back but also a test in the front end to make sure it was being displayed. As our front end is all webby, as most front ends tend to be these days. Years ago I did some work with Mercury QuickTest as it was then called(I believe it is now owned by HP which makes it pure evil, just like HP-UX). I looked at a couple of tools for automating the browser and decided on selenium partially because it appeared to be the most developed and partially because I couldn't refuse a tool with an atomic mass of 78.96.

This is the part where I warn you that I really have no idea what I'm doing, I'm no selenium expert and I'll probably change my mind about how to best implement it. I'll post updates as I change my mind.

I started by installing the selenium IDE and, on another computer, the remote control. The IDE is just a plugin for firefox while the RC is a java based tool which happily runs on windows. I opened up a port for it in the windows firewall... oh who am I kidding I turned the damn thing off.

I recorded a number of actions using the IDE and added a number of verifications(you can add a check by right clicking on an element and selecting one of the verify options in there). I don't consider these to be unit tests since they tend to cross over multiple pages. What I'm creating are work flow tests or behavioral tests. These could be used as part of BDD in conjunction with a tool such as Cucumber or SpecFlow. As such I don't really care how long these tests take to run, I don't envision them being run as part of our continuous integration process but rather as part of hourly builds. If things really get out of hand(and they won't our application isn't that big) then the tests can be distributed over a number of machines and even run in the cloud on some Amazon instances.

In the Selenium IDE I exported the test to C# so they could be run in our normal build process. Unfortunately the default export template export nunit tests and our project makes use of the visual studio test suite. There are no technical reasons for not having both testing frameworks but I don't want to burden the other developers too much with a variety of frameworks. So my job for tomorrow is to explore alternatives to the standard export. I am also not crazy about having to maintain the tests in C#, it requires that a developer be involved in testing changes when it should really be easy enough for a business person to specify the behaviour. I have some ideas about running the transformations from Selenium file format(which is an html table) into C# using either T4 or the transformation code extracted from the IDE running in a server side javascript framework.

I'll let you know what I come up with. Chances are it will be crazy, because sensible isn't all that fun.

Useful Blogs

Design of Selenium tests for Asp.netA blog with some specific suggestions about how to work with selenium and asp.net. I'm not in 100% agreement with his ideas but I might change my mind later.

Adam Goucher's blog - Adam is a hell of a nice guy who has been helping me out with Selenium on the twitter.

Web.config Tranformation Issues

2010-11-19T14:06:00.001-08:00

As you all should I am using web.config transformations during the packaging of my ASP.net web applications. Today I was working with a project which didn't have transformations defined previously so I thought I would go ahead and add them. All went well until I built a deployment package and noticed that my transformations were not being applied. Looking at the build log I found warnings like these


C:\Users\stimms\Documents\Visual Studio 2010\Projects\SockPuppet\Web.Debug.config(5,2): Warning : No element in the source document matches '/configuration'
Not executing SetAttributes (transform line 18, 12)
Output File: obj\Debug\TransformWebConfig\transformed\Web.config

This started a fun bit of debugging during which I removed everything from the web.config and built it from the ground up. Eventually I traced the problem to a hold over from some previous version of .net, in the Web.config the configuration was defined as


<configuration xmlns="http://schemas.microsoft.com/.NetConfiguration/v2.0">

Changing that to just


<configuration>

Solved the problem. I imagine this is some sort of odd XML namespace issue, hopefully if you run into this you'll find this post and not waste an hour like me.

MySQL in an NServiceBus Handler

2010-10-05T21:55:00.000-07:00

I have an autonomous component in my NServiceBus implementation which needs to talk to a MySQL database. When I first implemented the handler I configured the end point to be transactional, mostly because I wasn't too sure about the difference between configuring AsA_Service and AsA_Client and transactions sounded like they might be something I would like. What the transactional endpoint does is wrap the endpoint in a distributed transaction. A distributed transaction is a mechanism which allows you to share a transaction between a number of databases so that if you're writing to several databases when you commit to one database and it fails then the transactions in the other databases will rollback.

However when I went to test the handler it failed with an error:

MySql /Net connector does not support distributed transactions

I solved it by configuring the endpoint AsA_Client. The problem with that configuration is that the handler wipes the queue on startup which isn't ideal. None of the built in configurations are quite right for our situation so we override the configuration as described here: http://www.nservicebus.com/GenericHost.aspx.

Fixing Table Identities

2010-06-28T08:42:00.000-07:00

I make heavy use of Red Gate's excellent SQL Compare tools. I know I'm a bit of a shrill for them but they are time savers when dealing with multiple environments(development, testing, production) which is a pretty common occurrence in any sort of agile development. One flaw in them is that they often mess up the sequences in the destination database. Say you have a table Students with 15 records in it in development and 30 in production then performing a copy often brings along the sequence even if you don't select syncing that table. This results in duplicate key errors whenever a new record is inserted.

For weeks I have been saying "I should write a script to check and fix that". Well I finally did it


SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

create PROCEDURE dbo.FixTableIdentities
 
AS
BEGIN
 SET NOCOUNT ON;
 declare @currentKeyValue int
 declare @currentIdentityValue int
 declare @toRun nvarchar(500)
 declare @tableToCheck nvarchar(500)
 declare @idColumnCount int
    declare db_cursor cursor for select name from sysobjects where type='U'
    
    open db_cursor
    fetch next from db_cursor into @tableToCheck
    
    while @@FETCH_STATUS = 0
    BEGIN
  select @idColumnCount = count(*) from syscolumns where id=object_id(@tableToCheck) and name='id'
  if(@idColumnCount = 1)
  BEGIN
    select @currentKeyValue = ident_current(@tableToCheck) 
    set @toRun = N'select @currentIdentityValue = max(id) from ' + @tableToCheck;
    EXEC sp_executesql @toRun, N'@currentIdentityValue int OUTPUT', @currentIdentityValue OUTPUT;
    if(@currentIdentityValue <> @currentKeyValue)
    BEGIN
    DBCC CHECKIDENT (@tableToCheck,reseed, @currentIdentityValue) 
    END
   END
   FETCH NEXT FROM db_cursor into @tableToCheck
 END
 CLOSE db_cursor
 deallocate db_cursor
END
GO

When run this procedure will go through all your tables and ensure that the id column is in sync with the sequence. At the moment it just looks at the column called id and manipulates that.

Measuring Language Productivity

2009-12-13T09:17:00.000-08:00

I recently asked a question over at stackoverflow about the productivity gains in various languages.


Does anybody know of any research or benchmarks of how long it takes to develop the same
application in a variety of languages?  Really I'm looking for Java vs. C++ but any
comparisons would be useful.  I have the feeling there is a section in Code Complete
about this but my copy is at work.

I was really asking because I wanted to help justify my use of the Pellet semantic reasoner over the FaCT++ reasoner in a paper.

What emerged from the question was that there really was not much good research into the topic of language productivity and that any research which had been done was from the 2000 time-frame. What makes research like this difficult is finding a large sample size and finding problems which don't favour one class of language greatly over another. That got me thinking, what better source of programmers is there than stackoverflow? There are developers from all across the spectrum of languages and abilities; there is even a pretty good geographic disbursement.

Let's do this research ourselves! I propose a stackoverflow language programming contest. We'll develop a suite of programming tasks which try as hard as possible to not focus on the advantages of one particular language and gather metrics. I think we should gather

Time taken to develop

Lines of code required

Runtime over the same input

Memory usage over the same input

Other things I haven't thought of

I'll set up a site to gather people's solutions to the problems and collate statistics but the problems should be proposed by the community. We'll allow people to checkout the problem set, time how long it takes to the to complete it and then submit the code for their answers. I'll run the code and benchmark the results and after, say two weeks of having the contest open, publish my results as well as the dataset for anybody else to analyze.

Abuse of Extension Methods

2009-12-09T07:01:00.000-08:00

In the code base I'm working with we have a number of objects which augment existing objects. For instance I have a User object which is generated by my ORM so it looks like


   string userName;
   string firstName;
   string lastName;
   int companyID;
   int locationID;

In order to display user objects it is useful to have the name of the company and the location which are stored in another table. To limit the amount of stuff being passed around we defined an ExtendedUser which extends User and adds the fields


   string companyName;
   string locationName;

Creating these extended classes requires passing in a base class and then pulling all the properties off of it and assigning them to the extended class. This is suboptimal because it means that when a new property is added to the bass class it has to be added to the code which extracts the properties in the extended class. To address this I created a method which iterates over the properties in the base class and assigns them to the extended class.


public static void CopyProperties(this object destination, object source, List<string> ignoreList)
        {
            foreach (var property in destination.GetType().GetProperties())
            {
                if (source.GetType().GetProperties().Select(p => p.Name).Contains(property.Name) && !ignoreList.Contains(property.Name))
                {
                    var sourceProperty = source.GetType().GetProperty(property.Name);
                    if (property.CanWrite && sourceProperty.GetType() == property.GetType() && sourceProperty.GetValue(source, null) != null)
                        property.SetValue(destination, sourceProperty.GetValue(source, null), null);
                }
            }
        }

If you have sharp eyes you'll notice that I've defined this method as an extension method. This allows me to do insane things like


ExpandedClass expandedClass = new ExpandedClass();
expandedClass.CopyProperties(cls);
expandedClass.LocationName = GetLocationNameFromID(cls.LocationID);
expandedClass.CourseName = GetCourseNameFromID(cls.CourseID);
expandedClass.InstructorName = GetInstructorNameFromID(cls.InstructorID);
expandedClass.CompanyName = GetCompanyNameFromID(cls.CompanyID);

I can also do this for any other two classes which share property names.

Persisting in MVC

2009-11-14T17:27:00.000-08:00

Rob Conery, who is a giant in the ASP.net MVC world(he wrote the ASP.net store front and is also the author of a 200 page book on inversion of control) is calling for suggestions about an active record persistence engine. I wanted to present how I think it should be done which is just a bit too long for a comment on the tail end of Rob's blog. I've been reading a lot lately about areas in MVC2 and the portable areas project which is part of MVC contrib project. Now the portable areas aren't yet finalized but the idea is that these areas will be able to be dropped into a project and will provide some set of controllers and views which will provide a swack of functionality.

The example I've seen bandied about is that of a forum. Everybody has written a forum or two in their time now you can just drop in an area and get the functionality for free. I can see a lot of these components becoming available on codeplex or git hub. Component based software like this is "the future" just like SOA was the future a few years ago. The problem with components like this is that it is difficult to keep things consistent across the various components. At one end of the spectrum of self containment If each component is self contained then it has to provide for its own data persistence as well as any other services it consumes.

I have helpfully harnessed the power of MS Paint to create an illustration of the spectrum between a component being self contained and being reliant on services being provided for it. If anybody is interested my artistic skills are available for hire. The further to the left the more portable the further to the right the more reliant the components are on services being provided for them and the less portable. We want to be towards the left, because left is right.

If these components really are the future then we need to find a way to couple the components and provide communication between them. This is where MEF steps up to the plate. What I'm proposing is that rather than spending our time creating unified interfaces for storing data we create a method agnostic object store. Components would call out to MEF for a persistence engine and then pass in whatever it was they wanted to save. The engine should handle the creation of database tables on the fly or files or web service callouts to a cloud. That is what I believe should exist instead of a more concrete IActiveRecordEngine.

What's the advantage? We keep the standard interface for which Rob is striving but we can now have that interface implemented by a plugable component rather than having it hard coded into a web.config.

The second part of Rob's post is about creating opinionated controllers. I have to say I'm dead against that. I agree with the goal of implementing the basic CRUD operations for people, in fact I'm in love with it. What I don't like is that it is implemented in a base class from which my controllers descend. If I'm reading the post correctly then the base controller is implementing actual actions. It is dangerous to implement actions willy nilly, actions which could be dangerous and people wouldn't even realize the actions exist. Chances are very good that users are just going to leave the actions implemented rather than overriding them with noop actions.

Another option is that I'm reading this post incorrectly and the methods in the base class are private and not actions. I like that a lot more, but even more I like generating template controllers. Subsonic 3 follows this methodology and it is really nice to be able to twiddle with bits of the implementation. What's more the generation doesn't have to stop at the controller. If the implementation in the controller is known then why not generate the views as well?

All in all I like the idea of improving the object store support in ASP.net MVC but I would like it to be as flexible as possible.

Quick post on getting node information from Umbraco with IronPython

2009-10-28T13:30:00.001-07:00

I was just working with the IronPython page type in umbraco and needed to get a property from the page I was on. This can be done by accessing the Node API found in umbraco.presentation.nodeFactory. In order to be able to pull a value you will need pull in that part of the API


import umbraco.presentation.nodeFactory
from umbraco.presentation.nodeFactory import *

Now you can get the current node and query its properties


print  Node.GetCurrent().GetProperty("Address").Value

xVal PostSharp 1.0 Demo Project

2009-10-07T21:37:00.000-07:00

After my last post I thought I would look at the demo project for xVal 1.0 and see if I could get it working with PostSharp. It was a little bit differnt in how it way set up from my projects but I figured it could still be improved with PostSharp. My first issue was that the method I was intercepting was in the entity itself rather than in a repository. This meant that there were methods in the entity which I didn't wish to intercept. There were two classes of those

Accessor methods - we don't need to intercept getters

Internal methods - ASP.net MVC uses reflection to examine the internals of the data classes in order to bind form results to them. We can avoid intercepting these by ignoring methods which start with '.'

Next because the entity already contained all of the data it needed to persist the persistance method didn't have any arguments. In the previous post I assumed that this would always be the case. You know what they say about assuming: if you assume you make a jerk out of everybody in Venice. Pretty sure that is the saying.

This required and expansion of the current validator


public override void OnEntry(MethodExecutionEventArgs eventArgs)
        {
            if (IsAccessor(eventArgs) || IsInternal(eventArgs))
                return;
            if (HasArguments(eventArgs))
            {
                CheckPassedInformation(eventArgs);
            }
            else
            {
                CheckSelfContainedEntity(eventArgs);
            }
            base.OnEntry(eventArgs);
        }

        private static bool HasArguments(MethodExecutionEventArgs eventArgs)
        {
            return eventArgs.GetReadOnlyArgumentArray() != null && eventArgs.GetReadOnlyArgumentArray().Count() &rt; 0;
        }

        private static void CheckPassedInformation(MethodExecutionEventArgs eventArgs)
        {
            var toValidate = eventArgs.GetReadOnlyArgumentArray()[0];
            var errors = DataAnnotationsValidationRunner.GetErrors(toValidate);
            if (errors.Any())
                throw new RulesException(errors);
        }

        private static void CheckSelfContainedEntity(MethodExecutionEventArgs eventArgs)
        {
            var toValidate = eventArgs.Instance;
            var errors = DataAnnotationsValidationRunner.GetErrors(toValidate);
            if (errors.Any())
                throw new RulesException(errors);
        }

        public bool IsAccessor(MethodExecutionEventArgs eventArgs)
        {
            if (eventArgs.Method.Name.StartsWith("get_"))
                return true;
            return false;

        }

        public bool IsInternal(MethodExecutionEventArgs eventArgs)
        {
            if (eventArgs.Method.Name.StartsWith("."))
                return true;
            return false;

        }

You can see I did a little bit of clean code refactoring in there to extract some methods. Now two different methods of saving information are checked.

The lesson here seems to be that the way in which you construct your data persistance layer has an effect on the construction of the validator such that there is no generic aspect which you can download and use.

You can download my modified xVal project here. In order to get it running you'll need to have PostSharp installed but you're going to want it for lots of other stuff so get going on installing it.

Oh just one more note, when you're trying it out be sure to disable javascript so that the page actually posts back and doesn't validate using the javascript validation.

Cleaning Up xVal Validation With PostSharp

2009-09-16T21:10:00.001-07:00

Even though the new ASP.net MVC 2.0 framework comes with built in validation it is useful to look at some alternatives. Afterall you don't want Microsoft telling you how to do everything, do you? One of the better validation frameworks is the xVal framework which just went 1.0. In this release there has been added support for a number of new features, probably the coolest of which is AJAX based validation for complex input.

However xVal does have one drawback, it is quite verbose to implement. In every method which alters data you will have to put something like


            var errors = DataAnnotationsValidationRunner.GetErrors(this).ToList();
 
            if(errors.Any())
                throw new RulesException(errors);

This is a bit repedative and kind of tiresome. Sure we could extract a method from this and just call that each and every time we edit data but that doesn't really solve the underlying problem of having code which is repeated.

Enter PostSharp.

PostSharp is an aspect oriented addon for .net languages. It acutally does most of its weaving in a post build step modifying the MSIL the compiler generates. We can extract the validation into an aspect.


 [Serializable]
    public class ValidateAttribute : OnMethodBoundaryAspect
    {
        public ValidateAttribute(){}

        public override void OnEntry(MethodExecutionEventArgs eventArgs)
        {
            if (eventArgs.GetReadOnlyArgumentArray() != null && eventArgs.GetReadOnlyArgumentArray().Count() > 0)
            {
                var toValidate = eventArgs.GetReadOnlyArgumentArray()[0];
                var errors = DataAnnotationsValidationRunner.GetErrors(toValidate);
                if (errors.Any())
                    throw new RulesException(errors);
            }
        }
    }

Here we create an OnEntry method which is called before the method we are intercepting. We skip any method with no arguments since it isn't likely to be updating data. Then we extract the argumetns and pass them into the validator for it to do its business.

Finaly we give the PostSharp framework a bit of information about where to use this aspect

AssemblyInfo.cs


[assembly: CertificateSearch.Aspects.Validate(AttributeTargetTypes = "CertificateSearch.Models.*Repository")]

I have applied it to all the methods in files which end with Repository the Models namespace. That covers all the data modification methods in the project. I can now add new methods without the cost of ensuring that I validate each one.

IE Caches JSON

2009-09-16T20:51:00.000-07:00

I ran into an interesting problem today on everybody's favorite browser Internet Explorer. At issue was a page I had which was partially populated using jQuery's getJSON function. As it turns out even though I had the caching turned to no-cache on the server IE was perfectly happy to cache the document because it was fetched using GET. Apparently this is OK to do. Obviously this ruins my site's functionality so I instructed jQuery to override it by setting


   $.ajaxSetup({ cache: true });

This function works by adding a nonsense value to the end of the request. Looking at the actual jQuery source we can see that the current time is appened to the request which makes the browser believe it is a new URL.


  if ( s.cache === false && type === "GET" ) {
   var ts = now();

   // try replacing _= if it is there
   var ret = s.url.replace(rts, "$1_=" + ts + "$2");

   // if nothing was replaced, add timestamp to the end
   s.url = ret + ((ret === s.url) ? (rquery.test(s.url) ? "&" : "?") + "_=" + ts : "");
  }

Problem solved

Caching and Why You Shouldn't Listen to Blogs

2009-08-28T16:24:00.000-07:00

A while ago I wrote and article entitled "HTML Helper for Including and Compressing Javascript", no don't click on that link because it is all wrong. The gist of the article was that in order to save clients from downloading a bunch javascript files each time they visited the site opening up a bunch of costly connections a handler would include all those files in the HTML page and, as an added bonus, compress them. I forgot one key thing and leddt was good enough to point it out. Because each page you load on the site has the javascript inlined there is no way to cache it.

So what do we do? I think the best solution is to stop compressing the javascript on request. Instead combine and compress the javascript as part of the build process and then serve it up as a separate request. The disadvantage here is that if you use a library like jquery-ui on only one page you end up downloading it for any page the user visits. However the price you pay in getting that is a one time cost while using the terrible solution I suggested before you pay for it again and again and again. In that way it is much like not taking out the trash, I have to hear about it every day plus it smells. You wouldn't believe how hard it is to get the smell out of the fur of the 40 dogs in the puppy mill I run in my garage.

How things are cached on a browser has always been a mystery to me and there isn't a whole lot on the internet about the technicalities of what browsers do and do not cache. Basically it seems to come down to the cache-control header which govern how devices retain content. The ever so verbose W3C HTTP 1.1 spec defines the grammar for cache-control as


Cache-Control   = "Cache-Control" ":" 1#cache-directive
    cache-directive = cache-request-directive
         | cache-response-directive
    cache-request-directive =
           "no-cache"                          ; Section 14.9.1
         | "no-store"                          ; Section 14.9.2
         | "max-age" "=" delta-seconds         ; Section 14.9.3, 14.9.4
         | "max-stale" [ "=" delta-seconds ]   ; Section 14.9.3
         | "min-fresh" "=" delta-seconds       ; Section 14.9.3
         | "no-transform"                      ; Section 14.9.5
         | "only-if-cached"                    ; Section 14.9.4
         | cache-extension                     ; Section 14.9.6
     cache-response-directive =
           "public"                               ; Section 14.9.1
         | "private" [ "=" <"> 1#field-name <"> ] ; Section 14.9.1
         | "no-cache" [ "=" <"> 1#field-name <"> ]; Section 14.9.1
         | "no-store"                             ; Section 14.9.2
         | "no-transform"                         ; Section 14.9.5
         | "must-revalidate"                      ; Section 14.9.4
         | "proxy-revalidate"                     ; Section 14.9.4
         | "max-age" "=" delta-seconds            ; Section 14.9.3
         | "s-maxage" "=" delta-seconds           ; Section 14.9.3
         | cache-extension                        ; Section 14.9.6
    cache-extension = token [ "=" ( token | quoted-string ) ]

Pretty simple. The fields you have to watch out for are max-age and public/private. The age is how long the cache is permitted to retain the document before it must rerequest it and public indicates the page is public and should be cached for all users. How the actual browsers will implement these is a function of the users so all you can do is make sure your site obeys the standard.

But don't trust me, I'm just a blogger and I already lied to you once. In my next post I'll talk a bit about caching in ASP.net and how to save database trips.

Bookmarklet for MSDN

2009-08-19T12:37:00.000-07:00

Today I was cruising the old MSDN using their much better low bandwidth version when I stumbled across a page on events in C#. What got my attention was the example code all grey and boring not to mention hard to follow. What this page needed was a little bit of SyntaxHighlighter Alex Gorbatchev's glorious javascript library which add syntax to source code. I use it right here on my blog as does everybody else who passes the coolness test. The test, of course, being the use of SyntaxHighlighter. I hacked at the jQueryify bookmarklet and managed to get it to load the correct SyntaxHighlighter libraries and stylesheets. The next time you're on MSDN squinting at a piece of code try hitting MSDN Style. ~~It only works on firefox at the moment but I'll update it to work on IE as well.~~ Simply drag this link to your bookmarks bar and you're good to go:

MSDN Style

Source


javascript:%20(function(){
 function getScript(url,success){
  var script=document.createElement('script');
  script.src=url;
  var head=document.getElementsByTagName("head")[0], done=false;
  script.onload=script.onreadystatechange = function(){
    if ( !done && (!this.readyState ||
     this.readyState == "loaded" || this.readyState == "complete") ) {
   done=true;
   success();
    }
  };
  head.appendChild(script);
  };
  getScript('http://alexgorbatchev.com/pub/sh/2.0.320/scripts/shCore.js',function(){});
  getScript('http://alexgorbatchev.com/pub/sh/2.0.320/scripts/shBrushCSharp.js',function(){});
  getScript('http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js',function() {
  loadStyle('http://alexgorbatchev.com/pub/sh/2.0.320/styles/shCore.css');
  loadStyle('http://alexgorbatchev.com/pub/sh/2.0.320/styles/shThemeDefault.css');
      return completeLoad();
  });
  function loadStyle(url)
 {
  style=document.createElement('link');
  style.href = url;
  style.rel = "stylesheet";
  style.type="text/css";
  document.getElementsByTagName("head")[0].appendChild(style);
    };

  function completeLoad() {
     $(".libCScode:not(div)").addClass("brush: csharp");
     
     SyntaxHighlighter.highlight();
  };
})();

A Wonderful Example of Comments Causing Problems

2009-08-07T10:42:00.001-07:00

Current thinking is that comments in code should be helpful and should only document why and not how, that is what the code is for. That wasn't always the case, for example


$courseNick = $_POST['courseNick']; //Course nick
$phone = $_POST['phone'];           //Course nick

See how useless that is?

A watershed moment

2009-08-07T07:10:00.000-07:00

News in the twitterverse today is all about this article at inforworld or, more precisely, the guidelines produced by the American Law Institute. In a small section they suggest that software companies should be held liable for shipping software with known bugs. Damn right they should.

The software world has long survived protected from their own mistakes through the EULA. Let's look at a license agreement, perhaps the Windows XP license agreement as a typical example. Here is a excerpt from section 2.15:

LIMITATION ON REMEDIES; NO CONSEQUENTIAL OR OTHER DAMAGES. Your exclusive remedy for any breach of this Limited Warranty is as set forth below. Except for any refund elected by Microsoft, YOU ARE NOT ENTITLED TO ANY DAMAGES, INCLUDING BUT NOT LIMITED TO CONSEQUENTIAL DAMAGES

So if Windows XP crashes and you loose an assignment or if your battleship sinks Microsoft is sorry but they aren't going to stand up and take responsibility. At least not more than the purchase price of Windows and I'll bet you it is a trial to get them to cough up even that. A lot of people are upset about this guideline. By a lot of people I, of course, mean software vendors. I suppose I too would be upset if I shipped known bad software, but I don't because my customers deserve more than that. I'm not saying my software is perfect, it isn't, however it has no known bugs. And that's the key right there: know bugs.

All software is going to have bugs in it, even with the most rigorous testing and test driven development there are still going to be issues. Most of these bugs are not covered under the guidelines because a concerted effort was made to find bugs and fix them during the development process. This doesn't mean that you can't ship software with know issues, it just means that now the risk of doing so is spread more evenly between you and your client. Which is only right.

Drawing again on the tired car analogy people would be outraged if Ford shipped a car which they knew imploded in the rain. Sure the car industry isn't a perfect analogy but it is pretty good. Lots of software is responsible for our lives in the same way as cars, heck there is a huge amount of software in cars. Why should software vendors be any less liable?

What does this really mean for companies? In a word: transparency. Let's say that I ship some software with an issue and it hurts somebody and they decide to sue. It is going to cost me money in lawyers even if I had no idea that defect existed. I've got to show that I didn't know the defect existed at shipping time. That is going to require a bunch of e-discovery and searching of e-mails, costly. What can I do to try to avoid being sued in the first place? Easy, publish all the bugs in my software in a system the public can see. I mean internal bugs as well as external bugs, everything. Next I fix bugs before I write new code and I fix them promptly. If I can gain a reputation as open and responsive people are far more likely to write off my mistakes as just that.

Now somebody is going to argue that publishing every defect puts me at a PR disadvantage compared to the guys down the street who don't advertise their bugs. I don't believe that for a second. People who buy software are generally not dumb, they know that bugs exist and they know that they're probably going to find some in the software they buy. Do you want to deal with a company which won't admit that they have bugs and even threatens to sue people who bring them issues or with one which has shown itself to be responsive to customer issues? I'll take responsive every time.

The time has passed when software was used only for esoteric purposes by men in white coats and it is time the industry grew up and learned that if you get a paper route you can't dump the papers in the garbage without consequences. Stop dumping bad code on the world.

HTML Helper for Including and Compressing Javascript

2009-06-23T18:29:00.000-07:00

This article is outdated and ill advised

If you're like me then you probably have a whole bunch of javascript includes on your site cluttering up the top of your page. I tend to keep most of mine in Site.Master, it isn't so much that I use them all on every page but I do make use of a large enough subset that I can't be bothered to load them in each view. For example here is what I have at the top of a Site.Master page for my Activity Tracker project:


 <script src="<%=ResolveClientUrl("~/Scripts/jquery-1.3.2.js") %>" language="javascript" type="text/javascript"></script> <br/>   <script src="<%=ResolveClientUrl("~/Scripts/jquery.tablesorter.min.js") %>" language="javascript" type="text/javascript"></script> <br/>   <script src="<%=ResolveClientUrl("~/Scripts/extjs/ext-jquery-adapter.js")%>" language="javascript" type='text/javascript' ></script> <br/>   <script src="<%=ResolveClientUrl("~/Scripts/extjs/ext-all-debug.js")%>" language="javascript" type='text/javascript' ></script> <br/>   <script src="<%=ResolveClientUrl("~/Scripts/jquery.autocomplete/js/jquery.autocomplete.js")%>" language="javascript" type='text/javascript' ></script> <br/>   <br/>   <!--Validation--><br/>   <script src="<%=ResolveClientUrl("~/Scripts/xVal.jquery.validate.js") %>" language="javascript" type="text/javascript"></script><br/>   <script src="<%=ResolveClientUrl("~/Scripts/jquery.validate/jquery.validate.js") %>" language="javascript" type="text/javascript"></script><br/>   <br/>   <!--Graphs--><br/>   <script src="<%=ResolveClientUrl("~/Scripts/flot/jquery.flot.js") %>" language="javascript" type="text/javascript"></script> <br/>   <script src="<%=ResolveClientUrl("~/Scripts/flot/excanvas.js") %>" language="javascript" type="text/javascript"></script> <br/>   <script src="<%=ResolveClientUrl("~/Scripts/flot/ext.flot.js") %>" language="javascript" type="text/javascript"></script> <br/>   <script src="<%=ResolveClientUrl("~/Scripts/jquery-ui-1.7.1.js") %>" language="javascript" type="text/javascript"></script>

This thing was not only a mess but it was also amazingly inefficient. Each time the web browser opens a connection to the server there is overhead, if you put everything in one file then you can save on this overhead. Apparently you can only have two open connections per domain so you don't really get any advantage from having multiple .js file which run parallel. I was starting to notice the overhead just opening up pages from my localhost. Things would get bad if I deployed this to the real world. I went through a couple of possible solutions:

1. Manually combine the javascript files, they change infrequently
2. Combine the javascript during the build and produce just one file
3. Use edge caches of the javascript files

Manual combination of the files was out right away, I don't like doing work over and over again that is what computers are for. I liked the idea of 2 because I'm a huge build guy and to do that I would get to play with msbuild or nant or something like that. However I liked being able to make javascript change without rebuilding the site.

Solution #3, using edge caches, seemed like a great plan. The issue is that these systems are not always reliable, not from a always up point of view but from a changes point of view. I couldn't govern when they made changes to the scripts and I would have to use stock versions of all the scripts. The autocomplete script I used is slightly modified for my purposes so #3 was out.

I decided that what I needed was a home grown solution so I came up with Html.IncludeScriptLibraries. This is a new extension method (I'm in to those at the moment) which loaded javascript files for me.


 public static string IncludeScriptLibraries(this HtmlHelper helper, string basePath, params string[] scripts)
        {
            String contents = "";
            foreach(String script in scripts)
            {
                String fileName = basePath + "\\" +  script.Replace("/", "\\");
                if (File.Exists(fileName))
                {
                    StreamReader reader = File.OpenText(fileName);
                    contents += reader.ReadToEnd();
                reader.Close();
            }
            }
            return "<script language=\"javascript\" type=\"text/javascript\">" + contents + "</script>";
        }

This simple function opens a list of script files from some directory and concatenates them into one string. This is then returned. The master page now has


<%= Html.IncludeScriptLibraries(Server.MapPath(ResolveClientUrl("~/Scripts")), 
        "jquery-1.3.2.js",
        "jquery.tablesorter.min.js",
        "extjs/ext-jquery-adapter.js",      
        "extjs/ext-all-debug.js",
        "jquery.autocomplete/js/jquery.autocomplete.js",
        "xVal.jquery.validate.js",
        "jquery.validate/jquery.validate.js",
        "flot/jquery.flot.js",
        "flot/excanvas.js",
        "flot/ext.flot.js",
        "jquery-ui-1.7.1.js"
           )%>

Pretty nifty. Now we have this library inclusion abstracted into a function we can add some other cool stuff. I grabbed a .net port of the pack javascript library from Dean Edwards and turned it into a library in my project. I changed the extension method to return


ECMAScriptPacker packer = new ECMAScriptPacker();
return "<script language=\"javascript\" type=\"text/javascript\">" + packer.Pack(contents) + "</script>";

Which should have nicely packed up the files. However the library seems to munge javascript as does the sample application, which is a pitty. If it had worked that would be pretty cool.

This extension method is a way to centralize the loading of javascript onto a pages and provides the ability to preprocess it through compression or anything else. A caching level should be added to avoid hitting the disk with every query.

As always I'm curious about possible improvements to this.

Bar Graphs Using Flot and ASP.net MVC - Part II

2009-06-22T19:29:00.000-07:00

I wasn't really planning a part II to this article but I left the code in something of a mess and we can do much better than having to write a bunch of javascript each time we need to make a graph. Let's use the HtmlHelper and extension methods. The HtmlHelper is a class in System.Web.Mvc which is often extended to supply shortcuts for writing out HTML. xVal extends it to add client side validation and we're going to do the same thing to write out the javascript for our graphs. We start by creating a new extension class to complement HtmlHelper


public static class HTMLExtensions
    {
        public static string BarGraph(this HtmlHelper helper, string divName, IDictionary<String, int> dataPoints)
        {
            String dataPointsArrayName = divName + "Values";
            String dataPointLabelsArrayName = divName + "Labels";
            
            
            String returnText = @"<script language=""javascript"">
                                    var {0} = ["; 

            int counter = 0;
            foreach (KeyValuePair<string, int> kvp in dataPoints)
            {

                if (counter > 0)
                    returnText += ",";
                returnText += "[" + counter + ".5, " + kvp.Value + "]";
                counter++;
            }
            returnText += "];\n";

            returnText += @"var {1} = ["; 
            counter = 1;

            foreach (KeyValuePair<string, int> kvp in dataPoints)
            {
                if (counter > 1)
                    returnText += ",";
                returnText += "[" + counter + ",\"" + kvp.Key + "\"]";
                counter++;
            }
            returnText += "];\n";

            returnText += @"$(document).ready(function() 
                            {
                                $.plot($(""#{2}""), [{0}], { 
                                  series: {
                                    data: {0}, 
                                    color: ""rgb(182,188,194)""  
                                    }, 
                                  bars: { show: true, barWidth: 1.0 }, 
                                  yaxis: { min: 0 },
                                  xaxis: { ticks: {1}} 
                                  });
                            });";

            returnText += "</script>";

            returnText = returnText.Replace("{0}",dataPointsArrayName).Replace("{1}", dataPointLabelsArrayName).Replace("{2}", divName);

            return returnText.ToString();
        }

    }

This class is a bit of a mess of string appending but hopefully changes to it will be infrequent. I know somebody is going to jump on my string appending but before you do take a read of Jeff Atwood's excellent examination of micro optimization. The key take away is that all the messy javascript generation which we did have in the views is now centralized into one place.

So now we have an extension method we can call from our View, so let's pop back over to that. The page has two graphs on it and it use to have a huge amount of hand coded javascript. That can all be replaced with


<%Dictionary<string, int> topTeamsDictionary = (ViewData["TopTeams"] as List<TeamTotal>).ToDictionary(n => n.team.teamName, n => n.total);
Dictionary<string, int> userActivityTotals = (ViewData["UserActivityTotals"] as List<UserActivityTotal>).ToDictionary(n => n.ActivityName, n => n.TotalPoints);%>

<%= Html.BarGraph("teamPointsGraph", topTeamsDictionary)%>
<%= Html.BarGraph("UserActivityTotalsGraph", userActivityTotals)%>

The data in the ViewData was in a list and that list is need by other things on the page so we quickly transform the List into a Dictionary using LINQ. That is pretty much it. So long as the div referenced in the call to BarGraph exists you should get the same stylish graphs as yesterday. Obviously there are a lot of other options which can be passed through to the graph, these are left as an exercise for the reader. I've always wanted to say that.

Bar Graphs Using Flot and ASP.net MVC

2009-06-21T18:51:00.001-07:00

There are a bunch of posts out there about using flot to create nifty HTML graphs but the Internet is all about reiteration and I need to start posting more frequently. Flot is a javascript library which uses HTML Canvas from HTML 5 to draw simple vector graphics. It works natively on good browsers and with a little bit of hacking on IE8 too. At its most simple one needs only include the jquery libraries and the flot libraries. Technically the excanvas library needs only be included for IE but it doesn't seem to do any harm to include it all the time and the packed version is just a few kb.


<script src="<%=ResolveClientUrl("~/Scripts/jquery-1.3.2.js") %>" language="javascript" type="text/javascript"></script>



<script src="<%=ResolveClientUrl("~/Scripts/flot/jquery.flot.js") %>" language="javascript" type="text/javascript"></script><script language="javascript" type="text/javascript" src="<%=ResolveClientUrl("~/Scripts/flot/excanvas.pack.js")%>"></script>

As you can see I always use the ResolveClientURL function to resolve paths to javascript libraries. This section is added to my Site.Master but if your site doesn't include graphs on every page then it would be more efficient to include it only on the pages with graphs. For the purposes of this post we're also going to add


<script src="<%=ResolveClientUrl("~/Scripts/flot/ext.flot.js") %>" language="javascript" type="text/javascript"></script>

which adds some extensions to the normal flot package. We're most interested in tick labels on the x-axis. More on that later. Now we need to get some data ready to be graphed. I have a handy function in the repository for fetching a list of data objects, in the controller for the page I call


public ActionResult Index()
        {
           ViewData["UserActivityTotals"] repository.getUserActivityTotals(user.userID);
           return View()
        }

This returns a collection of UserActivityTotal objects which are lightweight objects


namespace ActivityTracker.Models

{
    public class UserActivityTotal
    {
        public string ActivityName { get; set; }
        public int TotalPoints { get; set; }
    }
}

Now in the view I print out this information into javascript


var userActivitiesNames = [<% 
            counter = 1;
            if(null != ViewData["UserActivityTotals"])
            {
                foreach(UserActivityTotal uat in ViewData["UserActivityTotals"] as List<UserActivityTotal>)
                {
                    if(counter > 1)
                        Response.Write(",");
                    Response.Write("[" + counter + ",\"" + uat.ActivityName + "\"]");
                    counter ++;
                }
            }
        

         %>];

That counter thing is in there because javascript can't handle a trailing comma in array definitions unlike Perl or Ruby. That would be a really nice thing to fix for the next version of javascript. Anyway this results in something which looks like


var userActivities = [[0.5, 1899],[1.5, 157],[2.5, 904]];

We're just putting in a monatomically increasing series for the x value. These ensure that the columns don't overlap or have large gaps between them. Why don't we just put in the values which we have in the ActivityName field? Because flot doesn't handle x-axis labels which are not numerical or dates. For that we have to start using the extension to flot which we included earlier. First let's just get the graph onto the page.

Add a div to define the size and location of the graph

and a snippet of javascript which sets up the graph object and assigns it to the div


$.plot($("#UserActivityTotalsGraph"), [userActivities], {
                                label: "Points",
                                series: {
                                    data: userActivities,
                                    color: "rgb(182,188,194)"
                                },
                                bars: { show: true, barWidth: 1.0 },
                                yaxis: { min: 0 }
                            });

That looks pretty nice except for the labels on the bottom. To get those working we hop back into the view and add an xaxis line to the plot call.


$.plot($("#UserActivityTotalsGraph"), [userActivities], {                                label: "Points",
                                series: {
                                    data: userActivities,
                                    color: "rgb(182,188,194)"
                                },
                                bars: { show: true, barWidth: 1.0 },
                                yaxis: { min: 0 },
                                xaxis: { ticks: userActivitiesNames}
                            });

This uses another array called userActivitiesNames which we define with


var userActivitiesNames = [<%
            counter = 1;
            if(null != ViewData["UserActivityTotals"])
            {
                foreach(UserActivityTotal uat in ViewData["UserActivityTotals"] as List<UserActivityTotal>)
                {
                    if(counter > 1)
                        Response.Write(",");
                    Response.Write("[" + counter + ",\"" + uat.ActivityName + "\"]");
                    counter ++;
                }
            }
       

         %>];

And we're done! The final product looks like this:

Building Adobe Air Files

2009-04-29T20:33:00.000-07:00

Assembling Adobe Air installers is pretty easy. In this article I'll show you how to put together an ant build.xml file which will build for you.

For the incurably impatient here is the whole thing:

Let's break it down a bit

Here we set up some properties to be used in the rest of the file. I'm sure I don't have to quote the Pragmatic Programmer about repeating yourself.

Air applications have to be signed. As part of a release process you'll want to generate a proper key and keep in somewhere safe like on a cd in the cat's litter box. However we don't want developer builds getting out into the wild so we'll just generate a key here and use it. You'll notice on line 12 we sleep for a couple of seconds. Why is that? It turns out that the key generator returns before it is actually done generating so we'll give it a few more seconds otherwise you'll get an error like


failed while unpackaging: [ErrorEvent type="error" bubbles=false cancelable=false eventPhase=2 text="invalid package signature" errorID=5022]
starting cleanup of temporary files
application installer exiting

in the Air installation log. (See http://kb.adobe.com/selfservice/viewContent.do?externalId=kb403123 to read how to enable logging of Air installations)

Finally we come to actually building the application

Here I'm replacing teh version number in the application.xml with the SVN_REVISION. Our builds run from inside the wonderful Hudson build management system which is kind enough to pass through a token containing the revision number from SVN. We finally execute adt the Air packaging tool giving it a list of the files we would like to have inside the package. We list them by hand as a double check against accidentally including extra files, like bankingInformation.xls.

That's it! The only real trick is pausing for the key generation to complete.

ASP.net MVC returning JSONP

2009-04-18T14:28:00.000-07:00

I've been working on a piece of code which returns JSON in response to some request. It has all being going fine on the local host but I've just deployed it up to my new test server and started to access it using JQuery and it stooped working. This is, of course, typical. This time however, my problem was ignorance rather than a programming blunder. As it turns out when returning JSON across domains you need to actually return JSONP in order to get call backs working. What is JSONP? I'm glad you asked, because I had no idea either. Basically it is the same JSON you love but wrapped with a bit of text which specifies the name of the function to call upon returning.

JSON:

{"userID":"00000000-0000-0000-0000-000000000000","success":"false"}

JSONP:

somefunction({"userID":"00000000-0000-0000-0000-000000000000","success":"false"})

Easy enough. In JQuery you need to just add another parameter to the JSON call in order to pass the name of the function to the server

$.getJSON(URL + "/json/Message/sendMessage?userName=" + $("#userName3").val() + 
                                                    "&messageText=" + $("#message").val() +
                                  "&userKey=" + key + 
                                          "&jsoncallback=?", 
                            function(json)
                            {
                            }
                );

JQuery will automatically replace the ? with the name of your callback function, in this case an anonymous function.

Having discovered all of this I realized that I now had a ton of code returning JsonResults which needed to be changed. I figured the best way to do this was to actually create a JsonpResult which was based off of the JsonResult. So I did just that, basing it off of the now open sourced ASP.net MVC JsonResult

public class JsonpResult : System.Web.Mvc.JsonResult
    {
        public override void ExecuteResult(ControllerContext context)
        {
            if (context == null)
            {
                throw new ArgumentNullException("context");
            }
            
            HttpResponseBase response = context.HttpContext.Response;

            if (!String.IsNullOrEmpty(ContentType))
            {
                response.ContentType = ContentType;
            }
            else
            {
                response.ContentType = "application/json";
            }
            if (ContentEncoding != null)
            {
                response.ContentEncoding = ContentEncoding;
            }
            if (Data != null)
            {
                // The JavaScriptSerializer type was marked as obsolete prior to .NET Framework 3.5 SP1
#pragma warning disable 0618
                HttpRequestBase request = context.HttpContext.Request;
                
                JavaScriptSerializer serializer = new JavaScriptSerializer();
                if (null != request.Params["jsoncallback"])
                    response.Write(request.Params["jsoncallback"] + "(" + serializer.Serialize(Data) + ")");
                else
                    response.Write(serializer.Serialize(Data));
#pragma warning restore 0618
            }
        }
    }

I then extended Controller

public class WopsleController : Controller
    {
        protected internal JsonpResult Jsonp(object data)
        {
            return Jsonp(data, null /* contentType */);
        }

        protected internal JsonpResult Jsonp(object data, string contentType)
        {
            return Jsonp(data, contentType, null);
        }

        protected internal virtual JsonpResult Jsonp(object data, string contentType, Encoding contentEncoding)
        {
            return new JsonpResult
            {
                Data = data,
                ContentType = contentType,
                ContentEncoding = contentEncoding
            };
        }

    }

and altered all my controllers to extend WopsleController rather than Controller. Seems to work pretty well.

Best Bug I've Written Today

2009-02-28T19:21:00.000-08:00

Today I create a handy little class in my attempt to rewrite wordle. It encapsulated a word and a word count and implemented comparable

1 package com.simontimms.wordle;
2
3 public class WordCountElement implements Comparable<WordCountElement>{
4
5
6 private int count = 0;
7 private String word;
8
9
10 public WordCountElement (String word, int count)
11 {
12 this.word = word;
13 this.count = count;
14 }
15
16 public int compareTo(WordCountElement toCompare)
17 {
18 return this.count - toCompare.getCount();
19
20 }
21
22 public void setCount(int count) {
23 this.count = count;
24 }
25
26 public int getCount() {
27 return count;
28 }
29
30 public void setWord(String word) {
31 this.word = word;
32 }
33
34 public String getWord() {
35 return word;
36 }
37 }

At another point in my code I created a sorted set out of these WordCountElements

1 public TreeSet<WordCountElement> getSortedSetOfWordCounts(String textToAnalyze)
2 {
3 HashMap<String, WordCountElement> unsortedSet = getWordCounts(textToAnalyze);
4 TreeSet<WordCountElement> sortedSet = new TreeSet<WordCountElement>();
5
6 for(String s : unsortedSet.keySet())
7 {
8 sortedSet.add(unsortedSet.get(s));
9 System.out.println(s);
10 }
11 return sortedSet;
12 }

I couldn't get my unit tests to pass when I had more than one word in the unsortedSet. I ended up debugging it and found that even though I was adding two different words only the first was present in the sorted set. Well select might not be broken but perhaps sorted sets were. Can you see the bug?

That's right, the custom comparator resulted in two words with equal counts being equal. So select wasn't broken. Drat.

Tweetdeck on 64bit Ubuntu Llinux

2009-02-26T20:39:00.000-08:00

Tweetdeck is a very powerful twitter client which is written in the rather nifty Adobe Air framework. Unfortunately the air platform does not yet have support for 64-bit Linux. I imagine it will come along once flash 10 for 64 bit Linux comes out of beta. There is a rather long tutorial on how to work around the limitations and run it in 32-bit mode on 64-bit linux right here but that isn't enough to run tweetdeck. You'll also need to pull down a copy of 32 bit libgnome-keyring to allow it to access your keyring.

wget http://ubuntu.interlegis.gov.br/ubuntu/pool/main/g/gnome-keyring/libgnome-keyring0_2.22.2-0ubuntu1_i386.deb
ar x libgnome-keyring0_2.22.2-0ubuntu1_i386.deb data.tar.gz
tar -xzvf data.tar.gz
cp usr/lib/* /usr/lib32/

On a related note it is a shame that the default Air page is so pedestrian. I am filled with no excitement upon visiting it.