2020-06-24

Running Dockerized ELK Stack on Windows

The ELK stack is a combination of 3 tools which make log ingestion and search a snap. I needed to do some debugging on a Kibana dashboard for a client so I tried to stand up a quick docker container on Windows to try out the queries. To do this I used

sudo docker pull sebp/elk
docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

During startup Elasticsearch failed to start with

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elasticsearch.log

Fortunately this is really quick to fix with my WSL based docker set up. I just started WSL directly and issued

> sudo sysctl vm.max_map_count
vm.max_map_count = 65530

This found that indeed I had too few of a max_map_count. Fixing it required issuing

> sudo sysctl -w vm.max_map_count=262144

Elasticsearch and, in fact, all of ELK started up nicely after that.

2020-05-10

Azure App Configuration Expired Token

I moved one of the products I help maintain over to using Azure’s excellent App Configuration tool. But one of the other developers on the team started to have problems with it. When loading the configuration in Azure Functions they got an error: The access token has expired

Read More

2020-04-29

Rename Azure DevOps Pipeline

Simple things should be simple. But they aren’t always. In today’s challenge all I wanted was to rename a build definition in Azure devops. I spent ages fumbling around the edit screen for the build definition trying to find how to rename it.

No way to rename here

There is no way to do it from there. So I googled it and found a raft of great content on how to change the build number but nothing on how to change the build definition name. Eventually I found it. You need to go back up one level by clicking on the breadcrub from that edit screen.

Click on the project name in the breadcrumbs

Next you can rename by clicking on the ... menu

Click on the project name in the breadcrumbs

2020-04-01

Flutter unit testing with native channels

Today I was digging through some unit tests in our flutter project that seemed to be failing on my machine but not necessarily in other places like our build pipeline. The problem was that we had some calls to async methods which were not being awaited properly. I fixed those up and they uncovered a bunch of more serious problems in our tests. We were calling out to validate a phone number with libphonenumber and now we were actually awaiting the call properly we saw this error

[master ≡ +0 ~2 -0 !]> flutter test
00:03 +27 -1: test\unit\providers\create_account_provider_test.dart: Real mobile number - is valid [E]
  MissingPluginException(No implementation found for method isValidPhoneNumber on channel codeheadlabs.com/libphonenumber)
  package:blah/providers/create_account_provider.dart 101:9  CreateAccountProvider.setMobileNumber

00:03 +28 -2: test\unit\providers\create_account_provider_test.dart: Valid state if properties are valid [E]
  MissingPluginException(No implementation found for method isValidPhoneNumber on channel codeheadlabs.com/libphonenumber)
  package:blah/providers/create_account_provider.dart 101:9  CreateAccountProvider.setMobileNumber

As it turns out libphonenumber is actually a native implementation wrapped up with flutter. To communicate with this native code isn’t possible in a test environment so it needs to be mocked. This can be done by mocking the channel.

In the setUp() for the unit tests I added a call to setMockMethodCallHandler like so

const _channel = const MethodChannel('codeheadlabs.com/libphonenumber');
setUp(() async {
_channel.setMockMethodCallHandler((MethodCall methodCall) async {
    return true;
    });  
});

tearDown((){
_channel.setMockMethodCallHandler(null);
});

With this call in place I was able to run the test without issue.

2020-03-31

Solve WebForms Errors with PreCompilation

I have a webforms application that I help maintain. Today I made some change and managed to break one of the pages on the site. The error was unbelievably unhelpful.

Wut? 500 error with no useful details

In older versions of ASP.NET it is nearly impossible to diagnose these sorts of errors. Was it something with the web.config? Did I mess up the dependency injection? I messed about a bit and found that if I deleted everything out of the .aspx file things worked. So it was the view. But what?

Read More

2020-01-15

Flutter Widget Testing

I’m sure I don’t have to tell you how important automated tests are to the software development process. Even if they didn’t have other advantages I’d write tests just so I could have that sense of security when changing things in an application. Most every framework and language has some sort of testing ability built into it and Flutter is no different. However the documentation for widget testing in flutter is pretty poor. This post should get you over some of the gotchas I faced.

Read More

2020-01-10

Allow hosted agents through firewall

I have an on-premise (well in a third party data center but close enough) database which I’d like to update via a build in a hosted agent on Azure. We’ve done this before in Jenkins by just allowing a specific IP address through the firewall. However we’re in the process of moving to DevOps for this build. Unfortunately, the hosted build agents don’t have entirely predictable IP addresses. Every week Microsoft publishes a list of all the IP addresses in Azure. It is a huge json document and for our region (Central Canada) there are about 40 IP addresses ranges the build agent could be in. We want an automated way to update our firewall rules based on this list.

To do so we make use of the Azure Powershell extensions. The commandlet Get-AzNetworkServiceTag is an API based way to get the IP ranges. You can then pass that directly into the firewall rules like so

$addrs = ((Get-AzNetworkServiceTag -Location canadacentral).Values|Where-Object { $_.Name -eq "AzureCloud.canadacentral" }).Properties.AddressPrefixes
Set-NetFirewallRule -DisplayName "Allow SQL 1433 Inbound" -RemoteAddress $addrs

Running this once a week lets us keep the firewall up to date with the hosted agent ranges.

2019-12-30

UNION vs. UNION ALL in SQL Server

I really dislike database queries which are slow for no apparent reason. I ran into one of those today. It queries over a few thousands of well indexed rows and returned a handful, perhaps 3, records. Time to do this? 33 seconds. Well that’s no good for anybody. Digging into the query I found that it actually used a UNION to join 3 sets of similar data together. I go by the rule of thumb that SQL operations which treat data as sets and do things with that in mind are efficient. I’m not sure where I read that but it has stuck with me over the years. What it suggests is that you should avoid doing things like looping over rows or calling functions on masses of data.

As it turns out there are actually two different UNION operators in SQL Server: UNION and UNION ALL. They differ in how they handle duplicate entries. UNION will check each entry to ensure that it exists in the output only one time.

Read More

2019-12-05

Machine Learning for Boring Applications

C# Advent

This post is one among many which is part of 2019’s C# Advent. There are a ton of really great posts this year and some new bloggers to discover. I’d strongly encourage you to check it out.


If you read at all about the myriad of applications for machine learning you’ll find that there are a lot of people out there building really cool stuff. Cars which drive themselves. Things which write tweets based on an AI’s interpretation of thousands of tweets about venture capitalists. Unfortunately I, like a lot of developers, earn our bread and butter by writing what is mostly boring forms over data applications. This is a space in which there is 0 application for machine learning. Or is there?

Read More

2019-10-12

Bulk Load and Merge Pattern

The more years you spend programming the more you run into situations you’ve run into before. Situations you now know, instinctively, how to address. I suppose this is “experience” and is what I’m paid the medium dollars for. One such problem I’ve solved at least a dozen times over the years is updating a bunch of data in a database from an external source. This, as it turns out, can be a great source of poor performance if you don’t know how to address it. Let’s dig into my approach.

Read More