Blog

A catalogue of my discoveries in software development and related subjects, that I think might be of use or interest to everyone else, or to me when I forget what I did!

Batching Async Calls

April 25, 2018

To reduce the payload size of individual calls when loading up resources by ID, sometime you want to send multiple async requests in smaller batches. If we don't need to worry about local/remote resources (i.e. don't need intelligent partitioning or resource friendly approach), the easiest way is to fire off a load of tasks which consume a small batch from the superset. Here is a simple re-usable implementation:
public class BatchContentRequestor<TId, TValue>
{
    private readonly int _batchSize;
    private readonly Func<IEnumerable<TId>, Task<IEnumerable<TValue>>> _getContentAsyncFunc;

    public BatchContentRequestor(int batchSize, Func<IEnumerable<TId>, Task<IEnumerable<TValue>>> getContentAsyncFunc)
    {
        if (batchSize <= 0)
        {
            throw new ArgumentOutOfRangeException(nameof(batchSize), "Batch size must be a positive integer value.");
        }

        _batchSize = batchSize;
        _getContentAsyncFunc = getContentAsyncFunc ?? throw new ArgumentNullException(nameof(getContentAsyncFunc));
    }

    public async Task<IEnumerable<TValue>> GetContentBatchedAsync(IEnumerable<TId> allContentIds)
    {
        var allContentIdsList = allContentIds?.ToList();

        if (allContentIdsList == null || !allContentIdsList .Any())
        {
            return await _getContentAsyncFunc(allContentIdsList );
        }

        var allContentValues = new List<TValue>();

        var getBatchTasks = new List<Task<IEnumerable<TValue>>>();
        for (var batchStart = 0;
            batchStart < allContentIdsList.Count;
            batchStart += _batchSize)
        {
            var batchIds = allContentIdsList
                .Skip(batchStart)
                .Take(_batchSize)
                .ToList();

            getBatchTasks.Add(_getContentAsyncFunc(batchIds));
        }

        await Task.WhenAll(getBatchTasks).ConfigureAwait(false);

        foreach (var completedBatch in getBatchTasks)
        {
            allContentValues.AddRange(await completedBatch.ConfigureAwait(false));
        }

        return allContentValues;
    }
}
You can call it with your superset and it will automatically hit your callback function with the batches of IDs, will collect the results and return the superset of values. If the calling code passes a null or empty value this will still be passed to your callback for handling, making this a transparent proxy for the calling code. e.g.
var items = await new BatchContentRequestor<int, Item>(10, GetItemsByIdAsync).GetContentBatchedAsync(allItemIds).ConfigureAwait(false);
Permalink: Batching Async Calls

Fast and Easy GetHashCode

April 17, 2018

Often when you create a class you need to override the the GetHashCode method, to easily compare instances using a custom Equals operator or to use the class as a key in dictionaries etc. I've seen various ways of doing this, but they are usually either wrong, slow, complex or difficult to remember (things like string concatenation, bit shifting, xor, using prime numbers etc.) As ever with programming, if there is a complex problem to solve then somebody has probably already solved it (and I don't mean copy/pasting GetHashCode algorithms from stack overflow!). One place this has already been done within the .NET framework is anonymous types. The algorithm is fast, we don't need to understand it's complexity and given that it in the framework we can assume it's well tested. So my new favourite way of implementing GetHashCode is to simply project the fields into an anonymous type!
public class Example
{
  private string someField1;
  private int someField2;

  public override int GetHashCode()
  {
      return new { someField1, someField2 }.GetHashCode();
  }
}
Permalink: Fast and Easy GetHashCode

Bio - Pragmatech Software Solutions

January 01, 2018

From the beginning of 2018 I have taken the leap into self-employment. I am now working as a consultant/contract software developer through Pragmatech Software Solutions. Primarily I expect this will involve applying more of my technical expertise in coding and software development which is what I really enjoy and where I excel. I also hope to be able to help clients in any way I can such as providing input on solution architecture, software design, performance optimization, Azure cloud, VSTS, CI/CD, Agile or any other areas where I have experience!
Permalink: Bio - Pragmatech Software Solutions

Adding HATEOAS Links to Web API Resources

December 01, 2017

I was recently working with a React developer on a data driven single page application using an ASP.NET WebAPI backend. I wanted the API to be RESTful to simplify the design, since React can provide client side state and in fact is driven by application state and the API was only there to provide data and persistence. We decided to use the HATEOAS convention, of simply adding a "links" section to each of the objects, in order to inform the client of the available associated resource URIs. Based loosely on the HAL standards we agreed that we simply wanted to add some basic links within the normal JSON response as opposed to negotiating new media types etc. For this very basic requirement then, comes a very simple solution. I created a base class in my API Models folder which would support links:
public class LinkModel
{
    public LinkModel()
    {
    }

    public LinkModel(string href, string method, string rel)
        : this()
    {
        this.Href = href;
        this.Method = method;
        this.Rel = rel;
    }

    public string Method { get; set; }
    public string Rel { get; set; }
    public string Href { get; set; }
}
And then a generic class which would contain the payload object, or "content" and the "links":
public class ContentWithLinks<T>
{
    public ContentWithLinks()
    {
    }

    public ContentWithLinks(T content, IEnumerable<LinkModel> links)
        : this()
    {
        this.Content = content;
        this.Links = links;
    }

    public T Content { get; set; }
    public IEnumerable<LinkModel> Links { get; set; }
}
Each of my actions that wanted to use this strategy would return this type with the calculated links:
[HttpGet]
[Route]
public IEnumerable<ContentWithLinks<DataSetModel>> GetAllDataSets()
{
    return new ContentWithLinks<DataSetModel>[]
    {
        new ContentWithLinks<DataSetModel>(new DataSetModel("Example"), this.GetLinks("Example"))
    };
}

private IEnumerable<LinkModel> GetLinks(string dataSetName)
{
    return new LinkModel[]
    {
        new LinkModel(this.Url.Link(nameof(this.GetAnalysisSchemaForDataSet), new { dataSetName = dataSetName }), "GET", "Schema"),
        new LinkModel(this.Url.Link(nameof(FiltersController.GetAllFilters), new { dataSetName = dataSetName }), "GET", "Filters"),
        new LinkModel(this.Url.Link(nameof(FilesController.GetAllFiles), new { dataSetName = dataSetName }), "GET", "Files")
    };
}
Using the "Url.Link" syntax ensure the route is calculated by ASP.NET. In fact, I went ahead and used the "nameof" operator too, to ensure any code refactorings of these methods would be taken in account automatically. In the client side the various links were then bound to the different actions a user could perform, which would trigger a call to the next URI in the workflow, generating more links and actions for the React components to bind.
Permalink: Adding HATEOAS Links to Web API Resources

Object Oriented Azure Search Client

June 01, 2017

There are a few options already for wrapping calls to Elastic Search (such as ElasticSearch.Net and NEST) however, I wanted to create an object oriented strongly type interface for some simple queries - (with a view to also being able to convert to this from a CDSA "where clause"). My particular implementation of Elastic search, Azure Search, can be called using a restful API interface with OData and Lucene syntax. So my idea was then eventually any OO representation of a query can eventually then be boiled down to a URL with a querystring (similar to how CDSA converts OO clauses to SQL). My first step was to create an abstraction around the concept of a "filter" that can be applied to a search:
public abstract class Filter
{
    public abstract string GetQueryString();
}
Since we are most likely going to be dealing with collections of filters, I created an extension to provide easy access to this method:
internal static class FilterExtensions
{
    public static string AsQueryString(this IEnumerable<Filter> filters)
    {
        return string.Join(" and ", filters.Select(f => f.GetQueryString()));
    }
}
The first concrete implementation is always required by Azure Search, it's the API version you intend to work with:
internal sealed class ApiVersionFilter : Filter
{
    public override string GetQueryString()
    {
        return $"api-version=2016-09-01";
    }
}
Most fields you are querying will either be some filterable scalar value like strings and numbers or collection of these values in a collection column. I represented these types of query using two classes:
public class FieldValuesFilter : Filter
{
    private IEnumerable<IFieldValue> selectedValues;
    private string @operator;

    public FieldValuesFilter(string fieldName, IEnumerable<IFieldValue> selectedValues)
        : this(fieldName, "eq", selectedValues)
    {
    }

    public FieldValuesFilter(string fieldName, string @operator, IEnumerable<IFieldValue> selectedValues)
    {
        this.FieldName = fieldName;
        this.selectedValues = selectedValues;
        this.@operator = @operator;
    }

    public string FieldName { get; private set; }

    public override string GetQueryString()
    {
        return "(" + string.Join(" or ", this.selectedValues.Select(v => $"{this.FieldName} {this.@operator} {v.GetFormattedValue()}")) + ")";
    }
}

public class ArrayValueFilter : Filter
{
    private IEnumerable<IFieldValue> selectedValues;
    private string @operator;

    public ArrayValueFilter(string fieldName, IEnumerable<IFieldValue> selectedValues)
        : this(fieldName, "any", selectedValues)
    {
    }

    public ArrayValueFilter(string fieldName, string @operator, IEnumerable<IFieldValue> selectedValues)
    {
        this.FieldName = fieldName;
        this.selectedValues = selectedValues;
        this.@operator = @operator;
    }

    public string FieldName { get; private set; }

    public override string GetQueryString()
    {
        return "(" + string.Join(" or ", this.selectedValues.Select(v => $"{this.FieldName}/{this.@operator}(t: t eq {v.GetFormattedValue()})")) + ")";
    }
}
You will notice that to abstract the field type I use an interface for IFieldValue for passing in the filter values, this is because depending on whether the data type is a string or a number the formatting will change. The interface and the two implementing classes are below:
public interface IFieldValue
{
    object GetValue();
    string GetFormattedValue();
}

public class StringFieldValue : IFieldValue
{
    private string value;

    public StringFieldValue(string value)
    {
        this.value = value;
    }

    public string GetFormattedValue()
    {
        return $"'{this.value}'";
    }

    public object GetValue()
    {
        return this.value;
    }
}

public class IntegerFieldValue : IFieldValue
{
    private int value;

    public IntegerFieldValue(int value)
    {
        this.value = value;
    }

    public string GetFormattedValue()
    {
        return this.value.ToString();
    }

    public object GetValue()
    {
        return this.value;
    }
}
You can create other filters, such as a "well known text" or "proximity" filter to query spatial data:
public class WktFilter : Filter
{
    private string wkt;

    public WktFilter(string wkt)
    {
        this.Wkt = wkt;
    }

    public string Wkt
    {
        get
        {
            return this.wkt;
        }

        set
        {
            this.wkt = value;
        }
    }

    public override string GetQueryString()
    {
        return $"geo.intersects(location, geography'{this.Wkt}')";
    }
}

public class ProximityFilter : Filter
{
    private double latitude;
    private double longitude;
    private double radiusInMeters;

    public ProximityFilter(double latitude, double longitude, double radiusInMeters)
    {
        this.latitude = latitude;
        this.longitude = longitude;
        this.radiusInMeters = radiusInMeters;
    }

    public override string GetQueryString()
    {
        // Azure Search works with KM not M, so div by 1000
        return $"geo.distance(location, geography'POINT({this.longitude} {this.latitude})') le {this.radiusInMeters / 1000d}";
    }
}
Now that we have the ability to create filters using C# classes and for more derivatives to be added if you more more strong typing, then we need a way of converting this to an Azure Search query. For simplicity I again started with a base class representing a given service which encapsulates the ability to convert a set of filters into a query string or post body and holds the config and endpoint for the index:
public abstract class SearchServiceBase
{
    private readonly AzureServiceConfig config;

    public SearchServiceBase(AzureServiceConfig config)
    {
        this.config = config;
    }

    public abstract string Api { get; }

    public AzureServiceConfig Config
    {
        get
        {
            return this.config;
        }
    }

    protected string GetQueryString(string name, IEnumerable<Filter> filters, int? top, int? skip)
    {
        // parse the input parameters
        StringBuilder requestParameters = new StringBuilder();

        // name param
        if (!string.IsNullOrEmpty(name))
        {
            requestParameters.Append($"search={name}&");
        }

        // add API version by default
        requestParameters.Append(new ApiVersionFilter().GetQueryString() + "&");

        if (top.HasValue)
        {
            requestParameters.Append($"$top={top.Value.ToString()}&");
        }

        if (skip.HasValue)
        {
            requestParameters.Append($"$skip={skip.Value.ToString()}&");
        }

        if (skip.HasValue && skip.Value == 0)
        {
            requestParameters.Append($"$count=true&");
        }

        // filters could be none, one or many
        if (filters != null && filters.Any())
        {
            requestParameters.Append($"$filter={filters.AsQueryString()}&");
        }

        // get the resource
        return requestParameters.Length > 0 ? "?" + requestParameters.ToString().TrimEnd('&') : string.Empty;
    }

    protected string GetPostQueryString()
    {
        return "?" + new ApiVersionFilter().GetQueryString();
    }

    protected string GetPostBody(string name, IEnumerable<Filter> filters, int? top, int? skip)
    {
        // parse the input parameters
        StringWriter sw = new StringWriter();
        JsonTextWriter writer = new JsonTextWriter(sw);

        // {
        writer.WriteStartObject();

        if (skip.HasValue && skip.Value == 0)
        {
            writer.WritePropertyName("count");
            writer.WriteValue("true");
        }

        if (!string.IsNullOrEmpty(name))
        {
            writer.WritePropertyName("search");
            writer.WriteValue(name);

            writer.WritePropertyName("searchMode");
            writer.WriteValue("all");
        }

        if (top.HasValue)
        {
            writer.WritePropertyName("top");
            writer.WriteValue(top.Value.ToString());
        }

        if (skip.HasValue)
        {
            writer.WritePropertyName("skip");
            writer.WriteValue(skip.Value.ToString());
        }

        // filters could be none, one or many
        if (filters != null && filters.Any())
        {
            writer.WritePropertyName("filter");
            writer.WriteValue(filters.AsQueryString());     // querystring is same format as POST property value
        }

        // }
        writer.WriteEndObject();

        return sw.ToString();
    }
}
So an example subclass which consumes this functionality:
public class ExampleSearchService : SearchServiceBase
{
    private readonly string indexName;

    public ExampleSearchService(AzureServiceConfig config, string indexName)
        : base(config)
    {
        this.indexName = indexName;
    }

    public override string Api
    {
        get
        {
            return $"indexes/{this.indexName}/docs/search";
        }
    }

    public async Task<IEnumerable<ExampleSearchResult>> GetResultsAsync(string name, IEnumerable<Filter> filters, int maxResultCount, bool usePost = false)
    {
        using (var webApiClient = new AzureElasticSearchClient(this.Config))
        {
            webApiClient.Timeout = new TimeSpan(0, 20, 0);

            var results = new List<ExampleSearchResult>();

            int pageSize = 1000;
            int pagesToRetrieve = 1;
            int pagesRetrieved = 0;

            // at least one page, but may be more..
            while (pagesRetrieved < pagesToRetrieve)
            {
                HttpResponseMessage result = null;
                if (usePost)
                {
                    string requestUrl = $"{this.Api}{this.GetPostQueryString()}";
                    string requestBody = this.GetPostBody(name, filters, pageSize, pagesRetrieved * pageSize);

                    HttpContent content = new StringContent(requestBody, System.Text.Encoding.UTF8, "application/json");

                    // call the API and increment received count
                    result = await webApiClient.PostAsync(requestUrl, content).ConfigureAwait(false);
                }
                else
                {
                    // build the query url from the filters
                    string requestUrl = $"{this.Api}{this.GetQueryString(name, filters, pageSize, pagesRetreived * pageSize)}";

                    result = await webApiClient.GetAsync(requestUrl).ConfigureAwait(false);
                }
                
                pagesRetrieved++;

                // if it was successful, we can process it
                if (result.IsSuccessStatusCode)
                {
                    // parse the JSON response
                    var jResponse = JObject.Parse(await result.Content.ReadAsStringAsync().ConfigureAwait(false));

                    // if this was the first page, we can parse the item count
                    if (pagesRetrieved == 1)
                    {
                        int count = (int)jResponse.GetValue("@odata.count");

                        // check against max result count and throw exception if over that
                        if (maxResultCount > 0)
                        {
                            if (count > maxResultCount)
                            {
                                throw new InvalidOperationException($"Search result count of {count} was greater than the maximum of {maxResultCount}");
                            }
                        }

                        pagesToRetrieve = (int)Math.Ceiling((double)count / (double)pageSize);
                    }

                    // now get the value, which is the array of results
                    JArray jsonResults = (JArray)jResponse.GetValue("value");

                    // loop over the JSON array and deserialise each result obejct
                    foreach (var resultData in jsonResults)
                    {
                        var result = resultData.ToObject<ExampleSearchResult>();

                        results.Add(result);
                    }
                }
            }

            return results;
        }
    }
}
You will notice I created a wrapper for HttpClient, this is simply to encapsulate adding the base address and API key:
internal class AzureElasticSearchClient : HttpClient
    {
        public AzureElasticSearchClient(AzureServiceConfig config)
        {
            this.BaseAddress = new Uri($"https://{config.AzureSearchServiceName}.search.windows.net/");
            this.DefaultRequestHeaders.Add("api-key", config.AzureSearchApiKey);
        }
    }
Here are some examples of how to create and add filters then call the example search service:
[TestFixture]
public class TestExampleSearchService
{
    private ExampleSearchService sut = new ExampleSearchService();

    [Test]
    public async Task TestGetResultsInWktWithAdditionalFilter_Elastic()
    {
        // arrange
        var testPolygonWkt = "POLYGON ((-1.6259765625 53.74404116282134, -1.6005706787109375 53.76089000834015, -1.5696716308593748 53.73876182109416, -1.6036605834960935 53.72799803200196, -1.6259765625 53.74404116282134))";
        var polyFilter = new WktFilter(testPolygonWkt);
        var examplePropertyFilter = new FieldValuesFilter("SomeIntProperty", new IntegerFieldValue[] { new IntegerFieldValue(1) });
        
        // act
        var startTime = DateTime.Now;
        var result = await this.sut.GetResultsAsync(string.Empty, new Filters.Filter[] { polyFilter, examplePropertyFilter });
        Trace.WriteLine($"Time taken: {DateTime.Now.Subtract(startTime).TotalMilliseconds} ms");
        Trace.WriteLine($"# recs {result.Count()}");
        // assert
        Assert.Greater(result.Count(), 0);
    }

    [Test]
    public async Task TestGetResultsInProximityWithName_Elastic()
    {
        // arrange
        var longi = -1.6259765625;
        var lati = 53.74404116282134;
        var dist = 200;
        var proxFilter = new ProximityFilter(lati, longi, dist);

        // act
        var startTime = DateTime.Now;
        var result = await this.sut.GetResultsAsync("example name", new Filters.Filter[] { proxFilter });

        Trace.WriteLine($"# recs {result.Count()}");
        Trace.WriteLine($"Time taken: {DateTime.Now.Subtract(startTime).TotalMilliseconds} ms");

        // assert
        Assert.Greater(result.Count(), 0);
    }

    [Test]
    public async Task TestGetResultsInArrayFilter_Elastic()
    {
        // arrange
        var possibleValues = {"hello", "world"}.Select(s => new StringFieldValue(s));
        var exampleArrayPropertyFilter = new FieldValuesFilter("SomeStringArrayProperty", possibleValues);
        
        // act
        var startTime = DateTime.Now;
        var result = await this.sut.GetResultsAsync(string.Empty, new Filters.Filter[] { exampleArrayPropertyFilter });
        Trace.WriteLine($"Time taken: {DateTime.Now.Subtract(startTime).TotalMilliseconds} ms");
        Trace.WriteLine($"# recs {result.Count()}");
        // assert
        Assert.Greater(result.Count(), 0);
    }
}
With this structure all now in place, I wanted to plug this into an existing CDSA application which was currently using SQL for performing queries. I therefore needed a way to convert from traditional "CDSA WhereClause" objects to my new Azure Search filter structure. I created a basic implementation, which isn't 100% compatible with all clauses yet, but for most use cases it works fine:
internal class WhereClauseParser
{
    Regex isNumeric = new Regex("^\\d+$");

    public IEnumerable<FieldValuesFilter> ParseWhereClause(WhereClause clause)
    {
        if (clause == null || clause.RecursiveClauseList.Count == 0)
        {
            return null;
        }

        return this.ParseWhereClauseWithSubgroups(clause);
    }

    private IEnumerable<FieldValuesFilter> ParseWhereClauseWithSubgroups(WhereClause clause)
    {
        if (clause.ConjunctionOperator == ConjunctionOperator.And)
        {
            var myFilters = new List<FieldValuesFilter>();
            foreach (var whereClauseElement in clause.ClauseList)
            {
                myFilters.Add(this.ParseWhereClauseElement(whereClauseElement));
            }

            if (clause.SubGroups.Count > 0)
            {
                foreach (var subClause in clause.SubGroups)
                {
                    myFilters.AddRange(this.ParseWhereClauseWithSubgroups(subClause));
                }
            }

            return myFilters;
        }
        else
        {
            throw new NotImplementedException("Elastic search clause parser currently only supports the 'AND' conjunction.");
        }
    }

    private FieldValuesFilter ParseWhereClauseElement(WhereClauseElement whereClauseElement)
    {
        // start with the defaults
        var fieldName = whereClauseElement.CompareItem;
        var values = new object[] { whereClauseElement.CompareValue };

        var @operator = string.Empty;

        // don't need qualified paths, remove the dots
        fieldName = fieldName.Replace(".", "");

        switch (whereClauseElement.Operator)
        {
            case Operator.Equals:
            case Operator.Like: // todo: this should be a wildcard search, not an exact equality
                @operator = "eq";
                break;
            case Operator.NotEqual:
                @operator = "ne";
                break;
            case Operator.GreaterThan:
                @operator = "gt";
                break;
            case Operator.LessThan:
                @operator = "lt";
                break;
            case Operator.GreaterThanEqualTo:
                @operator = "ge";
                break;
            case Operator.LessThanEqualTo:
                @operator = "le";
                break;
            case Operator.In:
                @operator = "eq";
                values = values[0].ToString().Split(',');

                // if it was an array of numbers stored as CSV, then unpack the numbers as ints.
                if (values.All(v => this.isNumeric.IsMatch(v.ToString())))
                {
                    values = values.Select(v => (object) int.Parse(v.ToString())).ToArray();
                }

                break;
            default:
                throw new NotImplementedException("Elastic search clause parser currently does not support this operator");
        }

        return new FieldValuesFilter(fieldName, @operator, values.Select(v => this.FieldValueFactory(v)));

    }

    private IFieldValue FieldValueFactory(object v)
    {
        // a dirty type check, lets call it a "factory"
        if (v is int)
        {
            return new IntegerFieldValue((int)v);
        }
        else
        {
            return new StringFieldValue(v.ToString());
        }
    }
}
Permalink: Object Oriented Azure Search Client

Async all the way conventions

May 23, 2017

Having already been caught out by the ASP.NET deadlock issue when using async code synchronously I did a lot of reading on the subject, from various sources and ended up compiling a list of rules/conventions to follow when dealing with async code. This is by no means a gospel list and I don't elaborate on why each guideline exists, but it's just a concise set of reminders of things to consider when writing async code, since the compiler will not pick up warnings for a lot of these things. The list is shared below:
  • Don't use .Result or .Wait() - use async/await (unless absolutely necessary - e.g. destructor, domain value factory - see final point)
  • Don't used async void
  • Always use "ConfigureAwait(false)" inside async library code (unless you need the context)
  • Don't "await Task.FromResult" - remove the async and just return the Task directly (unless async null or type casting).
  • When returning Task, don't return "null" - you must return Task.FromResult<T>(null);
  • Make sure any function returning Task is suffixed as "Async" to guide the caller
  • Don't use parallel for each, use tasks and Task.WhenAll (unless need custom partitioning)
  • Constructors and destructors cannot be marked async, therefore should not call async methods (no compiler warning)
  • For ctors - consider making it a factory with CreateInstanceAsync
  • For destructors - try to design against it, or fall back to Task.Run().GetAwaiter().GetResult() syntax (see final point)
  • When using .Wait() and .Result syntax, ensure it runs on a background thread with no context e.g. ... Task.Run(() => theMainAsync()).GetAwaiter().GetResult();
General approach: In general, start off by making the interface return Task<> .. update the classes to return Task, then look to see if anything inside the function is truly async. If yes - mark the function async and use await (unless you can pass through).. if No, then just return a Task.FromResult to satisfy the interface.. Don't use async/await inside of a parallel loop. You can block here since its already async and on another thread, or change to tasks. If you *need* to convert to synchronous (such as for passing a value factory) then use Task.Run to encapsulate the blocking call, so that its on another thread with a different context and use .GetAwaiter().GetResult() to unwrap exceptions.
Permalink: Async all the way conventions

Spatial Queries using Entity Framework without Microsoft.SqlServer.Types.dll

May 18, 2017

You can use entity framework to make spatial queries against your Microsoft SQL Server database, which is great when the majority of the ORM code is using EF. What isn't great is that for this to work you have to add some native DLLs into your project, namely the "Microsoft.SqlServer.Types.dll" - this comes in 32 bit and 64 bit variants. Seemingly this also needs to appear in any top level application container, which is usually a different project to the one housing your ORM code. In the end these DLLs seem to get everywhere, except the deployed environment and cause a lot of problems at runtime if they're missing. For my purposes though, I didn't need full in-memory spatial support, I simply wanted to add spatial predicates to my where clauses and return normal "non-spatial" objects back from SQL Server. This is really easy to do in isolation, just write the SQL text and execute it directly using EF. However, most of the time you actually want to combine this with another set of undeterminate predicates which have been applied to the IQueryable in other parts of the code. To solve this issue, I wrote a helper which allows me tag on my spatial query as the last execution step of the IQueryable (e.g. replacing the ToList()) as follows:
internal static class EFSpatialHelper
{
        public static Task<List<T>> QueryByProximityAsync<T>(DbContext context, IQueryable<T> query, Proximity proximity)
        {
            string sqlPointGeogPreamble = $@"
Declare @GeogPoint as geography
Declare @GeogShape as geography

SET @GeogPoint = geography::STGeomFromText('POINT (' + Cast(@longitude as varchar) + ' ' + Cast(@latitude as varchar) + ')', 4326)
SET @GeogShape = @GeogPoint.STBuffer(@buffer)

";
            string sqlGeogPredicate = "@GeogShape.STIntersects(Point) = 1";

            List<SqlParameter> sqlParams = new List<SqlParameter>();
            sqlParams.Add(new SqlParameter("@longitude", proximity.Longitude));
            sqlParams.Add(new SqlParameter("@latitude", proximity.Latitude));
            sqlParams.Add(new SqlParameter("@buffer", proximity.RadiusInMeters));

            return InjectAndExecuteSql(context, query, sqlPointGeogPreamble, sqlGeogPredicate, sqlParams.ToArray());
        }

        internal static Task<List<T>> GetByWellKnownTextAsync<T>(DbContext context, IQueryable<T> query, string wellKnownText)
        {
            string sqlWktGeogPreamble = @"
Declare @GeogPolygon as geography
Declare @GeomPolygon as geometry

Set @GeomPolygon = Geometry::STGeomFromText(@WKT, 4326)
Set @GeogPolygon = Geography::STGeomFromWKB(@GeomPolygon.MakeValid().STUnion(@GeomPolygon.STStartPoint()).STAsBinary(), 4326)

";
            string sqlGeogPredicate = "Point.STIntersects(@GeogPolygon)=1";

            return InjectAndExecuteSql(context, query, sqlWktGeogPreamble, sqlGeogPredicate, new SqlParameter("@WKT", wellKnownText));
        }

        private static Task<List<T>> InjectAndExecuteSql<T>(DbContext context, IQueryable<T> query, string spatialPreamble, string spatialPredicate, params SqlParameter[] spatialParams)
        {
            ObjectQuery<T> objectQuery = GetQueryFromQueryable(query);
            string sqlQryText = $"{spatialPreamble}{objectQuery.ToTraceString()}";

            // the query text in here is quite simple, we know it doesnt have an order by or a group by, so just stick predicates on the end.
            if (!sqlQryText.ToUpper().Contains("WHERE"))
            {
                sqlQryText = $"{sqlQryText} WHERE {spatialPredicate}";
            }
            else
            {
                sqlQryText = $"{Regex.Replace(sqlQryText, "WHERE", "WHERE (", RegexOptions.IgnoreCase)}) AND {spatialPredicate}";
            }

            List<SqlParameter> sqlParams = new List<SqlParameter>();
            if (spatialParams != null)
            {
                sqlParams.AddRange(spatialParams);
            }

            foreach (var p in objectQuery.Parameters)
            {
                sqlParams.Add(new SqlParameter(p.Name, p.Value));
            }

            return context.Database.SqlQuery<T>(
                sqlQryText,
                sqlParams.ToArray()).ToListAsync();
        }

        private static ObjectQuery<T> GetQueryFromQueryable<T>(IQueryable<T> query)
        {
            var internalQueryField = query.GetType().GetFields(System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance).Where(f => f.Name.Equals("_internalQuery")).FirstOrDefault();
            var internalQuery = internalQueryField.GetValue(query);
            var objectQueryField = internalQuery.GetType().GetFields(System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance).Where(f => f.Name.Equals("_objectQuery")).FirstOrDefault();
            return objectQueryField.GetValue(internalQuery) as ObjectQuery<T>;
        }
}
Here is an example of the modified logic for executing a query:
if (wellKnownText != string.Empty)
{
    return await this.GetItemsInWellKnownTextAsync(context, fullQuery, wellKnownText).ConfigureAwait(false);
}
else if (proximity != null)
{
    return await this.GetItemsInProximityAsync(context, fullQuery, proximity).ConfigureAwait(false);
}
else
{
    return await fullQuery.ToListAsync().ConfigureAwait(false);
}
Permalink: Spatial Queries using Entity Framework without Microsoft.SqlServer.Types.dll

Dealing With Map Related Data on the Server in C#

February 07, 2017

It is possible to create interactive graphical maps in many front end frameworks. In the past I have written mapping applications using Virtual Earth (JavaScript), Bing Maps (Silverlight), Bing Maps (JavaScript) and MapBox (JavaScript). The client side implementations vary, but there are common themes especially from a server side point of view. There are a lot of concepts that be represented in re-usable C# code. For this reason, I have created a class library of ".NET Mapping" objects and helpers, which represent the mapping concepts such as geographical points, bounding boxes, proximities, distance calculation and zoom level calculation. You can view or download the library on GitHub.
Permalink: Dealing With Map Related Data on the Server in C#

Azure AAD Login Redirect Loop

January 11, 2017

Sometimes when using Microsoft Azure AAD for authentication the website goes into a redirect permaloop. So far, this has been caused by two different things for me - so I will collate the causes and solutions below:
  1. HTTP vs HTTPS - Since the Microsoft login server is running on HTTPS you must also run your website on HTTPS, since the login cookie will be issued using the secure flag, which prevents it from being read using HTTP. There are various ways to force SSL on your website, from global filters to URL re-writing, but in general make sure to setup your site URLs using HTTPS in the Azure portal and to browse to HTTPS.
  2. OWIN Middleware Cookie issue - There is a known issue with the OWIN middleware which means that it's cookies are sometimes lost when issued in conjunction with some other cookie, e.g. the session cookie. The easiest solution to this one I have found is to install a NuGet package called "Kentor.OwinCookieSaver" and then add the following line to startup.auth.cs
     public void ConfigureAuth(IAppBuilder app)
            {
                app.SetDefaultSignInAsAuthenticationType(CookieAuthenticationDefaults.AuthenticationType);
                app.UseKentorOwinCookieSaver(); // <-- ADD THIS LINE
    
Permalink: Azure AAD Login Redirect Loop

C# HttpClient for Calling Azure based WebAPI with OAuth Client Credentials

January 06, 2017

The below C# code creates a sub-class of the System.Net.Http.HttpClient that deals with adding the "Authorization" header to subsequent calls, using OAuth client credentials grant.
using System;
using System.Collections.Generic;
using System.Net.Http;
using Newtonsoft.Json.Linq;

namespace SomeProject.WebAPI.Client
{
    internal sealed class SomeWebApiHttpClient : HttpClient
    {
        private static string authToken;
        private static DateTime authTokenTime;

        private SomeWebApiHttpClient(ISomeWebApiClientConfig config)
        {
            this.BaseAddress = new Uri(config.SomeWebApiBaseAddress);
        }
        
        public static async Task<SomeWebApiHttpClient> CreateInstanceAsync(ISomeWebApiClientConfig config)
        {
            var instance = new SomeWebApiHttpClient(config);
            instance.DefaultRequestHeaders.Add("Authorization", await GetAuthAsync(config.SomeWebApiTenantId, config.CallingClientId, config.CallingClientKey));

            return instance;
        }

        private static async Task<string> GetAuthAsync(string tenantId, string callingClientId, string callingClientKey)
        {
            if (string.IsNullOrEmpty(authToken) || DateTime.Now.Subtract(authTokenTime).TotalMinutes > 45)
            {
                using (var authClient = new HttpClient())
                {
                    authClient.BaseAddress = new Uri("https://login.microsoftonline.com/");

                    Dictionary<string, string> bodyContent = new Dictionary<string, string>();
                    bodyContent.Add("grant_type", "client_credentials");
                    bodyContent.Add("client_id", callingClientId);
                    bodyContent.Add("client_secret", callingClientKey);
                    bodyContent.Add("resource", "https://somedomain.onmicrosoft.com/APP-URI-ID");

                    var result = await authClient.PostAsync(
                        $"{tenantId}/oauth2/token",
                        new FormUrlEncodedContent(bodyContent)).ConfigureAwait(false);

                    if (result.IsSuccessStatusCode)
                    {
                        dynamic tokenResult = (dynamic)JValue.Parse(await result.Content.ReadAsStringAsync().ConfigureAwait(false));

                        authToken = $"{tokenResult.token_type} {tokenResult.access_token}";
                        authTokenTime = DateTime.Now;
                    }
                    else
                    {
                        Trace.WriteLine($"Failed to login to API - {await result.Content.ReadAsStringAsync().ConfigureAwait(false)}");
                        
                        return string.Empty;
                    }
                }
            }

            return authToken;
        }
    }
}
In the above code you must correctly set the domain and APP-URI-ID for the WebAPI this client is supposed to be using (or pass them in the config if they are variable). The configuration is passed in via an interface so that the calling code can inject whatever implementation they have for storing the configuration. E.g. In the web project composition root, a class that reads from the web.config using ConfigurationManager. The interface is defined as below:
public interface ISomeWebApiClientConfig
    {
        string SomeWebApiBaseAddress { get; }
        string SomeWebApiTenantId { get; }
        string CallingClientId { get; }
        string CallingClientKey { get; }
    }
These settings can be found in the Azure portal within the active directory that defines the client/server applications. Now the rest of the code base can use this client to call the API's endpoints, without worrying about injecting the auth header. Example usage of the HttpClient to call an example API endpoint:
using System.Collections.Generic;
using System.Diagnostics;
using Newtonsoft.Json.Linq;

namespace SomeProject.WebAPI.Client
{
    public class ExampleDataClient
    {
        private readonly ISomeWebApiClientConfig config;

        public ExampleDataClient(ISomeWebApiClientConfig config)
        {
            this.config = config;
        }

        public async Task<IEnumerable<SomeExampleData>> GetSomeExampleData(string someParam)
        {
            using (var client = await SomeWebApiHttpClient.CreateInstanceAsync(this.config).ConfigureAwait(false))
            {
                var result = await client.GetAsync($"api/SomeExampleData").ConfigureAwait(false);

                if (result.IsSuccessStatusCode)
                {
                    var returnValue = JValue.Parse(await result.Content.ReadAsStringAsync().ConfigureAwait(false));

                    if (returnValue.HasValues)
                    {
			List<SomeExampleData> outputList = new List<SomeExampleData>();
                       foreach (var item in returnValue)
                       {
                           outputList.Add(SomeExampleData.FromJObject(item));
                       }
                       
                       return outputList.AsEnumerable();
                    }
                    else
                    {
                        return Task.FromResult<IEnumerable<SomeExampleData>>(null);
                    }
                }
                else
                {
                    Trace.WriteLine($"HTTP error: {result.RequestMessage.RequestUri} ({result.StatusCode}) - {await result.Content.ReadAsStringAsync().ConfigureAwait(false)}");
                    return Task.FromResult<IEnumerable<SomeExampleData>>(null);
                }
            }
        }
    }
}
Permalink: C# HttpClient for Calling Azure based WebAPI with OAuth Client Credentials