Blog

Useful tidbits related to software development, that I think might be of use or interest to everyone else (or to me when I forget what I did!)

Exposing Kafka from Rancher/K8S VM to Local Machine

March 01, 2019

Following on from my previous post on setting up Rancher/K8S on RancherOS in a VM in Windows for local development, a common task will be setting up container services within the cluster but then accessing those services from your local Windows machine (e.g. while developing in Visual Studio). In a lot of cases this is probably straightforward, either exposing ports directly using a service or using Ingress to route host headers to the correct internal service. However in the case of Kafka it's a bit more complex due to the way in which the brokers address themselves when the initial connection is received and the broker list is sent back. In a nutshell, the default Kafka setup from the Catalog Apps in Rancher binds the brokers to their POD IP, when the broker list is sent to Windows it cannot address these IPs (unless you want to set up some kind of natting). After some Googling and help from the following posts: https://rmoff.net/2018/08/02/kafka-listeners-explained/ https://github.com/helm/charts/issues/6670 I came up with the following instructions: STEP 1 (install Kafka in cluster): Install Kafka from the Rancher catalogue
  1. your-dev-cluster > default > Catalog Apps > Launch
  2. find and select "Kafka"
  3. switch off the "Topics UI Layer 7 Loadbalancer" (near the bottom) - don't need it in dev.
  4. click "Launch"
  5. .. Wait until all the kafka services are running ..
  6. You can now verify that the Landoop UI is running and seeing brokers by visiting the endpoint is has produced, e.g. http://rancherdev.yourdomain:30188 <-- random port, check what it says!!
Kafka is now available in the cluster, but not from Windows. Continue with step 2 --> STEP 2 (expose Kafka externally): Change the Kafka startup command for multiport listening
  1. your-dev-cluster > default > workloads > kafka-kafka
  2. Three dots, click "Edit"
  3. Click "show advanced options"
  4. Under Command > Entrypoint - paste the following:
    sh -exc 'export KAFKA_BROKER_ID=${HOSTNAME##*-} && \export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_IP}:9092,EXT://rancherdev.yourdomain.com:$((9093 + ${KAFKA_BROKER_ID})) && \export KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXT:PLAINTEXT && \export KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT && \exec /etc/confluent/docker/run'
  5. Click "Upgrade"
Add service discovery for the new ports
  1. your-dev-cluster > default > Service Discovery
  2. Click "View/Edit YAML" on kafka-kafka..
  3. Use the following lines for section "spec > ports" (assuming you have 3 instances of Kafka)
    ports:
      - name: broker
        port: 9092
        protocol: TCP
        targetPort: 9092
      - name: broker-ext0
        port: 9093
        protocol: TCP
        targetPort: 9093
      - name: broker-ext1
        port: 9094
        protocol: TCP
        targetPort: 9094
      - name: broker-ext2
        port: 9095
        protocol: TCP
        targetPort: 9095
    
Configure nginx to use TCP ConfigMap
  1. your-dev-cluster > system > workloads > nginx-ingress-controller
  2. Three dots > edit
  3. Environment variables:
  4. "Add from Source" > "Config Map" > "tcp-services"
  5. Click "Upgrade"
Expose the port using Ingress TCP ConfigMap
  1. your-dev-cluster > system > resources > config maps > ns: ingress-nginx > tcp-services
  2. Three dots, click "Edit"
  3. Add the following entries:
            - key = 9093
            - value = kafka/kafka-kafka:9093
            - key = 9094
            - value = kafka/kafka-kafka:9094
            - key = 9095
            - value = kafka/kafka-kafka:9095
    
Reboot the kafka services
  1. your-dev-cluster > default > workloads > tick all and click 'redeploy'
Now from Windows try telnet to rancherdev.yourdomain.com 9093/9094/9095 or even better from WSL bash, install kafkacat and run: kafkacat -b rancherdev.yourdomain.com:9093 -L

Setting up a Kubernetes cluster using Rancher on RancherOS

February 17, 2019

Little cheat sheet for setting up a single node Kubernetes/Rancher on a developer machine using Hyper-V without tying it to the DHCP IP address that was issued at the time of creation. Setup Rancher on RancherOS
  1. Download the RancherOS Hyper-V ISO image from the GitHub repo
  2. Setup a Hyper-V VM with the bootable ISO set as the boot device (with Internet connectivity - I used 4 vCPU, 16GB RAM and 500GB vHDD)
  3. Boot the VM and allow Linux to boot
  4. Type the following command (uses a password to avoid SSH keys):
    sudo ros install -d /dev/sda --append "rancher.password=yourpassword"
    
  5. Reboot and skip the CD boot step (i.e. boot from the hard disk)
  6. Login with "rancher" and "yourpassword" - at this point you may wish to get the IP and switch to another SSH client such as PuTTY and login from there.
  7. Create an SSL certificate for your "rancherdev" domain - from your rancher home directory
    docker run -v $PWD/certs:/certs -e SSL_SUBJECT="rancherdev.yourdomain.com" paulczar/omgwtfssl
    
  8. Optionally, you can now delete this container/image from Docker
  9. Run the following command to start Rancher in a Docker container (with persistent storage and custom SSL certificate)
    docker run -d -v /mnt/docker/mysql:/var/lib/mysql -v $PWD/rancher:/var/lib/rancher -v $PWD/certs/cert.pem:/etc/rancher/ssl/cert.pem -v $PWD/certs/key.pem:/etc/rancher/ssl/key.pem -v $PWD/certs/ca.pem:/etc/rancher/ssl/cacerts.pem --restart=unless-stopped -p 8080:80 -p 8443:443 rancher/rancher
    
  10. In order to internally resolve the custom rancherdev domain in RancherOS, add a loopback record it to the hosts file
    echo "127.0.0.1 rancherdev.yourdomain.com" | sudo tee -a /etc/hosts > /dev/null
    
  11. Rancher should now be running on the VM's public IP (run "ifconfig" to get your VM IP if you don't have it already)
  12. On your host OS (e.g. Windows) add this IP to the hosts file against "rancherdev.yourdomain.com" (c:\windows\system32\drivers\etc\hosts)
  13. Browse to the https://rancherdev.yourdomain.com:8443 in your web browser
  14. Follow the wizard to setup password/servername etc. for Rancher
Create a new Kubernetes cluster using Rancher
  1. In the Rancher browser UI - select to add a new cluster
  2. Choose "Custom" and use all the defaults, no cloud provider, [I disabled recurring etcd snapshots in the advanced options since this is a dev setup] - click Next
  3. In the next screen, choose all the Node Roles (etcd, Control Plane, Worker) - expand Advanced options and set the public and internal address to be 127.0.0.1 to ensure the node can survive an external IP change (or another copy running)
  4. Copy the generated Docker command to the clipboard and press Done - it should look something like this:
    sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.1.6 --server https://rancherdev.yourdomain.com:8443 --token XXX --ca-checksum XXX --node-name my-dev-node --address 127.0.0.1 --internal-address 127.0.0.1 --etcd --controlplane --worker
  5. Paste and run the command in the RancherOS shell
  6. Rancher should then provision the Kubernetes cluster
NB. Any links generated by the Rancher UI to containers you install will use "127.0.0.1" as the URL which is of course wrong from your host OS. You will need to manually enter the URL as rancherdev.yourdomain.com Surving an IP Change If you fire up the VM for the first time on another machine or your DHCP recycles and your external IP changes, you will need to follow these steps to get up and running:
  1. Run the VM as normal in Hyper-V
  2. Login via the Hyper-V console with rancher/yourpassword
  3. Get the IP address of the running RancherOS
    ifconfig
  4. Update your Windows host file (c:\windows\system32\drivers\etc\hosts) with and entry for rancherdev.yourdomain.com pointing to the VM IP
  5. Browse to the rancher URL and give it some time to come back online

StackExchange.Redis Wrapper for JSON Chunking

January 07, 2019

I have used Redis caching with the StackExchange.Redis client in .NET across various projects and each time I find myself solving the same problems. The main problem, aside from abstracting the client and solving a few other issues (see below), is usually that my JSON data is bigger than Redis would like and it starts to perform badly or throws errors because the "qs" is full. I know there are other serialisation formats to try which might save some space, but my preference is to continue with JSON. I have created a GitHub repository called ChunkingRedisClient, which wraps up this boilerplate functionality in a central place. You can also install the current build as a NuGet package. Below is the write-up from the README: ---
# Chunking Redis Client
A library which wraps the StackExchange.Redis client, specifically using JSON serialisation, and adds functionality such as chunked reading/writing and sliding expiration.

The purpose of this library is to create a re-usable library of code (NB. which I need to put into a NuGet package) for wrapping the StackExchange.RedisClient and solving the issues I usually need to solve.

Those being:

* IoC wrappers/abstractions
   - Just take your dependency on "IRedisClient<TKey, TItem>"
   - By default you should configure your DI container to inject the provided RedisClient<TKey, TItem>
   - Since IoC is used throughout you also need to configure:
     ~ IRedisWriter<TKey, Item> -> JsonRedisWriter or ChunkedJsonRedisWriter
     ~ IRedisReader<TKey, Item> -> JsonRedisReader or ChunkedJsonRedisReader
     ~ IRedisWriter<TKey, Item> -> JsonRedisDeleter or ChunkedJsonRedisDeleter
     (note: for one combination of TKey, TItem - ensure the decision to chunk or not is consistent)
     ~ IKeygen<TKey> to an object specific implementation, like GuidKeygen
     ~ For chunking, locking is required:
             IRedisLockFactory -> RedisLockFactory
             To override the default of InMemoryRedisLock, call RedisLockFactory.Use<IRedisLock>() <-- your class here
     
* Strongly typed access to the cache
  - Use any C# object as your TKey and TItem, given that:
      ~ Your TKey is unique by GetHashCode(), or implement your own Keygen
      ~ Your TItem is serialisable by Newtonsoft.Json
      
* Implementing the StackExchange Connection Multiplexer
  - This is handled by the RedisDatabaseFactory
  - Not using the usual "Lazy<ConnectionMulitplexer>" approach, as I want to support one multiplexer per connection string (if your app is dealing with more than 1 cache)
  - The multiplexers are stored in a concurrent dictionary where the connection string is the key
  - The multiplexer begins connecting asynchronously on first use
    
* Sliding expiration of cache keys
  - Pass in the optional timespan to read methods if you want to use sliding expiration
  - This updates the expiry when you read the item, so that keys which are still in use for read purposes live longer
  
* Chunked JSON data
  - This solves a performance issue whereby Redis does not perform well with large payloads.
  - Sometimes you may also have had errors from the server when the queue is full.
  - The default chunk size is 10KB which can be configured in the ChunkedJsonRedisWriter
  - The JSON data is streamed from Newtonsoft into a buffer. Every time the buffer is full it is written to Redis under the main cache key with a suffix of "chunkIndex"
  - The main cache key is then written to contain the count of chunks, which is used by the reader and deleter.
  
* Generating keys for objects
  - I don't like using bytes for keys as they are not human readable, so I like to generate unique strings
  - There is no none-intrusive way of providing a type agnostic generic keygen, therefore you must write your own. If you write something for a CLR type, considering contributing it to the project!
  - Since we know Guids are unique, I have demonstrated the ability to create custom keygens.


The code can be extended to support other serialisation types (TODO), distributed locks (TODO), different ways of generating keys or whatever you need it to do.

Equality for Value Objects and Entities in DDD

December 12, 2018

In DDD most objects can be categorised as either value types or entities. Value types being objects where there is not one identifier, but simply a collection of related properties; entities being where the ID of the object is the ultimate identifier and all other properties are attributes of this entity. For me, the desired functionality in terms of equality comparisons is that entities are "Equal" when they have the same ID.. Value types are equal when they have matching "composite key" - i.e. all the properties of the object. To model this I have created a base class for enforcing value equality and a more specialised base for an entity:
public abstract class ValueEqualityObject<T> : IEquatable<T>
{
    public sealed override bool Equals(object obj)
    {
        if (obj is null)
            return false;

        if (ReferenceEquals(obj, this))
            return true;

        if (GetType() != obj.GetType())
            return false;

        return Equals((T)obj);
    }

    public sealed override int GetHashCode()
    {
        return TupleBasedHashCode();
    }

    public abstract bool Equals(T other);

    protected abstract int TupleBasedHashCode();
}

public abstract class Entity<TId> : ValueEqualityObject<Entity<TId>>
    {
        protected Entity(TId id)
        {
            Id = id;
        }

        public TId Id { get; }

        protected override int TupleBasedHashCode()
        {
            return (Id).GetHashCode();
        }

        public override bool Equals(Entity<TId> other)
        {
            return other != null 
                && other.Id.Equals(Id);
        }
    }
Now for each domain type I can choose which base to inherit from. For entities I simply define the ID, for value types I am prompted to define a TupleBasedHashCode and Equals method. The TupleBasedHashCode is a reminder to myself on a my preferred strategy for GetHashCode which is use the built-in Tuple implementation :)

Lodash Memoize Wrapper for Caching Multiple Args

July 31, 2018

The Lodash memoize function caches a function call and can vary the cache items based on the parameters. By default the "cache key" is the first parameter, but often it's useful to vary by all parameters. Here is a simple wrapper that will use a custom resolver to always cache based on all args passed to the function. With this code in place, simply import this file instead of lodash version into your consuming code.
import _memoize from 'lodash-es/memoize';

export default function memoize(func)
{
    const resolver = (...args) => JSON.stringify(args);

    return _memoize(func, resolver);
}

Batching Async Calls

April 25, 2018

To reduce the payload size of individual calls when loading up resources by ID, sometime you want to send multiple async requests in smaller batches. If we don't need to worry about local/remote resources (i.e. don't need intelligent partitioning or resource friendly approach), the easiest way is to fire off a load of tasks which consume a small batch from the superset. Here is a simple re-usable implementation:
public class BatchContentRequestor<TId, TValue>
{
    private readonly int _batchSize;
    private readonly Func<IEnumerable<TId>, Task<IEnumerable<TValue>>> _getContentAsyncFunc;

    public BatchContentRequestor(int batchSize, Func<IEnumerable<TId>, Task<IEnumerable<TValue>>> getContentAsyncFunc)
    {
        if (batchSize <= 0)
        {
            throw new ArgumentOutOfRangeException(nameof(batchSize), "Batch size must be a positive integer value.");
        }

        _batchSize = batchSize;
        _getContentAsyncFunc = getContentAsyncFunc ?? throw new ArgumentNullException(nameof(getContentAsyncFunc));
    }

    public async Task<IEnumerable<TValue>> GetContentBatchedAsync(IEnumerable<TId> allContentIds)
    {
        var allContentIdsList = allContentIds?.ToList();

        if (allContentIdsList == null || !allContentIdsList .Any())
        {
            return await _getContentAsyncFunc(allContentIdsList );
        }

        var allContentValues = new List<TValue>();

        var getBatchTasks = new List<Task<IEnumerable<TValue>>>();
        for (var batchStart = 0;
            batchStart < allContentIdsList.Count;
            batchStart += _batchSize)
        {
            var batchIds = allContentIdsList
                .Skip(batchStart)
                .Take(_batchSize)
                .ToList();

            getBatchTasks.Add(_getContentAsyncFunc(batchIds));
        }

        await Task.WhenAll(getBatchTasks).ConfigureAwait(false);

        foreach (var completedBatch in getBatchTasks)
        {
            allContentValues.AddRange(await completedBatch.ConfigureAwait(false));
        }

        return allContentValues;
    }
}
You can call it with your superset and it will automatically hit your callback function with the batches of IDs, will collect the results and return the superset of values. If the calling code passes a null or empty value this will still be passed to your callback for handling, making this a transparent proxy for the calling code. e.g.
var items = await new BatchContentRequestor<int, Item>(10, GetItemsByIdAsync).GetContentBatchedAsync(allItemIds).ConfigureAwait(false);

Fast and Easy GetHashCode

April 17, 2018

Often when you create a class you need to override the the GetHashCode method, to easily compare instances using a custom Equals operator or to use the class as a key in dictionaries etc. I've seen various ways of doing this, but they are usually either wrong, slow, complex or difficult to remember (things like string concatenation, bit shifting, xor, using prime numbers etc.) As ever with programming, if there is a complex problem to solve then somebody has probably already solved it (and I don't mean copy/pasting GetHashCode algorithms from stack overflow!). One place this has already been done within the .NET framework is the Tuple type. The algorithm is fast, we don't need to understand it's complexity and given that it in the framework we can assume it's well tested. So my new favourite way of implementing GetHashCode is to simply project the fields into a Tuple!
public class Example
{
  private string someField1;
  private int someField2;

  public override int GetHashCode()
  {
      return (someField1, someField2).GetHashCode();
  }
}
This isn't quite so straightforward when one of your fields is a collection type, however it can be worked around using IStructuralEquatable as per this MSDN link: https://docs.microsoft.com/en-us/dotnet/api/system.collections.istructuralequatable?redirectedfrom=MSDN&view=netcore-2.2#remarks So as long as your collection types are backed by an Array, you can use the following syntax to generate your hash code based on the array values:
public class Example
{
  private string someField;
  private int[] someArrayField;

  public override int GetHashCode()
  {
      return ((someField, someArrayField) as IStructuralEquatable).GetHashCode(StructuralComparisons.StructuralEqualityComparer);
  }
}

Bio - Pragmatech Software Solutions

January 01, 2018

From the beginning of 2018 I have taken the leap into self-employment. I am now working as a consultant/contract software developer through Pragmatech Software Solutions. Primarily I expect this will involve applying more of my technical expertise in coding and software development which is what I really enjoy and where I excel. I also hope to be able to help clients in any way I can such as providing input on solution architecture, software design, performance optimization, Azure cloud, VSTS, CI/CD, Agile or any other areas where I have experience!

Adding HATEOAS Links to Web API Resources

December 01, 2017

I was recently working with a React developer on a data driven single page application using an ASP.NET WebAPI backend. I wanted the API to be RESTful to simplify the design, since React can provide client side state and in fact is driven by application state and the API was only there to provide data and persistence. We decided to use the HATEOAS convention, of simply adding a "links" section to each of the objects, in order to inform the client of the available associated resource URIs. Based loosely on the HAL standards we agreed that we simply wanted to add some basic links within the normal JSON response as opposed to negotiating new media types etc. For this very basic requirement then, comes a very simple solution. I created a base class in my API Models folder which would support links:
public class LinkModel
{
    public LinkModel()
    {
    }

    public LinkModel(string href, string method, string rel)
        : this()
    {
        this.Href = href;
        this.Method = method;
        this.Rel = rel;
    }

    public string Method { get; set; }
    public string Rel { get; set; }
    public string Href { get; set; }
}
And then a generic class which would contain the payload object, or "content" and the "links":
public class ContentWithLinks<T>
{
    public ContentWithLinks()
    {
    }

    public ContentWithLinks(T content, IEnumerable<LinkModel> links)
        : this()
    {
        this.Content = content;
        this.Links = links;
    }

    public T Content { get; set; }
    public IEnumerable<LinkModel> Links { get; set; }
}
Each of my actions that wanted to use this strategy would return this type with the calculated links:
[HttpGet]
[Route]
public IEnumerable<ContentWithLinks<DataSetModel>> GetAllDataSets()
{
    return new ContentWithLinks<DataSetModel>[]
    {
        new ContentWithLinks<DataSetModel>(new DataSetModel("Example"), this.GetLinks("Example"))
    };
}

private IEnumerable<LinkModel> GetLinks(string dataSetName)
{
    return new LinkModel[]
    {
        new LinkModel(this.Url.Link(nameof(this.GetAnalysisSchemaForDataSet), new { dataSetName = dataSetName }), "GET", "Schema"),
        new LinkModel(this.Url.Link(nameof(FiltersController.GetAllFilters), new { dataSetName = dataSetName }), "GET", "Filters"),
        new LinkModel(this.Url.Link(nameof(FilesController.GetAllFiles), new { dataSetName = dataSetName }), "GET", "Files")
    };
}
Using the "Url.Link" syntax ensure the route is calculated by ASP.NET. In fact, I went ahead and used the "nameof" operator too, to ensure any code refactorings of these methods would be taken in account automatically. In the client side the various links were then bound to the different actions a user could perform, which would trigger a call to the next URI in the workflow, generating more links and actions for the React components to bind.

Object Oriented Azure Search Client

June 01, 2017

There are a few options already for wrapping calls to Elastic Search (such as ElasticSearch.Net and NEST) however, I wanted to create an object oriented strongly type interface for some simple queries - (with a view to also being able to convert to this from a CDSA "where clause"). My particular implementation of Elastic search, Azure Search, can be called using a restful API interface with OData and Lucene syntax. So my idea was then eventually any OO representation of a query can eventually then be boiled down to a URL with a querystring (similar to how CDSA converts OO clauses to SQL). My first step was to create an abstraction around the concept of a "filter" that can be applied to a search:
public abstract class Filter
{
    public abstract string GetQueryString();
}
Since we are most likely going to be dealing with collections of filters, I created an extension to provide easy access to this method:
internal static class FilterExtensions
{
    public static string AsQueryString(this IEnumerable<Filter> filters)
    {
        return string.Join(" and ", filters.Select(f => f.GetQueryString()));
    }
}
The first concrete implementation is always required by Azure Search, it's the API version you intend to work with:
internal sealed class ApiVersionFilter : Filter
{
    public override string GetQueryString()
    {
        return $"api-version=2016-09-01";
    }
}
Most fields you are querying will either be some filterable scalar value like strings and numbers or collection of these values in a collection column. I represented these types of query using two classes:
public class FieldValuesFilter : Filter
{
    private IEnumerable<IFieldValue> selectedValues;
    private string @operator;

    public FieldValuesFilter(string fieldName, IEnumerable<IFieldValue> selectedValues)
        : this(fieldName, "eq", selectedValues)
    {
    }

    public FieldValuesFilter(string fieldName, string @operator, IEnumerable<IFieldValue> selectedValues)
    {
        this.FieldName = fieldName;
        this.selectedValues = selectedValues;
        this.@operator = @operator;
    }

    public string FieldName { get; private set; }

    public override string GetQueryString()
    {
        return "(" + string.Join(" or ", this.selectedValues.Select(v => $"{this.FieldName} {this.@operator} {v.GetFormattedValue()}")) + ")";
    }
}

public class ArrayValueFilter : Filter
{
    private IEnumerable<IFieldValue> selectedValues;
    private string @operator;

    public ArrayValueFilter(string fieldName, IEnumerable<IFieldValue> selectedValues)
        : this(fieldName, "any", selectedValues)
    {
    }

    public ArrayValueFilter(string fieldName, string @operator, IEnumerable<IFieldValue> selectedValues)
    {
        this.FieldName = fieldName;
        this.selectedValues = selectedValues;
        this.@operator = @operator;
    }

    public string FieldName { get; private set; }

    public override string GetQueryString()
    {
        return "(" + string.Join(" or ", this.selectedValues.Select(v => $"{this.FieldName}/{this.@operator}(t: t eq {v.GetFormattedValue()})")) + ")";
    }
}
You will notice that to abstract the field type I use an interface for IFieldValue for passing in the filter values, this is because depending on whether the data type is a string or a number the formatting will change. The interface and the two implementing classes are below:
public interface IFieldValue
{
    object GetValue();
    string GetFormattedValue();
}

public class StringFieldValue : IFieldValue
{
    private string value;

    public StringFieldValue(string value)
    {
        this.value = value;
    }

    public string GetFormattedValue()
    {
        return $"'{this.value}'";
    }

    public object GetValue()
    {
        return this.value;
    }
}

public class IntegerFieldValue : IFieldValue
{
    private int value;

    public IntegerFieldValue(int value)
    {
        this.value = value;
    }

    public string GetFormattedValue()
    {
        return this.value.ToString();
    }

    public object GetValue()
    {
        return this.value;
    }
}
You can create other filters, such as a "well known text" or "proximity" filter to query spatial data:
public class WktFilter : Filter
{
    private string wkt;

    public WktFilter(string wkt)
    {
        this.Wkt = wkt;
    }

    public string Wkt
    {
        get
        {
            return this.wkt;
        }

        set
        {
            this.wkt = value;
        }
    }

    public override string GetQueryString()
    {
        return $"geo.intersects(location, geography'{this.Wkt}')";
    }
}

public class ProximityFilter : Filter
{
    private double latitude;
    private double longitude;
    private double radiusInMeters;

    public ProximityFilter(double latitude, double longitude, double radiusInMeters)
    {
        this.latitude = latitude;
        this.longitude = longitude;
        this.radiusInMeters = radiusInMeters;
    }

    public override string GetQueryString()
    {
        // Azure Search works with KM not M, so div by 1000
        return $"geo.distance(location, geography'POINT({this.longitude} {this.latitude})') le {this.radiusInMeters / 1000d}";
    }
}
Now that we have the ability to create filters using C# classes and for more derivatives to be added if you more more strong typing, then we need a way of converting this to an Azure Search query. For simplicity I again started with a base class representing a given service which encapsulates the ability to convert a set of filters into a query string or post body and holds the config and endpoint for the index:
public abstract class SearchServiceBase
{
    private readonly AzureServiceConfig config;

    public SearchServiceBase(AzureServiceConfig config)
    {
        this.config = config;
    }

    public abstract string Api { get; }

    public AzureServiceConfig Config
    {
        get
        {
            return this.config;
        }
    }

    protected string GetQueryString(string name, IEnumerable<Filter> filters, int? top, int? skip)
    {
        // parse the input parameters
        StringBuilder requestParameters = new StringBuilder();

        // name param
        if (!string.IsNullOrEmpty(name))
        {
            requestParameters.Append($"search={name}&");
        }

        // add API version by default
        requestParameters.Append(new ApiVersionFilter().GetQueryString() + "&");

        if (top.HasValue)
        {
            requestParameters.Append($"$top={top.Value.ToString()}&");
        }

        if (skip.HasValue)
        {
            requestParameters.Append($"$skip={skip.Value.ToString()}&");
        }

        if (skip.HasValue && skip.Value == 0)
        {
            requestParameters.Append($"$count=true&");
        }

        // filters could be none, one or many
        if (filters != null && filters.Any())
        {
            requestParameters.Append($"$filter={filters.AsQueryString()}&");
        }

        // get the resource
        return requestParameters.Length > 0 ? "?" + requestParameters.ToString().TrimEnd('&') : string.Empty;
    }

    protected string GetPostQueryString()
    {
        return "?" + new ApiVersionFilter().GetQueryString();
    }

    protected string GetPostBody(string name, IEnumerable<Filter> filters, int? top, int? skip)
    {
        // parse the input parameters
        StringWriter sw = new StringWriter();
        JsonTextWriter writer = new JsonTextWriter(sw);

        // {
        writer.WriteStartObject();

        if (skip.HasValue && skip.Value == 0)
        {
            writer.WritePropertyName("count");
            writer.WriteValue("true");
        }

        if (!string.IsNullOrEmpty(name))
        {
            writer.WritePropertyName("search");
            writer.WriteValue(name);

            writer.WritePropertyName("searchMode");
            writer.WriteValue("all");
        }

        if (top.HasValue)
        {
            writer.WritePropertyName("top");
            writer.WriteValue(top.Value.ToString());
        }

        if (skip.HasValue)
        {
            writer.WritePropertyName("skip");
            writer.WriteValue(skip.Value.ToString());
        }

        // filters could be none, one or many
        if (filters != null && filters.Any())
        {
            writer.WritePropertyName("filter");
            writer.WriteValue(filters.AsQueryString());     // querystring is same format as POST property value
        }

        // }
        writer.WriteEndObject();

        return sw.ToString();
    }
}
So an example subclass which consumes this functionality:
public class ExampleSearchService : SearchServiceBase
{
    private readonly string indexName;

    public ExampleSearchService(AzureServiceConfig config, string indexName)
        : base(config)
    {
        this.indexName = indexName;
    }

    public override string Api
    {
        get
        {
            return $"indexes/{this.indexName}/docs/search";
        }
    }

    public async Task<IEnumerable<ExampleSearchResult>> GetResultsAsync(string name, IEnumerable<Filter> filters, int maxResultCount, bool usePost = false)
    {
        using (var webApiClient = new AzureElasticSearchClient(this.Config))
        {
            webApiClient.Timeout = new TimeSpan(0, 20, 0);

            var results = new List<ExampleSearchResult>();

            int pageSize = 1000;
            int pagesToRetrieve = 1;
            int pagesRetrieved = 0;

            // at least one page, but may be more..
            while (pagesRetrieved < pagesToRetrieve)
            {
                HttpResponseMessage result = null;
                if (usePost)
                {
                    string requestUrl = $"{this.Api}{this.GetPostQueryString()}";
                    string requestBody = this.GetPostBody(name, filters, pageSize, pagesRetrieved * pageSize);

                    HttpContent content = new StringContent(requestBody, System.Text.Encoding.UTF8, "application/json");

                    // call the API and increment received count
                    result = await webApiClient.PostAsync(requestUrl, content).ConfigureAwait(false);
                }
                else
                {
                    // build the query url from the filters
                    string requestUrl = $"{this.Api}{this.GetQueryString(name, filters, pageSize, pagesRetreived * pageSize)}";

                    result = await webApiClient.GetAsync(requestUrl).ConfigureAwait(false);
                }
                
                pagesRetrieved++;

                // if it was successful, we can process it
                if (result.IsSuccessStatusCode)
                {
                    // parse the JSON response
                    var jResponse = JObject.Parse(await result.Content.ReadAsStringAsync().ConfigureAwait(false));

                    // if this was the first page, we can parse the item count
                    if (pagesRetrieved == 1)
                    {
                        int count = (int)jResponse.GetValue("@odata.count");

                        // check against max result count and throw exception if over that
                        if (maxResultCount > 0)
                        {
                            if (count > maxResultCount)
                            {
                                throw new InvalidOperationException($"Search result count of {count} was greater than the maximum of {maxResultCount}");
                            }
                        }

                        pagesToRetrieve = (int)Math.Ceiling((double)count / (double)pageSize);
                    }

                    // now get the value, which is the array of results
                    JArray jsonResults = (JArray)jResponse.GetValue("value");

                    // loop over the JSON array and deserialise each result obejct
                    foreach (var resultData in jsonResults)
                    {
                        var result = resultData.ToObject<ExampleSearchResult>();

                        results.Add(result);
                    }
                }
            }

            return results;
        }
    }
}
You will notice I created a wrapper for HttpClient, this is simply to encapsulate adding the base address and API key:
internal class AzureElasticSearchClient : HttpClient
    {
        public AzureElasticSearchClient(AzureServiceConfig config)
        {
            this.BaseAddress = new Uri($"https://{config.AzureSearchServiceName}.search.windows.net/");
            this.DefaultRequestHeaders.Add("api-key", config.AzureSearchApiKey);
        }
    }
Here are some examples of how to create and add filters then call the example search service:
[TestFixture]
public class TestExampleSearchService
{
    private ExampleSearchService sut = new ExampleSearchService();

    [Test]
    public async Task TestGetResultsInWktWithAdditionalFilter_Elastic()
    {
        // arrange
        var testPolygonWkt = "POLYGON ((-1.6259765625 53.74404116282134, -1.6005706787109375 53.76089000834015, -1.5696716308593748 53.73876182109416, -1.6036605834960935 53.72799803200196, -1.6259765625 53.74404116282134))";
        var polyFilter = new WktFilter(testPolygonWkt);
        var examplePropertyFilter = new FieldValuesFilter("SomeIntProperty", new IntegerFieldValue[] { new IntegerFieldValue(1) });
        
        // act
        var startTime = DateTime.Now;
        var result = await this.sut.GetResultsAsync(string.Empty, new Filters.Filter[] { polyFilter, examplePropertyFilter });
        Trace.WriteLine($"Time taken: {DateTime.Now.Subtract(startTime).TotalMilliseconds} ms");
        Trace.WriteLine($"# recs {result.Count()}");
        // assert
        Assert.Greater(result.Count(), 0);
    }

    [Test]
    public async Task TestGetResultsInProximityWithName_Elastic()
    {
        // arrange
        var longi = -1.6259765625;
        var lati = 53.74404116282134;
        var dist = 200;
        var proxFilter = new ProximityFilter(lati, longi, dist);

        // act
        var startTime = DateTime.Now;
        var result = await this.sut.GetResultsAsync("example name", new Filters.Filter[] { proxFilter });

        Trace.WriteLine($"# recs {result.Count()}");
        Trace.WriteLine($"Time taken: {DateTime.Now.Subtract(startTime).TotalMilliseconds} ms");

        // assert
        Assert.Greater(result.Count(), 0);
    }

    [Test]
    public async Task TestGetResultsInArrayFilter_Elastic()
    {
        // arrange
        var possibleValues = {"hello", "world"}.Select(s => new StringFieldValue(s));
        var exampleArrayPropertyFilter = new FieldValuesFilter("SomeStringArrayProperty", possibleValues);
        
        // act
        var startTime = DateTime.Now;
        var result = await this.sut.GetResultsAsync(string.Empty, new Filters.Filter[] { exampleArrayPropertyFilter });
        Trace.WriteLine($"Time taken: {DateTime.Now.Subtract(startTime).TotalMilliseconds} ms");
        Trace.WriteLine($"# recs {result.Count()}");
        // assert
        Assert.Greater(result.Count(), 0);
    }
}
With this structure all now in place, I wanted to plug this into an existing CDSA application which was currently using SQL for performing queries. I therefore needed a way to convert from traditional "CDSA WhereClause" objects to my new Azure Search filter structure. I created a basic implementation, which isn't 100% compatible with all clauses yet, but for most use cases it works fine:
internal class WhereClauseParser
{
    Regex isNumeric = new Regex("^\\d+$");

    public IEnumerable<FieldValuesFilter> ParseWhereClause(WhereClause clause)
    {
        if (clause == null || clause.RecursiveClauseList.Count == 0)
        {
            return null;
        }

        return this.ParseWhereClauseWithSubgroups(clause);
    }

    private IEnumerable<FieldValuesFilter> ParseWhereClauseWithSubgroups(WhereClause clause)
    {
        if (clause.ConjunctionOperator == ConjunctionOperator.And)
        {
            var myFilters = new List<FieldValuesFilter>();
            foreach (var whereClauseElement in clause.ClauseList)
            {
                myFilters.Add(this.ParseWhereClauseElement(whereClauseElement));
            }

            if (clause.SubGroups.Count > 0)
            {
                foreach (var subClause in clause.SubGroups)
                {
                    myFilters.AddRange(this.ParseWhereClauseWithSubgroups(subClause));
                }
            }

            return myFilters;
        }
        else
        {
            throw new NotImplementedException("Elastic search clause parser currently only supports the 'AND' conjunction.");
        }
    }

    private FieldValuesFilter ParseWhereClauseElement(WhereClauseElement whereClauseElement)
    {
        // start with the defaults
        var fieldName = whereClauseElement.CompareItem;
        var values = new object[] { whereClauseElement.CompareValue };

        var @operator = string.Empty;

        // don't need qualified paths, remove the dots
        fieldName = fieldName.Replace(".", "");

        switch (whereClauseElement.Operator)
        {
            case Operator.Equals:
            case Operator.Like: // todo: this should be a wildcard search, not an exact equality
                @operator = "eq";
                break;
            case Operator.NotEqual:
                @operator = "ne";
                break;
            case Operator.GreaterThan:
                @operator = "gt";
                break;
            case Operator.LessThan:
                @operator = "lt";
                break;
            case Operator.GreaterThanEqualTo:
                @operator = "ge";
                break;
            case Operator.LessThanEqualTo:
                @operator = "le";
                break;
            case Operator.In:
                @operator = "eq";
                values = values[0].ToString().Split(',');

                // if it was an array of numbers stored as CSV, then unpack the numbers as ints.
                if (values.All(v => this.isNumeric.IsMatch(v.ToString())))
                {
                    values = values.Select(v => (object) int.Parse(v.ToString())).ToArray();
                }

                break;
            default:
                throw new NotImplementedException("Elastic search clause parser currently does not support this operator");
        }

        return new FieldValuesFilter(fieldName, @operator, values.Select(v => this.FieldValueFactory(v)));

    }

    private IFieldValue FieldValueFactory(object v)
    {
        // a dirty type check, lets call it a "factory"
        if (v is int)
        {
            return new IntegerFieldValue((int)v);
        }
        else
        {
            return new StringFieldValue(v.ToString());
        }
    }
}