Blog

Useful tidbits related to software development, that I think might be of use or interest to everyone else (or to me when I forget what I did!)

Writing an NPM startup script in TypeScript to support complex scenarios

December 19, 2022

NPM scripts are often used to abbreviate longer commands with many arguments or to sequence multiple commands that are required to compile or run a Node based system. Sometimes they can get complex; if you have multiple moving parts to consider, such as conditionally setting environment variables, conditional compilation steps or multiple sequential and parallel stages of the build. This is especially true if you trying to write a startup script for automation tests, which probably need to spin up mocked APIs, build and start the system under test, wait for startup to complete then execute tests. It is equally as useful when you have multiple configurations of the application and it's supporting local development services (such as running a mock, changing an environment flag etc.) This can be achieved using daisy chained NPM scripts and helper packages such as "npm-run-all", "concurrently", "start-server-and-test", but I found that this doesn't scale well if you have multiple options for developers to choose from when running the app. A contrived example of some options you might want to give developers: - run in "development" or "production" mode (usually determined by running either "npm run dev" or "npm run build/start") - setting environment variables based on the mode (e.g. you might have used "cross-env-shell" in your npm script) - start a mocked api, or don't (maybe you'd spin this up asynchronously using "npm-run-all" or "concurrently") - build for "flavour 1" or "flavour 2" of the code (say, for example, you can choose whether to compile for REST or GraphQL at build time) You might also have some automation tests that need to: - start a mocked api (and wait for it to load) - build and run the system under test in production mode (and wait for it to load) - run the tests (in local or browserstack mode) - kill all the child processes Aside from the complexities of describing all of the above in NPM scripts, it gets very repetitive which each variation. Even with the limited choices described above you get the Cartesian product of all combinations, represented as scripts e.g.: "development mode", "with mock", "GraphQL" "development mode", "with mock", "REST" "development mode", "no mock", "GraphQL" "development mode", "no mock", "REST" "production mode", "with mock", "GraphQL" "production mode", "with mock", "REST" "production mode", "no mock", "GraphQL" "production mode", "no mock", "REST" You can remove one level of complexity by using NPM configuration settings, to switch between modes, that live in your .npmrc file such as:
mock-api='true'
api-mode='REST'
use-browserstack='false'
And then only having the usual 3 scripts in your package.json (for example using Next.js) that take into account the config settings: "dev", "build", "start" By using settings in the .npmrc file we get all the power of NPM configuration, which means by default it will use the values defined in .npmrc, but these can be overridden with environment variables (so maybe you'd set different defaults in your Dockerfile or build pipeline, than what local devs might use), or can be override with CLI arguments (so maybe you'd do this when spinning up a system from an automation test suite). The next complexity to solve is how to interpret the NPM configuration settings, such that the right build steps are executed and are sequential or parallel in nature accordingly. This where I decided that NPM scripts still weren't the best choice and it would be easier to write that logic as TypeScript (and have npm simply execute the script). The below example shows how this can work for the automation test scenario, making use of a "test-runner" script and re-using "build" and "start" scripts from the system under test: package.json snippet:
"test:automation": "cross-env-shell NODE_ENV='production' \"ts-node --project ./tsconfig.commonjs.json test-runner.ts\"",
tsconfig snippet:
"compilerOptions": {
	"module": "commonjs"
}
test-runner.ts:
#!/usr/bin/env ts-node
import { subProcess, subProcessSync } from 'subspawn';

// example of reading NPM config from within a script
let serverHost = 'localhost';
if (process.env.npm_config_use_browserstack === 'true') {
  serverHost = 'bs-local.com';
}

// example of overriding NPM config from within a script (to disable system's built in mock api when it's script runs)
process.env.npm_config_mock_api = 'false';

// example of setting general environment variables used by your application (override api to point at the mock)
process.env.NEXT_PUBLIC_API_HOST = `http://${serverHost}:5038`;

// example of spinning up background services and waiting for them to load
subProcess('automation-tests', 'npm run start-wiremock', true);
subProcessSync('npx wait-on tcp:5038', false);

// example of re-using scripts that exists for spinning up the system
process.chdir('../../src');

if (process.env.npm_config_skip_build !== 'true') {
  process.env.PUBLIC_URL = `http://${serverHost}:3000`;

  require('../../src/build'); // pull in the build script for the SUT
}
// start the SUT
require('../../src/start'); // pull in the start script for the SUT
process.chdir('../tests/integration');

// begin the test execution
subProcessSync('npm run execute-tests', true);

// exiting the process will also kill all the background processes
process.exit(0);

export {};

You'll notice the use of "npx wait-on" which is a handy package for testing when a dependency has become available. You'll also notice the use of "subspawn" which is an NPM package I created specifically for this use case to address the complexities of spawning, killing and integrating the stdout of child processes in Node in a cross-platform way.

Proxying clientside API traffic from deployed website to local machine

November 08, 2022

Recently I ran into a situation where I wanted to proxy the calls made by a client application (i.e. javascript AJAX calls) for a web application that was hosted on another machine. So the configuration looked something like below: https://www.mywebsite.com -> user machine (javascript) -> https://api.mywebsite.com/api-1/endpoint I didn't want to run "www.mywebsite.com" code on my machine, I wanted to run the deployed website in my browser, but I wanted all calls to "api.mywebsite.com/api-1" to be routed to my local development environment so I could either debug or mock the API responses as I wanted. The solution comprised of 3 basic elements:
  1. An Nginx reverse proxy running in Docker on my machine
  2. Self signed SSL certificates for TLS termination in Nginx
  3. Running Chrome with custom host resolver rules (this could also be done in /etc/hosts but I only wanted a temporary solution)
If your API calls don't use HTTPS then you don't need the TLS termination, but in my case I do need it so I created some self-signed certificates that I will later trust within Chrome:
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout nginx-selfsigned.key -out nginx-selfsigned.crt
Now create a reverse-proxy Nginx configuration file, like below:
server {
    listen 0.0.0.0:80;
    listen 0.0.0.0:443 ssl;
    server_name host.docker.internal;
    ssl_certificate       /etc/ssl/certs/server.crt;
    ssl_certificate_key   /etc/ssl/certs/server.key;
    location /api-1/ {
        proxy_pass http://host.docker.internal:5001/;
    }
}
This essentially now routes the traffics from "https://localhost/api-1" to "http://host-machine:5001/" which is where I can run the development mode API. With those things in place, whenever I want to the run the deployed website against my local machine APIs, I can use the below commands:
docker run --rm \
-p 80:80 -p 443:443 \
--name nginx-reverse-proxy \
--add-host=host.docker.internal:host-gateway \
-v $(pwd)/nginx.conf:/etc/nginx/conf.d/default.conf \
-v $(pwd)/nginx-selfsigned.crt:/etc/ssl/certs/server.crt \
-v $(pwd)/nginx-selfsigned.key:/etc/ssl/certs/server.key \
nginx &
google-chrome https://www.mywebsite.com/ --host-resolver-rules="MAP api.mywebsite.com 127.0.0.1"
The first time you run it, Chrome won't trust your SSL certs, but if you open an API URL in a new tab and manually trust it, then the clientside calls will work :)

Creating additional monitors in Immersed for Linux

July 19, 2022

Immersed is an application that allows you to connect to your computer from within Virtual Reality and use the full power of your Windows/Mac/Linux machine to run apps etc. but using virtual screens that are in a VR world. This can be useful if you want to get away from the home office environment and into a world where you can fully concentrate on the task at hand, or if you're travelling and can't take around your usual multi-screen physical setup. The Windows and Mac versions of Immersed allow you to create additional "virtual" monitors so that when you enter into VR you can see not only your physical display but one or more "virtual" displays too. Unfortunately the Linux version of the desktop agent does not yet support this feature, so if you have a 2 screen setup in the real world, then you'll see 2 screens in VR. However, there is a way of adding these screens in Linux using a tool called "xrandr" (works with X11). I originally tried the methods documented on the virtual-display-linux GitHub page, however this doesn't work when using Cinnamon desktop on Linux Mint as the additional virtual screen causes Cinnamon to crash on startup. It actually turns out that you don't need to create a virtual screen as long as you have some unused graphics ports on your machine, for example on my machine running xrandr -q shows:
Screen 0: minimum 320 x 200, current 3200 x 1200, maximum 16384 x 16384
eDP-1 connected (normal left inverted right x axis y axis)
   1920x1080     60.02 +  60.01    59.97    59.96    59.93    48.00  
   ...
DP-1 disconnected (normal left inverted right x axis y axis)
HDMI-1 disconnected (normal left inverted right x axis y axis)
DP-2 disconnected (normal left inverted right x axis y axis)
HDMI-2 disconnected (normal left inverted right x axis y axis)
DP-3 disconnected (normal left inverted right x axis y axis)
HDMI-3 disconnected (normal left inverted right x axis y axis)
DP-3-1 disconnected (normal left inverted right x axis y axis)
DP-3-2 connected primary 1600x1200+0+0 (normal left inverted right x axis y axis) 367mm x 275mm
   1600x1200     60.00*+
   ...
DP-3-3 connected 1600x1200+1600+0 (normal left inverted right x axis y axis) 367mm x 275mm
   1600x1200     60.00*+
   ...
The "eDP-1" adapter is my physical laptop screen (which is connected but with the lid closed is not active). Then my two physical monitors plugged into my docking station are both running under the "DP-3" display port, as "DP3-2" and "DP3-3".. This means I have "HDMI-1", "HDMI-2", "DP3-1" all available to "plug something in". You don't actually need to physically plug something in to use these in VR though, so I can just activate one (or more) of them at my desired resolution and position it wherever I'd like it to appear when I enter VR. In my case I like to split my two physical monitors apart with a 3rd low res, wide monitor (1400x900) that makes working in VR easier. For example:
Real world:
/-------\ /-------\
| DP3-2 | | DP3-3 |
\-------/ \-------/

Virtual world:
/-------\ /--------\ /-------\
| DP3-2 | | HDMI-1 | | DP3-3 |
\-------/ \--------/ \-------/
To achieve this, I've written a shell script which will add the new display settings before starting up the Immersed Agent, and will then reset the settings when the process finishes:
#!/bin/sh
xrandr --addmode HDMI-1 1400x900
xrandr --output DP-3-2 --pos 0x0 --output HDMI-1 --mode 1400x900 --right-of DP-3-2 --output DP-3-3 --right-of HDMI-1
~/.local/bin/Immersed/Immersed-x86_64.AppImage
xrandr --output HDMI-1 --off --output DP-3-3 --right-of DP-3-2

Easily Wrapping Standalone Components in Redux Provider

January 28, 2022

In order to setup a context when using React hooks related to a Redux store (e.g. useDispatch, useSelector) you need to have your component nested inside of a "Provider" component (from the react-redux package). This isn't always possible as not all applications are built as a single "app" with components nested under a single root. In my case I am using ReactJs.Net together with a CMS to allow the end user to defined any combination of a number of pre-defined "components" on a page. It turns out that you don't need all components to be nested inside of the same "Provider" instance, as long as the "store" itself is a singleton then you can have many "Provider" instances on the page all sharing the same "store". I wanted an easy way to start wrapping my existing components in a "Provider" component without having to change too much about my non-Redux application structure. What I came up with was creating a simple high order component as a function and then simply wrapping my exported component with a call to the HOC when I want to wrap it with the Redux provider, e.g.
import React from 'react';
import { Provider } from 'react-redux';
import { store } from './store';

export const withReduxStore = Component => ({ ...props }) =>
    (<Provider store={store}><Component {...props} /></Provider>)
This assumes you have singleton store, for example:
import { createStore, applyMiddleware } from 'redux';
import thunk from 'redux-thunk';
import rootReducer from './ducks/rootReducer'

const store = createStore(rootReducer, applyMiddleware(thunk));

export { store };
And now to update a component that previously didn't have access to the store context and give it access:
import React from 'react';
import { useDispatch } from 'react-redux';
import { withReduxStore } from './state/withReduxStore.jsx';

const MyExampleComponent = (props) => {
    const dispatch = useDispatch();

    return <>
        <button onClick={() => dispatch({hello: "world"})} type="button">Dispatch Something</button>
    </>
}

export default withReduxStore(MyExampleComponent); <-- simply wrap it with a call to "withReduxStore"

Combining Two Objects in C# Similar to JavaScript Spread Operator

September 16, 2021

I wouldn't necessarily recommend doing something like this in production code however I find this useful when writing SpecFlow tests - I want the Gherkin to call out a few key properties of a class, then I want to generate a "valid" instance (to pass any validation) but using the test data supplied. Imagine the following scenario:
public class Customer
{
   public string Firstname { get; set; }
   public string Surname { get; set; }
   public string EmailAddress { get; set; }       // validation states that this must be a valid email address
}

// imagine some kind of "valid instance builder" used in testing
public static class TestCustomerBuilder
{
      private static readonly Fixture Fixture = new();
      public static Customer AnyValidInstance()
     {
         return Fixture.Build<Customer>()
                   .With(c => c.EmailAddress, _fixture.Create<MailAddress>().Address) // make sure this passes validation by default
                   .Create();
     }
}
Now imagine you're writing some Gherkin that doesn't care about email - you're just testing something to do with Firstname and Surname, so you might write:
Given the create customer request contains the below details
| Firstname | Surname |
| Hello     | World   |
When the create customer endpoint is called
Then a new customer is created with the following details
| Firstname | Surname |
| Hello     | World   |
It's a contrived example, but you should see the point when it comes to implementing the step definitions I like to use the built in helpers from SpecFlow rather than "magic strings" as much as possible (as it makes the steps more re-usable) so how about the below:
[Given("the create customer request contains the below details")]
public void GivenTheCreateCustomerRequestContainsTheBelowDetails(Table table)
{
   _testContext.CustomerRequest = table.CreateInstance<Customer>();
}
The problem with the above is the created instance won't be valid, on account of it having no email address. You could code around this by manually only setting certain properties but that introduces the re-usability problem again. Enter the "model combiner" which is designed to copy all non-null properties from a source instance to a destination instance, e.g.:
[Given("the create customer request contains the below details")]
public void GivenTheCreateCustomerRequestContainsTheBelowDetails(Table table)
{
   var testDataInstance  = table.CreateInstance<Customer>();
   var validInstance = TestCustomerBuilder.AnyValidInstance();

   ModelCombiner.Comine(testDataInstance, validInstance);

   _testContext.CustomerRequest = validInstance;
}
Now the request contains a "valid" customer but also has our specific data taken from the Gherkin. The model binder class looks as below (which I got from an idea seen here: https://stackoverflow.com/questions/8702603/merging-two-objects-in-c-sharp)
public static class ModelCombiner
{
	private static readonly HashSet<Type> SupportedTypes = new();

	private static Mapper Mapper { get; } = new(new MapperConfiguration(expression =>
	{
		Setup<Customer>(expression);
		Setup<SomeOtherType>(expression);
	}));

	public static T Combine<T>(T source, T destination)
	{
		if (!SupportedTypes.Contains(typeof(T)))
			throw new InvalidOperationException(
				$"Cannot combined unsupported type {typeof(T).FullName}. Please add it to the setup in {nameof(ModelCombiner)}");

		return Mapper.Map(source, destination);
	}

	private static void Setup<T>(IProfileExpression expression)
	{
		SupportedTypes.Add(typeof(T));

		expression.CreateMap<T, T>()
			.ForAllMembers(opts => opts
				.Condition((_, _, srcMember) => srcMember != null));
	}
}
Another option I found online that looks worth a look: https://github.com/kfinley/TypeMerger

Using Fluent Assertions Inside of a Moq Verify

September 06, 2021

FluentAssertions adds many helpful ways of comparing data in order to check for "equality" beyond a simple direct comparison (for example check for equivalence across types, across collections, automatically converting types, ignoring elements of types, using fuzzy matching for dates and more). Making a "fluent assertion" on something will automatically integrate with your test framework, registering a failed test if something doesn't quite match. e.g. to compare an object excluding the DateCreated element:
actual.Should()
	.BeEquivalentTo(expected, cfg => cfg.Excluding(p => p.DateCreated));
However, sometimes the "actual" value you want to make the assertion on is only available as part of a Moq Verify statement, which only supports matching based on a boolean return type. e.g.
myMock.Verify(m => 
	m.Method(It.Is<MyData>(actual => 
		actual == expected));
As you can see above, replacing "==" with a "Fluent" assertion is not possible out of the box. However there is a trick you can use by setting up the below helper method:
public static class FluentVerifier
{
	public static bool VerifyFluentAssertion(Action assertion)
	{
		using (var assertionScope = new AssertionScope())
		{
			assertion();

			return !assertionScope.Discard().Any();
		}
	}
}
Now you can nest the Fluent Assertion inside of the Verify statement as follows:
myMock.Verify(m => 
	m.Method(It.Is<MyData>(actual => 
		FluentVerifier.VerifyFluentAssertion(() => 
			actual.Should()
			.BeEquivalentTo(expected, cfg => cfg.Excluding(p => p.DateCreated), ""))));
Note however that since Lambda expressions can't contain calls to methods with optional parameters, you must specify the "becauseArgs" parameter of the "BeEquivalentTo" method.

Test Data Definition Framework for .NET

August 20, 2021

I have just released the initial version of my new open source project which is designed to allow one set of integration tests to run against in-memory fakes and against a "real" repository using only pre-compiler directives. This is useful when you have a suite of SpecFlow tests that you want to quickly run locally whilst developing (e.g. with NCrunch) and on a cloud build server (e.g. GitHub) where you don't always want to hit "real" data resources, but want the flexibility of occasionally switching to "real data mode" and running the same set of tests against a real MongoDB or SQL server (for example). The initial version has one backing store implementation, for MongoDB, but I'm hoping by making this open source other backing stores can be added over time. You can read more about it on the GitHub page here: https://github.com/craigwardman/TestDataDefinitionFramework

Run Windows Command from WSL Bash

August 18, 2021

Since the advent of WSL I spend most of my time using Bash to perform my CLI tasks in Windows. Sometimes however, I'd like to run a command as though I were running it in CMD.. (I'm not talking about wslview though). The example that springs to mind is when starting a dotnet web application, where I'd like it to bind on the Windows IP/Port not on the WSL one. So although I could run "dotnet run" from Bash, I actually want to run "dotnet run" from Windows (with minimal effort of course) For this I've created a bash alias called "winrun" which looks as follows:
alias winrun='cmd.exe /c start cmd /k'
So now if I'm in Bash and want to run a dotnet project I just type:
winrun dotnet run

Cloning a Visual Studio Project With A New Name

July 21, 2021

Often when I'm working on solutions I have a lot of projects that are all pretty much starting from the same place as another. For example, "unit test project targeting .net5, with FluentAssertions, Moq, Autofixture" - but also sometimes projects with a similar class structures too like a "repository that talks to MongoDB, with an ISomethingConfig", or a "Lambda with DI container". Of course I could create Visual Studio templates for these, but often templates fall out of date or I'd end up still needing to copy in some specific classes from a recently used project that I want to replicate. In a lot of cases I found myself copying and pasting and entire project folder, renaming the files accordingly then going through the csproj/cs files to replace the namespace and/or class names to target my "new" project. To automate this I wrote a Bash script, which might not be the most elegant or robust script in the world, but it does the job so I thought I'd share it here:
# Script to create a new Visual Studio project folder from an existing one
# Usage: ./csprojclone.sh Your.Namespaced.Source Your.Namespaced.Destinationn

if [ $# -lt 2 ]
	then
		echo "Please pass source and destination namespace arguments"
		return -1
fi
source=${1%%/}
dest=${2%%/}

if [[ ! -d $source ]]
	then
		echo "Source directory does not exist or isn't a folder - make sure you are in the correct working directory"
		return -1
fi

if [[ -e $dest ]]
        then
                echo "Destination folder already exists in the working directory"
                return -1
fi

oldproject=${source%%.UnitTests}
oldproject=${oldproject##*.}
newproject=${dest%%.UnitTests}
newproject=${newproject##*.}

cp -r $source $dest
rm -r $dest/bin/
rm -r $dest/obj/
find $dest -iname "*$source*" -exec rename -v "s/$source/$dest/" {} \;
find $dest -iname "*$oldproject*" -exec rename -v "s/$oldproject/$newproject/" {} \;
find $dest -type f -print0 | xargs -0 sed -i "s/$source/$dest/g"
find $dest -type f -print0 | xargs -0 sed -i "s/$oldproject/$newproject/g"
As per the script, the usage is to call the script from within your solution directory, passing in the name of the existing project folder and then a destination one. The script will clone the source, rename the files, search and replace the namespace change and update any classes that had the specific project name in. If you're using it on a unit tests project, it will strip of ".UnitTests" from the path - so if that's not your naming convention then feel free to edit those bits. Here's an example of how it might work: PWD = /path/to/SolutionFolder
    My.Namespace.UserRepository 
    My.Namespace.UserRepository/My.Namespace.UserRepository.csproj
    My.Namespace.UserRepository/MongoStuff/...
    My.Namespace.UserRepository/IUserRepositoryConfig.cs
    etc..
. /path/to/script/csprojclone.sh My.Namespace.UserRepository My.Namespace.OrderRepository
Will create:
    My.Namespace.OrderRepository
    My.Namespace.OrderRepository/My.Namespace.OrderRepository.csproj
    My.Namespace.OrderRepository/MongoStuff/...
    My.Namespace.OrderRepository/IOrderRepositoryConfig.cs
With all namespaces also updated.

Easy Way To Test All Constructor Null Argument Checks

February 09, 2021

Often C# classes will have several dependencies passed into the constructor and some (maybe all) of these will be mandatory for the class to function. In this case, ordinarily you'd add a null argument check in the ctor and if something is null throw an "ArgumentNullException". This is pretty boiler plate and is usually auto-generated code, but still it needs to be tested in your unit tests to assert that indeed all mandatory options have been checked (and conversely that all optional dependencies can indeed be null). This can be quite tedious and repetitive and later changing the signature of the ctor can result in many tests that requiring fixing up. To make these tests smaller and more concise I've come up with a new strategy using test cases and nullable mocks, as follows:
// SomeClass.cs
public class SomeClass
{
	private readonly IDependency1 _dependency1;
	private readonly IDependency2 _dependency2;
	private readonly IDependency3 _dependency3;

	public SomeClass(IDependency1 dependency1, IDependency2 dependency2, IDependency3 dependency3)
	{
		_dependency1 = dependency1 ?? throw new ArgumentNullException(nameof(dependency1));
		_dependency2 = dependency2 ?? throw new ArgumentNullException(nameof(dependency2));
		_dependency3 = dependency3 ?? throw new ArgumentNullException(nameof(dependency3));
	}
}

// SomeClass.tests.cs
[TestFixture]
public class SomeClassTests
{
	private Mock<IDependency1> _dependency1Mock;
	private Mock<IDependency2> _dependency2Mock;
	private Mock<IDependency3> _dependency3Mock;

	[SetUp]
	public void SetUp()
	{
		_dependency1Mock = new Mock<IDependency1>();
		_dependency2Mock = new Mock<IDependency2>();
		_dependency3Mock = new Mock<IDependency3>();
	}

	[TestCase("dependency1")]
	[TestCase("dependency2")]
	[TestCase("dependency3")]
	public void Ctor_RequiredDependencyNull_ThrowsException(string dependency)
	{
		var setup = new Dictionary<string, Action>
		{
			{"dependency1", () => _dependency1Mock = null },
			{"dependency2", () => _dependency2Mock = null },
			{"dependency3", () => _dependency3Mock = null }
		};
		setup[dependency]();

		Func<SomeClass> act = GetDefaultSut;

		act.Should().Throw<ArgumentNullException>().And.ParamName.Should().Be(dependency);
	}

   private SomeClass GetDefaultSut()
	{
		return new SomeClass(_dependency1Mock?.Object, _dependency2Mock?.Object, _dependency3Mock?.Object);
	}
}
My above example is for NUnit, using FluentAssertions and Moq but can be converted to your testing tools of choice. An even easier option, when all your dependencies are required is to use the "GuardClauseAssertion" from "AutoFixture.Idioms"
private readonly Fixture _fixture;

public SomeClassTest(){
     _fixture = new Fixture();
     _fixture.Customize(new AutoMoqCustomization());
}

[Test]
public void Ctor_NullDependency_Throws()
{
        new GuardClauseAssertion(_fixture)
                .Verify(typeof(SomeClass).GetConstructors());
}