This is the second part of a multi-part post series I’m going to write about Code Generation using C#. In the previous part I’ve shown how to invoke C# Scripts (and CSX files) from PowerShell and Visual Studio. In this part, I’ll create a simple extractor to read the physical schema of a SQL Server Database and save it as a JSON file.

Creating the Project

The first step is creating a .NET project.

Running CSX scripts do not require csproj (or Visual Studio), but it’s very helpful to have a csproj for many reasons:

  • You get intellisense (mostly on CS, not so much on CSX - so try to use CSX as little as possible)
  • You can check for compilation-errors
  • You can configure a Console App (or any other entry point) to launch and debug your scripts
  • csproj is required for automatically restoring NuGet packages (although if you already have the NuGet packages you may just use them without csproj)

All libraries which I’ll use in my generator (Dapper and Newtonsoft) are compatible with netstandard2.0.
This means that the project type can be either .NET Framework (net472, which used to be called “full framework”) or .NET Core (netcore21 or netcore31) - both should work.

NuGet Packages

I’ll use two third-party libraries: Dapper (micro-ORM, for loading objects from SQL Queries) and Newtonsoft.Json (JSON library, just for serialization). All we need is their dlls, but the easiest way to getting those is by installing their NuGet packages to our csproj (which will automatically install those packages to anyone who tries to build our csproj, so it’s hassle-free collaboration among multiple developers).
If you’re not using Visual Studio IDE you can install those packages by running
dotnet add package Dapper and dotnet add package Newtonsoft.Json
in the same folder of your csproj.
If you’re using Visual Studio you can use the Package Manager Console and run this:

Install-Package Dapper
Install-Package Newtonsoft.Json

Sql Server Schema objects

Next, I’ll create my objects that will represent my physical SQL database. One important thing to remember is that C# scripting engines do NOT allow the use of namespaces.

SqlServerTable.cs:

using System;
using System.Collections.Generic;

public class SqlServerTable
{
    public string Database { get; set; }
    public string TableSchema { get; set; }
    public string TableName { get; set; }

    /// <summary>
    /// Can be "TABLE" or "VIEW"
    /// </summary>
    public string TableType { get; set; }

    public string TableDescription { get; set; }

    public List<SqlServerColumn> Columns { get; set; } = new List<SqlServerColumn>();

    /// <summary>
    /// FKs which point from THIS (Child) table to the primary key of OTHER (Parent) tables
    /// </summary>
    public List<SqlServerForeignKey> ForeignKeys { get; set; } = new List<SqlServerForeignKey>();

    /// <summary>
    /// FKs which point from OTHER (Child) tables to the primary key of THIS (Parent) table
    /// </summary>
    public List<SqlServerForeignKey> ChildForeignKeys { get; set; } = new List<SqlServerForeignKey>();

}

I’ll omit other classes for brevity, but you can refer to all code here (classes SqlServerColumn.cs, SqlServerForeignKey.cs, SqlServerForeignKeyMember.cs ).

Schema Reader

Finally, I’ll create the class which will read SQL metadata - SqlServerSchemaReader.cs:

using Dapper;
using System;
using System.Data;
using System.IO;
using System.Linq;

public class SqlServerSchemaReader
{
  public Func<IDbConnection> CreateDbConnection { get; set; }

  public SqlServerSchemaReader(Func<IDbConnection> createDbConnection)
  {
    CreateDbConnection = createDbConnection;
  }

  public void ExportSchemaToJSON(string outputJsonSchema)
  {
    Console.WriteLine("Reading Database...");

    using (var cn = CreateDbConnection())
    {
      var tables = cn.Query<SqlServerTable>(@"
        SELECT 
          t.TABLE_CATALOG as [Database], 
          t.TABLE_SCHEMA as [TableSchema], 
          t.TABLE_NAME as [TableName], 
          CASE WHEN t.TABLE_TYPE='VIEW' THEN 'VIEW' ELSE 'TABLE' END as [TableType],
          ep.value as [TableDescription]
		    FROM  INFORMATION_SCHEMA.TABLES t
		    INNER JOIN sys.schemas sc ON t.TABLE_SCHEMA = sc.[name]
          ... full code omitted for brevity - please refer to: 
          ... https://github.com/Drizin/CodegenCS/tree/master/src/CodegenCS.SqlServer
      ").AsList();

      var allColumns = cn.Query<SqlServerColumn>(@"
          ... full code omitted for brevity... 
      ").AsList();

      var fks = cn.Query<SqlServerForeignKey>(@"
          ... full code omitted for brevity... 
      ").AsList();

      var fkCols = cn.Query<SqlServerForeignKeyMember>(@"
          ... full code omitted for brevity... 
      ").AsList();
      foreach (var fk in fks)
      {
        fk.Columns = fkCols.Where(c => 
            c.ForeignKeyConstraintName == fk.ForeignKeyConstraintName && 
            c.FKTableSchema == fk.FKTableSchema
        ).OrderBy(c => c.PKColumnOrdinalPosition).ToList();
      }

      foreach (var table in tables)
      {
        table.Columns = allColumns.Where(c => c.TableSchema == table.TableSchema && c.TableName == table.TableName).ToList();
        foreach(var column in table.Columns)
          column.ClrType = GetClrType(table, column);
        table.Columns.ForEach(c => { c.Database = null; c.TableSchema = null; c.TableName = null; });

        // We copy FKs and remove redundant properties of the parent object (table) which we're attaching this FK into
        table.ForeignKeys = Clone(fks.Where(fk => fk.FKTableSchema == table.TableSchema && fk.FKTableName == table.TableName).ToList());
        table.ForeignKeys.ForEach(fk => { fk.FKTableSchema = null; fk.FKTableName = null; });

        // We copy FKs and remove redundant properties of the parent object (table) which we're attaching this FK into
        table.ChildForeignKeys = Clone(fks.Where(fk => fk.PKTableSchema == table.TableSchema && fk.PKTableName == table.TableName).ToList());
        table.ChildForeignKeys.ForEach(fk => { fk.PKTableSchema = null; fk.PKTableName = null; });

      }

      SqlServerDatabaseSchema schema = new SqlServerDatabaseSchema()
      {
        LastRefreshed = DateTimeOffset.Now,
        Tables = tables,
      };

      Console.WriteLine($"Saving into {outputJsonSchema}...");
      File.WriteAllText(outputJsonSchema, Newtonsoft.Json.JsonConvert.SerializeObject(schema, Newtonsoft.Json.Formatting.Indented));
    }

    Console.WriteLine("Success!");
  }

  string GetClrType(SqlServerTable table, SqlServerColumn column)
  {
    string sqlDataType = column.SqlDataType;
    switch (sqlDataType)
    {
      case "bigint":
        return typeof(long).FullName;
      case "smallint":
        return typeof(short).FullName;
      case "int":
        return typeof(int).FullName;
      case "uniqueidentifier":
        return typeof(Guid).FullName;
      case "smalldatetime":
      case "datetime":
      case "datetime2":
      case "date":
      case "time":
        return typeof(DateTime).FullName;
      case "datetimeoffset":
        return typeof(DateTimeOffset).FullName;
      case "float":
        return typeof(double).FullName;
      case "real":
        return typeof(float).FullName;
      case "numeric":
      case "smallmoney":
      case "decimal":
      case "money":
        return typeof(decimal).FullName;
      case "tinyint":
        return typeof(byte).FullName;
      case "bit":
        return typeof(bool).FullName;
      case "image":
      case "binary":
      case "varbinary":
      case "timestamp":
        return typeof(byte[]).FullName;
      case "nvarchar":
      case "varchar":
      case "nchar":
      case "char":
      case "text":
      case "ntext":
      case "xml":
        return typeof(string).FullName;
      default:
        Console.WriteLine($"Unknown sqlDataType for {table.TableName}.{column.ColumnName}: {sqlDataType}");
        return null;

      // Vendor-specific types
      case "hierarchyid":
        return "Microsoft.SqlServer.Types.SqlHierarchyId"; // requires Microsoft.SqlServer.Types.dll (EF or Dapper 1.34+)
      case "geography":
        return "Microsoft.SqlServer.Types.SqlGeography";  // requires Microsoft.SqlServer.Types.dll (EF or Dapper 1.32+)
      case "geometry":
        return "Microsoft.SqlServer.Types.SqlGeometry";  // requires Microsoft.SqlServer.Types.dll (EF or Dapper 1.33)+
    }
  }

  public static T Clone<T>(T source)
  {
    var serialized = JsonConvert.SerializeObject(source);
    return JsonConvert.DeserializeObject<T>(serialized);
  }

}

CSX

The idea is that we should put as much as possible into .cs files (and as little as possible in csx script), since intellisense and compile-time checks work better than in the CSX. The script file RefreshDatabaseSchema.csx will basically load libraries, define connection strings and paths, and invoke SqlServerSchemaReader.cs:

/// <summary>
/// This CSX Script will invoke SqlServerSchemaReader, which extracts the schema of SQL database and saves into a JSON file.
/// The easiest way to launch csi.exe (which is shipped with Visual Studio) to run this script is by using PowerShell script RefreshDatabaseSchema.ps1
/// You can do that from Visual Studio (see instructions in RefreshDatabaseSchema.ps1) or you can just execute "Powershell RefreshDatabaseSchema.ps1"
/// </summary>

// System libraries
#r "System.Data.dll"

// Load 3rd-party libraries by their relative paths, relative to "$Env:userprofile\.nuget\packages\"
#r "dapper\2.0.35\lib\netstandard2.0\Dapper.dll"
#r "newtonsoft.json\12.0.3\lib\netstandard2.0\Newtonsoft.Json.dll"

// CS files are better than CSX because Intellisense and Compile-time checks works better. 
#load "SqlServerTable.cs"
#load "SqlServerColumn.cs"
#load "SqlServerForeignKey.cs"
#load "SqlServerForeignKeyMember.cs"
#load "SqlServerDatabaseSchema.cs"
#load "SqlServerSchemaReader.cs"

using System;
using System.IO;
using System.Runtime.CompilerServices;
using System.Data;
using System.Data.SqlClient;

// Helpers to get the location of the current CSX script
public static string GetScriptPath([CallerFilePath] string path = null) => path;
public static string GetScriptFolder([CallerFilePath] string path = null) => Path.GetDirectoryName(path);


// location relative to the CSX script
string outputJsonSchema = Path.GetFullPath(Path.Combine(GetScriptFolder(), "AdventureWorksSchema.json")); 
string connectionString = @"Data Source=MYWORKSTATION\SQLEXPRESS;
                            Initial Catalog=AdventureWorks;
                            Integrated Security=True;";

Func<IDbConnection> connectionFactory = () => new SqlConnection(connectionString);
var reader = new SqlServerSchemaReader(connectionFactory);
reader.ExportSchemaToJSON(outputJsonSchema);

Powershell

Last, I’ll create a PowerShell to invoke the CSX file, which is useful because it can locate the csi.exe in multiple locations and because it can provide to csi.exe the location of per-user NuGet packages, so that CSX can load libraries by their relative-locations, without having to hard-code user-specific folders. RefreshDatabaseSchema.ps1:

# To Execute Powershell Scripts from Visual Studio:
# 1) Right-button PS1 file - "Open With...""
# 2) Configure:
#      Program: Powershell.exe
#      Arguments: -noexit -File %1
#      Friendly Name: Execute PowerShell Script

# To execute CSX scripts you'll need CSI.EXE (C# REPL) which is shipped with Visual Studio
# but can also be installed by using the NuGet package Microsoft.Net.Compilers.Toolset - https://www.nuget.org/packages/Microsoft.Net.Compilers.Toolset/

# For more info about launching CSX scripts from PowerShell or from Visual Studio, check https://drizin.io/code-generation-csx-scripts-part1/

$dir = Split-Path $MyInvocation.MyCommand.Path 
$script = Join-Path $dir ".\RefreshDatabaseSchema.csx"


# Locate CSI.EXE by searching common paths
$csi = ( 
    "$Env:userprofile\.nuget\packages\microsoft.net.compilers.toolset\3.6.0\tasks\net472\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Roslyn\csi.exe"
) | Where-Object { Test-Path $_ } | Select-Object -first 1

if (!$csi)
{
    Write-Host "---------------------------------------" -for red
    Write-Host "Can't find csi.exe" -for red
    Write-Host "Please fix search paths above, or install NuGet Microsoft.Net.Compilers.Toolset" -for red
    Write-Host "---------------------------------------" -for red
    Exit 1
}


$stopwatch = [System.Diagnostics.Stopwatch]::StartNew()

Write-host "Starting csi.exe $script ..." -for yellow
& $csi /lib:"$Env:userprofile\.nuget\packages\" $script

Write-Host "Finished in $($stopwatch.Elapsed.TotalMilliSeconds) milliseconds"

# Since I configured "-noexit" parameter in Visual Studio I don't need this
#if ($host.Name -notmatch 'ISE') { Write-Host -NoNewLine "(Just press Enter to exit)" -for cyan; read-host; }  

Running Powershell:

Result file AdventureWorksSchema.json:

Full Source code available here

This is the second part of a multi-part post series I’m going to write about Code Generation using C#:

This is the first part of a multi-part post series I’m going to write about Code Generation using C#.

I’ve mentioned before that I’ve been a fan of code generation for a long-time. I have used CodeSmith generator, MyGeneration Code Generator, and in the past few years, I’ve been using T4 templates.

The main advantage of code generation (which includes DB scaffolding, also called database first by some ORMs) is that it does the work for you, is repeatable, and less error-prone than manually writing everything. So it’s mostly about productivity and consistency. If you’re writing repetitive code by hand, you’re stealing from your employer or clients. And if you have a data-oriented application, almost certainly you have repetitive code to some degree.

The right tool for the job

Until recently the gold standard for code generation in Microsoft stack was using T4 templates, which is the out-of-the-box tool provided by Microsoft and shipped with Visual Studio. The major problem with T4 is that it has terrible syntax, terrible tooling and debugging support, and can get as ugly as this:

C# Script Files (CSX files)

In the search of a modern code-generation tool I’ve explored Razor Templates, Mustache, and Mustache-like templates (like DotLiquid and Handlebars), and others. I’ve noticed that there’s a growing popularity of using C# code (Roslyn or not) to do the code-generation using pure C# - which makes sense, since you can get strong typing, compile-time checking, full IDE support (with debugging), cross-platform (dotnet core), full access to all .NET Framework (SqlServer, Dapper, Newtonsoft JSON, etc). So you get a full-fledged language instead of using a templating-engine which only offers a subset of features of the underlying language.

C# Script Files (CSX) were introduced with Roslyn, and can be executed in Roslyn or in other compatible cross-platform scripting engines like dotnet-script or even with C# REPL called csi.exe. Those scripting engines certainly have some limitations (like using namespaces), but they allow us to virtually invoke any C# code, with essential features like loading external assemblies, loading code organized across multiple source files, etc. And it’s much easier than using Powershell to invoke C#.

Sample CSX Script

CSX scripts inside Visual Studio have some support for Intellisense (auto-completion) and compile-time checks, but those features work much better in CS files. So it’s a good idea to put as much as possible into cs files and as little as possible in CSX scripts. I like to use CSX only for basic things like loading libraries, setting connection strings, settings paths, and invoking the real code in CS files.

MyProgram.cs:

public class MyProgram
{
   public void MyMethod()
   {
      Console.WriteLine("Hello from MyMethod");
   }  
}

MyScript.csx:

#load "MyProgram.cs" 

new MyProgram().MyMethod(); 
Console.WriteLine("Hello Code-Generation!");

Running CSX Script using C# REPL (CSI.EXE)

Visual Studio ships with a command line REPL called CSI that can be used to run .csx scripts.

You can run CSI.EXE directly from Visual Studio Developer Command Prompt (csi MyScript.csx):

Assembly References

In the same sense that it’s a good idea to use simple statements in CSX to invoke more complex CS code, it’s also a good idea to load external assemblies when you can rely on existing libraries.

CSX allows loading assembly references by using the #r directive in the top of your scripts:

// CSI.EXE requires absolute paths for loading external assemblies: 
#r "C:\Users\drizin\.nuget\packages\dapper\2.0.35\lib\netstandard2.0\Dapper.dll" 

#load "File1.cs" 
#load "File2.cs" 
#load "MyProgram.cs" 

new MyProgram().MyMethod(); 
Console.WriteLine("Hello Code-Generation!");

NuGet Packages

If you need to reference a NuGet package, you can just rely on NuGet tools (and Visual Studio build process) to automatically restore the packages required by your script. For achieving that, you can just add the CSX as part of a Visual Studio project, so when each developer tries to build the project Visual Studio will download the missing packages, and the developer just needs to fix the assemblies location.

Another alternative, which does not require to use csproj at all, is using PowerShell to download the required nugets (see script at the end of this post).

Invoking C# REPL (CSI running CSX scripts) from PowerShell

Although you can run CSI.exe directly from Visual Studio Developer Command Prompt, invoking it through PowerShell is very helpful for a few reasons:

  • You can run outside of Visual Studio. You don’t even need Visual Studio to run CSX.
  • Allows us to find NuGet locations and reference external assemblies with relative paths on the CSX (more about this below).
  • Restoring NuGet files (example script at the end of the post)

To invoke CSI using Powershell, we must know the location of csi.exe.

CSI is shipped with Visual Studio but can also be installed by using the NuGet package Microsoft.Net.Compilers.Toolset (warning: as described here the installation script for the package changes .csproj file to use the compiler provided by the package - this is likely not what you want, so make sure to roll-back such changes if they are made during installation).

So the first step is to search for csi.exe in multiple locations as I show in the sample Powershell script RunMyScript.ps1 below:

# Locate CSI.EXE by searching common paths
$csi = ( 
    "$Env:userprofile\.nuget\packages\microsoft.net.compilers.toolset\3.6.0\tasks\net472\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Roslyn\csi.exe"
) | Where-Object { Test-Path $_ } | Select-Object -first 1

$dir = Split-Path $MyInvocation.MyCommand.Path 
$script = Join-Path $dir "MyScript.csx"

& $csi $script

To launch the PowerShell script from the command line, it’s just about running Powershell Full-Path-To-Your-Script-ps1.

Running from Visual Studio IDE

To run from Visual Studio, you can just add the PS1 to your project or solution, right-click the file, and click the option “Open with PowerShell ISE”, which is the IDE for editing/running PowerShell scripts.

Another alternative is that you can add new actions to your right-button actions - you can click “Open With…” and configure PowerShell to be executed directly from Visual Studio:

The list of possible actions will include this new option of invoking PS1 scripts directly from the IDE, and you can also set this as the default action for opening PS1 files.

Allowing Unsigned Scripts

If you have never executed unsigned PowerShell scripts you’ll have to enable PowerShell unsigned scripts by running Powershell as Administrator and running this command:
Set-ExecutionPolicy -ExecutionPolicy Unrestricted.
Don’t forget to enable for both Windows PowerShell (64-bits) and for Windows PowerShell (x86), which is the one that is invoked from inside Visual Studio IDE.

Relative Assembly References

As we’ve seen before, CSX accepts absolute references like this:

#r "C:\Users\drizin\.nuget\packages\dapper\2.0.35\lib\netstandard2.0\Dapper.dll"

One of the major problems with CSI is that the #r directive (for loading assembly references) doesn’t accept nuget-like references or environment variables so all assembly references should be specified with full paths. This is not a showstopper but it’s a little annoying since it makes it harder to share code among multiple developers since each developer would have to fix their references.

One of the advantages of using PowerShell (as described above) is that we can use environment-variables and use #r directive with relative paths. In the PowerShell script, we just have to locate the base path where your assemblies are located and pass that to CSI so it can use this folder to search for referenced assemblies, like this:

$assemblies = "${env:userprofile}\.nuget\packages\";
& $csi /lib:"$assemblies" $script

And then in the CSX, you can use relative paths like this:

#r "dapper\2.0.35\lib\netstandard2.0\Dapper.dll"

PackageReference (NuGet 4) vs packages.config (NuGet 3)

The new MSBuild format (“SDK-Style”, which uses PackageReference inside the csproj) installs the NuGet packages in this per-user folder.

The old MSBuild format (“non-SDK-Style”, before Visual Studio 2017, which uses packages.config) installs the NuGet packages in the “packages” folder under the Solution folder.

We can adjust our PowerShell scripts according to where our project will restore NuGet packages:

$csi = ... # (locate your csi.exe)
$dir = Split-Path $MyInvocation.MyCommand.Path 
$script = Join-Path $dir "MyScript.csx"

# Call csi.exe and specify that libraries referenced by #r directives 
# should search in a few nuget locations

# New NuGet 4.0+ (PackageReference) saves User-specific packages
# in "%userprofile%\.nuget\packages\"
$nuget1 = "${env:userprofile}\.nuget\packages\";

# New NuGet 4.0+ (PackageReference) saves Machine-wide packages 
# in "%ProgramFiles(x86)%\Microsoft SDKs\NuGetPackages\"
$nuget2 = "${env:ProgramFiles(x86)}\Microsoft SDKs\NuGetPackages\";

# Old NuGet (packages.config) saves packages in "\packages" folder at solution level.
# Locate by searching a few levels above
$nuget3 = ( 
    (Join-Path $dir ".\packages\"),
    (Join-Path $dir "..\packages\"),
    (Join-Path $dir "..\..\packages\"),
    (Join-Path $dir "..\..\..\packages\"),
    (Join-Path $dir "..\..\..\..\packages\")
) | Where-Object { Test-Path $_ } | Select-Object -first 1

# if you're using new NuGet format (PackageReference defined inside csproj) 
& $csi /lib:"$nuget1" $script  

# if you're using old NuGet format (packages.config)
# & $csi /lib:"$nuget3" $script  

And our CSX would use relative references:

// CSX can load libraries by defining their relative paths

// New NuGets (PackageReference) are installed under "${env:userprofile}\.nuget\packages\" 
// or "${env:ProgramFiles(x86)}\Microsoft SDKs\NuGetPackages\")
// and have this format:
#r "dapper\2.0.35\lib\netstandard2.0\Dapper.dll"

// Old NuGets (packages.config) are installed under "(SolutionFolder)\packages"
// and have this format
// #r "Dapper.2.0.35\lib\netstandard2.0\Dapper.dll"

//...
new MyProgram().MyMethod();
Console.WriteLine("Hello Code-Generation!");

So cool and so easy, isn’t it?

Other tools

I have played with other tools like dotnet-script, nake, and the popular scriptcs. For different reasons I couldn’t make any of them work fine (some weren’t even installing in a traditional .net framework project, some had complex methods for using nugets, and some simply weren’t working (not finding include files, etc)). So I decided to stick with the plain csi.exe, which most users will already have installed.

Final PowerShell Script

This script below has some nice features:

  • Allows to search for assemblies in multiple paths, including all NuGet locations
  • Can pass multiple paths to csi.exe, even though csi does not accept spaces in paths
  • Can restore missing NuGet packages (will even download nuget.exe)
# To Execute Powershell Scripts from Visual Studio:
# 1) Right-button PS1 file - "Open With...""
# 2) Configure:
#      Program: Powershell.exe
#      Arguments: -noexit -File %1
#      Friendly Name: Execute PowerShell Script

# To execute CSX scripts you'll need CSI.EXE (C# REPL) which is shipped with Visual Studio
# but can also be installed by using the NuGet package Microsoft.Net.Compilers.Toolset - https://www.nuget.org/packages/Microsoft.Net.Compilers.Toolset/

# For more info about launching CSX scripts from PowerShell or from Visual Studio, check https://drizin.io/code-generation-csx-scripts-part1/

$dir = Split-Path $MyInvocation.MyCommand.Path 
$script = Join-Path $dir ".\GenerateSimplePOCOs.csx"
$requiredLibs = @(
    @{ Name = "Newtonsoft.Json"; Version = "12.0.3" },
    @{ Name = "CodegenCS"; Version = "1.0.5" }
);

# By default we'll only use NuGet 4 locations. But you can change to 3 if you're hosting 
# your scripts in a project with the old packages.config format and want to rely on existing project packages
$NuGetVersion = 4; 



$ErrorActionPreference = "Stop"

# Locate CSI.EXE by searching common paths
$csi = ( 
    "$Env:userprofile\.nuget\packages\microsoft.net.compilers.toolset\3.6.0\tasks\net472\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Roslyn\csi.exe",
    "$Env:programfiles (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Roslyn\csi.exe"
) | Where-Object { Test-Path $_ } | Select-Object -first 1

if (!$csi)
{
    Write-Host "---------------------------------------" -for red
    Write-Host "Can't find csi.exe" -for red
    Write-Host "Please fix search paths above, or install NuGet Microsoft.Net.Compilers.Toolset" -for red
    Write-Host "---------------------------------------" -for red
    Exit 1
}
Write-Host "Found csi.exe: $csi" -for cyan

# List of locations to search for assembly references
$libPaths = @()
$libPaths += $dir

if ($NuGetVersion -eq 4)
{
    # New NuGet 4.0+ (PackageReference) saves User-specific packages in %userprofile%\.nuget\packages\
    $libPaths += "${env:userprofile}\.nuget\packages";
    if (Test-Path "${env:userprofile}\.nuget\packages") { $missingNuGetPackagesLocation = "${env:userprofile}\.nuget\packages" }

    # New NuGet 4.0+ (PackageReference) saves Machine-wide packages in %ProgramFiles(x86)%\Microsoft SDKs\NuGetPackages\"
    $libPaths += "${env:ProgramFiles(x86)}\Microsoft SDKs\NuGetPackages";
}

if ($NuGetVersion -eq 3)
{
    # Old NuGet (packages.config) saves packages in "\packages" folder at solution level.
    # Locate by searching a few levels above the script
    $missingNuGetPackagesLocation = ( 
        (Join-Path $dir ".\packages"),
        (Join-Path $dir "..\packages"),
        (Join-Path $dir "..\..\packages"),
        (Join-Path $dir "..\..\..\packages"),
        (Join-Path $dir "..\..\..\..\packages")
    ) | Where-Object { Test-Path $_ } | Select-Object -first 1
    $libPaths += $missingNuGetPackagesLocation
}

# where to download missing NuGet packages
if ((Test-Path $missingNuGetPackagesLocation) -eq $false)
{
    $missingNuGetPackagesLocation = $dir
}


# csi /lib parameter allows multiple paths but does not accept spaces (or quotes) so we have to use short DOS 8.3 paths
$fso = New-Object -ComObject Scripting.FileSystemObject
$libPaths = ($libPaths | Where-Object { Test-Path $_ } | ForEach { $fso.GetFolder($_).shortpath  });


Write-Host "CSI will use the following paths to search for assembly references:`r`n   - $($libPaths -Join "`r`n   - ")" -for cyan


$missingLibs = @()
$requiredLibs | foreach {
    $requiredLib = $_;
    Write-Host "Checking for $($requiredLib.Name) version $($requiredLib.Version)..." -for Cyan -NoNewLine

    if ($NuGetVersion -eq 4)
    {
        # NuGet 4+ format
        $found = $libPaths | 
        ForEach { Join-Path $_ ($requiredLib.Name + '\' + $requiredLib.Version) } | 
        Where-Object { Test-Path $_ } | Select-Object -first 1

        if ($found -eq $null)
        {
            Write-Host "`n$($requiredLib.Name) not found. Will install using NuGet" -for Yellow
            $missingLibs += $requiredLib
        }
        else
        {
             Write-Host "Found: $found" -for Cyan
        }
    }

    if ($NuGetVersion -eq 3)
    {
        # NuGet <=3 format
        $found = $libPaths | 
        ForEach { Join-Path $_ ($requiredLib.Name + '.' + $requiredLib.Version) } | 
        Where-Object { Test-Path $_ } | Select-Object -first 1

        if ($found -eq $null)
        {
            Write-Host "`n$($requiredLib.Name) not found. Will install using NuGet" -for Yellow
            $missingLibs += $requiredLib
        }
        else
        {
             Write-Host "Found: $found4 $found3" -for Cyan
        }
    }
}

if ($missingLibs)
{
    $nuget = Join-Path $env:TEMP "nuget.exe"
    if ((Test-Path $nuget) -eq $False)
    {
        Write-Host "Downloading NuGet.exe into $nuget" -for cyan
        $webClient = New-Object System.Net.WebClient 
        $webClient.DownloadFile("https://dist.nuget.org/win-x86-commandline/latest/nuget.exe", $nuget)
    }

    $missingLibs | foreach {
        $missingLib = $_
        Write-Host "Downloading $missingLib...";
        $libName = $missingLib.Name
        $libVersion = $missingLib.Version
        if ($libVersion -eq $null)
        {
            & $nuget install $libName -OutputDirectory $missingNuGetPackagesLocation
        }
        else
        {
            & $nuget install $libName -Version $libVersion -OutputDirectory $missingNuGetPackagesLocation
        }
        if ($lastExitCode -ne 0)
        {
            Write-host "-------------`nError downloading $missingLib - aborting...`n-------------" -for red
            Exit 1
        }
    }
}


$stopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Write-host "Starting csi.exe $script ..." -for yellow
& $csi /lib:$($libPaths -Join ';') $script

$stopwatch.Stop()
Write-Host "Finished in $($stopwatch.Elapsed.TotalMilliSeconds) milliseconds"

# Since I configured "-noexit" parameter in Visual Studio I don't need this
#if ($host.Name -notmatch 'ISE') { Write-Host -NoNewLine "(Just press Enter to exit)" -for cyan; read-host; }  

And my final CSX

/// <summary>
/// This CSX Script will invoke SimplePOCOGenerator, which builds simple POCO classes based on a JSON file with schema of SQL database
/// The easiest way to launch csi.exe (which is shipped with Visual Studio) to run this script is by using PowerShell script GenerateSimplePOCOs.ps1
/// You can do that from Visual Studio (see instructions in RefreshDatabaseSchema.ps1) or you can just execute "Powershell GenerateSimplePOCOs.ps1"
/// </summary>

// System libraries
#r "System.Data.dll"

// Load third-party libraries by their relative paths, relative to "$Env:userprofile\.nuget\packages\"
#r "newtonsoft.json\12.0.3\lib\netstandard2.0\Newtonsoft.Json.dll"
#r "codegencs\1.0.5\lib\netstandard2.0\CodegenCS.dll"

// CS files are better than CSX because Intellisense and Compile-time checks works better. 
#load "DatabaseSchema.cs"
#load "SimplePOCOGenerator.cs"

using System;
using System.IO;
using System.Runtime.CompilerServices;
using System.Data;
using System.Data.SqlClient;

// Helpers to get the location of the current CSX script
public static string GetScriptPath([CallerFilePath] string path = null) => path;
public static string GetScriptFolder([CallerFilePath] string path = null) => Path.GetDirectoryName(path);


// locations relative to the CSX script
string inputJsonSchema = Path.GetFullPath(Path.Combine(GetScriptFolder(), "AdventureWorksSchema.json"));
string targetFolder = Path.GetFullPath(Path.Combine(GetScriptFolder(), @".\POCOs\"));
string csProj = Path.GetFullPath(Path.Combine(GetScriptFolder(), @".\CSX-PowerShell-VisualStudio2015.csproj"));

var generator = new SimplePOCOGenerator(inputJsonSchema);
generator.Namespace = "CSX_PowerShell_VisualStudio2015";
generator.Generate(targetFolder, null);

Full source code is available here, with a sample project that reads a JSON file with AdventureWorks database schema and generates POCOs, both for SDK-Style (new csproj format) and for non-SDK-style (old Visual Studio 2015 format).

So cool and so easy, isn’t it?

In my next post, I’ll create a SQL schema extractor, and this schema will be used in next posts to build code generator for POCOs, EFCore, and other related services.

Enjoy!

This was the first part of a multi-part post series I’m going write do about Code Generation using C#:

I’ve been a fan of code generation (mostly for scaffolding data access objects from databases) for a long-time. I have used CodeSmith generator, MyGeneration Code Generator, LINQ to SQL, Rockford Lhotka CSLA, Subsonic Active Record Templates, and more recently Simon Hughes Entity Framework T4 templates.

The main advantage of code generation (which includes “db scaffolding”, also known as database first in some ORMs) is that it does the work for you, is repeatable, and less error-prone than manually writing everything. So it’s mostly about productivity and consistency. If you’re writing repetitive code by hand, you’re stealing from your employer or from your client. And if you have a data-oriented application, almost certainly you have repetitive code to some degree.

I frequently need to make some customizations over the code generation templates, and it’s always a hard task. Most code generators do not have good support for one or more of the following important features:

  • Indentation and whitespace control
  • Debugging
  • Includes (subtemplates)
  • Intellisense (code completion)
  • Compile-time checking (strong typing)
  • Integration with CSPROJ files

Among those, I had one important requirement: the included subtemplates should respect parent indentation, and even if they were multi-line templates those lines should be rendered in the correct position. Or as handlerbars describe , “By default, an indented partial-call causes the output of the whole partial being indented by the same amount.”.

While trying to find a good code generator (to replace what I currently do with T4 Templates) I explored the most popular free tools/libraries for code generation and text templating.

T4 Templates

T4 templates are based on ASPX control blocks <% %>. They tend to get very ugly, have low degree of intellisense/typing (it’s easy to write invalid code and you can only find out debugging which is even harder), have terrible support for indentation, and everytime that I upgrade my Visual Studio the T4 templates totally break, since they are very environment dependent.

And they look as ugly as this:

Razor Engines

Antaris RazorEngine is the top-leading razor engine, but there are other razor engines like RazorLight and WestWind.RazorHosting.

Razor is more concise (and less ugly) than ASPX control blocks, but for achieving concise code they have to make some “assumptions” about what is control logic and what is output code, and therefore it tries to automagically guess C# code that is intended to control logic so if you write Debug.WriteLine it will assume that this should be interpreted (and not written to output).
If you type an XML tag inside a code block it will automagically assume that it should be written to output (which makes sense, afterall XML tags are not control logic, but yet this gets very confusing). So to make it work you end up using escape characters like @, @{} and @: everywhere.

Razor templates are nice because they have good Visual Studio support (with intellisense and object models, although those unclear distintions between control logic and output code usually make intellisense fail), and allow partials, but unfortunately for having intellisense in partials you would have to split them into indivual files.

RazorEngine is very complete, very stable, and somehow easy to extend, but they have some minor annoyances like saving temporary DLLs (which don’t get deleted) everywhere.
But the major problems with all Razor-based engines is that Razor uses angle-based tags, and therefore is targeted at writing XML/HTML, and a little annoying to write other outputs. Additionally, the Partials (includes) do NOT respect parent indentation level - so if you include a multi-line subtemplate, you’ll get a messy indentation.

Scripty

Scripty describes itself as Tools to let you use Roslyn-powered C# scripts for code generation. You can think of it as a scripted alternative to T4 templates.
It can manipulate CSPROJ files, has support for both .NET Framework and .NET core, and has good indentation support: it has its own TextWriter which keeps tracks of current indent level, and restores this indent level when you start a new line). It also has great intellisense since it’s scripted using pure C#.
Unfortunately, it does not have integrated templating engine, so writing complex templates can get ugly and cumbersome. Additionally it looks like the author not only does not intend to support any specific templating engine, but also has somehow abandoned the project (and complex open source projects when abandoned usually decline fast unless a new group of supporters fork it into a new project) - so I was a little afraid of putting all my coins into a complex codebase which is not being maintained.

Templating Engines

Currently the most popular templating engines are based on the mustache standard, which is very nice because it uses “double-braces” to define control logic with tags {{ this }}, so you can still use regular single-braces for C# (or Java or any C-Style) code. Although mustache is the first engine using this, it was quickly followed by better alternatives like handlebars.js and Shopify liquid, which are the two most popular engines, and they were both respectively ported to .NET in the libraries Handlebars.Net and Dotliquid. There’s also a new kid on the block called Scriban which is based on Dotliquid but with some improvements, and growing fast.

Although the escape character in all those engines does not conflict with C# code, their syntax is very different from regular C# code (probably easier to parse), so you have to learn a new syntax. Learning a new syntax is not a problem, afterall we all have learned so many different syntaxes over time and old dogs love to learn new tricks, but the problem is that those engines also do NOT have any debugging support, which is a must, specially when you’re not familiar with the syntax, or when you’re writing complex templates. Debugging templates only by the error outputs is like debugging code only with Console.WriteLine when you could be using a real IDE.

Jinja2

Since none of the .NET alternatives fully matched what I was looking for, I also checked the most popular templating engine for non-NET languages.
Jinja2 engine is written in Python, and is similar to Handlebars and Liquid. They have good whitespacing control, good indentation, and a good ecosystem (parsers, editors, etc). However, similarly like Dotliquid and Handlebars.Net, it looks like all those templating tools are NOT focused on code generation but on used-defined-templates (like configuring automatic emails, configuring pages for your ecommerce, etc), so they have a different set of requirements in mind. For those tools it really makes sense to have a simple syntax, a sandboxed security model (isolated AppDomain), and loading templates from strings stored anywhere (like your database). A code generator does not need any of that, but needs good debugging support, good intellisense, and (for those of us who like it) strong typing.
Last, since I’m mostly into C#/.NET, it didn’t look like a good fit to start using a Python engine - especially if I want my templates to be easily adopted by other C# users. I was looking for something similar to T4 templates but with better readability, better tooling, and better reuse.

Writing code using C#

From the tools above, the only one which integrates with msbuild projects (in other words can write into csproj files) is Scripty, so this is where I started my tests. Scripty basically have their own TextWriter (Scripty.Core.Output.OutputFile) and some helpers like a collection of TextWriters for writing to multiple files, and for choosing how each file should be treated by the csproj, etc.
This is Scripty syntax:

w.WriteLine();
w.WriteLine("///<summary>");
w.WriteLine("/// {0}", System.Security.SecurityElement.Escape(c.SummaryComments));
w.WriteLine("///</summary>");

For writing indented code in Scripty you could manually indent your strings like this:

w.WriteLine("finally");
w.WriteLine("{");
w.WriteLine("    dbConnection.Close();");
w.WriteLine("}");

… or you could use their nice indentation helpers which control indentation level for you:

w.WriteLine($"public {Settings.DbContextName}(string connectionString) : base(connectionString)");
w.WriteLine("{");
using (w.WithIndent())
{
    w.WriteLine("InitializePartial();");
}
w.WriteLine("}");

As you can notice, this can get very ugly and cumbersome to write:

w.WriteLine($"{Settings.MigrationClassModifiers} class {Settings.MigrationConfigurationFileName}: System.Data.Entity.Migrations.DbMigrationsConfiguration<{Settings.DbContextName }> ");
w.WriteLine("{");
using (w.WithIndent())
{
    w.WriteLine($"public {Settings.MigrationConfigurationFileName}()");
    w.WriteLine("{");
    using (w.WithIndent())
    {
        w.WriteLine($"AutomaticMigrationsEnabled = { Settings.AutomaticMigrationsEnabled.ToString() };");
        w.WriteLine($"AutomaticMigrationDataLossAllowed = { Settings.AutomaticMigrationDataLossAllowed.ToString() };");
        if (!string.IsNullOrEmpty(Settings.ContextKey))
            w.WriteLine([email protected]"ContextKey = ""{ Settings.ContextKey }"";");
    }
    w.WriteLine("}");
}
w.WriteLine("}");

Pretty ugly, right?

More concise manual indentation

The first helper method that I added to Scripty was something to make manual-indentation less verbose when all I need is to create C-style blocks ( { } ).

With my helpers, creating an indented code block became a little easier because a single statement will both write the line before the block starts, will write the brace which starts the block, will increase indent, and when being disposed it will decrease the indent and will also output the brace which closes the block. So it’s 3 lines (and increasing/decreasing inner indentation) in a single line.

using (w.WithCStyleBlock($"namespace {myNamespace}"))
{
    using (w.WithCStyleBlock($"public class {myClass}"))
    {
         w.WriteLine("// My Properties start here");
    }
}

…this generates this code:

namespace codegencs
{
    public class Test1
    {
        // My Properties start here
    }
}

Another example:

w.WriteLine($"{Settings.MigrationClassModifiers} class {Settings.MigrationConfigurationFileName}: System.Data.Entity.Migrations.DbMigrationsConfiguration<{Settings.DbContextName }> ");
w.WriteLine("{");
using (w.WithIndent())
{
    w.WriteLine($"public {Settings.MigrationConfigurationFileName}()");
    w.WriteLine("{");
    using (w.WithIndent())
    {
        w.WriteLine($"AutomaticMigrationsEnabled = { Settings.AutomaticMigrationsEnabled.ToString() };");
        w.WriteLine($"AutomaticMigrationDataLossAllowed = { Settings.AutomaticMigrationDataLossAllowed.ToString() };");
        if (!string.IsNullOrEmpty(Settings.ContextKey))
            w.WriteLine([email protected]"ContextKey = ""{ Settings.ContextKey }"";");
    }
    w.WriteLine("}");
}
w.WriteLine("}");

Can be simplified into:

using (w.WithCStyleBlock($"{Settings.MigrationClassModifiers} class {Settings.MigrationConfigurationFileName}: System.Data.Entity.Migrations.DbMigrationsConfiguration<{Settings.DbContextName }> ))
{
    using (w.WithCStyleBlock($"public {Settings.MigrationConfigurationFileName}()"))
    {
        w.WriteLine($"AutomaticMigrationsEnabled = { Settings.AutomaticMigrationsEnabled.ToString() };");
        w.WriteLine($"AutomaticMigrationDataLossAllowed = { Settings.AutomaticMigrationDataLossAllowed.ToString() };");
        if (!string.IsNullOrEmpty(Settings.ContextKey))
            w.WriteLine([email protected]"ContextKey = ""{ Settings.ContextKey }"";");
    }
}

As you can see, the boilerplate was condensed into a single statement, and yet the control logic braces/indentation clearly match the output code braces which are implicitally rendered.
(I know that some people will prefer opening braces in the same line - so sorry, you won’t see that helper in my code generator, ever!! :-) just kidding - let’s create a WithJavascriptStyleBlock() for opening block in same line ok? :-) )

Multiline statements and the Mixed Indentations Problem

Even with C-block helpers, writing line by line like previous examples can be cumbersome - you’ll probably want to write small blocks in a single multiline statement like this:

if (writeClass)
{
    if (writeDispose) 
    {
        w.WriteLine(
        @"protected virtual void Dispose(bool disposing)
        {
            base.Dispose()
        }");
    }
}

The problem is that in the code above I made the string aligned with the control logic, to be visually clear. But the output would be this (assuming that TextWriter is at IndentLevel=0):

protected virtual void Dispose(bool disposing)
        {
            base.Dispose()
        }

To make the output code correctly indented, you would have to use Mixed Indentations, where control logic has its own indentation, and output code has another:

if (writeClass)
{
    if (writeDispose) 
    {
        w.WriteLine(
@"protected virtual void Dispose(bool disposing)
{
    base.Dispose()
}");
    }
}

Looks ugly, isn’t it? This mixed-indentation problem also affects T4 templates, as you can see in the previous T4 template sample screenshot that I posted above. And as you’ll notice, T4 templates can get even worse because the syntax highlighting may get totally crazy (or not work at all) and you won’t be able to tell apart what is control logic and what is output code.

As you can see, when you write multiline statements if you want to correctly control indentation and whitespace you’ll have to use mixed indents of the generating code and the generated code, and that gets very confusing. If you indent the strings to the right (to be aligned with the outer control code), you’ll get extra indents in your output, which may not be desirable.
(And, yes, I’m aware that there are code formatting tools like Roslyn, but yet, having manual control over code is a good idea, and more than this - you may want to write non-C# code that can’t be automatically formatted)

Solving multi-line strings

To make the multi-line blocks a little less confusing (avoid “mixed indent levels”) I created a helper method which would allow me to write multiline string with any number of padding spaces (to make it aligned with the outer generating code), and yet those spaces will be ignored. Whenever I write a multi-line block, my helper assumes that the block should follow the current indentation level (of Scripty TextWriter), and do not add any further indentation.

if (something)
{
    if (something)
    {
        if (something)
        {
            w.WriteLine(@"
                namespace codegencs
                {
                    public class Test1
                    {
                        // My Properties start here
                    }
                }");
        }
    }
}

In case, my helper method will realign this whole block to the left, docking the outermost line to the left, while respecting “internal” indentation. So assuming that the current TextWriter was at IndentLevel 0 we get this output:

namespace codegencs
{
    public class Test1
    {
        // My Properties start here
    }
}

And if the TextWriter was at IndentLevel 1 we get this output:

    namespace codegencs
    {
        public class Test1
        {
            // My Properties start here
        }
    }

PS: Not only the block is docked to the left (removing from all lines the maximum number of spaces) but also we ignore the first empty line, which allows us to put the opening@" in a single line, making the first line correctly-aligned with the following lines.

// this
w.WriteLine(@"
        public class Test1
        {
            // My Properties start here
        }");

// is a little better than this, right?
w.WriteLine(
        @"public class Test1
        {
            // My Properties start here
        }");

Integrating a real Templating Engine

Now that indentation and multiline strings were more readable, the next step was to allow templates with more logic, and mostly with reusable code (functions or subtemplates). As I explained above, Scripty does not have ambition to be a Templating Engine, or even to have one integrated, so it was clear that I should integrate into Scripty a real engine like RazorEngine, Scriban or Dotliquid.

RazorEngine has good extensibility, it looked promising. I wrote my own BaseTemplate and registered it to be injected for all templates. The idea was to get the underlying Scripty TextWriter.

Basically I had this:

public class RazorIndentedTemplate<T> : TemplateBase<T>
{
    private readonly TextWriter _tw;

    public RazorIndentedTemplate(TextWriter tw)
    {
        _tw = tw;
    }
    public override void Write(object value)
    {
        _tw.Write(value);
    }

    public override void WriteLiteral(string literal)
    {
        _tw.Write(literal);
    }
...
    public override TemplateWriter Include(string name, object model, Type modelType)
    {
        return new TemplateWriter(w =>
        {
            var scripty  = ((Scripty.Core.Output.OutputFile)this._tw);
            scripty.IndentLevel++;
            Razor.Run(name: name, writer: this.CurrentWriter, modelType: modelType, model: model);
            scripty.IndentLevel--;
        });
    }
}

And then in your template:

var config = new FluentTemplateServiceConfiguration(c =>
    c
    //.WithEncoding(RazorEngine.Encoding.Raw)
    .WithBaseTemplateType(typeof(RazorIndentedTemplate.RazorIndentedTemplate<>))
    .ActivateUsing(context => (ITemplate)Activator.CreateInstance(context.TemplateType, writer))
);
using (var service = RazorEngineService.Create(config))
{
...
}

With the code above, everytime that I included a subtemplate (using @Include("Subtemplate")) I could find the TextWriter, and increase the indent level.

That’s not what I wanted.

I wanted that indent to be increased according to the inline position of the cursor:

string razorTemplate = @"
    public class Class1
    {
        @Include("Class1Method1") // this SHOULD increase indent level.
    }
    @Include("otherclass") // this should NOT increase indent level.
";

In other words, I wanted includes to respect parent indentation even if the parent indentation was inline defined by some spaces right before the include.

So I needed to get current cursor position (how many characters after the last line break, but not counting indent spaces added by Scripty). Scripty OutputFile does NOT allow me to get that, and also does not allow me to replace it’s OutputFile by my own implementation (like RazorEngine does), but it allows to pass an inner text writer.
I don’t remember if I really managed to make it work (I think I just gave up and decided to write my own before it), but it was certainly turning into a Frankestein:

  • In my template (outside Scripty) I would had multi-line helper.
  • In my template (outside Scripty) I would had indentation helper.
  • In my template I would have a PositionableTextWriter (to keep track of cursor position in current line), and would pass this as inner writer of Scripty OutputFile.
  • In another assembly I had to create my new RazorBaseTemplate, and register it to RazorEngine. Which in case would depend on my PositionableTextWriter.
  • Razor includes would automatically decide if they should increase indentation.

And in the end, it would still be hard to debug, bad intellisense (unless I was willing to create an individual cshtml file for each reusable block ever), and totally coupled.

Besides Scripty OuputFile which was limited, I also faced another dozen of weird problems when running Scripty (which uses Roslyn scripting, which have some serious issues like not allowing namespaces) like assembly conflicts, namespaces problems, etc, all hard to solve.

So after frustration with both Scripty and RazorEngine, I decided to write my own TextWriter not only to keep track of current cursor position, but also to completely replace Scripty indent-control.

My own TextWriter

Writing my TextWriter was basically to keep track of current cursor position, so I would have to count number of characters written after each line break. And then I added the indentation-control, similar to Scripty. And then I added my helpers methods: the one that will “trim” multi-line strings, and the one that is a shorthand for C-blocks.

Ok, all working great - Now that indentation and multiline strings were more readable, it was time to create a robust way of using subtemplates. Probably I could use RazorEngine more easily, but since I was already rewriting from scratch I thought I could just use pure C# strings as my templating engine.

This is what I had in mind:

// This is a reusable method which you can embed anywhere inside your string-interpolated templates
string RenderProperties(List<Property> props)
{
    return () => [email protected]"
        {string.Join(Environment.NewLine, props.Select(prop => $"public {prop.Type} {prop.Name} {{ get; set; }}"))}";
}
public void GenerateMyClass()
{
    List<Property> props = new List<Property>() { new Property() { Name = "Name", Type = "string" }, new Property() { Name = "Age", Type = "int" } };
    var w = new TemplateTextWriter();
    string myNamespace = "codegencs";
    string myClass = "Test1";
    w.Write(@"
        namespace {myNamespace}
        {
            public class {myClass}
            {
                // My Properties start here
                { RenderProperties(props) }
            }
        }");
}

The problem with this is that my Write method (yes, that one which can handle multiline strings) will receive as argument a single string. And this is what it would output, since the included method renders a linebreak and has no idea of inline indentation:

namespace codegencs
{
    public class Test1
    {
        // My Properties start here
        public string Name { get; set; }
public int Age { get; set; }
    }
}

And as you may have noticed, I’m a little obsessed with whitespace control, so the whitespeak freak got angry again because this way of using subtemplates wouldn’t work. Subtemplates should “respect current indent” even if the indent was implicitally defined by some spaces before the subtemplate was included.

Why not C# ?

Since my TextWriter was fully C# (no templating language), and it was doing a good part of what I needed, why not do the whole subtemplating thing in C#? All I had to do is finding a way to capture current cursor position before starting a subtemplate. In other words, I wanted “inline templates” which should be executed while preserving the current indentation.

If it sounds weird to output C# code using C# code (really?), let me give some good reasons:

  • Compiled, typed, compile-time checking
  • Full IDE support. The IDE which you are already familiar. With debugging support.
  • Runs in .NET Standard and .NET Core
  • Full access to all C# Framework. You can connect to SqlServer, you can use Newtonsoft JSON, etc.
  • Full C# framework - you’re not limited to a small subset of string-manipulation functions.

After all those reasons, all I can think is “Why hadn’t this been done before”.

Interpolated Strings to the Rescue

As I explained above my inline templates would have to “capture” the current cursor position, and therefore I would have to process the inner template only after the outer template (at least the parts which precede the inner template) had been processed/written.

Now comes the interesting part…

As soon I started exploring how I could control the rendering of the interpolated strings, I discovered that string interpolation is not only syntactic-sugar offered by the compiler, but it also translates into a class FormattableString which offers the format and the arguments individually, and that would be a perfect use for outer template (format) and inner templates (arguments):

Func<List<Property>, FormattableString> RenderProperties = (props) => ...
string myClass = "Test1";
writer.Write([email protected]"
    public class {myClass}
    {{
        // My Properties start here
        { RenderProperties(props) }
    }}");

… is translated into this:

Func<List<Property>, FormattableString> RenderProperties = (props) => ...
string myClass = "Test1";
writer.Write(FormattableStringFactory.Create(@"
    public class {0}
    {{
        // My Properties start here
        {1}
    }}", new object[] { myClass, RenderProperties(props) } );

PS: since interpolated strings use braces as control character, we have to use double braces for escaping braces.

FormattableStringFactory.Create creates an object of type FormattableString, which has the format and the arguments.
If my Write method expects a string, the FormattableString is implicitally converted into a regular string (and in case double braces will be rendered as single braces).
On the other hand, if method expects a FormattableString, we can use the format and arguments explicitly - so we can process the format block by block, and when we reach each argument {X} we can “save” the current position and therefore preserve the current indentation.
So all I have to do is process (writing to output) the string format until I found the embedded argument, then process the inline argument, and repeat until the end of the string format. So with string interpolation we can embed complex subtemplates inside the parent template, while respecting the “cursor position” where the inner template starts.

// This is a reusable method which you can embed anywhere inside your string-interpolated templates
Func<FormattableString> RenderProperties(List<Property> props)
{
    return () => [email protected]"
        {string.Join(Environment.NewLine, props.Select(prop => $"public {prop.Type} {prop.Name} {{ get; set; }}"))}"
    ;
}
public void GenerateMyClass()
{
    List<Property> props = new List<Property>() { new Property() { Name = "Name", Type = "string" }, new Property() { Name = "Age", Type = "int" } };
    var writer = new TemplateTextWriter();
    string myNamespace = "codegencs";
    string myClass = "Test1";
    writer.Write([email protected]"
        namespace {myNamespace}
        {{
            public class {myClass}
            {{
                // My Properties start here
                { RenderProperties(props) }
            }}
        }}");
}

And the output is:

namespace codegencs
{
    public class Test1
    {
        // My Properties start here
        public string Name { get; set; }
        public int Age { get; set; }
    }
}

As you can see, the inner block has multiple lines, and yet all those lines were written in the same position where the first line started. In other words, the inner block was fully written in the “current cursor position”. And again, if the text writer had indent level 1, all that output (outer and inner template) would have 4 more spaces before each line. Cool, uh?

For my inner template I used a Func<FormattableString> but it could be other types like string, Func<string>, or FormattableString itself. They would all be evaluated “on demand”, only by the moment we need to output those parameters.

FormattableString is automatically created by the compiler when you write an interpolated string.

The core method of my TextWriter is below, and basically it will get the format string, split by the arguments ({0}, {1}, etc, which are automatically generated by the compiler when you write a regular interpolated string), and will write (respecting current indent level, etc) a small part of the string format “up to the next argument”, then it will process the argument, which might be another interpolated string (entering into recursion), or might be a more reusable callback function, or just a plain string, and repeat until the end of the string.

#region WriteFormattable(string format, params object[] arguments) - Basically, we split any interpolated string, and write block by block, doing lazy-evaluation of arguments. 
/// <summary>
/// This is the "heart" of this class. Basically, we split any interpolated string, and write block by block, doing lazy-evaluation of arguments. 
/// The idea of writing Func<FormattableString> is that we do NOT evaluate the {arguments} BEFORE the outer string is being written - they are only evaluated when needed
/// so we can capture the cursor position in current line, and preserve-it if the arguments render multi-line strings
/// </summary>
public void WriteFormattable(FormattableString formattable)
{
    string format = formattable.Format;
    if (_trimBlockPaddings)
        format = TrimLeftPadding(format);
    if (_trimFirstEmptyLine)
        format = this.TrimFirstEmptyLine(format);
    object[] arguments = formattable.GetArguments();
    var matches = Regex.Matches(format, @"{\d}");
    int lastPos = 0;
    for (int i = 0; i < matches.Count; i++)
    {
        // unescape escaped curly braces
        string text = format.Substring(lastPos, matches[i].Index - lastPos).Replace("{{", "{").Replace("}}", "}"); 
        base.WriteWithIndents(text);
        // arguments[i] may not work because same argument can be used multiple times
        var arg = arguments[int.Parse(matches[i].Value.Substring(1, 1))];
        if (arg as FormattableString != null)
        {
            base.ExecuteInlineAction(() =>
            {
                FormattableString subText = (FormattableString)arg;
                this.Write(subText);
            });
        }
        else if (arg as Func<FormattableString> != null)
        {
            base.ExecuteInlineAction(() =>
            {
                Func<FormattableString> subText = ((Func<FormattableString>)arg);
                this.Write(subText);
            });
        }
        else if (arg as Action<TemplateTextWriter> != null)
        {
            base.ExecuteInlineAction(() =>
            {
                Action<TemplateTextWriter> action = ((Action<TemplateTextWriter>)arg);
                action(this);
            });
        }
        else if (arg as Action<TextWriter> != null)
        {
            base.ExecuteInlineAction(() =>
            {
                Action<TextWriter> action = ((Action<TextWriter>)arg);
                action(this);
            });
        }
        else if (arg as Func<string> != null)
        {
            base.ExecuteInlineAction(() =>
            {
                string exec = ((Func<string>)arg)();
                this.Write(exec);
            });
        }
        else if (arg as string != null)
        {
            base.ExecuteInlineAction(() =>
            {
                string subText = ((string)arg);
                this.Write(subText);
            });
        }
        else
        {
            base.ExecuteInlineAction(() =>
            {
                this.Write(arg.ToString());
            });
        }

        lastPos = matches[i].Index + matches[i].Length;
    }
    string lastPart = format.Substring(lastPos).Replace("{{", "{").Replace("}}", "}");
    base.WriteWithIndents(lastPart);
}
#endregion

And the indentation control basically will split all writes line-by-line, and for each new line will add indent before the text:

#region WriteWithIndents(string value): writes line by line, and writes the indent strings before each new line
/// <summary>
/// This is the "heart" of this class. Basically, we split any multi-line string, and write line by line. 
/// Before writing each new line we write the indent block, which could for example be 8 spaces (4 spaces in first indent level and 4 spaces for second indent level),
/// or 2 tabs (one for each indent level), or any combination.
/// </summary>
/// <param name="value"></param>
protected void WriteWithIndents(string value)
{
    if (string.IsNullOrEmpty(value))
        return;

    var matches = _lineBreaksRegex.Matches(value);

    int lastPos = 0;
    for (int i = 0; i < matches.Count; i++)
    {
        string line = value.Substring(lastPos, matches[i].Index - lastPos);
        string lineBreak = value.Substring(matches[i].Index, matches[i].Length);
        lastPos = matches[i].Index + matches[i].Length;

        // if _dontIndentCurrentLine is set, it's because we're starting an inner block right "at cursor position"-  no need to indent again - we're already positioned!
        if (line.Length > 0 && _currentLine.Length == 0 && !_dontIndentCurrentLine)
            this.WriteIndent();

        this.WriteRaw(line);
        if (_normalizeLineEndings)
            this.WriteRaw(_innerWriter.NewLine);
        else
            this.WriteRaw(lineBreak);
        _currentLine.Clear();
        _dontIndentCurrentLine = false;
    }
    string lastLine = value.Substring(lastPos);

    if (lastLine.Length > 0 && _currentLine.Length == 0 && !_dontIndentCurrentLine)
        this.WriteIndent();
    this.WriteRaw(lastLine);
    _currentLine.Clear().Append(lastLine);
    _dontIndentCurrentLine = false;
}
#endregion

Other than this, the last magic part is this function, which will “save current cursor position”, and will start a new “inline write”. For example, suppose that the outer template has implicitally written 4 spaces and then we’re starting an inline action, the TextWriter will “save” this 4 spaces, write the first line (no need to indent because it was “implicitally indented”), and then will explicitly indent the next lines with this same 4 spaces. (Would also work for tabs, or any other indentation string).

/// <summary>
/// If the current line has some manually-written whitespace, this method will "save" current cursor position, and in case the inline action writes a multi-line text,
/// the subsequent lines (after the first) will "preserve" the cursor position by adding this extra indent.
/// ("manually-written whitespace" does not count the automatic indents written by this class).
/// </summary>
protected void ExecuteInlineAction(Action inlineAction)
{
    string indent = _currentLine.ToString();
    if (indent != null && indent.Length > 0 && string.IsNullOrWhiteSpace(indent))
    {
        this._levelIndent.Push(indent); // we could convert tabs to spaces or vice-versa
        _dontIndentCurrentLine = true;
        _currentLine.Clear();
        inlineAction();
        this._levelIndent.Pop();
    }
    else
    {
        inlineAction();
    }
}

Hope you enjoyed as much as I did.

On next posts I may explore some new ideas, or maybe just develop some nice templates.

Full source code here