Helping the Queryable Provider sort out interpolated strings

Subtleties of Queryable Provider

The Queryable Provider cannot handle this:

 var result = _context.Humans
                      .Select(x => $"Name: {x.Name}  Age: {x.Age}")
                      .Where(x => x != "")
                      .ToList();

It will not cope with any expression that will use the interpolated string, but it will parse this without difficulty:

 var result = _context.Humans
                      .Select(x => "Name " +  x.Name + " Age " + x.Age)
                      .Where(x => x != "")
                      .ToList();

It is especially painful to correct bugs after turning on ClientEvaluation (an exception when calculating on the client), all profiles of the auto-mapper should be subjected to rigorous analysis to find this interpolation. Let's figure out what’s the matter and offer our solution to the problem.

We correct

The interpolation in the Expression Tree is translated like this (this is the result of the ExpressionStringBuilder.ExpressionToString method, it omitted some nodes, but for us it is
not fatal):

// для x.Age требуется boxing
Format("Name:{0} Age:{1}", x.Name, Convert(x.Age, Object)))

Or so, when there are more than 3 arguments

Format("Name:{0} Age:{1}", new [] {x.Name, Convert(x.Age, Object)))

We can conclude that the provider simply did not learn how to handle such cases, but they could teach it to reduce these cases to the good old ToString (), which is sorted like this:

((("Name: " + x.Name) + " Age: ") + Convert(x.Age, Object)))

I want to write a Visitor that will go through the Expression Tree, namely through the nodes of the MethodCallExpression and replace the Format method with concatenation. If you are familiar with Expression Trees, then you know that C # offers us its visitor for traversing the tree - ExpressionVisitor, for those who are not familiar it will be interesting .

It is enough to override only the VisitMethodCall method and slightly modify its return value. The method parameter is of type MethodCallExpression, which contains information about the method itself and about the arguments that are passed to it.

Let's split the task into several parts:

1. Determine that it was the Format method that came to VisitMethodCall
2. Replace this method with string concatenation
3. Process all overloads of the Format method that can be received
4. Write an Extension method in which our visitor will call

The first part is quite simple, the Format 4 method has overloads that will be built
in the Expression tree

 public static string Format(string format, object arg0)  
 public static string Format(string format, object arg0,object arg1)  
 public static string Format(string format, object arg0,object arg1,object arg2)
 public static string Format(string format, params object[] args)

We get using the reflection of their MethodInfo

private IEnumerable FormatMethods =>
            typeof(string).GetMethods().Where(x => x.Name.Contains("Format"))
//первые три
private IEnumerable FormatMethodsWithObjects => 
   FormatMethods
         .Where(x => x.GetParameters()
         .All(xx=> xx.ParameterType == typeof(string) || 
                        xx.ParameterType == typeof(object))); 
//последний
private IEnumerable FormatMethodWithArrayParameter => 
   FormatMethods
        .Where(x => x.GetParameters()
                              .Any(xx => xx.ParameterType == typeof(object[])));

Class, now we can determine that the Format method has "come" to MethodCallExpression.

When traversing a tree, VisitMethodCall may "come":

1. Format method with object arguments
2. Format method with object [] argument
3. Not the Format method at all

A bit custom Pattern Maching

So far, only 3 conditions can be resolved with the help of if, but we, assuming that in the future we will have to expand this method, put all the cases in such a data structure:

 public class PatternMachingStructure
 {
    public Func FilterPredicate { get; set; }
    public Func> 
                                       SelectorArgumentsFunc { get; set; }
    public Func, Expression> 
                                       ReturnFunc { get; set; }
 }
var patternMatchingList = new List()

Using FilterPredicate, we determine which of the 3 cases we are dealing with. The SelectorArgumentFunc is needed to bring the arguments of the Format method to a uniform form, the ReturnFunc method, which will return the new Expression to us.

Now let's try to replace the interpolation representation with concatenation, for this we will use the following method:

private Expression InterpolationToStringConcat(MethodCallExpression node,
            IEnumerable formatArguments)
{
  //выбираем первый аргумент
  //(example : Format("Name: {0} Age: {1}", x.Name,x.Age) -> 
  //"Name: {0} Age: {1}"
  var formatString = node.Arguments.First();
  // проходим по паттерну из метода Format и выбираем все 
  // строки между аргументами передаем их методу ExpressionConstant
  // example:->[Expression.Constant("Name: "),Expression.Constant(" Age: ")]
  var argumentStrings = Regex.Split(formatString.ToString(),RegexPattern)
                             .Select(Expression.Constant);
  // мерджим их со значениями formatArguments
  // example ->[ConstantExpression("Name: "),PropertyExpression(x.Name),
  // ConstantExpression("Age: "),
  // ConvertExpression(PropertyExpression(x.Age), Object)]
  var merge = argumentStrings.Merge(formatArguments, new ExpressionComparer());
  // склеиваем так, как QueryableProvider склеивает простую конкатенацию строк
  // example : -> MethodBinaryExpression 
  //(("Name: " + x.Name) + "Age: " + Convert(PropertyExpression(x.Age),Object))
  var result = merge.Aggregate((acc, cur) =>
                    Expression.Add(acc, cur, StringConcatMethod));
  return result;
 }

InterpolationToStringConcat will be called from Visitor, it is hidden behind ReturnFunc
(when node.Method == string.Format)

protected override Expression VisitMethodCall(MethodCallExpression node)
{
  var pattern = patternMatchingList.First(x => x.FilterPredicate(node.Method));
  var arguments = pattern.SelectorArgumentsFunc(node);
  var expression = pattern.ReturnFunc(node, arguments);
  return expression;
}

Now we need to write logic to handle different overloads of the Format method, it is quite trivial and is located in patternMachingList

patternMatchingList = new List
{
    // первые три перегрузки Format
   new PatternMachingStructure
   {
        FilterPredicate = x => FormatMethodsWithObjects.Contains(x),
        SelectorArgumentsFunc = x => x.Arguments.Skip(1),
        ReturnFunc = InterpolationToStringConcat
    },
    // последняя перегрузка Format, принимающая массив
    new PatternMachingStructure
    {
        FilterPredicate = x => FormatMethodWithArrayParameter.Contains(x),
        SelectorArgumentsFunc = x => ((NewArrayExpression) x.Arguments.Last())
                                                            .Expressions,
        ReturnFunc = InterpolationToStringConcat
     },
     // node.Method != Format
    new PatternMachingStructure()
    {
        FilterPredicate = x => FormatMethods.All(xx => xx != x),
        SelectorArgumentsFunc = x => x.Arguments,
         ReturnFunc = (node, _) => base.VisitMethodCall(node)
     }
};

Accordingly, in the VisitMethodCall method, we will go through this sheet until the first positive FilterPredicate, then convert the arguments (SelectorArgumentFunc) and execute ReturnFunc.

Let's write Extention, calling that we can replace the interpolation.

We can get Expression, pass it to our Visitor, and then call the IQuryableProvider CreateQuery interface method, which will replace the original expression tree with ours:

public static IQueryable ReWrite(this IQueryable qu)
{
  var result = new InterpolationStringReplacer().Visit(qu.Expression);
  var s = (IQueryable) qu.Provider.CreateQuery(result);
  return s; 
}

Pay attention to Cast qu.Provider.CreateQuery (result) of type IQueryable in IQueryablethis is generally standard practice for c # (look at IEnumerable), it arose because of the need to process all generic interfaces in one class that wants to accept IQueryable / IEnumerable, and process it using common interface methods.
This could have been avoided by casting T to the base class, this is possible using covariance, but it also imposes some restrictions on interface methods (more on this in the next article).

Total

Apply ReWrite to the expression at the beginning of the article

 var result = _context.Humans
                      .Select(x => $"Name: {x.Name}  Age: {x.Age}")
                      .Where(x => x != "")
                      .ReWrite()
                      .ToList();
// correct
// [Name: "Piter" Age: 19]

Github