AScript如何实现LINQ语法

AScript是一个开源的C#动态脚本解析执行引擎，支持扩展自定义语法，本篇将详细讲解如何扩展实现LINQ语法。

一、原理

LINQ语句会被编译为调用Queryable/Enumerable扩展方法中的SelectMany/Where/Join/GroupJoin/GroupBy/OrderBy/OrderByDescending/Select等方法。

Queryable/Enumerable扩展方法已通过AddFunc方式注入到了CSharpLang语言中：

1 // IEnumerable<T>扩展方法
2 AddFunc(typeof(System.Linq.Enumerable));
3 // IQueryable<T>扩展方法
4 AddFunc(typeof(System.Linq.Queryable));

LINQ To SQL通过分析Queryable扩展方法及Lambda参数生成对应的SQL语句来执行数据库查询操作。

我们只要实现从LINQ语句生成调用Queryable/Enumerable扩展方法即可。

二、示例

 1 using (var context = new TestSqliteContext())
 2 {
 3     string s = @"
 4 var q = from p in context.Persons
 5     join a in context.AddressInfos on p.Id equals a.UserId into aa
 6     from a in aa.DefaultIfEmpty()
 7     select new { p.Id, p.Name, p.Age, MyAddress = a.Address };
 8 q.ToList();
 9 ";
10     var script = new Script();
11     script.Context.SetVar("context", context);
12     var list = script.Eval(s);
13     Console.WriteLine(JsonConvert.SerializeObject(list, Formatting.Indented));
14 }

生成的sqlite查询语句为：

1 SELECT "p"."Id", "p"."Name", "p"."Age", "a"."Address" AS "MyAddress"
2    FROM "Persons" AS "p"
3    LEFT JOIN "AddressInfos" AS "a" ON "p"."Id" = "a"."UserId"

下面我们来看看如何实现LINQ语法。

三、定义QueryNode节点

首先，定义一个LINQ查询语法树节点，用于管理LINQ语法中的from/where/join/group/orderby/select语句，并在执行或编译时生成对应的方法调用。

 1 public class QueryNode : TreeNode
 2 {
 3     // 变量所属上级（变量聚合）
 4     private readonly Dictionary<string, string> _VarParentDict = new Dictionary<string, string>();
 5     // 变量所属上级计数（变量聚合计数）
 6     private int _ParentCounter = 0;
 7     // 当前变量名
 8     private string _CurrentVarName;
 9     // 当前数据源
10     private ITreeNode _Source;
11     public override Expression Build(BuildContext buildContext, ScriptContext scriptContext, BuildOptions options)
12     {
13         return _Source.Build(buildContext, scriptContext, options);
14     }
15     public override object Eval(ScriptContext context, BuildOptions options, EvalControl control, out Type returnType)
16     {
17         return _Source.Eval(context, options, control, out returnType);
18     }
19 }

1、添加from语句

在QueryNode中添加AddFrom方法，增加LINQ中的from语句：

 1 /// <summary>
 2 /// from varName in source
 3 /// </summary>
 4 public void AddFrom(string varName, ITreeNode source)
 5 {
 6     if (_Source == null)
 7     {
 8         // 第1个from语句
 9         _Source = source;
10         _CurrentVarName = varName;
11         return;
12     }
13     // _Source.SelectMany(_CurrentVarName => source, (_CurrentVarName, varName) => new { _CurrentVarName, varName })
14     var selectMany = new CallFuncNode
15     {
16         Name = "SelectMany",
17         Args = new ITreeNode[]
18         {
19             _Source,
20             // _CurrentVarName => source
21             new DefineFuncNode
22             {
23                 Args = new [] { new DefineVarNode(_CurrentVarName) },
24                 Body = TryVisitAndReplace(source)
25             },
26             // (_CurrentVarName, varName) => new { _CurrentVarName, varName }
27             new DefineFuncNode
28             {
29                 Args = new []
30                 {
31                     new DefineVarNode(_CurrentVarName) },
32                     new DefineVarNode(varName) }
33                 },
34                 Body = new NewNode
35                 {
36                     InitProperties = new ITreeNode[]
37                     {
38                         new VariableNode(_CurrentVarName),
39                         new VariableNode(varName)
40                     }
41                 }
42             }
43         }
44     };
45     // 更新当前数据源
46     _Source = selectMany;
47     // 变量聚合
48     var oldCurrentName = _CurrentVarName;
49     _CurrentVarName = $"<>h__TransparentIdentifier{_ParentCounter++}";
50     _VarParentDict[oldCurrentName] = _CurrentVarName;
51     _VarParentDict[varName] = _CurrentVarName;
52 }

多个from语句生成SelectMany方法调用，方法调用对应的是CallFuncNode节点，Lambda参数或者Func<>参数对应的是DefineFuncNode节点。

SelectMany方法第2个参数为变量聚合，所以我们需要生成聚合变量名并跟踪当前变量聚合路径，TryVisitAndReplace方法用于替换数据源中的变量聚合路径，这里不作说明，详情可查看QueryNode源码。

2、添加where语句

 1 /// <summary>
 2 /// from a in query1
 3 /// from b in query2
 4 /// where a.Age == b.Age
 5 /// </summary>
 6 public void AddWhere(ITreeNode condition)
 7 {
 8     if (_Source == null)
 9     {
10         throw new Exceptions.ScriptAnalyzingException("invalid expression where");
11     }
12     // _Source.Where(<>h__TransparentIdentifier0 => (<>h__TransparentIdentifier0.a.Age == <>h__TransparentIdentifier0.b.Age))
13     var whereNode = new CallFuncNode
14     {
15         Name = "Where",
16         Args = new ITreeNode[]
17         {
18             _Source,
19             // <>h__TransparentIdentifier0 => (<>h__TransparentIdentifier0.a.Age == <>h__TransparentIdentifier0.b.Age)
20             new DefineFuncNode
21             {
22                 Args = new [] { new DefineVarNode(_CurrentVarName) },
23                 Body = TryVisitAndReplace(condition)
24             }
25         }
26     };
27     // 更新当前数据源
28     _Source = whereNode;
29 }

where语句生成Where方法调用，<>h__TransparentIdentifier0为聚合参数，a和b是where语句中的参数，TryVisitAndReplace方法就是把where语句中的a、b参数替换为聚合参数路径：<>h__TransparentIdentifier0.a和<>h__TransparentIdentifier0.b。

3、添加select语句

 1 /// <summary>
 2 /// select new { a.Name, b.Age }
 3 /// </summary>
 4 public void AddSelect(ITreeNode selector)
 5 {
 6     if (_Source == null)
 7     {
 8         throw new Exceptions.ScriptAnalyzingException("invalid expression select");
 9     }
10     // _Source.Select(<>h__TransparentIdentifier0 => new <> f__AnonymousType0`2(Name = <> h__TransparentIdentifier0.a.Name, Age = <> h__TransparentIdentifier0.b.Age))
11     var selectNode = new CallFuncNode
12     {
13         Name = "Select",
14         Args = new ITreeNode[]
15         {
16             _Source,
17             new DefineFuncNode
18             {
19                 Args = new [] { new DefineVarNode(_CurrentVarName) },
20                 Body = TryVisitAndReplace(selector)
21             }
22         }
23     };
24     // 更新当前数据源
25     _Source = selectNode;
26     // 重置变量
27     _VarParentDict.Clear();
28     _ParentCounter = 0;
29     _CurrentVarName = null;
30 }

select语句表示一个查询语句的结束，所以要重置变量，后续查询时重新跟踪变量。

4、添加join语句

 1 public void AddJoin(string varName, ITreeNode source, ITreeNode key1, ITreeNode key2, string intoName = null)
 2 {
 3     if (_Source == null)
 4     {
 5         throw new Exceptions.ScriptAnalyzingException("invalid expression join");
 6     }
 7     if (string.IsNullOrEmpty(intoName))
 8     {
 9         AddJoin1(varName, source, key1, key2);
10     }
11     else
12     {
13         AddJoin2(varName, source, key1, key2, intoName);
14     }
15 }

join语句分inner join和left join两种语法：

• inner join对应Join方法

 1 /// <summary>
 2 /// join varName in source on a.Age equals varName.Age
 3 /// </summary>
 4 private void AddJoin1(string varName, ITreeNode source, ITreeNode key1, ITreeNode key2)
 5 {
 6     // _Source.Join(source, a => a.Age, varName => varName.Age, (a, varName) => new <> f__AnonymousType2`2(a = a, varName = varName))
 7     var joinNode = new CallFuncNode
 8     {
 9         Name = "Join",
10         Args = new ITreeNode[]
11         {
12             _Source,
13             TryVisitAndReplace(source),
14             // key1: a => a.Age
15             new DefineFuncNode
16             {
17                 Args = new [] { new DefineVarNode(_CurrentVarName) },
18                 Body = TryVisitAndReplace(key1)
19             },
20             // key2: varName => varName.Age
21             new DefineFuncNode
22             {
23                 Args = new [] { new DefineVarNode(varName) },
24                 Body = TryVisitAndReplace(key2)
25             },
26             // (a, varName) => new { a, varName })
27             new DefineFuncNode
28             {
29                 Args = new []
30                 {
31                     new DefineVarNode(_CurrentVarName),
32                     new DefineVarNode(varName)
33                 },
34                 Body = new NewNode
35                 {
36                     InitProperties = new ITreeNode[]
37                     {
38                         new VariableNode(_CurrentVarName),
39                         new VariableNode(varName)
40                     }
41                 }
42             }
43         }
44     };
45     // 更新当前数据源
46     _Source = joinNode;
47     // 变量聚合
48     var oldCurrentName = _CurrentVarName;
49     _CurrentVarName = $"<>h__TransparentIdentifier{_ParentCounter++}";
50     _VarParentDict[oldCurrentName] = _CurrentVarName;
51     _VarParentDict[varName] = _CurrentVarName;
52 }

• left join对应GroupJoin方法，后续紧跟着from语句对应SelectMany及DefaultIfEmpty方法

 1 /// <summary>
 2 /// join varName in source on a.Age equals varName.Age into intoName
 3 /// </summary>
 4 private void AddJoin2(string varName, ITreeNode source, ITreeNode key1, ITreeNode key2, string intoName)
 5 {
 6     // _Source.GroupJoin(source, a => a.Age, varName => varName.Age, (a, intoName) => new <> f__AnonymousType2`2(a = a, intoName = intoName))
 7     var joinNode = new CallFuncNode
 8     {
 9         Name = "GroupJoin",
10         Args = new ITreeNode[]
11         {
12             _Source,
13             TryVisitAndReplace(source),
14             // key1: a => a.Age
15             new DefineFuncNode
16             {
17                 Args = new [] { new DefineVarNode(_CurrentVarName) },
18                 Body = TryVisitAndReplace(key1)
19             },
20             // key2: varName => varName.Age
21             new DefineFuncNode
22             {
23                 Args = new [] { new DefineVarNode(varName) },
24                 Body = TryVisitAndReplace(key2)
25             },
26             // (a, intoName) => new { a, intoName })
27             new DefineFuncNode
28             {
29                 Args = new []
30                 {
31                     new DefineVarNode(_CurrentVarName),
32                     new DefineVarNode(intoName)
33                 },
34                 Body = new NewNode
35                 {
36                     InitProperties = new ITreeNode[]
37                     {
38                         new VariableNode(_CurrentVarName),
39                         new VariableNode(intoName)
40                     }
41                 }
42             }
43         }
44     };
45     // 更新当前数据源
46     _Source = joinNode;
47     // 变量聚合
48     var oldCurrentName = _CurrentVarName;
49     _CurrentVarName = $"<>h__TransparentIdentifier{_ParentCounter++}";
50     _VarParentDict[oldCurrentName] = _CurrentVarName;
51     _VarParentDict[intoName] = _CurrentVarName;
52 }

left join语句我们不需要跟踪varName变量，而是跟踪intoName变量。

5、添加group语句

 1 /// <summary>
 2 /// group a.Name by a.Age into intoName
 3 /// </summary>
 4 public void AddGroup(ITreeNode key, ITreeNode element, string intoName = null)
 5 {
 6     if (_Source == null)
 7     {
 8         throw new Exceptions.ScriptAnalyzingException("invalid expression group");
 9     }
10     // _Source.GroupBy(a => a.Age, a => a.Name)
11     bool hasElement;
12     if (element == null) hasElement = false;
13     else if (element is VariableNode elementVarNode && elementVarNode.Name == _CurrentVarName)
14     {
15         hasElement = false;
16     }
17     else
18     {
19         hasElement = true;
20     }
21     ITreeNode group;
22     if (hasElement)
23     {
24         group = new CallFuncNode
25         {
26             Name = "GroupBy",
27             Args = new ITreeNode[]
28             {
29                 _Source,
30                 // key: a => a.Age
31                 new DefineFuncNode
32                 {
33                     Args = new [] { new DefineVarNode(_CurrentVarName) },
34                     Body = TryVisitAndReplace(key)
35                 },
36                 // element: a => a.Name
37                 new DefineFuncNode
38                 {
39                     Args = new [] { new DefineVarNode(_CurrentVarName) },
40                     Body = TryVisitAndReplace(element)
41                 },
42             }
43         };
44     }
45     else
46     {
47         group = new CallFuncNode
48         {
49             Name = "GroupBy",
50             Args = new ITreeNode[]
51             {
52                 _Source,
53                 // key: a => a.Age
54                 new DefineFuncNode
55                 {
56                     Args = new [] { new DefineVarNode(_CurrentVarName) },
57                     Body = TryVisitAndReplace(key)
58                 }
59             }
60         };
61     }
62     // 更新当前数据源
63     _Source = group;
64     // 重置变量
65     _CurrentVarName = intoName;
66     _VarParentDict.Clear();
67     _ParentCounter = 0;
68 }

跟select语句类似，group语句也要重置变量，后续语句重新从group数据源查询并跟踪变量。

6、添加orderby语句

orderby语句有2种排序模式：ascending和descending，默认为ascending，AScript脚本支持简化asc和desc 。

 1 /// <summary>
 2 /// orderby a.Age descending
 3 /// </summary>
 4 public void AddOrderby(ITreeNode key, string mode)
 5 {
 6     if (_Source == null)
 7     {
 8         throw new Exceptions.ScriptAnalyzingException("invalid expression orderby");
 9     }
10     // _Source.OrderByDescending(a => a.Age)
11     var orderby = new CallFuncNode
12     {
13         Name = mode == "desc" || mode == "descending" ? "OrderByDescending" : "OrderBy",
14         Args = new ITreeNode[]
15         {
16             _Source,
17             // key: a => a.Age
18             new DefineFuncNode
19             {
20                 Args = new [] { new DefineVarNode(_CurrentVarName) },
21                 Body = TryVisitAndReplace(key)
22             }
23         }
24     };
25     // 更新当前数据源
26     _Source = orderby;
27 }

四、定义LINQ语法解析

LINQ语法是from语句开头，select或group语句结尾，AScript脚本对结尾语句不做约束。

我们创建FromTokenHandler类，用于解析LINQ语句。

并在CSharpLang中注册解析器：AddTokenHandler("from", FromTokenHandler.Instance) 。

 1 /// <summary>
 2 /// from a in query1
 3 /// from b in query2
 4 /// join c in query3 on a.Id equals c.Id into cc
 5 /// from c in cc.DefaultIfEmpty
 6 /// orderby b.Age desc
 7 /// group a by a.Name into g
 8 /// select new { Name = g.Key, Count = g.Count() }
 9 /// </summary>
10 public class FromTokenHandler : ITokenHandler
11 {
12     public static readonly FromTokenHandler Instance = new FromTokenHandler();
13     private static readonly HashSet<string> _OnTokens = new HashSet<string> { "on" };
14     private static readonly HashSet<string> _EqualsTokens = new HashSet<string> { "equals" };
15     private static readonly HashSet<string> _ByTokens = new HashSet<string> { "by" };
16     private static readonly HashSet<string> _Keywords = new HashSet<string> { "from", "where", "join", "select", "orderby", "group" };
17     private static readonly HashSet<string> _JoinEndTokens = new HashSet<string> { "from", "where", "join", "select", "orderby", "group", "into" };
18     private static readonly HashSet<string> _OrderbyEndTokens = new HashSet<string> { "from", "where", "join", "select", "orderby", "group", "ascending", "descending", "asc", "desc" };
19     public void Build(DefaultSyntaxAnalyzer analyzer, TokenAnalyzingArgs e)
20     {
21         e.IsHandled = true;
22         var queryNode = e.Ignore ? null : new QueryNode();
23         var createFullOptions = (e.Options.CreateFullTreeNode ?? false) ? e.Options : new BuildOptions(e.Options) { CreateFullTreeNode = true };
24         // 解析from语句
25         BuildFrom(analyzer, e, createFullOptions, queryNode);
26         // 解析后续linq语句
27         while (true)
28         {
29             var token = e.TokenReader.Read();
30             if (!token.HasValue) break;
31             if (token.Value.IsSymbol(";"))
32             {
33                 e.TokenReader.Push(token.Value);
34                 break;
35             }
36             if (token.Value.IsSymbol("from"))
37             {
38                 BuildFrom(analyzer, e, createFullOptions, queryNode);
39             }
40             else if (token.Value.IsSymbol("join"))
41             {
42                 BuildJoin(analyzer, e, createFullOptions, queryNode);
43             }
44             else if (token.Value.IsSymbol("where"))
45             {
46                 BuildWhere(analyzer, e, createFullOptions, queryNode);
47             }
48             else if (token.Value.IsSymbol("select"))
49             {
50                 BuildSelect(analyzer, e, createFullOptions, queryNode);
51             }
52             else if (token.Value.IsSymbol("group"))
53             {
54                 BuildGroup(analyzer, e, createFullOptions, queryNode);
55             }
56             else if (token.Value.IsSymbol("orderby"))
57             {
58                 BuildOrderby(analyzer, e, createFullOptions, queryNode);
59             }
60             else
61             {
62                 throw new Exceptions.ScriptAnalyzingException($"invalid expression near from, unknow {token.Value.Value} at ({token.Value.Line},{token.Value.Column})");
63             }
64         }
65         // 将LINQ语句添加到语法树中
66         if (!e.Ignore)
67         {
68             e.TreeBuilder.AddData(e.BuildContext, e.ScriptContext, e.Options, e.Control, queryNode);
69         }
70     }
71 }

1、解析from语句

 1 /// <summary>
 2 /// from a in query1
 3 /// </summary>
 4 private void BuildFrom(DefaultSyntaxAnalyzer analyzer, TokenAnalyzingArgs e, BuildOptions createFullOptions, QueryNode queryNode)
 5 {
 6     var varToken = analyzer.ValidateNextToken(e.TokenReader, ETokenType.Word);
 7     analyzer.ValidateNextToken(e.TokenReader, "in");
 8     var source = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _Keywords);
 9     queryNode?.AddFrom(varToken.Value.Value, source);
10 }

2、解析where语句

1 private void BuildWhere(DefaultSyntaxAnalyzer analyzer, TokenAnalyzingArgs e, BuildOptions createFullOptions, QueryNode queryNode)
2 {
3     var condition = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _Keywords);
4     queryNode?.AddWhere(condition);
5 }

3、解析select语句

1 private void BuildSelect(DefaultSyntaxAnalyzer analyzer, TokenAnalyzingArgs e, BuildOptions createFullOptions, QueryNode queryNode)
2 {
3     var selector = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _Keywords);
4     queryNode?.AddSelect(selector);
5 }

4、解析join语句

 1 /// <summary>
 2 /// from a in q1
 3 /// join b in q2 on a.Id equals b.Id into cc
 4 /// </summary>
 5 private void BuildJoin(DefaultSyntaxAnalyzer analyzer, TokenAnalyzingArgs e, BuildOptions createFullOptions, QueryNode queryNode)
 6 {
 7     var varToken = analyzer.ValidateNextToken(e.TokenReader, ETokenType.Word);
 8     analyzer.ValidateNextToken(e.TokenReader, "in");
 9     var source = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, e.Options, e.TokenReader, e.Control, e.Ignore, _OnTokens);
10     analyzer.ValidateNextToken(e.TokenReader, "on");
11     var key1 = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _EqualsTokens);
12     analyzer.ValidateNextToken(e.TokenReader, "equals");
13     var key2 = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _JoinEndTokens);
14     string intoName = null;
15     var intoToken = e.TokenReader.Read();
16     if (intoToken.HasValue)
17     {
18         if (intoToken.Value.IsSymbol("into"))
19         {
20             var intoNameToken = analyzer.ValidateNextToken(e.TokenReader, ETokenType.Word);
21             intoName = intoNameToken.Value.Value;
22         }
23         else
24         {
25             e.TokenReader.Push(intoToken.Value);
26         }
27     }
28     queryNode?.AddJoin(varToken.Value.Value, source, key1, key2, intoName);
29 }

5、解析group语句

标准LINQ语法中，group后面必需写element，即：group a by a.Age，在AScript脚本中可以简化不写element：group by a.Age 。

 1 /// <summary>
 2 /// from a in q1
 3 /// group a.Name by a.Age into g
 4 /// </summary>
 5 private void BuildGroup(DefaultSyntaxAnalyzer analyzer, TokenAnalyzingArgs e, BuildOptions createFullOptions, QueryNode queryNode)
 6 {
 7     var element = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _ByTokens);
 8     analyzer.ValidateNextToken(e.TokenReader, "by");
 9     var key = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _JoinEndTokens);
10     string intoName = null;
11     var intoToken = e.TokenReader.Read();
12     if (intoToken.HasValue)
13     {
14         if (intoToken.Value.IsSymbol("into"))
15         {
16             var intoNameToken = analyzer.ValidateNextToken(e.TokenReader, ETokenType.Word);
17             intoName = intoNameToken.Value.Value;
18         }
19         else
20         {
21             e.TokenReader.Push(intoToken.Value);
22         }
23     }
24     queryNode?.AddGroup(key, element, intoName);
25 }

6、解析orderby语句

增加了asc/desc关键字来简化ascending/descending。

 1 /// <summary>
 2 /// from a in q
 3 /// orderby a.Age ascending
 4 /// </summary>
 5 private void BuildOrderby(DefaultSyntaxAnalyzer analyzer, TokenAnalyzingArgs e, BuildOptions createFullOptions, QueryNode queryNode)
 6 {
 7     var key = analyzer.BuildOneStatement(e.BuildContext, e.ScriptContext, createFullOptions, e.TokenReader, e.Control, e.Ignore, _OrderbyEndTokens);
 8     var token = e.TokenReader.Read();
 9     string mode = null;
10     if (token.HasValue)
11     {
12         if (token.Value.IsSymbol("ascending") || token.Value.IsSymbol("asc") || token.Value.IsSymbol("descending") || token.Value.IsSymbol("desc"))
13         {
14             mode = token.Value.Value;
15         }
16         else
17         {
18             e.TokenReader.Push(token.Value);
19         }
20     }
21     queryNode?.AddOrderby(key, mode);
22 }

五、总结

我们通过自定义QueryNode和FromTokenHandler，扩展实现了LINQ语法，支持LINQ to Object和LINQ to SQL。

AScript开源地址：https://gitee.com/rockey627/AScript