Files
sqltoolsservice/src/Microsoft.InsightsGenerator/RulesEngine.cs
Karl Burtram 5cf5b59a0d Insights generator project (#1066)
* InsightsGenerator project template files

* Add insights projects to SLN

* Setting up siggen class (#1003)

* Setting up siggen class

* fixed the learn method

* Refactoring code
Fixed compile errors

* renamed results to result

* Basic transformation logic (#1004)

* Fix a couple bugs and add a simple test (#1007)

* Fix a couple bugs and add a simple test

* More tests and bug fix

* Nara/workflow (#1006)

* added a queue processor

* ordered  using statements

* Armemon/analytics (#1008)

* Basic transformation logic

* changed some structure of siggen

* added sum and average method, as well as select rows by input name

* add insights to results

* min, max added

Co-authored-by: Karl Burtram <karlb@microsoft.com>
Co-authored-by: Aasim Khan <aasimkhan30@gmail.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>

* Added rules engine base implementation (#1005)

* Added rules engine base implementation

* update comments

* addressing comments

* adding template text to columnheaders object

* adding template text to columnheaders object

* fixing columnheaders class

* Added test

* Added Template Parser unit test in Test project

* Deleted unnecessary files and reverted the files that were modified by mistake

Co-authored-by: Jinjing Arima <jiarima@microsoft.com>

* Insights generator message handler placeholder (#1013)

* Aasim/insights/insight methods (#1014)

* Basic transformation logic

* changed some structure of siggen

* Added top and bottom insight functions

* Added top, bottom insights
Added tests for top, bottom insights

* Armemon/insights2 (#1011)

* max and min insightsperslice, and tests

* got rid of unneccesssary function

* get indexes

Co-authored-by: Arslan Memon <armemon@microsoft.com>

* Armemon/insights2 (#1012)

* max and min insightsperslice, and tests

* got rid of unneccesssary function

* get indexes

* learn for stringinputtyype

* add learn implentation

Co-authored-by: Arslan Memon <armemon@microsoft.com>

* Added Tests
Removed duplicate methods

Co-authored-by: Karl Burtram <karlb@microsoft.com>
Co-authored-by: arslan9955 <53170027+arslan9955@users.noreply.github.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>

* Armemon/insights2 (#1016)

* Basic transformation logic

* changed some structure of siggen

* Added top and bottom insight functions

* Added top, bottom insights
Added tests for top, bottom insights

* max and min insightsperslice, and tests

* got rid of unneccesssary function

* get indexes

* learn for stringinputtyype

* add learn implentation

* add unique inputs

* fix merge error

* add to result

Co-authored-by: Karl Burtram <karlb@microsoft.com>
Co-authored-by: Aasim Khan <aasimkhan30@gmail.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>

* Added all the templates (#1015)

* Added all the templates
Added a method to find matched template

* Added a function to replace # and ## values in a template

* Added ReplaceHashesInTemplate call

* Added comments

* Updated the template txt

* Updated GetTopHeadersWithHash function to add #toplist

* Updated the logic per offline discussion with Hermineh

* Update request handler contract to take array (#1020)

* added rulesengine findmatchingtemplate (#1019)

* Add support for getting DacFx deploy options from a publish profile (#995)

* add support for getting options from a publish profile

* update comments

* set values for default options if they aren't specified in the publish profile

* addressing comments

* Updating to latest DacFx for a bug fix (#1010)

* added rulesengine findmatchingtemplate

* Update DacFx deploy and generate script with options (#998)

* update deploy and generate script to accept deployment options

* add tests

* add test with option set to true

* merge

* merge

* incorporated FindMatchedTemplate

Co-authored-by: Kim Santiago <31145923+kisantia@users.noreply.github.com>
Co-authored-by: Udeesha Gautam <46980425+udeeshagautam@users.noreply.github.com>

* -Added logic for Insights Generator Service Handler (#1017)

* -Added logic for Insights Generator Service Handler

* Fixed some logic

* Adding workflow test

* Update transform and add tests (#1024)

* Jiarima/fix rules engine logic (#1021)

* Added all the templates
Added a method to find matched template

* Added a function to replace # and ## values in a template

* Added ReplaceHashesInTemplate call

* Added comments

* Updated the template txt

* Updated GetTopHeadersWithHash function to add #toplist

* Updated the logic per offline discussion with Hermineh

* Update with the fixes

* Updated template and foreach conditions

* Added distinct

* Updated tests according to the logic change (#1026)

* Nara/remove queing (#1023)

* loc update (#914)

* loc update

* loc updates

* Add support for getting DacFx deploy options from a publish profile (#995)

* add support for getting options from a publish profile

* update comments

* set values for default options if they aren't specified in the publish profile

* addressing comments

* Updating to latest DacFx for a bug fix (#1010)

* Update DacFx deploy and generate script with options (#998)

* update deploy and generate script to accept deployment options

* add tests

* add test with option set to true

* intermediate check in for merge, transformed not working

* intermediate check in for merge, transformed not working

* added test case

* merged

Co-authored-by: khoiph1 <khoiph@microsoft.com>
Co-authored-by: Kim Santiago <31145923+kisantia@users.noreply.github.com>
Co-authored-by: Udeesha Gautam <46980425+udeeshagautam@users.noreply.github.com>

* Output data types from transform (#1029)

* Fix bug process input_g (#1030)

* Fixed the insight generator service (#1028)

* Jiarima/added more testings (#1031)

* Added another test
Updated ReplaceHashesInTemplate function to return string instead of Template

* Added third test

* Merged

* Reverted the workflow file to match with the one in hack/insights

* Bugs fixes to hook insights up to ADS (#1033)

* Bug fixes for hack insights (#1032)

* Fixed the minColumn index bug in Data Transformation
Fixed the template matching logic.

* Adding changes from PR

* Try to fix Workflow tests

* Readd workflow tests

* Fix template load location

Co-authored-by: Aasim Khan <aasimkhan30@gmail.com>
Co-authored-by: Nara <NaraVen@users.noreply.github.com>
Co-authored-by: arslan9955 <53170027+arslan9955@users.noreply.github.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>
Co-authored-by: gadudhbh <68879970+gadudhbh@users.noreply.github.com>
Co-authored-by: Jinjing Arima <jiarima@microsoft.com>
Co-authored-by: jiarima <68882862+jiarima@users.noreply.github.com>
Co-authored-by: Kim Santiago <31145923+kisantia@users.noreply.github.com>
Co-authored-by: Udeesha Gautam <46980425+udeeshagautam@users.noreply.github.com>
Co-authored-by: khoiph1 <khoiph@microsoft.com>
2020-09-03 14:15:51 -07:00

213 lines
7.6 KiB
C#

//
// Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.
//
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
namespace Microsoft.InsightsGenerator
{
public class RulesEngine
{
public static List<Template> Templates;
public static List<string> TopListHashHeaders = new List<string>{ "#top", "#averageSlice", "#topPerslice" , "#bottom"};
public RulesEngine()
{
Templates = GetTemplates();
}
public static ColumnHeaders TemplateParser(string templateContent)
{
ColumnHeaders ch = new ColumnHeaders();
var processedText = Regex.Replace(templateContent, @",|\\n", "");
ch.Template = templateContent;
List<string> keyvalue = processedText.Split(' ').Select(s => s.Trim()).ToList();
foreach (string s in keyvalue)
{
if (s.StartsWith("#"))
{
string headers = s.Substring(1, s.Length - 1);
if (headers.StartsWith("#"))
{
ch.DoubleHashValues.Add( "##" + headers.Substring(1, headers.Length - 1));
}
else
{
if (headers != "placeHolder")
{
ch.SingleHashValues.Add("#" + headers);
}
}
}
}
return ch;
}
/// <summary>
/// Find a matched template based on the single hash values
/// </summary>
/// <param name="singleHashHeaders"></param>
/// <returns></returns>
public static string FindMatchedTemplate(List<List<string>> singleHashHeaders, DataArray columnInfo)
{
var resultTemplate = new StringBuilder();
if (Templates == null)
{
Templates = GetTemplates();
}
var headersWithSingleHash = GetTopHeadersWithHash(singleHashHeaders);
foreach (var template in Templates)
{
var columnHeaders = TemplateParser(template.Content);
var singleHashFromTemplate = columnHeaders.SingleHashValues.Distinct();
var isMatched = true;
// all the values from template needs to be found in the input from SigGen
foreach (var hashFromTemplate in singleHashFromTemplate)
{
if (!headersWithSingleHash.Contains(hashFromTemplate.ToLower()))
{
isMatched = false;
break;
}
}
if (isMatched)
{
// Replace # and ## values in template with actual values
resultTemplate.AppendLine(ReplaceHashesInTemplate(singleHashHeaders, columnInfo, template) + "\n");
}
}
// No matched Template found
return resultTemplate.ToString();
}
private static string ReplaceHashesInTemplate(List<List<string>>singleHashList, DataArray columnInfo, Template template)
{
StringBuilder modifiedTemp = new StringBuilder(template.Content);
// Replace single hash values
foreach (var line in singleHashList)
{
// Example of a single list (line):
// "top", "3", " input (value) %OfValue ", " input (value) %OfValue ", " input (value) %OfValue "
var headerInputs = line.ToArray();
string header = "#" + headerInputs[0];
if (TopListHashHeaders.Contains(header))
{
if(!modifiedTemp.ToString().Contains(header))
{
continue;
}
//First replace the header with the second value in the list
modifiedTemp.Replace(header, headerInputs[1]);
StringBuilder remainingStr = new StringBuilder();
for (int i = 2; i < headerInputs.Length; i++)
{
// Append all the rest of the elemet in the array separated by new line
remainingStr.AppendLine(headerInputs[i]);
}
modifiedTemp.Replace("#placeHolder", remainingStr.ToString());
}
else
{
modifiedTemp.Replace("#" + headerInputs[0], headerInputs[1]);
}
}
// Replace double hash values
var transformedColumnArray = columnInfo.TransformedColumnNames.ToArray();
var columnArray = columnInfo.ColumnNames.ToArray();
for (int p = 0; p < columnInfo.TransformedColumnNames.Length; p++)
{
modifiedTemp.Replace("##" + transformedColumnArray[p], columnArray[p]);
}
return modifiedTemp.ToString();
}
private static List<string> GetTopHeadersWithHash(List<List<string>> singleHashHeaders)
{
var topHeaderList = new List<string>();
foreach (var list in singleHashHeaders)
{
topHeaderList.Add("#" + list.First().ToLower());
}
return topHeaderList;
}
/// <summary>
/// Retrieve all the templates and template ids
/// </summary>
/// <returns>All the template & id comnbination </returns>
public static List<Template> GetTemplates()
{
var templateHolder = new List<Template>();
string assemblyPath = System.Reflection.Assembly.GetExecutingAssembly().Location;
string assemblyDirectoryPath = System.IO.Path.GetDirectoryName(assemblyPath);
string templateFilePath = Path.Combine(assemblyDirectoryPath, "Templates", "templates.txt");
using (StreamReader streamReader = new StreamReader(templateFilePath, Encoding.UTF8))
{
int temId = 0;
var wholeText = streamReader.ReadToEnd();
var templateArray = wholeText.Split(new[] { "Template " }, StringSplitOptions.None).ToList();
foreach (var line in templateArray.Where(r => r != string.Empty))
{
var parts = line.Split(new[] { Environment.NewLine }, StringSplitOptions.None).ToList();
temId = int.Parse(parts[0]);
templateHolder.Add(new Template(temId, parts[1]));
}
return templateHolder;
}
}
}
/// <summary>
/// Template container to hold the template id + template body combination
/// </summary>
public class Template
{
public Template(int id, string content)
{
Id = id;
Content = content;
}
public int Id { get; set; }
public string Content { get; set; }
}
public class ColumnHeaders
{
public ColumnHeaders()
{
SingleHashValues = new List<string>();
DoubleHashValues = new List<string>();
Template = null;
}
public List<string> SingleHashValues { get; set; }
public List<string> DoubleHashValues { get; set; }
public string Template { get; set; }
}
}