Month: September 2015

Using JQuery DataTables , Entity Framework (EF) and Dynamic LINQ in conjunction


The Problem

Few months ago I started using DataTables (A plugin for JQuery in order to create neat tables in HTML). However, I noticed that twice I wrote code kind of similar for addressing the same issue. Thus with this project I’d like to set a base for extensible code which deals with the common request and response work related to DataTable API. The goal is to quickly provide an endpoint which provides data compatible with DataTable data source.
All the filtering, sorting and projection should happen at server side.

For this solution I’ll use the following components:
DataTables see https://www.datatables.net/
Dynamic LINQ see http://dynamiclinq.azurewebsites.net/

My project is inspired by https://github.com/kendo-labs/dlinq-helpers. Kendo Grid is a very robust table for HTML.

Remarks

– Filtering logic has not been taking into consideration, mainly because I didn’t find a common mechanism for receive filter information at server side. If this is a requirement, I will consider to mimic Kendo UI protocol for sending filtering information.
– Order and Search have been implemented
– Implementations for Array of Array and Array of Objects have been provided, however the developer is responsible of calling the proper methods.

The Solution

First of all The idea is to create an extensible module. At this time the filtering logic is pending to really have a useful. Thus I will start by showing the code of the DataTables module. The following code holds the models used by the model these classes are basically POCOS to be used to receive or send information.

using System.Runtime.Serialization;

namespace DataTables.Models
{
    /// <summary>
    /// Holds the information required for a given column
    /// </summary>
    [DataContract]
    public class ColumnRequest
    {
        /// <summary>
        /// Specific search information for the column
        /// </summary>
        [DataMember(Name = "search")]
        public SearchRequest Search { get; set; }

        /// <summary>
        /// Column's data source, as defined by columns.data.
        /// </summary>
        [DataMember(Name = "data")]
        public string Data { get; set; }

        /// <summary>
        /// Column's name, as defined by columns.name.
        /// </summary>
        [DataMember(Name = "name")]
        public string Name { get; set; }

        /// <summary>
        /// Flag to indicate if this column is search-able (true) or not (false). This is controlled by columns.searchable.
        /// </summary>
        [DataMember(Name = "searchable")]
        public bool? Searchable { get; set; }

        /// <summary>
        /// Flag to indicate if this column is orderable (true) or not (false). This is controlled by columns.orderable.
        /// </summary>
        [DataMember(Name = "orderable")]
        public bool? Orderable { get; set; }

    }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;

namespace DataTables.Models
{
    /// <summary>
    /// 
    /// </summary>
    [DataContract]
    public class SearchRequest
    {
        /// <summary>
        /// Search value
        /// </summary>
        [DataMember(Name = "value")]
        public string Value { get; set; }

        /// <summary>
        /// true if the search value should be treated as a regular expression for advanced searching, false otherwise. 
        /// </summary>
        [DataMember(Name = "regex")]
        public bool Regex { get; set; }

        public virtual string Operator { get; set; }

        public SearchRequest()
        {
            Operator = @"contains";
        }

        private static readonly IDictionary<string, string> operators = new Dictionary<string, string>
        {
            {"eq", "="},
            {"neq", "!="},
            {"lt", "<"},
            {"lte", "<="},
            {"gt", ">"},
            {"gte", ">="},
            {"startswith", "StartsWith"},
            {"endswith", "EndsWith"},
            {"contains", "Contains"},
            {"doesnotcontain", "Contains"}
        };

        public virtual string ToExpression(DataTableRequest request)
        {
            if (Regex)
                throw new NotImplementedException("Regular Expression is not implemented");

            string comparison = operators[Operator];
            List<string> expressions = new List<string>();
            foreach (var searchableColumn in request.Columns.Where(c => c.Searchable.HasValue && c.Searchable.Value))
            {
                if (Operator == "doesnotcontain")
                {
                    expressions.Add(string.Format("!{0}.ToString().{1}(\"{2}\")", searchableColumn.Data, comparison, Value));
                }
                else if (comparison == "StartsWith" || comparison == "EndsWith" || comparison == "Contains")
                {
                    expressions.Add(string.Format("{0}.ToString().{1}(\"{2}\")", searchableColumn.Data, comparison, Value));
                }
                else
                {
                    expressions.Add(string.Format("{0} {1} \"{2}\"", searchableColumn.Data, comparison, Value));
                }
            }
            return string.Join(" or ", expressions);
        }
    }
}

using System.Runtime.Serialization;

namespace DataTables.Models
{
    /// <summary>
    /// Contains the information about the sort request
    /// </summary>
    [DataContract]
    public class OrderRequest
    {
        /// <summary>
        /// Indicates the orientation of the sort "asc" for ascending or "desc" for desc
        /// </summary>
        [DataMember(Name = "dir")]
        public string Dir { get; set; }

        /// <summary>
        /// Column which contains the number of column which requires this sort.
        /// </summary>
        [DataMember(Name = "column")]
        public int? Column { get; set; }

        public string ToExpression(DataTableRequest request)
        {
            return string.Concat(request.Columns[Column.Value].Data, " ", Dir);
        }
    }
}


using System.Collections;
using System.Runtime.Serialization;

namespace DataTables.Models
{
    /// <summary>
    /// Encapsulates a Data Table response format
    /// </summary>
    /// <typeparam name="TData"></typeparam>
    [DataContract]
    public class DataTableResponse 
    {
        /// <summary>
        /// The draw counter that this object is a response to - from the draw parameter sent as part of the data request. Note that it is strongly recommended for security reasons that you cast this parameter to an integer, rather than simply echoing back to the client what it sent in the draw parameter, in order to prevent Cross Site Scripting (XSS) attacks.
        /// </summary>
        [DataMember(Name = "draw")]
        public int? Draw { get; set; }

        /// <summary>
        /// The data to be displayed in the table. 
        /// <remarks>
        /// This is an array of data source objects, one for each row, which will be used by DataTables. 
        /// Note that this parameter's name can be changed using the ajax option's dataSrc property.
        /// </remarks>
        /// </summary>
        [DataMember(Name = "data")]
        public IEnumerable Data { get; set; }

        /// <summary>
        /// Total records, before filtering (i.e. the total number of records in the database)
        /// </summary>
        [DataMember(Name = "recordsTotal")]
        public int? RecordsTotal { get; set; }

        /// <summary>
        /// Total records, after filtering (i.e. the total number of records after filtering has been applied - not just the number of records being returned for this page of data).
        /// </summary>
        [DataMember(Name = "recordsFiltered")]
        public int? RecordsFiltered { get; set; }

        /// <summary>
        /// Optional: If an error occurs during the running of the server-side processing script, you can inform the user of this error by passing back the error message to be displayed using this parameter. Do not include if there is no error.
        /// </summary>
        [DataMember(Name = "error")]
        public string Error { get; set; }
    }
}

using System.Collections.Generic;
using System.Runtime.Serialization;

namespace DataTables.Models
{
    /// <summary>
    /// This class holds the minimum amount of information provided by DataTable to perform a request on server side.
    /// </summary>
    [DataContract]
    public class DataTableRequest
    {
        /// <summary>
        /// Column data request description
        /// </summary>
        [DataMember(Name = "columns")]
        public List<ColumnRequest> Columns { get; set; }

        /// <summary>
        /// Column requested order description
        /// </summary>
        [DataMember(Name = "order")]
        public List<OrderRequest> Order { get; set; }

        /// <summary>
        /// Global search value. To be applied to all columns which have search-able as true.
        /// </summary>
        [DataMember(Name = "search")]
        public SearchRequest Search { get; set; }

        /// <summary>
        /// Paging first record indicator. This is the start point in the current data set (0 index based - i.e. 0 is the first record).
        /// </summary>
        [DataMember(Name = "start")]
        public int? Start { get; set; }

        /// <summary>
        /// Number of records that the table can display in the current draw. It is expected that the number of records returned will be equal to this number, unless the server has fewer records to return. 
        /// </summary>
        [DataMember(Name = "length")]
        public int? Length { get; set; }

        /// <summary>
        /// Draw counter. This is used by DataTables to ensure that the Ajax returns from server-side processing requests are drawn in sequence by DataTables (Ajax requests are asynchronous and thus can return out of sequence). 
        /// This is used as part of the draw return parameter. 
        /// </summary>
        [DataMember(Name = "draw")]
        public int? Draw { get; set; }
    }
}

They way I have decided to implement logic is through extension methods.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Dynamic;
using DataTables.Models;

namespace DataTables.Extensions
{
    public static class IQueryableExtensions
    {
        /// <summary>
        /// Computes a DataTables response based on the request. This response is compatible with an array of arrays
        /// </summary>
        /// <remarks>
        /// This response is compatible with an array of arrays
        /// </remarks>
        /// <typeparam name="TEntity">Entity to be returned in the data</typeparam>
        /// <param name="query">Initial data source. It will be part of the response after computations</param>
        /// <param name="request">Holds information about the request</param>
        /// <param name="getEntityFieldNamesFunc">Function which provides a sequence of fields of the entity to be used</param>
        /// <param name="getEntityAsEnumerableFunc">Function which converts an entity to an array</param>
        /// <returns></returns>
        public static DataTableResponse GetDataTableResponse<TEntity>(this IEnumerable<TEntity> query, DataTableRequest request,
            Func<TEntity, IEnumerable<string>> getEntityFieldNamesFunc, Func<TEntity, System.Collections.IEnumerable> getEntityAsEnumerableFunc)
        {
            // When using Arrays set the columns data before performing operations.
            SetColumnData(request, getEntityFieldNamesFunc);

            var response = GetDataTableResponse(ref query, request);

            // Converts the result compatible with an array (required by default for DataTables)
            response.Data = query.Select(getEntityAsEnumerableFunc);
            return response;
        }

        /// <summary>
        /// Computes a DataTables response based on the request. This response is compatible with an array of objects.
        /// </summary>
        /// <remarks>
        /// This response is compatible with an array of objects.
        /// </remarks>
        /// <typeparam name="TEntity">Entity to be returned in the data</typeparam>
        /// <param name="query">Initial data source. It will be part of the response after computations</param>
        /// <param name="request">Holds information about the request</param>
        /// <returns></returns>
        public static DataTableResponse GetDataTableResponse<TEntity>(this IEnumerable<TEntity> query, DataTableRequest request)
        {
            // When using Arrays set the columns data before performing operations.
            var response = GetDataTableResponse(ref query, request);

            // Converts the result compatible with an array (required by default for DataTables)
            response.Data = query;
            return response;
        }

        private static DataTableResponse GetDataTableResponse<TEntity>(ref IEnumerable<TEntity> query, DataTableRequest request)
        {
            var response = new DataTableResponse();
            // Setting up response
            response.Draw = request.Draw;
            response.RecordsTotal = query.Count();

            // sorting
            query = query.OrderBy(string.Join(",", request.Order.Select(o => o.ToExpression(request))));

            // search
            if (request.Search != null && !string.IsNullOrEmpty(request.Search.Value))
            {
                query = query.Where(request.Search.ToExpression(request));
            }

            // filtering


            // Counting results
            response.RecordsFiltered = query.Count();

            // Returning page (subset of filtered records)
            if (request.Start.HasValue && request.Start.Value > 0)
                query = query.Skip(request.Start.Value);
            if (request.Length.HasValue && request.Length.Value >= 0)
                query = query.Take(request.Length.Value);
            return response;
        }

        private static void SetColumnData<TEntity>(DataTableRequest request, Func<TEntity, IEnumerable<string>> func)
        {
            SetColumnData(request, func(default(TEntity)).ToArray());
        }

        private static void SetColumnData(DataTableRequest request, params string[] fields)
        {
            for (int index = 0; index < fields.Length && index < request.Columns.Count; index++)
            {
                var fieldIndex = 0;
                if (int.TryParse(request.Columns[index].Data, out fieldIndex))
                    request.Columns[index].Data = fields[fieldIndex];
            }
        }
    }
}

The previous code takes advantage Dynamic LINQ to compute queries, and using as input the request information. With the previous code in place, we can use in our Controller the following code:

using DataTableSample.Services;
using DataTables.Models;
using DataTableSample.Models;
using System.Linq.Dynamic;
using DataTables.Extensions;

namespace DataTableSample.Controllers
{
    public class EmployeesController : ApiController
    {
        private CompanyModel db = new CompanyModel();

        // GET: api/Employees
        public async Task<IHttpActionResult> GetEmployees([FromUri]DataTableRequest request)
        {
            var isUsingArrayOfArrays = request.Columns.Where(c => { var n = 0; return int.TryParse(c.Data, out n); }).Any();
            var response = isUsingArrayOfArrays
                ? db.Employees.AsNoTracking().AsQueryable().GetDataTableResponse(request, GetEntityFieldNames, GetEntityAsEnumerable)
                : db.Employees.AsNoTracking().AsQueryable().GetDataTableResponse(request)
                ;
            return Json(response);
        }

        /// <summary>
        /// Converts the Entity to a sequence of values (an Array)
        /// </summary>
        /// <param name="e"></param>
        /// <returns></returns>
        private System.Collections.IEnumerable GetEntityAsEnumerable(Employee e)
        {
            yield return e.FirstName;
            yield return e.LastName;
            yield return e.Position;
            yield return e.Office;
            yield return e.StartDate;
            yield return e.Salary;
        }

        /// <summary>
        /// Provides the name of the Columns used in the Array
        /// </summary>
        /// <param name="e"></param>
        /// <returns></returns>
        private IEnumerable<string> GetEntityFieldNames(Employee e)
        {
            yield return nameof(e.FirstName);
            yield return nameof(e.LastName);
            yield return nameof(e.Position);
            yield return nameof(e.Office);
            yield return nameof(e.StartDate);
            yield return nameof(e.Salary);
        }
    }
}

The last portion of code is

<!DOCTYPE html>
<html>
<head>
    <title></title>
    <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.9/css/jquery.dataTables.min.css" />
    <meta charset="utf-8" />
</head>
<body>
    <h1>Using Objects</h1>
    <table id="example1" class="display" cellspacing="0" style="width:100%;">
        <thead>
            <tr>
                <th>First name</th>
                <th>Last name</th>
                <th>Position</th>
                <th>Office</th>
                <th>Start date</th>
                <th>Salary</th>
            </tr>
        </thead>
        <tfoot>
            <tr>
                <th>First name</th>
                <th>Last name</th>
                <th>Position</th>
                <th>Office</th>
                <th>Start date</th>
                <th>Salary</th>
            </tr>
        </tfoot>
    </table>
    <h1>Using Arrays</h1>
    <table id="example2" class="display" cellspacing="0" style="width:100%;">
        <thead>
            <tr>
                <th>First name</th>
                <th>Last name</th>
                <th>Position</th>
                <th>Office</th>
                <th>Start date</th>
                <th>Salary</th>
            </tr>
        </thead>
        <tfoot>
            <tr>
                <th>First name</th>
                <th>Last name</th>
                <th>Position</th>
                <th>Office</th>
                <th>Start date</th>
                <th>Salary</th>
            </tr>
        </tfoot>
    </table>
</body>
</html>
<script src="Scripts/jquery-1.7.js"></script>
<script src="Scripts/DataTables/jquery.dataTables.js"></script>
<script type="text/javascript">
    $(document).ready(function () {
        $('#example1').DataTable({
            "processing": true,
            "serverSide": true,
            "ajax": "api/Employees",
            "columns": [
                { "data": "FirstName" },
                { "data": "LastName" },
                { "data": "Position" },
                { "data": "Office" },
                { "data": "StartDate" },
                { "data": "Salary" }
            ]
        });

        $('#example2').DataTable({
            "processing": true,
            "serverSide": true,
            "ajax": "api/Employees"
        });
    });
</script>

By running the application the two tables will be populated, one of them will be using Array of Arrays and the second will be using an Array of objects.

The full source code can be downloaded from:
https://github.com/hmadrigal/playground-dotnet/tree/master/MsWebApi.DataTables

Cheers,
Herb

Advertisements

Generating like-JIRA tickets using Entity Framework (EF6) and MS SQL (TSQL)


The problem

We have a Master/Detail relationship with two entities, and we want to identify detail records by using a friendly ID. The friendly ID is based on a master code (which it is specified in the Master Table) and a Detail Number (which works like a counter on each given master code). For example

Master Table ( When IsAutoGen is 1, then Details should auto generate an friendly ID using the AutoGenPrefix)

Id          IsAutoGen AutoGenPrefix
----------- --------- -------------
1           1         GRPA
2           1         GRPB
5           0         NULL

Detail table AutoGenPreix-AutoGenNumber identify uniquely each ticket by providing a friendly-auto-Incremental Id.

Id          MasterId    AutoGenPrefix AutoGenNumber
----------- ----------- ------------- -------------
361         1           GRPA          1
362         2           GRPB          1
363         5           NULL          1
364         1           GRPA          2
365         2           GRPB          2
366         5           NULL          2
367         1           GRPA          3
368         2           GRPB          3

Remarks

Certainly this problem may have many different solutions. For example it is technically possible to do all the computations only using C# and Entity Framework, thread locks and database transactions to support concurrency. However by doing it so, the developer will be responsible for dealing with concurrency, it will lock the database for longer periods since changes happens at business logic layer. Thus, this approach attempts to delegate the concurrency handling task to the database engine and reduce the lock time when possible. This solution is very particular to Entity Framework and Microsoft SQL, although ORMs and Triggers are concepts available in most of the platforms nowadays.

The solution

Just in case, you have skipped the test before, I encourage you to read the remarks. That said, First lets create our sample tables by using the following T-SQL script:

CREATE TABLE [dbo].[Master] (
    [Id]            INT         IDENTITY (1, 1) NOT NULL,
    [IsAutoGen]     BIT         CONSTRAINT [DF_Master_IsAutoGen] DEFAULT ((0)) NOT NULL,
    [AutoGenPrefix] VARCHAR (5) NULL,
    CONSTRAINT [PK_Master] PRIMARY KEY CLUSTERED ([Id] ASC)
);
GO
CREATE TABLE [dbo].[Detail] (
    [Id]            INT         IDENTITY (1, 1) NOT NULL,
    [MasterId]      INT         NOT NULL,
    [AutoGenPrefix] VARCHAR (5) NULL,
    [AutoGenNumber] INT         NULL,
    CONSTRAINT [PK_Detail] PRIMARY KEY CLUSTERED ([Id] ASC),
    CONSTRAINT [FK_Detail_Master] FOREIGN KEY ([MasterId]) REFERENCES [dbo].[Master] ([Id])
);
GO

As simple as two tables, in order to establish a master-detail relationship. Now with the same ease, lets see the suggested trigger:

CREATE TRIGGER  [dbo].[Insert_Detail_ReadableId]
ON [dbo].[Detail]
INSTEAD OF INSERT
AS
BEGIN
   SET NOCOUNT OFF;
	INSERT INTO [Detail] ([MasterId],[AutoGenPrefix], [AutoGenNumber])
	SELECT 
		I.[MasterId] AS [MasterId],
		M2.AutoGenPrefix AS [AutoGenPrefix],
		ISNULL(ROW_NUMBER() OVER(ORDER BY I.[Id]) + M.[AutoGenNumber],1) AS [AutoGenNumber]
	FROM INSERTED  I
	LEFT JOIN [Master] M2 ON I.[MasterId] = M2.[Id]
	LEFT JOIN (
		SELECT 
			[MasterId],
			MAX(ISNULL([AutoGenNumber],0)) AS [AutoGenNumber]
		FROM [AutoGenDB].[dbo].[Detail]
		GROUP BY [MasterId]
	) M ON I.[MasterId] = M.[MasterId]

	SELECT [Id] FROM [Detail] WHERE @@ROWCOUNT > 0 AND [Id] = SCOPE_IDENTITY();
END

Notice that the trigger is responsible of performing the INSERT since it has been configured to be INSTEAD OF INSERT. The triggers uses the INSERTED variable to hold a table with all the records that should have been inserted. Thus the application performs a INSERT INTO using as data source the INSERTED table, the inserted max AutoGenNumber per AutoGenPrefix and a row number to be added to the Max AutoGenNumber.
Pay attention to the last line, which returns the inserted ID, this is required by Entity Framework to detect as successful the INSERT operation (see StackOverflow | OptimisticConcurrencyException — SQL 2008 R2 Instead of Insert Trigger with Entity Framework or StackOverflow | error when inserting into table having instead of trigger from entity data framework )

This is all we need in the TSQL level. Every time we attempt to INSERT a new record, right after the INSERT request, our code will perform the inserted by computing some additional data.

In our C# code, I’ll assume we’re using an EDMX file and using DB first approach. Thus, we usually only update our DbContext by updating our models from Visual Studio (VS).

using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations.Schema;
using System.Data.Entity.Infrastructure;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace AutoGenDB
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var context = new AutoGenDB.AutoGenDBEntities())
            {
                // if master table is empty, then it inserts some master records.
                if (!context.Masters.Any())
                {
                    for (int masterIndex = 0; masterIndex < 5; masterIndex++)
                    {
                        var isAutoGenEnabled = masterIndex % 2 == 0;
                        var newMaster = new Master { IsAutoGen = isAutoGenEnabled, AutoGenPrefix = isAutoGenEnabled ? "GRP" + Convert.ToChar((masterIndex % 65) + 65) : default(string) };
                        context.Masters.Add(newMaster);
                    }
                    context.SaveChanges();
                }

                // Inserts some details on different groups
                List<Detail> details = new List<Detail>();
                List<Master> masters = context.Masters.AsNoTracking().ToList();
                for (int i = 0; i < 100; i++)
                {
                    var masterId = masters[i % masters.Count].Id;
                    details.Add(new Detail() { MasterId = masterId });
                }
                context.Details.AddRange(details);
                context.SaveChanges();

                // Prints 10 auto generated IDS
                foreach (var master in context.Masters.AsNoTracking().Take(3))
                    foreach (var detail in master.Details.Take(5))
                        Console.WriteLine(master.IsAutoGen ? "Ticket:{0}-{1}" : @"N/A", detail.AutoGenPrefix, detail.AutoGenNumber);
            }
            Console.ReadKey();
        }
    }
}

The previous C# code inserts some records and prints the information generated by the Trigger. At this point there is only one missed component.
If we run the code as it is. The application won’t load the auto generated values, this is because Entity Framework does not know that these columns were populated. An option is to instruct Entity Framework about when it should reload the entities. The following code, decorates the Detail class with an IDetail interface which can be used in the DbContext to identify the entity type that should be reloaded.

using System;
using System.Collections.Generic;
using System.Data.Entity;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace AutoGenDB
{
    interface IDetail
    {
        int? AutoGenNumber { get; set; }
        string AutoGenPrefix { get; set; }
        int Id { get; set; }
        Master Master { get; set; }
        int MasterId { get; set; }
    }

    public partial class AutoGenDBEntities : DbContext
    {
        public override int SaveChanges()
        {
            var entriesToReload = ChangeTracker.Entries<IDetail>().Where(e => e.State == EntityState.Added).ToList();
            int rowCount = base.SaveChanges();
            if (rowCount > 0 && entriesToReload.Count > 0)
                entriesToReload.ForEach(e => e.Reload());
            return rowCount;
        }

    }

    public partial class Detail : AutoGenDB.IDetail
    {

    }
}

Please notice how the subclass of the DbContext is overriding the SaveChanges method in order to request to reload the instances of IDetail. Reloading entities may or may not be desirable in depending on the application.
With these overrides, the DbContext will know when a given entity should be reloaded.

As usual the code sample can be check at https://github.com/hmadrigal/playground-dotnet/tree/master/MsDotNet.EntityFrameworkAndTriggers