ESENT – the „ancient NoSQL DB“ made by Windows

 

image.png

Those of you who use RavenDB might have heard about ESENT already. In the inside RavenDB uses the “Extensible Storage Engine” which is included into Windows since XP. Read here why Ayende is looking for an alternative for a while – although ESENT is basically reliable but that’s just a side note.

ESENT is used by Microsoft in many different areas – like the Active-Directory or the Exchange Mailbox information’s. So basically it is used whenever a huge amount of information’s need some structure. I’m going to quote this Blogpost because it describes the features of ESENT pretty well.

Features

Significant technical features of ESENT include:

- ACID transactions with savepoints, lazy commits, and robust crash recovery.
- Snapshot isolation.
- Record-level locking (multi-versioning provides non-blocking reads).
- Highly concurrent database access.
- Flexible meta-data (tens of thousands of columns, tables, and indexes are possible).
- Indexing support for integer, floating point, ASCII, Unicode, and binary columns.
- Sophisticated index types, including conditional, tuple, and multi-valued.
- Columns that can be up to 2GB with a maximum database size of 16TB.

Note: The ESENT database file cannot be shared between multiple processes simultaneously. ESENT works best for applications with simple, predefined queries; if you have an application with complex, ad-hoc queries, a storage solution that provides a query layer will work better for you.

 

First steps with ESENT – ManagedESENT

Of course it is possible to talk in C++ to ESENT but it just doesn’t look right. A better alternative is:

 

image

The project consists of two different parts:

- A .NET wrapper to ESENT.dll

- PersistentDictionary which takes the .NET wrapper and offers a nice API

Additionally you’ll get some additional documentation.

 

.NET wrapper

Who plans to use ESENT in a more “pure” way could use the .NET wrapper. But you better know the insides of ESENT before you do so. Here is the code of a low-level sample-application.

 

namespace EsentSample { using System; using System.Text; using Microsoft.Isam.Esent.Interop; public class EsentSample { /// <summary> /// Main routine. Called when the program starts. /// </summary> /// <param name="args"> /// The arguments to the program. /// </param> public static void Main(string[] args) { JET_INSTANCE instance; JET_SESID sesid; JET_DBID dbid; JET_TABLEID tableid; JET_COLUMNDEF columndef = new JET_COLUMNDEF(); JET_COLUMNID columnid; // Initialize ESENT. Setting JET_param.CircularLog to 1 means ESENT will automatically // delete unneeded logfiles. JetInit will inspect the logfiles to see if the last // shutdown was clean. If it wasn't (e.g. the application crashed) recovery will be // run automatically bringing the database to a consistent state. Api.JetCreateInstance(out instance, "instance"); Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null); Api.JetInit(ref instance); Api.JetBeginSession(instance, out sesid, null, null); // Create the database. To open an existing database use the JetAttachDatabase and // JetOpenDatabase APIs. Api.JetCreateDatabase(sesid, "edbtest.db", null, out dbid, CreateDatabaseGrbit.OverwriteExisting); // Create the table. Meta-data operations are transacted and can be performed concurrently. // For example, one session can add a column to a table while another session is reading // or updating records in the same table. // This table has no indexes defined, so it will use the default sequential index. Indexes // can be defined with the JetCreateIndex API. Api.JetBeginTransaction(sesid); Api.JetCreateTable(sesid, dbid, "table", 0, 100, out tableid); columndef.coltyp = JET_coltyp.LongText; columndef.cp = JET_CP.ASCII; Api.JetAddColumn(sesid, tableid, "column1", columndef, null, 0, out columnid); Api.JetCommitTransaction(sesid, CommitTransactionGrbit.LazyFlush); // Insert a record. This table only has one column but a table can have slightly over 64,000 // columns defined. Unless a column is declared as fixed or variable it won't take any space // in the record unless set. An individual record can have several hundred columns set at one // time, the exact number depends on the database page size and the contents of the columns. Api.JetBeginTransaction(sesid); Api.JetPrepareUpdate(sesid, tableid, JET_prep.Insert); string message = "Hello world"; Api.SetColumn(sesid, tableid, columnid, message, Encoding.ASCII); Api.JetUpdate(sesid, tableid); Api.JetCommitTransaction(sesid, CommitTransactionGrbit.None); // Use JetRollback() to abort the transaction // Retrieve a column from the record. Here we move to the first record with JetMove. By using // JetMoveNext it is possible to iterate through all records in a table. Use JetMakeKey and // JetSeek to move to a particular record. Api.JetMove(sesid, tableid, JET_Move.First, MoveGrbit.None); string buffer = Api.RetrieveColumnAsString(sesid, tableid, columnid, Encoding.ASCII); Console.WriteLine("{0}", buffer); // Terminate ESENT. This performs a clean shutdown. Api.JetCloseTable(sesid, tableid); Api.JetEndSession(sesid, EndSessionGrbit.None); Api.JetTerm(instance); } } }

All it does is to save one value – that’s it. Uhhh… I know but as I said before it is Low-Level ;-)

A better inside is available in the Stock-Sample. In this example a complex data base is created by using the ManagedEsent API and it saves stock information’s. But there is still an easier way:

 

Persistent Dictionary

This alternative behaves mainly like a dictionary – but saved as an Esent-database. There are some more details in the documentation.

 

public static void Main(string[] args) { var dictionary = new PersistentDictionary<string, string>("Names"); Console.WriteLine("What is your first name?"); string firstName = Console.ReadLine(); if (dictionary.ContainsKey(firstName)) { Console.WriteLine("Welcome back {0} {1}", firstName, dictionary[firstName]); } else { Console.WriteLine( "I don't know you, {0}. What is your last name?", firstName); dictionary[firstName] = Console.ReadLine(); } }

The code is quite simple and in the background a folder named “Names” is created. That’s where the Esent database is situated:

image

Unfortunately I don’t know if it’s possible to readout the information with a tool (without code).

 

About the Performance

It takes a while to enter 1.000.000 entries into a database with using a PersistentDictionary and it takes even longer with using GUIDs instead of Integer. Maybe it takes some time to get expressive numbers. The team published some performance information here and here.

 

LINQ support!

There is a LINQ support for the PersistentDictionary since Version 1.6.

 

Windows Store Apps

Since Esent is a part of Windows and some sections might use the API it is possible to use Esent with Windows Store Apps. It should work since version 1.8 but it seems like the PersistentDictionary is not supported actually. Read more about it here.

 

More links about ManagedEsent

- Performance of Persistent Dictionary (CodePlex Discussion)

- Ayendes Fork

- Blog by a Microsoft employee about Esent and the ManagedEsent API

 

Esent Serialization

Beside the “PersistentDictionary” there is another project using the ManagedEsent:

 

image

 

Basically the project helps to integrate data structures easily in Esent. Have a look on it in this demo.

 

Result

Esent has a nasty API but is still used even in Windows 8. With the help of tools like ManagedEsent API, PersistentDictionary and Esent Serialization the first steps are quite easy. If they are the tools of choice depends mainly on the purpose – since there are numerous alternatives. But the fact that RavenDB is partly build on Esent is a proof that it is at least worth a shot.

P.S.: Since the code in this blogpost is mainly from official sources I won’t upload anything on GitHub. The necessary ManagedEsent.dll is available on NuGet.

 

Question: Anyone some experience with Esent to share?

If you enjoyed this post, please consider leaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.

About the author

Written by

Learn more about our team.

Comment on this post

Recent Posts

  • How to access an Azure Website with the local IIS Manager

      Since the end of February it is possible to access an Azure Website with the IIS Manager. Although the Azure Management site offers some information’s there are more details visible at the IIS Manager.   For the connection you will need an IIS Manager and the IIS Manager for Remote Administration Extension. It’s also […]

  • image1929-570x143_thumb.png
    Create and validate own Json-Web-Tokens (JWTs)

    If you are interested in web authentication you probably have heard about JSON Web tokens (JWT). What is a JWT? Maybe I’m not using the correct security termination but however: JWTs are used to exchange claims between two systems. For example: You want to log on to a service (like Facebook, Twitter, etc.) and want […]

  • Micro-Optimization: how to shrink or „embed“ pictures

      I’m currently working on the “CodeInside Dashboard” and since the page structure isn’t that difficult it should be possible to fulfill all of Google Pagespeed or Yahoos YSlow recommendations. One of the rules was to optimize the 4 PNGs that are embedded on the page.   Before – without optimization: Below you can see […]

  • image1979-570x194.png
    Move to Windows Azure – VMs, Word Press Migration, DNS changes

    Since mid January this blogs runs on a WordPress installation in an Azure VM. Because I always thought that the subject is quite complicated this blogpost offers a view behind the scenes. Why this move? So far this blog (both German and English Version) runs on a hoster somewhere in Germany. The main problem with this […]

  • Windows Azure Active Directory – CRUD for users and groups

      Windows Azure Active Directory? If you are not informed about the subject I recommend you to have a look on this Azure Info site. Which resources are there? The Azure AD contains the following entities: - Users - Groups - Contacts - Roles Access to the directory or on the “directory graph” Although the […]

Support us