Data Exchange Framework:Simple Import

In this post we will create a simple DXF process to import data from a SQL database to sitecore. This post will help us understand few core concepts of DXF like processor, convertor and plugin.

The code sample and examples in this post just explains the concepts. It should not be considered for a real DXF implementaion. In subsequent posts we will refine the code to understand other core concepts and incorporate them.


Source: SQL database

Target: Sitecore

  • To process the data import, we need to read from a SQL table and create coresponding items in sitecore.To read the value from SQL, we would need connection string and a SQL script. We can create a template to hold these information. A converter is used to read data from sitecore and create a Plugin object. A plugin is a simple PoCo object.
  • Converter writes the plugin to PipelineStepContext. 
  • DXF pass the PipelineStepContext to processor.
  • Processor reads the plugin and process the data.

Step 1 : Create a New Provider

Navigate to "/sitecore/templates/Data Exchange". Click on "Create Provider" in the Data Exchange Ribbon.


Enter the name "DbProvider" and click on "OK". It will initiate a command template and create set of folders and templates for the new provider under "/sitecore/templates/Data Exchange/Providers/DbProvider".



Step 2: Create a new Processor Template

Create a new template ReadDatabaseConfigurationProcessorStep using the BaseTemplate "/sitecore/templates/Data Exchange/Framework/Pipeline Steps/Base Pipeline Steps/Base Pipeline Step". Create the template fields as defined below.


Step 3: Define a Plugin

Create a new visualstudio project "Creekworm.DXF.DatabaseProvider". Add reference to "Sitecore.DataExchange", "Sitecore.DataExchange.DataAccess", "Sitecore.DataExchange.Local" and "Sitecore.DataExchange.Local.Sdk" nuget packages.

Create a new folder "Plugin" and add a new C# file called "DatabaseSourceSetting" inheriting Sitecore.DataExchange.IPlugin

public class DatabaseSourceSetting : IPlugin
    {
        public string SQLScript { get; set; }
        public string ConnectionString { get; set; }
        public NameValueCollection Parameters { get; set; }
    }

Step 4: Define a Converter

Create a new folder Converter and add a new C# filed called ReadDatabaseConfigurationConverter inheriting Sitecore.DataExchange.Converters.PipelineSteps.BasePipelineStepConverter.

Note: The attribute "SupportedIds" should contain the id of prevously created template. DXF validated all convertes and match the supported Id with the templateid before processing.

    [SupportedIds("{12C0F5DF-307C-4C4B-89B5-56F7A2F547AC}")]
    public class ReadDatabaseConfigurationConverter : BasePipelineStepConverter
    {
        public ReadDatabaseConfigurationConverter(IItemModelRepository repository) : base(repository)
        {
        }
        protected override void AddPlugins(ItemModel source, PipelineStep pipelineStep)
        {
            DatabaseSourceSetting settings = new Plugin.DatabaseSourceSetting();
            settings.ConnectionString = GetStringValue(source, "ConnectionString");
            settings.SQLScript = GetStringValue(source, "SQLScript");
            var parametrs = GetStringValue(source, "Parameters");
            settings.Parameters = StringUtil.GetNameValues(parametrs, '=', '&');
            pipelineStep.Plugins.Add(settings);
        }
    }

Step 5: Define a Processor

Create a new folder Processor and add a new C# file called ReadDatabaseConfigurationProcessor inheriting from Sitecore.DataExchange.Processors.PipelineSteps.
Processor should contain actual business logic to read/write the data. Here we are reading the plugin and excute the script to read data from database. Then we are reading the data and creating coresponding sitecore items.

I have created a BlogTemplate to hold imported data.

    public class ReadDatabaseConfigurationProcessor : BasePipelineStepProcessor
    {
        public override void Process(PipelineStep pipelineStep, PipelineContext pipelineContext)
        {
            Assert.ArgumentNotNull(pipelineStep, nameof(pipelineStep));
            Assert.ArgumentNotNull(pipelineContext, nameof(pipelineContext));

            var settings = pipelineStep.GetPlugin();
            Assert.IsNotNull(settings, $"{settings} is Null");

            var dbItems = SqlHelper.DbReader(settings.ConnectionString, settings.SQLScript, settings.Parameters);

            var templateId = "{EC13D0E0-16F4-47D6-8413-B0A13552AAE1}";
            var parentId = "{C5F6205D-2CA2-4ED9-83C7-A8696A71E3DC}"; //sitecore/content/home/blogs
            var db = Sitecore.Data.Database.GetDatabase("master");
            var parentItem = db.GetItem(parentId);
            using (new SecurityDisabler())
            {
                foreach (var item in dbItems)
                {
                    ItemManager.CreateItem(
                        ItemUtil.ProposeValidItemName(Convert.ToString(item["Title"]))
                        , parentItem, ID.Parse(templateId));
                }
            }
        }
    }

Step 6: Create Data Exchange Tenant

Now we need to create a new DXF tenant to hold DXF items to process the data. Navigate to /sitecore/system/Data Exchange and click on Empty Data Exchange Tenant. Provide the name "SimpleDemo".

Navigate to the newly created tenant (/sitecore/system/Data Exchange/SimpleDemo) , goto Content Tab and enable the Tenant.


Step 7: Create the Processor

Navigate to "/sitecore/system/Data Exchange/SimpleDemo/Pipelines" and Insert a Pipeline with name "SimpleDatabaseImport". Now right click on the Pipeline and we can find previously created processor template in the list.

Add a pipeline step of type "ReadDatabaseConfigurationProcessorStep" and name it "ReadDatabaseConfigurationProcessorStep". We can find the template fields "ConnectionString","SQLScript" and "Parameters". Along with these we can find two more fields ("Converter Type" and "Processor Type") inherited from BasePipelineStep template. Update these fields as shown below.


Step 8: Create PipelineBatch

The last step is to create the PipelineBatch. Navigate to "/sitecore/system/Data Exchange/SimpleDemo/Pipeline Batches" and Insert a Pipeline batch "Import Blogs". Select the previous pipeline for the "Pipelines" multilist field.

At this point the content tree for the tenant should look like,


Step 9: Run Processor

Now click on "Run Pipeline Batch" command from top ribbon. This will initiate the process and data will be imported.



In this post we looked into few basic concepts of DXF like Processor, Converter, Plugin,PipelineBatch and PipelineStep Context.

  • Processor: Contains the business logic
  • Converter: Reads sitecore configuration and creates plugin object
  • PlugIn: Simple PoCo class to hold the configuration
  • PipelineStepContext: Context shared between a onverter and its processor.
  • Pipeline Batch: Initiates the DXF process

Note:

Pipeline batch looks into all the linked processors and converters. If converter doesnot have valid allowedId attribute or have some exception, the processor will be skip

Download

Source Code: https://1drv.ms/u/s!Aj3yU0TIffgru06zO4rPfdDXej9g