Spike exploration: sftp, zip, config and Windows Service

I did a spike exploration to find out what is involved in creating a Windows Service with event logging. I also wanted to find out what is involved in working with sftp, zip and configuration with C#.  The code is available on github: https://github.com/k0emt/sftp-windows-service-spike

sftp

In order to test the sftp code I set up a Linux VM in the Azure cloud environment and created a user account on it.

I use the ssh component from http://sshnet.codeplex.com/ for sftp.  It is very straight forward to use this component.  The key for sftp is to use the SftpClient class.  Upload a file with the UploadFile method.  The sample code is in the ConsoleSftp projects Program.cs: https://github.com/k0emt/sftp-windows-service-spike/blob/clean/ConsoleSftp/Program.cs

zip

Use System.IO.Compression for creating/expanding zip files.

To use the ZipFile class, you must reference the System.IO.Compression.FileSystem assembly in your project.

http://msdn.microsoft.com/en-us/library/system.io.compression.zipfile(v=vs.110).aspx

I use ZipFile.CreateFromDirectory in the code above to zip an entire directory before sending it over sftp.

configuration

ConfigurationManager is the class that is used for working with application settings.

http://msdn.microsoft.com/en-us/library/system.configuration.configurationmanager.appsettings(v=vs.110).aspx

I used it in the service code to pull in a setting called user: https://github.com/k0emt/sftp-windows-service-spike/blob/clean/TransferService/TransferService.cs

In order for this to come together, you must create an app.config file in your project and change the property setting in Visual Studio to tell it to deploy when the project is built. https://github.com/k0emt/sftp-windows-service-spike/blob/clean/TransferService/App.config

Windows service with event logging

Create a Windows Service by choosing the Windows Service project template.

The Windows Service code must be installed and uninstalled.  It is possible to create an installer/uninstaller.  I opted to use the command line for this spike.

http://msdn.microsoft.com/en-us/library/y817hyb6(v=vs.110).aspx

http://msdn.microsoft.com/en-us/library/zt39148a(v=vs.110).aspx

I went here to learn about Service Methods – that is the kind of events to which the service can respond.  You can use the Visual Studio designer to enable the events you want handle.  Then you’ll need to code the events.  For this spike code I have it make an entry in the event log for each event type that is triggered.

http://msdn.microsoft.com/en-us/library/System.ServiceProcess.ServiceBase_methods(v=vs.110).aspx

Installing and Uninstalling the service

http://msdn.microsoft.com/en-us/library/sd8zc8ha(v=vs.110).aspx

Install with installutil.exe TransferService.exe

Go to Windows Services to see that the installed service is running.  See, the Spike Transfer Service below.

Uninstalling with installutil.exe /u TransferService.exe

Event Logging

For event logging a new EventLog is created.  An event source must be created and the source has to be wired up to the EventLog. 

Once this is done, the WriteEntry method can be used for writing to the event log.

You can go to the system event log panel to see where the events have been logged.

Summary

Overall, I was pleasantly surprised with the ease of accomplishing the spikes target tasks.  There was also decent documentation and examples available.

Task Parallel Computing

There are two primary parallel computing techniques: task parallel and data parallel.  Some key aspects to consider if tasks can be run in parallel:

• Are there dependencies between tasks?
• Does the result of the operations change if you change the execution order?
• Will there be contention for data or some other resource?
• Will we get a return on investment for the overhead of configuring and running in parallel?
• Do we have a significant number of tasks to run?  We should have more tasks to run than we have cores.

Batch processing

I created a BatchProcessor solution with a console application called BPRun.  This project runs a setup portion, the parallel tasks and then the teardown jobs.   The tasks to run are specified in corresponding configuration files.  I use code similar to this in production for batch processing ~20 DOS .bat jobs.  Being able to have jobs run in a parallel fashion has resulted in a significant time savings.

Setup and Teardown

The setup and teardown portions of the application are run in a serial fashion.  The items in the configuration files are executed one after another.  Comments may be placed in the file with // prefixing.

Task Parallel

The task parallel section executes the items in the corresponding configuration file.  As a task finishes the scheduler picks up and runs the next job on that process.  The code that does this is in a corresponding Processor class.

Tasks with dependencies

What if you have a task that has a dependency?  You can chain them.  That is, have the first process or task call the second or dependent task upon completion.  For instance, you may have a job that does some data prep and then chain it to call a task that does analysis.

Keeping Busy

On *nix systems we have the sleep command which we can use to pause for the given # of seconds.  However, this doesn’t place any load on the machine.  I wanted the machine to have a load placed on it by the tasks so that we could see that it is really taxing the machine.  To that end I created a small console application called Busy.  This takes the number of seconds to be busy as a parameter.

Download

You can get the source code for the Batch Processor and the Busy application here: https://github.com/k0emt/BatchProcessor

Video

See a video tour of the code for Busy and the Batch Processor along with demos of the applications here: