Syslog service

This work was done based on requirements on the task: http://www.freelancer.com/projects/Delphi-Software-Architecture/Windows-Log-Syslog-Service.html

The project was done during 1 month. This is twice longer than the bid I did, but there was a lot of low-level WinAPI details. This is more low-level windows-based work than the Delphi native code. Therefore the project was remade 2 times. I refined their classes several time to come for better, reliable, clear and to be more understood results.

Ok. Here I want to describe this work in details. How it's working, correct, or, not.

Let's go. Main parts:

• The Eventlog monitoring block
• The Files monitor
• The communication part
• The debug logging messages
• The interface part
• The service part
• InnoSetup installer

This block is working based on universal 'eventlog' unit, described here

Main features:

• Check the windows eventlog facility to take new messages from it
• Select every message based on selection criterias
• Parse the message and convert it into needed-to-show format

Ok. This block creates the list of messages in the internal proprietary format. These message ready to be sending out inside “Transaction”.

How it's working:

Every known branch (“Application”, “Sytem”, and “Security”) has the variable “last_seen” – TDateTime, this is the date/time when this event was added into the windows eventlog facility. Everytime the block read all the messages from the log, and take only those that have the generated time >= “last_seen” time.

These messages are added into the list, and the “last_seen” variable is updated by the value of last generated message (not “now” because when the program is working newer messages could be generated).

Also when the program takes messages – it check the selected messages parameters based on http://msdn.microsoft.com/en-us/library/windows/desktop/aa363648(v=vs.85).aspx (or any other)

After that the message is converted into the simple string line on the next pattern:

'record number: ' + IntToStr(Number) + #9 + 'source: ' + SourceName + #9 + 'message: ' + event_message + #9 + 'event id: ' + IntToStr(Code) +#9 + 'event type: '+IntToStr(EventType);

Where

• Number – is the message number from the 'eventlog' facility
• Source – one of values 'Application', 'Security', 'System'
• Message – this is the human-friendly probably long string described what happen.
  The parser replace these characters – #13 (CR), #10 (EOLN), #27 (EOF), #9 (TAB), #26 by spaces for better handling and avoid any conflicts
• Event id – zero if this is non-DLL message (free message), or the evet_id inside the DLL is bound with the message
• Event type – one of known event types: 'EVENTLOG_SUCCESS' (0), 'EVENTLOG_ERROR_TYPE' (1), 'EVENTLOG_WARNING_TYPE' (2), 'EVENTLOG_INFORMATION_TYPE' (4), 'EVENTLOG_AUDIT_SUCCESS' (8), 'EVENTLOG_AUDIT_FAILURE' (10), also the code “16”

Example of the message placed into the output text log file:

record number: 6604	source: Security	message: The description for Event ID ( 593 ) in Source ( Security ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. The following information is part of the event: 3820C:\WINDOWS\system32\arp.exesysadminSERVER(0x0,0x12995)	 event id: 593	 event type: 8

This part do the monitoring of the files based on the list of files added with help of the The interface part

We apply the next common parameters for all the files together:

• Skip all lines on startup
• Check interval

How it's working:

1. On the program start – read 'last_size' variable from the registry
2. If the 'Skip all lines on startup' is ON – set 'last_size' into the current size in bytes of every files
3. Every time after the program read messages from the eventlog facility – it go to the list of monitored files, and check the size of the file – if it greater than 'last_size'
4. It open the file and read all bytes after the 'last_size' position into the binary 'buffer'
5. Then it split 'buffer' into usual lines by split delimiter = [#10, #13]
6. Store 'new_last_size' for every monitored file

On successful Transaction

1. Assign all 'new_last_size' into the 'last_size' variables for every monitored file
2. Save new file sizes the into the registry

UDP / TCP communications with the syslog service. Also here's email sending block on TCP errors.

Used components – Indy TIdSyslog for UDP syslog communication, and TIdTCPClient – for TCP syslog communication. TIdSMTP – for simple email sending procedure based on simple communication with an external SMTP-server.

Since this part was not described in the initial technical requirements – it was made with minimum features using the well-known indy project.

This is the simplest variant to communicate with Syslog service, and not reliable as well – messages could be omitted due protocol realization.

Better way. The connection will be established and be more reliable. Also transaction will be not completed when the TCP connection will fail or broken, therefore syslog service will wait until the connection will success or it'll send out mail message about failed connection and stop the monitoring.

Is working when the 'debug.txt' file is exists in the directory with the program. Needed for testing / debugging purposes.

File 'debug.txt' – being exists in the directory with the service will enable the service to write out the file 'yeti_debug#####.txt' in the append mode where it'll log every points in/out into procedures to have a log. Where ##### is a timestamp when the program was started.

There is a set of 3 user friendly forms based on the additional customer requirements.

Also the application has an icon in the windows tray and simple popup menu with next items:

• Start the service
• Stop the service
• About
• Setting
• ——-
• Exit

The interface part read the status of the service part (the running status is shown in the interface part header). It can also start or stop it.

Hidden service software

The *.iss source that compile the program into self-installed software.

In block descriptions we defined some things, here's their descriptions.

The process of sending out of taken data. On success – it update the variables and the registry. Otherwise – send out email (for TCP 2 syslog monitoring)

When the service read messages from the eventlogNT, and lines from monitored files – it should send them out to syslog. We can define this process as a “transaction”. If transaction fail – the program will not update the Last_events[k] and lastsize[k] variables.

After 'try_cnt' fails – it sends out an email about this on defined email if any.

Successful – the program update the values:

• Last_events[k] (k=0..2) – the datetime from where the program should take new messages from the eventlogNT; this parameter it keeps in the registry in the 'General' folder of the service application – 3 parameters “Last_events_0”, “Last_events_1”, “Last_events_2” format binary (datetime/double, 8 bytes)
• Files[k].Last_Size (k=0..monitored_files_cnt-1] – the last size of the monitored file. We can think about if the file was appended – we read all new lines and send them into the syslog. If the file become smaller then their last_size – the program load all lines from this file.

All messages from the eventlogNT are read in the reverse order (from latest back to oldest), result – the TStringList variable. On successful transaction – it takes the DateTime “Generated” value from the latest message and place it into the “Last_events” variable.

Ok, as I mentioned before, every monitored file has the last size – the length of the file in bytes. The program is thinking about the file size is grows up. Every time the program open the file, seek into the position of previous size, and read all new bytes into the “buffer” in the binary mode. The it split this buffer into TStringList variable by #10 and #13 (usual windows line breaks).

In the case if last_size less then the current size – the program thinks about the monitored file was rewritten from scratch and read all lines from their beginning.

The program keeps all the options and work variables in the windows registry. Both Service block and the Interface block use the same registry branch.

Here's the description of every parameter in registry.

Contain two classes that read data from registry into the properties.

Also it save changed variable back into registry.

This unit is used by both parts – by the interface and the service.

The interface form that consist of 3 pages based on initial 3 gifs.

The service unit

Used as syslog facility, email about bugs, etc. For now – it write messages into the windows/temp directory, file “test_output.txt”. (note: in Win 2008 R2 – a 'temp' directory is under /users/…/application data/)

Interface part contains one form with 3 tabs and the popup menu that used to start/stop service, show About page and exit.

All changes are saved into the registry.

Service part is working in hidden (system) mode. It uses the settings from registry. If they were not added – it uses the settings by default.

By default: ipv4; UDP:127.0.0.1:1468; all checkboxes are checked; the program use the testing smtp-service for email connection.

In the case the TCP is not working, and the service send an email about it. Then the service stop their work because of no destination with an appropriate message into eventlog. To turn it ON – change the settings and start the service.

Create the file 'debug.txt' (any size) and place it into the service directory (by default “C:\Program Files\Yetidi syslog”). Now during the service work – it'll write the temporary file “test_output.txt” into “C:\Windows\Temp” (or another place based on system settings). Also every bug will be written into the file “log_file.txt” in the windows temporary directory.