Development of tools for working with binary data


If you wish to leave comments or ask questions, please log on or sign up.
This feature is implemented in 1C:Enterprise version 8.3.9.1818.

We have implemented a number of low-level tools for working with binary data. Now you can handle such tasks as:

  • Interaction with specialized devices using a binary protocol
  • File parsing and operations with files in different formats
  • Converting text data to binary data directly, for example, to send reports
  • Working with binary data in the memory

Previously, the platform had a number of methods available for working with files and the BinaryData type. But they did not provide any easy way to analyze the internal contents or modify it. All actions were performed for all data at the same time. The only operation that could be done with a portion of the data is splitting a file into parts and putting it back together.

Now the platform provides tools both for sequential operations with large amounts of binary data and random access to relatively small amounts of binary data entirely in RAM.

Main types of sequential operations with data

A specific example is the easiest way to demonstrate the purpose and interconnection of new objects. The example splits a WAV file into the same 1,000-byte parts.

The diagram shows the usage sequence for 1C:Enterprise script objects corresponding to the listing.

10d9336e7c33b8ccd2cecfdc55da5f96.png

Example: Split a WAV file into parts
// Split a WAV file into parts.
SourceStream       = FileStreams.OpenForRead(FileName);
DataReader         = New DataReader(SourceStream);
SourceHeaderBuffer = DataReader.ReadIntoBinaryDataBuffer(44);

// Decode the header.
DataSize    = SourceHeaderBuffer.ReadInt32(4);    
NumChannels = SourceHeaderBuffer.ReadInt16(22);   
SampleRate  = SourceHeaderBuffer.ReadInt32(24);   
ByteRate    = SourceHeaderBuffer.ReadInt32(28);   

// Based on information from the header, calculate the chunk size.
// In this example, for simplicity sake, let us take a fixed number,
// because it is not related to working with binary data.    

// Chunks are an array of values of ReadDataResult type.
ChunksArray = DataReader.SplitInPartsOf(1000);

ChunkNumber = 0;
For Each ChunkReadResult In ChunksArray Do
   ChunkHeaderBuffer = SourceHeaderBuffer.Copy();
   
   ChunkSize = ChunkReadResult.Size + 40;
   ChunkHeaderBuffer.WriteInt32(4, ChunkSize);
   
   ChunkNumber = ChunkNumber + 1;
   ChunkFileName = "C:\Chunks\" + String(ChunkNumber) + ".wav";
   
   TargetStream = FileStreams.OpenForWrite(ChunkFileName); 
   
   DataWriter = New DataWriter(TargetStream);
   DataWriter.WriteBinaryDataBuffer(ChunkHeaderBuffer);
   DataWriter.Write(ChunkReadResult);
   DataWriter.Close();
   
   TargetStream.Close();
   
EndDo;

SourceStream.Close();

The basis for working with binary data is a group of new types, which can be referred to as "streams". There are three types of them: Stream, FileStream, and MemoryStream. The example uses one of them, FileStream, but the rest mainly operate in the same way.

Streams are intended for sequential reading/writing of large amounts of binary data. Their advantage is that they allow you to work with arbitrary amounts of data streams. But they provide only basic features, such as reading from a stream, writing into a stream, and changing the current position (seek).

Streams can be created by file name or from the BinaryData object. In the example, the stream is created by file name (FileStreams.OpenForRead(FileName)) using one of the FileStreamsManager object methods. This is a new way to create streams, and we will tell you more about it in the future.

Then, in order to have more extensive opportunities for work, DataReader (New DataReader(SourceStream)) object is created from the file stream. This object allows you to read the individual bytes, characters, and numbers. You can use it to read a string taking into account the encoding, or read the data up until a certain marker known in advance. This object has its "opposite", DataWriter, which is created in a similar way, but writes data instead of reading. Because these objects read/write data from/to streams, they are also doing it sequentially, which allows you to work with streams of arbitrary size.

In the example, DataReader is used for two purposes. Firstly, in order to read and edit the file header, and secondly, in order to split the file into several parts.

The file header is the BinaryDataBuffer object. This is an important object, but we will talk about it a little later. The body of the file is divided into parts of equal size (DataReader.SplitInPartsOf(1000)) that are obtained in the form of ReadDataResult objects. This type has no special features, and mostly just stores the read data.

Next, for every such part of the file in the example, FileStream for writing is created, and on its basis, DataWriter is created. Writing the data writes a new header and a new body to the stream, and closes it. At the end of the cycle, the result is a set of several files.

Byte operations

In the example, the file header is read into BinaryDataBuffer object (ReadIntoBinaryDataBuffer (44)). The main difference of this object from the above is that it provides not sequential, but random access to data, and lets you modify the data at any position.

All data from this object are entirely located in the RAM. Therefore, on the one hand, it is designed for analyzing and editing of not very large amounts of binary data. But on the other hand, it provides convenient options for random reading and writing of bytes represented by numbers, for splitting the buffer into several parts and combining multiple buffers in one, as well as for getting a part of the buffer of the specified size.

Turning a picture by 90 degrees can be used as an example to illustrate capabilities of the binary data buffer.

Example: rotating an image
/ /For simplicity sake, let us drop the issue of formats and headers, 
// and assume that we already have the image pixels by lines.
// We are not going to discuss saving the image in this example.

// In this example, we assume that the size of the image is relatively 
// small and allows you to store the entire image in the memory.

// We get the image bytes in the BinaryDataBuffer object.
PictureBuffer = GetPictureBytes();

// We assume that we know the image dimensions in pixels.
Width = 200;
Height = 100;

// We create a new byte array for the rotated image.
// Let us assume that each pixel is represented by 4 bytes. Since in this example 
// we are not changing the pixels themselves, but only changing their order, 
// the internal pixel presentation is not important to us.
RotatedImageBuffer = New BinaryDataBuffer(Width * Height * 4);

// We rotate the original image by 90 degrees. For this, let’s go over
// all the pixels of the original image, calculate the new index of the pixel
// in the rotated image and write the pixel to the new byte array.
For WidthIndex = 0 To Width-1 Do
   For HeightIndex = 0 To Height - 1 Do
      PixelIndex    = 4 * (WidthIndex + HeightIndex * Width);
      NewPixelIndex = 4 * (HeightIndex + WidthIndex * Width);
      
      // We use GetSlice (), rather than Read() to avoid
      // extra copying, because the source buffer does not change.
      Pixel = PictureBuffer.GetSlice(PixelIndex, 4);
      RotatedImageBuffer.Write(NewPixelIndex, Pixel);
      
   EndDo;
   
EndDo;

Synchronous and asynchronous work

Streams (Stream, FileStream, and MemoryStream), DataReader, DataWriter, and ReadDataResult have pairs of synchronous and asynchronous methods. For example, Write () – BeginWrite (), Close () – BeginClose().

Asynchronous methods are needed in order to be able to work in the same way in the thin client, as well as in the web client (because browsers use an asynchronous model).

Synchronous methods are required for working in the server context (because the server uses only the synchronous model).

Stream manager and asynchronous constructors

In addition to the listed objects, we have also implemented FileStreamsManager that exists in a single instance and is available through the FileStreams property of the global context.

As we have previously mentioned, streams require both synchronous and asynchronous operation. We created asynchronous counterparts for the synchronous methods, however, this approach is not suitable for constructors. Object constructors can only run synchronously. Therefore, in order to enable asynchronous object creation, we implemented a manager with asynchronous methods that, essentially, use different ways to create a FileStream object.

And for the sake of aesthetics and symmetry, we have added similar synchronous methods. In order to be able to simplify an entry, instead of a constructor with four parameters:
New FileStream(<FileName>, <OpeningMode>, <AccessLevel>, <BufferSize>)
you can use a suitable method with one parameter:
OpenForAppend(<FileName>)

Example of reading and writing a multipart HTTP message

In order to illustrate the new features for working with binary data, we would like to refer to two more listings. They illustrate implementation of a task, which we were often asked about.

Example of reading a multipart HTTP message

// Procedure - Reading a multipart HTTP message
//
// Parameters:
//  Headers        - Map -  Request headers.
//  BinaryDataBody - BinaryData - Request body.
//
Procedure Read_MultipartHTTP(Headers, BinaryDataBody) Export
   
   Separator = GetMultipartMessageSeparator(Headers);
   
   Markers = New Array();
   Markers.Add("==" + Separator);
   Markers.Add("==" + Separator + Chars.LF);
   Markers.Add("==" + Separator + Chars.CR);
   Markers.Add("==" + Separator + Chars.CR + Chars.LF);
   Markers.Add("==" + Separator + "==");
   
   Address = Undefined;
   Text    = Undefined;
   Image1  = Undefined;
   Image2  = Undefined;
   
   DataReader = New DataReader(BinaryDataBody);
   
   Size = BinaryDataBody.Size();
   
   // Let us move to the beginning of the first part.
   DataReader.SkipTo(Markers);
   
   // Next, let us read all the parts in the cycle.
   While True Do
      ReadResultPart = DataReader.ReadTo(Markers);
      
      If Not ReadResultPart.MarkerFound Then
         
         // A malformed message.
         Break;
         
      EndIf;
      
      PartReader  = New DataReader(ReadResultPart.OpenStreamForRead());
      PartHeaders = ReadHeaders(PartReader);
      PartName    = GetMessageName(PartHeaders);
      
      If PartName = "DestAddress" Then
         Address = PartReader.ReadChars();
         
      ElsIf PartName = "MessageText" Then
         Text = PartReader.ReadChars();
         
      ElsIf PartName = "image1" Then
         Image1 = PartReader.Read().GetBinaryData();
         
      ElsIf PartName = "image2" Then
         Image2 = PartReader.Read().GetBinaryData();
         
      EndIf;
      
      If ReadResultPart.MarkerIndex = 4 Then
         
         // The last part is read.
         Break;
         
      EndIf;
      
   EndDo;
   
EndProcedure

// Function - Read headers
//
// Parameters:
//   PartReader    - DataReader - Request part.
// 
// Return value:
//   Map - Part headers.
//
Function ReadHeaders(PartReader)
   
   Headers = New Map();
   
   While True Do
      Str = PartReader.ReadLine();
      
      If Str = "" Then
         
         Break;
         
      EndIf;
      
      Parts      = StrSplit(Str, ":");
      HeaderName = TrimAll(Parts[0]);
      Value      = TrimAll(Parts[1]);
      
      Headers.Insert(HeaderName, Value);
      
   EndDo;
   
   Return Headers;
   
EndFunction

// Function - Get multipart message separator.
//
// Parameters:
//  Headers - Map - Request headers.
// 
// Return value:
//  String - Multipart message separator.
//
Function GetMultipartMessageSeparator(Headers)
   ContentType = Headers.Get("Content-Type");
   
   Properties = StrSplit(ContentType, ";", False);
   Boundary = Undefined;
   
   For Each Property In Properties Do
      Parts = StrSplit(Property, "=", False);
      PropertyName = TrimAll(Parts[0]);
      
      If PropertyName <> "boundary" Then
         
         Continue;
         
      EndIf;
      
      Boundary = TrimAll(Parts[1]);
      
      Break;
      
   EndDo;
      
   Return Boundary;
   
EndFunction

// Function - Get message name.
//
// Parameters:
//  Headers - Map - Part headers.
// 
// Return value:
//  String - Message name.
//
Function GetMessageName(Headers)

   Disposition = Headers.Get("Content-Disposition");
   
   Properties = StrSplit(Disposition, ";", False);
   Name = Undefined;
   
   For Each Property In Properties Do
      Parts = StrSplit(Property, "=", False);
      PropertyName = TrimAll(Parts[0]);
      
      If PropertyName <> "name" Then
         
         Continue;
         
      EndIf;
      
      Name = TrimAll(Parts[1]);
      
      Break;
      
   EndDo;
      
   Return Name;
   
EndFunction

Example of writing a multipart HTTP message


// Creating a test multipart HTTP message.
// 
// Return value:
//  Structure - {Headers, BinaryData}.
//
Function Write_MultipartHTTP() Export
   
   // Let us create embedded messages.
   BinaryDataAddress = CreateMessage_Text("DestAddress", "brutal-jonh@example.com");   
      
   BinaryDataText = CreateMessage_Text("MessageText", 
      "Hello, John!" + Chars.LF + 
      "Your pet lion you left at my place last week tore up my entire sofa into pieces." + Chars.LF +
      "Hurry up and take him away from here, please!" + Chars.LF +
      "Two pictures with the consequences are attached.");
      
   BinaryDataPenguins = CreateMessage_Image("image1", "penguins.jpg", PictureLib.Penguins);
   BinaryDataKoala    = CreateMessage_Image("image2", "koala.jpg",    PictureLib.Koala);
   
   // Let us create the main multipart message.
   Separator = "Asrf456BGe4h";
   
   Result = New Structure();
   Headers = New Map();
   
   Result.Insert("Headers", Headers);
   Headers.Insert("Content-Type", "multipart/form-data; boundary=" + Separator);
   
   StreamBody = New MemoryStream();
   DataWriter = New DataWriter(StreamBody);
   
   DataWriter.WriteLine("==" + Separator);
   DataWriter.Write(BinaryDataAddress);
   
   DataWriter.WriteLine("==" + Separator);
   DataWriter.Write(BinaryDataText);
   
   DataWriter.WriteLine("==" + Separator);
   DataWriter.Write(BinaryDataPenguins);


   DataWriter.WriteLine("==" + Separator);
   DataWriter.Write(BinaryDataKoala);
   
   DataWriter.WriteLine("==" + Separator + "==");

   DataWriter.Close();
   
   BinaryDataBody = StreamBody.CloseAndGetBinaryData();
   
   Result.Insert("Body", BinaryDataBody);
   
   Return Result;
   
EndFunction

// Returns an HTTP message as BinaryData.
//
// Parameters:
//   MessageName – String. 
//   Text        - String. 
// 
// Return value:
//   BinaryData – HTTP message.
//
Function CreateMessage_Text(MessageName, BinaryDataText)
   
   Stream = New MemoryStream();
   DataWriter = New DataWriter(Stream);
   
   // Headers.
   DataWriter.WriteLine("Content-Disposition: form-data; name=" + MessageName);
   DataWriter.WriteLine("");
   
   // Body.
   DataWriter.WriteLine(BinaryDataText);
   
   DataWriter.Close();
   
   Return Stream.CloseAndGetBinaryData();
   
EndFunction

// Returns an HTTP message as BinaryData.
//
// Parameters:
//   MessageName - String
//   FileName    - String  
//   Image       - Picture 
// 
// Return value:
//   BinaryData - HTTP message.
//
Function CreateMessage_Image(MessageName, FileName, Image)
   
   Stream = New MemoryStream();
   DataWriter = New DataWriter(Stream);
   
   // Headers.
   DataWriter.WriteLine("Content-Disposition: form-data; name=" + MessageName + "; filename=" + FileName);
   DataWriter.WriteLine("Content-Type: image/jpeg");
   DataWriter.WriteLine("");

   // Body.
   DataWriter.Write(Image.GetBinaryData());
   
   DataWriter.Close();
   
   Return Stream.CloseAndGetBinaryData();
   
EndFunction