Catalog writing speed issues


If you wish to leave comments or ask questions, please log on or sign up.

The 1C:Enterprise developers forum

#1
People who like this:0Yes/0No
Interested
Rating: 33
Joined: Oct 27, 2011
Company: Abaco Soluciones S.A.

We have crated a really small configuration from scratch (just 2 catalogs). These 2 catalogs are populated from text files by searching matches and updating them if no match was found.
I have noticed (with the performance profiler) significant write speed difference with 2 catalogs. Writing to the second catalog took 97,8% of all the time of the program. 2nd catalog have almost the same number of attributes (around 10) and all of them are of simple types, like string, date or number.

What could affect writing performance by that much considering that:
1. catalogs does not have complex type attributes
2. they are not hierarchical
3. there are no links from or to the catalogs
4. there are more than 200K+ records in each of them
5. infobase was put on a RAM-drive to eliminate any disk subsystem problems
6. Infobase was relatively small (200M in size)

Could variable length strings affect performance?
Is there any maximum attributes summary length?
If there is more than one index could it degrade write performance by much?
Something else, that I have forgotten to mention?
Could it be platform version related problem? I was testing on 8.3.12.1586

Please, help me with any ideas. It is so painfully slow now, that is going to take some days to populate Infobase with data.

Also I have noticed that while testing typically 1c app consumes more and more memory at the begging of ejection, then reaches around 450-600Mb of RAM and starts decreasing use of memory, staying at around 90Mb of RAM.

 
#2
People who like this:1Yes/0No
Active user
Rating: 4
Joined: Mar 10, 2017
Company: Rufinor

Dear Alexey,

Please share with us code listing which you use for upload.

Some recommendation that you can use:

1. Update 1C platform version to 8.3.13;
2. Try to avoid text fields with unlimited length;
3. Do upload in exclusive mode;
4. Try to use transaction during upload;
5. Switch off regisration in the exchange plan during save;
5. Use SQL version of 1C platform.

Kind regards,
Alex

 
#3
People who like this:0Yes/0No
Interested
Rating: 33
Joined: Oct 27, 2011
Company: Abaco Soluciones S.A.

Quote
Alex Ogourok wrote:
1. Update 1C platform version to 8.3.13;
Well, it is a universal suggestion, I still think of 8.3.13 as beta, but we could try.
Quote
2. Try to avoid text fields with unlimited length;
No unlimited text fields were used, also we have tried to used only fixed length fields but no change in write time
Quote
3. Do upload in exclusive mode;
Yes, active
Quote
4. Try to use transaction during upload;
What is your suggestion? Count to 1000 and commit?
Quote
5. Switch off registration in the exchange plan during save;
Switched off, also switched off full text search, data history, no additional indexes
Quote
5. Use SQL version of 1C platform.
I thought about this one, but I can not explain why writing to one catalog only takes 1% of execution time and another one - 97%. I measured less than 20 writes per second on 3Ghz CPU and RAM-drive for the Infobase.

 
#4
People who like this:0Yes/0No
Interested
Rating: 33
Joined: Oct 27, 2011
Company: Abaco Soluciones S.A.

Quote
Alex Ogourok wrote:
Please share with us code listing which you use for upload.
I don't have code with me right now, but it is basically like this:


Code
For Each Row In file Do
  Fields = GetFieldsFromRow(row);
  If Catalog1.Select(Fields.Code).IsEmpty() Then
    Obj1 = Catalog1.GetObject();
    ...
    Obj1.Write();
    
    Obj2 = Catalog2.GetObject();
    Obj2.Write();
  EndIf;
EndDo;

 
#5
People who like this:0Yes/0No
Active user
Rating: 4
Joined: Mar 10, 2017
Company: Rufinor

Dear Alexey,

It is not clear fr om this code example wh ere do you take reference to Catalog2.

It would be good if you could send full code exerpt.

You may try to use transactions by 1000 and commit.

Also if these are 2 catalogs that are not linked to each other why not to try to save Catalog2 in a separate cycle?

With SQL you could use profiler to understand the exact reason for slow performance.

Kind regards,
Alex

 
Subscribe