One of the most important issues (VIB) was severe performance insufficiency reported by everyone who tried to use the system during its acceptance testing. Literally every user action (like opening a form or refreshing a list) used to take a few seconds while the more massive operations (like posting a heavy document) could take up to 30 seconds.
This kind of issues usually points out that there is a single place (called “bottleneck”) impacting the whole system and causing all user operations’ slowdowns. The bottleneck can be encountered at any system level, namely:
- 1C application code which might contain suboptimal fragments that are used everywhere and cause all the symptoms. This was a fairly improbable scenario in our case because suboptimal code usually impacts some of the operations whereas all others work fine.
- 1C:Enterprise platform that might have a bug influencing the general system performance. This would be one more long shot in the case because the system was using a platform version known for its stable and fast operation for a few months by the time.
- DBMS performance problems and bottlenecks hardly needed to be taken into consideration, because the users experienced slowdowns everywhere – even on the operations not sending any DBMS requests (like lists scrolling)
- Operating system and middleware (drivers, virtual resources and so on) seemed to be a good candidate because this level performance problem could cause the symptoms we observed. Moreover, the system was running on the virtual servers that made this layer more vulnerable to errors. On the other hand, there were slowdowns on client-side operations (like list scrolling) but the server issues could also affect them because the system was using a thick client whose interaction with server-side could occur at any moment regardless of the operation type.
- Hardware bottleneck could also cause all the symptoms but this was hardly possible because of the low workload during this test period and the fact, that slowdowns were easily reproducible in single-user mode.
That allowed us to exclude the hardware from further investigation and left basically one candidate – operating system/middleware.
To confirm the bottleneck location we made an exact copy of the system except for the hardware and OS/middleware, i.e. we deployed the same system on another computer. Tests showed much better performance even though the hardware we used had significantly lower performance than the client’s servers. It meant the OS/middleware is a significant part of the problem but there still was a chance that the problem occurs only with 1C:Enterprise systems, i.e. there is something in the interaction between 1C and OS/middleware that causes the problem.
To exclude this possibility we ran a few different benchmark tests on the client’s hardware and compare their results with the same test performed on other computers. All tests showed that client-server performance is much lower in comparison with other computers even though they have less powerful hardware.
To illustrate the difference we included in this post file “SQLIO .xlsx” showing the difference in disk array performance of client’s hardware and two other servers, having much less powerful disks. To get this data we used SQLIO tool by Microsoft as it is shown in this .
These results and conclusions were presented to the client, who decided to postpone further investigation of this OS/middleware performance issues and move the live system into other hardware. Immediately after that, the performance went up to the level that met users' requirements.
