[CLOSED] Performance bottlenecks clues

Page 1 of 3 123 LastLast
  1. #1

    [CLOSED] Performance bottlenecks clues

    Hello,

    We've been running into consistent performance issues while load testing a new Ext.Net v1.7 application on the web server. SQL Server traces indicate that there's no database related bottleneck or it's negligible at this point. The web server performance, however, is pretty weak and it degrades rapidly when loading over 10 concurrent Vusers using HP LoadRunner. Our development environment is created with Hyper-V and the virtual web server specs are: Windows Server 2008 R2, 8 GB of RAM and 4 logical cores running at 2 GHz. SAN storage is implemented on the host as RAID10 with the underlying hard disks rotating @15K rpm. The web server throughput (bytes per second received by all the Vusers) stalls after running 10 Vusers at a very low amount just below 700 KB and after that drops and everything becomes erratic: hits per second, response time, etc. The test results indicate that the web server doesn't scale up, I get that. One telling story is high CPU utilization (70-90%) and Processor Queue length (number of threads not able to execute at a time) of over 2 on average and spiking.

    Now, based on the info above, what do you think the culprit or culprits of poor performance might be? Is our hardware adequate, in your opinion? Could it be a matter of poor application design only? Can it be attributable to the Ext.Net/Sencha technologies implementation details?

    I would be attaching the load testing report generated by LoadRunner for your reference if the website would allow me to do that. Your advice would be highly appreciated. Please let me know what other information may be useful to consider.
    Last edited by Daniil; Jun 15, 2015 at 2:45 PM. Reason: [CLOSED]
  2. #2
    What happens if you load test another web application with no Ext.Net present?
  3. #3
    I've load tested a little Telerik sample application available at http://demos.telerik.com/crmsample/ for download. The web server handled the load well although I don't think such a comparison would be accurate. Granted, the sample application is pretty simplistic compared to the real one I've been having problems with.
  4. #4
    Thanks for testing that. I am trying to understand if it is a server/network or application issue.

    Personally, I do have a ext.net 1.7, database driven (sqlserver) web application hosted online -on a shared (!) web server (iis 7.5) and hit by more than 10 users on a daily basis. As far as I can remember, I hadn't done anything special (e.g. to configure ext.net or other parts of the application) but was careful about optimizing ext.net, especially the number of controls per page and the amount of transferred data in every (ajax) call (even though you already said this is probably not your problem). Also, viewstate is used (it is a somewhat old implementation).

    Having said that, the 1.7 components are behaving extremely well. Now, it is important to note that the web server is not setup and configured by me but by the data center company. I clearly remember that the same application was awfully slow when hosted on another server, which means that the web server setup and the networking infrastructure made the difference in my case.

    Bottom line, I shared my personal experience to underline the fact that the 1.7 version behaves as expected unless used with no care at all :)
  5. #5
    Thanks for sharing your experience, Dimitris! Have you actually load tested your application in dev/qa/prod environment with concurrent Vusers? If so, what were your observations of the web and/or application server performance counters and CPU activity %? Could you please share the specs of your dev and prod web servers?
    How many concurrent requests does your application get in production?
    By suggesting that you were careful about optimizing Ext.Net, could you mention the steps you took outside of keeping the Ext.Net controls on the page to the minimum? Does your app make use of any TabPanel controls?

    Edit In:
    I was unable to attach the LoadRunner report that I emailed to Daniil the other day. Let me know if you need a copy of it.
    Last edited by vadym.f; May 11, 2015 at 5:57 PM.
  6. #6
    By suggesting that you were careful about optimizing Ext.Net, could you mention the steps you took outside of keeping the Ext.Net controls on the page to the minimum? Does your app make use of any TabPanel controls?
    Actually, yes it does. For example, in a 9-tab TabPanel, each tab contains various child controls all loading data. So, I took care not to load everything at once but rather use the Activate DirectEvent (GridPanelTests is the tab itself):

    <DirectEvents>
        <Activate OnEvent="GridPanelTests_Activate" />
    </DirectEvents>
    protected void GridPanelTests_Activate(object sender, DirectEventArgs e)
    {
        TestService service = new TestService();
        this.StoreTests.DataSource = service.GetTests();
        this.StoreTests.DataBind();
    }
    One of the tabs is a FormPanel with 60 fields. Due to its size, the form is dynamically created on the server side.

    Last but not least, as far as Ext.NET is concerned, I applied server side data paging wherever possible (in GridPanels, etc).

    Application design-wise, I've used layers (Repository, Service, Presentation, Infrastructure etc). This is a big topic on its own but I cannot help but emphasize on the fact that there is good performance gain in separating things.

    Could you please share the specs of your dev and prod web servers?
    - Windows Server 2008 R2
    - IIS 7.5
    - 16 GB RAM
    - Intel Xeon 1270 Quad Core

    How many concurrent requests does your application get in production?
    20, more or less.

    Edit In:
    I was unable to attach the LoadRunner report that I emailed to Daniil the other day. Let me know if you need a copy of it.
    It can be communicated to me directly from Daniil.
    Last edited by Dimitris; May 14, 2015 at 7:42 AM.
  7. #7
    Hello, I would like to share some thoughts about this issue but, as there lacks a lot of information (as your application might be a complex one), all I can do is explore guesses.

    - One: Memory usage
    You said you have 8GB of RAM. Maybe your SQL server runs in the same machine and is allocating or limited to almost or more than it has. SQL Server tends to try and keep pages open for faster data access.
    This would then leave less than enough memory for the IIS process, system begins to need too much memory-page-picking from the hard disk. This will dramatically increase load and context switch times thus leaving processes extremely slow.
    This does not really explain the fact that you get 70-90% CPU usage though.

    - Two: Load balancing
    Depending on the application's complexity, you could have built a greatly-performing data access to the database (it could be fast as it keeps pages open), but maybe you leave too much work on the application level. This usually is not a problem, as CPU (bus transfers) goes thousands of times faster than data transfers. But maybe the calculations overwhelm data transfer due to the amount of data, required memory to store all worked data. If possible, trimming the database queries (if the case, of course) to the minimum and clearing big arrays/lists after they are no longer necessary might do the trick.

    - Three: More on load balancing
    Otherwise, maybe there's too much, maybe unnecessary postback in every request. Every postback would require reallocating resources, preparing data so, even if an individual sql request is fast but several postbacks are made upon page load, you might end up re-reading the same data, re-interpreting/calculating it, so wasting resources you could otherwise parse just once or twice per VUser.

    It is not much to be expected the problem to be the actual extJS controls in the page. Well, if you have many controls, of course, it would increase amount of postback for their loading and events, but the control loading itself weighs on client-side only.
    I don't know whether HP LoadRunner actually emulates the real page loading, thus sending postbacks and all, at all. HP LoadRunner if run in the same machine (I doubt that's the case) could be taking the resources needed to stall the machine should it really interpret and build each page's request.

    It could prove useful if you could empirically do some tests if you have better hardware available -- or the option to build a huger/weaker virtual machine on your physical host. Changing one or the other variable and retaining the others, and watching what gives more performance difference. You can also compare running LoadRunner against the page when it points to a full-fledged database, and also to a dummy bare database i.e. change server available data to process, in order to evaluate any possible changes.

    Also if you could come up with a sample reproducing the performance hit on your side we would actually be able to run and either state the reasons it performs poorly after that number of users or suggest reviews to improve the code. Of course, we are specialized in Ext.NET and depending on the technology you're using, it would be beyond the scope we are able to help you with, but we'll try our best.

    Hope something among all that helps in figuring out the issues you have. I am somewhat more inclined to believe the load balancing is the case -- but that's a guess among guesses.
    Fabrício Murta
    Developer & Support Expert
  8. #8
    Thanks for your pointers, Dimitris and Fabricio! We're generally following the common sense best practices, e.g. the SQL Server is on a separate VM and we use a layered web application architecture relying on well defined data access, service, business and presentation components.
    I believe the memory is a non-issue because its utilization stays stable and well below the amount of RAM available on the web server at the time of testing. Something like 2.5 GB out of 8 GB gets used up and it doesn't spike.
    I've tried to carefully craft and examine the various web requests that the LoadRunner has produced when my scenarios got recorded. Yes, it's taken a couple of tricks to make them AJAX compliant so that full post backs don't fire erroneously. Also, think time markers should be used appropriately to emulate the real users experiences. I'm reasonably confident that those calls are fairly close to the real ones although there's always room for improvement. :)
    I agree that it would be great to load test the application against a different infrastructure but we don't seem to have much leeway there. In our Hyper-V environment, VM specs allow up to 64 GB of RAM with maximum of 4 logical cores. The deployment environment for the web is a stand alone VM server with no load balancer. Could this be an issue? I'd be curious what you deployment web/application server configuration looks like.
    Last edited by vadym.f; May 13, 2015 at 12:55 PM.
  9. #9
    I believe your hardware setup is quite decent and well designed.

    But something must be the cause -- given your scenario really reproduces close to actual user access -- I believe just 10 connections at once is too low a volume for bottlenecking and getting random error messages.

    You probably already checked this but
    - Do you have governance over the whole physical host? No chance some VMs are well overloaded thus hitting the overall performance of the remaining VMs? That's a common issue on virtual environments, specially VPS hostings. Specially disk access is compromised this way.
    - No chance the total VMs exceed the 64GB limit -- or 60GB as probably the host OS should need some memory for it to actually work?

    At least I can't see a reason ext.net would demand -- by itself -- so much resources just to build up the dynamic javascript ExtJS code for the page. Did you test how high on usage it gets when you try just one connection repeating the load over and over (not in parallel)?
    Fabrício Murta
    Developer & Support Expert
  10. #10
    Just an idea, could it be a problem with HyperV and the NICs you are using?
Page 1 of 3 123 LastLast

Similar Threads

  1. [CLOSED] EXT .net performance
    By NECSL in forum 2.x Legacy Premium Help
    Replies: 15
    Last Post: May 06, 2014, 7:04 AM
  2. [CLOSED] Low performance
    By mohan.bizbites in forum 2.x Legacy Premium Help
    Replies: 3
    Last Post: Dec 11, 2013, 9:32 PM
  3. Performance
    By Puia in forum 1.x Help
    Replies: 2
    Last Post: Jul 28, 2009, 10:51 AM
  4. GWT EXT performance
    By jeybonnet in forum Open Discussions
    Replies: 6
    Last Post: Mar 25, 2009, 8:39 AM
  5. AjaxEvent Performance
    By jchau in forum 1.x Help
    Replies: 2
    Last Post: Oct 13, 2008, 1:02 PM

Posting Permissions