PDA

View Full Version : [CLOSED] Memory fragmentation problems with Transormer.NET



dlefevre
Apr 29, 2016, 11:35 AM
Hi,

I issue a performance pb with an application due to large object heap consuming memory in the IIS process

The analyse of a memory dump of the w3wp process show that Transformer.NET.ItemTag is storing most of the LOH html pages in form of strings.

So my questions are :
Are you aware on such an issue ?
Is there a way to avoid this ?


Here is a short resume of the analysys

As seen in the summary table below, free objects are taking up almost 1 GB and System.String object are taking 652 Mb.

00007ffaf7e8fc38 31020 4026528 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[Avanteam.Formulars.IApsOption, Avanteam.Formulars]][]
00007ffaf67a3c20 33754 4050480 System.Web.UI.LiteralControl
00007ffaf7605418 173251 5544032 System.Web.UI.StateBag
00007ffaf67a3f98 103033 5769848 System.Web.UI.ControlCollection
00007ffaf4cacf78 124757 5988336 System.Text.StringBuilder
00007ffaf772f720 27078 6065472 System.Data.SqlClient._SqlMetaData
00007ffaf6527ab0 13098 6706176 System.Data.DataTable
00007ffaf66f7620 57201 6864120 System.Data.SimpleType
00007ffaf7d89ea0 21229 7623408 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.Object, mscorlib]][]
00007ffaf4f96838 211520 8460800 System.Collections.Specialized.HybridDictionary
00007ffaf4f96a98 178691 8577168 System.Collections.Specialized.ListDictionary
00007ffaf4e9e0e8 217760 8710400 System.Collections.ArrayList
00007ffaf85d5a58 225196 9007840 Avanteam.Kernel.Schema.APSColumn
00007ffaf5099978 201501 9672048 System.Collections.ArrayList+ArrayListEnumeratorSi mple
00007ffaf76059b8 326320 10442240 System.Web.UI.StateItem
00007ffaf85d29c8 208441 13340224 Avanteam.Kernel.Schema.APSField
00007ffaf7a52b60 354541 14181640 System.Web.UI.ControlCollection+ControlCollectionE numerator
00007ffaf7752258 74408 14286336 System.Data.SqlClient.SqlParameter
00007ffaf5044d18 376816 15072640 System.Collections.Specialized.ListDictionary+Dict ionaryNode
00007ffaf66f2238 27283 15127920 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Data.DataColumn, System.Data]][]
00007ffaf512de08 15935 16543888 System.Double[]
00007ffaf678e710 215139 17211120 System.Data.Common.StringStorage
00007ffaf66cdf70 193828 18607488 System.Data.DataRow
00007ffaf4e58b80 62970 21432912 System.Collections.Hashtable+bucket[]
00007ffaf851b7b8 209310 21768240 Avanteam.Documents.DataSets.DSDocuments+FieldDefin itionsRow
00007ffaf52f4ac8 13650 22535376 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.String, mscorlib]][]
00007ffaf4f77cf8 30079 31762024 System.DateTime[]
00007ffaf4e57990 432588 40824924 System.Int32[]
00007ffaf66f44f8 32066 54797104 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
00007ffaf669c7d8 326615 70548840 System.Data.DataColumn
00007ffaf6789070 49280 72912896 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
00007ffaf54ff2e8 2504955 80158560 System.ResolveEventArgs
00007ffaf4cf1538 58152 123882204 System.Byte[]
00007ffaf4cac5c8 422155 132354080 System.Char[]
00007ffaf4c5be30 1351930 409909384 System.Object[]
00007ffaf4ca4d18 3909705 652648888 System.String
0000005934696110 23551 1056192938 Free


When I dumped out System.String objects in LOH, I see these are whole page html stored as a string object.

<!DOCTYPE html>
<html>
<head>
<link type="text/css" rel="stylesheet" href="/eCS/extjs/resources/ext_theme_gray/ext-theme-gray-all-embedded-css/ext.axd?v=31210" id="ext-theme" />
<link type="text/css" rel="stylesheet" href="/eCS/extnet/resources/css/extnet-all-embedded-css/ext.axd?v=31210" id="extnet-styles" />
<script type="text/javascript" src="/eCS/extjs/ext-all-js/ext.axd?v=31210"></script>
<script type="text/javascript" src="/eCS/extnet/extnet-all-js/ext.axd?v=31210"></script>
<script type="text/javascript" src="/eCS/extnet/locale/ext-lang-fr-js/ext.axd?v=31210"></script>
<link rel="stylesheet" type="text/css" href="/eCS/DXR.axd?r=1_8-yQjza" /><link rel="stylesheet" type="text/css" href="/eCS/DXR.axd?r=1_9-yQjza" /><link rel="stylesheet" type="text/css" href="/eCS/DXR.axd?r=1_1-yQjza" /><title>

</title><link href="../../Styles/css/document.css" type="text/css" rel="stylesheet" /><link href="../../Styles/StyleSheet.css" type="text/css" rel="stylesheet" /><style type="text/css">.hidden { display:none; }</style><link href="/eCS/Styles/Onglets.css" type="text/css" rel="stylesheet" />
<script type="text/javascript">
//<![CDATA[
Ext.net.ResourceMgr.init({id:"DocumentMenu_1$ResourceManager1",aspForm:"form1",theme:"gray",ns:[""],appName:"eCS"});Ext.onReady(function(){Ext.ns("Ext.net.DirectMethods");Ext.apply(Ext.net.DirectMethods, { DoArchiveDocument:function(comment,config){return Ext.net.DirectMethod.request("DoArchiveDocument",Ext.applyIf(config || {}, {params:{comment:comment},control:"DocumentMenu_1_Toolbox1"}));} });});
//]]>
</script>
</head>
<body>
<form name="form1" method="post" action="./M_page1.aspx?Open&amp;IdDoc=6c25d2a0-b87f-4ddc-ab73-b66e58328936&amp;tabkey=6c25d2a0-b87f-4ddc-ab73-b66e58328936&amp;_dc=1461066481311&amp;_dc=1461066483858" onsubmit="javascript:return WebForm_OnSubmit();" id="form1" enctype="multipart/form-data">
<div>
<input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value="" />
<input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value="" />
<input type="hidden" name="ctl04_HiddenField" id="ctl04_HiddenField" value="" />
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKLTMxNjQ3MDk0Mw8WBh4QRG9jdW1lbnRVbmlxdWVJZAU kNmMyNWQyYTAtYjg3Zi00ZGRjLWFiNzMtYjY2ZTU4MzI4OTM2H g1Jc05ld0RvY3VtZW50aB4JTGFzdExvZ0lkBSRjNjc2NmJmOC1 hMzFlLTRiNWMtYjBmMC00MzE5ZGI5OGUyNDgWAgIDDxYCHgdlb mN0eXBlBRNtdWx0aXBhcnQvZm9ybS1kYXRhFiYCBA9kFh4CCg8 PF....
.
.
.

I ran few scripts to find where are these large strings created or rooted from and I see most of them are coming from Transformer.NET.ItemTag operation.


Thread 2704:
0000005c763bdc70 00007ffaf82f3664 Transformer.NET.TokenSelector.Place(System.Text.St ringBuilder, Transformer.NET.Token)
r13:
-> 0000005982a488f8 Transformer.NET.ItemTag
-> 0000005982a3ffc0 Ext.Net.ExtNetTransformer
-> 0000005982a40088 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[Transformer.NET.TokenSelector, Transformer.NET]]
-> 0000005982a404b8 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[Transformer.NET.TokenSelector, Transformer.NET]][]
-> 0000005982a400d8 Transformer.NET.TokenSelector
-> 00000059b6834b88 System.Text.RegularExpressions.Regex
-> 00000059b6834c40 System.Text.RegularExpressions.ExclusiveReference
-> 00000059b686ac90 System.Text.RegularExpressions.CompiledRegexRunner
-> 0000005c3c119c68 System.String

fabricio.murta
Apr 29, 2016, 6:15 PM
Hello!

We recently analyzed examples explorer and we could get that the IIS process were growing but the APP pool had its own cycling mechanism reloading the worker processes in such a way we couldn't really grab a huge 1GB+ memory usage for the whole process. Down to 2.x also, Ext.NET stored in its assembly more resources than it was really necessary, so between 2.x to 3.x releases, there were a major cleanup reducing considerably the size of the assembly thus its memory usage (as most of the memory used were caching embedded resources on it).

Maybe that's the reason we couldn't notice a so huge memory usage in the recent tests. I presume you did all your investigation using 2.x assemblies, right? This System.String high usage might also come from text-encoded embedded resources besides compiled pages. I'm not sure the summary results you show are limited to the Transformer.NET.itemTag space, are them?

geoffrey.mcgill
Apr 29, 2016, 7:16 PM
Hello dlefevre,

Thanks for the information on the performance report.

There is a lot of string manipulation going on within the final processing of the Html, mostly within the HttpHandler.

That code has not been reviewed in a long time, so we'll take a good look.

dlefevre
May 09, 2016, 3:10 PM
Hello Fabricio and Geoffrey

Indeed we did all our investigations using 2.x assemblies which is the version we use. We did a quick test with version 3 but detected several issues and side effects when so it is not possible for us to migrate quickly to the new version.
We did further investigations and always detected LOH generated by Transformer.NET.itemTag.
Actually, the w3wp dumps are analysed by Microsoft for an important project for a customer of us. The microsoft escalade team is ready to exchange with you if necessary.
Any help will be appreciated.

Didier

fabricio.murta
May 10, 2016, 2:57 PM
Hello Didier!

About the pages being stored in text format, I'm not sure it can be helped, or else it would just slow things down zipping/unzipping the page.

I would like you to clarify what exactly the problem is. Are you noticing these memory fragments are not being erased, ever? Your testbed is your production system, or did you write a test page for this purpose (so you could share the test case with us for us to also check by our side)?

Do you let IIS worker processes to refresh over time (short or reasonable session inactivity timeout)? Or do you set up IIS to keep processes open for very long times?

fabricio.murta
May 24, 2016, 8:51 PM
Hello Didier! Are you still interested in this issue? We certainly are and would appreciate your feedback!