[CLOSED] Memory fragmentation problems with Transormer.NET

**dlefevre** · Apr 29, 2016, 10:35 AM

Hi,

I issue a performance pb with an application due to large object heap consuming memory in the IIS process

The analyse of a memory dump of the w3wp process show that Transformer.NET.ItemTag is storing most of the LOH html pages in form of strings.

So my questions are :
Are you aware on such an issue ?
Is there a way to avoid this ?

Here is a short resume of the analysys

As seen in the summary table below, free objects are taking up almost 1 GB and System.String object are taking 652 Mb.

00007ffaf7e8fc38    31020      4026528 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[Avanteam.Formulars.IApsOption, Avanteam.Formulars]][]
00007ffaf67a3c20    33754      4050480 System.Web.UI.LiteralControl
00007ffaf7605418   173251      5544032 System.Web.UI.StateBag
00007ffaf67a3f98   103033      5769848 System.Web.UI.ControlCollection
00007ffaf4cacf78   124757      5988336 System.Text.StringBuilder
00007ffaf772f720    27078      6065472 System.Data.SqlClient._SqlMetaData
00007ffaf6527ab0    13098      6706176 System.Data.DataTable
00007ffaf66f7620    57201      6864120 System.Data.SimpleType
00007ffaf7d89ea0    21229      7623408 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.Object, mscorlib]][]
00007ffaf4f96838   211520      8460800 System.Collections.Specialized.HybridDictionary
00007ffaf4f96a98   178691      8577168 System.Collections.Specialized.ListDictionary
00007ffaf4e9e0e8   217760      8710400 System.Collections.ArrayList
00007ffaf85d5a58   225196      9007840 Avanteam.Kernel.Schema.APSColumn
00007ffaf5099978   201501      9672048 System.Collections.ArrayList+ArrayListEnumeratorSimple
00007ffaf76059b8   326320     10442240 System.Web.UI.StateItem
00007ffaf85d29c8   208441     13340224 Avanteam.Kernel.Schema.APSField
00007ffaf7a52b60   354541     14181640 System.Web.UI.ControlCollection+ControlCollectionEnumerator
00007ffaf7752258    74408     14286336 System.Data.SqlClient.SqlParameter
00007ffaf5044d18   376816     15072640 System.Collections.Specialized.ListDictionary+DictionaryNode
00007ffaf66f2238    27283     15127920 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Data.DataColumn, System.Data]][]
00007ffaf512de08    15935     16543888 System.Double[]
00007ffaf678e710   215139     17211120 System.Data.Common.StringStorage
00007ffaf66cdf70   193828     18607488 System.Data.DataRow
00007ffaf4e58b80    62970     21432912 System.Collections.Hashtable+bucket[]
00007ffaf851b7b8   209310     21768240 Avanteam.Documents.DataSets.DSDocuments+FieldDefinitionsRow
00007ffaf52f4ac8    13650     22535376 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.String, mscorlib]][]
00007ffaf4f77cf8    30079     31762024 System.DateTime[]
00007ffaf4e57990   432588     40824924 System.Int32[]
00007ffaf66f44f8    32066     54797104 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
00007ffaf669c7d8   326615     70548840 System.Data.DataColumn
00007ffaf6789070    49280     72912896 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
00007ffaf54ff2e8  2504955     80158560 System.ResolveEventArgs
00007ffaf4cf1538    58152    123882204 System.Byte[]
00007ffaf4cac5c8   422155    132354080 System.Char[]
00007ffaf4c5be30  1351930    409909384 System.Object[]
00007ffaf4ca4d18  3909705    652648888 System.String
0000005934696110    23551   1056192938      Free

When I dumped out System.String objects in LOH, I see these are whole page html stored as a string object.

<!DOCTYPE html>
<html>
<head>
    <link type="text/css" rel="stylesheet" href="/eCS/extjs/resources/ext_theme_gray/ext-theme-gray-all-embedded-css/ext.axd?v=31210" id="ext-theme" />
    <link type="text/css" rel="stylesheet" href="/eCS/extnet/resources/css/extnet-all-embedded-css/ext.axd?v=31210" id="extnet-styles" />
    <script type="text/javascript" src="/eCS/extjs/ext-all-js/ext.axd?v=31210"></script>
    <script type="text/javascript" src="/eCS/extnet/extnet-all-js/ext.axd?v=31210"></script>
    <script type="text/javascript" src="/eCS/extnet/locale/ext-lang-fr-js/ext.axd?v=31210"></script>
<link rel="stylesheet" type="text/css" href="/eCS/DXR.axd?r=1_8-yQjza" /><link rel="stylesheet" type="text/css" href="/eCS/DXR.axd?r=1_9-yQjza" /><link rel="stylesheet" type="text/css" href="/eCS/DXR.axd?r=1_1-yQjza" /><title>
                
</title><link href="../../Styles/css/document.css" type="text/css" rel="stylesheet" /><link href="../../Styles/StyleSheet.css" type="text/css" rel="stylesheet" /><style type="text/css">.hidden { display:none; }</style><link href="/eCS/Styles/Onglets.css" type="text/css" rel="stylesheet" />
    <script type="text/javascript">
    //<![CDATA[
        Ext.net.ResourceMgr.init({id:"DocumentMenu_1$ResourceManager1",aspForm:"form1",theme:"gray",ns:[""],appName:"eCS"});Ext.onReady(function(){Ext.ns("Ext.net.DirectMethods");Ext.apply(Ext.net.DirectMethods, { DoArchiveDocument:function(comment,config){return Ext.net.DirectMethod.request("DoArchiveDocument",Ext.applyIf(config || {}, {params:{comment:comment},control:"DocumentMenu_1_Toolbox1"}));} });});
    //]]>
    </script>
</head>
<body>
<form name="form1" method="post" action="./M_page1.aspx?Open&amp;IdDoc=6c25d2a0-b87f-4ddc-ab73-b66e58328936&amp;tabkey=6c25d2a0-b87f-4ddc-ab73-b66e58328936&amp;_dc=1461066481311&amp;_dc=1461066483858" onsubmit="javascript:return WebForm_OnSubmit();" id="form1" enctype="multipart/form-data">
<div>
<input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value="" />
<input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value="" />
<input type="hidden" name="ctl04_HiddenField" id="ctl04_HiddenField" value="" />
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKLTMxNjQ3MDk0Mw8WBh4QRG9jdW1lbnRVbmlxdWVJZAUkNmMyNWQyYTAtYjg3Zi00ZGRjLWFiNzMtYjY2ZTU4MzI4OTM2Hg1Jc05ld0RvY3VtZW50aB4JTGFzdExvZ0lkBSRjNjc2NmJmOC1hMzFlLTRiNWMtYjBmMC00MzE5ZGI5OGUyNDgWAgIDDxYCHgdlbmN0eXBlBRNtdWx0aXBhcnQvZm9ybS1kYXRhFiYCBA9kFh4CCg8PF....
.
.
.

I ran few scripts to find where are these large strings created or rooted from and I see most of them are coming from Transformer.NET.ItemTag operation.

Thread 2704:
    0000005c763bdc70 00007ffaf82f3664 Transformer.NET.TokenSelector.Place(System.Text.StringBuilder, Transformer.NET.Token)
        r13: 
            ->  0000005982a488f8 Transformer.NET.ItemTag
            ->  0000005982a3ffc0 Ext.Net.ExtNetTransformer
            ->  0000005982a40088 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[Transformer.NET.TokenSelector, Transformer.NET]]
            ->  0000005982a404b8 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[Transformer.NET.TokenSelector, Transformer.NET]][]
            ->  0000005982a400d8 Transformer.NET.TokenSelector
            ->  00000059b6834b88 System.Text.RegularExpressions.Regex
            ->  00000059b6834c40 System.Text.RegularExpressions.ExclusiveReference
            ->  00000059b686ac90 System.Text.RegularExpressions.CompiledRegexRunner
            ->  0000005c3c119c68 System.String

**fabricio.murta** · Apr 29, 2016, 5:15 PM

Hello!

We recently analyzed examples explorer and we could get that the IIS process were growing but the APP pool had its own cycling mechanism reloading the worker processes in such a way we couldn't really grab a huge 1GB+ memory usage for the whole process. Down to 2.x also, Ext.NET stored in its assembly more resources than it was really necessary, so between 2.x to 3.x releases, there were a major cleanup reducing considerably the size of the assembly thus its memory usage (as most of the memory used were caching embedded resources on it).

Maybe that's the reason we couldn't notice a so huge memory usage in the recent tests. I presume you did all your investigation using 2.x assemblies, right? This System.String high usage might also come from text-encoded embedded resources besides compiled pages. I'm not sure the summary results you show are limited to the Transformer.NET.itemTag space, are them?

**geoffrey.mcgill** · Apr 29, 2016, 6:16 PM

Hello dlefevre,

Thanks for the information on the performance report.

There is a lot of string manipulation going on within the final processing of the Html, mostly within the HttpHandler.

That code has not been reviewed in a long time, so we'll take a good look.

**dlefevre** · May 09, 2016, 2:10 PM

Hello Fabricio and Geoffrey

Indeed we did all our investigations using 2.x assemblies which is the version we use. We did a quick test with version 3 but detected several issues and side effects when so it is not possible for us to migrate quickly to the new version.
We did further investigations and always detected LOH generated by Transformer.NET.itemTag.
Actually, the w3wp dumps are analysed by Microsoft for an important project for a customer of us. The microsoft escalade team is ready to exchange with you if necessary.
Any help will be appreciated.

Didier

**fabricio.murta** · May 10, 2016, 1:57 PM

Hello Didier!

About the pages being stored in text format, I'm not sure it can be helped, or else it would just slow things down zipping/unzipping the page.

I would like you to clarify what exactly the problem is. Are you noticing these memory fragments are not being erased, ever? Your testbed is your production system, or did you write a test page for this purpose (so you could share the test case with us for us to also check by our side)?

Do you let IIS worker processes to refresh over time (short or reasonable session inactivity timeout)? Or do you set up IIS to keep processes open for very long times?

**fabricio.murta** · May 24, 2016, 7:51 PM

Hello Didier! Are you still interested in this issue? We certainly are and would appreciate your feedback!

Thread: [CLOSED] Memory fragmentation problems with Transormer.NET

Thread Tools

Search Thread

Display

[CLOSED] Memory fragmentation problems with Transormer.NET

Similar Threads

[CLOSED] Memory leak

[1.5]Big bug for ext memory leak!

[CLOSED] [1.2] Memory

Out of memory

Memory Leak in I.E.

Posting Permissions

Sitemap

Downloads

Resources

Thread: [CLOSED] Memory fragmentation problems with Transormer.NET

[CLOSED] Memory fragmentation problems with Transormer.NET

Similar Threads

Posting Permissions

Sitemap

Downloads

Resources

Stay connected with us