PDA

View Full Version : [OPEN] [#356] Gzipped extnet static resources not persistanced



michaeld
Sep 23, 2013, 1:35 AM
I guess this is more of a feature request, but it seems to me that the static ext.net resources call CompressionUtils.GZip on the static bytestream each time a resource is requested (when GZip="true"). It seems to me that it would be 20 times more efficient if the requested resources were already gzipped as a static file resource (part of build project) and transmitted directly instead of processed then transmitted. The second best option would allow the resources to persist to a writable physical location on the server the first time requested then transmitted. As extnet is already pretty heavy on the cpu given the number of transformations, this would significantly task the cpu less from a scalability standpoint.

I realize once-each client has cached the resource, it doesn't need to repeatedly process and transfer. I also verified that the processed output is server-side cached to prevent the zip being reprocessed again. However, as these fragments can be fairly large, I'm finding that my server is blowing away these cached resources frequently due to restricted access to memory. In other words, the server-side cache is more-often rendered null and void and has to reprocess again-and-again. Moreover, every worker thread keeps its own cached reference in memory. A different cache-provider might help but also add its own performance weight. This issue hasn't been one on my development environment, but I noticed the issue when I ran a trace on my production server. The short term answer would be make more memory available on my production environment, but the BEST solution of all worlds would be my original request for the resources to be static persisted files or resources like the other static ext.net content.

michaeld
Sep 23, 2013, 2:30 AM
I also looked into the fact that all resources are cached, compressed or not, which also may be part of the reason the server-cache is so eager to release memory.

Also note the following code in ResourceHandler.cs can be improved.


try
{
this.sm = new ResourceManager();
this.compress = CompressionUtils.IsGZipSupported && this.sm.GZip;


this.SetWebResourceName(file);


this.stream = this.GetType().Assembly.GetManifestResourceStream( this.webResource);
string ext = this.webResource.RightOfRightmostOf('.');
this.compress = this.compress && !this.IsImage(ext);


switch (ext)
{
case "js":
this.WriteFile("text/javascript");
break;
case "css":
this.WriteFile("text/css");
break;
case "swf":
this.compress = false;
this.WriteImage("application/x-shockwave-flash");
break;
case "gif":
this.WriteImage("image/gif");
break;
case "png":
this.WriteImage("image/png");
break;
case "jpg":
case "jpeg":
this.WriteImage("image/jpg");
break;
}
}
catch (Exception e)
{
string s = this.IsDebugging ? e.ToString() : e.Message;
context.Response.StatusDescription = s.Substring(0, Math.Min(s.Length, 512));
this.context.Response.Redirect(Page.ClientScript.G etWebResourceUrl(this.sm.GetType(), this.webResource));
}
finally
{
if (this.stream != null)
{
this.stream.Close();
}
}
}


You're getting the Resource byte stream every time a resource is requested, even though you test later to see if you already have it in memory.
Instead you should call this.GetCache() before this entire section of code instead of inside of each WriteFile/WriteImage. You can pass the content instead as an additional parameter to WriteFile/WriteImage.

Revised:


try
{
this.sm = new ResourceManager();
this.SetWebResourceName( file );
string ext = this.webResource.RightOfRightmostOf( '.' );
this.compress = this.sm.GZip && CompressionUtils.IsGZipSupported; // changed order to prevent IsGZipSupported test unless required
this.compress = this.compress && ( !this.IsImage( ext ) && ext != "swf" ); // moved swf test here to load correct cache ref


// Load stream only if not already available in server-cache
this.output = this.GetCache();
if( this.output == null )
this.stream = this.GetType().Assembly.GetManifestResourceStream( this.webResource );


switch( ext )
{
case "js":
this.WriteFile( "text/javascript" );
break;
case "css":
this.WriteFile( "text/css" );
break;
case "swf":
this.WriteImage( "application/x-shockwave-flash" );
break;
case "gif":
this.WriteImage( "image/gif" );
break;
case "png":
this.WriteImage( "image/png" );
break;
case "jpg":
case "jpeg":
this.WriteImage( "image/jpg" );
break;
}
}
catch( Exception e )
{
string s = this.IsDebugging ? e.ToString() : e.Message;
context.Response.StatusDescription = s.Substring( 0, Math.Min( s.Length, 512 ) );
this.context.Response.Redirect( Page.ClientScript.GetWebResourceUrl( this.sm.GetType(), this.webResource ) );
}
finally
{
if( this.stream != null )
{
this.stream.Close();
}
}
}




I also fixed some other small ordering issues.


Also, had to pull GetCache call from WriteImage and WriteFile:


private void WriteFile(string responseType)
{


if (this.output != null)
{
this.Send(this.output, responseType);

return;
}


this.sb = new StringBuilder(4096);
...


private void WriteImage(string responseType)
{


if (this.output == null)
{
this.length = Convert.ToInt32(this.stream.Length);
this.output = new Byte[this.length];
this.stream.Read(this.output, 0, this.length);


this.SetCache(this.output);
}


this.Send(this.output, responseType);
}

michaeld
Sep 23, 2013, 2:45 AM
^^^
Sorry edited till I got the bugs out.

Daniil
Sep 25, 2013, 5:30 AM
Hi @michaeld,

Thank you for the suggestion! We will review.

michaeld
Oct 08, 2013, 12:07 AM
Thank you for the suggestion! We will review.

I haven't seen this hit the trunk yet. Might it make 2.3? It would be nice not having to load the file resources on every load. It's not a major issue, but some of the streams are very large and can benefit from this edit by a a few tens of milliseconds. I've been testing my branch of this edit on production and dev for the past week; it working reliably.

geoffrey.mcgill
Oct 08, 2013, 3:22 AM
I don't think this revision has been made in Svn. For some reason I missed this thread when originally posted.

We're going to try and implement this immediately and get into the 2.3.0 release.


It seems to me that it would be 20 times more efficient if the requested resources were already gzipped as a static file resource (part of build project) and transmitted directly instead of processed then transmitted.

This never crossed my mind before, but it makes perfect sense. The pre-compressed files will probably not make it into 2.3.0, but we will investigate.

michaeld
Oct 08, 2013, 4:39 AM
I don't think this revision has been made in Svn. For some reason I missed this thread when originally posted.
We're going to try and implement this immediately and get into the 2.3.0 release.

Cool. I observed the latest SVN updates were suggesting you were probably days away from that posting which is why I have been reminding you guys on some of the topics like this (oh and the Ext.Net.Utilities stringutils.cs).


This never crossed my mind before, but it makes perfect sense. The pre-compressed files will probably not make it into 2.3.0, but we will investigate.

Your current implementation is pretty sufficient. It creates the gzipped version to cache only once which puts you already ahead of most competitive libraries which barely do a fraction. If it weren't for caches being independent per worker thread (garden) and instance (farm), I wouldn't even have been worth mention. Though, I would understand if you didn't have the pre-compressed versions in resources on any but release minified versions given the number of permutations of debug and dev, but since you already have a good handle on builder extensions in VS, it might not be that difficult to do them all.

michaeld
Oct 08, 2013, 5:12 AM
Geoffrey, on the topic of subtle improvements in resource/performance deployments, you might also want to check in on this thread too.
http://forums.ext.net/showthread.php?25367-SSL-CDN/

Daniil
Oct 10, 2013, 4:14 PM
It should be addressed in the SVN trunk revision #5391. Could you update and give it a try, please?

michaeld
Oct 11, 2013, 12:30 AM
I observed vladimir changed the section to prevent reload of resources. I'll give it a try once I've fixed my extjs cdn code.

michaeld
Oct 11, 2013, 6:29 AM
Seems to work. I know Geoffrey had mentioned considering implementing the pre-gzipped resources, so I don't know if you guys added that to your tracker but you can close this item unless you want to keep it open till after that implementation. From what I can see, it looks like the foundation for the option is there now.

Daniil
Oct 15, 2013, 11:55 AM
Created an Issue.
https://github.com/extnet/Ext.NET/issues/356