Solving long JSON return from a webservice

I was being plagued by this error intermittently:
Error during serialization or deserialization using the JSON JavaScriptSerializer. The length of the string exceeds the value set on the maxJsonLength property

Some Googling led me to realise that there is indeed a limit, and this limit is quite low.
Most of the posts explained how to increase the limit but also explained why this was a bad idea and that you should instead reduce your JSON size.

I agree with this, better to return what you can to the client and tell them to "try again" for whatever is left.
But this poses a problem - how do you know how much data you can return at once?
Google drew a blank on this one.

This little function here will do the trick:

using System.Configuration; using System.Web.Configuration; public int GetMaxJSONLength() { Configuration cConfig = WebConfigurationManager.OpenWebConfiguration(null); return ((ScriptingWebServicesSectionGroup) cConfig. SectionGroups["system.web.extensions"]. SectionGroups["scripting"]. SectionGroups["webServices"] ).JsonSerialization.MaxJsonLength; }
This can be executed from anywhere, it doesn't have to be part of the web project itself.

Something else to be aware of - any strings in your return value will be encoded.
That means characters like < and > will become \u003c and \u003e - 6 times longer than they were previously.
So if you have a string containing html this can make a substantial difference to the length of the string.

To account for this make sure you use the length of the encoded string, not the original.

The woes of inline functions

OK; so I like inline fucntions really - especially in javascript. But now I am realising their evil intent after I spent a not insubstantial amount of time tracing a bug.
here is the scenario:
A simple method with tbh not a whole lot in it.
In fact the entire method is taken by a single if/else block. Regardless of the outcome of both if blocks it is necassary to call a method afterwards.
So obviously that call goes outside the if.
Normally that would be fine.
But in my case the else block contained an inline function used as a callback.
A callback that will occur some undetermined (but very small) amount of time after the if block has exited.
Of course I needed to wait until this callback had been executed before calling the final method.
function DoSomething() { if (condition) { DoOneThing(); } else { AsyncronuslyDoAnother(function (result){ DoSomethingWithTheResultWhenTheAsyncCallCompletes(); }); } DoFinalThing(); } Now the simple solution in this instance was to move the call to DoFinalThing() to be inside the if block, and also inside the anonymous callback function.
function DoSomething2() { if (condition) { DoOneThing(); DoFinalThing(); } else { AsyncronuslyDoAnother(function (result){ DoSomethingWithTheResultWhenTheAsyncCallCompletes(); DoFinalThing(); }); } } However in my case the problem only arose in the first place because I simply didn't notice the inline function when I added the call to DoFinalThing() and assumed that it was all just part of the else logic.
So additionally I removed the inline function and henceforth declare all such things to be evil!
(ok well not quite, but at least I am more wary now!)
function AsyncCallback(result) { DoSomethingWithTheResultWhenTheAsyncCallCompletes(); DoFinalThing(); } function DoSomething3() { if (condition) { DoOneThing(); DoFinalThing(); } else { AsyncronuslyDoAnother(AsyncCallback); //N.B. no parens on AsyncCallback - we are passing a reference to the method not invoking it } }

Ajaxable Web Control in a Class Library

So I recently went on a course to get my AJAX knowledge up to scratch.
The course was fine and taught me quite a bit that I didn't know but the most important thing for me was missing - how do you do all this in a class library?

Basically I want the ability to have some control which has both server and client side components, for these 2 parts to talk to eachother with ajax, and I want to be able to wrap the whole thing up into a DLL file that can be dropped into any web application and with no user modifications required.

The instructor gave me this link which was helpful but didn't go the whole way.
I knew there had to be a better, more complete way, and so I set to work.

The result is what you see below.

Add the C# code in the block below to a class library and inherit from it for your server control.
You'll need a WebService and you'll have to provide that as the type parameter to the base class.
And that's about it!
The class does require some tweaking to the web.config file - but it will do that for you the first time you run it.

Here you can find the compiled DLL: http://dev.dj-djl.com/AjaxableServerControlWithWebService.zip
And a Template project for Visual Studio 2012: dev.dj-djl.com/AjaxableServerControlWithWebServiceTemplate.zip
The template control contains a single web-exposed method which is triggered when the button is clicked and the result is given to a simple alert box.

The code below contains some code taken from this Code Project post, which was modified by me.
The rest of the code is written by me. It is all licecned under an LGPL3.0 licence.   /* LGPL3.0 Classes enabling embedding of a fully AJAXable ASPX Server Control into a DLL for reuse or redistribution. Most of the code below is LGPL3.0 - Copyright (C) 2012 Lee Smith http://dev.dj-djl.com http://blog.dj-djl.com Contains some LGPL3.0 code As indicated - (C) 2007 James Ashley http://www.codeproject.com/Articles/22384/ASP-NET-AJAX-Controls-and-Extenders#http_handler LGPL3.0: This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not please see <http://www.gnu.org/licenses/>. */ using System; using System.Collections.Generic; using System.Reflection; using System.Text; using System.Web; using System.Web.Services; using System.Web.SessionState; using System.Web.UI; namespace DJL { internal static class Utils { public static string ToDelimitedstring<T>(this List<T> lst, string Delimiter = ",") { System.Text.StringBuilder sbReturn = new StringBuilder(); Boolean flgFirst = true; foreach (T Item in lst) { if (!flgFirst) { sbReturn.Append(Delimiter); } sbReturn.Append(Item.ToString()); flgFirst = false; } return sbReturn.ToString(); } public static string CompressAndBase64(string Value) { System.IO.MemoryStream mst = new System.IO.MemoryStream(); System.IO.Compression.GZipStream gz = new System.IO.Compression.GZipStream(mst, System.IO.Compression.CompressionLevel.Optimal, true); byte[] bytValue = System.Text.Encoding.UTF8.GetBytes(Value); gz.Write(bytValue,0 , bytValue.Length); gz.Flush(); gz.Close(); return Convert.ToBase64String(mst.ToArray()).Replace("+", "_").Replace("/", "-"); } public static string DeBase64AndDecompress(string Base64Value) { //return System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(Base64Value)); byte[] bytValue = Convert.FromBase64String(Base64Value.Replace("-", "/").Replace("_", "+")); System.IO.MemoryStream mst = new System.IO.MemoryStream(); System.IO.Compression.GZipStream gz =null; try { mst.Write(bytValue, 0, bytValue.Length); mst.Seek(0, System.IO.SeekOrigin.Begin); gz = new System.IO.Compression.GZipStream(mst, System.IO.Compression.CompressionMode.Decompress, false); byte[] bytDecompressedValue = new byte[102400]; //100k - should be enough! return System.Text.Encoding.UTF8.GetString(bytDecompressedValue, 0, gz.Read(bytDecompressedValue, 0, 102400)); } finally { gz.Close(); mst.Close(); } } public static string GetURLWithoutAsmx(Type type) { //System.Security.Cryptography.MD5 md5 = System.Security.Cryptography.MD5.Create(); //byte[] bytTypeName = System.Text.Encoding.UTF8.GetBytes(type.Assembly.FullName + "/" + type.FullName); //return typeof(AjaxableScriptContolWithWebserviceHandlerFactory).Name + "/" + Convert.ToBase64String(md5.ComputeHash(bytTypeName)); return typeof(AjaxableScriptContolWithWebserviceHandlerFactory).Name + "/" + CompressAndBase64(type.Assembly.FullName + "/" + type.FullName); } public static string GetURLWithoutAsmx<T>() { return GetURLWithoutAsmx(typeof(T)); } } /// <summary> /// A serverside control which can be included into a DLL, with full ajax. /// It uses a webservice embedded into the same DLL to perform ajax operations. /// </summary> /// <typeparam name="TService">The type of the Webservice</typeparam> public abstract class AjaxableScriptContolWithWebservice<TService> : ScriptControl where TService : System.Web.Services.WebService { private bool flgConfigChanged = false; /// <summary> /// When set indicates that the web.config file was altered /// </summary> protected bool ConfigChanged { get { return flgConfigChanged; } } public AjaxableScriptContolWithWebservice() { CheckConfig(); this.Load += AjaxableScriptContolWithWebservice_Load; } private void CheckConfig() { //System.Configuration blah = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~"); System.Configuration.Configuration WebConfig = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~"); System.Xml.XmlDocument xmdWebConfig = new System.Xml.XmlDocument(); xmdWebConfig.Load(WebConfig.FilePath); if (System.Web.HttpRuntime.UsingIntegratedPipeline) { SetIntegratedConfig(xmdWebConfig, WebConfig.FilePath); } else { SetClassicConfig(xmdWebConfig, WebConfig.FilePath); } } private void SetIntegratedConfig(System.Xml.XmlDocument WebConfig, string WebConfigPath) { /* <system.webServer> <handlers> <add name="AjaxServerControl1.ServerControl1" verb="*" path="AjaxServerControl1.ServerControl1.asmx" type="AjaxServerControl1.ServerControl1" /> </handlers> </system.webServer> */ System.Xml.XmlElement xmeSystemWeb = (System.Xml.XmlElement)WebConfig.SelectSingleNode("/configuration/system.webServer"); if (xmeSystemWeb == null) { xmeSystemWeb = WebConfig.CreateElement("system.webServer"); WebConfig.DocumentElement.AppendChild(xmeSystemWeb); } System.Xml.XmlElement xmeHandlers = (System.Xml.XmlElement)xmeSystemWeb.SelectSingleNode("handlers"); if (xmeHandlers == null) { xmeHandlers = WebConfig.CreateElement("handlers"); xmeSystemWeb.AppendChild(xmeHandlers); } string strURL = Utils.GetURLWithoutAsmx<TService>(); System.Xml.XmlElement xmeAdd = (System.Xml.XmlElement)xmeHandlers.SelectSingleNode("add[@name='" + strURL + "']"); if (xmeAdd == null) { xmeAdd = WebConfig.CreateElement("add"); xmeAdd.SetAttribute("verb", "*"); xmeAdd.SetAttribute("name", strURL); xmeAdd.SetAttribute("path", strURL + ".asmx"); xmeAdd.SetAttribute("type", typeof(AjaxableScriptContolWithWebserviceHandlerFactory).FullName); xmeHandlers.AppendChild(xmeAdd); WebConfig.Save(WebConfigPath); flgConfigChanged = true; } } private void SetClassicConfig(System.Xml.XmlDocument WebConfig, string WebConfigPath) { throw new Exception(); /* <system.web> <httpHandlers> <add verb="*" path="AjaxServerControl1.ServerControl1.asmx" type="AjaxServerControl1.ServerControl1" validate="true" /> </httpHandlers> </system.web> * */ System.Xml.XmlElement xmeSystemWeb = (System.Xml.XmlElement)WebConfig.SelectSingleNode("/configuration/system.web"); if (xmeSystemWeb == null) { xmeSystemWeb = WebConfig.CreateElement("system.web"); WebConfig.DocumentElement.AppendChild(xmeSystemWeb); } System.Xml.XmlElement xmeHTTPHandlers = (System.Xml.XmlElement)xmeSystemWeb.SelectSingleNode("httpHandlers"); if (xmeHTTPHandlers == null) { xmeHTTPHandlers = WebConfig.CreateElement("httpHandlers"); xmeSystemWeb.AppendChild(xmeHTTPHandlers); } string strURL = Utils.GetURLWithoutAsmx<TService>(); System.Xml.XmlElement xmeAdd = (System.Xml.XmlElement)xmeHTTPHandlers.SelectSingleNode("add[@path='" + strURL + ".asmx']"); if (xmeAdd == null) { xmeAdd = WebConfig.CreateElement("add"); xmeAdd.SetAttribute("verb", "*"); xmeAdd.SetAttribute("path", strURL + ".asmx"); xmeAdd.SetAttribute("type", typeof(AjaxableScriptContolWithWebserviceHandlerFactory).FullName); xmeAdd.SetAttribute("validate", "true"); xmeHTTPHandlers.AppendChild(xmeAdd); WebConfig.Save(WebConfigPath); flgConfigChanged = true; } } protected bool AutoRefreshOnConfigChange { get; set; } protected override sealed void Render(HtmlTextWriter writer) { base.Render(writer); if (ConfigChanged && AutoRefreshOnConfigChange) { writer.WriteLine("<div style='position: fixed; height: 100%; width: 100%; top: 0; left: 0; background-color: rgba(128,128,128,0.85);'>\n" + " <div style='width: 40em; height: 5em; background-color: white; color: black; box-shadow: 2px 2px 0px black; text-align: center;" + "position: absolute; left: 50%; top: 50%; margin-left: -20em; margin-top: -2.5em;'><br/>" + "The web.config was updated. The page will now be reloaded.<br/>\n" + " N.B. You may have to check out the web.config file if it is under source control</div>\n" + "</div>"); writer.WriteLine("<meta http-equiv='refresh' content='1;" + this.Page.Request.Url.ToString() + "'/>"); } else { RenderContent(writer); } } protected virtual void RenderContent(HtmlTextWriter writer) { } void AjaxableScriptContolWithWebservice_Load(object sender, EventArgs e) { string strURL = Utils.GetURLWithoutAsmx<TService>(); ScriptManager.GetCurrent(this.Page).Services.Add(new ServiceReference(strURL + ".asmx")); } } // This class is largely written by James Ashley with some modifications by Lee Smith as indicated // see http://www.codeproject.com/Articles/22384/ASP-NET-AJAX-Controls-and-Extenders#http_handler for the original source code public class AjaxableScriptContolWithWebserviceHandlerFactory : IHttpHandlerFactory { #region IHttpHandlerFactory Members IHttpHandlerFactory factory = null; public IHttpHandler GetHandler(HttpContext context, string requestType, string url, string pathTranslated) { Assembly ajaxAssembly = typeof(System.Web.Script.Services.GenerateScriptTypeAttribute).Assembly; factory = (IHttpHandlerFactory)System.Activator.CreateInstance(ajaxAssembly.GetType("System.Web.Script.Services.RestHandlerFactory")); IHttpHandler restHandler = (IHttpHandler)System.Activator.CreateInstance(ajaxAssembly.GetType("System.Web.Script.Services.RestHandler")); ConstructorInfo WebServiceDataConstructor = ajaxAssembly.GetType("System.Web.Script.Services.WebServiceData").GetConstructor( BindingFlags.NonPublic | BindingFlags.Instance, null, new Type[] { typeof(Type), typeof(bool) }, null); MethodInfo CreateHandlerMethod = restHandler.GetType().GetMethod("CreateHandler", BindingFlags.NonPublic | BindingFlags.Static, null, new Type[] { ajaxAssembly.GetType("System.Web.Script.Services.WebServiceData"), typeof(string) }, null); IHttpHandler originalHandler = null; //LS Change the behvaiour for some URLs, original code contained some hard-coded assumptions and also did not provide the js and jsdebug methods. string strAssemblyName = context.Request.Url.Segments[2].Trim('/').Trim(); if (strAssemblyName.ToLower().EndsWith(".asmx")) { strAssemblyName = strAssemblyName.Substring(0, strAssemblyName.Length - 5); } strAssemblyName = Utils.DeBase64AndDecompress(strAssemblyName); string strTypeName = strAssemblyName.Split('/')[1].Trim(); strAssemblyName = strAssemblyName.Split('.')[0].Trim(); string strMethodName = context.Request.Url.Segments[3]; Assembly assAssembly = Assembly.Load(strAssemblyName); Type typWebServiceType = assAssembly.GetType(strTypeName); if (strMethodName == "js" || strMethodName == "jsdebug") { originalHandler = new WebserviceJSGenerator(typWebServiceType); } else { //LS End modifications originalHandler = (IHttpHandler)CreateHandlerMethod.Invoke(restHandler, new Object[]{ WebServiceDataConstructor.Invoke(new object[] { typWebServiceType, false }), strMethodName }); } //<-- LS Type t = ajaxAssembly.GetType("System.Web.Script.Services.ScriptHandlerFactory"); Type wrapperType = null; if (originalHandler is IRequiresSessionState) { wrapperType = t.GetNestedType("HandlerWrapperWithSession", BindingFlags.NonPublic | BindingFlags.Instance); } else { wrapperType = t.GetNestedType("HandlerWrapper", BindingFlags.NonPublic | BindingFlags.Instance); } return (IHttpHandler)System.Activator.CreateInstance(wrapperType, BindingFlags.NonPublic | BindingFlags.Instance, null, new object[] { originalHandler, factory }, null); } public void ReleaseHandler(IHttpHandler handler) { factory.ReleaseHandler(handler); } #endregion } public class WebserviceJSGenerator : IHttpHandler { public bool IsReusable { get { return false; } } public void ProcessRequest(HttpContext context) { if (context == null) { throw new ArgumentNullException("context"); } string clientProxyScript = GetClientProxyScript(type); if (clientProxyScript != null) { context.Response.ContentType = "application/x-javascript"; context.Response.Write(clientProxyScript); } } private string GetClientProxyScript(Type Type) { StringBuilder strScript = new StringBuilder(); strScript.AppendLine("// Generated by " + this.GetType().FullName); strScript.AppendLine("// http://dev.dj-djl.com"); strScript.AppendLine("// http://blog.dj-djl.com"); strScript.AppendLine("// loosely based on the code generated by Microsoft's System.Web.Script.Services.ClientProxyGenerator class"); strScript.AppendLine("// requires other Microsoft javascript libraries to be loaded and assumes that to be the case."); strScript.AppendLine("Type.registerNamespace('" + Type.Namespace + "');"); strScript.AppendLine(Type.FullName.Replace("+", "_") + "=function(){"); strScript.AppendLine(Type.FullName.Replace("+", "_") + ".initializeBase(this);"); strScript.AppendLine(" this._timeout=0,"); strScript.AppendLine(" this._userContext=null,"); strScript.AppendLine(" this._succeeded=null,"); strScript.AppendLine(" this._failed=null}"); strScript.AppendLine(Type.FullName.Replace("+", "_") + ".prototype={"); strScript.AppendLine(" _get_path:function(){"); strScript.AppendLine(" var p = this.get_path();"); strScript.AppendLine(" if (p) return p;"); strScript.AppendLine(" else return " + Type.FullName.Replace("+", "_") + "._staticInstance.get_path();"); strScript.AppendLine(" }"); foreach (System.Reflection.MethodInfo miMethod in Type.GetMethods(BindingFlags.Public | BindingFlags.Instance)) { if (miMethod.GetCustomAttributes(typeof(WebMethodAttribute), true).Length > 0) { strScript.Append(" ," + miMethod.Name + ":function("); List<string> lstParametersObject = new List<string>(); List<string> lstParametersXmlComments = new List<string>(); List<string> lstParametersJSFunction = new List<string>(); foreach (System.Reflection.ParameterInfo piParam in miMethod.GetParameters()) { if (!piParam.IsRetval) { lstParametersObject.Add(piParam.Name + ":" + piParam.Name); lstParametersXmlComments.Add(" /// <param name=\"" + piParam.Name + "\" type=\"" + piParam.ParameterType.Name + "\" optional=\"" + (piParam.IsOptional ? "true" : "false") + "\" mayBeNull=\"" + (piParam.ParameterType.IsValueType ? "true" : "false") + "\"></param>"); lstParametersJSFunction.Add(piParam.Name); } } if (lstParametersJSFunction.Count > 0) { lstParametersJSFunction.Add(""); }; //add a blank parameter so that we do infact get a trailing comma, but only if there are items strScript.AppendLine(lstParametersJSFunction.ToDelimitedstring(", ") + "succeededCallback, failedCallback, userContext) {"); strScript.AppendLine(lstParametersXmlComments.ToDelimitedstring("\n")); strScript.AppendLine(" /// <param name=\"succeededCallback\" type=\"Function\" optional=\"true\" mayBeNull=\"true\"></param>"); strScript.AppendLine(" /// <param name=\"failedCallback\" type=\"Function\" optional=\"true\" mayBeNull=\"true\"></param>"); strScript.AppendLine(" /// <param name=\"userContext\" optional=\"true\" mayBeNull=\"true\"></param>"); strScript.Append(" return this._invoke(this.get_path(), '" + miMethod.Name + "', false, {"); strScript.Append(lstParametersObject.ToDelimitedstring(", ")); strScript.AppendLine("}, succeededCallback,failedCallback,userContext);} "); } } strScript.AppendLine(" }"); string strStatic = @" <<TypeName>>.registerClass('<<TypeName>>',Sys.Net.WebServiceProxy); <<TypeName>>._staticInstance = new <<TypeName>>(); <<TypeName>>.set_path = function(value) { <<TypeName>>._staticInstance.set_path(value); } <<TypeName>>.get_path = function() { /// <value type=""String"" mayBeNull=""true"">The service url.</value> return <<TypeName>>._staticInstance.get_path();} <<TypeName>>.set_timeout = function(value) { <<TypeName>>._staticInstance.set_timeout(value); } <<TypeName>>.get_timeout = function() { /// <value type=""Number"">The service timeout.</value> return <<TypeName>>._staticInstance.get_timeout(); } <<TypeName>>.set_defaultUserContext = function(value) { <<TypeName>>._staticInstance.set_defaultUserContext(value); } <<TypeName>>.get_defaultUserContext = function() { /// <value mayBeNull=""true"">The service default user context.</value> return <<TypeName>>._staticInstance.get_defaultUserContext(); } <<TypeName>>.set_defaultSucceededCallback = function(value) { <<TypeName>>._staticInstance.set_defaultSucceededCallback(value); } <<TypeName>>.get_defaultSucceededCallback = function() { /// <value type=""Function"" mayBeNull=""true"">The service default succeeded callback.</value> return <<TypeName>>._staticInstance.get_defaultSucceededCallback(); } <<TypeName>>.set_defaultFailedCallback = function(value) { <<TypeName>>._staticInstance.set_defaultFailedCallback(value); } <<TypeName>>.get_defaultFailedCallback = function() { /// <value type=""Function"" mayBeNull=""true"">The service default failed callback.</value> return <<TypeName>>._staticInstance.get_defaultFailedCallback(); } <<TypeName>>.set_enableJsonp = function(value) { <<TypeName>>._staticInstance.set_enableJsonp(value); } <<TypeName>>.get_enableJsonp = function() { /// <value type=""Boolean"">Specifies whether the service supports JSONP for cross domain calling.</value> return <<TypeName>>._staticInstance.get_enableJsonp(); } <<TypeName>>.set_jsonpCallbackParameter = function(value) { <<TypeName>>._staticInstance.set_jsonpCallbackParameter(value); } <<TypeName>>.get_jsonpCallbackParameter = function() { /// <value type=""String"">Specifies the parameter name that contains the callback function name for a JSONP request.</value> return <<TypeName>>._staticInstance.get_jsonpCallbackParameter(); } <<TypeName>>.set_path(""/<<URL>>""); "; string strURL = Utils.GetURLWithoutAsmx(Type); strScript.AppendLine(strStatic.Replace("<<NameSpace>>", Type.Namespace).Replace("<<TypeName>>", Type.FullName.Replace("+", "_")).Replace("<<URL>>", strURL + ".asmx")); foreach (System.Reflection.MethodInfo miMethod in Type.GetMethods(BindingFlags.Public | BindingFlags.Instance)) { if (miMethod.GetCustomAttributes(typeof(WebMethodAttribute), true).Length > 0) { List<string> lstParametersJSFunction = new List<string>(); List<string> lstParametersXmlComments = new List<string>(); foreach (System.Reflection.ParameterInfo piParam in miMethod.GetParameters()) { if (!piParam.IsRetval) { lstParametersJSFunction.Add(piParam.Name); lstParametersXmlComments.Add(" /// <param name=\"" + piParam.Name + "\" type=\"" + piParam.ParameterType.Name + "\" optional=\"" + (piParam.IsOptional ? "true" : "false") + "\" mayBeNull=\"" + (piParam.ParameterType.IsValueType ? "true" : "false") + "\"></param>"); } } if (lstParametersJSFunction.Count > 0) { lstParametersJSFunction.Add(""); } string strMethodStatic = @" <<TypeName>>.<<MethodName>>= function(" + lstParametersJSFunction.ToDelimitedstring(", ") + @"onSuccess,onFailed,userContext) { " + lstParametersXmlComments.ToDelimitedstring("\n") + @" /// <param name=""succeededCallback"" type=""Function"" optional=""true"" mayBeNull=""true""></param> /// <param name=""failedCallback"" type=""Function"" optional=""true"" mayBeNull=""true""></param> /// <param name=""userContext"" optional=""true"" mayBeNull=""true""></param> <<TypeName>>._staticInstance.<<MethodName>>(" + lstParametersJSFunction.ToDelimitedstring(", ") + @" onSuccess,onFailed,userContext); }"; strScript.AppendLine(strMethodStatic.Replace("<<NameSpace>>", Type.Namespace).Replace("<<TypeName>>", Type.FullName.Replace("+", "_")).Replace("<<MethodName>>", miMethod.Name)); } } return strScript.ToString(); } Type type; public WebserviceJSGenerator(Type type) { this.type = type; } } }  

Changing the OS on a remote server with no physical access

So this is a rather odd thing to want to do but actually with VPS providers having a limited range of pre-installed OSes available - which are not always the ones we are familar with, perhaps its not that odd after all.


So I have a number of VPS servers with CentOS installed.
Now I'm sure Centos is fine for a lot of people but personally I just can't get on with it.

I am a .NET developer, which means I need mono.
The version of mono which ships with CentOS 5 is just too far out of date for my needs now.
And CentOS 6 doesn't seem to have mono at all?!?

So after much battling with trying to compile applications from source, trying to add redhat repositories and install mono from there, trying a number of different solutions I finally gave up and decided to attempt to switch the OS to openSUSE, which I am familar with and I know has an up-to-date mono available with all the bells and whistles that I require.

So now the big question is - how to switch the OS on a remote (other side of world) server with no physical access?

Again I have tried a number of methods most of which ended in a dead-end but the solution I settled on seems to be quite functional, while not ideal.

IMPORTANT: before doing any of the things mentioned in this article please, please, (I cannot stress this enough) PLEASE! do a backup!

Fortunately my VPS provider has a handy "re-install" OS option in case things went spectacularly bad, but I hope never to have to use it.

Also I was working on a new VPS with nothing actually running on it yet, so if I had to delete the VPS and make a new one - it wouldn't be the end of the world.

I strongly recommend that you do NOT attempt any of the following on anything that even remotely resembles a live production server.


If you choose to ignore this warning - don't come running to me!

OK with that (somewhat verbose) disclaimer out of the way - let's get on with it!

N.B. For all the following I wil assume that you are familiar with YAST

So SUSE has this handy feature in YAST named "install to directory".

Sadly I couldn't find any way to run yast from within CentOS.

So you need to have an existing installation of SUSE somewhere, on some other server or computer.

So the first few steps are run from an existing installation of SUSE, somewhere.
Optionally you can skip to Step 11.

1. Open yast and install the package "yast2-dirinstall"
2. Close yast
3. Open yast again (this is necassary to load the new module)
4. Under software you should now have a new entry "Installation into directory" - open this.
5. Set the target directory
6. Turn OFF the option to run YAST and SUSEconfig.
7. Create Image: No (actually I had this on, but it didn't seem to do anything for me?)
8. Set the software.
    At this point I recommend having a very minimal install to reduce the size of the files to be copied onto the server to be changed.
    With a minimal software selection the installed size is almost 900MB (SUSE 12.1)
    Also I think it is unlikely that you will be able to get a working X install (although I have not tried) with this method.
    You will however need at least the following packages:

• bash
• grep
• sed
• yast2
• vim (or your favourite command line editor)
• zypper
• iputils
     that is a nice short list, but of course they pull in a thousand dependencies...

9. Let that run for a while then quit YAST.
10. Package the installed to directory into an archive and copy the archive to the server to be changed.
11. You can optionally skip steps 1-10. By downloading the image I have created here: http://dev.dj-djl.com/TinySuse12.1.tgz

The following steps should be performed on the server to be changed.

1. Unpack the archive somewhere sensible on your system.
   Do NOT unpack it to the root directory!
2. Use your favorite editor to create a short script which will open the suse environment in a chroot session:
3. user@centos# vim runsusechrooted.sh

   #!/bin/bash echo "mounting dev"
sudo mount -o bind /dev tinysuse/dev
echo "mounting proc"
sudo mount -o bind /proc tinysuse/proc
echo "mounting sys"
sudo mount -o bind /sys tinysuse/sys

echo "chrooting"
sudo chroot tinysuse
echo "chroot environment closed."
echo "umounting dev"
sudo umount tinysuse/dev
echo "unmounting proc"
sudo umount tinysuse/proc
echo "unmounting sys"
sudo umount tinysuse/sys
   

   

4. set the script as executable
   chmod +x runsusechrooted.sh 
5. run the script (your system may require you to run this as root):
   user@centos# ./runsusechrooted.sh
mounting dev
mounting proc
mounting sys
chrooting
suse:/ #
6. If you get this far then you now have a successful SUSE environment running inside the parent OS.
   Now the fun part begins!
   
   At this point in the process I, at first, attempted to get the VPS to boot from the suse system, with not much luck.
   I tried using yast within the suse system to overwrite the bootloader on the VPS.
   I tried editing the grub menu from the host centos system.
   Nothing that I tried would result in the VPS booting the suse system, it always booted CentOS. Unfortunately I can't even see the grub screen to see what errors may or may not be there. My suspicion is that the VPS provider is somehow skipping the bootloader and going straight into a predefined kernel.
   If anyone knows a way to rig it so that the VPS can boot from the new system it would save on all the faff that follows!
   The first thing is we need to get a working internet connection. so:
7. Exit out of the chroot environment,copy the resolv.conf from the host, then go back into the chrooted environment:
   suse:/ # exit
user@centos# cp /etc/resolv.conf tinysuse/etc/resolv.conf
cp: overwrite `tinysuse/etc/resolv.conf'? y
user@centos#  ./runsusechrooted.sh
suse:/ # ping -c 1 www.google.com
PING www.l.google.com (74.125.227.51) 56(84) bytes of data.
64 bytes from dfw06s06-in-f19.1e100.net (74.125.227.51): icmp_seq=1 ttl=56 time=1.83 ms

--- www.l.google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.834/1.834/1.834/0.000 ms
8. N.B. If you think your resolv.conf might change, you might want to try hard-linking it. I've not tested this so I have no idea how well it may work? 
9.  Now we need to install any additional software that may be required:
10. suse:/ # yast
11. First open the Software->Software Repositories section and add at least the standard SUSE OSS repository by using the community repositories option. you may want to add other repos at this point too.
12. Once you have done this you can install software using either yast or zipper.

So now we have a working environment, but what about services?
Thankfully I've devised a cunning hack to get services running in suse too.

1. First make sure you don't have a conflicting service running in the host environment, I chose to completely uninstall  apache from the host to ensure there are no conflicts.
2. user@centos# yum remove httpd
3. Next configure the service in suse as normal (I used yast for the initial config, but will manually edit the config files later to suit my setup)
4. finally we need a script to start the services.
   suse:/ # vim chrootboot.sh
#!/bin/bash
NEW=$1
OLD=$( cat /runlevel )
 [ "a${OLD}a" == "aa" ] && OLD=0
IGNORE="network halt reboot"
echo Changing runlevel from $OLD to $NEW

        START=$(diff <(ls /etc/init.d/rc${OLD}.d -1 | grep "^S" | sed "s/^S[0-9]*//" |sort ) <(ls /etc/init.d/rc${NEW}.d -1 | grep "^S" | sed "s/^S[0-9]*//" | sort ) | grep "^>" | sed "s/^> //")

        STOP=$( diff <(ls /etc/init.d/rc${OLD}.d -1 | grep "^K" | sed "s/^K[0-9]*//" |sort ) <(ls /etc/init.d/rc${NEW}.d -1 | grep "^S" | sed "s/^S[0-9]*//" | sort ) | grep "^<" | sed "s/^< //")

echo Stopping services in runlevel $OLD

        echo ${STOP} | grep -v "^ *$" | sed "s/ /\n/g" |  grep -v "$(echo $IGNORE | sed "s/ /\\\|/g")" |  sed -e "s#^.*\$#ls -1 /etc/rc.d/rc${OLD}.d/K*\0#" | bash | sort | sed -e "s#^/etc/rc.d/rc${OLD}.d/K[0-9]*\([^0-9].*\)\$#/sbin/service \1 stop#"  | bash

echo Starting Services in runlevel $1

        echo ${START} | grep -v "^ *$" | sed "s/ /\n/g" |  grep -v "$(echo $IGNORE | sed "s/ /\\\|/g")" | sed -e "s#^.*\$#ls -1 /etc/rc.d/rc${NEW}.d/S*\0#" | bash | sort | sed -e "s#^/etc/rc.d/rc${NEW}.d/S[0-9]*\([^0-9].*\)\$#/sbin/service \1 start#"  | bash

echo $1 > /runlevel
   N.B. You may want to disable some services to avoid them erroring when they try to start.
5. Now if you exit from the chroot environment at this point you will notice that the unmounting fails - this is because the services remain running (this is good!)
6. But - if you reboot the host, you will also notice that the services are not started again.
7. So now we need a way to start the chrooted services on system boot, and close them cleanly when we reboot.
8. For this we will create a new host service.
   user@centos# cd /etc/rc.d/init.d
user@centos# vim susechroot

   #!/bin/bash . /etc/init.d/functions #echo -e "\n\n $(date +%Y-%m-%dT%H%M%S ) $0 $*" >> /susechroot.log case $1 in start)  mount  -o bind /dev /tinysuse/dev         mount -o bind /proc /tinysuse/proc         mount -o bind /sys /tinysuse/sys         mount none -t  devpts /tinysuse/dev/pts         chroot /tinysuse /chrootboot.sh 5         touch /var/lock/subsys/susechroot ;; stop)   chroot /tinysuse /chrootboot.sh 0         sleep 1         umount /tinysuse/dev         umount /tinysuse/proc         umount /tinysuse/sys         umount /tinysuse/dev/pts         rm -f /var/lock/subsys/susechroot ;; status) chroot /tinysuse service --status-all ;; esac

   cd ../rc3.d
ln -s ../init.d/susechroot S99susechroot
cd ../rc5.d
ln -s ../init.d/susechroot S99susechroot
cd ../rc6.d
ln -s ../init.d/susechroot K0susechroot
cd ../rc0.d
ln -s ../init.d/susechroot K0susechroot
/sbin/service susechroot start

eh Voila!

Now whenever we reboot the server it executes the scripts for runlevel 5 in the suse environment on startup.
So if we have, e.g. Apache installed in the suse environment and configured for runlevel 5. We can reboot the server and apache will be started with no manual intervention.

Now if anyone can work out how to use Suse's init system, rather than my custom script - please let me know!

Also just a note that it appears SUSE's and CentOS's init systems are actually quite different.
In SUSE the kill scripts live in the same runlevel directory as the Start scripts, but are only executed when there is no matching Start script in the target runlevel.

In CentOS the kill scripts live in all runlevel directories which do not have a start script.

This confusion delayed me in coming up with the above solution.

Solving "Could not load file or assembly" problems on mono

So once again I have been tearing my hair out for hours trying to solve this problem.
Specifically in my case it was mod-mono-server2.exe that was producing this error. Now I *know* that that file exists - I checked, in fact that was the assembly I was executing.
So it must be one of the dependencies - but which one?
It turns out there IS a way to find out.
By setting 2 environment variables before attempting to run the application you can get some debugging information which tells you exactly what files mono was trying to find when it errored.

The 2 variables in question are:
MONO_LOG_LEVEL=info MONO_LOG_MASK=asm
So if you run
MONO_LOG_LEVEL=info MONO_LOG_MASK=asm mono myapp.exe
You will get a stack load of debugging info on the console, and just before the exception you should see some lines which looks something similar to this:
Mono-INFO: Assembly Loader probing location: '/usr/lib64/mono/gac/mod-mono-server2/2.10.0.0__0738eb9f132ed756/mod-mono-server2.dll'. Mono-INFO: Assembly Loader probing location: '/usr/lib64/mod-mono-server2.dll'. Mono-INFO: Assembly Loader probing location: '/usr/lib64/mono/gac/mod-mono-server2/2.10.0.0__0738eb9f132ed756/mod-mono-server2.exe'. Mono-INFO: Assembly Loader probing location: '/usr/lib64/mod-mono-server2.exe'. Handling exception type FileNotFoundException
Bingo! It's looking for an assembly called "mod-mono-server2"
But wait a minute! That's the assembly I'm running!

And no - I have no idea why it can't find it when I've actually given it the path to look in!
But since I can see where it is looking a quick symbolic link solves my issue:

ln -s "/usr/local/lib/mono/2.0/mod-mono-server2.exe" /usr/lib64/
This is probably a bad idea, there must be a good reason why it can't find it in the correct location.

Now I'm getting a different error about non-blocking sockets.

*sigh*

Installing MONO on CENTOS

I've spent literally hours tearing my hair out trying to get this sorted so I thought I would share what I have found here:

Most of the search results on google are out of date and/or contain links which no longer work.

I also tried compiling everything from source but ran into trouble here too.

Eventually after much headscratching I found a solution :)

Take a look at this page: http://fedoraproject.org/wiki/EPEL

Ignore that this is a fedora project as it works just fine on centos.

Part way down the page under the "How can I use these extra packages?" heading are a couple of links. Choose the one that is appropriate to your version of centos. e.g. I am using centos 6 so I used the "newest version of 'epel-release' for EL6" link.

Download the rpm file and install it.
You should probably verify the rpm before installing, but I couldn't work out how and I was loosing patience by this point!

rpm -i epel-release-*.noarch.rpm
Now you should be able to search for and install the mono packages (and others) using yum.

e.g.
yum install mono-core mono-data mono-devel mono-extras mono-web mono-winforms monodoc mono-web-devel
N.B. there is no monodevelop included in this lot.
And I can't find anywhere to get it either :(

Fixing: Msg 15137, Level 16, State 1, Procedure sp_xp_cmdshell_proxy_account, Line 1 An error occurred during the execution of sp_xp_cmdshell_proxy_account. Possible reasons: the provided account was invalid or the '##xp_cmdshell_proxy_account##' credential could not be created. Error code: '0'.

This one has been driving my crazy for ages. There are plenty of blog posts out there explaining how to fix this one by running SSMS as an administrator but in my case this did not help. The problem I was having was actually a really subtle one - getting the username right. In all other contexts I am able to use 'ip.add.re.ss\username' Not here so to fix the error in my case I had to use 'servername\username' Hopefully this post will save someone else a major headache!

Saving open files remotely (kate)

So I neglected to save a file on my home pc before leaving. Then at work I wanted to continue editing the same file. My home desktop was not configured for VNC so what to do? I investiagted attaching a vnc server to a running desktop, without access to that desktop but quickly came to the conclusion that it would just be a massive security flaw and hence not allowed. I was just about to give up when I remembered dbus With an ssh session open I was able to tell the instance of kate running on the home pc to save the file without ever seeing the desktop. And it's actually really easy! First you need to get the right display - for 99% of cases that will be display 0 export DISPLAY=:0 now you need to find the running instance of kate qdbus | grep kate hopefully there is only one Now identify the correct document (N.B. replace the number below with the once found from the previous statement) qdbus org.kde.kate-28399 | grep Document If there is only one document open then its easy, otherwise you need to go through each document and see if its the right one. Unfortunatly there doesn't appear to be a way to get the filename, so you'll have to inspect the content qdbus org.kde.kate-28399 /Kate/Document/1 org.kde.KTextEditor.Document.text Repeat this for all the documents until you find the one you want then issue: qdbus org.kde.kate-28399 /Kate/Document/13 org.kde.KTextEditor.Document.save If you get the response true then the document is now saved! I did investigate whether this same method could be used to attach a vnc server using krfb but (thankfully) it can't This method almost certainly can be adapted to other applications and editors though - as long as they support dbus

Why I hate Imports/using

I'm sure other people must find this annoying, not just me.
And even Microsoft do it on MSDN.

What am I on about?

Posting code that requires importing a namespace - without specifying what namespace(s) need to be imported.

Imagine the following code:

Stream stMyStream = new FileStream("SomeFile.txt", FileMode.Open);

Now I know, and you know, that Stream exists within System.IO
but without a using directive at the top of your file, the compiler doesn't know this and it gives you a whole bunch of red squiggly lines.

In the above case it is obvious which namespace you need, but that isn't always so, and you can be left scratchign your head for ages trying to track down which namespace you are missing.

So please people: when you post code example - either use a fully qualified type name, or specify the appropriate namespaces to be used.
e.g.

System.IO.Stream stMyStream = new System.IO.FileStream("SomeFile.txt", System.IO.FileMode.Open);

Just a note to anyone who doesn't know:
the C# directive
using System.IO;
is equivalant to the VB statement
Imports System.IO


To add further confusion both VB and C# have a using statement with is to do with scoping and disposing of variables.


I use both C# and VB so I tend to interchange terms quite a lot!

</Rant>

Cheetsheet

This is my "cheetsheet" or reference card.
I'll update this post as and when but really its just a place to store the stuff I always forget how to do or the exact syntax involved.

Bash:

For:

for f in dir/file*  do    echo "Processing $f"    # do something on $f  done
for i in {0..10..2}    do echo "Welcome $i times"  done

Ascii dec/hex/binary table

Key:

decimal numbers (base 10) hexadecimal numbers (base 16) ascii character or control code binary digits (base 2). The top table has binary 0 as the first digit, the bottom table has binary 1
The binary numbers across the top are the next most significant digits
The binary numbers down the left side are the final 4 digits
so for example: 182 (B6) has a 1 for the first digit as its in the bottom table, Next look to the top of the column to get 011
and finally look at the row to get 0110
So the entire sequence is 10110110

 

0000

0001

0010

0011

0100

0101

0110

0111

1000

1001

1010

1011

1100

1101

1110

1111

000

000�

101

202

303

404

505

606

707

808

909

100A

110B

120C

130D

140E

150F

001

1610

1711

1812

1913

2014

2115

2216

2317

2418

2519

261A

271B

281C

291D

301E

311F

010

3220

3321!

3422"

3523#

3624$

3725%

3826&

3927'

4028(

4129)

422A*

432B+

442C,

452D-

462E.

472F/

011

48300

49311

50322

51333

52344

53355

54366

55377

56388

57399

583A:

593B;

603C<

613D=

623E>

633F?

100

6440@

6541A

6642B

6743C

6844D

6945E

7046F

7147G

7248H

7349I

744AJ

754BK

764CL

774DM

784EN

794FO

101

8050P

8151Q

8252R

8353S

8454T

8555U

8656V

8757W

8858X

8959Y

905AZ

915B[

925C\

935D]

945E^

955F_

110

9660`

9761a

9862b

9963c

10064d

10165e

10266f

10367g

10468h

10569i

1066Aj

1076Bk

1086Cl

1096Dm

1106En

1116Fo

111

11270p

11371q

11472r

11573s

11674t

11775u

11876v

11977w

12078x

12179y

1227Az

1237B{

1247C|

1257D}

1267E~

1277F

 

0000

0001

0010

0011

0100

0101

0110

0111

1000

1001

1010

1011

1100

1101

1110

1111

12880€

12981

13082‚

13183ƒ

13284„

13385…

13486†

13587‡

13688ˆ

13789‰

1388AŠ

1398B‹

1408CŒ

1418D

1428EŽ

1438F

14490

14591‘

14692’

14793“

14894”

14995•

15096–

15197—

15298˜

15399™

1549Aš

1559B›

1569Cœ

1579D

1589Ež

1599FŸ

160A0 

161A1¡

162A2¢

163A3£

164A4¤

165A5¥

166A6¦

167A7§

168A8¨

169A9©

170AAª

171AB«

172AC¬

173AD­

174AE®

175AF¯

176B0°

177B1±

178B2²

179B3³

180B4´

181B5µ

182B6¶

183B7·

184B8¸

185B9¹

186BAº

187BB»

188BC¼

189BD½

190BE¾

191BF¿

192C0À

193C1Á

194C2Â

195C3Ã

196C4Ä

197C5Å

198C6Æ

199C7Ç

200C8È

201C9É

202CAÊ

203CBË

204CCÌ

205CDÍ

206CEÎ

207CFÏ

208D0Ð

209D1Ñ

210D2Ò

211D3Ó

212D4Ô

213D5Õ

214D6Ö

215D7×

216D8Ø

217D9Ù

218DAÚ

219DBÛ

220DCÜ

221DDÝ

222DEÞ

223DFß

224E0à

225E1á

226E2â

227E3ã

228E4ä

229E5å

230E6æ

231E7ç

232E8è

233E9é

234EAê

235EBë

236ECì

237EDí

238EEî

239EFï

240F0ð

241F1ñ

242F2ò

243F3ó

244F4ô

245F5õ

246F6ö

247F7÷

248F8ø

249F9ù

250FAú

251FBû

252FCü

253FDý

254FEþ

255FFÿ

For a more complete Ascii table, look here

Binding a datagridview to a list

Ok, so binding a grid to a list is easy, but what if you want changes in the list to be shown in the grid?
Well actually, it turns out, this is easy too - just not obvious.

Instead of using System.Collections.Generic.List<T>
use System.ComponentModel.BindingList<T>
It's exactly the same in all important regards, but it has the methods necessary for the grid to know that the list has changed

Easy huh!

Editing xml in vim

Yes, I use vim!

And large html, xml and xsl documents can be frustrating so here are a few things I sussed out today to make it easier:

1. None of the commandline tools I could find were capable of cleanly formatting an xml document and putting comments in the correct alignment. All of them put the comments with no indentation.
   Some of you may prefer this but it stosp the following tips from working correctly in vim
   So I wrote a quick 5 min tool to do this, and here it is.
   Extract that archive somewhere into your path and then simply cat unformatted.xml | xmltidy > formatted.xml
   easy :)
   N.B. You'll need to have mono installed.
2. Next I've assigned this new tool to an easy key in vim
   In your .vimrc file enter the following:
   :map <F2> :1,$!xmltidy<Enter>
   You can use any key you like of course
3. Next I enabled code-folding
   Now there are plenty of posts out on the web about getting code folding with xml to work however I had issues with all of them.
   What I found to work was to enable folding by indentation (hence why comments have to be in the correct alignment).
   These 2 lines of vimrc sorted that out for me:
   au FileType htm,html,xml.xsl,xslt exe ":set foldmethod=indent"
au BufNewFile,BufRead *.htm,*.html,*.xml,*.xsl,*.xslt exe ":set shiftwidth=2"
   N.B. I have my tabstop set to 2 so that pressing the tabkey indents the line by 2 columns instead of the usual 
4.  Finally; if, like me, you don't like that ugly banner on the code folding lines then you'll want to change the colours with this line:
   :highlight Folded ctermbg=black

Having enabled the code folding you can use zo and zc to open and close a fold respectively.
N.B. when folding it folds everything at that level, not everything that is a child.
In other words if you had this document:
<xml>   <node1>     <node1a/>     <node1b/>   </node1>   <node2>     <node2a/>     <node2b/>   </node2> </xml>
and you press zc when the cursor is on the line with node1 then rather than hiding node1a and node1b like you might expect - it hides node1 and node2 (as well as their children)
 

Distinct in XSL

So I frequently have a need to do this but I can never remember how, so I figure why not use my shiny new blog and write about it here!

The problem:

• You have some node list which contains many, many items.
• You want to group items based on part of their content
• You want to do it in xsl

Example:
The xml:
<phrases>   <node type="Greeting">Hello</node>   <node type="Farewell">Goodbye</node>   <node type="Greeting">Hi</node>   <node type="Greeting">Howdy</node>   <node type="Farewell">See you later</node> </phrases>
the result:
Greeting Farewell
The solution:
You can use the generate-id() function.
This function is part of xsl so you don't need any funky includes, but it means you can't use it from XPath (e.g. from C#'s SelectNodes() method)

<xsl:if test="generate-id(.) = generate-id(../node[@type=current()/@type])>   <xsl:value-of select="."/> </xsl:if>

And that's it!
Let's break it down

• generate-id(.)
  Does exactly what you might expect i.e. it generates a unique ID which represents the current node. Calling generate-id on the same node twice will return the same id.
• ../node[@type=current()/@type]
  This bit gets all the other nodes which have the same value based on the key we are grouping by (in this case

Visit my webites

As well as this blog I have 2 main websites currently.

• My DJing website; where you can download plenty of my mixes:
  http://www.dj-djl.com 

• My development website which lists the personal software projects I have developed
  http://dev.dj-djl.com

I've also got a new site, still under construction dedicated to the radio switcher project which has been my main focus of late.
http://dev.dj-djl.com/radioswitcher.aspx

When this project goes live I will probably give it its own domain name

Welcome to my blog

I've just created this blog today so bare with me while the content slowly builds!
I guess the idea behind this blog is to share some of the cool things I've done or discovered in the development world.
Hopefully someone somewhere will find it useful!