10 February 2007

JSON web apps without Ajax

This article first appeared in the December 2006 issue of Visual Systems Journal (VSJ).

In the July/August 2006 issue of VSJ, Mike James pondered how best to design new web applications. Good responsiveness means that more code has to be moved client-side into the browser, making Ajax requests to the server and updating the page on the fly or in response to user actions. However, good design also implies a separation between presentation HTML, presentation code, business logic and the data handling layers, as is possible with various server-side technologies such as ASP.NET.

My idea presented here is to keep the client-side code as separate from the server-side code as possible, and to keep on using the good design practices techniques server-side.

If you generate Ajax requests yourself and update the page DOM, then the data can get out of sync with the server-side representation. For example, in ASP.NET a server-side variable <asp::Label .. /> is rendered as an HTML <SPAN>; if you update the SPAN in JavaScript then the new value is not necessarily present server-side at the next post-back. I tried to resolve this a few months back and got in a terrible mess, so I gave up. Nowadays, for ASP.NET, various frameworks such as Atlas hide these difficulties; essentially you carry on coding server-side as usual and the framework generates the right JavaScript and Ajax behind the scenes to improve responsiveness for your GridView or whatever.

My most recent project required good interactivity on the client-side but also access to good programmability and a database on the server. On the server I decided to stick with what I now know best, ASP.NET and SQL Server. Using stored procedures effectively provided a data access API, as well as improving security. I also used the ASP.NET membership and profile features which provided various levels of abstraction above SQL Server database views, stored procedure and tables. My data tables were keyed off the ASP.NET user information, so my stored procedures looked up some data in the ASP.NET membership tables.

There are two client applications: a specialised photo viewer and photo designer. These did not correspond to any available server controls so there was no quick way to implement these with high responsiveness.

My first reaction was to make Ajax requests to find which photo to show next. However I eventually realised that the client apps do not need to ask the server for this information because it could be included with the page when it is loaded. This makes the pay-load for the page a bit bigger but the pay-off is a better response time because it does not have to query the server - this also reduces the load on the server. This approach also gives better compatibility with some older browsers that do not support Ajax requests.

The design could be summed up as having client applications that were largely separate from the server code. Most pages on the site could be handled by standard ASP.NET code using the full monty of server-side facilities. However the crucial client applications were kept standalone as far as possible - this should also help to achieve the aim of operating standalone off-site or as a mashup on other pages.

I had previously decided that the photo information should be stored on the server in an XML file. Some of this information is also contained in the site database, but I thought that it was important to have all the photo-site data together in one easy-to-use place. As a consequence there is a need to maintain consistency between the database and the XML, eg if a user deletes a photo then the database entry is removed; the XML also needs to be updated.

Client photo viewer app

Anyway, I needed to pass the XML data to the web page so that the photo viewer client app knew what to do. I decided to do this by converting the XML to JSON and passing it to the client app JavaScript. In the codebehind C# for the ASP.NET page, I load the XML and convert it to JSON using the code I described in last month's article. The JSON is passed to the page using a ClientScriptManager RegisterStartupScript call which eventually invokes the page processJSON() JavaScript function. The one trick to this technique is to realise that the JSON is interpreted twice, once when processJSON() is called, and again when the JSON parser is called. To resolve this issue, simply replace each \ with a \\ before sending the JSON string off.

    XmlDocument doc = new XmlDocument();
string JSON = XmlToJSON(doc);
JSON = JSON.Replace(@"\", @"\\");
ClientScriptManager cs = Page.ClientScript;
cs.RegisterStartupScript(GetType(), "Startup", "processJSON('" + JSON + "');", true);

ASP.NET inserts the processJSON() call at the end of the web page so that it is called after almost everything else has been loaded. I'm not sure if this corresponds exactly to the BODY onload event, but it seems to work OK.

The page HTML should include any JavaScript source files that the project requires:

    <script src="json.js" type="text/javascript"></script>
<script src="ClientApp.js" type="text/javascript"></script>

The ClientApp.js should process the JSON as is appropriate for your application:

    var obj;
function processJSON( JSON)
obj = JSON.parseJSON();
if( !obj)
alert("JSON decode error");

Client designer app

The photo viewer client-side app does not need to interact with server-side code (all it does is request new images) although it could be enhanced to log the actions that a user has taken. However the matching designer client-side app does need to interact with the server to store the design that a user is making. I hold any pending user changes in a JavaScript array and report them to the server when the user clicks on the Done button or opts to move to a new photo. There is a visual indication when there are unsaved changes; if the user navigates away then any unsaved changes are lost.

The changes are reported by storing them in JSON format in a hidden form variable called Changes. This code shows how the Done button and Changes are defined in the .aspx file:

    <asp:Button ID="btnDone" Text="Done" OnClick="btnDone_Click" runat="server" />
<input id="Changes" type="hidden" runat="server" />

In case you are not familiar with this syntax, btnDone and Changes are page variables that can used in server-side code. When the ASPX page is rendered, they are converted into HTML that can be accessed using JavaScript and the DOM. Note that when rendered, the DOM ids may be different, eg if the page uses a master page layout.

If the user clicks on Done, I want to store any changes before posting back to the server, so I add an onclick handler to the Done button in the .aspx.cs Page_Load():

btnDone.Attributes["onclick"] = "javascript:return btnDone_OnClick()";

When the page is rendered, btnDone becomes an INPUT form field with its onclick handler set. The JavaScript onclick handler stores the JSON string version of the ChangesDone variable and then returns true to let the page form be submitted:

    function btnDone_OnClick()
ChangesField.value = ChangesDone.toJSONString();
return true; // Don't cancel btnDone form submit

For the above to work, the JavaScript ChangesField variable needs to be set correctly to the Changes hidden field. Remember that Changes runs server-side as well. This means that ASP.NET may have decorated the generated HTML field name with various prefixes, eg if the page uses a master page layout. To get the correct name, my aspx.cs code registers some startup script to tell the JavaScript the correct name for Changes.ClientID. The JavaScript also needs the btnDone.UniqueID so this is passed as well:

    ClientScriptManager cs = Page.ClientScript;
cs.RegisterStartupScript(GetType(), "SetServerControls", "SetServerControls("+
"'" + Changes.ClientID + "','" + btnDone.UniqueID + "');", true);

This is the JavaScript code that is called at startup to store the control names:

    var ChangesField = false;
var btnDoneUniqueId = false;

function SetServerControls( ChangesField_ClientID, _btnDoneUniqueId)
ChangesField = document.getElementById(ChangesField_ClientID);
btnDoneUniqueId = _btnDoneUniqueId;

I mentioned earlier that I also wanted to postback to the server on events other than pressing the Done button. This is achieved in the JavaScript handler for each event by storing the ChangesDone data and then simulating a press of the Done button. This is done by calling the __doPostBack() function that ASP.NET will have generated, passing the required Done button UniqueId, eg:

    function Move_dblclk()
ChangesField.value = ChangesDone.toJSONString();

OK, let's move back to the server and see how the Done click is handled server-side. This simply picks up the Changes value and decodes it using the Nii.JSON.JSONArray parser along the following lines:

    protected void btnDone_Click(object sender, EventArgs e)
string sChanges = Changes.Value;
if (!string.IsNullOrEmpty(sChanges))
JSONArray Changes = new JSONArray(sChanges);

There is one complication to the above process for my designer application. In the viewer, the startup JavaScript can be registered in the Page_Load() method. However the Done button handler btnDone_Click() will be called after Page_Load(). If the Done handler alters the information that you want to send to the client, then you must generate the startup JavaScript later. I found that this could be done in the page prerender function Page_Prerender() as follows:

protected void Page_Prerender(object sender, EventArgs e)
string JSON = XmlToJSON(SiteXmlDocument);
JSON = JSON.Replace(@"\", @"\\");
ClientScriptManager cs = Page.ClientScript;
cs.RegisterStartupScript(GetType(), "SetServerControls", "SetServerControls("+
"'" + Changes.ClientID + "','" + btnDone.UniqueID + "');", true);
cs.RegisterStartupScript(GetType(), "Startup", "processJSON('" + JSON + "');", true);

On the server, I made the viewer and designer into .ascx user controls. Incidentally the standard Visual Studio File in Files did not find text in C# .ascx.cs files even though the filter includes *.cs. I reported this a bug but Microsoft bizarrely said that this was by design; fix it by adding *.ascx.cs to the filter. I also add *.js to find text in JavaScript and *.master to find master page markup.

Here's another JavaScript tip: include a version number in the filename, eg ClientApp_v1_01.js if the code ever changes. If you don't do this, a returning user's browser may use a cached and out-of-date version. When you do a change, rename the file and update all the code that references it.


Whew! But not too bad really. With this tricky code out of the way, I could concentrate on the client and server code separately which I think was a wise choice. Coming back to JavaScript for a large project was not as bad as I thought it might be, as it was more expressive than I had realised and DOM interactions worked well and pretty consistently across most browsers. However I did revert to alert() box debugging. A search found various JavaScript debuggers but I never got round to trying them. For JavaScript, ASP.NET and other technologies, I refer quite often to help sites on the web, so thanks for all the fish.

On the server, separation of the codebehind from the presentation HTML is definitely a good thing. If I am honest however, I have to report that my .aspx files still contain a tiny amount of inline code to access certain page fields or fill GridView TemplateFields. Declarative design can only ever take you so far, so my .aspx.cs code is largely linked to the presentation HTML, eg filling labels or coping with SelectedIndexChanged. I probably do not make enough of an effort to separate the business logic from the presentation logic, although using stored procedures as a data access layer is a good discipline for security as well as design-separation reasons.

04 January 2007

JavaScript Object Notation (JSON)

This article first appeared in the November 2006 issue of Visual Systems Journal (VSJ).

In the July/August 2006 issue of VSJ, Mike James showed the basics of how to make Asynchronous JavaScript and XML (Ajax) calls from a web page to a server using an XMLHttpRequest object. This returns either plain text in responseText or XML in responseXML properties. Unfortunately, XML can be a little cumbersome to use, which isn't going to make your JavaScript any easier to understand. However there is an alternative in the form of JavaScript Object Notation (JSON, pronounced Jason) which is a string representation of structured data using JavaScript's object literal notation, ie name/value pair objects.

Here's some XML:
<?xml version='1.0' encoding='UTF-8'?>
<name>A Bee Co</name>
<contact type='email'>sales@abee.co.uk</contact>
<employee type='Manager'>
Alison Bee
<employee type='worker'>
John Sloop

In JSON the same data might look like this:
{ "business":
{ "name": "A Bee Co",
"contact": {"type": "email", "value": "sales@abee.co.uk" },
"employee": [ {"type": "Manager", "value": "Alison Bee" },
{"type": "worker", "value": "John Sloop" }

If the above JSON string is in a var called Business, then it can be parsed very simply in JavaScript:
    var biz = eval("(" + Business + ")");

The following easy-to-use values are then available:
•  biz.business.name: A Bee Co
•  biz.business.contact.type: email
•  biz.business.employee.length: 2
•  biz.business.employee[0].value: Alison Bee
As you can see, the JSON string has been converted into an object tree. If you remove the superfluous "business" object from the JSON, then the values would be even simpler, eg biz.name.

Note that the multiple "employee" XML elements were translated into a JSON/JavaScript array. If there were just one employee, then biz.business.employee would simply be a property string. To cope with this, I use my ObjectToArray function to convert this property into an array which lets me handle it more simply:

function ObjectToArray( obj)
if( !obj) return new Array(); // zero element array
if( !obj.length) return new Array(obj); // one element array
return obj; // array already
var employees_array = ObjectToArray(biz.business.employee);

This approach works well as long as you know the structure that you are expecting, which will usually be the case. Obviously you will then want to do something with your data, ie show it to the user. Be careful if you are using server-side processing such as ASP.NET because changing a page object will not necessarily change the value seen at the next real postback, or worse, validation errors can occur. Handling interactions between client and server variables can be a messy kettle of fish which I don't want to go into here, but can be solved using Atlas and other Ajax frameworks. In next month's article I suggest a simple client-server scenario which makes use of JSON, XML, JavaScript and ASP.NET2.

Anyway, back at the bit-face, you can find out more about JSON at www.json.org. JSON supports these types of values: string, number, object, array, boolean and null. Strings are in Unicode and various escape sequences are supported. JSON doesn't support binary data so you cannot embed images. JSON seems to be supported by all versions of JavaScript in browsers.

JSON can be used in other languages apart from JavaScript. It's main use seems to be data-interchange, particularly in Ajax requests. Note that Ajax requests are not supported by older browsers. Although ease of use was the original raison d'etre for JSON, it is now also commonly used because it is more compact than XML resulting in reduced data transfer requirements.

Parsing time is also reputedly less for JSON. However you need to be careful using eval() against an untrusted source. The JSON web site provide a file json.js that includes a safer parser string method prototype parseJSON(), so replace the above eval() call with:
var biz = Business.parseJSON();

This still doesn't stop potentially dangerous values appearing as parsed values, so be careful if you stuff values into DOM innerHTML values for example. The JSON JavaScript code also defines toJSONString() to convert JavaScript objects and arrays into JSON.

C# XML to JSON conversion

In a recent web application I decided to store and process data server-side in XML because this is a standard storage type and ASP.NET2 provides good routines for handling an XmlDocument in memory. However I wanted to transfer information to the web page client in JSON, and decode returned information in JSON.

Good old www.json.org provided classes to convert JSON into C# objects at www.json.org/cs.zip. As an example, in JavaScript convert your employee array to a JSON string:

var JSON_employees = biz.business.employee.toJSONString();

Once this is safely server-side, use C# code like this to retrieve the name/value pairs for each employee, slightly laboriously:
    using Nii.JSON;
JSONArray employees = new JSONArray(JSON_employees);
for (int eno = 0; eno < employees.Count; eno++)
JSONObject jemployee = (JSONObject)employees[eno];
for (int ano = 0; ano < jemployee.Count; ano++)
string name = jemployee[ano].ToLower();
string value = jemployee[name].ToString();

I couldn't find a means of converting XML into JSON in C#, so I thought that it must be straight forward to write my own converter. It turned out that there were more XML scenarios than I had originally envisaged. For simple XML elements containing text there seems an obvious translation:
 XML: <xx>yyy</xx>
JSON: "xx":"yyy"

If the XML element has attributes or contains child elements then these are converted into object name/value pairs, with text usually converted into a field called "value". (Life gets complicated if there is an attribute or child elements called "value".)
 XML: <xx w='zzz'><aa>bb</aa>yyy</xx>
JSON: "xx": { "aa":"bb", "w":"zzz", "value":"yyy" }

Duplicate elements are converted into an array:
 XML: <xx><aa>bb</aa><aa>cc</aa></xx>
JSON: "xx": { "aa": ["bb","cc"] }

One example of "equivalent" XML and JSON online cleverly contracted this:
 XML: <employees><employee>bb</employee><employee>cc</employee></employees>
JSON: "employees": ["bb","cc"]

Note that XML and JSON conversions are not necessarily reversible, ie you cannot convert one to the other and back again and guarantee to get the same as you started.

You can get my code online www.phdcc.com/xml2json.htm. The code also makes sure that characters are escaped if necessary for correct representation in JSON.

www.json.org, www.json.org/js.html, www.json.org/cs.zip, www.crockford.com, www.phdcc.com/xml2json.htm