Embracing the Web Part 3: Paradigm Shifts and Maturity Factors in Web-based Application Developmen

Part 1 of this series describes how distributed computing simplifies client and server programming, and Part 2 makes the points that componentization simplifies software plug-n-play, and that enterprise services simplify large-scale software development. Unlike a client/server or an enterprise application, both of which typically support only a small number of users, Web applications must be robust and scalable enough to support thousands or even millions of users. In this article, I'll review the paradigm shifts and maturity factors in Web-based application development.

Related Reading .Net Framework Essentials By Thuan L. Thai, Hoang Lam Table of Contents Index Sample Chapter Read Online--Safari Search this book on Safari:   Only This Book All of Safari Code Fragments only

Web Paradigm Shifts

If you've been following the Internet for the last ten years, you may have noticed that the paradigm has shifted from connectivity to presentation to programmability. In this section, I'll briefly discuss these different shifts, and introduce you to a simple yet extremely practical protocol called SOAP (Simple Object Access Protocol).

Connectivity

Before 1993, network computing was pretty much all about connecting computers together to form local area networks (LAN). Engineers soon realized that they could connect many LANs together to form a wide area network (WAN). Connecting all these WANs together has given us the highly interconnected Internet we have today.

TCP/IP was the most popular protocol for connecting these networks together. Because TCP/IP was the de facto standard, many software vendors developed client and server applications using TCP/IP. This was good because as long as you publish the application-level communication protocol specification, other people may use your services. Examples of published application-level protocols include the Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), Telnet, and finger, all of which are based on TCP/IP. Around 1993, the Hypertext Transfer Protocol (HTTP), which assumes the use of TCP/IP, became extremely popular, and that's when the software industry entered a new era.

Presentation

HTTP was invented especially to transfer HTML documents across the Internet. As you may already know, HTML is a simple syntax for specifying the layout and presentation of the data stored in an HTML document. Abiding by the HTTP specification, any software vendors could develop Web servers and Web browsers. Because the data exchanged between the servers and browsers was in the HTML format, browsers could parse the HTML, format the data in the correct look and feel, and present the thin, graphical user interface to the user.

HTTP and HTML made thin user interfaces possible. HTML allowed you to insert hyperlinks and graphics into the HTML files. Hyperlinks made it easy for the users to surf the Web, and embedded graphics were an eye opener to the user. Not long after the introduction of HTTP and HTML, every company in the world was putting up Web sites, and everyone else was using some sort of Web browser.

Worth noting is that HTTP is a stateless protocol. For each HTTP request, a client connects to the server, retrieves some data from the server, and disconnects from the server. You may think that the required connection and disconnection are expensive, but doing these two things for each request allows for scalability. By making HTTP a stateless protocol, a Web server can support an unlimited number of users, barring hardware constraints.

Programmability

Building Web sites was easy, but building Web applications, such as e-commerce sites, was not so easy, and the stateless nature of HTTP didn't make it easier. As a result, several efforts were put in place to make it easier to develop Web applications. On the server side, software vendors introduced tools that supported dynamic Web page development, including ColdFusion and Active Server Pages (ASP). ASP allowed you to develop Web pages that looked and behaved differently depending on the state or the data of the request. To make ASP even more powerful, developers could use JavaScript or VBScript to add server-side and client-side logic. These scripting languages added programmability to both the Web browser and server.

Check out O'Reilly's new .NET Resource Center for the latest articles and books covering Microsoft's .NET technology.

But ASP and scripting weren't enough, because people wanted to make Web applications function just like a normal Windows application, with rich user interfaces and powerful business logic. Moreover, they wanted to be able to exchange structured data with different B2B (business-to-business) partners, and pass the same data to the browsers. The Extensible Markup Language (XML) was then introduced as a mechanism for electronic data interchange, and it quickly became a standard.

XML is simple to use and extend because the syntax looks similar to HTML. The exception is that you can use any tag you want, as long as you have a beginning tag, such as <car>, and an ending tag, such as </car>. The following is a snippet of valid XML:

<car>
  <vin>VI00000383148374</vin>
  <make>Accura</make>
  <model>Integra</model>
  <year>1995</year>
</car>

As you can see, XML really is simple. This XML snippet can represent a car object that either the Web server or the Web browser is using. Looking at it from a different angle, this XML snippet is structured data that can be shared among different applications.

With the advent of XML, many developers invented their own mechanisms for transferring XML between the Web server and the browsers. Microsoft helped developers by shipping two very important COM classes that started a new breed of Web application development. The programmatic identifiers (ProgID) of these COM classes are Microsoft.XMLHTTP and Microsoft.XMLDOM. The former, Microsoft.XMLHTTP, allows the Web browser to make a Web server request programmatically without leaving the current Web page. During the process, the Web browser can send along any XML buffer that will be deciphered by the target ASP page on the Web server. Microsoft.XMLDOM, on the other hand, allows a programmatic way to assemble and manipulate an XML buffer. Using these two COM objects, anyone can transfer XML data between the browser and the server.

Here's a code snippet showing how to send a request to a Web server and get back an XML buffer. You'll notice that I first create a Microsoft.XMLDOM object, oDOM, to store our XML buffer, which represents a request to get more information on a specific car. I then create a Microsoft.XMLHTTP object, oHTTP, and simply send oDOM to the Web server:

<HTML><BODY>

<SCRIPT>
function GetCar()
{
  // Create a DOM object to store XML.
  var oDOM =  new ActiveXObject("Microsoft.XMLDOM");
  oDOM.loadXML("<GetCar><vin>VI00000383148374</vin></GetCar>");
			
  // Create HTTP object to send XML to server.
  var oHTTP = new ActiveXObject("Microsoft.XMLHTTP");
  oHTTP.open("POST","http://localhost/thuan/GetCar.asp", false);
  oHTTP.send(oDOM);

  // Put the return XML into msg.		
  msg.innerText = oHTTP.ResponseText;
}
</SCRIPT>

<INPUT TYPE="button" Value="GetCar" onclick="GetCar()">
<hr>
<div id="msg"></div>

</BODY></HTML>

The previous code shows that it requests a page called GetCar.asp, which is shown here. This page simply checks to see whether the requested vehicle identification number (VIN) is VI00000383148374. If so, it returns more information to the browser; otherwise it returns an error XML buffer.

<%@LANGUAGE=JScript%>
<%
  var oDOM = Server.CreateObject("Microsoft.XMLDOM");
  oDOM.load(Request);

  var strVIN = oDOM.selectSingleNode("GetCar/vin")

  if (strVIN.text == "VI00000383148374")
  {
    Response.Write("<car><model>Integra</model></car>")
  }
  else
  {
    Response.Write("<error>Invalid car</error>");
  }
%>

The important thing to note is that when the browser sends the request to the server and receives the response, there is no page transition. The first page remains in view and only the data (msg) of the page is modified. This simple example shows that XML is very powerful. XML permeates the .NET platform.

For more in-depth coverage of Microsoft's .NET Framework, visit the O'Reilly Network's .NET DevCenter.

SOAP Overview

Before the introduction of SOAP, competent software shops developed their own SOAP-like protocols using their own custom XML formats and two well-liked COM objects, Microsoft.XMLHTTP and Microsoft.XMLDOM, similar to what I've shown in the previous section. The question is why must you invent this plumbing again and again if you can do it once and be done with it? SOAP is the answer, and it's based on the XML standard. XML allows all kinds of new, simple, and yet powerful concepts, including XML Schema Definition (XSD), Extensible Style Language (XSL), XSL Transformation (XSLT), SOAP, and others. A discussion of all these topics is beyond the scope of this article, so I'll just concentrate on a brief discussion of XSD and the SOAP protocol, since .NET takes these concepts into consideration from the beginning.

XSD

XSD is a specification for defining application-specific, XML-based, data types. Interesting to note is that XSD is expressed entirely in XML. Think of XSD as a way to define an abstract data type, similar to defining a struct in C or a class in C++. The only difference is that you are expressing this definition in XML. For example, given a C++ class as follows:

class CCar {
public:
  string vin;
  string make;
  string model;
  int year;
};

you can express the same type in XML using XSD, as follows:

<schema xmlns="http://www.w3.org/2001/XMLSchema"
 targetNamespace="urn:book:car"
 xmlns:t="urn:book:car">
  <element name="car" type="t:CCar"/>
  <complexType name="CCar>"
    <all>
      <element name="vin" type="string"/>
      <element name="make" type="string"/>
      <element name="model" type="string"/>
      <element name="year" type="int"/>
    </all>
  </complexType>
</schema>

An instance of this XSD can be expressed as follows:

<c:car xmlns:c="urn:book:car">
  <vin>VI00000383148374</vin>
  <make>Accura</make>
  <model>Integra</model>
  <year>1995</year>
</c:car>

Because of the ability to create new application-specific types, XSD is an extremely important specification. For example, SOAP is expressed entirely using SOAP-specific XSDs.

SOAP

The previous XSD example shows that an XSD definition maps directly to a data type, such as a C++ class, and that an XML buffer based on a specific XSD maps directly to an object or a variable of a specific type. So far, you have seen type definitions, but you haven't seen a way to express method invocation. SOAP is an open standard for expressing remote method invocations in XML, without prejudice to any specific computing platforms.

Instead of relying on a heavyweight RPC-based protocol or infrastructure, SOAP specifies a serialization format that will work with any type of transport. For this reason, a SOAP message can be easily transported as part of an HTTP message, making SOAP a viable protocol for the Web. In fact, one of the motivations for SOAP is that an RPC mechanism such as DCOM doesn't work through firewalls and NAT (Network Address Translation) software.

Having said that, let's talk a bit about the format of a SOAP message. A SOAP message is made up of a serialized XML buffer called a SOAP envelope, which is composed of an optional header and a required body. The SOAP header allows you to store protocol extension information. This can be whatever information you like, but typically this is where to insert your own authentication, authorization, or cryptographic information. The SOAP body is important because this is where you store structured XML data pertaining to your method invocation, including the details of your input and output parameters.

Here's an example of a SOAP message that you can use to invoke a method call GetCar. Notice that there is no header because it is optional. Also, notice that the information pertaining to our method invocation is encapsulated inside the SOAP body, as required by the specification.

<SOAP:Envelope 
 xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP:Body>
    <GetCar xmlns="http://tempuri.org/">
      <vin>VI00000383148374</vin>
    </GetCar>
  </SOAP:Body>
</SOAP:Envelope>

If you send this serialized XML buffer to a SOAP-supported platform, you will get back another serialized XML buffer that contains the outputs of the GetCar method. For example, you may get back a SOAP envelope with a body element that contains the XML buffer associated with the results of the method call, as shown here:

<SOAP:Envelope 
 xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP:Body>
    <GetCarResponse xmlns="http://tempuri.org/">
      <car><model>Integra</model></car>
    </GetCarResponse>
  </SOAP:Body>
</SOAP:Envelope>

You'll notice that this example is similar to the earlier example that made use of Microsoft.XMLHTTP and Microsoft.XMLDOM. This is intentional because I wanted you to compare a homegrown version of SOAP against standard SOAP.

Maturity Factors

With only about seven years of Web application development, the software industry has become more mature in architecting and developing Web-based systems. The major lessons learned include:

• Interoperability
• Scalability
• Availability
• Manageability

Let's briefly discuss these.

• Interoperability. Software has to support plug-n-play, so that it can interoperate with other software at a binary level, requiring zero source code. This requires the software to contain metadata about itself so that tools can inspect and pull out type information that can be used to integrate with the software. COM is the predominant technology that supports interoperability, and COM+ component services extend this notion to large-scale, enterprise-wide applications.

---

O'Reilly will soon release (September 2001) COM and .NET Component Services, which provides both traditional COM programmers and new .NET component developers with the information they need to begin developing applications that take full advantage of COM+ services.

---

• Scalability. A Web application may have a hundred users at initiation, but it may have a hundred thousand users two months after initiation. Because of the nature of the Web, Web applications must scale over time. You can scale your applications by either adding more hardware or adding smarter software. In hardware scaling, you can either scale up by adding more CPU and memory, or scale out by adding more physical machines, creating a farm of duplicate machines. You create a farm of machines, so that you can equally load balance the hits that enter your Web site, thus preventing bottlenecks and increasing responsiveness. Doing this permits the application to scale out in order to support a massive number of users as time matures. However, before you can scale out, you must architect your Web application correctly. The key to scalability is simple: design and implement totally stateless Web applications on the Web servers.

• Availability. Scaling to massive numbers of users is not enough. You have to design your applications to be available every minute of the day because users may use your Web applications at any time. You must ensure that your application will have high uptimes and low downtimes. To do this, you must add facilities for failover, in case your application goes down or hardware fails to operate. For example, you can take extra measures to ensure that your software is robust, add intelligent hardware load balancing to not only provide scalability but also failover, and add clustering to your database systems.

• Manageability. In enterprise or Web systems, you often have farms and clusters of machines. Managing and administering a few machines is easy, but managing farms or clusters of them can be very hectic. In an environment with many server machines, you don't travel around to fix problems. Instead, it is important to implement a central point for administering all the servers in the enterprise. Often you have to do this remotely, so a product such as Terminal Services is extremely important.

Summary

In this three-part series, I've taken you from simple distributed computing to large-scale, Web-based computing. I talked about SOAP and how it's a platform-agnostic protocol for distributed computing, thus appropriately catered for the Web. I've also discussed the lessons the software industry has learned from developing large-scale applications.

I have taken you through these concepts because they have all proven to be valuable to the software industry. The principles embodied within these technologies or concepts are important enough for Microsoft to consider all of these ideas. In general, Microsoft has taken these ideas and transformed them into something practical and applicable, a platform called Microsoft .NET.

As this and the previous two articles have shown, the software industry has put great effort into creating an open, Web-based computing platform. Microsoft is a part of this crowd because the .NET platform truly embraces the Web. If you're interested in a technical introduction to the .NET Framework, see .NET Framework Essentials, an O'Reilly book I coauthored with my colleague, Hoang Lam.

Thuan Thai is the author of Learning DCOM and coauthor of .NET Framework Essentials. He has been giving technical presentations on the .NET platform to clients since announcement of the initiative in August 2000.

O'Reilly & Associates recently released (June 2001) .NET Framework Essentials.

• Sample Chapter 6, Web Services, is available free online.

• You can also look at the Table of Contents, the Index, and the Full Description of the book.

• For more information, or to order the book, click here.