Pragmatic API design

In a comment on a previous post, Carfield Yim suggests an alternative style for API requests. We could dub this the URL-encoded style, since the content type associated with the request format is application/x-www-form-urlencoded. (Carfield's example involves updating a user's name, so the hypothetical service ought to require that the request be submitted using HTTP POST method rather than GET, since the action is unsafe.)

There is eternal debate in the software and web communities on the question of what APIs should look like, involving an acronym soup of terms such as SOAP, REST, SOA, POX and ROA. Without invoking too many buzzwords, we can observe that the following are both desirable properties of real-world APIs.

1. Support for invoking an operation, and viewing the response, directly in a browser. Typically the requests would need to be submitted using an HTML form. This can be very useful to a developer trying to make use of the API, since it becomes incredibly easy to test its behavior. Learning by experiment is often easier and more effective than reading a specification.
2. Reuse of existing data formats. For example, one of the examples in the original post was concerned with creating a user account in EKP. The body of the request message contained CSV data in the format EKP expects for files uploaded using the User Data Loader. Hence, if an organization has already written code to generate data in that format (extracted from their HR system for example), it needs relatively little work to switch between either generating files for manual upload, or submitting the data directly using the API. On the other hand, the application/x-www-form-urlencoded content type isn't normally used outside of HTTP requests.

(In passing, it's worth noting that SOAP does not satisfy either of these requirements particularly well. It fails completely with the first. As for the second, it's reasonably easy to reuse an existing format if it is XML-based and literal encoding is used rather than SOAP encoding. On the other hand, using a format that is not XML-based—because it predates XML for example—typically requires sending the data as some kind of SOAP attachment, which raises a whole other set of issues.)

In some cases, support for batch operations might also be a requirement. For example, if I need to update 1,000 user records, this will be considerably more efficient if the data can be sent in a single request. If 1,000 separate requests are required, then network latency will become a serious concern. (However, HTTP pipelining could help if supported.)

As I see it, the URL-encoded request style meets the first requirement (it supports invocation from a browser), but fails to meet the second (it fails to reuse an existing data format for the request) and third (it is not well suited to batch operations). On the other hand, the CSV request style described in the original post fails to meet the first requirement, but meets the second and third. This seems like a reasonable trade-off in this case. However, the URL-encoded style seems like a reasonable approach in cases where support for large batch updates isn't a critical requirement, and there isn't an existing data format that obviously matches the desired content of the request message.

Code samples: invoking APIs using Java

Following on from my previous post, I thought it might be useful to post some sample code that demonstrates how to invoke EKP's APIs, in this case using Java.

Every API request must include an HTTP basic authentication header. If you are using Java's built-in HttpURLConnection class, you would need to construct and add this header manually. This is not hard, and the EKP APIs and Web Services Overview explains how to do it. However, it's even easier to use a library that supports HTTP basic authentication directly.

One such library is Jakarta Commons HttpClient. HttpClient is free, open source, and distributed under version 2.0 of the Apache License, meaning that it can be used with minimal obligations in software that is not itself open source. (Note that HttpClient has dependencies on both Commons Logging and Commons Codec, so you will also need both of those to run this sample code.)

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.StringRequestEntity;

We begin by creating an instance of HttpClient, which is straightforward.

HttpClient httpClient = new HttpClient();

We need to initialize the HttpClient object with the appropriate user name and password. The user name is ignored by EKP, so we simply use the empty string. The password must match the value of the authentication.key property configured in the file WEB-INF/conf/ekp.properties.

String userName = "";
String key = "mysecretkey";
httpClient.getParams().setAuthenticationPreemptive(true);
httpClient.getState().setCredentials(AuthScope.ANY,
        new UsernamePasswordCredentials(userName, key));

As described in the previous post, we can create a user account (here, joestudent) by sending an HTTP POST request to /ekp/contentHandler/usersCsv, and including CSV-formatted data in the body of the request.

PostMethod postMethod1 = new PostMethod(
        "https://ekp.example.com/ekp/contentHandler/usersCsv");
String content1 = "Action,UserID,Password,FamilyName,GivenName\r\n"
                + "A,joestudent,dummypass,Student,Joe\r\n";
postMethod1.setRequestEntity(new StringRequestEntity(content1,
        "text/plain", "UTF-8"));
httpClient.executeMethod(postMethod1);

Similarly, we can enroll joestudent in a course with ID Derivatives_101 by sending an HTTP POST request to /ekp/enrollmentHandler, again with appropriately-formatted CSV data in the body of the request.

PostMethod postMethod2 = new PostMethod(
        "https://ekp.example.com/ekp/enrollmentHandler");
String content2 = "USERID,LEARNINGID\r\n"
                + "joestudent,Derivatives_101\r\n";
postMethod2.setRequestEntity(new StringRequestEntity(content2,
        "text/plain", "UTF-8"));
httpClient.executeMethod(postMethod2);

API 101

We just finished a fairly intensive training session here in Hong Kong with a couple of partners. The subject was ostensibly our customization toolkit, but we also spent quite a bit of time discussing APIs and how EKP can be integrated with existing web sites, portals and applications. Every time we conduct these training sessions, this topic expands.

Here's an outline of what we covered, with links to the relevant documentation.

1. The first step in seamlessly combining EKP functionality with an existing site is understanding how to implement web-based single sign-on.
2. With a single sign-on mechanism in place, it's possible to have external pages link directly to specific pages in EKP, and, in particular, to launch courses directly from external pages. (See also: How can I link directly to a page in EKP without losing the navigation frames?)
3. Data can be manipulated in, or extracted from, EKP using the APIs described in the EKP APIs and Web Services Overview. Examples:
   • To ensure that EKP has a user account for a specific user ID (as part of the single sign-on process for example), have the external site send a direct server-to-server HTTP POST request to /ekp/contentHandler/usersCsv, with CSV-formatted data (following the User Data Loader format) in the body of the request. Note that most of the columns are optional when the User Data Loader format is used, so the following is valid for example:

     Action,UserID,Password,FamilyName,GivenName
     A,joestudent,mypassword,Student,Joe

   • To ensure that a specific user is enrolled in a specific course (prior to launching the course for example), have the external site send a direct server-to-server HTTP POST request to /ekp/enrollmentHandler with CSV-formatted data in the body of the request. For example:

     USERID,LEARNINGID
     joestudent,Derivatives_101

   • To retrieve a complete training history, in XML format, for user joestudent (to display within an external page for example), have the external site send a direct server-to-server HTTP GET request to /ekp/contentGenerator/trainingHistoryXml?userId=joestudent.

Update: I posted Java code samples for creating a user account and enrolling a user in a course.

Update 2: Musings on the invocation style of the API operations discussed above, prompted by Carfield's comment.

NoSuchMethodError when upgrading to EKP 5.0

We've had a few reports of the following problem: when connecting to an EKP site that has been upgraded to EKP 5.0, a "Cannot connect to database" page is seen, and a message like the one below appears in the log file (WEB-INF/logs/ekp.log).

java.lang.NoSuchMethodError: javax.xml.parsers.DocumentBuilderFactory.getSchema()Ljavax/xml/validation/Schema;
 at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.<init>(DocumentBuilderImpl.java:110)
 at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(DocumentBuilderFactoryImpl.java:101)
 at com.sun.msv.verifier.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(DocumentBuilderFactoryImpl.java:145)

This appears to happen if EKP is running on an older version of Tomcat 4.1. Older versions of Tomcat 4.1 will override the XML parser libraries that are included in the Java runtime, and these libraries conflict with the versions included in J2SE 5.0, which EKP 5.0 requires.

The simplest solution is to delete the parser libraries that are included with Tomcat 4.1. These can be found in the common/endorsed/ directory under the Tomcat home directory. These libraries are not required when running under J2SE 5.0.

On Safari

I just sat down at my computer and was greeted with this.

I have never installed Apple's Safari browser, but I do have iTunes and QuickTime. Apparently Apple is trying to leverage iTunes and QuickTime to push Safari to as many PCs as possible.

If this is at all successful, Safari's usage share could increase significantly.

Custom login pages

Most EKP sites will use a custom login page rather than the standard page that ships with EKP. This is done by creating a file named login.wm and placing it in the WEB-INF/conf/ directory. The contents of this page are essentially HTML, with some placeholders for dynamic content that use the Velocity Template Language (VTL).

The standard login page, which can be used as a guide, can be extracted from the file WEB-INF/lib/ekp.jar using a zip utility such as WinZip or WinRAR. The location of the page within the ekp.jar file is com/netdimen/tx/auth/login/login.wm for EKP 4.7 and above, or com/netdimen/tx/login/login.wm for EKP 4.6.

One important point that is easy to overlook is that the page should contain the code shown below immediately after the opening <form> tag of the login form.

  #if ($target != "")
  <input type="hidden" name="target" value="$target" />
  #end

This code will ensure that, if a user who is not logged in attempts to access a page that requires authentication (which would happen if the user's session had timed out, for example), EKP will "remember" which page the user was trying to access and forward the user to that page after prompting her to log in. If users are always forwarded to the default start page after logging in, even if they were prompted to log in while trying to access another page, it is most likely because the above code is missing from the login page template.

(Note that the default static index.html page simply redirects the user to the dynamic login page described above. We recommend leaving the index.html page as-is and simply creating a login.wm page as described above, to avoid duplicating the custom login page HTML code.)

If your EKP site is hosted by NetDimensions, these steps will normally be performed for you by our staff.

Directory services and hosted EKP sites

Consider the following scenario, which is not uncommon. An organization wishes to make use of an EKP site that is hosted outside of their network. The organization has already implemented a directory service (Active Directory for example) that is most likely accessible via LDAP. Naturally, the organization would like their users to be able to access EKP using the usernames and passwords stored in the directory service. Below are three ways in which this could be accomplished.

1. Usernames and passwords could be exported from the directory service (using a tool such as CSVDE for Active Directory for example) and imported into EKP. The import could be performed manually using EKP's CSV User Data Loader, or it could be automated using EKP's APIs or import scheduler. Users would log into EKP using EKP's standard login page.
2. The EKP site could be configured to connect directly to the organization's directory service using LDAP to perform authentication. Again, users would log into EKP using EKP's standard login page.
3. A login page could be created on a web server in the organization's DMZ that connects to the organization's directory service and implements an authentication service for EKP as described in this previous post.

The first approach has a few disadvantages.

• It is not real-time. For example, a password change in the directory service will not take effect on the EKP site until the next time the usernames and passwords are synchronized.
• For extra security, systems that support password-based authentication might store a cryptographic hash of a password, from which it is infeasible to recover the original password, instead of storing the original password. (In fact EKP does exactly this, which is why it is not possible to export passwords from EKP.) Thus, it might not be possible to export passwords from the directory service.
• Even if it is technically possible to export passwords from the directory service, replicating passwords on a system outside of the organization's network might violate the organization's security policies.

The second approach avoids these problems. However, it does require that the organization's directory service be accessible from outside the organization's network, which often will not be the case. (This would of course be the preferred approach if the EKP site was hosted inside of the organization's network.)

The third approach has several advantages.

• It is real-time. For example, when a password is changed in the directory service, the new password will immediately be effective for accessing EKP.
• It does not require passwords to be exported from the directory service or replicated outside of the organization's network.
• It does not require that the directory service be accessible outside of the organization's network via LDAP.
• In cases where the organization already has a web site or portal that authenticates against the directory service, the authentication service for EKP can be combined with the existing site, thus providing single sign-on between the existing site and EKP.

This third approach should be the preferred approach in most cases where the EKP site is hosted outside of the organization's network.

Specifications and copyright

IMS Global Learning Consortium: Today the IMS Global Learning Consortium (IMS GLC) announced plans to initiate a pilot project in the distribution of interoperability specifications under a form of Creative Commons license.

There's a murky relationship between technical interoperability specifications and standard copyright. Here's an example.

Every SCORM-conformant content package needs to contain a set of W3C XML Schema documents. Since SCORM content packaging is derived from the IMS Content Packaging specification, one of these required documents is an IMS Content Packaging schema document, which contains an unambiguous IMS GLC copyright notice.

Based on this, it seems one could make a reasonable case that every SCORM-conformant content package is a derivative work, and hence an infringement of IMS' copyright.

Adopting a framework such as Creative Commons seems like a sensible way forward.

SCORM and AICC compared

GerryWaz: Anyone know of a good, dispassionate comparison of the two standards? Strengths and weaknesses of each?

In matters of technology, newer is frequently equated with better. However, while SCORM is the newer specification and is more widely supported, there are valid reasons for using courses and management systems based on the AICC specifications.

Below are some points to consider when deciding between AICC and SCORM. (We consider only SCORM 1.2 and SCORM 2004, since earlier versions of SCORM are effectively obsolete.)

Packaging and deployment

Thanks to SCORM's well-defined packaging model, deploying SCORM courses is often as simple as uploading a zip file. Historically, deploying AICC courses involved more steps due to the lack of a similar packaging model for these courses. In late 2006, AICC published a packaging specification for AICC courses. However, this is still not part of the core AICC specification, so neither AICC-conformant courses nor AICC-conformant LMSs are required to support it. (EKP supports AICC course packages.)

Ease of development

SCORM-conformant courses communicate with an LMS by calling methods of a JavaScript object called the API adapter. AICC-conformant courses communicate with an LMS by sending HTTP messages to the LMS, and interpretting the LMS's responses. (This communication method is known as the HTTP AICC CMI Protocol, or HACP.) Where courses are developed without the aid of specialized course authoring tools, it's usually considerably easier to implement SCORM communication than AICC communication. (However, most specialized course authoring tools shield the developer from these differences.)

Technical requirements

A conformant SCORM API adapter must implement complex rules for validating tracking data and reporting errors and, in the case of SCORM 2004, for evaluating and enforcing sequencing rules and processing navigation requests. Because of this complexity, many robust implementations use Java technology to implement the API adapter, meaning that Java must be available on the user's machine. (Both EKP and ADL's reference implementations use Java to implement the API adapter.) On the other hand, since AICC courses use direct HTTP-based communication, LMSs typically don't require Java to track AICC courses. (However, the courseware itself might still require Java.)

Cross-site scripting

Cross-site scripting occurs when content loaded from one web site in a browser attempts to read or manipulate content loaded from another web site in the same browser. In general, cross-site scripting presents a security risk, and browsers rightly restrict it in most cases. However, these restrictions can also interfere with tracking of courses where the courses reside on a different web site from the LMS. The problem is usually more serious for SCORM courses than for AICC courses. As described above, SCORM courses communicate with the LMS using JavaScript calls, and these are almost always blocked when the course and LMS reside on different sites. On the other hand, due to the nature of AICC communication, AICC courses have greater choice about the specific technology used to perform the communication, and different technologies enforce different policies regarding cross-site communicating. For example, an unsigned Java applet may not communicate with a site other than the one from which it was loaded, but a signed Java applet may do so if the user explicitly grants permission. Even without the use of browser plug-ins such as Java or Flash, an AICC course can implement course-to-LMS communication (but not LMS-to-course communication) simply by using standard HTML forms.

Sequencing

The AICC, SCORM 1.2 and SCORM 2004 specifications all provide sequencing mechanisms—that is, mechanisms for specifying rules concerning the order in which independently trackable units of content can be delivered. (In SCORM terminology, these units of content are called shareable content objects, or SCOs. In AICC terminology, they are known as assignable units, or AUs.) However, only the SCORM 2004 specification requires the LMS to support these mechanisms. On the other hand, it's always possible to implement sequencing in a way that is internal to the SCO or AU. So this is only an issue if you aim to reuse SCOs/AUs at a level of granularity below the course level.

Internationalization

The polite term for the AICC specification's support for internationalization is “boneheaded .” In particular, the AICC specification specifically prohibits the use of Unicode character encodings (e.g. UTF-8, UTF-16), instead requiring the use of one of the ISO 8859 family of encodings. Whereas Unicode encodings support virtually all modern languages, the ISO 8859 encodings support only a very limited set of languages, with Asian languages being particularly poorly supported. (Chinese, Japanese and Korean are all unsupported for example.) There's no good reason for this restriction, and in practice many systems "extend" the AICC specifications to support other encodings. (For example, EKP supports a wide range of encodings beyond the ISO 8859 family, including both UTF-8 and UTF-16.) However, a further problem is that the AICC mechanisms provide no clear way to specify which encoding is being used, so trial and error is often necessary in order to ensure non-English text displays as intended.