Transformations in PostGIS

Lately I've been doing a lot of development centered around PostGIS (http://postgis.refractions.net/) and ran across a problem that has a relatively easy solution (for those well versed in GIS) but finding an answer proved to be somewhat difficult.

Here are the assumptions and the requirements before we look at the problem:

• A point is defined by a latitude and longitude.
• A line is a defined by a series of points.
• A buffer is defined as a shape that envelopes a line. The width of the buffer is specified by the user in meters. The distance value the user supplies is a +/- distance from the supplied line.
• We are assuming WGS 84 as the target map projection.

PostGIS provides a handy buffer function called ST_Buffer and in order to use this function the geometry supplied to the buffer function along with the desired buffer distance need to be in the same coordinate system. See the problem yet? My coordinates are in decimal degrees (i.e., latitude and longitude) and my buffer distance is in meters. So I need to transform the line into a different coordinate system that will allow me represent my geography based shape in metric coordinates. This transformation is non-trivial since converting meters to decimal degrees relies on the use of a map projection because the world isn't flat (sorry!). Before we go any further let's look at the query I've developed so far:

SELECT ST_Buffer(GeomFromText('LINESTRING(-76.543 42.567, -76.012 42.345, -75.890 42.445, -75.543 42.330)'), 300.0) AS buffer_shape

The query is valid but the buffer distance is thought to be in decimal degrees (i.e., WGS84) with respect to PostGIS and therefore produces a shape that pretty much covers the world. I tried guesstimating the conversion from decimal degrees to meters but quickly realized, at best, a guesstimate would be terribly wrong.

To solve this problem - I employed the Transform function from PostGIS. In order to get this function to work properly I had to determine what coordinate system to transform my geometry into...this is where my Google-fu fell relatively short. Luckily I went to http://gis.stackexchange.com and found a reference to the proj4js.org project. Using proj4js I was able to determine the SRID of the desired metric coordinate system (that SRID is 900913). This coordinate system bases coordinates off of meters instead of decimal degrees. So I changed my original query to be the following:

SELECT Transform(ST_Buffer(Transform(GeomFromText('LINESTRING(-76.543 42.567, -76.012 42.345, -75.890 42.445, -75.543 42.330)'), 900913), 300.0), 4326) AS buffer_shape

The above statement transforms the line from WGS 84 to a Mercator projection, performs the buffer operation, and then does another transformation to project the resulting buffer shape back into WGS 84. Voila! Again, nothing earth shattering here but if you don't speak GIS all day then performing the transformations may not be entirely obvious. In my opinion the PostGIS documentation falls short when mentioning the transformation details.

In my last post I provided a small tutorial on how to use the Client-Side Event API within the ASPxGridView component from DevExpress. In this post I would like to cover another snippet that I came up with that may complement my previous tutorial. This snippet assumes that you have multiple selection enabled in the GridView and wish to preserve the selection as the user pages through results.

If Session("pks") IsNot Nothing Then
   Dim selectedPKs As List(Of Object) = 
     CType(Session("pks"), List(Of Object))
   For i As Integer = 0 To selectedPKs.Count - 1
     Dim selectedIdx As Integer = ASPxGridView1.
       FindVisibleIndexByKeyValue(selectedPKs(i))

     If selectedIdx >= ASPxGridView1.VisibleStartIndex And 
       selectedIdx <=   ASPxGridView1.VisibleStartIndex + 
       ASPxGridView1.VisibleRowCount Then         

       ASPxGridView1.FocusedRowIndex = selectedIdx         
       Exit For       
     End If     
    Next i   
End If  

This snippet goes in the Data_Bound event so that it executes each time the user moves to a different page within the grid view.

Command Scripts

Recently I had to create a test that simulated a catastrophic failure to ensure the integrity of an embedded database. Development time for the test was limited so I set out for the quickest approach and saved myself the hassle of mocking test cases for another day. The test would initialize the database and then insert a random number of rows. During the test the OS would kill the process while rows were being inserted and then restart the process to ensure the integrity of the database.

I ended up coming up with a command script in Windows XP that I felt was worth sharing due to its relative obscurity. I will save the majority of the details and only highlight the important points.

First, let's look at the script I made to launch the java process responsible for bootstrapping the database:

START java -cp . Sleeper
ping 1.0.0.0 -n 1 -w 5000 >NUL
FOR /F "tokens=1-2" in ('jps') DO (
IF "%%j" == "Sleeper" (
SET PID=%%i
)
)
TASKKILL /PID %PID%

For me, the coolest part of this script was using the jps command to get the PID for the process that I launched in the first step. My second favorite feature (*cough* hack *cough*) was the use of the ping command (not my idea see the link here). It's the only way I could get the script to wait for a few seconds before killing my process - allowing the test running a separate process to reach the portion of the test where the insertion was occurring. It's not the most reliable or sane approach but it worked perfectly for creating a quick simulation.

The second snippet isn't as exciting as the first but think it's a huge timesaver when trying to execute a Java program that has a very large number of dependencies all located in the same directory. This script scans the directory (e.g., the lib/ folder) and adds every filename ending with .jar to the classpath. Again, keep in mind that this isn't optimal - but it's something quick-and-dirty that you can use to get going:

::EnableDelayedExpansion must be turned on in order to
::programmatically append to the classpath
SETLOCAL EnableDelayedExpansion
SET CLASSPATH=.

FOR %%i IN (lib\*.jar) DO SET CLASSPATH=!CLASSPATH!;%%i

%JAVA_HOME%\bin\java -cp %CLASSPATH YourClassGoesHere

Be sure to include the SETLOCAL (http://ss64.com/nt/setlocal.html) directive and to use the ! operator instead of the % operator when doing the assignment.

ASPxGridView Client-Side API

When it comes to managing tabular data within an ASP.NET application I have found that the ASPxGridView control from DevExpress delivers a nice upgrade from the standard ASP.NET GridView control. In addition to the UI, the ASPxGridView control provides you with a seemingly endless number of customization options and provides great support for CRUD operations, exporting data, and filtering data with as little of ceremony (ok so maybe the ceremony is still there, it's just hidden inside of all of the design time configurations) that is possible with respect to an ASP.NET application.

Overall, I'm 85% satisified with how DevExpress implemented this control. The 15% of dissatisfaction that remains is distributed across missing features, method semantics, and the lack of documentation for the Client-Side API. Since my first two complaints are based on my opinion and I have no influence on how DevExpress conducts its business - I will use this post to cover the absolute basics for exercising the JavaScript API provided by the ASPxGridView control.

Accessing the Client-Side events is done by clicking on the smart dag on a ASPxGridView control while in design mode. With the "Tasks" panel open:

1. Click the link for "Client-Side Events..."
2. Select the event that you wish to customize.
3. You'll see an empty function:

function(s, e) {

}


4. The two parameters passed into the function are the sender object (that's the 's' parameter which is the ASPxGridView in our example) and the event object.
5. For this tutorial let's detect a selection change within the ASPxGridView and perform a simple validation on selection to enable or disable a button that's on the same form. After selecting the SelectionChanged event I provide the following JavaScript:
   

   
   function(s, e) {
   s.GetSelectedFieldValues("ID",
   function onGetValues(result){
      var button = document.getElementById("btnCreateID");
      if (s.GetSelectedRowCount() != 1) {
         button.disabled = true;
     } else {
        if (result[0] = null)
               button.disabled = false;
            else
           button.disabled = true; 
       }});
    }
   

   
   The above code invokes the GetSelectedFieldValues method from the Client-Side API and supplies the column value we wish to retrieve (a column named "ID" in this case) and a callback function that's used to process the result of getting the selected field values. In our example if we've selected one column and if that column doesn't already have an ID assigned to it, the create ID button is enabled.
   

That's all there is to it. The example is basic and somewhat contrived but hopefully it fills the void for your typical 'Getting Started' documentation that I'd like to see DevExpress provide.

My Top 10 Commonly Used Eclipse Shortcuts

Since getting reaquainted with the Eclipse IDE in the past few months I thought I'd share my top 10 most commonly used keyboard shortcuts (along with some commentary of course):

1. Alt + Up or Down Arrow: This shortcut moves a line of code up or down based on the directional arrow you choose. Identation is respected so if you move a line of code inside of something like a loop or conditional you maintain formatting.


2. CTRL + Shift + F: Formats the code. This is a real time saver especially if you have a custom formatter defined for your workspace.


3. CTRL + Alt + Up or Down Arrow: Duplicates a line of code. I find myself using this shortcut when making variable declarations.


4. CTRL + Shift + T: Opens a Type. Simliar to CTRL + Shift + R (open a resource) but this is more precise when you're truly looking to find a type and not a file in your workspace.


5. CTRL + Shift + O: Organizes imports. My code always feels amateur until I remove all of the unnecessary imports along with removing the blanket import statements (e.g., some.package.*).


6. Alt + Shift + R: Refactor. I'm not sure when Eclipse added the UI-sugar that live updates your code during a refactor operation but I like it!


7. CTRL + H: Search, not find. Not a huge fan of the search utility but sometimes it can't be avoided.


8. CTRL + /: Toggle Comment. Another huge time saver especially when you're trying to diagnose an offending block of code. For the purists at heart, the CTRL+Shift+\ and CTRL+Shift+/ add and remove (respectively) block comments.


9. CTRL + W: Close window. Definitely not revolutionary but I feel this one is often overlooked.


10. CTRL + L: Go to line. For the moments when your stack traces appear in a separate log file and not in a console inside of eclipse (where a hyperlink is usually provided if the source is available).

Content Repositories

Managing content vs. data is something that is becoming more clear to me in recent months. Over the last 2 years I’ve had the opportunity to get familiar with XML repositories (mainly MarkLogic Server) but did not really see the benefit of using such a technology outside of an enterprise-search-like application. MarkLogic provides a rich search API that includes content processing and enrichment along with some other pretty powerful features. All of this functionality is great but what if your project budget can’t accommodate the steep price? Are lesser known or open source content repositories still worth it?

I think so. First - what does it really mean when someone says their application manages content vs. data? In most cases, content and data can describe the same type of information but vary significantly in terms of extensibility. Data has a rigid structure that can be difficult to change over time while content tends to have a less rigid structure that can absorb additions and transformations more gracefully over the life of an application or solution. With that loose distinction between content and data in place picture using an XML repository to store messages exchanged in a SOA publish / subscribe paradigm. The messages will most likely evolve over time requiring developers to update the storage and retrieval mechanisms traditionally involving the update of relational queries in the application along with the structure of the database. Since XML repositories don’t have a set structure – developers are only concerned with how to retrieve the information and don’t need to concern themselves with the semantics behind storing the document (message validation isn’t included in this discussion).

This use case is fundamental and primitive but hopefully illustrates a key benefit for those looking to distinguish managing content from data.

The window.open Saga

A while back I encountered a strange problem with window.open() Javascript function. The offending Javascript looked like this:

  window.open('someurl.aspx', 'My Window');

The code seemed to work fine in Firefox but I got an error when trying to evaluate the expression in Internet Explorer 7. I went crazy trying to figure out what was wrong with my code and soon found the space ' ' in the second parameter ('My Window') was causing the problem. If replaced the space with an underscore (i.e., 'My Window' -> 'My_Window') the error disappeared. I noticed that if I would've read the Mozilla JS documentation (https://developer.mozilla.org/En/DOM/Window.open) I would've seen the warning about not putting a space in the window name but I didn't. However the issue was related to IE7 I turned to Microsoft's documentation (http://msdn.microsoft.com/en-us/library/ms536651%28VS.85%29.aspx) and it should be no surprise that the no-space-in-the-window-name isn't mentioned anywhere.

Quality of Documentation

I spend a lot of team reading API documentation to try and figure out how a particular function works, or most likely, I use the documentation to determine how a set of API calls work together to produce a desirable outcome. On most days I don't consider myself to be one of those developers "who just gets it" so I rely heavily on the completeness and quality of the documentation to complete my task in a timely fashion. Recently I have been developing with Java during the day and PHP (more specifically Drupal) by night. I thought juxtaposing (don't you just love that word?) the documentation for the Java platform with Drupal would provide some meaningful insight for updating Java's javadoc utility.

Since Java's inception, the language has grown to an unimaginable size encompassing everything from Desktop to Enterprise to Mobile development. The javadoc utility has remained consistent throughout each release and produces the same clean, intuitive documentation today as it did several years back. In my opinion Java makes it easy to create quality documentation and the platform documentation that Sun's staff puts out is clear, concise, and objective. Enter the API documentation for Drupal.

The documentation for Drupal is not auto-generated. Someone has to sit down and document a function and then publish the documentation for the rest of the world to see. Pulling yourself away from the code to the documentation will result in a lower quality of documentation because the developer will need to make a mental note to go back through code and extract the documentation - this may not happen as scheduled resulting in incomplete platform documentation. As a result of this, Drupal's documentation is concise but does not provide a big picture view of how a particular function interacts with related functions. Despite this shortcoming, the source code for each function is included with the documentation allowing a developer to dive right into the details to figure out how to best use the function. Additionally, the documentation includes a search utility that makes it easy to find the specific function you're looking for without knowing its exact signature.

PHP and Java differ greatly in functionality so I wouldn't want to see the javadoc utility get cute with an autocomplete, search utility but I would think Sun would update the javadoc utility and provide a "web 2.0" approach to the generation of their documentation. Scrolling through ALL the classes is getting a little old. Has anyone seen a third party javadoc utility that provides a "richer" experience?

Auto refresh div using jQuery and Drupal

Recently I had to figure out how to automatically refresh a Drupal block every 5 minutes. The concept is simple enough but since I am new to jQuery (javascript in general really) and Drupal it was difficult for me to find a straight forward example. What follows is step-by-step example of how I solved the problem...

1. Wrap the content you want to refresh in a <div> tag and give it a meaningful class name or ID (e.g., auto--refresh). The class name (or ID) is arbitrary as you will see in later steps. For example:
   
<div class="autorefresh">
Put something here. It will get updated at a set interval.
<div>

   


2. Create a menu function in Drupal to map the AJAX calback:
   
function blockupdate_menu() {
  $items = array();
  $items[] = array(
    'path' => 'blockupdate/update',
    'callback' => 'block_update',
    'type' => MENU_CALLBACK,
    'access' => TRUE,
  );
  return $items;
}

   


3. Implement the function that assembles the data to get sent back to the browser:
   
function block_update() {
  //The get_data() function is generic here
  //Use this function to get the updated data for display
  $html = get_data();
  print drupal_to_js(array('html' => $html));
  // The exit() call is critical!
  exit();
}

   


4. Implement the client side callback function in jQuery:
   
function autoupdate() {
  $.ajax(
  {
    type: "POST",
    url: "blockupdate/update",
    cache: false,
    success: function(data) {
      var result = Drupal.parseJson(data);
      $("div.autorefresh").fadeIn("slow").html(result['html']);
    }
  });
}

   


5. Use the setInterval Javascript function to perform the auto refresh:
   
if( Drupal.jsEnabled ) {
  $(document).ready(function() {
    setInterval("autoupdate()", 5 * 60000);
  });
}

   Note that the first parameter is the jQuery callback function and the second parameter is the time that will elapse in between a refresh, in milliseconds.

That's all there is to it really! To debug the new functionality make sure you have Firebug extension installed in Firefox and watch the traffic do a full roundtrip with updated content. It's also a good idea to set your interval to no more than a few seconds at first to ensure everything is working properly. Don't forget to read my last post on some of the lessons I learned while exploring jQuery inside of Drupal.

Things you should know about jQuery and Drupal

So in my last post I told you how much I love drupal. Although I had been using drupal to build a web app I hadn't been making use of jQuery at that point. jQuery is almost necessary if you want to add visual effects (client-side) and/or use AJAX within drupal. It was somewhat painful for me to get motivated enough to learn jQuery as I felt there was not a good starting point so in this post I'd like to establish some really basic guidelines for new Drupal developers who wish to use jQuery and do not wish to dig through the Drupal forums or sit through long, sleep-inducing screencasts.

Here's what I learned so far:

• Don't bother with "inline" jQuery via the drupal_add_js function...inline Javascript isn't cacheable and you WILL get bitten when trying to escape quotes. Create a separate .js file and put all of your jquery in it instead - then use drupal_add_js to link to your Javascript.


• Spend some time browsing visualjquery.com and don't move past it until you are comfortable with jQuery's syntax...you really will like it once you get going


• Watch out for relative vs. absolute paths when including external Javascript (jQuery) files within a Drupal module. Use drupal_get_path to build your paths for you.


• Firebug is your friend

Is this advice simple and perhaps somewhat contrived? You bet. Finding this advice all in one place isn't all that easy so I felt it was my duty to post it! And finally, for the love of god, if you are trying to create a sweet lightbox effect inside your web app, steer clear of Facebox and try your luck with ThickBox instead. :-)

Stay tuned as I raise the bar further and use jQuery inside of Drupal to bring my first AJAX module to life.