Web GIS Tutorial: WiGLE

This tutorial describes in detail the process of creating a Web GIS application. For this project we used a NAVTEQ map of Chicago as a base map and WiGLE data for GIS analysis.

Concentration of wireless networks in Chicago

WiGLE is an acronym for “Wireless Geographic Logging Engine”. This interesting project has the purpose of cataloging wireless networks across the world. Location of wireless network is calculated by triangulation using GPS traces that are submitted by users. The interesting feature of their service is the ability to query the database and get detailed data about wireless networks. We used this service to get the data for the purpose of this project.


Before we start, these are the requirements:

Software requirements:

Data requirements:

In this project the flashNavigator framework is used to visualize the map and the results of spatial database queries.

The sample map data of Chicago has been converted to Adobe Flash format by the flashNavigator framework. In this article the process of conversion isn’t covered, details about it can be found in one of previous articles.

WiGLE data is available free for download only to registered users. Because of their license this project has also been restricted to users registered at WiGLE. So if you would like to test this application please register at http://www.wigle.net/ and enter your credentials. Your credentials will be only used to query the WiGLE database.

Locations of wireless networks across Chicago are displayed in the flashNavigator client which is embedded into a web application and controlled by the JavaScript API.

The application has two main features:

Querying WiGLE data

WiGLE has a documented API for querying their database. Unfortunately this API isn’t stable, so be sure to check for latest API updates on their Wiki at https://wigle.net/wiki/index.cgi?API and forums at https://wigle.net/phpbb/. The database is only accessible by providing username and password. To reduce usage of the WiGLE database and their servers, we’ll create a local database which will cache requests to WiGLE. For our application we chose PostgreSQL database with the PostGIS extension.

Setting up the database

Before we can create WiGLE tables with PostgreSQL and its PostGIS extension, we need to create a PostGIS Mercator projection that will be used for stored data. We are using Mercator projection because the maps generated by flashNavigator are in Mercator projection too. It is crucial that the projections match.

This query adds Mercator projection to PostGIS:

INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid, srtext, proj4text)

VALUES (
54004,
'spatialreference.org',
54004,
'PROJCS["World_Mercator",GEOGCS["GCS_WGS_1984",DATUM["WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Mercator_1SP"],PARAMETER["False_Easting",0],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",0],PARAMETER["Standard_Parallel_1",0],UNIT["Meter",1],AUTHORITY["EPSG","54004"]]',
'+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs '

);

Next we need to create table in which the WiGLE data will be stored:

CREATE TABLE wigle
(
  netid character varying(17) NOT NULL,

  ssid character varying(256) NOT NULL,
  "comment" character varying(256) NOT NULL,

  name character varying(256) NOT NULL,
  "type" character varying(256) NOT NULL,

  freenet character varying(1) NOT NULL,
  paynet character varying(1) NOT NULL,

  firsttime timestamp without time zone NOT NULL,
  flags character varying(4) NOT NULL,

  wep character varying(1) NOT NULL,
  trilat double precision NOT NULL,

  trilong double precision NOT NULL,
  dhcp character varying(1) NOT NULL,

  lastupdt timestamp without time zone NOT NULL,
  channel integer NOT NULL,
  active character varying(1) NOT NULL,

  bcninterval integer NOT NULL,
  qos integer NOT NULL,
  userfound character varying(1) NOT NULL

);

Now we’ll add a PostGIS column in which we’ll store wireless network coordinates. We’ll add the column “point” which is of type POINT with the projection set to Mercator (54004).

SELECT
  AddGeometryColumn('wigle', 'wigle', 'point', 54004, 'POINT', 2);

After the table is created we can populate it using the WiGLE API. When we have all the data we need in our database, we need to set the PostGIS column point to WiGLE coordinate values. Wireless networks coordinates are stored in Lat/Long projection (id 4326). As we want to use Mercator projection (id 54004) we need to transform geometry to Mercator projection.

UPDATE wigle
   SET point = Transform(
       GeomFromText('POINT('|| trilong ||' ' || trilat || ')',4326),54004

);

The final step to get our “wigle” table ready is to create spatial index for faster access on the “point” column:

CREATE INDEX point_index ON wigle USING gist(point);

For creating the visual representation of wireless network concentration we’ll create another table that will be made out of grid cells. The idea is to assign to each cell the number of wireless networks that are inside it. With this information we can color each cell according to number of wireless networks that are in that area.

First we need to create table grid:

CREATE TABLE grid
(
  gid serial NOT NULL PRIMARY KEY,

  count integer
);
 
SELECT
  AddGeometryColumn('wigle', 'grid', 'cell', 54004, 'POLYGON', 2);

Now we need to create cells and store them into grid table. To create grid cells using PostGIS we’ll generate coordinates from the “wiggle” table by using the “generate_series” method. Then we’ll use those coordinate to create a cell that will be stored in the “grid” table. To make sure the projection is consistent with other tables and map data we will need to set the box to Mercator projection before storing it in the “grid” table.

We’ll create cells of 1 km2 by running this query:

INSERT INTO grid(cell)
  SELECT ST_SetSRID(

    makebox2d(
      st_makepoint(hor.n,ver.n),

      st_makepoint(hor.n+1000,ver.n+1000)
      ),54004) AS cell
  FROM

    generate_series(
        (SELECT ST_xmin(ST_Extent(point))-1000 FROM wigle)::integer,

        (SELECT ST_xmax(ST_Extent(point))+1000 FROM wigle)::integer,1000)

      AS hor(n),
    generate_series(
        (SELECT ST_ymin(ST_Extent(point))-1000 FROM wigle)::integer,

        (SELECT ST_ymax(ST_Extent(point))+1000 FROM wigle)::integer,1000)

      AS ver(n);

Finally after the grid is created we’ll populate it with number of wireless networks that are within each cell by running this simple query:

UPDATE grid
  SET count = ( SELECT count(*) FROM wigle WHERE St_Within(point,cell) );

Setting up flashNavigator framework viewer

We’ll use the flashNavigator framework (which is based on Adobe Flash technology) to display the map and WiGLE wireless networks. Working with this framework does not require any knowledge of Adobe Flash. FlashNavigator has an API for various platforms and for this project we’ll use JavaScript API.

To enable the JavaScript API, we must include one JavaScript file in the tag of the web page.

<script src="http://www.flashnavigator.net/wigle/fnViewer/flashNavigator.js" type="text/javascript"><!--mce:0--></script>

Now we can initialize flashNavigator. The namespace flashnavigator is static with static methods. With flashnavigator.include we define what plugins will be used in our project. The method flashnavigator.initialize has to be defined. This method will be called when flashNavigator core and plugins are loaded. Code that will initialize the viewer can be safely used in this method. In the code bellow we are creating a fnViewer object which will be loaded into a HTML container with the id “flashcontent”. Because we’ll be drawing something on the map we’ll use the canvas plugin.

// sets list of plugins that are used to extend
// flashNavigator functionality
flashnavigator.include("slider,scalebar,canvas,marker");
 
flashnavigator.initialize = function()

{
  // fnViewer constructor
  fn = new fnViewer("flashcontent");

 
  // canvas plugin constructor
  canvas = new fnCanvas(fn);
 

  // marker plugin constructor
  marker = new fnMarker(fn);
 
  // registers event listeners

  fn.addEventListener("load","fnOnLoad");
  fn.addEventListener("mouseup","fnOnMouseUp");

  fn.addEventListener("mousemove","fnOnMouseMove");
  fn.addEventListener("mousedown","fnOnMouseDown");

  fn.addEventListener("scalechange","fnUpdateScale");
 
  // loads map project
  fn.load("/wigle/maps/chicago.xml");

}

Next, it’s necessary to create a HTML element in which the map will be displayed. This element has to be reserved for flashNavigator by calling a special API method.

<div id="flashcontent" style="border: 1px solid #cccccc; width: 615px; height: 500px;">
  flashNavigator container</div>

<script type="text/javascript"><!--mce:1--></script>

GIS analysis (WiFi distribution in Chicago)

In this tutorial we’ll display distribution of wireless networks according to WiGLE data. The data has been stored into grid structure during database setup. So now we need a way to display that data using flashNavigator viewer. The backend part that will query database is written in PHP. The data will be formatted in JSON format for easy usage in JavaScript.

PHP script that queries database and outputs JSON formatted data:

  $xmin = floatval($_REQUEST["xmin"]);

  $xmax = floatval($_REQUEST["xmax"]);
  $ymin = floatval($_REQUEST["ymin"]);

  $ymax = floatval($_REQUEST["ymax"]);
 
  $link = pg_Connect("dbname=wigle user=user password=password");

 
  $query = "SELECT AsText(the_geom) as geom, count
            FROM grid
            WHERE ST_Intersects(the_geom,
                                ST_SetSRID(
                                   makebox2d(st_makepoint($xmin,$ymin),
                                             st_makepoint($xmax,$ymax))
                                   ,54004))
            ORDER BY count
            DESC LIMIT 1000";
 
   $result = pg_exec($link, $query);

 
   echo json_encode( pg_fetch_all($result) );
 
   pg_close($link);

Now we can display that data using JavaScript and flashNavigator API:

function getWirelessConcentration ()
{
  // we get current bound and send it to PHP script
  document.getElementById("message").innerHTML = "Loading...";

  var b = fn.getViewBound();
  var query = "&amp;xmin="+b.xMin+"&amp;xmax="+b.xMax+"&amp;ymin="+b.yMin+"&amp;ymax="+b.yMax;

  xmlhttpPost("/wigle/getAP2.php",query,drawWirelessDistribution);
}
 
function drawWirelessDistribution (cells)

{
 
  // first we delete everything that was displayed before
  canvas.deleteAll();
 
  // Json data needs to be evaluated before we can use it

  cells = eval(cells);
 
  // we truncate maximal number of wireless network per cell
  var maxCount = 2000;

 
  var frequency = (Math.PI/2)/maxCount;

  for (i=0; i &lt; cells.length; i++)

  {
    k = cells[i].count;
 
    if (k &gt; maxCount)

    k = maxCount;
 
    red   = Math.sin(frequency*k + 0) * 127 + 128;

    green = Math.sin(frequency*k + 2) * 127 + 128;

    blue  = Math.sin(frequency*k + 4) * 127 + 128;

 
    cells[i].color = RGB2Color(red,green,blue);

    cells[i].alpha = 60;
 
    canvas.deserializeGeometryOpenGIS(cells[i]);

  }
  document.getElementById("message").innerHTML = "";

}
 
function RGB2Color(r,g,b)
{
  // convets RGB formated color to integer

  return (r &amp; 0xff) *0x10000 | (g &amp; 0xff)*0x100 | (b &amp; 0xff);

}

WiFi Access Points display

To display wireless access points we need to define area in which we want to display them. Currently there are around 500 000 wireless networks in Chicago according to WiGLE database, so it would be impossible to display all of them at once.

Area will be defined by use of mouse. To define area it’s necessary to hold left mouse button down and then drag mouse to define circle radius. To display access points mouse button must be released. Wireless networks are display accordingly to zoom level. In some areas there is too many wireless networks so on in those areas some wireless networks will be discarded. To show more wireless networks it’s necessary to zoom in.

Wireless network markers are colored according to their QoS value. QoS is WiGLE value that define quality of access point location. Some access points have been maybe submitted few years ago and only by one user, so those access point will have low QoS and will be colored red.

Wireless access point display

Here is the JavaScript code for circle drawing using flashNavigator API:

function fnOnMouseDown(evt)
{
  /* on mouse down we store current mouse data
  like x and y coordinates */
 
  if (selectAreaTool)

    areaData = evt;
}
 
function fnOnMouseUp(evt)

{
  /* on mouse up we start loading access point data */
 
  if (selectAreaTool)
  {
    document.getElementById("message").innerHTML = "Loading...";

    query  = "&amp;x="+areaData.x+"&amp;y="+areaData.y+"&amp;r="+areaData.d;

    query += "&amp;s="+fn.getScale();
    xmlhttpPost("/wigle/getAP.php",query,getAP);

    fn.setTool('DragTool');
    selectAreaTool = 0;
  }

  else
    updateWithTimer();
}
 
function fnOnMouseMove(evt)

{
  /* during mouse move we need to calculate circle radius
     and draw the circle using flashNavigator API */
 
  if (!selectAreaTool || !areaData) return;

  var xx = areaData.x-evt.x;
  var yy = areaData.y-evt.y;

  var radius = Math.sqrt(xx*xx+yy*yy);

  if (radius &gt; 500) radius = 500;

 
  areaData.radius = radius;
 
  if (areaData.circle)

    canvas.remove(areaData.circle);
 
  var c = canvas.drawCircle(areaData.x, areaData.y, radius, 0x0000ff, 30);

  areaData.circle = c;
}

To gather wireless network location data we use following PHP script:

function dist($x1,$y1,$x2,$y2)

{
  $x = $x1-$x2;
  $y = $y1-$y2;

  return sqrt($x*$x+$y*$y);
}

 
$x = floatval($_REQUEST["x"]);
$y = floatval($_REQUEST["y"]);

$r = floatval($_REQUEST["r"]);
$scale = floatval($_REQUEST["s"]);

 
$bound_check = "";
if (isset($_REQUEST["xmin"]))

{
  $xmin = floatval($_REQUEST["xmin"]);
  $xmax = floatval($_REQUEST["xmax"]);

  $ymin = floatval($_REQUEST["ymin"]);
  $ymax = floatval($_REQUEST["ymax"]);

  $bound_check = " AND x(point) &gt; $xmin AND x(point) &lt; $xmax AND y(point) &gt;
                         $ymin AND y(point) &lt; $ymax ";
  }
 
$link = pg_Connect("dbname=wigle user=username password=password");

 
$query = "SELECT ssid,
                 qos,
                 netid,
                 x(point) as x,
                 y(point) as y
          FROM wigle
          WHERE st_dwithin(point,GeomFromText('POINT($x $y)',54004),$r)
                $bound_check
          ORDER BY qos DESC
         ";
 
$result = pg_exec($link, $query);

 
$ar = array();
$len = 0;

$d = 30*$scale/3779.527559055;
$count = count($rows);

 
for ($i = 0; $i &lt; $count; $i++)

{
  $data = $rows[$i];
  $flag = 1;

 
  for ($e = 0; $e &lt; $len; $e++)

  {
    if (dist($data["x"],$data["y"],$ar[$e]["x"],$ar[$e]["y"]) &lt; $d)

    {
      $flag = 0;
      break;
    }

  }
 
  if (!$flag) continue;
 
  $len++;

 
  array_push($ar,$data);
}
 
ob_start("ob_gzhandler");

 
echo json_encode($ar);
 
pg_close($link);

Finally to display gathered data we use following JavaScript:

//stores visible access points
var visibleAp = new Array();

 
function getAP(ap)
{
  /* we go through all received access points.
  if access point isn't already displayed
  we display it using marker.add method */
 
  var ap = eval(ap);

  var maxCount = 7;
  var frequency = (Math.PI/2)/maxCount;

  var tmpvisibleAp = new Array();
 
  for (i=0; i &lt; ap.length; i++)

  {
    if (!ap[i]) continue;

 
    if (visibleAp[ap[i].netid])
    {

      tmpvisibleAp[ap[i].netid] = visibleAp[ap[i].netid];

      visibleAp[ap[i].netid] = null;
      continue;

    }
 
    var k = ap[i].qos;

    if (k &gt; maxCount)
    k = maxCount;

 
    ap[i].icon = "/wigle/wifi.swf";
    ap[i].color = calculateColor( frequency * (maxCount-k) );

    ap[i].title = ap[i].ssid

    ap[i].content = "QoS: "+ap[i].qos;

    tmpvisibleAp[ap[i].netid] = marker.add(ap[i]);

  }
 
  for (h in visibleAp)
  {

    if (visibleAp[h])
      marker.remove(visibleAp[h]);

  }
 
  visibleAp = tmpvisibleAp;
  document.getElementById("message").innerHTML = "";

}

On mouse up and zoom event we’ll update visible access points and wireless network concentration by following code:

function Update()
{
  /*
  if access points are displayed then we refresh them
  by using current visible bound.
 

  if scale if less then 30000 then we refresh wireless
  concentration too.
  */
  var scale = fn.getScale();
  if (areaData)

  {
    var b = fn.getViewBound();
    var query  = "&amp;x="+areaData.x+"&amp;y="+areaData.y+"&amp;r="+areaData.d+"&amp;s="+fn.getScale();

    query += "&amp;xmin="+b.xMin+"&amp;xmax="+b.xMax+"&amp;ymin="+b.yMin+"&amp;ymax="+b.yMax;

    xmlhttpPost("/wigle/getAP.php",query,getAP);
  }
  if (scale &lt; 30000)

  getWirelessConcentration();
}
 
function updateWithTimer()
{

  window.clearTimeout(updateTimer);
  updateTimer = window.setTimeout(Update,500);

}

Update (4. August 2008): WiGLE login method written in PHP:

function http_parse_headers($header)
{
  $retVal = array();

  $fields = explode("\r\n", preg_replace('/\x0D\x0A[\x09\x20]+/', ' ', $header));

  foreach( $fields as $field ) 
  {
    if(preg_match('/([^:]+): (.+)/m', $field, $match)) 
    {

      $match[1] = preg_replace('/(?<=^|[\x09\x20\x2D])./e', 'strtoupper("\0")', strtolower(trim($match[1])));

      if(isset($retVal[$match[1]]))           
        $retVal[$match[1]] = array($retVal[$match[1]], $match[2]);

       else        
        $retVal[$match[1]] = trim($match[2]);     
    }

  }
  return $retVal;
}
 
function wigle_login($user,$pass)

{     
  $ch = curl_init();
 
  // set the target url
  curl_setopt($ch, CURLOPT_URL,"http://wigle.net/gps/gps/GPSDB/login");

 
  // how many parameters to post
  curl_setopt($ch, CURLOPT_POST, 3);

 
  // post parameters
  curl_setopt($ch, CURLOPT_POSTFIELDS,"credential_0=".$user."&credential_1=".$pass."&noexpire=off");      

 
  curl_setopt ($ch, CURLOPT_HEADER, 1);   
  curl_setopt($ch, CURLOPT_RETURNTRANSFER  ,1);

 
  $result = curl_exec($ch);               
 
  curl_close ($ch); 

 
  return $result;
}
 
function getWigleCookie($user,$pass,$noexpire=false) 

{        
  $cookie = wigle_login($user,$pass); 
  $headers = http_parse_headers($cookie);             

 
  if (!isset($headers['Set-Cookie']) || empty($headers['Set-Cookie'])) 
    return false;        

 
  $_SESSION["wigle_cookie"] = $headers['Set-Cookie'];
 
  return true;

}
 
session_start();
 
if (isset($_REQUEST["logout"])) 
  unset($_SESSION["wigle_cookie"]);

 
else if (getWigleCookie($_REQUEST["user"],$_REQUEST["pass"]))

  echo "ok";