Web GIS Tutorial: WiGLE

Omnisdata — Sat, 12 Jul 2008 20:07:27 +0000

Project URL: http://www.flashnavigator.net/wigle/

This tutorial describes in detail the process of creating a Web GIS application. For this project we used a NAVTEQ map of Chicago as a base map and WiGLE data for GIS analysis.

Concentration of wireless networks in Chicago

WiGLE is an acronym for “Wireless Geographic Logging Engine”. This interesting project has the purpose of cataloging wireless networks across the world. Location of wireless network is calculated by triangulation using GPS traces that are submitted by users. The interesting feature of their service is the ability to query the database and get detailed data about wireless networks. We used this service to get the data for the purpose of this project.

Before we start, these are the requirements:

Software requirements:

flashNavigator GIS/CAD framework (http://www.flashnavigator.net/ used version 2.0)

PHP (http://php.net/ used version 4.4.4-8)

PHP-JSON module for PHP (http://php.net/ used version 1.2.1)

PostgreSQL database (http://www.postgresql.org/ used version 8.1)

PostGIS extension for PostgreSQL database (http://postgis.refractions.net/ used version 1.3.3)

Data requirements:

NAVTEQ sample map of Chicago (http://www.nn4d.com/, ESRI Shapefile format)

WiGLE data for Chicago (http://www.wigle.net/)

In this project the flashNavigator framework is used to visualize the map and the results of spatial database queries.

The sample map data of Chicago has been converted to Adobe Flash format by the flashNavigator framework. In this article the process of conversion isn’t covered, details about it can be found in one of previous articles located at http://www.flashnavigator.net/resources/.

WiGLE data is available free for download only to registered users. Because of their license this project has also been restricted to users registered at WiGLE. So if you would like to test this application please register at http://www.wigle.net/ and enter your credentials at form located at http://www.flashnavigator.net/wigle/. Your credentials will be only used to query the WiGLE database.

Locations of wireless networks across Chicago are displayed in the flashNavigator client which is embedded into a web application and controlled by the JavaScript API.

The application has two main features:

Visual display of wireless network concentration in Chicago by coloring areas in different color. For this purpose red color is used for high and green color for low concentration.

Ability to dynamically query the database and display wireless networks in an area marked by a circle.

Querying WiGLE data

WiGLE has a documented API for querying their database. Unfortunately this API isn’t stable, so be sure to check for latest API updates on their Wiki at https://wigle.net/wiki/index.cgi?API and forums at https://wigle.net/phpbb/. The database is only accessible by providing username and password. To reduce usage of the WiGLE database and their servers, we’ll create a local database which will cache requests to WiGLE. For our application we chose PostgreSQL database with the PostGIS extension.

Setting up the database

Before we can create WiGLE tables with PostgreSQL and its PostGIS extension, we need to create a PostGIS Mercator projection that will be used for stored data. We are using Mercator projection because the maps generated by flashNavigator are in Mercator projection too. It is crucial that the projections match.

This query adds Mercator projection to PostGIS:

INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid, srtext, proj4text)
VALUES (
54004,
'spatialreference.org',
54004,
'PROJCS["World_Mercator",GEOGCS["GCS_WGS_1984",DATUM["WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Mercator_1SP"],PARAMETER["False_Easting",0],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",0],PARAMETER["Standard_Parallel_1",0],UNIT["Meter",1],AUTHORITY["EPSG","54004"]]',
'+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs '
);

Next we need to create table in which the WiGLE data will be stored:

CREATE TABLE wigle
(
  netid character varying(17) NOT NULL,
  ssid character varying(256) NOT NULL,
  "comment" character varying(256) NOT NULL,
  name character varying(256) NOT NULL,
  "type" character varying(256) NOT NULL,
  freenet character varying(1) NOT NULL,
  paynet character varying(1) NOT NULL,
  firsttime timestamp without time zone NOT NULL,
  flags character varying(4) NOT NULL,
  wep character varying(1) NOT NULL,
  trilat double precision NOT NULL,
  trilong double precision NOT NULL,
  dhcp character varying(1) NOT NULL,
  lastupdt timestamp without time zone NOT NULL,
  channel integer NOT NULL,
  active character varying(1) NOT NULL,
  bcninterval integer NOT NULL,
  qos integer NOT NULL,
  userfound character varying(1) NOT NULL
);

Now we’ll add a PostGIS column in which we’ll store wireless network coordinates. We’ll add the column “point” which is of type POINT with the projection set to Mercator (54004).

SELECT
  AddGeometryColumn('wigle', 'wigle', 'point', 54004, 'POINT', 2);

After the table is created we can populate it using the WiGLE API. When we have all the data we need in our database, we need to set the PostGIS column point to WiGLE coordinate values. Wireless networks coordinates are stored in Lat/Long projection (id 4326). As we want to use Mercator projection (id 54004) we need to transform geometry to Mercator projection.

UPDATE wigle
   SET point = Transform(
       GeomFromText('POINT('|| trilong ||' ' || trilat || ')',4326),54004
);

The final step to get our “wigle” table ready is to create spatial index for faster access on the “point” column:

CREATE INDEX point_index ON wigle USING gist(point);

For creating the visual representation of wireless network concentration we’ll create another table that will be made out of grid cells. The idea is to assign to each cell the number of wireless networks that are inside it. With this information we can color each cell according to number of wireless networks that are in that area.

First we need to create table grid:

CREATE TABLE grid
(
  gid serial NOT NULL PRIMARY KEY,
  count integer
);
 
SELECT
  AddGeometryColumn('wigle', 'grid', 'cell', 54004, 'POLYGON', 2);

Now we need to create cells and store them into grid table. To create grid cells using PostGIS we’ll generate coordinates from the “wiggle” table by using the “generate_series” method. Then we’ll use those coordinate to create a cell that will be stored in the “grid” table. To make sure the projection is consistent with other tables and map data we will need to set the box to Mercator projection before storing it in the “grid” table.

We’ll create cells of 1 km2 by running this query:

INSERT INTO grid(cell)
  SELECT ST_SetSRID(
    makebox2d(
      st_makepoint(hor.n,ver.n),
      st_makepoint(hor.n+1000,ver.n+1000)
      ),54004) AS cell
  FROM
    generate_series(
        (SELECT ST_xmin(ST_Extent(point))-1000 FROM wigle)::integer,
        (SELECT ST_xmax(ST_Extent(point))+1000 FROM wigle)::integer,1000)
      AS hor(n),
    generate_series(
        (SELECT ST_ymin(ST_Extent(point))-1000 FROM wigle)::integer,
        (SELECT ST_ymax(ST_Extent(point))+1000 FROM wigle)::integer,1000)
      AS ver(n);

Finally after the grid is created we’ll populate it with number of wireless networks that are within each cell by running this simple query:

UPDATE grid
  SET count = ( SELECT count(*) FROM wigle WHERE St_Within(point,cell) );

Setting up flashNavigator framework viewer

We’ll use the flashNavigator framework (which is based on Adobe Flash technology) to display the map and WiGLE wireless networks. Working with this framework does not require any knowledge of Adobe Flash. FlashNavigator has an API for various platforms and for this project we’ll use JavaScript API.

To enable the JavaScript API, we must include one JavaScript file in the tag of the web page.

 src="http://www.flashnavigator.net/wigle/fnViewer/flashNavigator.js" type="text/javascript">

Now we can initialize flashNavigator. The namespace flashnavigator is static with static methods. With flashnavigator.include we define what plugins will be used in our project. The method flashnavigator.initialize has to be defined. This method will be called when flashNavigator core and plugins are loaded. Code that will initialize the viewer can be safely used in this method. In the code bellow we are creating a fnViewer object which will be loaded into a HTML container with the id “flashcontent”. Because we’ll be drawing something on the map we’ll use the canvas plugin.

// sets list of plugins that are used to extend
// flashNavigator functionality
flashnavigator.include("slider,scalebar,canvas,marker");
 
flashnavigator.initialize = function()
{
  // fnViewer constructor
  fn = new fnViewer("flashcontent");
 
  // canvas plugin constructor
  canvas = new fnCanvas(fn);
 
  // marker plugin constructor
  marker = new fnMarker(fn);
 
  // registers event listeners
  fn.addEventListener("load","fnOnLoad");
  fn.addEventListener("mouseup","fnOnMouseUp");
  fn.addEventListener("mousemove","fnOnMouseMove");
  fn.addEventListener("mousedown","fnOnMouseDown");
  fn.addEventListener("scalechange","fnUpdateScale");
 
  // loads map project
  fn.load("/wigle/maps/chicago.xml");
}

Next, it’s necessary to create a HTML element in which the map will be displayed. This element has to be reserved for flashNavigator by calling a special API method.

 id="flashcontent" style="border: 1px solid #cccccc; width: 615px; height: 500px;">
  flashNavigator container

type="text/javascript">

GIS analysis (WiFi distribution in Chicago)

In this tutorial we’ll display distribution of wireless networks according to WiGLE data. The data has been stored into grid structure during database setup. So now we need a way to display that data using flashNavigator viewer. The backend part that will query database is written in PHP. The data will be formatted in JSON format for easy usage in JavaScript.

PHP script that queries database and outputs JSON formatted data:

  $xmin = floatval($_REQUEST["xmin"]);
  $xmax = floatval($_REQUEST["xmax"]);
  $ymin = floatval($_REQUEST["ymin"]);
  $ymax = floatval($_REQUEST["ymax"]);
 
  $link = pg_Connect("dbname=wigle user=user password=password");
 
  $query = "SELECT AsText(the_geom) as geom, count
            FROM grid
            WHERE ST_Intersects(the_geom,
                                ST_SetSRID(
                                   makebox2d(st_makepoint($xmin,$ymin),
                                             st_makepoint($xmax,$ymax))
                                   ,54004))
            ORDER BY count
            DESC LIMIT 1000";
 
   $result = pg_exec($link, $query);
 
   echo json_encode( pg_fetch_all($result) );
 
   pg_close($link);

Now we can display that data using JavaScript and flashNavigator API:

function getWirelessConcentration ()
{
  // we get current bound and send it to PHP script
  document.getElementById("message").innerHTML = "Loading...";
  var b = fn.getViewBound();
  var query = "&xmin="+b.xMin+"&xmax="+b.xMax+"&ymin="+b.yMin+"&ymax="+b.yMax;
  xmlhttpPost("/wigle/getAP2.php",query,drawWirelessDistribution);
}
 
function drawWirelessDistribution (cells)
{
 
  // first we delete everything that was displayed before
  canvas.deleteAll();
 
  // Json data needs to be evaluated before we can use it
  cells = eval(cells);
 
  // we truncate maximal number of wireless network per cell
  var maxCount = 2000;
 
  var frequency = (Math.PI/2)/maxCount;
  for (i=0; i &lt; cells.length; i++)
  {
    k = cells[i].count;
 
    if (k &gt; maxCount)
    k = maxCount;
 
    red   = Math.sin(frequency*k + 0) * 127 + 128;
    green = Math.sin(frequency*k + 2) * 127 + 128;
    blue  = Math.sin(frequency*k + 4) * 127 + 128;
 
    cells[i].color = RGB2Color(red,green,blue);
    cells[i].alpha = 60;
 
    canvas.deserializeGeometryOpenGIS(cells[i]);
  }
  document.getElementById("message").innerHTML = "";
}
 
function RGB2Color(r,g,b)
{
  // convets RGB formated color to integer
  return (r &amp; 0xff) *0x10000 | (g &amp; 0xff)*0x100 | (b &amp; 0xff);
}

WiFi Access Points display

To display wireless access points we need to define area in which we want to display them. Currently there are around 500 000 wireless networks in Chicago according to WiGLE database, so it would be impossible to display all of them at once.

Area will be defined by use of mouse. To define area it’s necessary to hold left mouse button down and then drag mouse to define circle radius. To display access points mouse button must be released. Wireless networks are display accordingly to zoom level. In some areas there is too many wireless networks so on in those areas some wireless networks will be discarded. To show more wireless networks it’s necessary to zoom in.

Wireless network markers are colored according to their QoS value. QoS is WiGLE value that define quality of access point location. Some access points have been maybe submitted few years ago and only by one user, so those access point will have low QoS and will be colored red.

Wireless access point display

Here is the JavaScript code for circle drawing using flashNavigator API:

function fnOnMouseDown(evt)
{
  /* on mouse down we store current mouse data
  like x and y coordinates */
 
  if (selectAreaTool)
    areaData = evt;
}
 
function fnOnMouseUp(evt)
{
  /* on mouse up we start loading access point data */
 
  if (selectAreaTool)
  {
    document.getElementById("message").innerHTML = "Loading...";
    query  = "&x="+areaData.x+"&y="+areaData.y+"&r="+areaData.d;
    query += "&s="+fn.getScale();
    xmlhttpPost("/wigle/getAP.php",query,getAP);
    fn.setTool('DragTool');
    selectAreaTool = 0;
  }
  else
    updateWithTimer();
}
 
function fnOnMouseMove(evt)
{
  /* during mouse move we need to calculate circle radius
     and draw the circle using flashNavigator API */
 
  if (!selectAreaTool || !areaData) return;
  var xx = areaData.x-evt.x;
  var yy = areaData.y-evt.y;
  var radius = Math.sqrt(xx*xx+yy*yy);
  if (radius &gt; 500) radius = 500;
 
  areaData.radius = radius;
 
  if (areaData.circle)
    canvas.remove(areaData.circle);
 
  var c = canvas.drawCircle(areaData.x, areaData.y, radius, 0x0000ff, 30);
  areaData.circle = c;
}

To gather wireless network location data we use following PHP script:

function dist($x1,$y1,$x2,$y2)
{
  $x = $x1-$x2;
  $y = $y1-$y2;
  return sqrt($x*$x+$y*$y);
}
 
$x = floatval($_REQUEST["x"]);
$y = floatval($_REQUEST["y"]);
$r = floatval($_REQUEST["r"]);
$scale = floatval($_REQUEST["s"]);
 
$bound_check = "";
if (isset($_REQUEST["xmin"]))
{
  $xmin = floatval($_REQUEST["xmin"]);
  $xmax = floatval($_REQUEST["xmax"]);
  $ymin = floatval($_REQUEST["ymin"]);
  $ymax = floatval($_REQUEST["ymax"]);
  $bound_check = " AND x(point) > $xmin AND x(point) < $xmax AND y(point) >
                         $ymin AND y(point) < $ymax ";
  }
 
$link = pg_Connect("dbname=wigle user=username password=password");
 
$query = "SELECT ssid,
                 qos,
                 netid,
                 x(point) as x,
                 y(point) as y
          FROM wigle
          WHERE st_dwithin(point,GeomFromText('POINT($x $y)',54004),$r)
                $bound_check
          ORDER BY qos DESC
         ";
 
$result = pg_exec($link, $query);
 
$ar = array();
$len = 0;
$d = 30*$scale/3779.527559055;
$count = count($rows);
 
for ($i = 0; $i &lt; $count; $i++)
{
  $data = $rows[$i];
  $flag = 1;
 
  for ($e = 0; $e &lt; $len; $e++)
  {
    if (dist($data["x"],$data["y"],$ar[$e]["x"],$ar[$e]["y"]) &lt; $d)
    {
      $flag = 0;
      break;
    }
  }
 
  if (!$flag) continue;
 
  $len++;
 
  array_push($ar,$data);
}
 
ob_start("ob_gzhandler");
 
echo json_encode($ar);
 
pg_close($link);

Finally to display gathered data we use following JavaScript:

//stores visible access points
var visibleAp = new Array();
 
function getAP(ap)
{
  /* we go through all received access points.
  if access point isn't already displayed
  we display it using marker.add method */
 
  var ap = eval(ap);
  var maxCount = 7;
  var frequency = (Math.PI/2)/maxCount;
  var tmpvisibleAp = new Array();
 
  for (i=0; i &lt; ap.length; i++)
  {
    if (!ap[i]) continue;
 
    if (visibleAp[ap[i].netid])
    {
      tmpvisibleAp[ap[i].netid] = visibleAp[ap[i].netid];
      visibleAp[ap[i].netid] = null;
      continue;
    }
 
    var k = ap[i].qos;
    if (k &gt; maxCount)
    k = maxCount;
 
    ap[i].icon = "/wigle/wifi.swf";
    ap[i].color = calculateColor( frequency * (maxCount-k) );
    ap[i].title = ap[i].ssid
    ap[i].content = "QoS: "+ap[i].qos;
    tmpvisibleAp[ap[i].netid] = marker.add(ap[i]);
  }
 
  for (h in visibleAp)
  {
    if (visibleAp[h])
      marker.remove(visibleAp[h]);
  }
 
  visibleAp = tmpvisibleAp;
  document.getElementById("message").innerHTML = "";
}

On mouse up and zoom event we’ll update visible access points and wireless network concentration by following code:

function Update()
{
  /*
  if access points are displayed then we refresh them
  by using current visible bound.
 
  if scale if less then 30000 then we refresh wireless
  concentration too.
  */
  var scale = fn.getScale();
  if (areaData)
  {
    var b = fn.getViewBound();
    var query  = "&x="+areaData.x+"&y="+areaData.y+"&r="+areaData.d+"&s="+fn.getScale();
    query += "&xmin="+b.xMin+"&xmax="+b.xMax+"&ymin="+b.yMin+"&ymax="+b.yMax;
    xmlhttpPost("/wigle/getAP.php",query,getAP);
  }
  if (scale &lt; 30000)
  getWirelessConcentration();
}
 
function updateWithTimer()
{
  window.clearTimeout(updateTimer);
  updateTimer = window.setTimeout(Update,500);
}

Update (4. August 2008): WiGLE login method written in PHP:

function http_parse_headers($header)
{
  $retVal = array();
  $fields = explode("\r\n", preg_replace('/\x0D\x0A[\x09\x20]+/', ' ', $header));
  foreach( $fields as $field ) 
  {
    if(preg_match('/([^:]+): (.+)/m', $field, $match)) 
    {
      $match[1] = preg_replace('/(?<=^|[\x09\x20\x2D])./e', 'strtoupper("\0")', strtolower(trim($match[1])));
      if(isset($retVal[$match[1]]))           
        $retVal[$match[1]] = array($retVal[$match[1]], $match[2]);
       else        
        $retVal[$match[1]] = trim($match[2]);     
    }
  }
  return $retVal;
}
 
function wigle_login($user,$pass)
{     
  $ch = curl_init();
 
  // set the target url
  curl_setopt($ch, CURLOPT_URL,"http://wigle.net/gps/gps/GPSDB/login");
 
  // how many parameters to post
  curl_setopt($ch, CURLOPT_POST, 3);
 
  // post parameters
  curl_setopt($ch, CURLOPT_POSTFIELDS,"credential_0=".$user."&credential_1=".$pass."&noexpire=off");      
 
  curl_setopt ($ch, CURLOPT_HEADER, 1);   
  curl_setopt($ch, CURLOPT_RETURNTRANSFER  ,1);
 
  $result = curl_exec($ch);               
 
  curl_close ($ch); 
 
  return $result;
}
 
function getWigleCookie($user,$pass,$noexpire=false) 
{        
  $cookie = wigle_login($user,$pass); 
  $headers = http_parse_headers($cookie);             
 
  if (!isset($headers['Set-Cookie']) || empty($headers['Set-Cookie'])) 
    return false;        
 
  $_SESSION["wigle_cookie"] = $headers['Set-Cookie'];
 
  return true;
}
 
session_start();
 
if (isset($_REQUEST["logout"])) 
  unset($_SESSION["wigle_cookie"]);
 
else if (getWigleCookie($_REQUEST["user"],$_REQUEST["pass"]))
  echo "ok";

About flashNavigator

Omnisdata — Sat, 12 Jul 2008 12:08:29 +0000

flashNavigator is a rapid development framework, used for publishing geospatial (GIS and CAD) data with Adobe Flash technology in web, desktop and mobile applications. Unlike other solutions it provides tiled vector to vector conversion of GIS and CAD files and database formats to Flash.

Adobe Flash utilizes a vector file format (SWF) which makes it great for GIS and CAD applications. The format is also optimized in consideriation to size, making it great for environments like the Internet.

flashNavigator viewer is an optimized Flash component. The viewer core has a total size of around 20 KB with a preloader of 4 KB. The core engine can be extended with custom plugins, which make the entire solution very flexible for usage in target applications.

flashNavigator is suited for use in simple (lite) projects utilizing only basic functionality, but it can also be used in enterprise projects which require complex functionality. It can be integrated into almost any current and upcoming web, desktop and mobile technology that supports Flash or Flash Lite.

Signed agreement with French company Kalyo

Omnisdata — Mon, 09 Jun 2008 12:40:28 +0000

Omnisdata has signed agreement with French company Kalyo about development of web-based GIS solution intended for telecommunication industry. The solution will be based on technology flashNavigator which will provide interactive visualization and analysis of telecommunication network information by implementing Geographical Information System (GIS) standards.

flashNavigator :: GIS/CAD rapid development framework

karabo.org.za

karteGradova.com

Web GIS Tutorial: WiGLE

Querying WiGLE data

Setting up the database

Setting up flashNavigator framework viewer

GIS analysis (WiFi distribution in Chicago)

WiFi Access Points display

Karabo

Kalyo

Immo-top Croatia

About flashNavigator

Erste nekretnine

WebMap service implementation - Erste nekretnine

Signed agreement with French company Kalyo