Open (Source) for Business?: My First Attempt at Deploying an Open Source Spatial Database

All modern geographers face the critical decision of  selection an appropriate GIS. There are many options from which to choose, and each offers unique benefits while invariable containing some drawbacks. Open source solutions  such as GrassGIS, Viking GIS, and my current favorite, Quantum GIS have the main advantage of being free to use, and allowing for great data portability between applications. In terms of close sourced GIS solutions, ESRI's ArcGIS rules the pack. ArcGIS is very prominently used by governments and other large organizations who want a solution that will offer reliability and that comes with support and complete documentation.

Well structured data is essential success in any GIS project.
All good databases are thoughtfully designed  in advnaced.

I cut my spatial teeth at University of California, Santa Barbara, where ESRI is the GIS of choice. Reflecting on my early GIS years, I feel somewhat shortchanged that my instructors did not expose me and my colleagues to open source GIS solutions. Once I graduated, I lost my school-provided ESRI seat license, and I needed to find a cheaper alternative to ESRI's products. And by cheaper, I mean free, because when it comes to GIS software, the options are either extremely expensive (ESRI) or free (most other GIS applications). 

After experimenting with a few options, I settled on Lisboa's Quantum GIS. Quantum GIS (qgis for short) has a clean and logical UI and is a comfortable switch for an ArcGIS user such as myself. I quickly discovered how to utilize the functionality I was looking for, and soon was using qgis to contently import, clip, rasterize, merge, and project, and export data.

Before too long, I had some large CSV datasets to work with for a consulting project. I converted the data into a shapefile, and quickly learned that shapefiles are not an effective storage format for 1.5 million data points, which unfortunately is the size of the data set with which I am working.

I wish I had more RAM. (Not actually me in this picture).

As my Dell laptop (with a humble Core i-3 chip and 4 gigs of slow RAM) tried desperately to crank away at spatial queries on this enormous shapefile, I become intimately acquainted with all varieties of program and system crashes. After a few unsuccessful days of booting and rebooting, I decided I needed to put my data in a more scalable format. 

Enter: The Spatial Database. 

Had I been using ESRI, a Geodatabase would have been just the ticket to manage such a large dataset. I fired up qgis and started looking for the "create GeoDB" function. Well, as it turns out, Geodatabases are a feature proprietary to ArcGIS, and ESRI does not embrace the sharing of functionality and open data standards which are common in the open source community. I was stunned to discover the Geodatabases as they exist in ArcGIS, are unique to ESRI products, and that a GeoDB cannot be access by any other application.

My assumption was that any database used to store spatial data was a Geodatabase. I was less than thrilled to learn that my relatively extensive experience working with Geodatabases would not apply to spatial databases in any other application... and every other spatial data application utilizes a common db protocol and ESRI alone does not embrace. Humph!

Soon I found myself in the world
of painful, general error messages. Help me, please!

How is a spatial database different from a "capital G" Geodatabase? After all, they serve largely the same function, in that they both are a fast way to store large amounts of spatial data in a hierarchical structure. ESRI's database solution is designed from the ground up to work with spatial data. It's very much a black-box solution - you give it data, and it lets you use that data in ArcGIS. A user does not have visibility into the configuration and implementation of the database - this is all handled under the hood by ArcGIS. As a user of open source GIS software, I learned that I would have to utilize the power of databases with my spatial data, I would have to roll my own database solution.

I have only a very general idea of how databases work: deploy the database, import your data, and then reference the database in an application to view and modify the data. So far, I have managed to get the database instance installed on my local machine. I selected POSTgres as my database system, and installed the POSTgis plugin to add spatial functionality to the database system. I used the included tool to store data (a single shapefile) in the database, and access that data from qgis. My understanding is that the main value of a database for spatial data comes in that data does not have to be contained within the shapefile format, but instead data of different types can be easily cross-referenced which allows for powerful analysis. Additionally, databases are much faster than shapefiles when manipulating large datasets, and are less prone to corruption during data writes.

I am still confused about the following:

• How does one relate shapefiles to each other in a spatial database?
• Can shapefiles of similar features (i.e. the same feature class) be combined upon import?
• What is the best way to import bulk geodata into a spatial database?
• How can I be sure data attributes are preserved?
• How do I move data from my database back to a shapefile for transfer to a collegue?
• Shapefiles are a great way to share data - is there a analog for sharing data from a database, or is export  to shapefile the best method?
• How can projections and coordinate reference issues best be handled?
• How do I modify data in a database (the db equivalent of editing a shapefile?)
• Do databases support topologically-aware feature editing?

As I continue to tinker and learn, I hope to answer most if not all of the above questions. In the mean time, if you have any suggestions, please do let me know!

Sailing on the San Joaquin River

With summer in full swing, there is no better time to get out for a sail! I recently went for a day cruise on the San Joaquin River in the heart of the region known locally as the delta. The winds were very light in the morning, so we had some time to float around and work on our tans, but thankfully a stiff breeze came in around 3 PM.

Overall, we covered 12 nautical miles, or about 14 miles, which is pretty good when you consider that we spent a lot of time sitting and waiting for a breeze.

Leaving Delta Sailing School's dock in Seven Mile Slough.

We got the sails up and made no progress upwind....

So we headed downwind for a nice cruise. 

Unfortunately, this meant we had to head
back upwind, requiring many tacks.

We finally got some nice log runs in before calling it a day.

Turkey Launches Satellite to Capture Earth Imagery

On 18 Dec 2012, Turkey successfully achieved orbit of their new earth observation satellite, Göktürk-2. According to an article on NASASpaceflight.com, the satellite was launched from the Jiuquan Launch Area 4 in China, and successfully achieved low earth orbit within hours.

Göktürk-2 launching aboard a two-stage Chinese rocket.

Göktürk-2 will capture both panchromatic (black and white) imagery with a resolution of 2.5 meters, and multi-spectral (color) imagery with a resolution of 10 meters. Multi-spectral imaging captures different wavelengths of light separately, allowing researchers to compare and analyze different types of light to gain detailed information about conditions across the earth's surface.  For example, multi-spectral satellites can map vegetation health by comparing how much visible light plants reflect with the amount of reflected  infrared light. Stressed vegetation reflects more visible light and absorbs more infrared light than healthy vegetation, allowing researchers to use imagery to identify areas where ailing plant health may hurt food production, or to identify the impacts of global climate change on sensitive habitats.

All of the software running on Göktürk-2 as well as 80% of the satellite's hardware was designed by the Scientific and Technological Research Council of Turkey (TÜBITAK), and assembly was completed by the TÜBITAK Space Technologies Research Institute (TÜBITAK UZAY) and Turkish Aerospace Industries (TUSAŞ). The satellite showcases Turkey's rising status as a technological powerhouse, and will prove to be a major strategic advantage by providing Turkey's Defense Department with remote imagery from countries around the globe.

Göktürk-2 in the lab before leaving earth.

Many countries already have satellites orbiting the globe for a variety of purposes, such as imagery collection, communications, and navigation. Over 200 publicly and privately owned satellites have been launched from the United States alone. Turkey no longer must rely solely on data provided by the international community to map and research important environmental phenomena, and can now produce such data first hand. Göktürk-2 is part of a family of satellites planned by Turkey to further facilitate data collection and augment existing intelligence resources.

Bill of Rights Does Not Apply to 66% of Americans

In their ongoing mission to defeat the terrorists, the Department of Homeland Security (DHS) has suspended the constitutional rights of 200 million Americans with a move that makes Senator Joseph McCarthy seem well reasoned and logical.

The Fourth Amendment of the Bill of Rights gives Americans protection against unreasonable search and seizure of their person and property. Just to review, here is the text of the fourth amendment as posted on the U.S. Government's document archive site (source).

Amendment IV

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

The DHS has claimed a long-standing right to search people crossing our international borders. Upon superficial consideration, this seems reasonable. But consider the smart phone, tablet, or laptop. These devices contain a person's most personal and sensitive data. If the data isn't stored on the device itself, than the device most likely will provide agents access to data stored online.

The DHS claims that these devices can be searched as part of a border crossing inspection. However, these devices contain access to an individual's most intimate and private data. By obtaining someone's device, one gains access to their email, phone records, bank accounts, photos, documents, and other private data. Therefore, searching one's laptop, tablet, or smartphone is not the same as searching their pockets and briefcase. In fact, searching an electronic device is more analogous to searching their entire home, office, and car all at once. By looking through a device, border agents can review and seize any and all of that person's private information.

An easy solution would be to only carry "dummy" devices when traveling internationally. For example, I can take a laptop with no personal data on it, and access my data online when I arrive at my destination. I can also encrypt my data, so if my computer is seized at the border, agents will only be able to see a scramble of nonsensical 1s and 0s.

Unfortuately, border searches are NOT limited to the border. The DHS claims that the internal border extends 100 miles inland, and therefore anyone within 100 miles of the border can be searched without probably cause and without a warrant. .66% of our nation's population (over 200 million people) lives within 100 miles of the border, and are therefore subject to search and seizure of their electronic devices and data at any time and place without a warrant.

Here is the ACLU's map of what it calls the "Constitution Free Zone". 

2 out of 3 Americans live in this Constitution Free Zone.
© American Civil Liberties Union

Do you live in it? Most of us do. 99% of Californians do, as do 88% of Arizonians. If you live in Connecticut, Delaware, Florida, Hawaii, Maine, Massachusetts, Michigan, New Hampshire, New Jersey or Rhode Island, then your entire state is a DHS search and seizure zone. So, you know, good luck with that.

 Therefore, most of us no longer have the right to avoid unreasonable search and seizure. So if and when you see those flashing reds and blues in your rear view mirror, and you're in the orange section on the map above, you'd best start deleting your hard drive and smart phone memory. Unless, of course, you are ok with the government having access to your entire digital life including your banking info, email, web browsing history, and location data. I, for one, do not consent to waive my 4th amendment rights. I have nothing to hide, but as the founding fathers intended, the U.S. Government cannot require me to prove that.

Thanks to Darlene Storm, whose ComputerWorld article on this topic inspired mine. Read her article here.

Why Can't (the carriers) Just Get Along?

No matter how hard they try, it seems that large companies have a tough time working together; as a company grows and its bureaucratic develops, the company seems less and less able to deliver products in a responsive manner.

Case in point: the wireless carriers. Sprint, Verizon, At&t and T-mobile all have network-based location capabilities, but none are available to app developers via a simple-to-use API. With this API, developers find it extremely difficult to utilize network-based location on their apps. The Wholesale Application Community (WAC) was a cross-carrier partnership who's goal was to allow any app to be used on any handset on any carrier. Under this Utopian coalition, a developer would build their app to WAC standards, and then would be able to deploy the app across carriers. It turns out that the carriers each had their own idea of how this should work, and after years of infighting, the WAC dissolved with few results for years of effort.

So what's a cross-carrier app developer to do? Well, if the end goal is to inoperative carrier-grade network location services, then companies like LOC-Aid are the answer.The carriers specialize in building infrastructure that operates with 'five 9s' reliability, and therefore innovation on the carrier side is a slow, painful process. LOC-Aid (for who I do contract work, full disclosure) specializes making carrier location services available through a modern, well-built API that app developers find easy to use. The carriers handle the back haul equipment and reliability, and LOC-Aid provides an innovative, high tech front which developers can use to bring the latest location technologies to their apps without direct carrier interaction.

The failure of the WAC shows how intermediary companies such as LOC-Aid continue to provide a highly valuable service to developers and end users. Maybe some day that telcos will learn to play nicely together (not likely), and until then, let someone like LOC-Aid handle the tough carrier negotiations!

Just How Big is Los Angeles?

Los Angeles is the urban sprawl capital of California. Simply climb to the top of any of the many mountain ranges which surround the city; one can see freeways, blocks of buildings, and endless ballet of cars, usually through a thick blanket of smog. 

Los Angeles Skyline and San Gabriel Mountains
Image © Charles Abbott, 99 Photography

Ask anyone who lives in the area: no on gives a second thought to driving 45 minutes at the top of the hat, or assuming that traffic will delay one's arrival for at least another 30 minutes. These constant traffic snarls and delays are simply a part of life for the average resident of the area.

The Hollywood Sign
Attribution: User Oreos at Wikimedia Commons

So why is traffic also so bad? How did Los Angeles earn the reputation as one of the most trafficked city in the country? Well, take a look at the map below and see how much larger of a geographic area the city covers when compared with other cities. A resident of LA simply has a much larger area which they will be expected to transverse for work, school, or pleasure. And because everything is so spread out, many areas are impractical for mass transit, keeping everyone stuck in their cars.

Size of Los Angeles vs. other urban centers
Attribution: Archinect News

Exploring Sonoma Valley Wine Country - On a Segway!

When it comes to wine tasting, I am traditional. I like to spend time taking in the rich aromas, colors, and tastes of each wine. I chat with the workers at the vineyard to get a sense of the love and care put into production. When possible, I always like to see the grapes themselves out in the field. (Not that I know much about wine tasting, but I try and copy what I see other 'wine snobs' doing!) 

Thankfully, my traditional tastes only apply in the tasting room and not to the method of transport used to get there, because I arrived to each vineyard on the back of a wild electronic steed, more commonly known as a Segway!

My dad had some about-to-expire coupons he'd bought on groupon for this wine tour by a local firm called Segway of Healdsburg, so he insisted I use them. "Ok, If you insist," was my reply. The tour started off in a park and ride parking lot in Healdsburg, right off HWY 101. The Segways come to you in a van, charged up and ready to go. They guides teach you how to ride in the safety of the big empty parking lot, which was great because I had never been on a Segway before, but thankfully it was super easy. We took a quick trip up a side street to make sure our Segwaying skills were up to snuff, and then we were off!

Location of the tour's starting point in relation to the bay.

Overview of our route.

My tracks when I was practicing back
and forth in the parking lot!

And the side road we practiced on.

After getting the hand of it, we headed down a main road to Limerick Lane, a quaint country road dotted with small vineyards. Segways are legally bicycles, so we stuck to the bike lane, but boy did we get some funny stares from passing motorists!

Aerial View of Viszlay Vineyards. 

Our first stop was Viszlay Vineyards. John, the owner, conducted our tasting himself around a shady outdoor table adjacent to the fields of vines. John told us he is a recent transplant from Chicago who decided to try his hand at wine making, and boy, he is doing something right! I am not a fan of Chardonnay, but we tasted a Chardonnay so crisp and delicious that I had to purchase a bottle. It felt great to buy something in cash directly from the person who had made it - this is how agriculture should work! And of course, we got him to sign the bottle.

Aerial View of Limerick Lane Cellars, with its bottle-shaped pool!

Our second and final stop was just down the road to a place called Limerick Lane Cellars. The tasting room here was much more formal, with dark woods and ornate decorations. The tasting assistant was very helpful and knowledgeable, and he gained my respect with his honesty; his personal favorite on their tasting list was also the cheapest, so he didn't seem to be trying to upsell us. 

Limerick Lane Cellars has been around for a while; some of their vines have been producing grapes since 1910! These old vines are unique because they were keep alive during prohibition when most grape vines in the area were ripped out. Due to their age, their roots reach down to the groudwater, meaning that these vines are non-irrigated. The reliance on groundwater gives the fruit a special kind of taste which you would not get with irrigated vines! We ended up buying a bottle of Zin from vines planted in the 1930's, and I cannot wait for a special occasion to drink it.

Overall, the Segway tour was a blast! I'd highly recommend taking the tour with Segway of Healdsburg, but even if you go it alone to Sonoma Wine Country, be sure to stop by Viszlay and Limerick Lane Cellars! Click here for photos of the trip taken by our tour guide!

Here is an interactive map of our route: The pins show the starting place and the location of both vineyards. Click on them for more info!

View Larger Map