Twitter Entices Users to Share Location Data

Twitter is reportedly experimenting with a new feature which will show nearby tweets on a map. This feature, currently referred to as "Nearby," appears designed to encourage users to geo-tag their tweets by providing the user value in sharing their location information.

Currently, users may geo-tag a tweet, but this feature is not enabled by default. Twitter is attempted to make money by selling advertising, and having more location info about more users will make Twitter's advertising inventory more value.

Users get the benefit of being able to see what is going on around them. This may be immensely useful for finding out about current news and further cements Twitter's position as the go-to source for news happening right now.

Imagine seeing a plume of smoke rising from ten blocks away - the first thing most of us tweeters would do is search Twitter for likely keywords, i.e. "smoke SF Financial district." Even the most artfully crafted searches still leave the user sorting through a huge amount of content. Twitter's new "nearby" feature will allow users to see tweets based on their geo-tagged location, making it much easier to see tweets being sent from the area of interest.

Note that geo-location enabled tweets have been possible for some time now. Several years ago, I worked with a developer who was making a "heat map" of tweets containing a given keyword. Higher tweet densities were represented with brighter colors on the map. I do not know the technical solution this developer had implemented, but so far as I can tell, Twitter's new API does not add new functionality, but rather, makes existing functionality available to users posting from 3rd party clients.

I am excitedly waiting to see the community's response to this new feature. New features often lead to new and previously un-imagined features, but the privacy challenges are daunting....


Good luck, Twitter!

Winter Cruise around Angel Island

We received our first winter storm recently here in the San Francisco Bay. A cold low pressure system moved in from Alaska and brought with it freezing temps, driving winds, and several inches of rain. I found this inclement weather to be a welcomed break from the typical winter pattern cool, calm, hazy days. The worse of the rain moved past after a few days but left behind strong winds. I couldn't help but take advantage of this break from typical winter calm to go for a sail.

After grabbing some sandwiches and bundling up against the cold, I headed out of Loch Lomond Marina in San Rafael and set a source south to Angel Island. Taking advantage of the unusual wind out of the north west, we had a leisurely cruise under the Richmond San Rafael Bridge and down towards the bustle of one of the world's most iconic waterways. 

Our route on the San Francisco Bay.

Leaving Loch Lomand Marina  - Must make sure to
make the turn WIDE to stay in deep water!

Loch Lomond Marina is difficult for sailboats. The marina sits about one mile from the deep water channel on the edge of a tidal mud flat. The water depth leading up to the marina is about two feet, which is tricky because the sailboat I use requires five feet of depth to pass safety. Fortunately, there is a channel dredged across the mud flat which is deep enough to navigate in a sailboat. Unfortunately, this channel is poorly marked and very narrow, and the only sure way to know you're not in the channel is once your vessel becomes mired in mud.

As you can see in the above picture, there is a small channel out of the marina which feeds into a slightly larger channel at a ninety degree angle (the blue line is the path taken by my vessel through these channels). Both channels are marked poorly, and the areas immediately surrounding this turn are especially shallow. Too wide of a turn will run one aground, as will too narrow of a turn. I have "played in the mud" here on several occasions, and have learned to not even attempt entry at low tide. 

Unusual NW winds allowed us to sail
under the Richmond Bridge.

Successful transit of the San Rafael Bridge under sail power was possible because of the storm's unusual winds out of the north and west. The prevailing wind, out of the south and west, is readily blocked by the bridge, making it difficult to safely cross without starting the motor. The wind out of the north, however, was able to keep the sails full all the way under the bridge - a rare treat!

Clearing Raccoon Straights.

Squeezed between Angel Island and passing some container ships.

Rounding Angel Island brought us head to head with a line of oncoming container ships. Luckily I was able to hug the coast and stay out of their way (and their wakes).

The fun upwind portion of the sail.

After rounding the southern tip of Angel Island and taking in some great views of San Francisco, we headed back upwind. Sailing against the wind is much rougher and more difficult than sailing with the wind. A sailboat cannot move directly into the wind, so we had to tact back and forth across the wind many times to get back upwind to where we started.

Going upwind is a lot more work than going downwind.

Check out our recorded speed (the blue line).

We averaged 4.4 mph, which is not bad for the entry-level 27 foot sailboat we were sailing. During some of the heavier gusts, we hit close to 8 mph!

Complete route map - zoomable and scrollable!

View Larger Map

Open (Source) for Business?: My First Attempt at Deploying an Open Source Spatial Database

All modern geographers face the critical decision of  selection an appropriate GIS. There are many options from which to choose, and each offers unique benefits while invariable containing some drawbacks. Open source solutions  such as GrassGIS, Viking GIS, and my current favorite, Quantum GIS have the main advantage of being free to use, and allowing for great data portability between applications. In terms of close sourced GIS solutions, ESRI's ArcGIS rules the pack. ArcGIS is very prominently used by governments and other large organizations who want a solution that will offer reliability and that comes with support and complete documentation.

Well structured data is essential success in any GIS project.
All good databases are thoughtfully designed  in advnaced.

I cut my spatial teeth at University of California, Santa Barbara, where ESRI is the GIS of choice. Reflecting on my early GIS years, I feel somewhat shortchanged that my instructors did not expose me and my colleagues to open source GIS solutions. Once I graduated, I lost my school-provided ESRI seat license, and I needed to find a cheaper alternative to ESRI's products. And by cheaper, I mean free, because when it comes to GIS software, the options are either extremely expensive (ESRI) or free (most other GIS applications). 

After experimenting with a few options, I settled on Lisboa's Quantum GIS. Quantum GIS (qgis for short) has a clean and logical UI and is a comfortable switch for an ArcGIS user such as myself. I quickly discovered how to utilize the functionality I was looking for, and soon was using qgis to contently import, clip, rasterize, merge, and project, and export data.

Before too long, I had some large CSV datasets to work with for a consulting project. I converted the data into a shapefile, and quickly learned that shapefiles are not an effective storage format for 1.5 million data points, which unfortunately is the size of the data set with which I am working.

I wish I had more RAM. (Not actually me in this picture).

As my Dell laptop (with a humble Core i-3 chip and 4 gigs of slow RAM) tried desperately to crank away at spatial queries on this enormous shapefile, I become intimately acquainted with all varieties of program and system crashes. After a few unsuccessful days of booting and rebooting, I decided I needed to put my data in a more scalable format. 

Enter: The Spatial Database. 

Had I been using ESRI, a Geodatabase would have been just the ticket to manage such a large dataset. I fired up qgis and started looking for the "create GeoDB" function. Well, as it turns out, Geodatabases are a feature proprietary to ArcGIS, and ESRI does not embrace the sharing of functionality and open data standards which are common in the open source community. I was stunned to discover the Geodatabases as they exist in ArcGIS, are unique to ESRI products, and that a GeoDB cannot be access by any other application.

My assumption was that any database used to store spatial data was a Geodatabase. I was less than thrilled to learn that my relatively extensive experience working with Geodatabases would not apply to spatial databases in any other application... and every other spatial data application utilizes a common db protocol and ESRI alone does not embrace. Humph!

Soon I found myself in the world
of painful, general error messages. Help me, please!

How is a spatial database different from a "capital G" Geodatabase? After all, they serve largely the same function, in that they both are a fast way to store large amounts of spatial data in a hierarchical structure. ESRI's database solution is designed from the ground up to work with spatial data. It's very much a black-box solution - you give it data, and it lets you use that data in ArcGIS. A user does not have visibility into the configuration and implementation of the database - this is all handled under the hood by ArcGIS. As a user of open source GIS software, I learned that I would have to utilize the power of databases with my spatial data, I would have to roll my own database solution.

I have only a very general idea of how databases work: deploy the database, import your data, and then reference the database in an application to view and modify the data. So far, I have managed to get the database instance installed on my local machine. I selected POSTgres as my database system, and installed the POSTgis plugin to add spatial functionality to the database system. I used the included tool to store data (a single shapefile) in the database, and access that data from qgis. My understanding is that the main value of a database for spatial data comes in that data does not have to be contained within the shapefile format, but instead data of different types can be easily cross-referenced which allows for powerful analysis. Additionally, databases are much faster than shapefiles when manipulating large datasets, and are less prone to corruption during data writes.

I am still confused about the following:

• How does one relate shapefiles to each other in a spatial database?
• Can shapefiles of similar features (i.e. the same feature class) be combined upon import?
• What is the best way to import bulk geodata into a spatial database?
• How can I be sure data attributes are preserved?
• How do I move data from my database back to a shapefile for transfer to a collegue?
• Shapefiles are a great way to share data - is there a analog for sharing data from a database, or is export  to shapefile the best method?
• How can projections and coordinate reference issues best be handled?
• How do I modify data in a database (the db equivalent of editing a shapefile?)
• Do databases support topologically-aware feature editing?

As I continue to tinker and learn, I hope to answer most if not all of the above questions. In the mean time, if you have any suggestions, please do let me know!

Sailing on the San Joaquin River

With summer in full swing, there is no better time to get out for a sail! I recently went for a day cruise on the San Joaquin River in the heart of the region known locally as the delta. The winds were very light in the morning, so we had some time to float around and work on our tans, but thankfully a stiff breeze came in around 3 PM.

Overall, we covered 12 nautical miles, or about 14 miles, which is pretty good when you consider that we spent a lot of time sitting and waiting for a breeze.

Leaving Delta Sailing School's dock in Seven Mile Slough.

We got the sails up and made no progress upwind....

So we headed downwind for a nice cruise. 

Unfortunately, this meant we had to head
back upwind, requiring many tacks.

We finally got some nice log runs in before calling it a day.

Turkey Launches Satellite to Capture Earth Imagery

On 18 Dec 2012, Turkey successfully achieved orbit of their new earth observation satellite, Göktürk-2. According to an article on NASASpaceflight.com, the satellite was launched from the Jiuquan Launch Area 4 in China, and successfully achieved low earth orbit within hours.

Göktürk-2 launching aboard a two-stage Chinese rocket.

Göktürk-2 will capture both panchromatic (black and white) imagery with a resolution of 2.5 meters, and multi-spectral (color) imagery with a resolution of 10 meters. Multi-spectral imaging captures different wavelengths of light separately, allowing researchers to compare and analyze different types of light to gain detailed information about conditions across the earth's surface.  For example, multi-spectral satellites can map vegetation health by comparing how much visible light plants reflect with the amount of reflected  infrared light. Stressed vegetation reflects more visible light and absorbs more infrared light than healthy vegetation, allowing researchers to use imagery to identify areas where ailing plant health may hurt food production, or to identify the impacts of global climate change on sensitive habitats.

All of the software running on Göktürk-2 as well as 80% of the satellite's hardware was designed by the Scientific and Technological Research Council of Turkey (TÜBITAK), and assembly was completed by the TÜBITAK Space Technologies Research Institute (TÜBITAK UZAY) and Turkish Aerospace Industries (TUSAŞ). The satellite showcases Turkey's rising status as a technological powerhouse, and will prove to be a major strategic advantage by providing Turkey's Defense Department with remote imagery from countries around the globe.

Göktürk-2 in the lab before leaving earth.

Many countries already have satellites orbiting the globe for a variety of purposes, such as imagery collection, communications, and navigation. Over 200 publicly and privately owned satellites have been launched from the United States alone. Turkey no longer must rely solely on data provided by the international community to map and research important environmental phenomena, and can now produce such data first hand. Göktürk-2 is part of a family of satellites planned by Turkey to further facilitate data collection and augment existing intelligence resources.

Bill of Rights Does Not Apply to 66% of Americans

In their ongoing mission to defeat the terrorists, the Department of Homeland Security (DHS) has suspended the constitutional rights of 200 million Americans with a move that makes Senator Joseph McCarthy seem well reasoned and logical.

The Fourth Amendment of the Bill of Rights gives Americans protection against unreasonable search and seizure of their person and property. Just to review, here is the text of the fourth amendment as posted on the U.S. Government's document archive site (source).

Amendment IV

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

The DHS has claimed a long-standing right to search people crossing our international borders. Upon superficial consideration, this seems reasonable. But consider the smart phone, tablet, or laptop. These devices contain a person's most personal and sensitive data. If the data isn't stored on the device itself, than the device most likely will provide agents access to data stored online.

The DHS claims that these devices can be searched as part of a border crossing inspection. However, these devices contain access to an individual's most intimate and private data. By obtaining someone's device, one gains access to their email, phone records, bank accounts, photos, documents, and other private data. Therefore, searching one's laptop, tablet, or smartphone is not the same as searching their pockets and briefcase. In fact, searching an electronic device is more analogous to searching their entire home, office, and car all at once. By looking through a device, border agents can review and seize any and all of that person's private information.

An easy solution would be to only carry "dummy" devices when traveling internationally. For example, I can take a laptop with no personal data on it, and access my data online when I arrive at my destination. I can also encrypt my data, so if my computer is seized at the border, agents will only be able to see a scramble of nonsensical 1s and 0s.

Unfortuately, border searches are NOT limited to the border. The DHS claims that the internal border extends 100 miles inland, and therefore anyone within 100 miles of the border can be searched without probably cause and without a warrant. .66% of our nation's population (over 200 million people) lives within 100 miles of the border, and are therefore subject to search and seizure of their electronic devices and data at any time and place without a warrant.

Here is the ACLU's map of what it calls the "Constitution Free Zone". 

2 out of 3 Americans live in this Constitution Free Zone.
© American Civil Liberties Union

Do you live in it? Most of us do. 99% of Californians do, as do 88% of Arizonians. If you live in Connecticut, Delaware, Florida, Hawaii, Maine, Massachusetts, Michigan, New Hampshire, New Jersey or Rhode Island, then your entire state is a DHS search and seizure zone. So, you know, good luck with that.

 Therefore, most of us no longer have the right to avoid unreasonable search and seizure. So if and when you see those flashing reds and blues in your rear view mirror, and you're in the orange section on the map above, you'd best start deleting your hard drive and smart phone memory. Unless, of course, you are ok with the government having access to your entire digital life including your banking info, email, web browsing history, and location data. I, for one, do not consent to waive my 4th amendment rights. I have nothing to hide, but as the founding fathers intended, the U.S. Government cannot require me to prove that.

Thanks to Darlene Storm, whose ComputerWorld article on this topic inspired mine. Read her article here.