Monday 29 December 2008

The more Microsoft tightens its grip, the more internet users will slip through its fingers

I was doing a fresh install of windows xp on my old laptop for my (not-so computer literate) mum on the weekend.

One of the first things she said to me was (paraphrased):

"So where's that Firefox that we had on the old one?"

So I happily went off and downloaded a copy straight away :-)

Friday 5 December 2008

Database Module: Going, Going, Gone!

Here's another little FDO Toolbox update with a small backstory.

When I first wrote the original ADO.net database module for FDO Toolbox, what I wanted was the ability to integrate FDO data sources with non-FDO data sources (eg. Joining an SDF file with a MS Access database).

For the next release of FDO Toolbox, this module will no longer exist.

The reason: OSGeo.ODBC

I have come to realise that I have been essentially duplicating functionality that already exists in the ODBC FDO Provider since the beginning of time. The provider already wrappers non-FDO data sources as FDO data sources, so I was basically reinventing the wheel (poorly).

So the plan now for non-spatial data support is currently the following:
  1. Give the ODBC Provider some special attention in the Express Module. There will be connection builders for commonly used formats like MS Access databases, Excel Spreadsheets, CSV files, DSNs, etc.
  2. Implementing Class to Feature Class conversion, which will replace the old DB Table to Point Feature conversion, since the ODBC provider already exposes the DB Tables as non-feature classes.
  3. Supporting joins between FDO data sources, again since the ODBC provider has done all the work for us.
  4. Implementing configuration support. This is just another FDO feature that has yet to be exposed by FDO Toolbox.
Items 2, 3 and 4 won't make the next release, but 1 will make it in some form or another.

Tuesday 2 December 2008

You are what you write

ISTJ - The Duty Fulfillers

The responsible and hardworking type. They are especially attuned to the details of life and are careful about getting the facts right. Conservative by nature they are often reluctant to take any risks whatsoever.

The Duty Fulfillers are happy to be let alone and to be able to work int heir own pace. They know what they have to do and how to do it.









What does your blog say about you?

Tuesday 11 November 2008

FDO Toolbox: Rebooted

Today I took off and nuked the site from orbit. After the dust settled, I landed a drop of the currently re-written FDO Toolbox into the now quite barren and empty trunk.

Here's a summary of what has changed.
  • The application now revolves around SharpDevelop Core. The decision to rewrite FDO Toolbox around SharpDevelop Core was an important (and quite easy) decision to make. Not only is SharpDevelop Core an extensibility framework, it also provides other helpful utilities like logging, file picking, unified resource access and other goodies. The application also uses the Text Editor component of SharpDevelop for editing of any text files (eg. XML feature schemas)
  • The application now uses FDO 3.4.0 (build G025), it will no longer compatible with FDO 3.3.x as it is using APIs new to 3.4.
  • Most of the UI components have been rewritten to be more cleaner and easier to maintain, with special mention to the Data Preview control.
  • The application is more friendly to localization as most hard-coded strings have been moved to resource files. There are still some strings floating around that will need to be moved.
  • The ETL API has been overhauled, the revised ETL API is based on Rhino ETL with modifications to support and handle geometries/features. The existing task infrastructure (UI and classes) is being rewritten to use this API.
There are still quite a few features from the previous version that have not made it into this version as I am still in the process of porting them over. You can now checkout or svn update from the trunk to see the current work in progress. For archival purposes, the 0.6.0 code-base now resides in a "legacy" branch

Here's some screenshots to tickle your fancy

The user interface (now with a toolbar!):

Add-In management using the SharpDevelop Add-In Manager

The new Data Preview control


Some small improvements to the Generic connect/create data store controls.








Wednesday 5 November 2008

Another FDO Toolbox update

Here's the current state of affairs on the FDO Toolbox front.

1. FDO Toolbox development is back in motion

Life has been less hectic than it was a few months ago, and as a result, you will be seeing more regular updates to FDO Toolbox.

2. FDO Toolbox is currently being re-written

This one will need a bit of explanation.

One of my original goals for FDO Toolbox has always been extensibility. This extensibility existed in the form of extension modules (case in point: The MapGuide extension module), but there wasn't really much thought put into the original design from an extensibility standpoint (I will admit that it was pretty much haphazardly put together).

Also most of the UI components of FDO Toolbox were developed in a similar ad-hoc fashion and as a result most of the UI code is a messy affair (example: The Data Preview component)

So I have made the tough (but ultimately beneficial) decision to take off and nuke the site (most of the code-base) from orbit. Here's a summary of the planned changes I intend to make:
  • SharpDevelop Core (ICSharpCode.Core) will be used as the extensibility framework for FDO Toolbox. This is a framework that has been proven to be both simple, powerful and versatile in my evaluation of the framework. Also it's one less component to worry about from a maintenance standpoint. I'm not a fan of reinventing the wheel. If a library/component exists that does what I require and has a proven track record, I will be using it. This is one such case.
  • On the UI front, most if not all of the components will be rewritten in a Model-View-Presenter fashion. This will make maintenance and testability much easier.
  • The FDO Toolbox Core library and command-line utilities will mostly survive intact, with refactorings to adapt to the new code base.
  • The ETL API will be overhauled and replaced with the proposed replacement API.

  • /branches/0.6 will be where the code base in its current form will reside. /trunk will be overhauled and replaced with the new code base.
As a result, v0.6.0 will be the last release of FDO Toolbox in its current form.

3. FDO Toolbox is almost certain to be moving to a new home.

A few weeks ago I submitted a RFC to make FDO Toolbox an official part of the FDO project. The gist of the RFC is that the FDO Toolbox codebase will move to OSGeo, while the existing Google Code repository will be the home for the development of FDO Toolbox extensions.

It is currently going through the voting phase, which looks to have unanimous approval. So expect an announcement soon from OSGeo or myself.

EDIT: Mono.AddIns is starting to look more attractive as it is based on SharpDevelop core and has gone through more refinements.

EDIT #2: After playing around with Mono.Addins, it doesn't look as attractive anymore (mainly because it requires a whole lot of Mono-specific libraries to get the whole thing working!)

Wednesday 29 October 2008

FDO Toolbox v0.6.0

It's been a while, but I've found some time to nut out some of the outstanding issues/enhancements since the last release. As a result, here's a new release coming at you!

New features

1. Visual data preview support

You can now preview spatial data the way it is meant to be previewed! :-) This uses the SharpMap library for rendering with a custom SharpMap provider that does the job of translating FDO geometries to SharpMap ones for rendering. Map previews are disabled for non-spatial queries.

2. MapGuide integration

There has been requests for some kind of support/integration with MapGuide (since they are closely related). This extension module allows you to connect to a MapGuide server and examine/preview its feature sources.

Please note that this is still experimental and still has some reliability issues. To use this extension module, you will have to manually load the MGModule.dll file or edit Modules.xml to auto-load the extension module.

3. Schema alteration support

Sometimes when bulk copying from one source to another, there are some things in the source schema that just cannot be applied to the target because it is unsupported. Now when you bulk copy from one source to another, any incompatibilities encountered will be presented to the user with the option of fixing said incompatibilities.


Other changes


  • Data Preview: You will be warned about potentially large result sets. This is controlled by a preference value: PREF_INT_WARN_DATASET
  • The MyMeta library has been replaced with raw ADO.net. What this means is that there is a much improved performance when retrieving table/column information from ADO.net data sources.
  • All application specific classes and components have been extracted out of FdoToolbox.Core.dll into FdoToolbox.Base.dll
  • Bulk copy overrides for SQL Server 2008
  • API Documentation for FdoToolbox.Core (CHM format)
  • Skeleton user documentation (also CHM format). More content will be added in the future.

Get it while it's hot!
Report bugs / feature requests

Saturday 25 October 2008

Brainstorming a cleaner ETL API for FDO Toolbox

Something that I've been brainstorming on is a cleaner API for doing ETL (Extract, Transform, Load) operations in FDO Toolbox.

Currently, you have a bunch of classes that does a series of pre-defined transformation tasks (eg. Bulk Copy, Join with Database, etc). There is very little you can do in terms of customisation, and there is no real way in the current API to "chain" certain tasks together.

Also, additional transformation classes would have to be written to cater for additional types of transformation.

The proposed solution

The revised ETL API is heavily influenced by the pipe-and-filters pattern. A bulk copy task for example would now be done like this:


//Setup
IConnection src = ...;
IConnection dest = ...;

ISelect select = src.CreateCommand(CommandType.CommandType_Select) as ISelect;
select.SetFeatureClass(daKlass);
IFeatureReader reader = select.Execute();

//Create the ETL pipeline

IFdoInput input = new FdoFeatureInput(reader);
IFdoOutput output = new FdoFeatureOutput(dest, daKlass);

FdoTransformationPipeline pipeline = new FdoTransformationPipeline();

pipeline.RegisterOperation(input); //Input source
pipeline.RegisterOperation(output); //Output source

pipeline.Execute();


That connects an input source with an output source and all features read from the input source are written to the output source.

Pluggability

What this revised API allows for is much finer control of the ETL process by allowing you to "plug in" additional transformation steps into the pipeline.

For example, sometimes copying data from one source to another requires renaming of property names. Modifying the above sample:


IFdoInput input = new FdoFeatureInput(reader);
IFdoOutput output = new FdoFeatureOutput(dest, daKlass);
IFdoRenameFilter ren = new FdoRenameFilter();
ren.AddRenameRule("NAME", "OWNER_NAME");

FdoTransformationPipeline pipeline = new FdoTransformationPipeline();

pipeline.RegisterOperation(input); //Input source
pipeline.RegisterOperation(ren); //Rename filter
pipeline.RegisterOperation(output); //Output source

pipeline.Execute();


That code will rename the property "NAME" of any features passing through to "OWNER_NAME"

Or perhaps, sometimes the actual property values themselves require substitution. Modifying the same sample as above:


IFdoInput input = new FdoFeatureInput(reader);
IFdoOutput output = new FdoFeatureOutput(dest, daKlass);
IFdoSubstitutionFilter sub = new FdoSubstituionFilter();
sub.AddSubstituion("NAME", "SMITH", "BLOGGS");

FdoTransformationPipeline pipeline = new FdoTransformationPipeline();

pipeline.RegisterOperation(input); //Input source
pipeline.RegisterOperation(sub); //Substitution filter
pipeline.RegisterOperation(output); //Output source

pipeline.Execute();


This will now copy from input to output, while replacing all instances of "SMITH" to "BLOGGS" in any "NAME" property that the substitution filter will encounter.

Splitting/Forking Pipelines

The IFdoFork interface allows you to split the pipeline into multiple processing pipelines. So say we wanted to segregate all the features where NAME = "SMITH", the above code snippet would then be:


IFdoInput input = new FdoFeatureInput(reader);
IFdoOutput output = new FdoFeatureOutput(dest, daKlass);

//Create fork
IFdoFork fork = new FdoFork();
//Create hell
IFdoOperation hell = new FdoNullOperation();

//Set the default pipeline
fork.SetDefaultPipeline("Default");
//Add the hell pipeline
fork.RegisterPipeline("Hell", hell);
//Send all features where NAME = 'SMITH' to hell. Send others through to the default pipeline
fork.RegisterAttributeCondition("NAME = 'SMITH'", "Hell");

FdoTransformationPipeline pipeline = new FdoTransformationPipeline();

pipeline.RegisterOperation(input); //Input source
pipeline.RegisterOperation(fork); //Fork, all smiths go to hell, others go to output source
pipeline.RegisterOperation(output); //Output source

pipeline.Execute();


This will read all features from the input source, send all features where NAME = 'SMITH' to the FdoNullOperation (like sending data to /dev/null in *nix) and send the rest on its merry way to the output source.

Extensibility

Here is where the revised API will be a much better. All pipeline operations implement a simple interface called IFdoOperation:


/// <summary>
/// FDO feature operation interface. A operation does a processing task on each
/// feature that is passed to it.
/// </summary>
public interface IFdoOperation
{
/// <summary>
/// Processes the set of features
/// </summary>
/// <param name="features">The set of features to process</param>
/// <returns>The processed set of features</returns>
IEnumerable<FdoFeature> Process(IEnumerable<FdoFeature> features);
}


Thus to create your own transformation operations, you just have to implement this interface and provide your own implementaion of Process()

You might be wonder what on earth is a FdoFeature? FdoFeature is the fundamental unit in the ETL pipeline. It encapsulates what is read from the FDO IFeatureReader from the input source and what gets inserted on the output source.

Finally it is the thing that travels through each operation in the pipeline, getting manipulated along the way. The interface of FdoFeature currently looks like this (conceptually):


public class FdoFeature : ICloneable
{
public FdoFeature();
public void Clear();
public OSGeo.FDO.Expression.ValueExpression this[string propertyName] { get; set; }
public void Rename(string propertyName, string newPropertyName);
public void Remove(string propertyName);
public object Clone();
public OSGeo.FDO.Commands.PropertyValueCollection ToValueCollection();
}


As you can see, it is effectively a ValueExpression dictionary.

Conclusion

As you can see, the revised ETL API is much more cleaner and modular than the current one. There are still a few things to iron out conceptually (eg. how to merge pipelines? Is it even possible?)

Also there is a minor thing of these code samples are living in my head and not in a Visual Studio project :-) So watch this space.

Tuesday 7 October 2008

I'll create a GUI interface in Visual Basic...

Warning: The following scene may be offensive to ALL programmers. It contains:

V) Visual Basic
C) Clueless crime scene investigators

Monday 29 September 2008

FDO Toolbox status update

It's been a while since the last release of FDO Toolbox. Previously, I stated that things would be business as usual. As ambitious as that statement was, the sad reality is that since starting my 2nd software development gig, my work-related projects have taken higher precedence over my personal projects, and when there are a lot of work related projects to get out the door, that leaves little room to continue development of FDO Toolbox (as you would imagine)

Does this mean the FDO Toolbox project is dead? No way! Rather the correct term would be that the project will be "on ice" to be thawed out (hopefully) in the near forseeable future. In the meantime I am happy to accept any patches or any applications for project membership (on a case-by-case basis of course) to continue development of this tool that has so much potential.

Thursday 18 September 2008

Gettin' swiggy wit it

Besides working on FDO Toolbox, another thing I've been doing lately is trying to learn SWIG and attempting to grok all the nice things you have to deal with from trying to wrap an API written in C++ :-)

One of the oddities I have noticed with the FDO API is that the managed bindings are coded "by-hand" unlike its geospatial cousin, which uses SWIG to do all the dirty work. The current bindings for FDO has a few shortcomings.

1) Because of the "coded by hand" nature, the Managed wrapper API is almost always behind the C++ version. Cases in point: Missing methods/properties and the FDO Expression Engine. Us .net people still can't get to use them yet!

2) The current .net wrapper is not inherently "portable", in the sense that I could run a certain .net application using the current managed FDO wrappers in an alternate .net environment. This is because the current FDO wrapper is a mixed-mode assembly, meaning it is inextricably tied to windows. At least with the SWIG approach, things are done via P/Invoke which is a more portable method of managed/unmanaged interop (in that we can attempt to make it portable).

3) Not a lot of language variety. Only .net and Python bindings exist, which are both in in various states of neglected/lagging maintenance and cruftiness. There are some really useful java geospatial applications out there that can be even more useful with FDO integration! Also, a good swig wrapper can serve as a useful template for creating bindings to other languages in the future.

FDO is a wonderful and ambitious API. But for FDO to really succeed, I truly believe that it needs more than a .net wrapper.

Wednesday 10 September 2008

Back to the front

Next week I am returning to the place I thought I would never be going back to. (I guess they got wind of what I've been working on since I left).

It's a strange, slightly awkward feeling going back to your previous job, seeing people who you thought you may never see again, and seeing the new faces that have popped up since you left.

In light of these turn of events, what will happen to the FDO Toolbox project?







































Nothing. Business as usual. :-)

Friday 5 September 2008

FDO, MapGuide. I pronounce you husband and wife.

One of the other major new features being currently worked on at the moment is integration with MapGuide Open Source/Enterprise.

Since MapGuide itself uses FDO for data access and its Feature Service API being similar to FDO. FDO Toolbox integration with MapGuide is a very useful feature to have. This integration exists in the form of an Extension Module.

One of the original design goals of FDO Toolbox apart from being a GUI front-end to FDO, is to be as extensible as possible. The MapGuide extension module is more-or-less a realisation of this goal.

To enable MapGuide integration, you would load the MGModule.dll via the Load Extension command.



Once loaded you'll notice something has changed. A new MapGuide application menu and a MapGuide Servers node in the Object Explorer



Now you can establish a connection to a MapGuide Server by right clicking the MapGuide Servers node and selecting the Connect to a remote MapGuide Server command (also available from the MapGuide menu)



From here it's like starting up MapGuide Studio or MapGuide Maestro. Provide the Site URL, Username, and Password.



And now you'll see all the feature sources for that particular server.

You can right click any feature source to bring up the Data Preview command


This will load up a Feature Source Preview tab for that particular feature source. This is very similar to the Data Preview tab for FDO connections, with the following differences.
  • There is no Expression Editor support for constructing filters. The reason is because the expression functions are not exposed via the mapagent, thus we cannot determine what expression functions are available for us to use. Nevertheless, the filter in MapGuide is the same as the filter in FDO. For MapGuide Open Source 1.2 or earlier expression functions do not exist.
  • The map preview tab will open the Schema Report page of the MapGuide web tier.

This is just only the tip of the iceberg as far as MapGuide integration goes. There are still many possible ideas yet to be realised.

If you work with MapGuide on a consistent basis then I invite you check out and play around with the latest trunk build from subversion and to contribute any suggestions and/or ideas on how we can best leverage this FDO/MapGuide "marriage" (and you were wondering what the title was all about!? :-)

Edit: A special thanks goes out to Kenneth Skovhede, author of the MapGuide Maestro API library for which this feature cannot exist without.

Visual Data Preview in FDO Toolbox

As a result of a successful proof of concept, I have just landed an update on the svn trunk that allows you to visually preview data sources in addition to the standard grid view.

Here's what a standard query looks like.


And now here's what that same query looks like in the map view.



The map view reflects the contents of the result set, so if I were to query all countries starting with A, I would get the following:


There are stills a few kinks to be ironed out. The map preview is known to fail if there happens to be invalid geometries in the data source. Also the map preview will be slower on large results (naturally).

Expect to see this feature in the next release or for the adventurous, you can check out the latest subversion revision.

Tuesday 26 August 2008

FDO Toolbox v0.5.8

Another release, another load of new features:
  • You can now save data preview results into a new SDF file
  • Bulk Copy: Source properties can now be mapped to most target properties regardless of data type. This can naturally lead to data conversion errors as a result. Any failed conversions are considered failed copies and they will be logged to a separate file on completion of the bulk copy for further analysis.
  • Bulk Copy: The source schema is now validated against the target to see if it can be applied. If it can't it will provide a detailed explanation as to why the schema cannot be applied.
  • A new ad-hoc data entry form for inserting new features into feature classes. This is currently very experimental and there is lots of room for improvement. This form can be brought up by right clicking the feature class you want to insert data into and choose "Insert Data"
  • New preference variable: pref_str_log_path. Any error logs are written to the path specified by this variable.
This release also fixes the following:
  • Prevent the creation of zero-length string, BLOB and CLOB data properties.
  • Enforce the name field requirement when creating new data properties.
  • Fix: Computed properties did not show in Data Preview (Standard Query mode)
Download
Report bugs or feature requests

Friday 22 August 2008

SELECT * INTO [sdf file] FROM [feature class]? Yes you can!

One of the long standing issues in FDO Toolbox (which isn't actually as serious as first thought) has been resolved. The resolution of this issue paves the way for some interesting ways to utilise the Data Preview feature, one way which I am introducing in this post.

The Data Preview tab now includes a "Save Query" (result) button, with the current available option being to a new SDF file. What this allows you to do is to save the current grid of a Data Preview to a new SDF file. For those SQL gurus out there, this effectively allows you to do the SQL equivalent of:

SELECT [columns/aliases] INTO [new SDF file] FROM [feature class]

... inside FDO Toolbox with FDO data sources!

Currently this feature is only enabled for the Standard query mode. Support for SQL query mode and Select Aggregates will come in due course. Support for other flat file output types (even non-FDO data sources) will also eventually come as well.

For the adventurous types. This feature is currently in the svn trunk. Otherwise expect to see this feature in the next release.

Sunday 17 August 2008

FDO Toolbox v0.5.6

This is a small bugfix release with one major bugfix:
  • Fix incorrect total for certain bulk copy task. For certain providers that support ISelectAggregates, a Count() was being executed with the first selected property name. This is not an identity property name and as a result, can result in inaccurate totals. This has been changed so that it now issues Count() using the first identity property from the source feature class.
Other changes:
  • Removed the (mandatory) feature limit in the Data Preview control, and make the querying be done on a background worker thread. This prevents locking up the UI while a query is executing on a large dataset. Queries in progress can be cancelled. This does not fix the "Memory allocation failed" problem but you can work around it by either turning off any geometry properties when quering in standard mode or to query via raw SQL.
  • Fixed incorrect labels in the Express bulk copy control.
  • Fixed potential crash in the Generic Connect/Create Data Store controls with certain providers.
Download
Report bugs or feature requests

Note: The changelog in the zip file is out of date. It has been updated in the svn repository.

Thursday 14 August 2008

FDO Toolbox v0.5.5

Changes in this release

Enhancements

  • Non-spatial data support: Database tables can now be converted to (Point) feature classes.
  • Bulk Copy now supports Batched insertions. Insert performance is better in batched mode (upwards of 10x in Oracle!), but not all providers support batched insertion.
  • Tasks and connections are persisted and restored on each session.
  • Save/Load support for non-spatial connections
  • Data Preview: Filtering support for select aggregates.
  • XML schemas for configuration files
  • New "Remove all" command for Connections (both types) and Tasks.
Fixes
  • Fix: Any tasks running can now be properly canceled.
  • MySQL copy spatial context fixes.
  • Fix: Saving schema was using the schema name instead of the chosen file name
  • Fix: Incorrect property mappings in Bulk copy control.
  • Bulk Copy: Oracle-specific overrides.
  • Bulk Copy: Improved counting process by choosing the most efficent method in this precedence: SQL - count() via ISelectAggregates - Brute Force
  • Bulk Copy: Auto-Generated IDs will be created for any class that has no identity properties that is to be applied to the target connection.
Other
  • Object Explorer menu updates.
  • UI tweaks in the Data Preview tab.
Download
Report bugs or feature requests

Thursday 7 August 2008

...and now for something completely different (ok, not really)

One of the main features I am currently researching and hoping to get into a future release of FDO Toolbox is the ability to visually preview FDO data sources.

The prime motivation for such a feature is that since we are working with geospatial data, there should be some way to see what the data actually looks like. This can pave the way for even more interesting features.

So one day (yesterday). I was playing around with good ol' SharpMap to see if I could get it to render data via FDO, after a day and bit of hacking around, the result is available here for your consumption.

To build the sample, build the SharpMapFdo.sln with Visual Studio 2005/2008 or msbuild and copy all the files in Thirdparty/Fdo to the output directory. Then run the TestApp.exe

Some notes about the FDO SharpMap data provider:

1. Only vector data source support has been implemented. Expect the unexpected if you init a VectorLayer with a raster FDO provider.

2. Due to the limitations of the SharpMap library, vector data sources must satisfy the following criteria:
  • The class to be rendered must be a feature class.
  • The feature class must have only one identity property and it must be either Int16, Int32 or Int64
3. This is only a proof of concept. Expect the thing to break and sometimes, break spectacularly.

4. I am aware of v2.0 being in development, but I chose the 0.9 branch because its code and interfaces are stable and simpler to comprehend. It was easier to implement a FDO data provider for the 0.9 branch than the 2.0 branch, whose APIs may still be in flux.

Tuesday 5 August 2008

FDO Toolbox v0.5.0

The changes in this release are huge enough to make the version jump to 0.5

New Features

1. Non-Spatial Data Support

You can now connect to non-spatial data sources using OLEDB. You can inspect the structure of non-spatial data sources just like the spatial ones. You can also preview non-spatial data sources using SQL.

2. Database Merge/Join Support

This is the feature that justifies the version jump.

You can join data from a FDO data source with data from a non-spatial data source by a common set of keys, and write the result to another (new) feature class.

The supported join types include:
  • Inner Join: Only records that match on both sides of the join are written
  • Left Outer Join: Only records from the left are written, any matching records on the right are also written
The supported cardinals include:
  • 1:1 For each matching record on the left, at most one record from the right is written
  • 1:m For each matching record on the left, each matching record from the right is written
For this implementation, the "Left" side of the join is always a FDO data source and the "Right" side of the join is always a non-spatial data source. The target is always a FDO data source.

When using this feature please be aware of the following:
  • The target connection must support IApplySchema (because a new class is being created)
  • An auto-generated ID will always be created for the joined feature class.
  • Properties which are also identity properties, can be copied over, but they won't be copied as identity properties to the joined feature class.
  • A column prefix is mandatory if there is a name clash between the chosen properties/columns
  • The joined feature class is always created, so it must not already exist.
3. Other changes
  • New FdoUtil.exe commands: RegisterProvider and UnregisterProvider. Consult the updated cmd_readme.txt for usage information.
  • FdoInfo.exe: Minor improvements in output display for the ListClassProperties command.
  • New UI front-end for the Preference dictionary. You will need to restart the application for most changes to be applied.
Download
Report bugs or feature requests

Sunday 3 August 2008

FDO Toolbox v0.4.2

Changes in this release.

New features/enhancements:
  • New per-class bulk copy option: Delete before copy. Please be warned that deleting existing data may take a while. Only use this option if necessary.
  • New global spatial filter bulk copy option. This is a textual spatial filter that is applied to each feature class. (non-feature classes are ignored). An example global spatial filter would be: INSIDE GeomFromText('POLYGON XY ((-124.07 49.1, -124.07 49.2, -123.8 49.2, -123.8 49.1, -124.07 49.1))') It is best to run this filter through the expression editor and/or data preview to verify it is a valid filter.
  • New command-line utility: FdoInfo.exe. Consult the updated cmd_readme.txt for usage information.
  • Class nodes now have a data preview option.
  • Expression Editor supports building spatial and distance operations.
  • Data preview in query mode now has support for property lists and computed fields.
Fixes:
  • Small UI changes in Spatial Context Dialog
  • Clicking cancel on the Generic Connect/Create Datastore dialogs will now actually close them.
  • Fixed incorrect spatial extents computation with certain data sources.
  • Fixed error about missing table "Projections" in cscatalog.sqlite
  • Connection-bound tabs are now single instance. For example: If you open the schema management tab for a given connection more than once, every attempt after the first will simply re-activate the existing tab.
Download
Report bugs or feature requests

Wednesday 30 July 2008

FDO Toolbox v0.4.1

Changes in this version.
  • Merged all the command-line utilities into a single utility: FdoUtil.exe. Consult the updated cmd_readme.txt for usage instructions.
  • New per-class bulk copy option: Attribute Filter. To migrate existing bulk copy definitions add a <SourceFilter> element (can be empty) as the last child element of each <Mapping> element in the bulk copy definition.
  • SchemaAttributeDictionary editing support for schemas and classes.
  • Schema and class nodes in the Object Explorer now have context menus.
  • Support for external FDO installations. When FdoToolbox launches for the first time it will look for the "FDO" subdirectory. If this directory doesn't exist, you will be prompted to specify the directory containing the FDO binaries. After this the directory will be written to a string preference pref_str_fdo_home. This value will be written to preferences.xml and used for every subsequent usage of FdoToolbox or its command-line utilities. Put it simply, you can download a Lite install, run FdoToolbox.exe or any command-line utility and point pref_str_fdo_home to an existing FDO directory/installation and Bob's your uncle.
  • "Apply" button in Schema Management Tab only activates if the underlying schema's ElementState is changed (ie. is dirty)
Download here

Sunday 27 July 2008

FDO Toolbox v0.4

In the "Release early, release often" fashion, a new release of FDO Toolbox is available, this time with a version number jump to 0.4. The new features include:
  • Command-line utilities: A series of small utility programs that implements discrete bits of the FDO API. Consult the cmd_readme.txt file included for descriptions and usage examples (or wait for another blog post :-))
  • Coordinate System Browser/Catalog: You can now store and manage your coordinate systems. When creating a spatial context, you can choose to look up this catalog and have it auto-fill part of the form for you. All the coordinate systems are stored in a sqlite database (cscatalog.sqlite)
Until v0.5 is released or a new version of FDO is released, this release will be the last one to have a "Full" configuration (because I don't want to exhaust my google code quota, and I don't want you to have to re-download FDO over and over again ;-))

Go get it!

Saturday 26 July 2008

FDO Toolbox v0.3.5

v0.3.5 fixes the save dialog bugs in the previous release (v0.3.4) and includes the following features:
  • New Program: TaskRun.exe, use it to load and run task definitions from the command line. NOTE both the source and target connections in the bulk copy task definition must be valid otherwise execution will fail.
  • Capability-aware menus: If a menu/toolbar item is disabled, it's because the command is not supported by the current selected connection.
  • New Data Store Manager for rdbms-based providers.
  • Bulk Copy Enhancement: Spatial Context List now in Bulk Copy Control. This means the Bulk Copy definition can now be executed without manual user input.
  • New Console Application framework. TaskRun.exe and other future command-line utilities will be built on this.
Also from now on, I will mainly be releasing "Lite" versions of FDO Toolbox, only bundling FDO when the major version number changes (eg. v0.4.0) or a new release of FDO is available.

Download

Friday 25 July 2008

Have Asus lost the plot (and a future customer)?

4 months ago, I was the happy owner of an Asus Eee PC 701

2 months ago, I was the happy owner of an Asus Eee PC 900 which resolved my main issue with the 701. The screen and resolution is actually of a usable size.

Today, I am a somewhat confused and rejected (in a "dumped in a relationship" kind of way) owner of my Eee PCs.

What on earth are Asus thinking?

I'm all for newer models. Progress and innovation is a good thing. But releasing FIVE new models in TWO months does not instill consumer confidence in your products.

And these models are not really different from each other, it's as though Asus had these big pile of components lying around and played "mix and match". Some have Intel Atom CPUs, some don't (so that's kind of a step back, seeing the Atom is the next big thing in mobile CPUs)

Kudos to Asus for innovation with the Eee PC and creating a new market of ultra-portable notebooks. But honestly, I think they've lost the plot. I doubt my next purchase will be from them. That MSI Wind looks very attractive.

P.S. Don't worry Eee PC 900. I won't dump you like your maker apparently did :-)

FDO Toolbox v0.3.4

A new release of FDO Toolbox is available. Changes which are mostly bug fixes and productivity enhancements, include:
  • Added a preferences dictionary. This gets saved to/loaded from Preferences.xml at application startup and shutdown. Currently two preferences are exposed: pref_str_working_directory and pref_bool_timestamp_console
  • Schema Management: Can now save selected schema to either an XML file or to a new SDF file
  • Geometric Property Dialog: FdoGeometryType is used instead of FdoGeometricType, allowing for more specific geometry types
  • You can now optionally connect to any SHP/SDF files created.
  • You can now optionally name any SDF/SHP connection created.
  • Class editing is activated again. Just be aware that removing properties won't work (the removal won't be applied)
  • Spatial Context Dialog: You can choose to set extents by computing it from a list of chosen feature classes.
  • Tabs will now close if their bound connection is closed.
  • Tabs will now have their title text renamed if their bound connection is renamed.
  • You can show the Object Explorer or Application Console if you accidentally close either one.
Download here:

http://code.google.com/p/fdotoolbox/downloads/list

Special thanks to Jason Birch and Kenneth Skovhede for suggestions (some of which made it into this release)

Edit: Most of the save functionality is broken and will be fixed in the next release

Tuesday 22 July 2008

FDO Toolbox v0.3.3

Just cut another release of FDO Toolbox. The following changes include:

  • Bulk Copy Control: Mappings can now be removed
  • Fixes for the Generic [Connect/Create Data Store] Dialogs for non-flatfile providers
  • Bulk Copy fixes for copying spatial contexts
  • Data Preview SQL support
  • Extension modules can now extend the application UI
  • Errors thrown by ISelectAggregate execution are now alerted to the user.
  • Expression Editor: Property List now is in a context menu.
  • Included missing SQL scripts from FDO distribution. These were needed by rdbms providers and was not copied during the build process.
Get it here:

http://code.google.com/p/fdotoolbox/downloads/list

Saturday 19 July 2008

FDO Toolbox v0.3.2

Another day, another update. The following changes include:
  • New Expression Editor. You can use this Expression Editor to build your query filters and/or aggregate expressions in the Data Preview Control. Currently, spatial and distance operations are not supported by the editor, but you can still enter your spatial/distance conditions manually.
  • Data Preview Control now supports Select Aggregate operations.
Get it here:

http://code.google.com/p/fdotoolbox/downloads/list

FDO Toolbox v0.3.1

This is a minor update to 0.3 with the following features/fixes
  • Bulk copy stability fixes
  • Bulk copy execution will not proceed if validation fails
  • Extension modules can be automatically loaded during startup by editing Modules.xml
  • Class Properties can be visible in the Object Explorer
  • Properties to map in the Bulk Copy control now have icons beside them to indicate property type
Get it here:

http://code.google.com/p/fdotoolbox/downloads/list

Sunday 13 July 2008

FDO Toolbox v0.3

FDO Toolbox v0.3 is out. The new features include:
  • Express Bulk Copy commands. Allows you to convert a SDF file to a SHP file and vice versa, and any other combinations thereof.
  • New Express Bulk Copy Control.
  • Can now refresh a selected connection from the Object Explorer
  • Can rename connections from the Object Explorer
  • Re-designed Data Preview Control.
  • New SHP (directory) express connection option
  • Re-designed Bulk Copy Control
  • Bulk Copy now has option: Copy Spatial Contexts
  • Support for multi-class bulk copy
  • Lots of minor improvements
This release also comes in binary form. You can either grab the source, or grab the binaries from http://code.google.com/p/fdotoolbox

Sunday 6 July 2008

FDO Toolbox

This is something that I have been working on and off for the past few months in my (now plentiful) spare time. So now I think it is time to share my work with you...

FDO Toolbox is an application to perform spatial data management and processing.

It is written in C# and uses the Feature Data Objects (FDO) API (http://fdo.osgeo.org)

FDO Toolbox requires .net Framework 2.0 and the Visual C++ 2005 SP1 Runtime Library.

There are currently no binaries available, but you can checkout the source from subversion and build the code yourself.

Some of the features of FDO Toolbox.

  • Connect to any FDO-supported data source.
  • Create new data stores for any FDO provider that supports the capability (eg. SDF, MySQL)
  • View and Edit schema and class definitions for any FDO data source.
  • Create custom extension modules for your own custom functionality. A test module (TestModule.dll) is included in the source code that demonstrates this functionality.
  • Save/Load schema definitions to and from XML
  • Bulk Copy from one FDO data source to another (very basic at the moment, still lots of work to be done).
  • Preview data from a feature class.
Get the code from here: http://code.google.com/p/fdotoolbox

Bug reports and suggestions are welcome.

DISCLAIMER: This is alpha-quality code. Do not try this application on production GIS data!!!

Fusion Wiki (or: Introduction to Fusion Parts 2 onwards)

I had prepared a bunch of posts about the new Fusion Framework for MapGuide, but it seems that the developers/users themselves have already put up similar material on the Fusion Trac site.

For those who don't know, the Fusion Trac site is here:

http://trac.osgeo.org/fusion/wiki

There is lots of material here to get you started and/or acquainted with developing fusion applications for MapGuide.

Monday 19 May 2008

A spatially-aware digital camera?

Disclaimer: I am not a photographer.

I'm currently in Hong Kong at the moment (having resigned from my job), and in the 2 weeks that I have stayed here so far I have been taking tons and tons of photos on my cousin's digital camera. One of the things that I would like to do (being the spatially-aware person I am) is to be able to geo-tag all the photos taken, providing a spatial context (where) to all the photos taken.

Unfortunately, geotagging to my current limited knowledge is a totally manual process and given the amount of photos I've taken so far, would take about the rest of my stay here (another 4 weeks and a bit :-)). This quandary had sparked my thoughts into motion. What would make an ideal spatially-aware digital camera that would make all of this much easier?

In my opinion, a spatially-aware or spatially-enabled digital camera should have the following:

A built-in GPS receiver

This is a no-brainer. A camera with a built-in GPS receiver would allow you to instantly capture and associate global coordinates to any photo taken.

A compass

Having a built-in compass can provide directional context to any photos taken. For some of the photos I have taken, it was hard to tell or remember which direction I was facing. Attaching directional information into the photos can help in these situations.

An intuitive image meta-data management system

For all of these features to work efficiently, a clever metadata management system needs to be available that can store and manage the spatial metadata attached to the photos you have taken. It should either utilise any metadata facilities built into the image format the camera supports (does JPEG have such capabilities?), or some kind of platform-neutral metadata format (XML perhaps?). Integration with popular photo management applications (eg. Picasa) is a bonus.


I'm gradually learning that photo albums are basically visual stories. Wouldn't it be nice to be able to not only say when, but also where an event took place with absolute ease? (emphasis on ease)

Monday 31 March 2008

The superficiality of Earth Hour

Some of you may know that Earth Hour happened this weekend.

Despite the claims about the amount of energy saved, and the amount of carbon emissions reduced, etc. I could not help but feel there was a certain superficiality to the whole event, as though it was a trendy, bandwagon-hopping thing to do.

Consider the following:
  • There are 24 x 365 = 8760 hours in a given year. 1 hour of energy savings in the grand scheme of things seems insignificant (assuming this will become a yearly event). Should we not try to work towards more permanent solution? (or to rephrase: Should we not be trying to make every hour like Earth Hour?)
  • As already mentioned, the event reeks of superficiality and tokenism. Call me cynical, but I doubt people will remember about Earth Hour in a month's time or two, and resume their normal energy wasting ways.
Earth Hour had great intentions, but there needs to be some solid plans of action to come out of this or it will be forgotten as some yearly special event, when it shouldn't. It should be something that we should be striving for on a more regular basis.

Saturday 29 March 2008

Speaking of rickrolls...

15 years ago, a dead hero had something relevant to say.

And for the record:

Rick Astley > Chuck Norris. Why?

Fact: Rick Astley is never gonna...
  1. Give you up
  2. Let you down
  3. Run around and desert you
  4. Make you cry
  5. Say goodbye
  6. Tell a lie, and hurt you
Can you say the same for Chuck?

Real life rickrolling?

On Thursday night, I decided to get out of my cave and go out with my colleagues and co-workers for a couple of rounds of bowling and karaoke. Being led into a false sense of expertise from extensively playing Wii Sports, I proceeded to comprehensively get my arse handed to me.

But that is not the purpose of this post for the fun part was the karaoke.

After gaining some liquid courage and belting out a couple of tunes myself from a mediocre list of songs (no metal or rock, boooo! I would've loved to belt out some AC/DC), I subversively added to the playlist, and consequently had to endure the most satanic rendition of Rick Astley's "Never gonna give you up". A real-life rickroll, if you will.

Now if only that was captured on video instead of me singing a duet of "The Bad Touch" by the bloodhound gang, it would've made for epic rickrolling material =)

Friday 28 March 2008

Perception of Time (or: The holy grail of productivity?)

Disclaimer: This post is not based on scientific fact. I am not a neurologist ;-)

There is something that has been on the back of my mind for some time now...

Ever had days at the office, where you keep watching the clock tick over to 5pm so you can then knock off?

Ever had days at the office, where you are totally in the zone and before you know it, the day has already finished.

I have this personal theory...

There (is | has to be) some part of our brain that can alter our perception of time. If there was a (legal ;-) ) drug or technique to control and master this perception of time, then we can truly be the master of our domain; To be able to be "in the zone" while at the same time telling yourself "geez, it's still not 5pm yet?" in the most constructive way possible ;-)

Tuesday 11 March 2008

Introduction to Fusion Pt 1: What is it?

One of the new features of the recently released MapGuide Open Source 2.0.0 is the Fusion Framework (hereby known as Fusion). Anyone who has worked with the existing AJAX viewer would know about some of the following limitations:
  • The user interface is not very customisable. You are stuck with the 3 column layout.
  • There is a 1:1 relationship between the Web Layout (that defines the AJAX viewer interface) and the Map Definition. What this means is that you don't have the ability to switch between maps within the same AJAX viewer, without some server-side hackery with dynamically created Web Layouts.
  • You cannot programmatically manipulate the state of user interface elements (eg. Enabling/Disabling toolbar buttons)
  • It is hard to interface with the AJAX viewer from JavaScript. For example, to listen to a selection event in the AJAX viewer, you currently have to use ugly hacks, such as overriding the AJAX viewer methods with your own.
  • It uses frames, which are absolutely evil
Fusion effectively addresses all these problems by allowing:
  • Discrete separation of functionality and style.
  • Flexible, free-form layout of user interface elements.
  • Componentisation of application behaviour (via widgets)
  • Styling of user interface elements via CSS.
With Fusion, you can create some compelling web mapping applications both in terms of functionality and visual style (case in point: compare the existing AJAX viewer with the sample Fusion Templates that come with the MapGuide Open Source installation).

Also, the widget-based approach allows a non-developer to quickly assemble a mapping application by choosing a Fusion template, and adding in the desired functionality via widgets. Fusion already comes bundled with lots of widgets covering many common functionalities and tasks (zoom, pan, buffer, measure, map switching, attribute browsing, and much more)

The architecture of the framework lends itself to clean separation of roles. Designers can focus on the design (templates), and developers can focus on the behaviour (widgets).

One of the (small) problems now with Fusion is the reliance on PHP to provide the required server-side behaviour for some of the Fusion widgets. Hopefully in the future, there will be complementary asp.net and java versions available.

As the title indicates, this will be the first of many posts I'll be writing about the Fusion Framework. Stay tuned.

Console.WriteLine("Hello World");

One of the first things you get introduced to in programming is learning how to write a program to say Hello World, so in the true fashion of a beginner programmer:

Hello World

This blog will mostly be technology based, with primary focus on a piece of software I work with regularly on a day-to-day basis: Autodesk MapGuide and MapGuide Open Source

So watch this space. Hope to see you around.