Data storage and retrieval


DSpatial is all about the manipulation of spatial data. The aim is to support as many of the multitude of spatial data formats as is humanly possible. DSpatial does not favour any particular format, and it is up to the application developer using DSpatial to provide the application with the necessary drivers to read and write data formats. Drivers for several file formats are included with DSpatial, including a driver for the versatile DSpatial format.

We can always use more formats! If you have a driver for a format not (yet) supported by DSpatial, consider making a contribution.


Pluggable Format Architecture

In order to shield the application developer from the details of every data format, a data abstraction layer is placed between the application and the drivers accessing the actual data files. This design is called the Pluggable Format Architecture (PFA). This rather fancy name derives from the quality that the drivers are independent of the user application. It will be entirely possible to build an application without any data drivers, but that will not be a very useful application from the DSpatial perspective. The developer can statically link data drivers, which will be permanently present in the executable, or load them dynamically at run-time using shared libraries (Win32 DLLs, or Linux so libraries), freeing resources on demand. The PFA supports an unlimited number of data drivers in the application.

Go to the Data driver sectionGo to the Data broker section

The PFA can be queried by the user application ("user") to find out which data formats are supported through a data broker. The user can then request the PFA to set up a data handler for a particular data source. The data handler connects to the data driver and sets up a source data set object. This source data set object calls the driver which accesses the actual physical resource (local file, database, etc.), passing the results back to the data handler, passing the results back to the user.

Sounds complicated? Yes, but the application developer sees very little of the guts of the PFA and there are several good reasons for this design. First, this design provides a consistent interface to the user, no matter what particular format is being supported. A uniform data handler is set up that can handle any kind of spatial data (vector, raster, attributes, etc), and the application developer needs only to worry about the public methods of the handler. Secondly, and more importantly, this design allows the user to access multiple data sources simultaneously. The PFA is a thread-safe environment and the user can attach as many handlers as necessary to as many drivers as it takes. The handlers are thread-safe such that true multi-tasking can be achieved on multi-processor systems (including hyperthreaded P4s), and virtual multi-tasking on all other systems (that is, while a handler is waiting for slow disk or network access, other handlers and the user thread can continue to run).


Data broker

The data broker is the primary interface of the PFA to the supported data formats. The data broker can be loaded statically with the executable, but it can also be loaded dynamically if the broker/driver pair is supplied as a dynamically linked library (dll or so). Dynamically loaded and static brokers are registered with the PFA, such that the PFA can expose the supported formats to the user. The user can query the PFA for available formats and driver capabilities (e.g. whether the reader can also save changes or write new files).

When the user requests a certain source to be opened, the PFA requests the data broker for an instance of the appropriate driver. The PFA then creates a handler, which receives a reference to the driver, and passes a reference to the handler back to the user. The user passes requests directly to the handler. The handler then passes the request on to the source data set that was set up for the handler. From the standpoint of the user, all communication is through the handler, the source data sets, brokers or drivers are never directly accessed.

Data driver

The data driver does the actual reading and writing of data sources. A data driver can be either statically linked in the application or dynamically loaded by the data broker when needed. Statically linking a driver would be useful to speed up access to a data format that is repeatedly used in an application, while dynamic loading is better for data formats that are used occasionally, and to conserve computer resources.

When the PFA creates a handler to serve a user request, a corresponding data driver is created by the data broker to serve the handler. Files are always opened for exclusive access (i.e. 1 driver for 1 source), but shared access is built in to the design by letting multiple user threads (e.g. a main thread for display, and a second thread for a slow analysis process) access a single handler and thus the driver. This is particularly handy for complex data formats, such as HDF files or external databases.


Go to SourceForge
Last modified: 22 November 2004. Page maintained by pvanlaake