Conduit Manager for Developers

(PC developers specifically)

Generic Conduit Manager for Developers

The Palm/USR conduit system is designed to allow a hotsync to occur with both native and foreign data. The designers assume that foreign data is in such a format that code will have to be written to implement a hotsync. Therefore, the conduit system assumes that each conduit will be different/unique. And that the data format for each conduit will be different/unique.

The designers of the HotSync system also provide a "backup conduit" which simply transfers raw data, in bulk.

I felt that these assumptions were a bit onerous. They assume that everybody who develops a PC app will be interested and capable of writing a DLL. I didn't think this was a reasonable assumption.

The Conduit Manager system uses a generic conduit, which can sync any Pilot database on a record-by-record basis. In order to do this, here are my assumptions:

This means that developers on the PC can work at a simpler level. There is no need to learn the entire Conduit management system. On the other hand, we (the PC developers) have to write the code to read and write MAC style numbers, and deal with fixed format records that are created by a foreign app. If you can live with this, climb abord.

In the coming weeks, I will bolster this package with documentation on the file formats, and sample code for the use of the GenCond.DLL and the Generic Conduit Manager.

Features for programmers


How to Write a Viewer

Have a look at the sample code to follow this discussion

Startup

At the start of the viewer program, you can automatically select the .dat file that the HotSync/GenCond writes to. Simply use the default directory and append your specific directory and filename. Alternatively, you can let the user specify the filename by presenting the user with the FileOpen dialog.

Be aware that the files GenCond.DLL generates are almost identical to those created by the Backup conduit that ships with the Pilot. If the user is not using the GenCond.dll, you can direct your file open to the Backup directory, and the approriate .pdb file.

Loading

I recommend that you load the whole file into memory. Allocate a single file header (78 bytes) and read it in. Then allocate room for the file index, which is 8 bytes times the number of records. In the examples, I call this aRecEntry[ ].

(optional) If your database uses the "attributes" or "categories" blocks, you should allocate these blocks at this point and read them from the file.

Now you are ready to read the actual records. In my examples, I read them into an array aRec[ ].

Display

Remember that some of the records you have loaded may be marked as deleted. These records must be preserved until the next hotsync, but should not be displayed.

What I usually do at this point is run through the aRecEntry[ ] and fill a list box with those records that are not deleted; the "deleted" flag is 0x80 in the aRecEntry[ ].attrib. I use the listbox .ItemData property to keep the appropriate index back into aRecEntry[ ].

This is the appropriate time to filter categories as well, or hide private/hidden records.

To sum up: at this point, you want to present your user with a subset of all the records.

Sync at Any Time

You never know when the user will start a hotsync. If he does, we have to assume that he wants any changes saved to disk before the hotsync starts. To do this, watch out for the DDE message, and

At this point, reload the file from disk, since it may have new data in it.

Record Management

The aRecEntry[ ] holds a lot of information. On disk, it is

long file pointer
byte attributes (a nibble of status bits and a nibble of category
3byte ID (unique ID on the Pilot)

The file pointer is the offset from the beginning of the file. Once you have loaded the data into program memory, you don't really need this file pointer anymore, so if you wish you may change this 'long' into a 'void *' and use it as a pointer to the raw record data.

The attributes high bits are 'deleted', 'modified', 'archive' and 'hidden'. The 'deleted' bit means that this record is due to be deleted. When this is set, you can discard the raw record data, but do not delete the aRecEntry[ ] element; you need to keep is so that the next hotsync can tell the Pilot to delete the corresponding record. As a general rule, your viewer/editor app never fully deletes an entry, it just marks it for deletion. Only the conduit does the actual deletion.

The 'modified' bit tells the conduit that this record needs to be sent to the Pilot. New records should always be marked 'modified'.

The 'archive' bit is not used in this version of the conduit. Use the 'private/hidden' bit as you see fit.

The lower nibble of the attribute byte is the category number.

The next 3 bytes are the ID as assigned by the Pilot. Whenever you make a new record on the PC, always assigne the ID as 0x000000. Be aware that the Backup conduit that comes with the Pilot creates files which are identical to those generated by the GenCond.dll conduit, except that all the ID's are flushed to 0. If you allow your users to view/edit these backup files, the result is only appropriate for a PCtoPilot one-way sync (because all the ID's are 0).

One thing obviously missing from the aRecEntry structure is the length of the raw data record. When you are reading the data from the file, you have to deduce the record lengths by noticing where the next record starts. The length of the last record is determined by the end-of-file.

When you read the raw data into memory, you should either read it into a data object that is aware of its length (such as a VB string, or a std::cstring) or else modify the aRecEntry structure with extra data:

long file-pointer/void-pointer
byte attribues
3byte ID
long record-length

Final Note

Remember that all 'numbers' in the file are stored in Mac format. You will almost certainly need a SwapShort() and SwapLong() routine.

Good Luck


Syncing with Existing Apps

Several people have asked me about grabbing data, or injecting data into the databases of the 4 main Pilot apps (address, datebook, todo & memopad). I would not recommend doing this. This is not what the generic conduit was designed for. The Pilot desktop apps have import and export facilities which should give you what you want.

The problem is twofold. First, the existing Pilot conduits significantly modify the data formats before saving the data to disk. Unless you know what these format are, you are risking data loss. If you over-ride the native conduits, your Pilot Desktop will not see the data.

Secondly, the entire Pilot sync mechanism is based on the assumption that one computer syncs with one Pilot. There are sub-mechanisms in place to deal with multi-computer, or multi-pilot, but there are not mechanisms in place to deal with a three-way-sync. What I mean by this is: if you decide to start syncing the Pilot address book to your down database, then you should kiss goodbye to the Pilot Desktop address book.


Tech Notes

(hackers only)

HotSync 1.0/Generic Conduit System 1.3

In order to write a generic conduit, I had to figure out how to make sure that the conduit gets called by the HotSync system. Using HotSync 1.0, that is not simple, since the HotSync.exe (actually syncmgr.dll) enumerates the databases in the Pilot, and only calls the DLL's that are required to service this list. If the app that the user has requested does not make it onto this enumeration, the DLL will not get called.

To get around this, I install a proxy in the MemoPad/Todo conduit, by inserting an ApplicationX registry entry. During the startup of the gencond.dll, I reach back into the Component3 registry entry and execute the memcond.dll first, and then run through the ConduitManager entries.

For each one, the gencond.dll uses the registry information to locate a file and reads it into memory. It then tries to sync up, record by record and creates a new local file. If all went well, the old file is renamed and the new file written.

HotSync 1.1/Generic Conduit System 2.0

The HotSync 1.1 system properly enumerates the databases on the Pilot, so there is no need for the above kluge. The conduits are installed as ApplicationX entries, as the Palm designers intended.

The registry entries have several keys that are standard:
Conduit: the name of the DLL
Creator: the DWORD signature for the app
Directory: where the data will be stored
File0: the name of the file
Module: not really used
Name: the string to be displayed in the HotSync.log
Remote0: the name of the database on the Pilot

It also has a few extended keys:
Backups: the number of backup copies of files
NotifyAppService & NotifyAppTopic: the DDE strings used to notify cooperating apps
Disabled: if true, this conduit is skipped
SyncDirection/NextSyncDirection: 0=normal sync, 1=careful, slow sync, 2=PilotToPC,3=PCtoPilot,-1=disabled

Note: in GCM1.3, the SyncDirection and NextSyncDirection were strings. They have been changed to DWORDS to align with other conduit designers.

DDE

If you have an application that is running, it will get a couple of DDE messages. Before the file is opened by the conduit, you will get a "application/notify" message command with the string "sync start". You have 1.5 seconds to write the file to disk before the gencond.dll will try to read it.

After the file has been written, you will get a "application/notify" message command with the string "sync finished". You can then re-read the file into the app. [All records will be marked clean, and deleted records will have been purged.] [Look at the source for cbasVw, in Form_LinkExecute.]

Locating User Files

In order to locate the files, each time the gencond.dll is executed, it puts an entry in the registry: HKCU/Software/Palm Computing/PilotDesktop/Preferences/LastUserDir ="name of user dir, without trailing backslash"

To locate the data files, simply concatenate a "\DirName" to this string. This should be the same DirName that the user put into the ConduitManager. Examples are "DinkyVw" or "cbasVw".

Services

If you wish to use it, the GenCond.DLL exports a simple little routine to build up multiple backup files. [This routine is also exported by GenCn20.DLL and DinkVwA.DLL The nice thing about using the DinkVwA.dll is that it has few dependencies, so can run in any directory.]

/*SDOC

EDOC*/

C/C++ int MakeBackupFiles(const char* sFileName, int nBackups);

VB Declare Function MakeBackupFiles lib "gencond.dll" alias _
"_MakeBackupFiles@8" (ByVal sFileName as String, _
ByVal nBackups as Long) as Long

In order to install conduits into the Pilot HotSync system, you have to insert keys like "ApplicationX", where X is 0,1,2... If you remove a conduit from the system, you must renumber any ApplicationX entries which use numbers higher than your own. Unfortunately, the Win32 API does not allow for renaming keys, so I have written (and tested) a routine, which is exported by GenCn20.DLL.

/*SDOC
EDOC*/
C/C++ int RegRenameKey(HKEY keyOldRoot, LPCSTR lpOldKeyName, 
              HKEY keyNewRoot, LPCSTR lpNewKeyName);
VB Declare Function RegRenameKey) lib "gencn20.dll" alias _
              "_RegRenameKey@16" (ByVal keyOldRoot as Long, _
              ByVal sOldKey as String, ByVal keyNewRoot as Long, _
              ByVal sNewNew as String) as Long

Typical usage:


Data Structures

All the .dat files saved on disk by the Generic Conduit (gencond.dll) follow a similar format. The file consists of a header, a record index (8 bytes per entry), an optional Attributes block, an optional Categories block, followed by the records themselves, concatenated.

The header is as follows:

followed by an array[nRecs] of these:

The lower 3 bytes of _id is the Pilot unique number for this record. The upper byte is two nibbles: