DP4 Auxiliary Database Controller Developers Manual

ADC - DP4 Database Middleware

An ADC (Auxiliary Database Controller) is a piece of software that sits in between a DP4 application and the DP4 database manager, and which extends or alters the standard DP4 functionality. Conceptually it is similar to the plug-ins used with Web Browsers or programs such as Adobe Photoshop. For the C Programmer, writing an ADC is similar to subclassing a pre-existing Window class in a Windows program or deriving one class from another in a C++ program.

When an ADC is plugged into the DP4 Database Manager, all calls made to the database manager by the DP4 application are first passed to the ADC for processing. The ADC is free to act on the intercepted call before returning control back to the database manager.

On intercepting a call the ADC may:

• Make DP4 calls
• Make native operating system calls or calls to third party software
• Return control back to the database manager

Purpose of ADCs

The DP4 Auxiliary Database Controller (ADC) is designed to meet three needs.

1. Firstly, you may require that some or all of your data should physically reside on the same machine as your DP4 database, but in a different, non-DP4 database or file system, perhaps because you have existing programs that can only access some other database. Or you may require to have some or all of your data on another machine altogether. But you want to make all the data appear to be logically in the DP4 database, because:

   • In this way, your application programmers need only learn one set of DP4 access methods, and only understand one kind of data structure. They can ignore the practicalities of how the data is actually accessed and stored.

   • You can then use QA Build, the DP4 Report Writer, and other DP4 utilities, on all the data.

   Since QA Build supports user exits written in C, you have a choice whether to use C exits or an ADC to access non-database files. Itim Technology Solutions generally prefers the ADC approach, as it provides a simpler model for the application programmer.

2. Secondly, you may want to put business logic into the database manager. You could use this to:

   • Support computed values on the database; for example, you could have order line value, computed at read time as quantity times product price.

   • Implement additional integrity constraints; for example, you could arrange so that a product can only be deleted if its replacement is in stock.

   • Implement views; for example, you could define a table 'overdue orders' in your data dictionary, but post no data to it. At read time the ADC would read the main orders table and return a record only if it was overdue.

   • Implement a sophisticated security system; for example, you could make a certain table accessible to different people at different times of the day.

   • Write a special audit file logging all reads of sensitive data.

3. Thirdly, you may wish to write some service routine of your own, for instance to support comms, and to do this in a way that allows your application programs to remain operating system independent. You can achieve this by putting your service routine in the ADC, and calling it through the provided special user function.

Of course you may do several of the above in a single ADC. For example, you could have a commodity data table, where the current price was looked up at read time from a remote information service if the price on the database was more than 2 hours out of date.

Loading an ADC

On Win32 platforms ADCs are loaded in the [startup] section of the DP4 configuration file as part of the DP4 service, and the command lines given here would be preceded with 1=, 2=, and so on. On other platforms they are loaded from the command line or a batch file/shell script. However the two processes are very similar, and the following instructions apply to all platforms unless otherwise noted. Remember that on Win32 platforms you can get the DP4 service to load alternative configurations of DP4 by specifying an an alternative name for the [startup] section of the DP4 configuration file, for example:
srvw32 -start -startup=startup_adc -preload=preload_adc. This may be convenient if you sometimes want to load with an ADC and sometimes not.

The database manager (srvX) or possibly a network requester such as tcpX must be loaded first, with a -aux command tail. The command tail −remain should also be used except on Windows platforms.

The ADC is then loaded, with or without a −remain command tail. If you load the ADC without −remain, it will unload when the next DP4 program terminates.

Loading Multiple ADCs - Chaining

A chain of ADCs may be loaded. This allows a cascade of interceptions. For example, with a chain consisting of the application, 3 ADCs and a server, it is possible for the first ADC to intercept a call from the application, and return 1 from the relevant u_ routines. The second ADC will then pick up the call and so on until the server is finally activated. The same principle of server-call interception operates as in a single ADC-server configuration.

The order of ADC loading is critical in a chaining situation:

1. Firstly the DP4 database manager must be loaded as:

   srvX -aux -chain n -remain
   or, on Win32 platforms:
   1=dp4srvr.w32 -aux -chain n

   where n is the highest value in the chain - that is, the chain consists of n+1 ADC nodes.

2. The highest ADC is then loaded:

   ADCName -chain n -remain
   or, on Win32 platforms:
   2=ADCName -chain n

   The remaining nodes are then run in descending order:

   ADCName -chain n-1 -remain
   .
   .
   .
   ADCName -chain 2 -remain
   ADCName -chain 1 -remain
   ADC -remain
   

   Note that the last ADC is run without a -chain tail.

3. Once the last ADC has been loaded DP4 applications may be loaded in the normal way.

The ability to chain ADCs is available in all 4.5xx or later versions of DP4. A decision to use chained ADCs should not be taken lightly. There is an inevitable performance penalty in using ADCs, because of the extra IPC required. Wherever possible you should build a single ADC combining all the extra functionality you need.

ADC Implementation

The ADC is supplied as a skeleton program, into which you can add whatever code you need. ADCS are loaded into memory 'on top of' the standard database manager (SRVn,or DP4SRVR.W32) as described above. Thereafter, all calls from application programs to the database manager are intercepted by the ADC. Depending on your code, the ADC can pass the call on to the standard database manager or deal with it itself.

The method of interception is as follows:

• On all operating systems except MS-DOS, some kind of message queue or pipe is used for IPC (Inter-Process Communication) between DP4 applications and the database manager. The details of this are of no relevance to ADC developers, (except that very occasionally some change may be made in the IPC mechanism used, which means that ADCs written for an earlier release of DP4 need to be relinked for the new release). The ADC takes over the regular queue, thus receiving all calls to the DP4 database manager made by applications. The ADC will handle certain calls itself (possible making other calls to the DP4 database manager in the process). Other calls will be passed onto the DP4 database manager. The DP4 database manager most use a different message queue from normal, so that the ADC can make calls to it without intercepting them again. The database manager is loaded with a command tail to tell it where to look for messages.

• Under MS-DOS, the details are generally similar except that IPC uses a software interrupt mechanism. The ADC redirects the software interrupt that normally points at SRV2 to point at itself, and saves the address of SRV2 so that database calls from within the ADC can be passed to SRV2.

The ADC is structured so that for each DP4 function that calls the database manager there is a corresponding function in a user written module of the ADC. (The names of these functions are fixed, and based on the pre 4.500 C library function names) The code of each user function will usually consist of two parts - the first part will examine the incoming call and decide whether to handle it or pass it on to the regular database manager. The second part of the function will contain code to handle the function. The next section explains the use of the user functions in detail.

The Skeleton ADC Functions

auxuser.c provides the skeleton ADC variants of standard DP4 Database Manager calls. You should modify the skeleton functions it contains to perform whatever actions are required. It is probably best to rename this module before using it in your own ADCs. If necessary or convenient you can of course divide its functions between more than one source file. It should not be necessary to modify any of the other ADC source modules. These are supplied in source form purely so that you can build your ADC using whatever compiler options you wish. If you modify other modules you will compromise the portability of your ADC both to other platforms, and to future releases of DP4.

The modified auxuser.c (or its replacement) is compiled in the normal ways, and linked with the other ADC source files appropriate to the platform you are creating an ADC for. The exact set of files used depends on the platform, as does the method of building the ADC. Information on building for the various platforms, and other platform specific information can be found at the end of this manual on these pages:

• Win32 Platforms
• Unix/Linux Platforms
• 16 bit Platforms

The functions in auxuser.c are usually called whenever the application program makes the corresponding call to the database manager. However, it is important to note that there is not necessarily a one to one correspondence between application level calls to the DP4 database manager and calls made to the ADC. For example the DP4 networking layers may translate a call to the rec_fetch() function into a blk_fetch() , so that subsequent calls to rec_fetch() can be handled locally, or queue updates until so that a "batch" of updates can be sent across the network in a single operation. These optimisations can be controlled for DP4 networking using the -noblock and/or -nofaster command tail. However, applications may also use the bf_fetch() function to perform similar record buffering at an application level, and in this case there is no way of disabling the behaviour without altering the application.

Each user written function should return 0 if processing for the call is complete, and the call should not be passed on to the original database manager (or next ADC in any chain). The functions return 1 to indicate that the call should be passed on to the original database manager. Where possible, it is better to code:

int u_fetch(flags,datarec,index,role,keys)
{
  return 1;
}

than to code:

int u_fetch(flags,datarec,index,role,keys)
{
   rec_fetch(flags,datarec,indexrec,role,keys);
   return 0;
}

The timestamp of all records fetched explicitly by an ADC as opposed to being read by the regular DP4 database manager as a result of returning 1 in a u_xxx function have their timestamp set to zero in case the timestamp is not valid for a corresponding DP4 database record. This can cause additional commit failures in an application. For further information on this consult the DP4 networking and Resilience manual. You can prevent this from happening by compiling the auxproc.c module with the symbol KEEP_TIMESTAMPS defined. In this case you must ensure that the timestamp is zeroed when necessary -i.e. when the timestamp of a record returned to by a fetch() function does not match the timestamp of a corresponding record on the real DP4 database and the record may subsequently be posted as a real record. It is especially important to zero timestamps where two different DP4 databases are presented as one logical database and records may be transferred between the databases.

The names of the functions in the user written module are given in the following table, together with the corresponding function(s) in the DP4 C library, and notes on their use.

Calls available to ADC functions

Your code can make full use of whatever standard DP4 database access functions are required. You can also use all the other DP4 library functions that do not use the DP4 terminal manager. (You cannot normally use the terminal manager functions because an ADC does not load this. There is more information about this in a later section, ADC calls to Terminal Manager Functions.) There may also be special considerations when using DP4 file handling functions in the sf_xx functions, or the srv_mem_xx memory workfile functions.

datafit.h lists all the DP4 C functions grouped by which, if any, of the database manger and terminal manager are used to implement them. Common database access functions that can be safely used are:

db_checkpoint()
db_close()
db_commit()
db_decommit()
db_open()
db_update()
rec_autoinc()
rec_fetch() or rec_fetch_main()
rec_kill()
rec_post()
rec_retrieve()
rec_verify()


Unless you are writing a 16 bit object code portable ADC you can use all standard C standard file functions of your operating system. Similarly you can use any functions in third party libraries except in object code portable ADCs.

With the exception of u_dbopen() the appropriate database is automatically open when the corresponding u_ functions are entered and the generation and number system are set to the values appropriate for the calling application program. The values of the global variables there, locked and so on will be passed back to the application program when these functions return.

Two additional calls to the original database manager are available, which cannot be made from application programs. These are chkpt_check() and chkpt_action().

chkpt_check() performs all the checking associated with db_update() and sets the value of fail_code appropriately; its returned value is TRUE if the db_update() would succeed.

chkpt_action() actually performs the db_update() without doing any checking first. Both functions require one parameter, which is the same as that supplied to db_update(). The user-supplied code for db_update() can therefore call chkpt_check() and also perform some other checking in order to decide whether to proceed. If it decides not to proceed, it should call decommit() to remove the pending updates from the work file in the original database manager, so that they are not actioned by the next successful call to db_update().

The function u_ftch2nd() corresponds to the application-program function ftch2nd(), which is a variant of rec_fetch() called by some old DP4 programs. This fetches a record on a secondary index, as rec_fetch() does, but the index and role are specified differently. For ftch2nd() the indexing table is determined by the type number of the 'indexrec' parameter, and the role is passed as a string. For rec_fetch() the index type number is passed directly, and the role is passed by number. Role numbers are associated with names by the @NAME_TABLE table on the database. rec_fetch() is more efficient than ftch2nd() and should be used in preference to it. Also programmers are occasionally confused by the two table records used with ftch2nd() and unintentionally attempt to pass data in the wrong copy.

In respect of all the various u_fetch() type functions, you may well find it convenient to write a single fetch function that accepts a combination of all the parameters: for example int u_fetch_common(int number, int flags, struct DATAREC * datarec, int index_table, int rolenr, char * rolename) and to call this function from all the various fetch functions. It is permissible for the blk_ functions to return a single record rather than the number requested. However, if at all possible you should return the number of records requested. DP4 programs that access the base dictionary using these functions may take fewer records being returned than requested as a signal that end of data has been reached,

Use of df_process variable

It should be noted that calls to DP4 library functions from inside an ADC are, by default, made on behalf of the application process. Therefore, if you want to access information in a temporary file on shared between all processes you should either create the file using native OS file handling functions, or if you wish to use DP4 functions, you should obtain a DP4 process number for the ADC itself by calling srv_init_process() at u_startup() time, and save the value this returns. Before calls to DP4 functions that logically belong to the ADC itself rather than the application, save the current value of the df_process variable, and temporarily set it to the ADCs saved process number for the duration of the call. If you do get a process number like this you must remember also to free it in u_terminate(). The special process number 1 is reserved for use by the DP4 database manager and may also be used by ADCs. However this will not work on a machine where the database manager is replaced with a network requester, nor can it safely be used with memory work files because of the risk of conflicts with the genuine database manager.

ADC calls to Terminal Manager Functions

In normal circumstances, ADCs should not call terminal manager functions. Under MS-DOS, it is usually harmless, because the call will appear to come from the application and the application's terminal manager will handle it. Under other operating systems, it is more serious because some kind of protection violation will most likely occur. (On the Win32 platform a few calls that do not depend on the normal initialisation routines might happen to work, but this cannot be relied upon.)

If you are not aware of which functions in the DP4 library can safely be called, you may find it helpful to refer to the datafit.h file. This lists functions in the following order:

• Functions not requiring SRVn or TRMn
• Functions requiring SRVn only
• Functions requiring TRMn

Try linking your ADC with an option to write a symbol or map file that you can examine (auxlink /m will work for 16 bit versions). If you find that _trm_ncall has been linked in the map file, your ADC is probably calling terminal manager functions. (The name of this symbol may be slightly different in the map file because of name mangling by the C compiler. The details of this will depend on the particular compiler in use.)

Note that tf_round_number() and tf_trunc_number() both require the terminal manager.

If your ADC does use terminal manager functions, take the following steps to remove the calls:

1 Replace calls to tf_ functions, by corresponding calls to sf_ functions where these exist. (If you are using the old 4.4xx function names, refer to the DP4 C Functions Reference Manual or dp4.h to see what the new names are)

2 If you are using tf_round_number(), tf_trunc_number() or bcd conversion functions, compile and link in the module notrmbcd.c . This will cause versions of these functions that do not use the terminal manager to be linked into your program.

If you wish to continue to use terminal manager functions from inside an ADC, you need, except in DOS, to load a copy of TRM for the ADC's use. To do this, you must (on 16 bit platforms) link with the appropriate operating system interface file ( mcwinif, mcos2if and so on) and call the following functions:

• trm_load_trm() to load terminal manager.

• tf_init_tf() to enable tf_ functions

• trm_init_trm() to enable other terminal manager functions

At unload time you must call:

trm_end_trm() and tf_end_tf()

Under Windows you may wish to skip the calls to trm_init_trm() and trm_end_trm(). In this case, only a restricted set of TRM functions are available, but TRM does not create a window.

Multi-Threading Issues

There is a possible problem in multi-user configurations, where the ADC is loaded either on the server machine or on a stand-alone machine. If the ADC performs some lengthy operation on behalf of one user, other users are prevented from simultaneously using the database manager. A possible solution to this, for operating systems that have the appropriate functionality, is to create a thread or to fork and to do the work in that. In this case the main thread must return to the application immediately, and the application will have to make a subsequent call to the ADC later on to retrieve the result of the operation. The DP4 IPC mechanism, and the DP4 database manager are single threading, and until the ADC returns other applications will be blocked from calling the database manager.

In the Unix version only, there is an alternative facility which allows the application to be suspended until the operation is complete without holding up other processes: if the user-supplied code is going to perform an operation which will take a significant time, such as accessing a remote host, it can first call the function afork(). This will fork the process and allow requests from other application programs to be processed, so that in this case the main thread does not need to return until the lengthy work is completed. afork() function returns -1 for an error ( EAGAIN or ENOMEM in errno ) and a non-negative value for success. If any of the user-supplied functions calls fork(), the user-supplied function must subsequently call exit() in all but one of the resulting processes. This feature is only available on Unix, where the network manager program itself is multi-threading. It cannot be used in configurations where there are chained ADCs, except for the top level ADC (the one loaded without −chain ). It should be noted that after the fork() the child process becomes the main ADC process. The parent process will terminate once it has completed the lengthy operation and returned to the calling application.

Debugging

ADCs normally run detached from any console which makes debugging them difficult. You may wish to consider developing any functions with serious functionality within a conventional program, and then move them into your ADC when they are working properly.

On Unix/Linux you can use printf() to insert diagnostics in your ADC. These will appear jumbled up with the regular output from applications. You could load the ADC on a different console from your application to avoid this problem.

You may also use printf() in ADCs for other operating systems, except that printf() cannot be used in object code portable programs unless they are built with Metaware C. On FlexOS ADCs accept the -nodetach command tail to prevent them from detaching from the console. In 32 bit Windows environments printf() output can be directed to the DP4 error log program, provided you link your ADC with the syslibt.lib library. Information on loading the error log program is contained in the Guide to DP4 configuration, and also here. In 16 bit Windows environments printf() output can be sent to DBWIN if you link with the wprintf module.

Example ADCS

Two example ADCs are supplied on the DP4 CD ROM.

1. AUXDEBUG
2. AUXDUPE

Another example ADC ram_db was also supplied in the past, but it has been removed, partly because it is no longer useful, and partly because some of the code it contained was unsafe.

AUDEBUG

AUXDEBUG is an ADC which can be used to trace all the calls made to the database manager. It is often useful for diagnostic purposes.

Normally it displays table numbers and suppresses information about base dictionary accesses.

AUXDEBUG accepts a number of command line options . These are detailed below:

• -name Gives table names not numbers (but reduces performance slightly).
• -time Adds time to all output lines (for performance measurement).
• -all Prints information for base dictionary accesses as well as user tables.
• -data nnnn Prints the first nnnn bytes of each data record in hex.

If you want to rebuild auxdebug there are a few complications.

1. If you want to build the DOS version with Microsoft C you need to remove the lines which call DosGetDateTime().

2. If you want to build the Win16 version you need to compile with WINDOWS defined.

3. If you want to build the Win32 version you need to link with syslibt.lib. This contains a replacement printf() routine which can output to a console or to the DP4 ERRLOG program. Click here for more information about ERRLOG.

AUDUPE

AUXDUPE is an example of a Y pipe ADC

AUXDUPE is an ADC which tracks all the records accessed by an application, and creates a small database called DUPEDB by default) containing exactly those records. If necessary table and PC information is copied first.

It is especially useful for creating a small example database that can be used to reproduce a problem when the real database is to big to send.

Records are only copied if the timestamp of the record is prior to the original timestamp of the database being duplicated. Hence DUPEDB always reflects the state of the original database at load time.

It is loaded as follows:

AUXDUPE -DB dbname [-newdb dbname]

If -newdb is not specified a database called DUPEDB is opened/created. (DUPEDB is created from TINYBD in the first instance). TINYBD is a BD with all but the core dictionary tables removed. You can use BD instead if you like.

Limitations

AUXDUPE attempts to copy records deleted as a result of a CHILD_DELETE operation. However when a CHILD_DELETE operation cascades through several levels only the first two levels of children are guaranteed to be duplicated.

AUXDUPE does not intercept any changes to the database made as a result of file operations.