ADC Developers Manual

DP4 Auxiliary Database Controller Developers Manual

ADC - DP4 Database Middleware

Diagram of ADC An ADC (Auxiliary Database Controller) is a piece of software that sits in between a DP4 application and the DP4 database manager, and which extends or alters the standard DP4 functionality. Conceptually it is similar to the plug-ins used with Web Browsers or programs such as Adobe Photoshop. For the C Programmer, writing an ADC is similar to subclassing a pre-existing Window class in a Windows program or deriving one class from another in a C++ program.

When an ADC is plugged into the DP4 Database Manager, all calls made to the database manager by the DP4 application are first passed to the ADC for processing. The ADC is free to act on the intercepted call before returning control back to the database manager.

On intercepting a call the ADC may:

  • Make DP4 calls
  • Make native operating system calls or calls to third party software
  • Return control back to the database manager

Purpose of ADCs

The DP4 Auxiliary Database Controller (ADC) is designed to meet three needs.

  1. Firstly, you may require that some or all of your data should physically reside on the same machine as your DP4 database, but in a different, non-DP4 database or file system, perhaps because you have existing programs that can only access some other database. Or you may require to have some or all of your data on another machine altogether. But you want to make all the data appear to be logically in the DP4 database, because:

    • In this way, your application programmers need only learn one set of DP4 access methods, and only understand one kind of data structure. They can ignore the practicalities of how the data is actually accessed and stored.

    • You can then use QA Build, the DP4 Report Writer, and other DP4 utilities, on all the data.

    Since QA Build supports user exits written in C, you have a choice whether to use C exits or an ADC to access non-database files. Itim Technology Solutions generally prefers the ADC approach, as it provides a simpler model for the application programmer.

  2. Secondly, you may want to put business logic into the database manager. You could use this to:

    • Support computed values on the database; for example, you could have order line value, computed at read time as quantity times product price.

    • Implement additional integrity constraints; for example, you could arrange so that a product can only be deleted if its replacement is in stock.

    • Implement views; for example, you could define a table 'overdue orders' in your data dictionary, but post no data to it. At read time the ADC would read the main orders table and return a record only if it was overdue.

    • Implement a sophisticated security system; for example, you could make a certain table accessible to different people at different times of the day.

    • Write a special audit file logging all reads of sensitive data.

  3. Thirdly, you may wish to write some service routine of your own, for instance to support comms, and to do this in a way that allows your application programs to remain operating system independent. You can achieve this by putting your service routine in the ADC, and calling it through the provided special user function.

Of course you may do several of the above in a single ADC. For example, you could have a commodity data table, where the current price was looked up at read time from a remote information service if the price on the database was more than 2 hours out of date.

Loading an ADC

On Win32 platforms ADCs are loaded in the [startup] section of the DP4 configuration file as part of the DP4 service, and the command lines given here would be preceded with 1=, 2=, and so on. On other platforms they are loaded from the command line or a batch file/shell script. However the two processes are very similar, and the following instructions apply to all platforms unless otherwise noted. Remember that on Win32 platforms you can get the DP4 service to load alternative configurations of DP4 by specifying an an alternative name for the [startup] section of the DP4 configuration file, for example:
srvw32 -start -startup=startup_adc -preload=preload_adc. This may be convenient if you sometimes want to load with an ADC and sometimes not.

The database manager (srvX) or possibly a network requester such as tcpX must be loaded first, with a -aux command tail. The command tail −remain should also be used except on Windows platforms.

The ADC is then loaded, with or without a −remain command tail. If you load the ADC without −remain, it will unload when the next DP4 program terminates.

Loading Multiple ADCs - Chaining

Chained ADCs A chain of ADCs may be loaded. This allows a cascade of interceptions. For example, with a chain consisting of the application, 3 ADCs and a server, it is possible for the first ADC to intercept a call from the application, and return 1 from the relevant u_ routines. The second ADC will then pick up the call and so on until the server is finally activated. The same principle of server-call interception operates as in a single ADC-server configuration.

The order of ADC loading is critical in a chaining situation:

  1. Firstly the DP4 database manager must be loaded as:

    srvX -aux -chain n -remain
    or, on Win32 platforms:
    1=dp4srvr.w32 -aux -chain n

    where n is the highest value in the chain - that is, the chain consists of n+1 ADC nodes.

  2. The highest ADC is then loaded:

    ADCName -chain n -remain
    or, on Win32 platforms:
    2=ADCName -chain n

    The remaining nodes are then run in descending order:

    ADCName -chain n-1 -remain
    .
    .
    .
    ADCName -chain 2 -remain
    ADCName -chain 1 -remain
    ADC -remain

    Note that the last ADC is run without a -chain tail.

  3. Once the last ADC has been loaded DP4 applications may be loaded in the normal way.

The ability to chain ADCs is available in all 4.5xx or later versions of DP4. A decision to use chained ADCs should not be taken lightly. There is an inevitable performance penalty in using ADCs, because of the extra IPC required. Wherever possible you should build a single ADC combining all the extra functionality you need.

ADC Implementation

The ADC is supplied as a skeleton program, into which you can add whatever code you need. ADCS are loaded into memory 'on top of' the standard database manager (SRVn,or DP4SRVR.W32) as described above. Thereafter, all calls from application programs to the database manager are intercepted by the ADC. Depending on your code, the ADC can pass the call on to the standard database manager or deal with it itself.

The method of interception is as follows:

  • On all operating systems except MS-DOS, some kind of message queue or pipe is used for IPC (Inter-Process Communication) between DP4 applications and the database manager. The details of this are of no relevance to ADC developers, (except that very occasionally some change may be made in the IPC mechanism used, which means that ADCs written for an earlier release of DP4 need to be relinked for the new release). The ADC takes over the regular queue, thus receiving all calls to the DP4 database manager made by applications. The ADC will handle certain calls itself (possible making other calls to the DP4 database manager in the process). Other calls will be passed onto the DP4 database manager. The DP4 database manager most use a different message queue from normal, so that the ADC can make calls to it without intercepting them again. The database manager is loaded with a command tail to tell it where to look for messages.

  • Under MS-DOS, the details are generally similar except that IPC uses a software interrupt mechanism. The ADC redirects the software interrupt that normally points at SRV2 to point at itself, and saves the address of SRV2 so that database calls from within the ADC can be passed to SRV2.

The ADC is structured so that for each DP4 function that calls the database manager there is a corresponding function in a user written module of the ADC. (The names of these functions are fixed, and based on the pre 4.500 C library function names) The code of each user function will usually consist of two parts - the first part will examine the incoming call and decide whether to handle it or pass it on to the regular database manager. The second part of the function will contain code to handle the function. The next section explains the use of the user functions in detail.

The Skeleton ADC Functions

auxuser.c provides the skeleton ADC variants of standard DP4 Database Manager calls. You should modify the skeleton functions it contains to perform whatever actions are required. It is probably best to rename this module before using it in your own ADCs. If necessary or convenient you can of course divide its functions between more than one source file. It should not be necessary to modify any of the other ADC source modules. These are supplied in source form purely so that you can build your ADC using whatever compiler options you wish. If you modify other modules you will compromise the portability of your ADC both to other platforms, and to future releases of DP4.

The modified auxuser.c (or its replacement) is compiled in the normal ways, and linked with the other ADC source files appropriate to the platform you are creating an ADC for. The exact set of files used depends on the platform, as does the method of building the ADC. Information on building for the various platforms, and other platform specific information can be found at the end of this manual on these pages:

  • Win32 Platforms
  • Unix/Linux Platforms
  • 16 bit Platforms

The functions in auxuser.c are usually called whenever the application program makes the corresponding call to the database manager. However, it is important to note that there is not necessarily a one to one correspondence between application level calls to the DP4 database manager and calls made to the ADC. For example the DP4 networking layers may translate a call to the rec_fetch() function into a blk_fetch() , so that subsequent calls to rec_fetch() can be handled locally, or queue updates until so that a "batch" of updates can be sent across the network in a single operation. These optimisations can be controlled for DP4 networking using the -noblock and/or -nofaster command tail. However, applications may also use the bf_fetch() function to perform similar record buffering at an application level, and in this case there is no way of disabling the behaviour without altering the application.

Each user written function should return 0 if processing for the call is complete, and the call should not be passed on to the original database manager (or next ADC in any chain). The functions return 1 to indicate that the call should be passed on to the original database manager. Where possible, it is better to code:

int u_fetch(flags,datarec,index,role,keys)
{
  return 1;
}

than to code:

int u_fetch(flags,datarec,index,role,keys)
{
   rec_fetch(flags,datarec,indexrec,role,keys);
   return 0;
}

The timestamp of all records fetched explicitly by an ADC as opposed to being read by the regular DP4 database manager as a result of returning 1 in a u_xxx function have their timestamp set to zero in case the timestamp is not valid for a corresponding DP4 database record. This can cause additional commit failures in an application. For further information on this consult the DP4 networking and Resilience manual. You can prevent this from happening by compiling the auxproc.c module with the symbol KEEP_TIMESTAMPS defined. In this case you must ensure that the timestamp is zeroed when necessary -i.e. when the timestamp of a record returned to by a fetch() function does not match the timestamp of a corresponding record on the real DP4 database and the record may subsequently be posted as a real record. It is especially important to zero timestamps where two different DP4 databases are presented as one logical database and records may be transferred between the databases.

The names of the functions in the user written module are given in the following table, together with the corresponding function(s) in the DP4 C library, and notes on their use.

User Functions in an ADC
ADC Function | DP4 Function | Comments
void u_startup( int argc, char **argv) -

Called when ADC is loaded. Used for "one time" initialisation.The command tail parameters argc and argv which are supplied to main() are passed to this function (the DP4 library tail_ functions can be used to access command line parameters as well).
void u_terminate(void) - Called when ADC is unloaded (at DP4 shutdown time). This function can be used to clear up things like allocated memory and temporary files that your ADC may have opened.
void u_pre_process( int in_net) - Called on entry for each call into the ADC. The global parameter block and data areas can be examined. Can be used for common functionality required for all calls to DP4 DBMS (for example audit trails or diagnostics). The parameter is always FALSE.
void u_post_process(int in_net) - Called on exit for each call into the ADC (after passing on to the regular DP4 DBMS). The global parameter block and data areas can be examined. Can be used in similar way to u_pre_process().
int u_init() | srv_enable() | Called when process connects to DP4 database manager. Can be used for per-process initialisation. You must either pass this function on, or call srv_enable(). If you want to handle calls to the system database, then perform any initialisation required for the system database here, as it is not normally opened explicitly by applications.
int u_finish() | srv_disable() | Called as part of disconnecting from DP4 database manager. You must either pass this function on, or call srv_disable() if u_init() enabled the DP4 database manager. This function may be called without databases being closed first.
int u_dbopen(char *dbname, int generation, int flags, int status) | db_open() | Called to open a specific DP4 database. You must pass this call on or call db_open() to open the real DP4 database if there is one. Commonly you will need code in this function which "remembers" the value for db_nr returned for a call to the DP4 DBMS db_open(), so that subsequent functions can decide whether a call is to a database of interest or not. Programs, especially QAB programs, may call this several times for a single database. Each call to db_open() "replaces" the open mode and generation for the program.
int u_dbclose(void) | db_close() | Called to close DP4 database. It is not guaranteed that programs call this function for all databases they open before terminating. The database should close (for the current user) when this function is called, even if db_open() has been called several times.
int u_update_db( int flags) | db_update(), db_commit(), db_checkpoint() | Called for all commit() type functions
int u_decommit(void) | db_decommit()
int u_retrieve( struct DATAREC *datarec) | rec_retrieve() | This function can only be called if RETRIEVE_ENABLE was specified to db_open(). No standard DP4 programs use this function, so you are unlikely to need to implement it.
int u_autoinc( struct DATAREC *datarec) | rec_autoinc()
int u_ftchprm( int flags, struct DATAREC *datarec, int keys) | rec_fetch_main() | Fetches on main index
int u_ftch2nd(int flags, struct DATAREC *datarec, struct DATAREC *indexrec, char *role, int keys) | ftch2nd() | Old function for fetch on secondary index,. This function is no longer used by standard DP4 programs.
int u_fetch( int flags, struct DATAREC *datarec, int index, int rolenr, int keys) | rec_fetch() | Fetch on secondary index. Note that if an application uses sec_fetch() on the main index the call will go to u_ftchprm() and not u_fetch().
int u_blk_fprm( int number, int flags, struct DATAREC *datarec, int keys) | blk_fprm(), bf_fetch() | Block fetch on main index. This function and the next will commonly be called from QAB for "multiple choice" and window access. Implementing this correctly helps improve performance of DP4 networking
int u_blk_fetch( int number, int flags, struct DATAREC *datarec, int index, int rolenr, int keys) | blk_fetch(), bf_fetch() | Block fetch on secondary index. Implementing this correctly helps improve performance of DP4 networking
int u_blk_f2nd( int number, int flags, struct DATAREC *datarec, struct DATAREC *indexrec, char *role, int keys) | blk_f2nd() | Old form of block fetch on secondary index. This function is never used by standard DP4 programs. Calls to ftch2nd() may be translated to this function by the network requester.
int u_copy2parent( struct DATAREC *from, struct DATAREC *to, char *role)
int u_copy2child( struct DATAREC *from, struct DATAREC *to, char *role) 		copy2parent()
copy2child() 		Obsolete key copy functions. These are never used by standard DP4 programs
int u_ncpy2parent( struct DATAREC *from, struct DATAREC *to, int role)
int u_ncpy2child( struct DATAREC *from, struct DATAREC *to, int role) 		rec_parent_copy()
rec_child_copy() 		Key copy functions. These are sometimes used by DP4 programs like QAB to copy keys from one table to another. Note that the parameter order of the destination and source tables is reversed in the ADC code compared with the C library functions.
int u_verify( struct DATAREC *datarec, int flags) | rec_verify() | Called by some applications to check a record has not changed during this transaction
int u_post( struct DATAREC *datarec, int flags) | rec_post() | Called to update records. Calls to this function may be delayed by buffering in the DP4 network requester. Be wary of updating external files in this function as transaction may not be committed later on.
int u_kill( struct DATAREC *datarec, int flags) | rec_kill() | Called to delete records. Calls to this function may be delayed by buffering in the DP4 network requester. Be wary of updating external files in this function as transaction may not be committed later on.
void u_user_auxdbms( void * data, int length) user_auxdbms()

This function is not used by standard DP4 programs. You can use it to implement any functionality you wish. However, another commonly used technique to implement additional functionality that does not correspond closely to DP4 functions is to add meta-tables to your database. These are tables defined using MAKEDB in the normal way, but do not contain data. Calls to regular DP4 functions for these tables are interpreted specially. The @network and @net_status tables used in DP4 resilience are an example of this type of table.

The DP4 library function user_auxdbms() returns TRUE to the application if the auxiliary db manager is running and the user-written code in this function has intercepted the call. If the call is passed on to the database manager, it will be ignored and the value FALSE will be returned to the application.
void u_other( struct SRV_PARA *para, void *data0, void * data1) | Various | Calls to various specialised functions such as db_get_generation(), and the sf_() file functions go through this function.

Calls available to ADC functions

Your code can make full use of whatever standard DP4 database access functions are required. You can also use all the other DP4 library functions that do not use the DP4 terminal manager. (You cannot normally use the terminal manager functions because an ADC does not load this. There is more information about this in a later section, ADC calls to Terminal Manager Functions.) There may also be special considerations when using DP4 file handling functions in the sf_xx functions, or the srv_mem_xx memory workfile functions.

datafit.h lists all the DP4 C functions grouped by which, if any, of the database manger and terminal manager are used to implement them. Common database access functions that can be safely used are:

db_checkpoint()
db_close()
db_commit()
db_decommit()
db_open()
db_update()
rec_autoinc()
rec_fetch() or rec_fetch_main()
rec_kill()
rec_post()
rec_retrieve()
rec_verify()

Unless you are writing a 16 bit object code portable ADC you can use all standard C standard file functions of your operating system. Similarly you can use any functions in third party libraries except in object code portable ADCs.

With the exception of u_dbopen() the appropriate database is automatically open when the corresponding u_ functions are entered and the generation and number system are set to the values appropriate for the calling application program. The values of the global variables there, locked and so on will be passed back to the application program when these functions return.

Two additional calls to the original database manager are available, which cannot be made from application programs. These are chkpt_check() and chkpt_action().

chkpt_check() performs all the checking associated with db_update() and sets the value of fail_code appropriately; its returned value is TRUE if the db_update() would succeed.

chkpt_action() actually performs the db_update() without doing any checking first. Both functions require one parameter, which is the same as that supplied to db_update(). The user-supplied code for db_update() can therefore call chkpt_check() and also perform some other checking in order to decide whether to proceed. If it decides not to proceed, it should call decommit() to remove the pending updates from the work file in the original database manager, so that they are not actioned by the next successful call to db_update().

The function u_ftch2nd() corresponds to the application-program function ftch2nd(), which is a variant of rec_fetch() called by some old DP4 programs. This fetches a record on a secondary index, as rec_fetch() does, but the index and role are specified differently. For ftch2nd() the indexing table is determined by the type number of the 'indexrec' parameter, and the role is passed as a string. For rec_fetch() the index type number is passed directly, and the role is passed by number. Role numbers are associated with names by the @NAME_TABLE table on the database. rec_fetch() is more efficient than ftch2nd() and should be used in preference to it. Also programmers are occasionally confused by the two table records used with ftch2nd() and unintentionally attempt to pass data in the wrong copy.

In respect of all the various u_fetch() type functions, you may well find it convenient to write a single fetch function that accepts a combination of all the parameters: for example int u_fetch_common(int number, int flags, struct DATAREC * datarec, int index_table, int rolenr, char * rolename) and to call this function from all the various fetch functions. It is permissible for the blk_ functions to return a single record rather than the number requested. However, if at all possible you should return the number of records requested. DP4 programs that access the base dictionary using these functions may take fewer records being returned than requested as a signal that end of data has been reached,

Use of df_process variable

It should be noted that calls to DP4 library functions from inside an ADC are, by default, made on behalf of the application process. Therefore, if you want to access information in a temporary file on shared between all processes you should either create the file using native OS file handling functions, or if you wish to use DP4 functions, you should obtain a DP4 process number for the ADC itself by calling srv_init_process() at u_startup() time, and save the value this returns. Before calls to DP4 functions that logically belong to the ADC itself rather than the application, save the current value of the df_process variable, and temporarily set it to the ADCs saved process number for the duration of the call. If you do get a process number like this you must remember also to free it in u_terminate(). The special process number 1 is reserved for use by the DP4 database manager and may also be used by ADCs. However this will not work on a machine where the database manager is replaced with a network requester, nor can it safely be used with memory work files because of the risk of conflicts with the genuine database manager.

ADC calls to Terminal Manager Functions

In normal circumstances, ADCs should not call terminal manager functions. Under MS-DOS, it is usually harmless, because the call will appear to come from the application and the application's terminal manager will handle it. Under other operating systems, it is more serious because some kind of protection violation will most likely occur. (On the Win32 platform a few calls that do not depend on the normal initialisation routines might happen to work, but this cannot be relied upon.)

If you are not aware of which functions in the DP4 library can safely be called, you may find it helpful to refer to the datafit.h file. This lists functions in the following order:

  • Functions not requiring SRVn or TRMn
  • Functions requiring SRVn only
  • Functions requiring TRMn

Try linking your ADC with an option to write a symbol or map file that you can examine (auxlink /m will work for 16 bit versions). If you find that _trm_ncall has been linked in the map file, your ADC is probably calling terminal manager functions. (The name of this symbol may be slightly different in the map file because of name mangling by the C compiler. The details of this will depend on the particular compiler in use.)

Note that tf_round_number() and tf_trunc_number() both require the terminal manager.

If your ADC does use terminal manager functions, take the following steps to remove the calls:

1 Replace calls to tf_ functions, by corresponding calls to sf_ functions where these exist. (If you are using the old 4.4xx function names, refer to the DP4 C Functions Reference Manual or dp4.h to see what the new names are)

2 If you are using tf_round_number(), tf_trunc_number() or bcd conversion functions, compile and link in the module notrmbcd.c . This will cause versions of these functions that do not use the terminal manager to be linked into your program.

If you wish to continue to use terminal manager functions from inside an ADC, you need, except in DOS, to load a copy of TRM for the ADC's use. To do this, you must (on 16 bit platforms) link with the appropriate operating system interface file ( mcwinif, mcos2if and so on) and call the following functions:

  • trm_load_trm() to load terminal manager.

  • tf_init_tf() to enable tf_ functions

  • trm_init_trm() to enable other terminal manager functions

At unload time you must call:

trm_end_trm() and tf_end_tf()

Under Windows you may wish to skip the calls to trm_init_trm() and trm_end_trm(). In this case, only a restricted set of TRM functions are available, but TRM does not create a window.

Multi-Threading Issues

There is a possible problem in multi-user configurations, where the ADC is loaded either on the server machine or on a stand-alone machine. If the ADC performs some lengthy operation on behalf of one user, other users are prevented from simultaneously using the database manager. A possible solution to this, for operating systems that have the appropriate functionality, is to create a thread or to fork and to do the work in that. In this case the main thread must return to the application immediately, and the application will have to make a subsequent call to the ADC later on to retrieve the result of the operation. The DP4 IPC mechanism, and the DP4 database manager are single threading, and until the ADC returns other applications will be blocked from calling the database manager.

In the Unix version only, there is an alternative facility which allows the application to be suspended until the operation is complete without holding up other processes: if the user-supplied code is going to perform an operation which will take a significant time, such as accessing a remote host, it can first call the function afork(). This will fork the process and allow requests from other application programs to be processed, so that in this case the main thread does not need to return until the lengthy work is completed. afork() function returns -1 for an error ( EAGAIN or ENOMEM in errno ) and a non-negative value for success. If any of the user-supplied functions calls fork(), the user-supplied function must subsequently call exit() in all but one of the resulting processes. This feature is only available on Unix, where the network manager program itself is multi-threading. It cannot be used in configurations where there are chained ADCs, except for the top level ADC (the one loaded without −chain ). It should be noted that after the fork() the child process becomes the main ADC process. The parent process will terminate once it has completed the lengthy operation and returned to the calling application.

Debugging

ADCs normally run detached from any console which makes debugging them difficult. You may wish to consider developing any functions with serious functionality within a conventional program, and then move them into your ADC when they are working properly.

On Unix/Linux you can use printf() to insert diagnostics in your ADC. These will appear jumbled up with the regular output from applications. You could load the ADC on a different console from your application to avoid this problem.

You may also use printf() in ADCs for other operating systems, except that printf() cannot be used in object code portable programs unless they are built with Metaware C. On FlexOS ADCs accept the -nodetach command tail to prevent them from detaching from the console. In 32 bit Windows environments printf() output can be directed to the DP4 error log program, provided you link your ADC with the syslibt.lib library. Information on loading the error log program is contained in the Guide to DP4 configuration, and also here. In 16 bit Windows environments printf() output can be sent to DBWIN if you link with the wprintf module.

Example ADCS

Two example ADCs are supplied on the DP4 CD ROM.

  1. AUXDEBUG
  2. AUXDUPE

Another example ADC ram_db was also supplied in the past, but it has been removed, partly because it is no longer useful, and partly because some of the code it contained was unsafe.

AUDEBUG

AUXDEBUG is an ADC which can be used to trace all the calls made to the database manager. It is often useful for diagnostic purposes.

Normally it displays table numbers and suppresses information about base dictionary accesses.

AUXDEBUG accepts a number of command line options . These are detailed below:

  • -name Gives table names not numbers (but reduces performance slightly).
  • -time Adds time to all output lines (for performance measurement).
  • -all Prints information for base dictionary accesses as well as user tables.
  • -data nnnn Prints the first nnnn bytes of each data record in hex.

If you want to rebuild auxdebug there are a few complications.

  1. If you want to build the DOS version with Microsoft C you need to remove the lines which call DosGetDateTime().

  2. If you want to build the Win16 version you need to compile with WINDOWS defined.

  3. If you want to build the Win32 version you need to link with syslibt.lib. This contains a replacement printf() routine which can output to a console or to the DP4 ERRLOG program. Click here for more information about ERRLOG.

AUDUPE

AUXDUPE is an example of a Y pipe ADC
Diagram of AUXDUPE

AUXDUPE is an ADC which tracks all the records accessed by an application, and creates a small database called DUPEDB by default) containing exactly those records. If necessary table and PC information is copied first.

It is especially useful for creating a small example database that can be used to reproduce a problem when the real database is to big to send.

Records are only copied if the timestamp of the record is prior to the original timestamp of the database being duplicated. Hence DUPEDB always reflects the state of the original database at load time.

It is loaded as follows:

AUXDUPE -DB dbname [-newdb dbname]

If -newdb is not specified a database called DUPEDB is opened/created. (DUPEDB is created from TINYBD in the first instance). TINYBD is a BD with all but the core dictionary tables removed. You can use BD instead if you like.

Limitations

AUXDUPE attempts to copy records deleted as a result of a CHILD_DELETE operation. However when a CHILD_DELETE operation cascades through several levels only the first two levels of children are guaranteed to be duplicated.

AUXDUPE does not intercept any changes to the database made as a result of file operations.


What's New in the DP4 ADC Developer's Guide

This page will tell you about recent changes to this manual. The most recent changes are mentioned first

New in 4.621

  • 18 Mar 2004 Description of ram_db removed

New in 4.620

  • 17 Oct 2002 What's New Page added

New in 4.619

  • 12 Apr 2001 New section added on data independence issues that arise with business logic ADCs.


Issues arising with Business Logic ADCs

The example ADCs supplied with DP4 are generic ADCs - they do not contain any code specific to a particular database, or require any knowledge of the structure of tables on a database. The same is unlikely to be true for an ADC which implements business logic. Typical tasks for such an ADC might be to maintain the total of a particular field across a group of records (or even a whole table) in a record in another table, or to create journal records containing information about fields updated in records in another table. When writing such an ADC you need to consider the problems that can arise through the use of data independence.

For example consider the following scenario:

  1. An order entry program is written to record sales orders. An ADC is written which intercepts updates to the order-line table and updates a quantity_in_stock field in the material records corresponding to order lines. Both the ADC and the order entry program use a header file that uses the database structures from generation 1.

  2. Later on it becomes necessary to add a new "status" field to the order line table. The "status" field is not required by the order-entry program itself, but by some batch program that is run later on in the life cycle of the order. Adding the "status" field causes data independence to be required for the order line table, and, as it happens, changes the position of the quantity field within the order line. (In fact adding a new field may well not change the position of the existing fields in a record, but it is likely to do so if the field was already used elsewhere on the database, or the record contains fields with roles and the new field does not have a role). The database generation is now 2

At this point the ADC and the order-entry program still both perceive the order-line table with the structure as it was at generation 1. Therefore the ADC will continue to update the stock table correctly when it intercepts calls from the order entry program. But what happens when the batch program is run? It will have been developed for generation 2, and will have a different view of the structure of the order line table. Perhaps we will be lucky and the ADC will somehow know that it does not have to update the stock table when intercepting calls from this program, but quite possibly we won't be lucky. It is very likely that the ADC will misbehave quite badly with this program.

The obvious solution is to run LIBMAKE to recreate the header files for both the order entry program and the ADC, and to rebuild both the program and the ADC. Then all the components in the system will use the most up to date structure for the table. In general you should always rebuild your programs so that the latest structure of the database is used for all tables, because relying on data independence to keep your program working will cause some performance degradation. Also updating a record from an old program will cause data to be lost from any fields added since that program was built.Prior to release 4.523 there are no other options: When you have a business logic ADC you must ensure that all programs, and the ADC itself, are compiled using header files appropriate to the latest structure of the database.

There is one exception to this - if the tables that have changed are accessed only by the ADC, then you do not need to recompile the applications, provided the ADC is careful to set and reset the database generation number when accessing the changed tables. For example, suppose the ADC writes data to a journal table which has had new fields added to it since the applications were written. You could update the journal tables inside the ADC with code something like this:

int u_post( struct DATAREC * datarec,int flags)
{
  if (db_nr == my_db_nr && datarec->typenr == _ORDER_LINE)
  {
     int save_generation = db_generation;
     struct ORDER_LINE * order_line = (struct ORDER_LINE *) datarec;
     db_generation = SALESORD_GENERATION;
     sf_get_date_time(&journal.update_date,&journal.update_time);
     journal.order_nr = order_line->order_nr;
     journal.linenr = order_line->linenr;
     MOVE(journal.username,username);
     rec_post(&journal.l);
     db_generation = save_generation;
  }
  return 1;
}

The expression in the if statement is typical of a business logic ADC. The variable my_db_nr would be set in the code for u_dbopen(), when the database for which journalling is required is opened. You can then test for tabels of interest by examining the typenr field of records.

The constant SALESORD_GENERATION represents the generation number for the database (SALESORD) at the time the ADC is compiled. Prior to release 4.523 you have to define this constant for yourself, but from this release LIBMAKE will define a constant DBNAME_GENERATION when generating C header files. It is much safer to rely on a constant like this than querying the database generation, (for example by calling db_get_generation()), because a program can only safely use the current generation number if it dynamically queries the data structure, which requires a lot of extra knowledge of how DP4 works.

It is very important only to use a modified generation number when accessing data private to the ADC. For example suppose the code above is slightly modified:

int u_post( struct DATAREC * datarec,int flags)
{
  if (db_nr == my_db_nr && datarec->typenr == _ORDER_LINE)
  {
     int save_generation = db_generation;
     struct ORDER_LINE * order_line = (struct ORDER_LINE *) datarec;
     db_generation = SALESORD_GENERATION;
     sf_get_date_time(&journal.update_date,&journal.update_time);
     journal.order_nr = order_line->order_nr;
     journal.linenr = order_line->linenr;
     MOVE(journal.username,username);
     rec_post(&journal.l);
     rec_post(&order_line->l);
     db_generation = save_generation;
     return 0;
  }
  return 1;
}

This code is potentially unsafe if the order_line table has been modified since the calling program was written. The order_line structure is in the application's view of the structure, possibly missing fields that have been added since (although this is contrary to our assumption that only the journal table has been modified, it is worth bearing this possibility in mind). Now however the database manager will believe the order_line structure is in the structure for the generation used by the ADC. This could lead to problems with data independence.

Use of Data Independence in an ADC

From version 4.523 it is possible, but by no means recommended, to use data independence in an ADC. In order to use this facility you have to link in the new ADC module AUXDI. This implements a function that can convert a DP4 record between any two generation numbers. The actual implementation of the function is inside the regular database manager. Using this function will certainly impact performance to some extent, so the function should be used sparingly. The following example shows how the code above could be adapted to make use of this facility:

int u_post( struct DATAREC * datarec,int flags)
{
  if (db_nr == my_db_nr && datarec->typenr == _ORDER_LINE)
  {
     int save_generation = db_generation;
     struct ORDER_LINE * order_line = (struct ORDER_LINE *) datarec;
     struct ORDER_LINE buffer;

     db_generation = SALESORD_GENERATION;
     sf_get_date_time(&journal.update_date,&journal.update_time);
     if (save_generation != SALESORD_GENERATION)
     {
       swap_data(&buffer.l,SALESORD_GENERATION,&order_line->l,save_generation);
       order_line = &buffer;
     }
     journal.order_nr = order_line->order_nr;
     journal.linenr = order_line->linenr;
     MOVE(journal.username,username);
     rec_post(&journal.l);
     db_generation = save_generation;
  }
  return 1;
}

swap_data() is declared like this:

struct DATAREC * void swap_data(short * to,int to_generation,short * from, int from_generation)

The function converts the from record, based on the structure at database generation from_generation, to the to record, based on the structure at database generation to_generation. You should always use a different record area for the from and to records. The return value is to. to_generation and from_generation can be any valid generations for the database. You should note that conversion always takes place by converting the structure from the "from" generation up to the current generation, and then converting back to the "to" generation, so the values in any fields common to the two generations but subsequently deleted will be lost. Internally the database manager only supports conversion between generations where one generation is the current generation.

Using this function does not free you from having to manage the generation number carefully. You must still ensure that the generation number has the appropriate value when calling the database manager, and that data is returned to the application in the correct form. There are two possible ways you could do this, illustrated for calls to u_ftchprm():

  • Convert all records of interest to the ADC to the generation expected by the ADC, make all calls to the database manager using this generation and the converted records. Convert the internal record back to the generation expected by the application prio to reurning data to it.

    int u_ftchprm(int flags,struct DATAREC * datarec,int keys)
{
  if (db_nr == my_dbnr)
  {
     static struct DATAREC my_datarec;
     int save_generation = db_generation;
     db_generation = SALESORD_GENERATION;
     if (SALESORD_GENERATION != save_generation)
       swap_data(&my_datarec.l,SALESORD_GENERATION,datarec,save_generation);
     if (rec_fetch_main(flags,&my_datarec.l,keys)) /* Read record in current generation */
       work(&my_datarec.l); /* Do special ADC work for record that has been fetched */

     if (SALESORD_GENERATION != save_generation)
     {
       /* convert record back to form expected by application */
       swap_data(datarec,save_generation,&my_datarec.l,SALESORD_GENERATION);
     }
     db_generation = save_generation;
     return 0;
  }
  return 1;
}

  • Make all calls to the database manager using the generation specified by the application. Convert records to the generation expected by the ADC as required.

    int u_ftchprm(int flags,struct DATAREC * datarec,int keys)
{
  if (db_nr == my_dbnr)
  {
     static struct DATAREC my_datarec;
     if (rec_fetch_main(flags,&datarec->l,keys)) /* Read record in old  generation */
     {
       if (SALESORD_GENERATION != db_generation)
       {
         /* convert to current generation */
         swap_data(&my_datarec.l,SALESORD_GENERATION,datarec,save_generation);
       }
       work(&my_datarec.l); /* Do special ADC work for record that has been fetched */
     }
     return 0;
  }
  return 1;
}

Of the two methods, the first is probably to be preferred for the following reasons:

  • Internally the effect of the first call is to move calls to swap_data() from inside the database manager to the ADC - using the first method, assuming the ADC is for the latest generation, the database manager will not need to invoke data independence for the call to rec_fetch_main(). With the second method there will be two calls to the data independence mechanism internal to the database manager, essentially the same as the calls made explicitly when using the first method.

  • Using the second method, any fields missing from the record in the generation used by the application will be given default values rather than the true value in the updated record. You have to use the first method to see the record accurately.


Building Win32 ADCs

The sample makefile supplied DP4ADC.MAK should be used either directly or as a template for your own. Remember that the auxuser.c module supplied is an empty source code template for you to add your own code. You may wish to rename this file for your own purposes. If you do this, do not link the auxuser.obj file supplied otherwise you will get multiply defined symbols.

To build the example AUXDEBUG program, use the following command line:

NMAKE -F DP4ADC.MAK AUXDEBUG.W32

Viewing Diagnostics with ERRLOG

On Win32 platforms the debug output from AUXDEBUG (and many other DP4 programs) is generated using special printf() statements, which are normally displayed in the DP4 error logging program ERRLOG.W32 . (The printf() used by the Windows AUXDEBUG program is implemented in syslibt.dll and linked in from syslibt.lib rather than from the normal C library. So this library must be linked before libc.lib.)

If you want to use ERRLOG, add/modify the following sections of your datafit.ini file:

[system]
load_errlog=1
[errlog]
debug_echo=1

You can configure ERRLOG's functionality by editing the lines in the [errlog] section. For example set debug_echo=0 to suppress the real-time window display. The other entries in this section control diagnostics from various DP4 libraries or programs.

It is also possible to output printf() to a special console window. Each DP4 process will have its own console window which will receive its diagnostics. (This works on all Win32 platforms, and is not NT specific as was stated in an earlier version of this document)

To do this add/modify the following section in your datafit.ini file:

[win32]
debug_console=1

On Window Nt and 2000 You must also ensure that the service startup parameters have the 'Allow service to interact with desktop' option enabled (using the 'Startup' button in the Service control panel).

You can use errlog diagnostics in any DP4 program on win32 platforms. Simply follow the same procedure of using printf() for diagnostics, and link with syslibt.lib before libc.lib. Please note that floating point output is not currently implemented in this version of printf().


Building Unix/Linux ADCs

The following source files are required for a Unix ADC:

aux3.c
auxproc.c
auxcall.c
auxcall3.c
user-defined module (for example, your adapted auxuser.c )
You will also need all the various files these include such as adc.h,ccsrvcal.c and so on.

Unix/Linux ADCs should be built using a native compiler. From 4.520 they cannot be built using the 16 bit object code portability feature. A sample makefile called unix_adc.mak is supplied. This can easily be adapted to build any ADC on any version of Unix or Linux. You will need to make sure the CFLAGS and LDFLAGS are set appropriately for your environment. The default value of these variables will probably select the one byte alignment required for compiling DP4 programs correctly. The file contains suitable values for the most popular varieties of Unix and Linux, and you should choose one of these.


Windows 3.x Auxiliary Database Controllers

With Microsoft C only, it is possible to create an ADC for Windows 3.x. When creating a Windows ADC you are strongly recommended to use small model if possible. It is advisable to compile all the ADC modules to match your chosen compiler version and compilation options. The instructions for building Windows 3.x ADCs are the same as for other 16 bit ADCS and can be found here.

You should not run a system including 16 bit Windows ADCs on any newer version of Windows that supports the Win32 APIs, especially in conjunction with DP4 networking programs. You should convert the system to use Win32 applications. We cannot officially support such configurations as there are insurmountable, though intermittent, problems with running them.

Calling Mechanism

The calling mechanism into the database manager was changed in 4.515 and again in 4.517. Prior to release 4.519 the DP4 database manager and 16 bit Windows ADCs also supported the old MS-DOS interrupt calling mechanism. This support was removed in 4.519, because it was considered obsolete, and also caused problems when attempting to run 16 bit Windows DP4 programs on other platforms that emulate Windows 3.x such as OS/2 or Unixware. Prior to 4.517 16 bit Windows ADCs did not require the -aux and −chain tails required for other platforms. Now they do need this information. However in order to maintain compatibility they attempt to discover their position in a chain automatically. You are recommended not to rely on this but to follow the same instructions as for other platforms.

Operational Considerations under Windows 3.x

Under Windows, when using ADCs, it is important that you are aware of the following operational considerations:

  • If you write an ADC for Windows, do not yield to another application unless absolutely necessary.

  • When a program calls the database manager, it may be busy doing processing for another application.(This does not happen unless a network requester or an ADC which yields is loaded). In this case the program loops repeatedly until the database manager can be called. Ideally, programs would do this by calling the GetMessage() function to yield. Unfortunately, this is not possible because the database manager may be called when there is no window for the application.

The command tails and loading order of them for the SRVW, NTBW, NTBMGR and AUXDISTR programs are the same as for other operating systems with the following difference:

  • Command tails for these programs can be specified in two ways:

    By a <program_name>=<command_line> section in the [progname] section of the datafit.ini file, for example:

    [NTBMGR]
    ntbmgr=‑server 2

By giving any command tails with the program manager icon or when the program is loaded from the File menu of the program manager. This overrides any command tails specified in datafit.ini for the program.


Compiling and Linking 16 bit ADCs

The following sections describe how to build an ADC on 16 bit Intel platforms. (MS-DOS, FlexOS, OS/2, Windows 3.x)

Linker option files are supplied which reference all the ADC modules except the user-defined module ( auxuser, auxdebug, auxdupe etc). These option files are named AUXn.LNK or AUXn.INP , where n relates to the operating system as per the above table. These option files are referenced by the AUXLINK batch files during linking. AUXLINK files are supplied for each environment supported for the ADC and are convenient tools for linking. It is important to use the correct parameters with AUXLINK, especially when the executable will later be converted for another operating system - run AUXLINK -HELP for advice.

The example link commands given in the following sections are for the example ADC AUXDEBUG . To build a different ADC replace " AUXDEBUG " with the name of your user-defined (auxuser-derived) module in the link command, or supply that name as a parameter to the appropriate AUXLINK file.

Compiler Support

ADCs can be built using any memory model supported by DP4 program libraries: small,medium, and large for Microsoft C; and the same models plus Big model for Metaware C. Microsoft C should be used for OS/2 and Windows ADCs. Metaware C should be used for FlexOS. MS-DOS ADCs can be built with either compier. ADC modules should be compiled with the same compiler options as normal DP4 programs, except that for 16 bit Windows programs you will also want to specify the options normally used for Windows programs if you are calling Windows functions directly. A batch file called auxlink.bat is supplied for each memory model, and can be used to link ADCs for any platform by specifying the ADC number for the target platform i.e.:

Nr | Platform
2 | MS-DOS
5 | FlexOS
7 | OS/2
w | Windows

The target ADC will default to the one appropriate to host operating system if not specified. When used with the /C option the small model version of AUXLINK can build an MS-DOS ADC program that can be converted to another operating system. It is vital to specify the correct platform in this case. Although theoretically possible It is highly inadvisable to attempt to build a Unix ADC using 16 bit compilation. Such ADCs will nor work beyond release 4.520 and will only work on platforms with the I286emul program.

You are strongly recommended to use or adapt the AUXLINK file. If for some reason this is inconvenient the following information may be helpful.

MS-DOS Development

Microsoft C

Memory models supported: Small, Medium, Large

Compiling

Compile all the source files (listed above):

Small Model:
cl -c -AS -FPa -Osi -Zelp filename.c
Medium Model:
cl -c -AM -FPa -Osi -Zelp filename.c
Large Model:
cl -c -AL -FPa -Osi -Gt16 -Zelp filename.c

Alternatively use the compiler batch file, MC.BAT, supplied by DP4 for each model.

Linking

Link using the Microsoft linker.

Small Model:
link /noe @aux2.lnk auxdebug, auxdebug, con, mcowns slibca
Medium Model:
link /se:256 /st:4096 /noe @aux2.lnk auxdebug, auxdebug, con, mcownm mlibca
Large Model:
link /se:256 /st:6144 /noe @aux2.lnk auxdebug, auxdebug, con, mcownl llibca

Alternatively use the AUXLINK.BAT file supplied by DP4 for use with the relevant Microsoft compiler memory model.

Metaware High C

Memory models supported: Small, Medium, Big, Large

Compiling

Compile all the modules using the hc compiler.

WARNING: It is vital that the HC.PRO files supplied by DP4 for your chosen Metaware compiler model(s) are referenced appropriately during Metaware compilation

Linking

Link using the Microsoft linker:

Small Model:
link /se:256 hcroot2 @aux2.lnk auxdebug, auxdebug, con, hcowns hcse
Medium Model:
link /se:256 /noe @aux2.lnk auxdebug, auxdebug, con, hcownm hcme
Big Model:
link /noe /se:256 init dosalloc @aux2.lnk auxdebug, auxdebug, con, hcownb hcbe

( dosalloc.obj will be supplied with your compiler and a modified init.obj supplied by DP4)

Large Model:
link /noe /se:256 init dosalloc @aux2.lnk auxdebug, auxdebug, con, hcownl hcle

( dosalloc.obj will be supplied with your compiler and a modified init.obj supplied by DP4.)

Alternatively use the AUXLINK.BAT file supplied by DP4 for use with the relevant Metaware compiler memory model.

FlexOS Development

The following source files comprise the FlexOS ADC:

aux5.c
auxproc.c
auxcall.c
auxcall5.c
user-defined module (for example, your adapted auxuser.c )

All FlexOS ADCs may be built native under FlexOS, or converted from MS-DOS.

Metaware High C

Memory models supported: Small, Medium, Big, Large

Compiling

Compile all the relevant source files using hc .

WARNING: It is vital that the HC.PRO files supplied by DP4 for the various Metaware compiler models are referenced appropriately during Metaware compilation

Linking

Link native under FlexOS using the Metaware linker:

Small Model:

link y=hcroot5,aux5[in],auxdebug,hcowns.l86[s],hcse.l86[s]

Medium Model:
link y=aux5[in],auxdebug,hcownm.l86[s],hcme.l86[s]
Big Model:
link y=aux5[inp],auxdebug,hcownb.l86[s],hcbe.l86[s] [DA[g[DGROUP],c[DATA],c[FAR_DATA]],st[m[180]]]
Large Model:
link y=aux5[inp],auxdebug,hcownl.l86[s],hcle.l86[s] [DA[c[DATA],c[FAR_DATA]],st[m[180]]]

Alternatively use the AUXLINK.BAT file supplied by DP4 for use with the Metaware compiler and relevant memory model.

OS/2 Development

The following source files comprise the OS/2 ADC:

aux7.c
auxproc.c
auxcall.c
auxcall7.c

user-defined module (for example, your adapted auxuser.c )

Microsoft C

Memory models supported: Small, Medium, Large

Compiling

Compile all the source files (listed above):

Small Model:
cl -c -AS -FPa -Osi -Zelp filename.c
Medium Model:
cl -c -AM -FPa -Osi -Zelp filename.c
Large Model:
cl -c -AL -FPa -Osi -Gt16 -Zelp filename.c

Alternatively use the compiler batch file ( MC.BAT ) supplied by DP4 for the each Microsoft model.

Linking

Small Model:
link /noe mcos2if @aux7.lnk auxdebug, auxdebug, con, mcowns slibcap
Medium Model:
link /se:256 /st:4096 /noe mcos2if @aux7.lnk auxdebug, auxdebug, con, mcownm mlibcap
Large Model:
link /se:256 /st:6144 /noe mcos2if @aux7.lnk auxdebug, auxdebug, con, mcownl llibcap

Alternatively use the AUXLINK.BAT file supplied by DP4 for use with the relevant Microsoft compiler memory model.