OADL Implementation Notes

Type System

In a 32-bit implementation, OADL objects are represented internally by 32-bit numbers. To maximize the precision of Float and Int variables, the type information for them is compressed. Float variables have 31 bits of value, and the type is indicated by having the least- significant bit set to '0'. Int variables have 30 bits of value, and the type is indicated by having the least-significant two bits set to '01' (or 1). The remaining types have up to 24 bits of value, six bits of type ID (from the table below), and the least-significant two bits are set to '11' (or 3).

It is illegal for least-significant bits of '11' (or 3) to be combined with the type bits 2-7 of Float (0) or Int (1). Attempting to create an OADL value of this form will result in undefined behavior.

In a 64-bit implementation of OADL, the first few types are arranged differently:

Additionally, in a 64-bit implementation, IDs and variable addresses are 32 bits rather than 24. The illegal type combinations are the least-significant bits of '11' (or 3) combined with Double (0) or Long (1) type bits 2-7. The value of nil is 0xFFFF_FFFF_FFFF_FFFFul (64 consecutive '1' bits).

There are several other dynamic types supported by OADL and returned by the typeof() intrinsic procedure; these types are synthized from a combination of OadlVar base type and information stored in the dynamic object referred to by the base value:

These pictures document the actual bit layout of the various types in a 32-bit implementation:

These pictures document the actual bit layout of the various types in a 64-bit implementation:

Several types refer to "IDs" - these are virtual indexes into a dynamic memory pool. Implementations must provide automatic garbage collection for these IDs, and all implementations must be able to distinguish read-only from writeable Array and Object contents.

Pointer values are provided only for the convenience of interfacing to system library routines. They are completely opaque to OADL programs. Pointer values are not saved to disk.

Generic arrays are arrays of 32-bit or 64-bit tagged values, as described above. Packed arrays are stored more efficiently, since the type of each element is homogeneous and stored as part of the array type.

Arguably, for Null values, one could store no data in a packed array. However, this is seen as an optimization of no value.

The type system is implemented in C at oadlvar.h .

OADL machine internals

In some cases, it can be useful to understand the OADL stack machine. Although it is not complete documentation, the opcodes are implemented in C at opcode.h and builtin.h and in OADL at oadlmach.oah . Additionally, note that some of the more complicated parts of OADL's machine are implemented in an extended form of OADL (extended by some intrinsics that are not supported for general use). See intrinsic.oad for more information.

External procedure libraries

External procedure libraries are typically implemented as dynamically loaded system libraries. These libraries are kept in a system-dependent location (frequently the syslib directory of the OADL installation location).

The dynamic library should implement an entry point named OADL_libname (where libname is the same as that used in the using extern statement). That entry point then performs callbacks to the OADL runtime to register external procedures, as well as to capture pointers to important OADL implementation functions. These callbacks are defined in oadlsys.h .

This entry point must be implemented with the following prototype (where "xxx" is the name of the dynamic library):

int OADL_xxx(void *ctx, OADL_FindProc_fp findProc);

The first argument passed to that entrypoint is a "context" pointer, which must be passed back to the findProc entry point. The second argument passed to the entry point is the findProc routine. This routine is used to find all the other OADL implementation procedures which may be needed by the utility library. The typedef for this function pointer is:

typedef int (*OADL_FindProc_fp)(void *ctx, const char *name,
                                int version, OADLproc *rProc);

The name is the name of the procedure that is desired (see the table below for the complete list). The OADL_SYS_VER constant must be given for version. The proc is returned in the rProc pointer. The OADLproc typedef exists only to provide a "generic" function pointer; the returned pointer should be typecast according to the table below before it is used.

The most important procedure which should be queried and used is the OADL_AddExtern() procedure. The utility library must use this procedure to define the Extern procs which are implemented by the library. The following fragment queries and calls the OADL_AddExtern procedure to implement an Extern named foo::bar"

OADL_AddExtern_fp OADL_AddExtern;
int foo_bar(void *ctx, OadlVar *pRes, int nargs, const OadlVar *args);

if (!findProc(ctx, "OADL_AddExtern",
               OADL_SYS_VER, (OADLproc *) &OADL_AddExtern))
{
    return 0;
}
if (!OADL_AddExtern(ctx, "foo::bar", foo_bar)) {
    return 0;
}

All Extern procs are implemented with the same function prototype:

typedef int (*OADL_Extern_fp)(void *ctx, OadlVar *pRes,
                              int nargs, const OadlVar *args);

The function returns 1 for success, and 0 if any error occurred (see ThrowError(), below, for how to raise an exception which will be handled after the function returns). The result OadlVar should be returned in the pRes pointer. nargs contains the number of arguments which were passed to the Extern, and args is an array containing all the arguments. See the oadlvar.h header file for the definition of the OadlVar structure. The ctx argument must be passed back to any callbacks called by this entry point.

If the external library has termination code that must be executed when the library is unloaded (generally, at OADL exit time) that code must be provided in the OADL_xxx_term() function (where "xxx" is, again, the name of the dynamic library as provided by the OADL using extern statement):

void OADL_xxx_term(void *ctx)

The following is the complete list of functions that may be queried by findProc. Generally, these functions return 1 on success and 0 if any error occurred.

External library support functions

OADL_AddExtern

typedef int (*OADL_AddExtern_fp)(void *ctx, const char *name, OADL_Extern_fp proc);

Defines an Extern entry point (see above for more information)

OADL_FatalError

typedef int (*OADL_FatalError_fp)(void *ctx, int errNum)

Raises fatal exception errNum from oadlerr.h

OADL_ThrowError

typedef int (*OADL_ThrowError_fp)(void *ctx, int errNum);

Schedules the error errNum (from oadlerr.h ) for an exception handling event after the Extern function returns.

OADL_GetGcCounter

typedef int (*OADL_GetGcCounter_fp)(void *ctx);

OADL keeps a count of how many times GC occurs. If the GC counter changes after calls to any heap allocation routine (e.g. OADL_WriteCopy(), OADL_CreateArray(), OADL_CreateObject(), OADL_CreateDict(), etc.) then a GC has occurred, and therefore OADL_GetArrayInfo() and OADL_GetObjectInfo() should be called again on all cached OADL_ArrayInfo and OADL_ObjectInfo structures.

OADL_TypeOf

typedef int (*OADL_TypeOf_fp)(void *ctx, int *pRes, OadlVar v);

Returns, in pRes, the derived OADL_VT_* type of v - exactly the same as the typeof intrinsic.

OADL_Writeable

typedef int (*OADL_Writeable_fp)(void *ctx, int *pRes, OadlVar v);

Queries whether array v is writeable (for example, whether SetIndex may be called on it). The result of the query is returned in pRes.

Object manipulation functions

OADL_ObjectInfo

The OADL_ObjectInfo structure contains information about an OADL object. See OADL_GetObjectInfo() for more details.

/* Object flags */
#define OADL_OBJ_READONLY               0x00000001      // Object is read-only
#define OADL_OBJ_IS_CLASS               0x00000002      // Object is a class template
#define OADL_OBJ_ZOMBIE                 0x00000004      // Object is a zombie

/* Public index access flags - top 2 bits of publicList[2*i+1] */
#define OADL_PUB_ACC_RW                 0x00000000      // Public is read/write
#define OADL_PUB_ACC_PROTECTED          0x40000000      // Public is protected
#define OADL_PUB_ACC_READONLY           0x80000000      // Public is read-only
#define OADL_PUB_ACC_MASK               0xA0000000      // Mask for public access bits

/* Public index type flags - next 2 bits of publicList[2*i+1] */
#define OADL_PUB_TYPE_NONE              0x00000000      // Public is heterogenous
#define OADL_PUB_TYPE_TYPE              0x10000000      // Public has a simple (scalar) type
#define OADL_PUB_TYPE_ARRAY             0x20000000      // Public has an array type
#define OADL_PUB_TYPE_CLASS             0x30000000      // Public has a class type
#define OADL_PUB_TYPE_MASK              0x30000000      // Mask for pub type bits

/* Public offset bits */
#define OADL_PUB_OFFS_MASK              0x0FFFFFFF      // Offset part of public index

/* Macros to access the publicList */
#define OADL_PUB_KEY(list,i)            ((list)[2*(i)])
#define OADL_PUB_VAL(list,i)            ((list)[2*(i)+1])

typedef struct {
    uint32_t numProps;                  /* Total number of public/private properties */
    uint32_t numPublics;                /* Total number of public indexes */
    uint32_t numTypes;                  /* Number of extra type words in pulicList */
    uint32_t objFlags;                  /* From OADL_OBJ_*, above */
    OadlVar *propList;                  /* List of object properties */
    const uint32_t *publicList;         /* Sorted list of [public idx,prop idx] pairs */
    const uint32_t *typeList;           /* Sorted list of [public idx,type info] pairs */
} OADL_ObjectInfo;

OADL_GetObjectInfo

int (*OADL_GetObjectInfo_fp)(void *ctx, OADL_ObjectInfo *pObj, OadlVar vObj);

Fills in an OADL_ObjectInfo structure with information about vObj. Note that the pointers in an OADL_ObjectInfo may be invlidated by a GC cycle; see OADL_GetGcCounter()

OADL_FindPublicIndex

int OADL_FindPublicIndex(const uint32_t *list, int numList, uint32_t pubNum)

Finds the index of pubNum in the sorted list, which contains [pubNum, pubVal] pairs, or -1 if the pubNum was not found in the list. This is a binary search, so the list MUST be sorted.

Note that this is not a function pointer which is queried from OADL; rather, it is an inline function.

OADL_PublicType

int OADL_PublicType(OadlVar *pRes, const OADL_ObjectInfo *pObj, OadlVar pub)

Returns the OadlVar type of the public property pub of pObj. Note that, if no type was specified for pObj.pub, OADL_NIL is returned as the type.

Note also that this is not a function pointer which is queried from OADL; rather, it is an inline function.

OADL_GetProp

typedef int (*OADL_GetProp_fp)(void *ctx, OadlVar *pRes, OadlVar propNum);

Returns, in pRes, the object property of self at offset propNum. Note that the layout of OADL object properties is undefined, and can change from compile to compile. It is generally better to use the GetPublic/SetPublic routines, below. That said, an OADL program may use the address-of operator & to pass the offset of a non-public property to an Extern for use with the GetProp / SetProp interface.

OADL_SetProp

typedef int (*OADL_SetProp_fp)(void *ctx, OadlVar propNum, OadlVar val);

Sets property number propNum of self to val. See caveats in GetProc, above.

OADL_GetPublic

typedef int (*OADL_GetPublic_fp)(void *ctx, OadlVar *pRes, OadlVar obj, OadlVar pub);

Returns, in pRes, the value of public property pub of object obj

OADL_SetPublic

typedef int (*OADL_SetPublic_fp)(void *ctx, OadlVar obj, OadlVar pub, OadlVar v);

Sets the public property pub of object obj to v. Note that SetPublic returns a status code, not a simple success/failure indication:

#define OADL_PUBLIC_OK       0       // Public was set OK
#define OADL_PUBLIC_ERROR    1       // Can't set a read-only constant - caller should throw OADL_E_ACCESS
#define OADL_PUBLIC_OVERLOAD 2       // Needs a call to := operator - responsibility of caller
#define OADL_PUBLIC_DO_TC    3       // Needs intrinsic type conversion - responsibility of caller

OADL_CreateObject

typedef int (*OADL_CreateObject_fp)(void *ctx, OadlVar *pRes, OadlVar vClass);

Creates a new instance of class vClass. Note that neither the create() nor the complete() method of the class will be invoked; this must be arranged by the caller.

OADL_GetSelf

typedef int (*OADL_GetSelf_fp)(void *ctx, OadlVar *pSelf);

Returns self in pSelf

OADL_NewPublic

typedef int (*OADL_NewPublic_fp)(void *ctx, OadlVar *pVar, OadlVar vName);

Creates a new public with name vName. Returns result in *pVar.

OADL_NewClass

typedef int (*OADL_NewClass_fp)(void *ctx, OadlVar *pVar, OadlVar vName, OadlVar vPublics);

Creates a new class with name vName (which can be nil) and initial key-value pairs vPublics (which must be a List). Returns result in *pVar.

Array manipulation functions

OADL_ArrayInfo

The OADL_ArrayInfo struct contains information about an OADL array. It is used both for information queries as well as for allocation for newly created arrays. See OADL_GetArrayInfo() and OADL_CreateArray() for more information. Here is the structure and its associated flags:

#define OADL_ARRAY_READONLY     0x01    /* Do not modify contents */
#define OADL_ARRAY_PERMANENT    0x02    /* Versus dyamic / GC memory */
#define OADL_ARRAY_USER_DATA    0x04    /* Points to user data, not OADL data */
#define OADL_ARRAY_ITERATOR     0x08    /* Iterator - only if not expanded */

/* If the array is an iterator and the caller requested the parameters,
 * the ArrData is a pointer to an array of the following items (in the
 * type given by ArrType):
 */
#define OADL_ARR_ITER_PARAM_FIRST       0
#define OADL_ARR_ITER_PARAM_LAST        1
#define OADL_ARR_ITER_PARAM_INCR        2
#define OADL_ARR_ITER_PARAM_COUNT       3 // Total size of iterator param array

typedef struct {
    uint32_t ArrSize;                   /* Total size of array, in elements */
    uint8_t ArrType;                    /* One of OADL_VT_* */
    uint8_t ElemSize;                   /* Size of a single element, in bytes */
    uint8_t ArrRank;                    /* Rank - number of dimensions */
    uint32_t ArrFlags;                  /* Flags from OADL_ARRAY_*, above */
    uint32_t ArrShape[OADL_MAX_RANK];   /* Shape of the array */
    uint32_t ArrStrides[OADL_MAX_RANK]; /* Dimension strides, in elements */
    OadlVar ArrVar;                     /* Only if iter. expanded to heap */
    void *ArrData;                      /* Pointer to array data or params */
    void *ArrAlloc;                     /* Only if iter. expanded to malloc */
} OADL_ArrayInfo;

OADL_GetArrayInfo

typedef int (*OADL_GetArrayInfo_fp)(void *ctx, OADL_ArrayInfo *pArr, OadlVar a, int flags);

Returns info about a in OADL_ArrayInfo structure *pArr. The flags are composed from the following bits:

#define OADL_AIF_ITERATOR_EXPANSION     0x00000007 // Mask for AIF_ITERATOR*
#define OADL_AIF_ITERATOR_HEAP          0x00000001 // ...expand into heap
#define OADL_AIF_ITERATOR_MALLOC        0x00000002 // ...expand into malloc
#define OADL_AIF_ITERATOR_SCRATCH       0x00000003 // ...expand into scratch
#define OADL_AIF_ITERATOR_PARAMS        0x00000004 // ...raw params in ArrData

OADL_CreateArray

typedef int (*OADL_CreateArray_fp)(void *ctx, OadlVar *pRes, OADL_ArrayInfo *pArr);

Creates an array with parameters taken from OADL_ArrayInfo *pArr. Returns the resulting array in *pRes. This does allocate heap memory which could trigger a GC and invalidate any cached heap pointers.

Note that pArr->ArrData and pArr->ArrStrides are ignored unless OADL_ARRAY_USER_DATA is specified in pArr->ArrFlags.

pArr->ArrSize and pArr->ElemSize are always ignored, as they are calculated from the input in pArr->ArrType and pArr->ArrShape.

The pArr structure is updated with the actual array created.

OADL_ArrayIncr

int OADL_ArrayIncr(int offs, int rank, const OADL_U32 strides[], const OADL_U32 shape[], OADL_U32 index[])

Increment a multidimensional array index and return the new offset to the current element. Note that a scalar may be aliased as an array by using a rank of 0. Note also that the proper offset to the next element (based on the contents of index[]) is computed if an offs of 0 is consistently provided in a loop over all elements.

Inputs:

offs - Current offset (start at 0)
rank - Rank (dimensionality) of array
strides[] - Array of strides, from OADL_ArrayInfo
shape[] - Shape of array, from OADL_ArrayInfo
index[] - Current indices of array (start at {0})

Note that this is not a function pointer which is queried from OADL; rather, it is an inline function. Updates the index[] array and returns an updated offs

OADL_IdxOffset

int OADL_IdxOffset(int rank, const uint32_t index[], const uint32_t strides[]);

Calculates the offset to an array element given an index, rank and strides. It is presumed that the index has been range-checked!

Note that this is not a function pointer which is queried from OADL; rather, it is an inline function.

OADL_GetIndex

typedef int (*OADL_GetIndex_fp)(void *ctx, OadlVar *pRes, OadlVar v, int nIdx, int idx[]);

Returns, in pRes, the contents of v[idx₀, idx₁, ...].

OADL_SetIndex

typedef int (*OADL_SetIndex_fp)(void *ctx, OadlVar v, int nIdx, int idx[], OadlVar val);

Sets v[idx₀, idx₁, ...] to val. Note that OADL_SetIndex() can trigger a GC.

OADL_GetSubr

typedef int (*OADL_GetSubr_fp)(void *ctx, OadlVar *pRes, OadlVar v, int numSub, OadlVar *subrs);

The same as v.subr(subrs₀, subrs₁, ...) Creates a new subrange of the given var v. The subrange is returned in pRes. Note that this allocates memory and may result in a GC cycle which may invalidate any locally cached pointers to OADL heap objects. See OADL_GetGcCounter() for more information.

OADL_WriteCopy

typedef int (*OADL_WriteCopy_fp)(void *ctx, int *pRes, OadlVar v);

To write directly into an array's data (i.e., instead of calling OADL_SetIndex()), a call to OADL_WriteCopy() must be made. The integer *pRes will be changed to 1 if a write copy was created and 0 otherwise. It is not necessary to call OADL_WriteCopy() on arrays created by the current Extern call. Note that OADL_WriteCopy() can trigger a GC.

Global variable manipulation functions

OADL_GetGlobal

typedef int (*OADL_GetGlobal_fp)(void *ctx, OadlVar *pRes, int idx);

Gets the global variable at index idx and puts it in pRes.

OADL_SetGlobal

typedef int (*OADL_SetGlobal_fp)(void *ctx, int idx, OadlVar val);

Sets the global variable at index idx to val.

OADL machine and procedure functions

OADL_AllocMach

typedef int (*OADL_AllocMach_fp)(void *ctx, void **pRes, int stackAlloc);

Creates an OADL Machine capable of calling procedures immediately. It allocates stackAlloc entries for the initial size of the execution stack.

OADL_FreeMach

typedef int (*OADL_FreeMach_fp)(void *ctx, void *ptr);
Frees a machine previously allocated by OADL_AllocMach

OADL_CallProc

typedef int (*OADL_CallProc_fp)(void *ctx, OadlVar *pRes, int typ, void *mach, OadlVar obj, OadlVar proc, int nargs, const OadlVar *args);

Makes an immediate or deferred call to the OADL procedure whose value is proc, or to the method whose value is obj.proc if obj is not nil (proc must be of type Public in that case). Returns 1 on success and 0 on failure.

The typ parameter defines how the call will be executed:

If typ is OADL_CALL_IMMEDIATE, an immediate call is performed. The mach parameter must be a machine allocated by OADL_AllocMach(). OADL_CallProc() will not return until the OADL procedure returns. The return value of the immediate call will be placed in the OadlVar pointed to by the pRes parameter. Most OADL procedures allocate heap objects. This may trigger a GC cycle which will invalidate any cached heap pointers. The calling routine must revalidate any pointers to heap objects after OADL_CallProc() returns.

If typ is OADL_CALL_DEFERRED, a deferred call is scheduled. A deferred call is a call placed on the current executing machine's call stack. OADL_CallProc() will return immediately after scheduling the call. Note that deferred calls do not return any value to the calling context. This allows multiple deferred calls to be scheduled simultaneously without overflowing the stack with unused results. Since the call is deferred, no heap allocations will occur at the time of the OADL_CallProc() call. If an Extern proc implementation defers one or more calls, it must return OADL_CALL_DEFERRED to OADL.

If typ is OADL_CALL_CHAINED, a chained call is scheduled. A chained call is also a deferred call; however, its result is returned to the calling context (which means that only one call may be chained). An Extern proc implementation must return OADL_CALL_CHAINED to OADL if it schedules a chained call.

OADL_AddProc

typedef int (*OADL_AddProc_fp)(void *ctx, OadlVar *pVar, const OADL_U8 instrs[], int numInst)

Adds a new intrinsic procedure based on the numInst OADL bytecodes found in the instrs array. The instrs / numInst parameters are typically obtained by using the cvtintr utility. The resulting intrinsic is returned in *pVar.

OADL_PushIntrinsic

typedef int (*OADL_PushIntrinsic_fp)(void *ctx)

Increments the intrinsic depth. Returns 1 on success, 0 if an error occurred. This function is useful for managing recursive intrinsic algorithms. If the intrinsic stack depth exceeds OADL_MAX_RANK an OADL_E_SHAPE exception will be thrown.

OADL_PopIntrinsic

typedef int (*OADL_PopIntrinsic_fp)(void *ctx)

Decrements the intrinsic depth. Returns 1 on success, 0 if an error occurred.

Dictionary manipulation functions

OADL_CreateDict

typedef int (*OADL_CreateDict_fp)(void *ctx, OadlVar *pRes, OadlVar *pList, int nList);

Creates a Dict holding exactly the key-value pairs found in the given list. nList must be a multiple of two.

OADL_EnumDict

typedef int (*OADL_EnumDict_fp)(void *ctx, OadlVar dict, void *arg, int (*func)(void *, OadlVar, OadlVar));

Enumerates the contents of a dictionary, passing the arg to the func with each key/value pair. Note that a nil key or value indicates an undefined entry. func should return 1 to terminate the enumeration early, and 0 otherwise.

Returns 1 if an invocation of func returned 1, and 0 otherwise (this facilitates the search for a specific set of keys).

func MUST NOT provoke a garbage collection cycle

Pointer manipulation functions

OADL_MakePointer

typedef int (*OADL_MakePointer_fp)(void *ctx, OadlVar *pRes, void *ptr);

Creates a new Pointer value from the given native ptr

OADL_DelPointer

typedef int (*OADL_DelPointer_fp)(void *ctx, OadlVar v);

Deletes the native Pointer variable v

OADL_FindPointer

typedef int (*OADL_FindPointer_fp)(void *ctx, void **pRes, OadlVar v);

Returns the native pointer *pRes associated with the Pointer v

Named object query functions

OADL_GetPubName

typedef int (*OADL_GetPubName_fp)(void *ctx, char **pRes, OadlVar v);

Returns the name of the Public v. Same as pubname(v)

OADL_GetExtName

typedef int (*OADL_GetExtName_fp)(void *ctx, char **pRes, OadlVar v);

Returns the name of the Extern v.

OADL_GetObjName

typedef int (*OADL_GetObjName_fp)(void *ctx, char **pRes, OadlVar v);

Returns the name of the object v. Same as objname(v)

OADL_FindPublic

typedef int (*OADL_FindPublic_fp)(void *ctx, OadlVar *pRes, const char *name);

Find the Public index named name and return its value in pRes. Note that name may be a UTF-8 encoded wide string.

OADL_FindExtern

typedef int (*OADL_FindExtern_fp)(void *ctx, OadlVar *pRes, const char *name);

Find the Extern named name and return its value in pRes. Note that name may be a UTF-8 encoded wide string.

OADL_FindGlobal

typedef int (*OADL_FindGlobal_fp)(void *ctx, OadlVar *pRes, const char *name);

Looks up name in the OADL global symbol talbe and returns the result in pRes

OADL_FindObject

typedef int (*OADL_FindObject_fp)(void *ctx, OadlVar *pRes, const char *name);

Find the object named name and return its value in pRes. Note that name may be a UTF-8 encoded wide string.

OADL_FindClass

typedef int (*OADL_FindClass)(void *ctx, OadlVar *pRes, const char *name);

Find the class named name and return its value in pRes. Note that name may be a UTF-8 encoded wide string.

OADL I/O functions

OADL_fgets

typedef int (*OADL_fgets_fp)(void *ctx, OadlVar *pRes, OadlVar f);

Reads a string from the OADL file f returning the resulting OADL String or WideString object in *pRes

OADL_fgetc

typedef int (*OADL_fgetc_fp)(void *ctx, int *pRes, OadlVar f);

Reads a UTF8 character from the OADL file f and returns the character as *pRes (note - returns the int, not the OadlVar)

OADL_fputc

typedef int (*OADL_fputc_fp)(void *ctx, OadlVar f, int c);

Puts the (possibly wide) character c into OADL file f

OADL_fputs

typedef int (*OADL_fputs_fp)(void *ctx, OadlVar f, const char *s, int len);

Puts the ASCII character string s into OADL file f

OADL_wfputs

typedef int (*OADL_wfputs_fp)(void *ctx, OadlVar f, const OADL_U32 *s, int len);

Puts the 32-bit Unicode character string s into OADL file f

OADL_ungetc

typedef int (*OADL_ungetc_fp)(void *ctx, OadlVar f, int c);

Pushes the character c back onto OADL file f to be returned by subsequent UTF8 reads

OADL_fopen

typedef int (*OADL_fopen_fp)(void *ctx, OadlVar *pRes, const char *name, const char *access);

Opens a UTF8 stream and returns the File result in *pRes

OADL_fclose

typedef int (*OADL_fclose_fp)(void *ctx, OadlVar f);

Close the OADL file f

OADL_fflush

OADL_fseek

OADL_ftell

OADL_fread

OADL_fwrite

OADL_feof

OADL_ferror

OADL_clearerr

typedef int (*OADL_fflush_fp)(void *ctx, OadlVar f);
typedef OADL_64 (*OADL_fseek_fp)(void *ctx, OadlVar f, OADL_64 offs, int seekType);
typedef OADL_64 (*OADL_ftell_fp)(void *ctx, OadlVar f);
typedef int (*OADL_fread_fp)(void *ctx, void *p, int n, int s, OadlVar f);
typedef int (*OADL_fwrite_fp)(void *ctx, void *p, int n, int s, OadlVar f);
typedef int (*OADL_feof_fp)(void *ctx, OadlVar f);
typedef int (*OADL_ferror_fp)(void *ctx, OadlVar f);
typedef int (*OADL_clearerr_fp)(void *ctx, OadlVar f);

Other OADL equivalents to stdio routines. A program which uses OADL_fopen must use these instead of real stdio routines.

OADL say() and read() replacement

typedef int (*OADL_GetChar_fp)(void *ctx, void *fp);

typedef void (*OADL_SayChar_fp)(void *ctx, int wch);

typedef void (*OADL_SayStr_fp)(void *ctx, char *s, int n);

typedef void (*OADL_SayLStr_fp)(void *ctx, OADL_U32 *s, int n);

These function pointers types are used by OADL built-in input and output routines, and can be updated via OADL_IoProcs(), below. Note that these functions are NOT queryable via the normal OADL_FindProc mechanism; use the OADL_GetIoProcs() function to get them instead.

OADL_IoProcs

typedef int (*OADL_IoProcs_fp)(void *ctx, OADL_GetChar_fp getChar, OADL_SayChar_fp sayChar, OADL_SayStr_fp sayStr, OADL_SayLStr_fp sayLStr);

If the extern library wants to hook/intercept say() and read(), it should call this routine with the four entry points specified.

OADL_GetIoProcs

typedef int (*OADL_GetIoProcs_fp)(void *ctx, OADL_GetChar_fp *pGetChar, OADL_SayChar_fp *pSayChar, OADL_SayStr_fp *pSayStr, OADL_SayLStr_fp *pSayLStr);

Retrieve the current IO procs

OADL UTF-8 support functions

OADL_ucCharType

typedef int (*OADL_ucCharType_fp)(int wch)

Determines the Unicode Character Type from a given WideChar value between 0 and 0x11000, inclusive. Returns -1 on error. See unicode.h for possible character types.

OADL_ucToUpper(int wch)

OADL_ucToLower(int wch)

typedef int (*OADL_ucToUpper_fp)(int wch)

typedef int (*OADL_ucToLower_fp)(int wch)

Converts a WideChar value betwen 0 and 0x110000, inclusive. Returns the original character if no single-character conversion if found.

OADL_utf8_bytelen

typedef int (*OADL_utf8_bytelen_fp)(void *ctx, int *pRes, OADL_U32 *src, int n);

Returns the number of bytes that would be required to encode the 32-bit string src

OADL_utf8_wcharlen

typedef int (*OADL_utf8_wcharlen_fp)(uint8_t *src, int n);

Returns the number of WCHARS that would be required to decode the UTF-8 encoded string src with length n.

OADL_utf8_encode

typedef uint8_t *(*OADL_utf8_encode_fp)(uint8_t *dst, uint32_t *src, int n);

Encodes the 32-bit wide string src with length n into the UTF-8 string dst which must have enough space allocated to hold the complete string. Returns a pointer to the byte after the last byte placed.

OADL_utf8_decode

typedef uint32_t *(*OADL_utf8_decode_fp)(uint32_t *dst, uint8_t *src, int n);

Decodes a UTF-8 encoded string src with length n into the 32-bit wide string dst which must have enough space allocated to hold the complete string. Returns a pointer to the uint32 after the last uint32 placed.

PCRE regular expression support functions

OADL_RegComp

typedef int (*OADL_RegComp_fp)(void *ctx, void **pRE, char *pattStr, int options);

Compiles the PCRE pattern pattStr into *pRE for subsequent use with OADL_RegEx(). Options are standard PCRE options, found in oadlpcre.h .

OADL_RegEx

typedef int (*OADL_RegEx_fp)(void *ctx, int *pRes, void *re, char *str, int first, int len, int options, int patOffs[], int patOffSize);

Executes the PCRE reg exp. machine with compiled expression RE. The matches are returned in patOffs. patOffs will be an array of begin,end+1 pairs indicating the captured substrings, with patOffs[0] and patOffs[1] indicating the entire match. first and len indicate the substring which is to be matched. Options are PCRE options, from the list found in oadlpcre.h . The number of pairs is returned in *pRes.

OADL_RegFree

typedef int (*OADL_RegFree_fp)(void *ctx, void *re);

Frees a PCRE regular expression previously created by OADL_RegComp()

Memory allocation and freeing

OADL_malloc

OADL_realloc

OADL_free

typedef int (*OADL_malloc_fp)(void *ctx, void **pRes, int nBytes);

typedef int (*OADL_realloc_fp)(void *ctx, void **pRes, void *ptr, int nBytes);

typedef int (*OADL_free_fp)(void *ctx, void *ptr);

Just like libc malloc/realloc/free but with OADL debugging hooks. Also more likely to work despite any interesting details about how malloc pools are implemented with shared libraries.

OADL_AllocScratch

typedef int (*OADL_AllocScratch_fp)(void *ctx, void **pRes, int nBytes);

Allocates scratch space that is only valid until the next OADL_AllocScratch. There is no need to free this memory; it is recycled by clients of OADL_AllocScratch

Simple External Library Example

The following is a very simple external library implementation. It can be compiled and used with the using extern example from the chapter on External procedures. The example should be compiled as follows:

Linux .so

clang -x c++ --std=c++11 -fPIC -c libfoo.c -I../../INCLUDE

clang++ --std=c++11 -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1 libfoo.o

Cygwin .dll

gcc -x c++ --std=c++11 -c libfoo.c -I../../INCLUDE

g++ -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1 libfoo.o -lc

Windows .dll

cl /c libfoo.c /I../../INCLUDE /DWIN32

link /dll libfoo.obj /out:libfoo.dll

MacOS .dylib

clang -x c++ --std=c++11 -c libfoo.c -I../../INCLUDE

clang++ --std=c++11 -dynamiclib -o libfoo.dylib libfoo.o -lc++

/*
 * Copyright (c) 2025 Ross Cunniff
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 */

#ifdef WIN32
#include <windows.h>
#endif

#include <stdio.h>

#include "oadlvar.h"
#include "oadlsys.h"

static OADL_GetArrayInfo_fp OADL_GetArrayInfo;
static OADL_AddExtern_fp OADL_AddExtern;
static OADL_TypeOf_fp OADL_TypeOf;

static int foo_bar(void *ctx, OadlVar *pRes, int nargs, const OadlVar *args)
{
    printf("foo::bar!\n");
    *pRes = OADL_NIL;
    return 1;
}

/* tables of extern functions defined */
static struct {
    const char *name;
    OADL_Extern_fp func;
} funcList[] = {
    { "foo::bar",       foo_bar },
    { 0, 0 }
};

/* table of OADL exports needed */
static struct {
    const char *name;
    OADLproc *func;
} oadlCB[] = {
    { "OADL_AddExtern",    (OADLproc *) &OADL_AddExtern },
    { "OADL_TypeOf",       (OADLproc *) &OADL_TypeOf },
    { "OADL_GetArrayInfo", (OADLproc *) &OADL_GetArrayInfo },
    { 0, 0 }
};

#ifdef __cplusplus
extern "C" {
#endif

#ifdef WIN32
__declspec(dllexport)
#endif
int OADL_libfoo(void *ctx, OADL_FindProc_fp findProc)
{
    int i;

    /* Get the OADL exports needed to run */
    for (i = 0; oadlCB[i].name; i++) {
        if (!findProc(ctx, oadlCB[i].name, OADL_SYS_VER, oadlCB[i].func)) {
            return 0;
        }
    }

    /* Define the extern routines */
    for (i = 0; funcList[i].name; i++) {
        if (!OADL_AddExtern(ctx, funcList[i].name, funcList[i].func)) {
            return 0;
        }
    }

    return 1;
}

#ifdef __cplusplus
}
#endif

#ifdef WIN32
// DLL entry function (called on load, unload, ...)
BOOL APIENTRY DllMain(HANDLE hModule, DWORD dwReason, LPVOID lpReserved)
{
    return TRUE;
}
#endif

Token state machine

Although much of OADL lexical analysis can be implemented via simple switch statements and single-character lookahead, distinguishing identifiers (names), integer constants, hexadecimal constants, and floating point constants from each other is more complicated. The lex program given in this document does not implement it fully correctly - it does not handle embedded underscores in integer/hex/float constants. The following picture shows a complete deterministic finite-state automaton (DFA) which will recognize these tokens. States with double circles around them are legitimate final states (and each is labelled with the type of the token recognized at that state). The following are the possible transtion labels:

Upper-case letter or range

The specified ASCII characters are recognized. Note that lower-case letters are also recognized by these transitions.

alph

Any alphabetic character is recognized, including non-ASCII UTF-8 characters with the Unicode alphabetic attribute.

alph(~L)

Any alphabetic character other than the letter L is recognized.

hex

Any hexadecimal digit (the digits 0 through 9 or the ASCII characters A through F) is recognized.

Digits or digit ranges

The specified digits are recognized.

The character "u"

The ASCII underline character '_' is recognized.

Punctuation (for example, ".", "$", "+", "-")

The given ASCII punctuation is recognized.

The state numbers correlate exactly to the DFA in OADL's reference implementation. In non-terminal states, any other character than those given in the outbound transitions is a lexical error condition. The terminal states are labelled with the token that they have recognized:

nam

An OADL identifier token

int

An OADL integer constant token

isz

An OADL sized integer constant token

bin

An OADL binary integer constant token

uns

An OADL unsigned integer constant token

hex

An OADL hexadecimal integer constant token

flt

An OADL floating point constant token

fsz

An OADL sized floating point constant token

hfp

An OADL hexadecimal floating point constant token

hfs

An OADL sized hexadecimal floating point constant token

lch

An OADL wide character token

lst

An OADL wide string token

dot

The OADL dot token "."

ell

The OADL ellipsis token "..."

The token state machine is directly implemented in C by tokmach.h and tokmach.cpp . The state numbers correspond directly to the state diagram.

Back to Glossary

Continue to Lex / Yacc Grammar

Return to Introduction