Methods and Programs used to Parse Data from Fixed Format Bulletins to the ISC Database

Contents

Introduction

As part of the development of the ISC data management system it is intended that all data will be maintained on the ISC's Oracle database. The database development process began with the development of a suitable schema, ISC1.0, to contain all required and useful data. The second stage which is described in this document was to parse all of the previously published data from the ISC bulletin into the database. This data was available from 1904 to the present day, though phase data was available only from the start of 1964. These data are in two file formats: historical files ( prior to 1964 ) and Fixed Format Bulletin ( FFB ) files.

The FFB and historical files are the only readily accessible source of electronic data covering the entire period over which the ISC has analysed and published data. There are some limitations in the content of this data, principally the absence of any un-associated phase data and phase data only associated with the primary hypocentre of any event. However this data is available and a project to extract the un-associated phase data from the ISC's tape archive exists. In the future as new data is loaded in the database rather than SIP all of the phase associations reported to the ISC will be preserved.

Loading Procedure

The loading of the database consists of 6 main stages:

  1. Parsing of data from the FFB files to a form suitable for SQL*Loader.
  2. Generation of a new set of temporary database tables.
  3. Load data into the temporary tables.
  4. Modify data in temporary tables.
  5. Data integrity checks.
  6. Insert temporary data into main database.

The first 5 stages are run from the script run-allx in the directory /export/home/oracle/ISCDB/LOAD. The final stage is carried out by the script insertx, in the same directory. As the scripts stand they require only 2 command line parameters, the year ( 4 digits ) and month ( 2 digits ) and will then run to completion.

ksh> run-allx 1997 06

ksh> insertx 1997 06

However they rely on the FFB being in the default directory of /export/iscftp/Bulletin/year. A few simple checks are made at the start of any run to ensure that the year and month in question have not been previously loaded.

A log file is kept of the output from each stage of the process and must be checked for error messages prior to the data being inserted into the main tables. A description of the important points of the log file will be given for each stage of the process. The log file for a particular job will be written to /export/home/oracle/ISCDB/LOAD/LOG/YEAR/YEARMONTH. After a month has been inserted the log file is made read-only to prevent accidental modifications.

For access to the main tables the ISC data administrator username and password are required to run any program that modifies the database tables. Although the code could be run for individual users to load their own tables. The scripts use environment variables to hold the username ISCUSER and password ISCPASS.

Notes:

All of the scripts were written for a Korn shell environment.

Where appropriate, files containing data or code have been named after their associated database table. With the exception of the pub_comments table which was originally called published where files are named after the original table.

Reading non-default FFB files

On occasion, an FFB other than those for published data will need to be read and loaded into a set of tables. To do this a number of parameters in the scripts need to be changed manually and insertx should never be run. The two parameters that need to be changed are the environment variables FILE ( location of the FFB ) and OUTFILE ( log file ).

The scripts in which this needs to be done are:

/export/home/oracle/ISCDB/LOAD/readx
/export/home/oracle/ISCDB/FFB/gethypx
/export/home/oracle/ISCDB/FFB/getphx
/export/home/oracle/ISCDB/LOAD/PROGS/bothx

Examples of the alternative settings for the variables will be found in the files.

Stage 1 Parsing of data from the FFB Files

Stage 1 Program Files

The parsing scripts were developed from the existing FFB search script developed for use on the ISC WEB site and use the nawk programming language. Nawk is a language designed to process sequential text files and as such was well suited to the fixed format text in the FFB files.

The following files are required to run stage 1 of the loading process and are called from run-allx:

File                Location                        Purpose
readx               ~/oracle/ISCDB/LOAD             Script to call main and set the
                                                    command line parameters.
main                ~/oracle/ISCDB/LOAD/PROGS       nawk programme to control the 
                                                    parsing of the FFB files.
read.nawk           ~/oracle/ISCDB/LOAD/PROGS       nawk subroutines for reading the
                                                    various parts of the FFB file.
insert.nawk         ~/oracle/ISCDB/LOAD/PROGS       nawk subroutines for writing the 
                                                    values extracted from the FFB 
                                                    into files for SQL*Loader.
phase_numbers       ~oracle/ISCDB/LOAD/DATA         Data file with ISC Phase ids.

The output from these programs is in 9 files, shown below, in the directory /export/home/oracle/ISCDB/LOAD/DAT. These do not need to be examined unless there is a problem with SQL*Loader and data has to be modified manually rather than through improvements to the nawk scripts.

published.dat                
association.dat
stamag.dat
amplitude.dat
event.dat
netmag.dat
hypoc_err.dat
phase.dat
hypocenter.dat

Stage 1 Description of Fixed Format Bulletin Files

The FFB files are sequentially written text files containing all of the data that was published in the printed bulletin. Data within the FFB is contained within 11 different records or formats:

Format 0        File Header
Format 1        Epicentre                                
Format 2        Epicentre Continuation
Format 3        Epicentre Comment
Format 4        Epicentre Comment Continuation
Format 5        Initial Phase
Format 6        Later Phase
Format 7        Phase Comment
Format 15       Initial Phase ( 5 character station code )
Format 90       Agencies
Format 91       Stations

A detailed description of the FFB can be found online at http://www.isc.ac.uk/doc/misc/fform.html or on the ISC CDs.

No information from the Header record is used for loading the database.

For the FFB the order in which the data is written is:

Header
Agencies                All agency numbers and information currently available
Stations                Only station information relevant to data within the FFB

Then for each event:

Secondary hypocentres   Formats 1 or 3
Primary hypocentre      Format 1
Phase data              Formats 5, 6, 7 and 15.

Each format 1 line may also have associated format 2, 3 records, and format 3 records may have continuation lines.

The data from the FFB is extracted and inserted into the database tables described briefly below. A complete description of the ISC schema is given in the document Database Schema, available online at http://www.isc.ac.uk/doc/database/schema.html.

Table             Contents
event             prime hypid
hypocenter        information relating to the details of all epicentres
hypoc_err         information relating to the accuracy of epicentre estimates
netmag            magnitude data from all associated phase data
association       data detailing the link between an epicentre and phase data
phase             station or phase data (associated and un-associated )
stamag            magnitudes from each phase data
amplitude         amplitude data from each phase data
pub_comments      comments, Format 3, 4 and 7 data
remark            Non-FFB data detailing any necessary about the data loaded
Appendix 1 details the relationship between the FFB variables and their destination attribute in the database tables. Importantly any modifications of the data or algorithms used to modify the data in the FFB are shown here.

Stage 1 Increment of ID Numbers ( External Keys )

All data within the database in identified by a unique ID number, this number provides the basic association or external key between all of the tables.

Event identification, evid, is incremented on each occurrence of a new event, indicated where the last primary epicentre flag is A', Format 1 position 26. This method is required because the primary epicentre is the last entry in an event and the FFB file is read sequentially.

Hypocentre IDs are incremented when a Format 1 line is read.

N.B. Though comments ( Format 3 ) can be included as an epicentre estimate in the FFB they contain no data beyond a time and a comment and as such were not entered into the hypocenter table. They are preserved in the pub_comments table as a unique comment associated with the appropriate event. prime_hyp is taken as the appropriate hypid when the prime epicentre flag is read.

Individual phases are identified by phid, which is incremented at each occurrence of a Format 5,6 or 15 line. A reading, a group of phases from the same station and event, is linked by rdid which is incremented at each Format 5 or 15 line.

magid, ampid and remid are incremented only when the appropriate data has been extracted from a phase line.

Stage 1 Comment lines

A large number of magnitudes are contained within the comments ( format 3 and 4 ). These magnitudes have been extracted and added to the netmag table. These magnitudes typically consist of only a magnitude, type and reporter. It is useful to note that the author for these netmag entries will be of the form 'NEIC;LYU' where the first part is the comment reporting agency and the second, if present the magnitude reporter.

Stage 1 Notes

For both hypocentres ( the prime must be within a specific month ) and phase dates where the date is outside of the month in question, the month is not incremented, but an extra day is created, e.g. 32nd January. This is not acceptable to general use and these dates have been identified and months and years incremented accordingly.

It has been important to differentiate between null values and 0 when parsing the data.

Log output from Stage 1 is as shown below:

Thursday September 16 10:07:30 BST 1999           Date and time of start of run
601B                                              ISC is not prime hypocentre
608B
920B
927B
Number of Hypocentres = 1217                                
Number of events = 661 
Number of Readings = 12439
Number of phases = 19820
01 1964                                           Date of historical file header
Number of events = 661
Number of Hypocentres = 1217

The features which need to be examined are the number of events and hypocenters in the FFB, first section, and in the historical file, second section. This is a rough guide only to the accuracy of the extraction program, discrepancies though are noted and are described later in this report. The non-prime ISC hypocentres are only present for information purposes.

Stage 2 Creation of Database Tables

A number of parameters in the database tables have to be set up after the FFB data has been loaded and a number of checks have to made on the data. Due to the size of the database the only efficient way to carry out these procedures was to have a set of tables containing only data from the month in question. Also this means that the main database contains only data that has already been checked.

A tablespace, load, was created solely for these temporary tables and the tables were created using scripts generated using make from controlled source files. All the SQL scripts for generating the load tables are in directory /export/home/oracle/ISCDB/CREATE, and the source files are in the sub-directory DESCRIP. The make command needs to be run from the directory DESCRIP.

The script create_loadx simply calls each SQL script in turn for each table.

The log file need only be examined for Oracle error messages. If all has gone well a series of 9 messages of Table dropped and 9 of Table created should be present. Typically only a message saying that a table could not be created is important, as if a run has been aborted at some point the tables may have been dropped but not recreated.

Stage 3 Loading of Temporary Database Tables

The script loadx, in directory /export/home/oracle/ISCDB/LOAD, calls the Oracle utility SQL*Loader for each of the 9 tables being loaded. A control file is required to provide information about the parameters read from the data files and their destination in the database. SQL*Loader produces log files describing the loading process and files containing any bad data. All of the SQL*Loader files are in the directory /export/home/oracle/ISCDB/LOAD/CTL.

SQL*Loader uses control files to set parameters such as the input file and database destinations. These files are named after their associated table with file extension .ctl. A log file, extension .log and a file containing any flawed data, extension .bad, will be created when SQL*Loader is run.

In the main log file the number of bad data lines and the data in question will be included. If there is any bad data the run will have to be made again with either the bad data, or parsing program modified as appropriate.

Stage 4 Final Modifications of Temporary Database Tables

A number of modifications need to be made to the tables which were not possible using the nawk program, essentially because the reading of the FFB is done sequentially rather than randomly. The various SQL and PL/SQL scripts are run by the script fixupx, as before from the directory /export/home/oracle/ISCDB/LOAD. The various scripts called by fixupx are detailed below in the order in which they are executed. All scripts are in the directory /export/home/oracle/ISCDB/LOAD/FIXUP.

non_prime-isc.sql (PL/SQL)

This update sets the phase association to be to the ISC hypocentre where the ISC hypocentre is not the prime. The ISC hypocenter is typically non-prime when a fixed location is know for the event, for instance if the event is a nuclear test.

Note though that a number of problems occur where on old data the number of observations contributing to an ISC magnitude does not match the number of stamag entries. This is because the algorithm used to calculate a network magnitude when the data was published has now changed. For instance, data used previously by the ISC is not now due typically to changes in distance constraints. There is typically little or no documentation about when and what changes were made to the code.

The log file contains a small description of this update and should be followed by the entry "PL/SQL procedure successfully completed.". If not then the reason for the failure should be investigated though at no point in loading of the database has an error occurred at this stage.

set_id_num.sql (PL/SQL)

All primary ID numbers used in the database have to be unique. This script sets all IDs in the temporary tables to start from the next available ID number from the main tables. It is not necessary to use a unique number generator because only 1 user has write access to the main tables and as long as standard scripts are used a non-unique ID number cannot be generated.

As each table is updated a comment to that effect is printed in the log. Then, as with the previous update the log file should have the entry "PL/SQL procedure successfully completed.".

index_load (SQL)

Despite the small size of the temporary tables a number of indexes are required to ensure efficient running of a number of later scripts. Though some further indexes are created and dropped as required by later scripts.

Consideration of the value of indexes has cut the time taken by some updates by as much as a factor of 10 reducing the time taken to run the updates from around 4 hours to just under 1 hour. Little is printed in the log file beyond confirmation of the index creation and the time taken to do so.

hypocenter.sql (SQL)

This SQL script carries out a number of updates to the hypocenter table. In a number of these cases the SQL code relies upon the id numbers being sequential. This assumption should not normally be relied upon, though for the load tables it is valid.

Each stage of the update script is indicated in the log file, followed in each case by the number of rows updated, the time taken and confirmation of the commit.

1

The hypocenter magid is set from the first appropriate entry in netmag.

N.B. Typically this will be mag1 from format 1. However please note that if the magnitude has come from a comment line the author in netmag will be in the form 'code;code', where the second code will be the hypocenter author.

2

The hypocenter magnitude and magtype are set from netmag using the magid set in the previous stage.

3

For events where there is no ISC hypocentre and ndef is not set then ndef is set to nass.

4

srn and grn were set for non-ISC hypocentres from the ISC entries.

N.B. This update is no longer used.

5

The velocity model used is set to 'JB' for all ISC hypocentres.

6

ndp and depdp are set null where both are 0.

7

For ISC hypocenters nass is set by counting the number of associated phases.

hypoc_err.sql

1

sdobs is set to null where it has been input as 0.

2

sdepth is set to null where it has been input as 0.

3

stime is set to null where it has been input as 0.

N.B. This update is no longer used.

4

smajax, sminax and strike are set null where smajax ( i.e. either ) is 0.

For the hypoc_err updates each stage is not labelled but the log file has the number of rows updated, the time taken and confirmation of the commit.

stamag.sql

This is a fairly major update that attempts to correctly associate all stamag magnitudes with the netmag magnitude that is calculated from them. The magtype of the stamag magnitudes also has to be generated using the appropriate algorithms. The association is not maintained in the FFB and the algorithm used in the ISC code had to be extracted and used to help in the association.

The magtype is 'mb':

if        period is <= 3 seconds or null
and       period > 0 or log( amplitude/period) is not null
and       distance >= 21 and distance <= 100

N.B. Notes ( From Robin ) suggest that prior to the October 1975 Bulletin that distance was not used in the 'mb' algorithm. However two exceptions were encountered:

October 1975 the distance had to be <= 100 to correctly associate the stamag entries.

June 1974 the distance had to be > 21 to correctly associate the stamag entries.

The magtype is 'MS':

if         period >= 10 and period <= 60 seconds
and        distance > 20 and distance <= 160

N.B. Depth <= 60 occasionally had to be used as an extra constraint.

N.B. Parts of the ISC association code suggest a minimum distance of 5 degrees was used, however other parts of the code eliminated magnitudes with a distance below 20. The algorithm was essentially tuned to suit the data where particular cases showed potential errors in the code. For instance phase magnitudes where the distance = 20 also had to be eliminated to make the associations correct.

MS magnitudes are calculated using the method shown below.

Magnitude from reading:
MREADING = ( MZMLR + MZ + MN +ME ) / ( No. magnitude components )

Typically only a magnitude from the Z component would be present.

Network Magnitude:
MS = ( MREADING1 + ... MREADINGn) / ( No. Reading magnitudes )

The above methods were used to calculate a netowrk magnitude from the associated stamag values, this magnitude is compared to the network magnitude from the FFB as part of the data integrity checks detailed later in the report.

For mb magtypes the magid is set for the first 'mb' magnitude in each reading. This is why the uncertainty column in stamag is set to the reading id of its appropriate phase line. The stamag uncertainty is not in the FFB. It was also important to remove any 0 magnitudes as these make the SQL impossible to construct.

MS magtypes are easy to associate as they are all required to be associated with the netmag entry.

For both of these magtypes a test is made that the association is correct, though only by number of associations and a relatively close magnitude, see test section later.

The script is carried out in a number of stages, with the log file containing a comment on which stage is being run and the usual output. The individual stages are described briefly below.

1

Uncertainty and magid are set to NULL ( should be the case anyway ).

2

Type of uncertainty is changed to hold a larger number.

3

Uncertainty is set to the reading id ( rdid ) of the corresponding phase id ( phid ).

4

Magnitude is set NULL where it is 0.

5

mb magtype is assigned where appropriate.

6

MS magtype is assigned where appropriate.

N.B. A temporary table is created with the phid of the first mb magnitude in each reading.

7

MS magids are associated from the netmag table.

8

mb magids are associated from the netmag table

Stage 5 Data Integrity Checks ( Errors and Inconsistencies )

Though as much effort as possible has gone into the accurate and automatic extraction of data from the FFB files, some errors or inconsistencies were found in the data which could not be resolved. These were shown up by a number of tests carried out on the table after all updates had been made.

Stage 5 Network Magnitudes

A number of netmag magnitudes could not be correctly associated with their stamag values, typically there were not enough values but in 3 cases there were too many suitable values. As a fraction of the total ( 0.04% ) the number of errors is small. The most recent error is in 1988 data and most are in the 1960s data suggesting either a change in the algorithm used to calculate netmag or improvements in data management as computing resources improved. All of these errors are shown in Appendix 2 and are identified in the database by a remid of 1 in netmag.

The second problem with netmag associations occurred where the netmag calculated from the correctly associated stamag entries did not agree to within 0.1 of the value of magnitude 1 or 2 in the FFB. This is only searched for where the network magnitudes have been associated correctly. The differences are still very small and likely to be simply rounding errors as only 1 decimal place is recorded for the stamag entries. All the occurrences of this error have a remid of 8225 for the netmag entry and a listing of these is given in Appendix 3.

In all cases where the netmag and stamag values gave an error an examination of the data was made to ensure no errors were present in the program used for association. To facilitate this the log file contains details of each incorrect netmag. Despite detailed investigations a number of errors could not be solved by modifying the program or by modification of the associations as sufficient data was not available in the FFB to indicate what was wrong.

Single error

16-11-1996 07:48:23, hypid 2157458, had an ML magnitude of 27. This is clearly wrong and appears to be a duplicate of the depth value. The database value has been set null with a remark, remid 8238 made to record the problem. The ML 27 value was in the published bulletin.

Stage 5 Phase entries

A later phase, phid 28104266, with the NDI event of 30-09-1965 08:48:27 has been printed and included in the FFB despite having a 22 hour time delay from the event. It is most likely that the day and hour time are incorrect and the time has been changed to correct it. The original time has been included in the remark for this phase entry, remid 10436.

Stage 5 Phase IDs

A number of operator phase IDs in the FFB are essentially rubbish, but these were generally allowed to go into the database as checking every possible phase type was not possible. The simple and oft repeated errors were eradicated automatically. A typical error was where a phase had a number at the end of the string e.g. 'P 4', the number was removed from the phase before entering it into the database.

However some, shown below had to be removed by hand as the speech marks in the string affect the function of SQL*Loader. All had the operator phase ( phase table ) set to null in the database.

Date                   phid      Operator id   Reporting Station   Database id
24-07-1976 06:22:21    22825445  2&"&          MAW                 Null
16-05-1976 22:21:11    22927145  )&"&          MAW                 Null
28-12-1975 15:37:41    23197144  2>"&          TOO                 Null
14-01-1979 12:40:19    21306116  S"P           BRS                 S'P'

Fake P

From 1996 the fake P phases supplied from the NEIC were identified in the FFB with an operator ID number of 111 and typically an alphanumeric ID of "P". In the database these phases have been given an ID of "PFAKE" to ensure their correct identification. A detailed description of the nature of fake P phases can be seen at http://www.isc.ac.uk/doc/analysis/1998p01.

Only 7 occurances of non P phase fake Ps were found, these are shown below. But it has been confirmed that all should have an operator ID of P. The ID number in all cases was 111 and they are clearly fake P from the printed bulletin.

DAY                       MSEC ISC_PHAS OP_PHASE STA          PHID
------------------- ---------- -------- -------- ------ ----------
11-01-1996 04:05:00          0 NULL     pP       TUC      29572290
11-01-1996 04:05:20          0 NULL     pP       ALQ      29572318
11-01-1996 04:10:20          0 NULL     pP       OBN      29572346
11-01-1996 04:10:30          0 NULL     pP       BOSA     29572385
30-01-1996 22:18:50          0 NULL     PKIKP    CCM      29667534
30-01-1996 22:18:50          0 NULL     PKIKP    FVM      29667538
30-01-1996 22:18:50          0 NULL     PKIKP    SLM      29667541

Stage 5 Hypocentre Discrepancies

As part of the testing procedure the historical file is compared to the FFB, this has shown up a number of inconsistencies between the two files.

One frequent inconsistency occurred where there were two near identical hypocentre entries in the FFB and one in the historical file. The only difference is that one entry would have a comment attached. . As all of these hypocentres were published both have been entered into the database, though a remark has been added. In a certain respect these errors do not affect the database as there is no inconsitency with the data in the same way as network magnitudes. However there is some possibility that the duplicates may need to be deleted at some point. A list of these hypocentres is given in Appendix 5.

However the opposite to the previous problem has also been encountered: in February 1969 the UPP epicentre ( 21-02-1969 10:58:24, hypid 1790034 ) appears twice in both the historical file and the printed bulletin but has been removed from the FFB file. It is this form of undocumented editing of the files which make it difficult to guarantee an exact match between data in the database and that in the printed bulletin.

The following 4 discrepancies occurred where an event consisted of a single hypocentre with no phase data. All of the hypocentres/events were deleted from the historical files.

DATE                msec Agency hypid     remid
31-10-1979 23:29:34 100  ADE    1457395   8237    
31-01-1979 23:41:43   0  ADE    1487619   8237   
31-03-1973 23:50:24 800  CAN    1691306   8245    
30-09-1975 22:00:36 800  PAS    1597140   8241    
February 1975

The ISC epicentre for 09-02-1975 21:18:10 has not been included in the historical file and the LAO epicentre has not been included in the printed bulletin. This particular epicentre should not have had the ISC epicentre published because there are only 4 associated readings, ISC epicentres usually have at least 5 readings. The most likely possibility is that after the first editing pass some readings were deleted but not the epicentre. Hypid 1615048, remid 8242

February 1969

In both the printed bulletin and the historical file the event with an ISC hypocentre at 21-02-1969 08:41:06 had an epicentre from UPP, though clearly in the wrong event. But unlike most other similar errors the incorrect epicentre has been deleted from the FFB. It does highlight the problem of knowing exactly what is being extracted from the FFB and how it compares to the printed bulletin.

April 1964

A completely spurious event appears in the FFB where the hypocenters, including the ISC hypocentre, appear to have become associated with a few phases from the two surrounding events. The printed bulletin does not contain these data and to load it correctly into the database it was deleted temporarily from the FFB file. Part of the FFB is shown in Appendix 4.

March 1964

An event with an ISC epicentre at 31-04-1964 00:44:52 also has a USGS epicentre in the printed bulletin at a time of 00:44:53. The FFB has no entry for the USCGS epicentre and the historical file shows the epicentre with a time of 03:44:53. As the epicentre is not in the FFB it must have at some time been deleted, as there is no explanation as to why there will be no entry in the database. Though there is no explanation of how the printed bulletin could be correct and the FFB from which it was created wrong.

December 1967

A hypocentre in the FFB and printed bulletin has been removed from the historical files ( From NOU at 31-12-1967 04:52:14 ). There is another NOU hypocentre at the same time but slightly different location, so the most reasonable explanation is that this was considered a bad duplicate, though only after publication of the data. It has been left in the database with a comment. The questionable hypocentre has hypid 1818835 with remid 8248 added to indicate some doubt about it's reliability.

July 1974

The event evid 746383 on 27-07-1974 21:24:33 had been deleted from the historical file. The hypocentres are shown below.

DATE                 Agency hypid     remid
27-07-1974 21:24:33  BCIS   1648828   8244
27-07-1974 21:24:36  HFS2   1648829   8244
27-07-1974 21:24:39  NEIS   1648830   8244 
27-07-1974 21:24:39  ISC    1648833   8244 
27-07-1974 21:24:41  MOS    1648831   8244
27-07-1974 21:24:43  LAO    1648832   8244

Stage 5 Misc

September 1976

ndef set to 0 in the FFB, not supposed to be a valid entry for this variable, set to null in the database.

August 1976

ndef set to 0 in the FFB, not supposed to be a valid entry for this variable, set to null in the database.

July 1976

ndef set to 0 in the FFB, not supposed to be a valid entry for this variable, set to null in the database.

ISC hypocentre for 05-06-1994 18:36:00 where there is no phase data in the FFB, nsta and nass are null but ndef is 17. The ISC epicentre has not been included in the historical file. ISC hypid 374624.

Stage 6 Loading Main Database Tables

This stage of the process involves a simple SQL script which first counts the number of rows in each load table and then inserts the whole load table into the main database table. The count of the rows is a simple check to tally against the number of rows inserted by the SQL statement, both of which are printed in the main log file. At this stage any failure to insert the rows would be messy to undo. Typically failures would only occur if there was insufficient space in the table or failure of the Oracle Instance.

Once loaded the log file is made read-only to prevent accidental modification or removal.

During the early stages of loading the main database tables were analysed after each month was loaded but this had to be reduced to each full year as the tables got bigger and finally dropped. This was because the time taken to estimate the statistics was too large an overhead on the loading procedure.

Appendix 1 Details of relationship between FFB data and the Attributes of Database Tables

Header Record: Format 0

Not used.

Epicentre Record: Format 1

FFB value                                          Database tables and column
Record category                                                NA
Next record category                                           NA
Year                                                     hypocenter.day
Month                                                    hypocenter.day
Day                                                      hypocenter.day
Hours                                                    hypocenter.day
Minutes                                                  hypocenter.day
Seconds                                                  hypocenter.day
                                                         hypocenter.msec
Precision of time                                              NA
Agency number                                            hypocenter.author
                                                         hypoc_err.author
                                                         pub_comments.author
                                                         netmag.author
Prime estimate flag                                      event.prime_hyp
Latitude                                                 hypocenter.lat
Precision of latitude                                          NA
Longitude                                                hypocenter.lon
Precision of longitude                                         NA
Depth in kilometres                                      hypocenter.depth
Precision of depth                                             NA
Magnitude 1                                              hypocenter.magnitude
                                                         netmag.magnitude
Magnitude 1 (end of range)                                     NA
Magnitude 1 precision                                          NA
Type of magnitude 1                                      hypocenter.magtype
                                                         netmag.magtype
No. observations contributing to mag 1                   netmag.nsta
Standard error for magnitude 1                           netmag.uncertainty
Precision of standard error for mag 1                          NA
Geographical region number                               hypocenter.grn
Seismic region number                                    hypocenter.srn
Number of observations                                   hypocenter.nsta
Standard deviation of one observation                    hypoc_err.sdobs
Precision of standard deviation of one ob                      NA
Obs contributing to standard deviation                   hypocenter.ndef

Magnitude type is set as shown below for any magnitude type in format 1,2,3 and 4:

FFB      magtype in Database
S        MS
W        MW
L        mL

All other FFB magnitude types are made lower case and prefixed by 'm'.

Epicentre Continuation Record: Format 2

FFB value                                          Database tables and column
Record category                                                NA
Next record category                                           NA
Year                                                           NA
Month                                                          NA
Magnitude 2                                              netmag.magnitude
                                         ( If no mag 1 ) hypocenter.magnitude
Magnitude 2 (end of range)                                     NA
Magnitude 2 precision                                          NA
Type of magnitude 2                                      netmag.magtype
                                         ( If no mag 1 ) hypocenter.magtype
Number of obs. contributing to mag 2                     netmag.nsta
Standard error for magnitude 2                           netmag.uncertainty
Precision of s.e. for magnitude 2                              NA
Standard error for origin time                           hypoc_err.stime
Precision of s.e. for origin time                              NA
Standard error latitude (degrees)                           see below
Precision of s.e. for latitude                                 NA
Standard error longitude (degrees)                          see below
Precision of s.e. for longitude                                NA
Standard error for depth (km)                            hypoc_err.sdepth
Precision of s.e. for depth                                    NA
Explosion/effects flag:                                  hypocenter.evtype
Mantissa of explosion charge size in tons                      NA
Exponent of explosion charge size                              NA
Precision of explosion charge size                             NA
Number of pP-P observation                               hypocenter.ndp
Std. deviation of 1 pP-P observation                           NA
Depth from pP-P observations (km)                        hypocenter_depdp
Std. error of depth from pP-P observations(km)                 NA
Maximum intensity                                              NA
Intensity scale                                                NA
Distance in degrees to closest observation               hypocenter.mindist
Degrees to most distant observation                      hypocenter.maxdist

The values of standard error for latitude and longitude are used as shown below:

if standard error latitude > standard error longitude
        hypoc_err.smajax = 111.0 * standard error latitude
        hypoc_err.sminax = 111.0 * standard error longitude 
        hypoc_err.strike = 0.0;
else
        hypoc_err.smajax = 111.0 * standard error longitude
        hypoc_err.sminax = 111.0 * standard error latitude 
        hypoc_err.strike = 90.0;

The explosion effects flag ( evtype ) is set to follow IMS1.0 as shown below:

Flag     evtype
D        de
F        fe
R        kr
N        kn
H        kh

The default is uk for unknown.

Epicentre Comment Record: Format 3

FFB value                                          Database tables and column
Record category                                                  NA
Next record category                                             NA
Year                                                             NA
Month                                                            NA
Days                                                             NA
Hours                                                            NA
Minutes                                                          NA
Seconds                                                          NA
Agency number                                             pub_comments.author
Prime-estimate flag                                              NA
Comment                                                   pub_comments.pubcomment
                                                          netmag.magnitude
                                                          netmag.magtype

Epicentre Comment Continuation Record: Format 4

FFB value                                          Database tables and column
Record category                                                  NA
Next record category                                             NA
Year                                                             NA
Month                                                            NA
Serial number                                                    NA
Comment                                                  pub_comments.pubcomment
                                                         netmag.magnitude
                                                         netmag.magtype

Initial Phase Record: Format 5 and 15

FFB value                                          Database tables and column
Record category                                                  NA
Next record category                                             NA
Year                                                     phase.day
Month                                                    phase.day
Station code                                             phase.sta
                                                         association.sta
Station number                                                   NA
Network code                                                     NA
Source code                                              stamag.author
Format received code                                             NA
Local - Teleseismic flag                                         NA
Azimuth in degrees from epicentre to station             association.esaz
Distance in degrees                                      association.delta
Number of phases in this observation                             NA
Days                                                     phase.day
Hours                                                    phase.day
Minutes                                                  phase.day
Seconds                                                  phase.day
                                                         phase.msec
Precision of time                                        phase.deltime
Operator's ID (numerical)                                        NA
Operator's ID (characters)                               phase.phase
Operator's residual                                              NA
ISC ID (numerical)                         ( as string ) association.phase
ISC residual                                             association.timeres
Direction of first motion                                     see below
Instrument type                                               see below
Component                                                phase.chan
Sharpness                                                     see below
Signal-to-noise ratio                                            NA
Log A/T                                                  amplitude.logat
Precision of log A/T                                             NA
Mantissa of amplitude                                    amplitude.amp        
Exponent of amplitude                                            "
amplitude units                                          amplitude.delamp
Period in seconds                                        amplitude.per
Precision of period                                              NA
Magnitude                                                stamag.magnitude
column 5 of station code ( Format 15 only )

The direction of first motion is set as shown below:

phase.lp_fm = direction of first motion when instrument type is B, b, L or l and
phase.sp_fm = null
otherwise
phase.lp_fm = null and
phase.sp_fm = direction of first motion

Sharpness is used to set the following variables:

phase.emergent = sharpness if sharpness is e or E and
phase.impulsive = null 
otherwise
phase.emergent = null and
phase.impulsive = sharpness if sharpness is I or I

phase.chan is set to '?' if component is null, and will always prefixed by '??' according to IMS1.0 format for S/H/I Channel Codes.

phase.deltime = 10^(-tim_prec)

stamag.author is set to NEIS if source code is 'U' and JMA if 'J'.

amplitude.delamp = 10^(-ampl_prec)

Later Phase Record: Format 6

FFB value                                          Database tables and column
Record category                                                  NA
Next record category                                             NA
Year                                                     phase.day
Month                                                    phase.day
Phase count                                                      NA
Days                                                     phase.day
Hours                                                    phase.day
Minutes                                                  phase.day
Seconds                                                  phase.day
                                                         phase.msec
Precision of time                                        phase.deltime
Operator's ID (numerical)                                        NA
Operator's ID (characters)                               phase.phase
Operator's residual                                              NA
ISC ID (numerical)                         ( as string ) association.phase
ISC residual                                             association.timeres
Direction of first motion                                   as format 5
Instrument type                                             as format 5
Component                                                phase.chan
Sharpness                                                   as format 5
Signal-to-noise ratio                                            NA
Log A/T                                                  amplitude.logat
Precision of log A/T                                             NA
Mantissa of amplitude                                    amplitude.amp        
Exponent of amplitude                                             "
Precision of amplitude                                   amplitude.delamp
Period in seconds                                        amplitude.per
Precision of period                                              NA
Magnitude                                                stamag.magnitude

See format 5/15 comments above for any data manipulation.

Phase Comment Record: Format 7

FFB value                                          Database tables and column
Record category                                                  NA
Next record category                                             NA
Year                                                             NA
Month                                                            NA
Count of comment records for this station                        NA
Comment                                                   pub_comments.pubcomment

Agency Record: Format 90

Used only to convert agency numbers in format 1,2,3 and 4 records to codes.

Station Record: Format 91

Not used.

Appendix 2 remid 1 netmag errors

DATE                     HYPID      MAGID AUTHOR  MAGTYP  MAG  NSTA STAMAGS                        
------------------- ---------- ---------- ------  ------ ----- ---- -------
13-05-1988 15:34:59     963721     538618 ISC     mb      4.8    20      19
20-06-1987 00:53:04    1023335     564573 ISC     mb      6.1   145     144
20-06-1987 00:53:04    1023335     564574 ISC     MS      4.5     4       5
15-12-1975 01:08:46    1588615     855336 ISC     mb      5.2    18      27
14-06-1975 19:14:02    1603944     862428 ISC     mb      5.2    44      43
17-05-1973 16:00:02    1684562     900303 ISC     mb      5.1    22      23
06-11-1971 21:59:56    1724241     917877 ISC     mb      6.6    69      69
02-10-1969 22:06:01    1773113     938604 ISC     mb      6.4    65      43
26-04-1968 15:00:01    1811308     956595 ISC     mb      6.2    46      45
10-12-1967 19:30:01    1817728     959343 ISC     mb      4.8    14       6
18-10-1967 14:30:02    1821427     961074 ISC     mb      5.7    26      20
07-09-1967 13:45:03    1822407     961529 ISC     mb      5       8       5
27-09-1967 17:00:02    1823617     962082 ISC     mb      5.7    41      34
18-08-1967 20:12:30    1824740     962572 ISC     mb      4.6     4       1
31-08-1967 16:30:00    1825487     962954 ISC     mb      5       6       1
22-07-1967 10:59:03    1826732     963537 ISC     mb      4.9     9       5
27-07-1967 13:00:02    1827125     963718 ISC     mb      5       5       3
26-06-1967 16:00:02    1829144     964640 ISC     mb      5.1     6       2
29-06-1967 11:25:03    1829289     964716 ISC     mb      4.6     4       2
10-05-1967 13:40:03    1829979     965001 ISC     mb      4.9     7       2
20-05-1967 15:00:01    1830619     965298 ISC     mb      5.8    44      36
23-05-1967 14:00:02    1830825     965397 ISC     mb      5.7    28      23
26-05-1967 15:00:05    1831008     965475 ISC     mb      5.4    15      12
23-02-1967 18:34:00    1836348     968036 ISC     mb      4.4     3       1
23-02-1967 18:50:02    1836350     968037 ISC     mb      5.6    31      25
19-01-1967 16:45:02    1838051     968881 ISC     mb      5.3    20      17
20-01-1967 17:40:05    1838111     968910 ISC     mb      5.3     9       6
13-12-1966 21:00:02    1839298     969569 ISC     mb      4.6     5       3
20-12-1966 15:30:01    1839613     969726 ISC     mb      6.3    50      45
12-09-1966 15:29:59    1844201     971979 ISC     mb      4.6     3       1
07-07-1966 09:59:57    1847771     973610 ISC     mb      4.1     8       2
08-07-1966 08:29:56    1847808     973631 ISC     mb      3.9     9       1
08-07-1966 09:29:56    1847810     973633 ISC     mb      4.1     8       2
09-07-1966 08:29:56    1847847     973651 ISC     mb      4       9       2
09-07-1966 09:29:56    1847849     973653 ISC     mb      4.2     7       1
10-07-1966 08:29:56    1847895     973679 ISC     mb      4.2     8       1
10-07-1966 09:29:56    1847897     973681 ISC     mb      4      10       3
11-07-1966 08:29:56    1847947     973703 ISC     mb      4       9       2
12-07-1966 08:29:55    1848015     973736 ISC     mb      4.1     8       1
12-07-1966 09:29:56    1848017     973738 ISC     mb      4.1     8       1
13-07-1966 08:29:57    1848061     973761 ISC     mb      4.1     8       2
13-07-1966 09:29:56    1848068     973766 ISC     mb      4      10       3
14-07-1966 08:29:57    1848113     973793 ISC     mb      4       9       2
14-07-1966 09:29:56    1848115     973795 ISC     mb      4.1     8       1
15-07-1966 08:29:56    1848160     973821 ISC     mb      3.9     6       2
18-07-1966 08:59:56    1848297     973897 ISC     mb      4.1     9       2
18-07-1966 09:59:57    1848301     973901 ISC     mb      4      10       3
19-07-1966 09:29:57    1848355     973931 ISC     mb      4       8       2
20-07-1966 08:29:57    1848400     973957 ISC     mb      4.1     8       1
20-07-1966 09:29:57    1848404     973960 ISC     mb      4       8       2
21-07-1966 08:29:56    1848480     974004 ISC     mb      4       7       2
21-07-1966 09:29:56    1848487     974009 ISC     mb      4.1     7       1
22-07-1966 08:29:57    1848529     974029 ISC     mb      4.2     7       1
22-07-1966 09:29:57    1848531     974031 ISC     mb      4.2     7       2
23-07-1966 08:29:56    1848572     974055 ISC     mb      4.4     5       1
23-07-1966 09:29:56    1848576     974058 ISC     mb      4.2     6       2
24-07-1966 08:29:57    1848639     974096 ISC     mb      4.2     6       1
24-07-1966 09:29:57    1848647     974104 ISC     mb      4.1     7       2
25-07-1966 08:29:57    1848688     974122 ISC     mb      4.1     7       1
25-07-1966 09:29:57    1848694     974128 ISC     mb      4       8       2
26-07-1966 09:29:56    1848736     974149 ISC     mb      4       6       2
27-07-1966 08:29:56    1848779     974177 ISC     mb      4.1     8       3
27-07-1966 09:29:56    1848781     974179 ISC     mb      3.9     7       3
28-07-1966 08:29:57    1848850     974210 ISC     mb      4.1     8       2
28-07-1966 09:29:57    1848852     974212 ISC     mb      4       8       2
02-06-1966 15:30:01    1849133     974340 ISC     mb      5.6    34      29
03-06-1966 14:00:02    1849184     974364 ISC     mb      5.7    33      29
30-06-1966 22:15:02    1850989     975272 ISC     mb      6.1    39      36
05-05-1966 14:00:03    1851243     975411 ISC     mb      4.4     3       2
06-05-1966 15:00:03    1851331     975457 ISC     mb      5.4    22      18
12-05-1966 19:37:26    1851698     975652 ISC     mb      4.3     4       1
13-05-1966 13:30:02    1851737     975674 ISC     mb      5.6    25      20
19-05-1966 13:56:30    1852039     975829 ISC     mb      5.9    36      29
27-05-1966 20:00:03    1852568     976114 ISC     mb      5       7       3
06-04-1966 13:57:19    1853086     976375 ISC     mb      4.4     5       2
14-04-1966 14:13:46    1853538     976610 ISC     mb      5.4     9       5
25-04-1966 18:38:03    1854143     976938 ISC     mb      4.5     4       3
24-02-1966 15:55:10    1857425     978613 ISC     mb      5       5       2
18-01-1966 18:35:03    1858391     979102 ISC     mb      5.2     5       4
03-12-1965 15:13:04    1859299     979524 ISC     mb      5.6    22      18
16-12-1965 19:15:02    1859745     979766 ISC     mb      5.3    12      10
29-10-1965 21:00:03    1862331     981235 ISC     mb      5.8    50      49
01-09-1965 20:07:59    1862422     981285 ISC     mb      4.2     3       2
10-09-1965 17:12:02    1862732     981447 ISC     mb      5.1     8       5
23-07-1965 17:00:02    1865706     983146 ISC     mb      5.4    20      17
21-04-1965 22:00:03    1870186     985932 ISC     mb      5       3       2
05-12-1964 21:15:03    1876895     990640 ISC     mb      4.8     8       4
05-11-1964 15:00:01    1878413     991269 ISC     mb      4.8     4       3
02-10-1964 20:03:00    1879669     991921 ISC     mb      4       3       2
09-10-1964 14:00:02    1879962     992061 ISC     mb      4.8     4       2
22-10-1964 15:59:59    1880664     992426 ISC     mb      4.6     7       3
24-04-1964 20:10:01    1889936     997416 ISC     mb      5.2     4       2
29-04-1964 20:47:00    1890158     997547 ISC     mb      4.1     3       1
20-02-1964 15:30:02    1893060     999297 ISC     mb      4.8     6       4
16-01-1964 16:00:02    1894058     999790 ISC     mb      4.9    11       5
16-01-1964 18:08:41    1894065     999795 ISC     mb      4.9     4       1
23-01-1964 16:00:01    1894377     999957 ISC     mb      3.9     3       1

Appendix 3 remid 8225 netmag errors

DATE                     HYPID      MAGID AUTHOR MAGTYP  MAG NSTA 
------------------- ---------- ---------- ------ ------ ---- ----
04-04-1991 18:59:59     727506     425568 ISC        mb  5.7   75 
13-04-1985 03:00:04    1163684     630356 ISC        MS  6.8   34 
09-03-1976 14:00:01    1579504     851503 ISC        mb  5.8   78 
14-02-1976 11:30:01    1582844     852761 ISC        mb  5.8   56 
20-11-1975 15:00:01    1591915     856853 ISC        mb  5.9   76 
28-10-1975 14:30:02    1594691     858070 ISC        mb  6.2   66 
19-06-1975 13:00:00    1604612     862742 ISC        mb  5.9   74 
14-05-1975 14:00:01    1607223     863900 ISC        mb  5.9   64 
28-02-1975 15:15:00    1616570     867998 ISC        mb  5.6   58 
30-08-1974 14:59:59    1632152     874525 ISC        mb  5.6   52 
14-10-1969 07:00:06    1773781     938922 ISC        mb  6.3   62 
16-09-1969 14:30:02    1775907     939954 ISC        mb  6.1   49 
07-05-1969 13:45:01    1783430     943857 ISC        mb  5.5   43 

Appendix 4 Spurious Event of April 1964

The data in bold shows the incorrect data and the phase data can be seen to have come from the surrounding events.
 1 21964 4241430 600-1  4B 291000-1 1304000-1   0 0        99         99               99
 2 11964 4        99         99  900-1   600-2  1800-2    99      99
 1 11964 4241430 720-1171C 293000-1 1301000-1 310 0 480    -1B        99             60-1  14
 1 11964 42414301000 0 15D 297000-1 1299000-1 33099        99         99               99
 1 21964 42414301190-1  1A 292500-2 1299600-2 710 0 460    -1B    4   99 238 20  29 130-2  29
 2 51964 4        99         99  240-2   420-3   770-3  58-1      99                      1 91
 5 61964 4YKS  570  1  21  128  22414303300 0101P/PKP   9999  0 -15        99      99    99
 6 51964 4 22414305100 0 35S        -129999999        99      99    99
 5 61964 4NGS  344  1 359  347  2241431 800 0101P/PKP   9999  0  32   e    99      99    99
.
.
.
 5 51964 4ORV  366  1  47 8464  12414423840-1101P/PKP   9999  0  -5   i    99      99    99
 5 51964 4FBC  139  1   8 8619  12414424500 0101P/PKP   9999  0 -10C  e    99      99    99
 5 11964 4UBO  528  1  41 9146  12414431300 0101P/PKP   9999  0  14   e   7-1      99    9946
 1 11964 42414402830-1171B 133000-1 -888000-11580 0 510    -1B     039-2            100-1  44
 1 11964 42414404200 0 19C 128700-2 -883500-2 800 0        99         99               99
 1 21964 42414403020-1  1A 136500-2 -885800-21530 0 420    -1B    1   99  73  6   4 145-2 102
 2 51964 4        99         99  150-2   410-3   440-3  41-1      99       14911  418     1 52
 5 61964 4SSS  473  1 273   60  22414405050-1101P/PKP   9999  0 -23   i    99      99    99
 6 51964 4 22414441100 0 35S       17959999999        99      99    99
 5 51964 4NCS  340  1 308   74  12414405220-1101P/PKP   9999  0 -14   i    99      99    99
 5 51964 4UBO  528  1 329 3232  12414431300 0101P/PKP   9999  09999   e   7-1      99    9942
 5 11964 4FBC  139  1  11 5196  12414424500 0101P/PKP   9999  09999C  e    99      99    99
 1 11964 42414402830-1171B 133000-1 -888000-11580 0 510    -1B     039-2            100-1  44
 1 11964 42414404200 0 19C 128700-2 -883500-2 800 0        99         99               99
 1 21964 42414403020-1  1A 136500-2 -885800-21530 0 530    -1B   17   99  73  6 112 145-2 102
 2 51964 4        99         99  150-2   410-3   440-3  41-1      99       14911  418     1163
 5 61964 4SSS  473  1 273   60  22414405050-1101P/PKP   9999  0 -23   i    99      99    99
 6 51964 4 22414441100 0 35S       17959999999        99      99    99
 5 51964 4NCS  340  1 308   74  12414405220-1101P/PKP   9999  0 -14   i    99      99    99
 5 61964 4LPS  287  1 319   85  22414405380-1101P/PKP   9999  0  -5   i    99      99    99
 6 51964 4 22414412200 0 35S         849999999        99      99    99
 5 51964 4BHP   51  1 117 1000  12414425100-1101P/PKP   9999  0   0   i  16-1      99    99
 5 61964 4TAC  483  1 301 1167  52414431500-1101P/PKP   9999  0  20   i    99      99    99
.
.

Appendix 5 Duplicated hypocenters

DATE                msec Agency hypid     remid
14-05-1977 03:39:23 100  LDG    1543028   8239    
14-05-1977 03:39:23 100  LDG    1543029   8239    
07-03-1977 09:41:39 500  SPGM   1547496   8240    
07-03-1977 09:41:39 500  SPGM   1547497   8240    
07-06-1980 18:35:02 800  LDG    1426142   8236    
07-06-1980 18:35:02 800  LDG    1426143   8236    
07-03-1981 00:51:36 700  TRN    1396067   8235    
07-03-1981 00:51:36 700  TRN    1396068   8235    
22-06-1981 10:21:01 700  PEK    1387317   8232    
22-06-1981 10:21:01 700  PEK    1387318   8232    
04-10-1981 00:01:28 800  MOS    1350451   8229    
04-10-1981 00:01:28 800  MOS    1350452   8229    
14-01-1982 11:36:05 300  PEK    1341371   8228    
14-01-1982 11:36:05 300  PEK    1341372   8228    
26-02-1983 03:01:25 400  MOS    1292029   8227    
26-02-1983 03:01:25 400  MOS    1292030   8227    
20-02-1983 10:49:50 300  MOS    1291206   8224
20-02-1983 10:49:50 300  MOS    1291207   8224
25-02-1973 20:04:28   0  WEL    1692982   8246    
25-02-1973 20:04:28   0  WEL    1692983   8246    
12-12-1970 10:03:00   0  TAP    1747049   8247    
12-12-1970 10:03:00   0  TAP    1747050   8247    
20-04-1967 01:42:42   0  BUL    1832550   8249    
20-04-1967 01:42:42   0  BUL    1832551   8249    
03-07-1981 17:14:50 200  PEK    1381758   8231
03-07-1981 17:14:50 200  PEK    1381759   8231
30-06-1981 21:55:50 600  MOS    1388342   8233
30-06-1981 21:55:50 600  MOS    1388343   8233
31-12-1967 04:52:14   0  NOU    1818834   8248
31-12-1967 04:52:14   0  NOU    1818835   8248
09-11-1974 10:12:57 900  PAS    1622242   10437
09-11-1974 10:12:57 900  PAS    1622243   10437