src.lei_generate_metadata_sql Namespace Reference

Functions
Path	get_repo_root ()
str	get_header ()
str	escape_sql (str s)
str	parse_as_of_date (str filename)
dict	find_csv_files (Path lei_dir)
Path|None	find_lei_bic_csv (Path lei_dir)
str	sql_value (str val, bool is_timestamp)
list	read_csv_rows (Path csv_file, dict column_map, set timestamp_cols)
	generate_catalog_sql (dict manifest, Path output_file)
	generate_methodology_sql (dict manifest, Path output_file)
	generate_dataset_sql (dict manifest, str as_of_date, Path output_file)
	generate_dataset_dependency_sql (dict manifest, Path output_file)
	generate_artefact_populate_sql (Path csv_file, str dataset_name, str dataset_code, str table_name, dict column_map, set timestamp_cols, Path output_file)
	generate_master_sql (Path output_file)
	main ()

Variables
dict	LEI2_COLUMN_MAP
dict	RR_COLUMN_MAP
dict	LEI2_TIMESTAMP_COLS
dict	RR_TIMESTAMP_COLS
dict	LEI_BIC_COLUMN_MAP

Detailed Description

Generates SQL populate scripts for GLEIF LEI metadata and artefact data.

Reads the manifest.json from external/lei/ and CSV subset files to generate:
  - lei_catalog_populate.sql
  - lei_methodology_populate.sql
  - lei_dataset_populate.sql
  - lei_dataset_dependency_populate.sql
  - lei_entities_small_artefact_populate.sql
  - lei_entities_large_artefact_populate.sql
  - lei_relationships_small_artefact_populate.sql
  - lei_relationships_large_artefact_populate.sql
  - lei_populate.sql (master include file)

Usage:
    python3 lei_generate_metadata_sql.py
    python3 lei_generate_metadata_sql.py --manifest-dir /path/to/external/lei
    python3 lei_generate_metadata_sql.py --output-dir /path/to/output

Function Documentation

get_repo_root()

Path get_repo_root

(

)

Get the repository root directory.

Here is the caller graph for this function:

get_header()

str get_header

(

)

Generate SQL file header.

Here is the caller graph for this function:

escape_sql()

str escape_sql

(

str

s

)

Escape single quotes for SQL strings.

Here is the caller graph for this function:

parse_as_of_date()

str parse_as_of_date

(

str

filename

)

Extract YYYYMMDD date from CSV filename and return as YYYY-MM-DD.

Here is the caller graph for this function:

find_csv_files()

dict find_csv_files

(

Path

lei_dir

)

Find LEI CSV subset files and return them grouped by type and size.

Here is the caller graph for this function:

find_lei_bic_csv()

Path | None find_lei_bic_csv

(

Path

lei_dir

)

Find the latest LEI-BIC mapping CSV file.

Here is the caller graph for this function:

sql_value()

str sql_value	(	str	val,
		bool	is_timestamp
	)

Format a CSV value as a SQL literal.

Here is the caller graph for this function:

read_csv_rows()

list read_csv_rows	(	Path	csv_file,
		dict	column_map,
		set	timestamp_cols
	)

Read CSV file and extract mapped columns as SQL value tuples.

Here is the caller graph for this function:

generate_catalog_sql()

generate_catalog_sql	(	dict	manifest,
		Path	output_file
	)

Generate the catalog populate SQL file.

Here is the caller graph for this function:

generate_methodology_sql()

generate_methodology_sql	(	dict	manifest,
		Path	output_file
	)

Generate the methodology populate SQL file.

Here is the caller graph for this function:

generate_dataset_sql()

generate_dataset_sql	(	dict	manifest,
		str	as_of_date,
		Path	output_file
	)

Generate the dataset populate SQL file.

Here is the caller graph for this function:

generate_dataset_dependency_sql()

generate_dataset_dependency_sql	(	dict	manifest,
		Path	output_file
	)

Generate the dataset dependency populate SQL file.

Here is the caller graph for this function:

generate_artefact_populate_sql()

generate_artefact_populate_sql	(	Path	csv_file,
		str	dataset_name,
		str	dataset_code,
		str	table_name,
		dict	column_map,
		set	timestamp_cols,
		Path	output_file
	)

Generate an artefact populate SQL file from a CSV subset.

Here is the caller graph for this function:

generate_master_sql()

generate_master_sql

(

Path

output_file

)

Generate the lei_populate.sql master include file.

Here is the caller graph for this function:

Variable Documentation

LEI2_COLUMN_MAP

dict LEI2_COLUMN_MAP

Initial value:

=  {
    'LEI': 'lei',
    'Entity.LegalName': 'entity_legal_name',
    'Entity.EntityCategory': 'entity_entity_category',
    'Entity.EntitySubCategory': 'entity_entity_sub_category',
    'Entity.EntityStatus': 'entity_entity_status',
    'Entity.LegalForm.EntityLegalFormCode': 'entity_legal_form_entity_legal_form_code',
    'Entity.LegalForm.OtherLegalForm': 'entity_legal_form_other_legal_form',
    'Entity.LegalJurisdiction': 'entity_legal_jurisdiction',
    'Entity.LegalAddress.FirstAddressLine': 'entity_legal_address_first_address_line',
    'Entity.LegalAddress.City': 'entity_legal_address_city',
    'Entity.LegalAddress.Region': 'entity_legal_address_region',
    'Entity.LegalAddress.Country': 'entity_legal_address_country',
    'Entity.LegalAddress.PostalCode': 'entity_legal_address_postal_code',
    'Entity.HeadquartersAddress.FirstAddressLine': 'entity_headquarters_address_first_address_line',
    'Entity.HeadquartersAddress.City': 'entity_headquarters_address_city',
    'Entity.HeadquartersAddress.Region': 'entity_headquarters_address_region',
    'Entity.HeadquartersAddress.Country': 'entity_headquarters_address_country',
    'Entity.HeadquartersAddress.PostalCode': 'entity_headquarters_address_postal_code',
    'Entity.EntityCreationDate': 'entity_entity_creation_date',
    'Registration.InitialRegistrationDate': 'registration_initial_registration_date',
    'Registration.LastUpdateDate': 'registration_last_update_date',
    'Registration.NextRenewalDate': 'registration_next_renewal_date',
    'Registration.RegistrationStatus': 'registration_registration_status',
    'Entity.TransliteratedOtherEntityNames.TransliteratedOtherEntityName.1': 'entity_transliterated_name_1',
    'Entity.TransliteratedOtherEntityNames.TransliteratedOtherEntityName.1.type': 'entity_transliterated_name_1_type',
}

RR_COLUMN_MAP

dict RR_COLUMN_MAP

Initial value:

=  {
    'Relationship.StartNode.NodeID': 'relationship_start_node_node_id',
    'Relationship.StartNode.NodeIDType': 'relationship_start_node_node_id_type',
    'Relationship.EndNode.NodeID': 'relationship_end_node_node_id',
    'Relationship.EndNode.NodeIDType': 'relationship_end_node_node_id_type',
    'Relationship.RelationshipType': 'relationship_relationship_type',
    'Relationship.RelationshipStatus': 'relationship_relationship_status',
    'Relationship.Period.1.startDate': 'relationship_period_1_start_date',
    'Relationship.Period.1.endDate': 'relationship_period_1_end_date',
    'Registration.InitialRegistrationDate': 'registration_initial_registration_date',
    'Registration.LastUpdateDate': 'registration_last_update_date',
    'Registration.RegistrationStatus': 'registration_registration_status',
    'Registration.ValidationSources': 'registration_validation_sources',
}

LEI2_TIMESTAMP_COLS

dict LEI2_TIMESTAMP_COLS

Initial value:

=  {
    'entity_entity_creation_date',
    'registration_initial_registration_date',
    'registration_last_update_date',
    'registration_next_renewal_date',
}

RR_TIMESTAMP_COLS

dict RR_TIMESTAMP_COLS

Initial value:

=  {
    'relationship_period_1_start_date',
    'relationship_period_1_end_date',
    'registration_initial_registration_date',
    'registration_last_update_date',
}

LEI_BIC_COLUMN_MAP

dict LEI_BIC_COLUMN_MAP

Initial value:

=  {
    'LEI': 'lei',
    'BIC': 'bic',
}