snowmobile.toml¶
The parsed and validated form of snowmobile.toml is a Configuration
object.
All parsing of the file is done within snowmobile.core.cfg
, in which sections are split at the root and fed into pydantic’s glorious API to define the schema and impose (evolving) validation where needed.
Once validated, the Configuration
object serves as a namespace for the contents/structure of the configuration file and utility methods implemented on top of them, with the rest of the API accessing it as the cfg
attribute of .
Inspecting sn.cfg
¶
The Configuration
model is accessed as the cfg
attribute of Snowmobile
; a straight-forward way to inspect its composition is to instantiate a delayed instance of :
import snowmobile
sn = snowmobile.Snowmobile(delay=True)
type(sn.cfg) #> snowmobile.core.configuration.Configuration
print(sn.cfg.location) # 'path/to/your/snowmobile.toml'
The following attributes of sn.cfg
map to the root configuration sections of snowmobile.toml:
type(sn.cfg.connection) #> snowmobile.core.cfg.connection.Connection
type(sn.cfg.loading) #> snowmobile.core.cfg.loading.Loading
type(sn.cfg.script) #> snowmobile.core.cfg.script.Script
type(sn.cfg.sql) #> snowmobile.core.cfg.other.SQL
type(sn.cfg.ext_sources) #> snowmobile.core.cfg.other.Location
Tip
The usage documentation contains detail on how changes to snowmobile.toml values flow through to and impact the its implementation.
Glossary¶
[connection]
Configuration options used by when establishing connections to Snowflake
default-creds
The credentials (alias) to use by default in absence of one provided to the creds
keyword argument to snowmobile.connect()
[connection.credentials]
Groups subsections of credentials, each declared with the structure of [connection.credentials.credentials_alias]
[connection.credentials.creds1]
Store your first set of credentials here; creds1
is a credentials alias
[connection.credentials.creds2]
Store as many credentials as you want following this format; aliases must be unique
[connection.default-arguments]
Credentials-agnostic keyword arguments to pass to snowflake.connector.connect()
[loading]
Configuration options for data loading used by snowmobile.Table
[loading.default-table-kwargs]
Default specifications for a snowmobile.Table object
[loading.put]
Default arguments to include in Snowflake’s put file from stage command
[loading.copy-into]
Default arguments to include in Snowflake’s copy into table command
[loading.save-options]
Groups subsections of save-options
[loading.save-options."snowmobile_default_csv"]
Default file-save options for snowmobile_default_csv
[loading.save-options."snowmobile_default_psv"]
Default file-save options for snowmobile_default_psv
[external-sources]
Defines paths to custom sources referenced by different snowmobile objects
ddl
Posix path to a sql file containing DDL for file formats
extension
Posix path to snowmobile-ext.toml
[script]
Configurations for snowmobile.Script
export-dir-name
Directory name for generated exports (markup and stripped sql scripts)
[script.patterns.core]
Core patterns used for markup identification
open-tag
Open-pattern for in-script tags
close-tag
Close-pattern for in-script tags
description-delimiter
Delimiter separating description from other statement attributes
description-index-prefix
String with which to prepend a statement’s index position when deriving desc_ge
[script.patterns.wildcards]
Defines wildcards for attribute names within script tags
wildcard-character
The literal character to use as a wildcard
wildcard-delimiter
The literal character with which to delimit wildcards
denotes-paragraph
Indicates the attribute value should be rendered as free-form markdown as opposed to a plain text bullet
denotes-no-reformat
Indicates the attribute name should be left exactly as it is entered in the script as opposed to title-cased
denotes-omit-name-in-output
Indicates to omit the attribute’s name in rendered output
[script.qa]
Default arguments for QA-Diff and QA-Empty Statements
partition-on
Pattern to identify the field on which to partition data for comparison
compare-patterns
Pattern to identify fields being compared
ignore-patterns
Pattern to identify fields that should be ignored in comparison
end-index-at
Pattern to identify the field marking the last index column
[script.qa.default-tolerance]
Default values for QA-Delta tolerance levels
relative
Default relative-difference tolerance
absolute
Default absolute-difference tolerance
[script.markdown]
Configuration for markdown generated from .sql files
default-marker-header
Header level for markers (h1-h6)
default-statement-header
Header level for statements (h1-h6)
default-bullet-character
Character to use for bulleted lists
wrap-attribute-names-with
Character to wrap attribute names with
wrap-attribute-values-with
Character to wrap attribute values with
include-statement-index-in-header
Denotes whether or not to include a statement’s relative index number in its header along with its name
limit-query-results-to
Maximum number of rows to include for a statement’s rendered Results
[script.markdown.attributes]
Configuration options for specific attributes
[script.markdown.attributes.markers]
Pre-defined marker configurations
[script.markdown.attributes.markers."__script__"]
Scaffolding for a template marker called ‘__script__’
as-group
The literal text within which to group associated attributes as sub-bullets
team
A sample attribute called ‘team’
author-name
A sample attribute called ‘author-name’
email
A sample attribute called ‘email’
[script.markdown.attributes.markers."__appendix__"]
Scaffolding for a second template marker called ‘__appendix__’
[script.markdown.attributes.reserved.rendered-sql]
Configuration options for a reserved attribute called ‘rendered-sql’
include-by-default
Include attribute by default for each Section
attribute-name
The attribute’s name as it is declared within a tag
default-to
The attribute name as it should be interpreted when parsed
[script.markdown.attributes.reserved.query-results]
Configuration for a reserved attributes called query-results
include-by-default
Include attribute by default for each Section
attribute-name
The attribute’s name as it is declared within a tag
default-to
The attribute name as it should be interpreted when parsed
format
Render format for the tabular results; markdown or html
[script.markdown.attributes.from-namespace]
List of Statement
attributes to include in its Section
; includes non-default attributes set on an instance
[script.markdown.attributes.groups]
Defines attributes to be grouped together within a sub-bulleted list
[script.markdown.attributes.order]
Order of attributes within a Statement
-level section
[script.tag-to-type-xref]
Maps tagged attributes to data types; will error if an attribute included here cannot be parsed into its specified data type
[sql]
SQL parsing specifications for a Statement
provided-over-generated
nm_pr
takes precedent over nm_ge
desc-is-simple
True
invokes additional parsing into desc
and obj
named-objects
Literal strings to search for matches that qualify as a Snowflake object if included within the first line of a statement’s sql and not equal to its first keyword
generic-anchors
Generic anchors to use for a given keyword; will be used for generated statements if desc-is-simple
is True
keyword-exceptions
Alternate mapping for first keyword found in a command
information-schema-exceptions
Map Snowflake objects to their information_schema.*
table name if different than the plural form of the object; (e.g. schema
information is in information_schema.schemata
not information_schema.schemas
)
File Contents¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | [connection]
default-creds = ''
[connection.credentials.creds1]
user = ''
password = ''
role = ''
account = ''
warehouse = ''
database = ''
schema = ''
[connection.credentials.creds2]
user = ''
password = ''
role = ''
account = ''
warehouse = ''
database = ''
schema = ''
[connection.default-arguments]
autocommit = true
authenticator = 'snowflake'
[loading]
[loading.default-table-kwargs]
file_format = 'snowmobile_default_psv'
validate_table = true
validate_format = true
if_exists = 'append'
keep_local = false
reformat_cols = true
upper_case_cols = true
check_dupes = true
load_copy = true
[loading.put]
auto_compress = true
[loading.copy-into]
on_error = 'continue'
[loading.save-options]
[loading.save-options."snowmobile_default_csv"]
index = false
header = false
quotechar = '"'
sep = ","
[loading.save-options."snowmobile_default_psv"]
index = false
header = false
quotechar = '"'
sep = "|"
[external-sources]
ddl = ''
extension = ''
sql-save-heading = ''
[script]
export-dir-name = '.snowmobile'
result-limit = -1
[script.patterns]
[script.patterns.core]
open-tag = '/*-'
close-tag = '-*/'
description-delimiter = '~'
description-index-prefix = "s"
[script.patterns.markup]
wildcard-character = '*'
wildcard-delimiter = '_'
denotes-paragraph = '*'
denotes-no-reformat = '**'
denotes-omit-name-in-output = '***'
[script.qa]
partition-on = 'src_description'
compare-patterns = ['.*_diff']
ignore-patterns = ['.*_tmstmp']
end-index-at = 'end_index'
[script.qa.default-tolerance]
relative = 0.0
absolute = 0.0
[script.markup]
default-marker-header = 'h1'
default-statement-header = 'h2'
default-bullet-character = '*'
wrap-attribute-names-with = '**'
wrap-attribute-values-with = '_'
include-statement-index-in-header = true
limit-query-results-to = 20
|
snowmobile-ext.toml¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | # =============================================================================
# ../snowmobile-ext.toml
# DO NOT DELETE UNLESS ALTERNATE EXTENSION FILE IS SPECIFED IN SNOWMOBILE.TOML
# =============================================================================
# todo: + 'tabs-to-spaces' and 'tab-size' for reserved attributes.sql
# -- Configuration options for snowmobile.script() ----------------------------
[script]
[script.markup.attributes]
[script.markup.attributes.markers]
[script.markup.attributes.markers."__script__"]
as-group = 'Author Information'
# ====/ start-attributes /====
team = 'Sample Team Name'
email = 'first.last@domain.com'
[script.markup.attributes.markers."__appendix__"]
as-group = ''
# ====/ marker-attributes /====
[script.markup.attributes.reserved.rendered-sql]
# The literal sql for a given statement.
include-by-default = true
attribute-name = 'sql'
default-to = 'SQL***'
# TODO: Make each of these a dictionary of derived classes like QA for Script
# with the .process() method being all the operations that have access to
# the tagged values from the script and the arguments in this block
# tabs-to-spaces = true
# tab-size = 2
[script.markup.attributes.reserved.query-results]
# The literal results returned by a given statement.
include-by-default = false
attribute-name = 'results'
default-to = 'results*_***'
format = 'markdown'
tabulate-format = 'grid'
[script.markup.attributes.from-namespace]
# TODO: period separated values for nested vals to vars(obj)[k]
# :: execution.time.str to check if str in obj.callables or obj.wrap
# :: * only applying if the bool(val) that's returned is True
execution_time_txt = 'Execution Time'
outcome_txt = 'Last Outcome'
[script.markup.attributes.groups]
"Execution-Information**" = [
# todo
# :: 'Execution Time', 'Last Outcome', etc
'execution_time_txt',
'outcome_txt'
]
"QA-Specifications**" = [
'partition-on',
'end-index-at',
'compare-patterns',
'ignore-patterns',
'absolute-tolerance',
'relative-tolerance',
]
[script.markup.attributes.order]
attribute-order = [
'authored-date',
'author-information',
'execution-information',
'qa-specifications',
'outcome_txt',
'execution_time_txt',
'description',
'p',
'sql',
'results'
]
# -- Type mapping of how attribute values should be parsed based on attr name -
[script.tag-to-type-xref]
string = [
'name', 'partition-on', 'end-index-at', 'description'
]
list = [
'compare-patterns', 'ignore-patterns'
]
float = [
'absolute-tolerance', 'relative-tolerance'
]
bool = [
'results', 'sql', 'transpose'
]
# -- Named object, generic anchors, and keyword-exceptions for sql parsing ----
[sql]
provided-over-generated = true
desc-is-simple = true
named-objects = [
# 'grant' statements
"select",
"all",
"drop",
# base objects
"temp table",
"transient table",
"table",
"view",
"schema",
"warehouse",
"file format",
# plural bases
"tables",
"views",
"schemas",
]
[sql.generic-anchors]
"select" = "select data"
"set" = "set param"
"unset" = "unset param"
"insert" = "insert into"
"delete" = "delete from"
[sql.keyword-exceptions]
"with" = "select"
[sql.information-schema-exceptions]
schema = "schemata"
|