snowmobile.toml

snowmobile.core.configuration snowmobile.core.cfg

The parsed and validated form of snowmobile.toml is a Configuration object.

All parsing of the file is done within snowmobile.core.cfg, in which sections are split at the root and fed into pydantic’s glorious API to define the schema and impose (evolving) validation where needed.

Once validated, the Configuration object serves as a namespace for the contents/structure of the configuration file and utility methods implemented on top of them, with the rest of the API accessing it as the cfg attribute of .

Inspecting sn.cfg

The Configuration model is accessed as the cfg attribute of Snowmobile; a straight-forward way to inspect its composition is to instantiate a delayed instance of :

import snowmobile

sn = snowmobile.Snowmobile(delay=True)

type(sn.cfg)              #> snowmobile.core.configuration.Configuration
print(sn.cfg.location)    #  'path/to/your/snowmobile.toml'

The following attributes of sn.cfg map to the root configuration sections of snowmobile.toml:

type(sn.cfg.connection)   #> snowmobile.core.cfg.connection.Connection
type(sn.cfg.loading)      #> snowmobile.core.cfg.loading.Loading
type(sn.cfg.script)       #> snowmobile.core.cfg.script.Script
type(sn.cfg.sql)          #> snowmobile.core.cfg.other.SQL
type(sn.cfg.ext_sources)  #> snowmobile.core.cfg.other.Location

inspect_configuration.py

Tip

The usage documentation contains detail on how changes to snowmobile.toml values flow through to and impact the its implementation.

Glossary


[connection]
Configuration options used by when establishing connections to Snowflake

default-creds
The credentials (alias) to use by default in absence of one provided to the creds keyword argument to snowmobile.connect()

[connection.credentials]
Groups subsections of credentials, each declared with the structure of [connection.credentials.credentials_alias]

[connection.credentials.creds1]
Store your first set of credentials here; creds1 is a credentials alias

[connection.credentials.creds2]
Store as many credentials as you want following this format; aliases must be unique

[connection.default-arguments]
Credentials-agnostic keyword arguments to pass to snowflake.connector.connect()

[loading]
Configuration options for data loading used by snowmobile.Table

[loading.default-table-kwargs]
Default specifications for a snowmobile.Table object

[loading.put]
Default arguments to include in Snowflake’s put file from stage command

[loading.copy-into]
Default arguments to include in Snowflake’s copy into table command

[loading.save-options]
Groups subsections of save-options

[loading.save-options."snowmobile_default_csv"]
Default file-save options for snowmobile_default_csv

[loading.save-options."snowmobile_default_psv"]
Default file-save options for snowmobile_default_psv

[external-sources]
Defines paths to custom sources referenced by different snowmobile objects

ddl
Posix path to a sql file containing DDL for file formats

extension
Posix path to snowmobile-ext.toml

[script]
Configurations for snowmobile.Script

export-dir-name
Directory name for generated exports (markup and stripped sql scripts)

[script.patterns.core]
Core patterns used for markup identification

open-tag
Open-pattern for in-script tags

close-tag
Close-pattern for in-script tags

description-delimiter
Delimiter separating description from other statement attributes

description-index-prefix
String with which to prepend a statement’s index position when deriving desc_ge

[script.patterns.wildcards]
Defines wildcards for attribute names within script tags

wildcard-character
The literal character to use as a wildcard

wildcard-delimiter
The literal character with which to delimit wildcards

denotes-paragraph
Indicates the attribute value should be rendered as free-form markdown as opposed to a plain text bullet

denotes-no-reformat
Indicates the attribute name should be left exactly as it is entered in the script as opposed to title-cased

denotes-omit-name-in-output
Indicates to omit the attribute’s name in rendered output

[script.qa]
Default arguments for QA-Diff and QA-Empty Statements

partition-on
Pattern to identify the field on which to partition data for comparison

compare-patterns
Pattern to identify fields being compared

ignore-patterns
Pattern to identify fields that should be ignored in comparison

end-index-at
Pattern to identify the field marking the last index column

[script.qa.default-tolerance]
Default values for QA-Delta tolerance levels

relative
Default relative-difference tolerance

absolute
Default absolute-difference tolerance

[script.markdown]
Configuration for markdown generated from .sql files

default-marker-header
Header level for markers (h1-h6)

default-statement-header
Header level for statements (h1-h6)

default-bullet-character
Character to use for bulleted lists

wrap-attribute-names-with
Character to wrap attribute names with

wrap-attribute-values-with
Character to wrap attribute values with

include-statement-index-in-header
Denotes whether or not to include a statement’s relative index number in its header along with its name

limit-query-results-to
Maximum number of rows to include for a statement’s rendered Results

[script.markdown.attributes]
Configuration options for specific attributes

[script.markdown.attributes.markers]
Pre-defined marker configurations

[script.markdown.attributes.markers."__script__"]
Scaffolding for a template marker called ‘__script__’

as-group
The literal text within which to group associated attributes as sub-bullets

team
A sample attribute called ‘team’

author-name
A sample attribute called ‘author-name’

email
A sample attribute called ‘email’

[script.markdown.attributes.markers."__appendix__"]
Scaffolding for a second template marker called ‘__appendix__’

[script.markdown.attributes.reserved.rendered-sql]
Configuration options for a reserved attribute called ‘rendered-sql’

include-by-default
Include attribute by default for each Section

attribute-name
The attribute’s name as it is declared within a tag

default-to
The attribute name as it should be interpreted when parsed

[script.markdown.attributes.reserved.query-results]
Configuration for a reserved attributes called query-results

include-by-default
Include attribute by default for each Section

attribute-name
The attribute’s name as it is declared within a tag

default-to
The attribute name as it should be interpreted when parsed

format
Render format for the tabular results; markdown or html

[script.markdown.attributes.from-namespace]
List of Statement attributes to include in its Section; includes non-default attributes set on an instance

[script.markdown.attributes.groups]
Defines attributes to be grouped together within a sub-bulleted list

[script.markdown.attributes.order]
Order of attributes within a Statement-level section

[script.tag-to-type-xref]
Maps tagged attributes to data types; will error if an attribute included here cannot be parsed into its specified data type

[sql]
SQL parsing specifications for a Statement

provided-over-generated
nm_pr takes precedent over nm_ge

desc-is-simple
True invokes additional parsing into desc and obj

named-objects
Literal strings to search for matches that qualify as a Snowflake object if included within the first line of a statement’s sql and not equal to its first keyword

generic-anchors
Generic anchors to use for a given keyword; will be used for generated statements if desc-is-simple is True

keyword-exceptions
Alternate mapping for first keyword found in a command

information-schema-exceptions
Map Snowflake objects to their information_schema.* table name if different than the plural form of the object; (e.g. schema information is in information_schema.schemata not information_schema.schemas)

File Contents

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
[connection]
    default-creds = ''

    [connection.credentials.creds1]
        user = ''
        password = ''
        role = ''
        account = ''
        warehouse = ''
        database = ''
        schema = ''

    [connection.credentials.creds2]
        user = ''
        password = ''
        role = ''
        account = ''
        warehouse = ''
        database = ''
        schema = ''

    [connection.default-arguments]
        autocommit = true
        authenticator = 'snowflake'

[loading]

    [loading.default-table-kwargs]
        file_format = 'snowmobile_default_psv'
        validate_table = true
        validate_format = true
        if_exists = 'append'
        keep_local = false
        reformat_cols = true
        upper_case_cols = true
        check_dupes = true
        load_copy = true

    [loading.put]
        auto_compress = true

    [loading.copy-into]
        on_error = 'continue'

    [loading.save-options]
        [loading.save-options."snowmobile_default_csv"]
            index = false
            header = false
            quotechar = '"'
            sep = ","
        [loading.save-options."snowmobile_default_psv"]
            index = false
            header = false
            quotechar = '"'
            sep = "|"

[external-sources]
    ddl = ''
    extension = ''
    sql-save-heading = ''

[script]
    export-dir-name = '.snowmobile'
    result-limit = -1

    [script.patterns]

        [script.patterns.core]
            open-tag = '/*-'
            close-tag = '-*/'
            description-delimiter = '~'
            description-index-prefix = "s"

        [script.patterns.markup]
            wildcard-character = '*'
            wildcard-delimiter = '_'
            denotes-paragraph = '*'
            denotes-no-reformat = '**'
            denotes-omit-name-in-output = '***'

    [script.qa]
        partition-on = 'src_description'
        compare-patterns = ['.*_diff']
        ignore-patterns = ['.*_tmstmp']
        end-index-at = 'end_index'

        [script.qa.default-tolerance]
            relative = 0.0
            absolute = 0.0

    [script.markup]
        default-marker-header = 'h1'
        default-statement-header = 'h2'
        default-bullet-character = '*'
        wrap-attribute-names-with = '**'
        wrap-attribute-values-with = '_'
        include-statement-index-in-header = true
        limit-query-results-to = 20

snowmobile-ext.toml

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
# =============================================================================
# ../snowmobile-ext.toml
# DO NOT DELETE UNLESS ALTERNATE EXTENSION FILE IS SPECIFED IN SNOWMOBILE.TOML
# =============================================================================

# todo: + 'tabs-to-spaces' and 'tab-size' for reserved attributes.sql

# -- Configuration options for snowmobile.script() ----------------------------
[script]

    [script.markup.attributes]

        [script.markup.attributes.markers]
            [script.markup.attributes.markers."__script__"]
                as-group = 'Author Information'
                # ====/ start-attributes /====
                team = 'Sample Team Name'
                email = 'first.last@domain.com'

            [script.markup.attributes.markers."__appendix__"]
                as-group = ''
                # ====/ marker-attributes /====

        [script.markup.attributes.reserved.rendered-sql]
            # The literal sql for a given statement.
            include-by-default = true
            attribute-name = 'sql'
            default-to = 'SQL***'
            # TODO: Make each of these a dictionary of derived classes like QA for Script
            #   with the .process() method being all the operations that have access to
            #   the tagged values from the script and the arguments in this block
            # tabs-to-spaces = true
            # tab-size = 2

        [script.markup.attributes.reserved.query-results]
            # The literal results returned by a given statement.
            include-by-default = false
            attribute-name = 'results'
            default-to = 'results*_***'
            format = 'markdown'
            tabulate-format = 'grid'

        [script.markup.attributes.from-namespace]
            # TODO: period separated values for nested vals to vars(obj)[k]
            #   :: execution.time.str to check if str in obj.callables or obj.wrap
            #   :: * only applying if the bool(val) that's returned is True
            execution_time_txt = 'Execution Time'
            outcome_txt = 'Last Outcome'

        [script.markup.attributes.groups]
            "Execution-Information**" = [
                # todo
                #   :: 'Execution Time', 'Last Outcome', etc
                'execution_time_txt',
                'outcome_txt'
            ]
            "QA-Specifications**" = [
                'partition-on',
                'end-index-at',
                'compare-patterns',
                'ignore-patterns',
                'absolute-tolerance',
                'relative-tolerance',
            ]

        [script.markup.attributes.order]
            attribute-order = [
                'authored-date',
                'author-information',
                'execution-information',
                'qa-specifications',
                'outcome_txt',
                'execution_time_txt',
                'description',
                'p',
                'sql',
                'results'
            ]


# -- Type mapping of how attribute values should be parsed based on attr name -
[script.tag-to-type-xref]
    string = [
        'name', 'partition-on', 'end-index-at', 'description'
    ]
    list = [
        'compare-patterns', 'ignore-patterns'
    ]
    float = [
        'absolute-tolerance', 'relative-tolerance'
    ]
    bool = [
        'results', 'sql', 'transpose'
    ]

# -- Named object, generic anchors, and keyword-exceptions for sql parsing ----
[sql]
 provided-over-generated = true
    desc-is-simple = true

    named-objects = [
        # 'grant' statements
        "select",
        "all",
        "drop",

        # base objects
        "temp table",
        "transient table",
        "table",
        "view",
        "schema",
        "warehouse",
        "file format",

        # plural bases
        "tables",
        "views",
        "schemas",
    ]

    [sql.generic-anchors]
        "select" = "select data"
        "set" = "set param"
        "unset" = "unset param"
        "insert" = "insert into"
        "delete" = "delete from"

    [sql.keyword-exceptions]
        "with" = "select"

    [sql.information-schema-exceptions]
        schema = "schemata"