{"pageProps":{"type":"HTML","data":{"body":"
\n

Utilities

\n
\n
\ndagster.file_relative_path(dunderfile, relative_path)[source]
\n

Get a path relative to the currently executing Python file.

\n

This function is useful when one needs to load a file that is relative to the position of\nthe current file. (Such as when you encode a configuration file path in source file and want\nin runnable in any current working directory)

\n
\n
Parameters:
\n
    \n

  • dunderfile (str) – Should always be __file__.

    • \n

  • relative_path (str) – Path to get relative to the currently executing file.

    • \n
\n
\n
\n

Examples:

\n
file_relative_path(__file__, 'path/relative/to/file')\n
\n
\n
\n\n
\n
\ndagster.config_from_files(config_files)[source]
\n

Constructs run config from YAML files.

\n
\n
Parameters:
\n

config_files (List[str]) – List of paths or glob patterns for yaml files\nto load and parse as the run config.

\n
\n
Returns:
\n

A run config dictionary constructed from provided YAML files.

\n
\n
Return type:
\n

Dict[str, Any]

\n
\n
Raises:
\n
    \n

  • FileNotFoundError – When a config file produces no results

    • \n

  • DagsterInvariantViolationError – When one of the YAML files is invalid and has a parse\n error.

    • \n
\n
\n
\n
\n\n
\n
\ndagster.config_from_pkg_resources(pkg_resource_defs)[source]
\n

Load a run config from a package resource, using pkg_resources.resource_string().

\n

Example

\n
config_from_pkg_resources(\n    pkg_resource_defs=[\n        ('dagster_examples.airline_demo.environments', 'local_base.yaml'),\n        ('dagster_examples.airline_demo.environments', 'local_warehouse.yaml'),\n    ],\n)\n
\n
\n
\n
Parameters:
\n

pkg_resource_defs (List[(str, str)]) – List of pkg_resource modules/files to\nload as the run config.

\n
\n
Returns:
\n

A run config dictionary constructed from the provided yaml strings

\n
\n
Return type:
\n

Dict[Str, Any]

\n
\n
Raises:
\n

DagsterInvariantViolationError – When one of the YAML documents is invalid and has a\n parse error.

\n
\n
\n
\n\n
\n
\ndagster.config_from_yaml_strings(yaml_strings)[source]
\n

Static constructor for run configs from YAML strings.

\n
\n
Parameters:
\n

yaml_strings (List[str]) – List of yaml strings to parse as the run config.

\n
\n
Returns:
\n

A run config dictionary constructed from the provided yaml strings

\n
\n
Return type:
\n

Dict[Str, Any]

\n
\n
Raises:
\n

DagsterInvariantViolationError – When one of the YAML documents is invalid and has a\n parse error.

\n
\n
\n
\n\n
\n
\ndagster.get_dagster_logger(name=None)[source]
\n

Creates a python logger whose output messages will be captured and converted into Dagster log\nmessages. This means they will have structured information such as the step_key, run_id, etc.\nembedded into them, and will show up in the Dagster event log.

\n

This can be used as a more convenient alternative to context.log in most cases. If log level\nis not set explicitly, defaults to DEBUG.

\n
\n
Parameters:
\n

name (Optional[str]) – If supplied, will create a logger with the name “dagster.builtin.{name}”,\nwith properties inherited from the base Dagster logger. If omitted, the returned logger\nwill be named “dagster.builtin”.

\n
\n
Returns:
\n

A logger whose output will be captured by Dagster.

\n
\n
Return type:
\n

logging.Logger

\n
\n
\n

Example

\n
from dagster import get_dagster_logger, op\n\n@op\ndef hello_op():\n    log = get_dagster_logger()\n    for i in range(5):\n        # do something\n        log.info(f"Did {i+1} things!")\n
\n
\n
\n\n
\n
\nclass dagster.ExperimentalWarning[source]
\n
\n\n
\n
\ndagster.make_email_on_run_failure_sensor(email_from, email_password, email_to, email_body_fn=<function _default_failure_email_body>, email_subject_fn=<function _default_failure_email_subject>, smtp_host='smtp.gmail.com', smtp_type='SSL', smtp_port=None, name=None, webserver_base_url=None, monitored_jobs=None, job_selection=None, monitor_all_repositories=False, default_status=DefaultSensorStatus.STOPPED)[source]
\n

Create a job failure sensor that sends email via the SMTP protocol.

\n
\n
Parameters:
\n
    \n

  • email_from (str) – The sender email address to send the message from.

    • \n

  • email_password (str) – The password of the sender.

    • \n

  • email_to (List[str]) – The receipt email addresses to send the message to.

    • \n

  • email_body_fn (Optional(Callable[[RunFailureSensorContext], str])) – Function which\ntakes in the RunFailureSensorContext outputs the email body you want to send.\nDefaults to the plain text that contains error message, job name, and run ID.

    • \n

  • email_subject_fn (Optional(Callable[[RunFailureSensorContext], str])) – Function which\ntakes in the RunFailureSensorContext outputs the email subject you want to send.\nDefaults to “Dagster Run Failed: <job_name>”.

    • \n

  • smtp_host (str) – The hostname of the SMTP server. Defaults to “smtp.gmail.com”.

    • \n

  • smtp_type (str) – The protocol; either “SSL” or “STARTTLS”. Defaults to SSL.

    • \n

  • smtp_port (Optional[int]) – The SMTP port. Defaults to 465 for SSL, 587 for STARTTLS.

    • \n

  • name – (Optional[str]): The name of the sensor. Defaults to “email_on_job_failure”.

    • \n

  • webserver_base_url – (Optional[str]): The base url of your dagster-webserver instance. Specify this to allow\nmessages to include deeplinks to the failed run.

    • \n

  • monitored_jobs (Optional[List[Union[JobDefinition, GraphDefinition, JobDefinition, RepositorySelector, JobSelector]]]) – The jobs that will be monitored by this failure sensor. Defaults to None, which means the alert will\nbe sent when any job in the repository fails. To monitor jobs in external repositories,\nuse RepositorySelector and JobSelector.

    • \n

  • monitor_all_repositories (bool) – If set to True, the sensor will monitor all runs in the\nDagster instance. If set to True, an error will be raised if you also specify\nmonitored_jobs or job_selection. Defaults to False.

    • \n

  • job_selection (Optional[List[Union[JobDefinition, GraphDefinition, JobDefinition, RepositorySelector, JobSelector]]]) – (deprecated in favor of monitored_jobs) The jobs that will be monitored by this failure\nsensor. Defaults to None, which means the alert will be sent when any job in the repository fails.

    • \n

  • default_status (DefaultSensorStatus) – Whether the sensor starts as running or not. The default\nstatus can be overridden from the Dagster UI or via the GraphQL API.

    • \n
\n
\n
\n

Examples

\n
email_on_run_failure = make_email_on_run_failure_sensor(\n    email_from="no-reply@example.com",\n    email_password=os.getenv("ALERT_EMAIL_PASSWORD"),\n    email_to=["xxx@example.com"],\n)\n\n@repository\ndef my_repo():\n    return [my_job + email_on_run_failure]\n
\n
\n
def my_message_fn(context: RunFailureSensorContext) -> str:\n    return (\n        f"Job {context.pipeline_run.job_name} failed!"\n        f"Error: {context.failure_event.message}"\n    )\n\nemail_on_run_failure = make_email_on_run_failure_sensor(\n    email_from="no-reply@example.com",\n    email_password=os.getenv("ALERT_EMAIL_PASSWORD"),\n    email_to=["xxx@example.com"],\n    email_body_fn=my_message_fn,\n    email_subject_fn=lambda _: "Dagster Alert",\n    webserver_base_url="http://mycoolsite.com",\n)\n
\n
\n
\n\n
\n
\nclass dagster._utils.forked_pdb.ForkedPdb(completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False, readrc=True)[source]
\n

A pdb subclass that may be used from a forked multiprocessing child.

\n

Examples:

\n
from dagster._utils.forked_pdb import ForkedPdb\n\n@solid\ndef complex_solid(_):\n    # some complicated stuff\n\n    ForkedPdb().set_trace()\n\n    # some other complicated stuff\n
\n
\n

You can initiate pipeline execution via the webserver and use the pdb debugger to examine/step through\nexecution at the breakpoint.

\n
\n\n
\n","toc":"\n"}},"__N_SSG":true}