aoutools (workbench)

Helpers for setting up a Hail session on the All of Us Researcher Workbench. None of these is required to use the rest of the library – they exist so that the Workbench-specific boilerplate (requester-pays billing, the reference genome, the Google billing project, and the locations of the VariantDataset and the workspace bucket) lives in one place rather than being copied into every notebook.

Note

The current All of Us platform (Verily) no longer exports the WGS_VDS_PATH and WORKSPACE_BUCKET environment variables that the older Workbench guaranteed. Code that reads them directly fails on a fresh workspace. These helpers resolve both without them.

Exporting a variable from a Jupyter terminal does not reach the notebook: the terminal and the kernel are sibling children of the Jupyter server, which captured its environment at startup, so the kernel never sees it – even after a restart. Set it from inside a notebook cell instead, or put it in ~/.ipython/profile_default/startup/00-aou-env.py.

aoutools.init_hail(reference='GRCh38', **kwargs)[source]

Initializes Hail for the All of Us Researcher Workbench.

Wraps hl.init with the two settings the Workbench needs:

  • Requester-pays billing. The bucket holding the VDS charges the reader, so Hail must be given a project to bill; without one, reading the VDS fails. The project is resolved by get_google_project, which checks GOOGLE_PROJECT, then GOOGLE_CLOUD_PROJECT, then the Workbench CLI – the older Workbench exported the first, the current (Verily) platform uses the second, and a clean image may set neither.

  • The reference genome. hl.init(default_reference=…) is deprecated; the reference is now set by calling hl.default_reference with an argument, after initialization. Doing it here keeps the deprecation warning out of your notebook.

Calling this off the Workbench, where no project can be found, is fine: the requester-pays setting is simply omitted, with a warning.

By default this function is safe to run more than once: re-running the cell that calls it will not raise an error.

Parameters:

  • reference (str, default="GRCh38") – The default reference genome. All of Us WGS data is GRCh38.

  • **kwargs – Passed through to hl.init. An explicit gcs_requester_pays_configuration or idempotent overrides the default set here.

Return type:

None

Examples

>>> from aoutools import init_hail
>>> init_hail()
aoutools.get_vds_path(path=None)[source]

Resolves the path to the All of Us WGS VariantDataset.

Checks three sources, in order, and returns the first one that is set:

  1. The path argument, if given.

  2. The WGS_VDS_PATH environment variable, exported by some Workbench images.

  3. DEFAULT_VDS_PATH, the current All of Us data release. Using this fallback emits a UserWarning, because it is pinned to a data version that this package cannot verify against your workspace.

Parameters:

path (str, optional) – An explicit path, which overrides both other sources. Passing the path you already have is always allowed, so a caller never has to branch on whether the environment happens to be configured.

Returns:

A GCS path to the VDS, suitable for hl.vds.read_vds.

Return type:

str

Examples

>>> import hail as hl
>>> from aoutools import get_vds_path, init_hail
>>> init_hail()
>>> vds = hl.vds.read_vds(get_vds_path())
aoutools.get_workspace_bucket(bucket=None)[source]

Resolves the GCS path of the workspace bucket, e.g. gs://my-bucket.

Checks four sources, in order, and returns the first that yields an answer:

  1. The bucket argument, if given.

  2. The WORKSPACE_BUCKET environment variable.

  3. The Verily Workbench CLI (wb resource list), which reports the workspace’s declared resources. This is the authoritative source, and it is what Verily’s own setup notebook uses.

  4. The gcsfuse mount table, as a fallback if the CLI is unavailable.

Why not just tell users to export the variable: exporting it from a Jupyter terminal does not work. The terminal and the kernel are sibling children of the Jupyter server, which captured its environment at startup, so the kernel never sees a variable exported later in a terminal – and restarting the kernel re-inherits that same, unchanged environment.

To set it from inside a notebook (which does work, since it mutates the kernel’s own environment):

import os
os.environ["WORKSPACE_BUCKET"] = "gs://your-bucket"

To make that persist across kernel restarts, drop the same two lines into ~/.ipython/profile_default/startup/00-aou-env.py, which every kernel runs at startup.

Notes

Passing bucket explicitly is the most predictable option: it skips every detection step above – no environment variable, no Workbench CLI, no mount table – so the result is exactly the value you give (with a gs:// prefix added if missing). Prefer it for a script you want to behave the same on every run. It is only as correct as the value you pass, though, so confirm the bucket exists before a long job writes into it; explicit passing removes the risk of the wrong bucket being guessed, not the risk of a typo.

A workspace commonly has more than one bucket, and picking the wrong one does not fail – the results simply are not where you think they are. Two in particular are excluded here:

  • the temporary workspace bucket, whose resource id also contains “workspace-bucket” and whose contents are garbage-collected;

  • a cloned workspace’s notebooks bucket (e.g. aou-tutorial-notebooks), which is fuse-mounted alongside your own.

Parameters:

bucket (str, optional) – An explicit bucket, with or without the gs:// prefix. Overrides every other source.

Returns:

The bucket as a GCS path, with no trailing slash (e.g. gs://my-bucket).

Return type:

str

Raises:

OSError – If the bucket cannot be determined, or if several are mounted and none can be identified as the workspace bucket. It will not guess.

aoutools.get_google_project(project=None)[source]

Resolves the Google project to bill requester-pays reads to.

Checks, in order: the project argument, $GOOGLE_PROJECT, $GOOGLE_CLOUD_PROJECT, and finally the Workbench CLI (wb workspace describe).

Two variable names are checked because the platforms disagree. The older All of Us Workbench exported GOOGLE_PROJECT; Verily’s setup notebook sets GOOGLE_CLOUD_PROJECT and leaves GOOGLE_PROJECT empty. Reading only the first meant a clean Verily image silently got no billing project at all, and every read of the requester-pays VDS bucket failed.

Parameters:

project (str, optional) – An explicit project, which overrides every other source.

Returns:

The project id, or None if no source knows one (e.g. off the Workbench).

Return type:

str or None

aoutools.DEFAULT_VDS_PATH

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.