Installing BigQuery clients¶
To interact with BigQuery from Python, install the google-cloud-bigquery library:
$ pip install google-cloud-bigquery
Typically, these credentials will either be per user or they will correspond to a service account (a system user who only has minimal permissions on the project):
- If you have your own credentials, we suggest storing these in your
home directory under
/home/faculty. For a description of how your home directory persists across servers, read the Your home directory section of the documentation.
- If the credentials belong to a service account linked to the project, we suggest storing the credentials in the project workspace so that everyone in the project can access them.
Once you have settled on a location for the credentials, make sure the
BigQuery client knows how to find them by setting the
GOOGLE_APPLICATION_CREDENTIALS environment variable. Run the
following commands, replacing
/path/to/credentials.json with the
absolute path to the credentials:
echo "export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json" > /etc/faculty_environment.d/gcloud-credentials.sh sudo sv restart jupyter # Restart Jupyter to make sure it has access to credentials.
To make setting up servers more reproducible, we recommend adding these commands to the scripts section of a custom environment. For further information on setting environment variables in Faculty, refer to the Environment variables section.
Accessing BigQuery from Python¶
from google.cloud import bigquery client = bigquery.Client() query = "SELECT * FROM `bigquery-public-data.london_bicycles.cycle_hire` LIMIT 10" df = client.query(query).to_dataframe()
This returns the result of the query as a Pandas dataframe.
Refer to the BigQuery documentation for other examples.