Storing Service Account JSON keys in plain text is not ideal to say the least. To protect that sensitive information, it is recommended the service account key be stored in the Data Fusion Secure Store to eliminate it from showing up in plain text in pipelines and logs. To recap…

TL;DR Use the Rest API to upload UDD’s across namespaces in a programatic way. While the Data Fusion UI has the ability to do this, it can be problematic when working across many independent namespaces. This allows a developer to quickly push new UDD’s across many namespaces. Artifact API I recently had…

TL;DR The 0.7.0 version of Livy was built for Spark 2.4.5 and Scala 11. Newer versions of Spark build on Scala 12 will not run the current Livy release. To get support for Spark 3.x …

TL;DR Just about anything you can do in the Data Fusion UI you can do with the vast REST interface. In the event you need to update your pipeline by adding the latest plugin release or updating a configuration parameter, you can easily do this through the application update API. Application Update API …

TL;DR Cloud Data Fusion uses macros to enable developers to build configuration driven data pipelines. Macros Macros are variables within Data Fusion that can be set at runtime when the pipeline runs. This allows developers to configure pipelines at runtime instead of hard coding values. If a configuration changes, like moving…

TL;DR The Data Fusion HTTP plugin enables access to web based data sources and can move the data to a variety of downstream data consumers. Developers can use custom pagination functionality to iterate through a list of API calls to enable large scale batch ingest of data. Business Challenge I was recently…

With Data Fusion, pipeline developers can create custom compute profiles to “right size” the Dataproc instance running the pipeline. These profiles are independent of the pipeline itself which enables developers to swap out profiles at runtime depending on how much data being processed. In the image above, a user can…

TL;DR Google Cloud Functions and Google Data Fusion’s REST API can be jointly used to deliver event based execution. Google Data Fusion Pipeline Execution In most typical deployments of Data Fusion, Pipelines are launched using the following three methods. In-built Data Fusion scheduler Google Cloud Composer (Fully managed Apache Airflow) 3rd party scheduler through Data…

TL;DR Google provides pre-built Dataflow templates to accelerate deployment of common data integration patterns in Google Cloud. This enables developers can quickly get started building pipelines without having to build pipelines from scratch. This article examines building a streaming pipeline with Dataflow templates to feed downstream systems. What is Apache Beam? Apache Beam is…