Cloud Data Fusion: Adding a Service Account to the Secure Store
Storing Service Account JSON keys in plain text is not ideal to say the least. To protect that sensitive information, it is recommended the service account key be stored in the Data Fusion Secure Store to eliminate it from showing up in plain text in pipelines and logs. To recap what a secure store is and the value it provides, please view the link below to get acquainted with this important part of the Data Fusion stack. The rest of this article will lean heavily on these concepts explored in this prior article.
Adding a Service Account
Adding a service account key can be a little bit tricky compared to a simple username or password. The key itself is a large JSON record and care needs to be taken in order to format it properly to the Secure Store. Here is a dummy key for this example.
{
"type": "service_account",
"project_id": "my-project",
"private_key_id": "my private key 123456",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvAIDFDFDFBADANBgkqhkiG9wQUITLOOKINGATMYKEYAoIBAQDYKWZDFDFDFDFDFDzPOkGcF\n7HuVC7WYDFDze3rBmAA4A6thinkagainthisisafakekey2sQz0/JnIZrdOd/\nEgpeFlDFEirQADFDvlzG__https://www.youtube.com/watch?v=dQw4w9WgXcQ__r3lLA\nEvf1p6tcn0qoareyoustillreadingthisVuPNy9X2sA3MYSpY\nBaGsFhrLDFDFEFL70o3Ri8kNseriouslyyoushouldstops37sWR1ouWUjfZinhnijxdmv\nQDup0PCUz2q3NsRNRJEYyjifstillherehuhRal8VAi88Fbs1psXcqR4iK\nIU1Lpzwa+deA3PEKh0GNZhellowworlduyYqG3WkH5WFo\n193VkUgS+l9UfbzcC/Dks4IliketurtlesF8atqHIWAp5IiXi\nvV/4II9jLkbDp1yiUaEbga0=DXg6j\n/MDLsrOji9BRVdGbdaUimBr22GGJktQI71IJb++cAPwBY/Ak2\norY47u008XhYBecVO+ixy4RYfvzBd/oAQxwvSaLlWQKBgQDuM5sCLBVMpWfd5rju\nm0Zd0w2GhyQ6gxoAI326JnKPPFjCbe+m3HWvUzN3jwAZFdPXT\n/ZQqSSV/qAq6lbzZzVxsAJ4MjJpcIslGPMIXYoyWIe1j3Bli5uv778EJCDxLcADA\nsr5YaMdoumL7lRMyNWtKvj1rBQKBgQDoUDWX3oM4HKM6rWZGlnn0FKlgqVKrTtKz\nkXQ/+f/sXlwbDpxhW1vuLUwRt3rLNEPwmJGimBvrSZRrzYV8PIohVq9WeE\ne6zPOIN5E6maAPduU3Q02t/Tj6d0LTGWnY9ZIQptrdFEAj6TcZ+fIf\ni7c9sigBB6tYCi+B5DWzo4hmH44vtvcYqh9d/h2mMEhOLEaSnl5W\neHWLEYdfYPDKWszYvbhUbSVrZ7K9OMDqKUPbVCNlV72JFWYc3Bj/txkK\n3wqAUZoL3fSmWLklBNwV+TfiOzrAgMzQhGuftBQtj67FjZAoGAQ3po\n1sAPJVFMKFMNjhtPx3uf8z4vLcdj\nJfNAyMsKj4EFASDFvR/8WOQjSOXruw\n+d3v9NMGyHorASDFSFAGFADSFAcDV+xd2rhmBk\nQqYflEiit9Pc6vO5ASDFADF3333l7dV7GBWN30S7+MaFE2SI/V\ncLsuOT+452bXl/51+W2X1ascpLV9\nsQRfWwlogctcg8C2QWK/rg==\n-----END PRIVATE KEY-----\n",
"client_email": "admin@allenparsensproject.iam.gserviceaccount.com",
"client_id": "my-client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/notmyproject.iam.gserviceaccount.com"
}To add anything to the secure store it must be added to the following template:
{
"description": "my service account key description",
"data": "where your service account json key goes",
"properties": {
"property-key": "service account demo key property"}
}The data key value pair is where the actual service account JSON will go. The property key value pair is additional metadata you can provide to add some context to the key, like the system it belongs to or the team that owns it.
In order to add the service account key to this template I typically format my service account key so that it all is on one line. You’ll also have to format the service account key so that all double quotes are escaped with \ . The other formatting that needs to be done is with newline characters in the private key. Those will also need to be escaped as well. Below is my example key properly formatted.
{
"description": "my fake key",
"data": ""{\"type\": \"service_account\",\"project_id\": \"my-project\",\"private_key_id\": \"my private key 123456\",\"private_key\": \"-----BEGIN PRIVATE KEY-----\\nMIIEvAIDFDFDFBADANBgkqhkiG9wQUITLOOKINGATMYKEYAoIBAQDYKWZDFDFDFDFDFDzPOkGcF\\n7HuVC7WYDFDze3rBmAA4A6thinkagainthisisafakekey2sQz0/JnIZrdOd/\\nEgpeFlDFEirQADFDvlzG__https://www.youtube.com/watch?v=dQw4w9WgXcQ__r3lLA\\nEvf1p6tcn0qoareyoustillreadingthisVuPNy9X2sA3MYSpY\\nBaGsFhrLDFDFEFL70o3Ri8kNseriouslyyoushouldstops37sWR1ouWUjfZinhnijxdmv\\nQDup0PCUz2q3NsRNRJEYyjifstillherehuhRal8VA1psXcqR4iK\\nIU1Lpzwa+deA3PEKh0GNZhellowworlduyYqG3WkH5WFo\\n193VkUgS+l9UfbzcC/Dks4IliketurtlesF8atqHIWAp5IiXi\\nvV/4II9jLkbDp1yiUaEbga0=DXg6j\\n/MDLsrOji9BR2GGJktQI71IJb++cAPwBY/Ak2\\norY47u008XhYBecVO+ixy4RYfvzBd/oAQxwvQDuM5sCLBVMpWfd5rju\\nm0Zd0w2ubWBB7VLGhyQ6gxoAI326JnKPPFjCzN3jwAZFdPXT\\n/ZQqSSV/qAq6lbzZzVxsAJ4MjJpcIslGPMIXYoyWIe1j3Bli5uv778EJCDxLcADA\\nsr5YaMdoumL7lRMyNWtKvj1rBQKBgQDoUDWX3oM4HKM6rWZGlnn0FKlgqVKrTtKz\\nkXQ/+f/sXlwbDpxhW1vuLUwRt3rLNEPwmJGimBvrSZRrzYVswF8N78PIohVq9WeE\\ne6zPOINAPduU3Q02t/Tj6d0LTGWYO5E8qFO2LnY9ZIQptrdFEAj6TcZ+fIf\\ni7c9siiijwKBgBB6tvcYqh9d/h2mMEhOLEaSnl5W\\neHWLEYdfYPDKWszYVjt0qO02vbV72JFWYc3Bj/txkK\\n3wqrILrKQSmWLklBNwV+TfiOzrAgMzQhGuftBQtj67FjZAoGAQ3po\\n1sAPJVFMKFMNjhtPORJ3UFheZtvyXFltnz5x3uf8z4vLcdj\\nJfNAyMsKj4EFASDFvR/8WOQjSOXruw\\n+d3v9NMGyHorASDFASDFADSFADSFAGFADSFAcDV+xd2rhmBk\\nQqYflEiit9Pc6vO5ASDFADF3333l7dV7GBWN30S7+MaFE2SI/V\\ncLsuOT+452bXl/51+W2X1ascpLV9\\nsQRfWwlogctcg8C2QWK/rg==\\n-----END PRIVATE KEY-----\\n\",\"client_email\": \"admin@allenparsensproject.iam.gserviceaccount.com\",\"client_id\": \"my-client-id\",\"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\"token_uri\": \"https://oauth2.googleapis.com/token\",\"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/notmyproject.iam.gserviceaccount.com\"}",
"properties": {
"property-key": "demo sa key"}
}If everything is formatted properly, the API will accept the submission of the template. In this example, I named the key sakey1234. The name for the key was provided as part of the API call.
To check that they key was formatted correctly, submit an API call to read the key.
Note that the response should look identical to your originally unformatted service account JSON. The big check here is that the private key should be all on one line. If the key is across multiple rows then the escape formatting is incorrect and you need to delete the key and resubmit.
Next Steps
With the key successfully uploaded, it can now be referenced in pipelines as well as compute profiles without having to explicitly list the key. In the example image below, the Shield Icon on some of the parameter fields allow you to select the key you wish to use. In other scenarios, just reference the key with in the secure macro format ${secure(my key name)}.
As a best practice, use secure keys for encrypting and securing your credentials you use in Data Fusion. Try to adopt this practice even in development to help protect your environments.
Below are some additional links to follow on Data Fusion Secure Store.