aki

chain of trust

As I mentioned before, scriptworkers allow for more control and auditability around sensitive release-oriented tasks in Taskcluster. The Chain of Trust allows us to trace requests back to the tree and verify each previous task in the chain.

We have been generating Chain of Trust artifacts for a while now. These are gpg-signed json blobs with the task definition, artifact shas, and other information needed to verify the task and follow the chain back to the tree. However, nothing has been verifying these artifacts until now.

With the latest scriptworker changes, scriptworker follows and verifies the chain of trust before proceeding with its task. If there is any discrepancy in the verification step, it marks the task invalid before proceeding further. This is effectively a second factor to verify task request authenticity.

scriptworker 1.0.0b1

1.0.0b1 is largely two pull requests: scriptworker.yaml, which allows for more complex, commented config, and chain of trust verification, which grew a little large (275k patch !).

This is running on signing scriptworkers which sign nightlies on date-branch. We still need to support and update the other scriptworker instance types to enable end-to-end chain of trust verification.

Tl;dr: I shipped scriptworker 0.8.2 (changelog) (github) (pypi) and scriptworker 0.7.2 (changelog) (github) (pypi) last Monday (Oct 24), and am only now getting around to blogging about them.

These are patch releases, and fix the polling loop.

scriptworker 0.8.2

The fix for bug 1310120 - puppet reinstalls scriptworker on every run exposed some scriptworker loop bugs: because puppet was restarting scriptworker regularly, we never had a long-running instance before.

:kmoir and :Callek noticed that signing scriptworker wasn't signing (nagios alerts on stuck queues are pending =\ ). It was clear that while git polling was continuing on its merry way, the task polling was dying fairly quickly. We also needed more logging around fatal exceptions.

I addressed these issues in scriptworker 0.8.2. We also have our third scriptworker contributor, :jlorenzo! (:jlund was #2)

scriptworker 0.7.2

Since we already had the 0.7.1 release to help other scriptworker instance types from having to deal with the churn from pre-1.0 changes, I backported the 0.8.2 fixes to the 0.7.x branch and released 0.7.2 off of it. This involved enough merge conflicts that I'm hoping to avoid too many more 0.7.x releases.

Tl;dr: I just shipped scriptworker 0.8.0 (changelog) (RTD) (github) (pypi).
This is a non-backwards-compatible release.

background

By design, taskcluster workers are very flexible and user-input-driven. This allows us to put CI task logic in-tree, which means developers can modify that logic as part of a try push or a code commit. This allows for a smoother, self-serve CI workflow that can ride the trains like any other change.

However, a secure release workflow requires certain tasks to be less permissive and more auditable. If the logic behind code signing or pushing updates to our users is purely in-tree, and the related checks and balances are also in-tree, the possibility of a malicious or accidental change being pushed live increases.

Enter scriptworker. Scriptworker is a limited-purpose taskcluster worker type: each instance can only perform one type of task, and validates its restricted inputs before launching any task logic. The scriptworker instances are maintained by Release Engineering, rather than the Taskcluster team. This separates roles between teams, which limits damage should any one user's credentials become compromised.

scriptworker 0.8.0

The past several releases have included changes involving the chain of trust. Scriptworker 0.8.0 is the first release that enables gpg key management and chain of trust signing.

An upcoming scriptworker release will enable upstream chain of trust validation. Once enabled, scriptworker will fail fast on any task or graph that doesn't pass the validation tests.