Bug 1289249 - Ignore ENOENT when chowning; r?garndt
When we switched builds to run-task, we started receiving tons of
"random" failures on OS X and some Android build tasks where the
recursive chown was failing to chown() files due to ENOENT that
a directory list had previously said existed. The most likely
explanation for this is that something outside of the task is
deleting files in the cache while the task is running. Another
explanation might be a Docker/AUFS bug.
Until we figure out the root cause, ignore ENOENT errors when
performing the recursive chown. To help us debug the issue, we
log when such an error occurs. If we see warnings for hundreds
or thousands of files in the same directories, that's a good
sign something else is racing to delete the directory tree.
MozReview-Commit-ID: DuYIfYI4VKN
--- a/testing/docker/recipes/run-task
+++ b/testing/docker/recipes/run-task
@@ -215,20 +215,36 @@ def main(args):
for path in args.chown_recursive or []:
print_line(b'chown', b'recursively changing ownership of %s to %s:%s\n' %
(path, user.pw_name, group.gr_name))
set_dir_permissions(path, uid, gid)
for root, dirs, files in os.walk(path):
for d in dirs:
- set_dir_permissions(os.path.join(root, d), uid, gid)
+ # TODO ENOENT is a workaround until we figure out why files
+ # are disappearing.
+ try:
+ set_dir_permissions(os.path.join(root, d), uid, gid)
+ except OSError as e:
+ if e.errno != errno.ENOENT:
+ raise
+ print_line(b'chown', b'warning: ENOENT on %s\n' %
+ os.path.join(root, d))
for f in files:
- os.chown(os.path.join(root, f), uid, gid)
+ # TODO ENOENT is a workaround until we figure out why files
+ # are disappearing.
+ try:
+ os.chown(os.path.join(root, f), uid, gid)
+ except OSError as e:
+ if e.errno != errno.ENOENT:
+ raise
+ print_line(b'chown', b'warning: ENOENT on %s\n' %
+ os.path.join(root, f))
def prepare_checkout_dir(checkout):
if not checkout:
return
# Ensure the directory for the source checkout exists.
try:
os.makedirs(os.path.dirname(checkout))