Use a pool for dart actions to avoid OOMs #27781
Conversation
flutter-dashboard
bot
added
the
platform-web
| @@ -0,0 +1,102 @@ | |||
| #!/usr/bin/env python3 | |||
| # Copyright 2019 The Fuchsia Authors. All rights reserved. | |||
There was a problem hiding this comment.
Is this copyright script meant to remain as-is? Presumably, this script is from the Fuchsia buildroot then?
There was a problem hiding this comment.
I see, it's this but with added Windows support. I'll review the delta.
There was a problem hiding this comment.
Yeah, it's from the Fuchsia buildroot. It looks like for other files from the Fuchsia buildroot we're leaving the Fuchsia copyright header.
build/get_concurrent_jobs.py
Outdated
| parser = argparse.ArgumentParser() | ||
| parser.add_argument( | ||
| "--memory-per-job", action=ParseSizeAction, default=[], nargs='*') | ||
| parser.add_argument("--reserve-memory", type=ParseSize, default=0) |
There was a problem hiding this comment.
I am not able to tell what the reserve memory flag is meant to do by just reading the flag name. Reserve memory for what? Perhaps add a help= argument to the parser? From my reading of the code, something like "Amount of total memory to reserve for activity unrelated to all jobs in the pool"? Come to think of it, since the caller doesn't know what the total memory is, perhaps this argument should be some percent of total memory? Say, something like 10% off the top.
There was a problem hiding this comment.
Added help text. Whether a percent or an absolute value, it's a bit of a blind guess at how much the system needs to keep for itself to avoid thrashing. I'd actually be more worried about a percent as far as that goes. On a small machine it might not be enough, and on a big machine it will be too much.
| _args = [ | ||
| "--reserve-memory=1GB", | ||
| "--memory-per-job", | ||
| "dart=1GB", |
There was a problem hiding this comment.
I was trying to figure out why memory-per-job was a key value pair but it looks like you are going to extend this for other things as well. Perhaps something for the linker when LTO is enabled maybe?
There was a problem hiding this comment.
Yeah, the Chromium and Fuchsia builds use this in LTO builds. We should probably consider doing that as well after profiling.
| parser.add_argument("--reserve-memory", type=ParseSize, default=0) | ||
| args = parser.parse_args() | ||
|
|
||
| total_memory = GetTotalMemory() |
There was a problem hiding this comment.
If total_memory is zero (as it could be in the error cases), perhaps just quit and return a default json with job counts of 1? I am not sure how robust GetTotalMemory is to error, or (for that matter) GN is to zero job counts.
* Use a pool for dart actions to avoid OOMs * Add Windows support
* Use a pool for dart actions to avoid OOMs * Add Windows support
Limits the level of concurrency of actions that run Dart programs during the build. This is to avoid e.g. too many invocations of the front end running at the same time and exhausting system memory, which results in build failures.
FIxes flutter/flutter#86946