A naukri.com initiative
Netflixtechblog
5d
317
Image Credit: Netflixtechblog
Introducing Configurable Metaflow
- Metaflow has introduced a new feature, Configs, to complement the existing Metaflow constructs of artifacts and Parameters, by allowing you to configure all aspects of the flow, decorators in particular, prior to any run starting.
- The new feature allows a flow to be configurable using configuration files, so variants can be defined without changing the code. Configurations can be used more widely in flow code and can be used to set defaults for parameters.
- Metaflow Configs are all stored as artifacts by Metaflow but differ in when they are persisted. Configuration parameters are resolved and persisted at the start of a run, while Metaflow Configs are resolved and persisted when the flow is deployed.
- Configurations can be read from a pleasantly human-readable configuration file and specifications can be made for a triggering ‘@schedule’ and ‘@resource’ requirements.
- Metaflow has always supported sweeping over parameter grids easily using foreaches, but altering the flow itself, for instance, to change @resources or @pypi/@conda dependencies for every experiment, hasn’t been easily possible. Configuration managers like Hydra allow for orchestration of experiments over multiple configurations or sweeping over parameter spaces.
- Metaflow has partnered with Netflix’s Metaboost, an internal Netflix CLI tool that helps ML practitioners manage, develop and execute their cross-platform projects, to provide a single interface to three internal platforms at Netflix that manage ETL/Workflows (Maestro), Machine Learning Pipelines (Metaflow) and Data Warehouse Tables (Kragle).
- Metaboost uses a configuration system that combines GIT-based parameters, global configurations and arbitrarily bound configuration files for use during execution against internal Netflix platforms.
- Metaflow has made it incredibly simple to integrate this configuration system with Configs. Users can simply add a mix-in class to their FlowSpec and reference the configuration values in steps or decorators.
- Configs allow for reproducible, consistent, low-boilerplate and easily configurable experiments and robust production deployments.
- Configs turn into dictionary artifacts, they get versioned and persisted automatically as artifacts. You can access Configs of any past runs easily through the Client API.
Read Full Article
18 Likes
For uninterrupted reading, download the app