A naukri.com initiative
Hackernoon
4d
370
Image Credit: Hackernoon
The $0 Scheduler That Almost Cost a Compny Everything
- A company faced severe CPU load spikes and system crashes during task triggers when migrating to DolphinScheduler.
- The issue was found to be related to improper scheduler configuration, not the number of tasks.
- They balanced machine load and task demands by adjusting thread settings in DolphinScheduler.
- The company transitioned from Alibaba Cloud DataWorks to a self-built big data platform due to budget constraints.
- Initially using 4 ECS servers for the platform, resources were gradually transitioned from DataWorks to DolphinScheduler.
- High CPU load spikes occurred during task scheduling intervals, causing system instability and crashes.
- Investigations revealed that adjusting thread settings and CPU limits in DolphinScheduler resolved the CPU spike problem.
- Configuration adjustments were made in DolphinScheduler's master and worker components to ensure smoother task execution.
- JVM memory settings were also optimized in the start.sh scripts for both master and worker components.
- Balancing thread counts and CPU limits helped stabilize CPU usage during task triggers and prevent system crashes.
Read Full Article
22 Likes
For uninterrupted reading, download the app