Batch Processing
Batch processing in MediaX SDK is handled through the Queue class, providing a mechanism to run multiple Job instances in parallel with configurable concurrency and robust error isolation.
Queue Class
Concurrent Job Processing
Manages multiple media processing jobs with configurable parallelism
Constructor
1
Parameters:
Methods
add()
add(job: Job): void
Add a single job to the queue for processing
queue.add(job);
on()
on(event: string, handler: Function): void
Register event listeners for job lifecycle and queue state
queue.on("jobStart", (job) => console.log("Started:", job));
Events
The queue emits the following events for comprehensive job lifecycle monitoring:
Usage Examples
Basic Queueing
123456789101112131415161718192021
Batching Multiple Jobs
12345678910111213
Note: All jobs will be executed with the specified concurrency limit until the queue is empty. Perfect for processing large batches of media files.
Batching Pipeline Jobs
Important: Queue does not directly accept Pipeline objects. However, since Pipeline.run() returns Jobs, those can be queued manually.
123456789101112
Queue Behavior
FIFO Processing
Jobs execute in First-In-First-Out order respecting concurrency limits
Slot Management
Completed jobs free up slots for queued jobs automatically
Error Isolation
Errors in one job do not stop the queue; subsequent jobs continue
Completion Detection
Queue is considered finished when all jobs complete and empty event fires
Key Characteristics:
- Jobs execute in FIFO order respecting concurrency limits
- Completed jobs automatically free up slots for queued jobs
- Errors in one job do not stop the queue; subsequent jobs continue
- Queue is finished when all jobs complete and empty event fires
Advanced Queue Management
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859
Performance Considerations
Optimal Concurrency
Set concurrency based on your system capabilities:
- • CPU-intensive tasks: Number of CPU cores
- • I/O-intensive tasks: 2-4x CPU cores
- • Mixed workloads: Start with 2-3, monitor and adjust
Memory Management
Large video files and high concurrency can consume significant memory. Monitor system resources and adjust concurrency accordingly.
Timeout Settings
Set appropriate timeouts for different job types. Video processing can take significantly longer than audio extraction.
Scale Your Media Processing
With batch processing mastered, explore other API references to build comprehensive media processing workflows with MediaX SDK.