sfcode
An Online Competing and Development Environment
|
Jobs is the Angular DevKit subsystem for scheduling and running generic functions with clearly typed inputs and outputs. A Job
instance is a function associated with metadata. You can schedule a job, synchronize it with other jobs, and use it to schedule other jobs.
The whole API is serializable, allowing you to use a Node Stream or message channel to communicate between the job and the job scheduler.
Jobs are lazy, cold, and guaranteed to execute exactly once when scheduled. Subscribing to a job returns messages from the point where the job is at.
A job receives a single argument when scheduled and can also listen to an input channel. It can emit multiple outputs, and can also provide multiple output channels that emit asynchronous JSON messages, which can be typed.
The I/O model is like that of an executable, where the argument corresponds to arguments on the command line, the input channel to STDIN, the output channel to STDOUT, and the channels would be additional output streams.
A Job
goes through multiple LifeCycle messages before its completion;
JobState.Queued
. The job was queued and is waiting. This is the default state from the scheduler.JobState.Ready
. The job's dependencies (see "Synchronizing and Dependencies") are done running, the argument is validated, and the job is ready to execute.JobState.Started
. The argument has been validated, the job has been called and is running. This is handled by the job itself (or createJobHandler()
).JobState.Ended
. The job has ended and is done running. This is handled by the job itself (or createJobHandler()
).JobState.Errored
. A unrecoverable error happened.Each state (except Queued
) corresponds to a JobOutboundMessage
on the outboundBus
observable that triggers the state change. The Scheduler
emits the Ready
and Errored
messages; the job implementation should not emit them, and if it does they are filtered out. You can listen for these messages or use the corresponding state member.
The job implementation should emit the Start
and End
messages when it is starting the job logic itself. Only the first Start
and End
messages will be forwarded. Any more will be filtered out.
The Queued
state is set as the job is scheduled, so there is no need to listen for the message.
The Job
object that is returned when you schedule a job provides access to the job's status and utilities for tracking and modifying the job.
id
. A unique symbol that can be used as a Map key.description
. The description of the job from the scheduler. See JobDescription
object.argument
. The argument value that was used to start the job.input
. An Observer
that can be used to send validated inputs to the job itself.output
. An Observable<OutputType>
that filters out messages to get only the returned output of a job.promise
. A promise that waits for the last output of a job. Returns the last value outputted (or no value if there's no last value).state
. The current state of the job (see LifeCycle
).channels
. A map of side channels the user can listen to as Observable
.ping()
. A function that can be used to ping the job, receiving a Promise
for when the ping is answered.stop()
. Sends a stop
input to the job, which suggests to stop the job. The job itself can choose to ignore this message.inboundBus
. The raw input Observer<JobInboundMessage>
. This can be used to send messages to the context.inboundBus
observable in the job. These are JobInboundMessage
messages. See "Communicating With Jobs".outboundBus
. The raw output Observable<JobOutput>
. This can be used to listen to messages from the job. See "Communicating With Jobs".The JobHandlerContext<>
is passed to the job handler code in addition to its argument. The context contains the following members:
description
. The description of the job. Its name and schemas.scheduler
. A Scheduler<>
instance that can be used to create additional jobs.dependencies
. A generic list of other job instances that were run as dependencies when scheduling this job. Their id
is not guaranteed to match the id
of the Job<>
instance itself (those Job<>
s might just be proxies). The state of those Job<>
is guaranteed to be JobState.Ended
, as JobState.Errored
would have prevented this handler from running.inboundBus
. The raw input observable, complement of the inboundBus
observer from the Job<>
.An example of a job that adds all input together and return the output value. We use a simple synchronous job registry and a simple job scheduler.
A job is at its core a function with a description object attached to it. The description object stores the JSON schemas used to validate the types of the argument passed in, the input and output values. By default, a job accepts and can output any JSON object.
This is a lot of boilerplate, so we made some helpers to improve readability and manage argument, input and output automatically:
You can also return a Promise or an Observable, as jobs are asynchronous. This helper will set start and end messages appropriately. It will also manage channels automatically (see below).
A more complex job can be declared like this:
Jobs can be started and updated in a separate process or thread, and as such communication with a job should avoid using global objects (which might not be shared). The jobs API and schedulers provide 2 communication streams (one for input and the other for output), named inboundBus
and outboundBus
.
The schedule()
function returns a Job<>
interface that contains a inboundBus
member of type Observer<JobInboundMessage>
. All messages sent to the job goes through this stream. The kind
member of the JobInboundMessage
interface dictates what kind of message it is sending:
JobInboundMessageKind.Ping
. A simple message that should be answered with JobOutboundMessageKind.Pong
when the job is responsive. The id
field of the message should be used when returning Pong
.JobInboundMessageKind.Stop
. The job should be stopped. This is used when cancelling/unsubscribing from the output
(or by calling stop()
). Any inputs or outputs after this message will be ignored.JobInboundMessageKind.Input
is used when sending inputs to a job. These correspond to the next
methods of an Observer
and are reported to the job through its context.input
Observable. There is no way to communicate an error to the job.Using the createJobHandler()
helper, all those messages are automatically handled by the boilerplate code. If you need direct access to raw inputs, you should subscribe to the context.inboundBus
Observable.
The Job<>
interface also contains a outboundBus
member (of type Observable<JobOutboundMessage<O>>
where O
is the typed output of the job) which is the output complement of inboundBus
. All messages sent from the job goes through this stream. The kind
member of the JobOutboundMessage<O>
interface dictates what kind of message it is sending:
JobOutboundMessageKind.Create
. The Job<>
was created, its dependencies are done, and the library is validating Argument and calling the internal job code.JobOutboundMessageKind.Start
. The job code itself should send that message when started. createJobHandler()
will do it automatically.JobOutboundMessageKind.End
. The job has ended. This is done by the job itself and should always be sent when completed. The scheduler will listen to this message to set the state and unblock dependent jobs. createJobHandler()
automatically send this message.JobOutboundMessageKind.Pong
. The job should answer a JobInboundMessageKind.Ping
message with this. Automatically done by createJobHandler()
.JobOutboundMessageKind.Output
. An Output
has been generated by the job.JobOutboundMessageKind.ChannelMessage
, JobOutboundMessageKind.ChannelError
and JobOutboundMessageKind.ChannelComplete
are used for output channels. These correspond to the next
, error
and complete
methods of an Observer
and are available to the callee through the job.channels
map of Observable.Those messages can be accessed directly through the job.outboundBus
member. The job itself should return an Observable<JobOutboundMessage<O>>
. The createJobHandler()
helper handles most of use cases of this and makes it easier for jobs to handle this.
Dispatchers are a helper that redirect to different jobs given conditions. To create a job dispatcher, use the createDispatcher()
function:
Jobs are always run in parallel and will always start, but many helper functions are provided when creating a job to help you control the execution strategy;
serialize()
. Multiple runs of this job will be queued with each others.memoize(replayMessages = false)
will create a job, or reuse the same job when inputs are matching. If the inputs don't match, a new job will be started and its outputs will be stored.These strategies can be used when creating the job:
Strategies can be reused to synchronize between jobs. For example, given jobs jobA
and jobB
, you can reuse the strategy to serialize both jobs together;
Even further, we can have package A and package B run in serialization, and B and C also be serialized. Running A and C will run in parallel, while running B will wait for both A and C to finish.
Jobs can be scheduled using a Scheduler
interface, which contains a schedule()
method:
The scheduler also has a getDescription()
method to get a JobDescription
object for a certain name; that description contains schemas for the argument, input, output, and other channels:
Finally, the scheduler interface has a pause()
method to stop scheduling. This will queue all jobs and wait for the unpause function to be called before unblocking all the jobs scheduled. This does not affect already running jobs.
When scheduling jobs, it is often necessary to run jobs after certain other jobs are finished. This is done through the dependencies
options in the schedule()
method.
These jobs will also be passed to the job being scheduled, through its context. This can be useful if, for example, the output of those jobs are of a known type, or have known side channels.
An example of this would be a compiler that needs to know the output directory of other compilers before it, in a tool chain.
When scheduling jobs, the user can add a dependencies
field to the scheduling options. The scheduler will wait for those dependencies to finish before running the job, and pass those jobs in the context of the job.
Jobs are called with a JobHandlerContext
as a second argument, which contains a dependencies: Job<JsonValue>[]
member which contains all dependencies that were used when scheduling the job. Those aren't fully typed as they are determined by the user, and not the job itself. They also can contain jobs that are not finished, and the job should use the state
member of the job itself before trying to access its content.
The JobHandlerContext
also contains a scheduler
member which can be used to schedule jobs using the same scheduler that was used for the job. This allows jobs to call other jobs and wait for them to end.
The Core Angular DevKit library provides 2 implementations for the Scheduler
interface:
Available in the jobs namespace. A registry that accept job registration, and can also schedule jobs.
Available through @angular-devkit/core/node
.
A scheduler that loads jobs using their node package names. These jobs need to use the createJobHandler()
helper and report their argument/input/output schemas that way.
job.promise
job.promise
waits for the job to ends. Don't rely on it unless you know the job is not watching and running for a long time. If you aren't sure, use job.output.pipe(first()).toPromise()
instead which will return the first next output, regardless of whether the job watches and rerun or not.Serialize Strategy vs Dependencies
Strategies are functions that transform the execution of a job, and can be used when declaring the job, or registering it. Dependencies, on the other hand, are listed when scheduling a job to order jobs during scheduling.
A job has no control over the way it's scheduled, and its dependencies. It can, however, declare that it shouldn't run at the same time as itself. Alternatively, a user could schedule a job twice and imply that the second run should wait for the first to finish. In practice, this would be equivalent to having the job be serialized, but the important detail is in whom is defining the rules; using the serialize()
strategy, the job implementation is, while when using dependencies, the user is.
The user does not need to know how to job needs to synchronize with itself, and the job does not need to know how it synchronizes with other jobs that it doesn't know about. That's part of the strength of this system as every job can be developed in a vacuum, only caring about its contracts (argument, input and output) and its own synchronization.