hdeeprm.agent module

Agents learn when interacting with the workload management environment, and take care of selecting the scheduling policies. This module defines superclasses for all the agents developed for HDeepRM.

class hdeeprm.agent.Agent

Bases: sphinx.ext.autodoc.importer._MockObject

Superclass for all Agents.

Any agent makes decisions, which are expressed in the decide() method. It uses PyTorch Module as a base class for defining forward passes.

rewards

List of rewards populated during the interaction with the environment. Rewards are feedback on agent’s behaviour.

Type:list

alter(action: int, environment) → None

Alters the environment by applying the action.

An alteration consists of stepping the environment. During this step, job scheduling and resource state changes are issued into Batsim.

Parameters:

• action (int) – Action ID to be applied.
• environment (Environment) – The environment to be altered.

decide(observation: numpy.ndarray) → int

Chooses an action given an observation from the environment.

Chosen actions are given by the action space describe in Environment.

Parameters:observation (ndarray) – The observation as a proxy of the environment’s state. Returns:Action ID to be taken over the environment.

observe(environment) → numpy.ndarray

Observes the current state of the environment.

Through observing, the agent gains information about the environment.

Parameters:environment (Environment) – The environment to be observed. Returns:A NumPy ndarray as an observation.

rewarded(environment) → None

Provides the agent with a reward for its action.

Rewards are provided from the environment, and they are adjusted to the optimization objective.

class hdeeprm.agent.ClassicAgent

Bases: hdeeprm.agent.Agent

Fixed-policy traditional agent.

Represents an agent based on a traditional fixed policy. It can be used as a comparison against deep reinforcement learning agents being developed. May implement any of the policy pairs in the action space.

decide(observation: numpy.ndarray) → int

Returns the fixed action, which is the only one available.

See hdeeprm.agent.Agent.decide().

class hdeeprm.agent.LearningAgent(gamma: float)

Bases: hdeeprm.agent.Agent

Agent which learns based on its actions.

A generic learning agent processes observations through its inner model. At the end of the, the loss given the decision chain is calculated, and utilized for updating the inner model parameters.

gamma

Hyperparameter, user provided. Discount factor for rewards, inbetween [0, 1). When close to 1, rewards from a more distant future will be considered for updating the model and viceversa.

Type:float

forward(observation: numpy.ndarray) → Any

Process the given observation by forwarding through inner model.

The inner model might be a neural network, which forwards the observation through its layers. The result is defined by the structure of last layers in the inner model, and can be defined by the user. Usual cases are policy learning and value learning, see hdeeprm.agent.PolicyLearningAgent.forward() and hdeeprm.agent.ValueLearningAgent.forward().

Parameters:observation (ndarray) – The observation as a proxy of the environment’s state. Returns:Defined by the user’s inner model.

loss() → float

Measures how good the agent has decided based on its rewards and decision outputs.

The loss is used for updating the agent’s inner model weights. Its definition depends on that of the model.

Returns:The loss value for the agent.

transform_rewards() → list

Discount and scale the rewards.

Discounting and scaling of rewards are applied for adjusting the relevance of recency in obtaining them. gamma is used as the discount factor.

Returns:A list with the transformed rewards for further processing.

class hdeeprm.agent.PolicyLearningAgent(gamma: float)

Bases: hdeeprm.agent.LearningAgent

Learns an adaptive policy as a probability distribution over actions.

In contrast with fixed-policy ClassicAgent, this agent learns to select different actions given distinct observations from the environment. This means that job and resource selections may be different depending on the environment state.

probs

First action preferences of the simulation for evolution insights.

Type:list

log_probs

Log probabilities for loss calculation at the end of the simulation.

Type:list

decide(observation: numpy.ndarray) → int

Process the observation and return the action ID.

Parameters:observation (ndarray) – The observation as a proxy of the environment’s state. Returns:Action ID for altering the environment.

forward(observation: numpy.ndarray) → <sphinx.ext.autodoc.importer._MockObject object at 0x7fc9ea51ac88>

Applies the forward pass of the inner model.

Outputs the probability distribution over actions.

Parameters:observation (ndarray) – The observation as a proxy of the environment’s state. Returns:Tensor with the probability distribution over actions.

forward_policy(observation: numpy.ndarray) → <sphinx.ext.autodoc.importer._MockObject object at 0x7fc9de07a320>

Forward pass of the inner model for learning the policy.

When the observation is processed, a probability distribution over the possible actions is given as output. The agent then chooses an action given this distribution, which depends on the observation.

Parameters:observation (ndarray) – The observation as a proxy of the environment’s state. Returns:Tensor with the probability distribution over the possible actions.

get_action(probabilities: <sphinx.ext.autodoc.importer._MockObject object at 0x7fc9de07a358>) → int

Gets an action based on the provided probabilities.

An action is sampled randomly from the probability distribution outputted from the forward pass. The log probability of the action is stored for further calculating the loss.

Parameters:probabilities (Tensor) – Tensor with the probability distribution over actions. Returns:Action ID to be taken over the environment.

loss() → float

Calculates the loss for the policy learning agent.

First it transforms the rewards, which result from the agent’s behaviour during the simulation. It calculates the losses for each step and then it sums them.

Returns:Sum of all the policy losses, which constitute the agent loss for the run.

policy_loss(rews_or_advs: list) → list

Calculates the losses based on the policy learned.

It is calculated based on log losses, and it is negative due to optimizers using gradient descent. For higher rewards, losses are lower.

Parameters:rews_or_advs (list) – List with transformed rewards or advantages (see Actor-Critic example). Returns:List with the policy losses.

save_log_prob(log_prob: <sphinx.ext.autodoc.importer._MockObject object at 0x7fc9de07a390>) → None

Saves a log prob in log_probs.

Parameters:log_prob (Tensor) – Log probability for the selected action to be stored.

class hdeeprm.agent.ValueLearningAgent(gamma: float)

Bases: hdeeprm.agent.LearningAgent

Learns a value estimation for being at a given observation (state).

The value is an approach to estimate the agent’s objective, which is the maximization of the discounted cumulative reward. It indicates the future value of being in the current state (observation). When the value is high, it means that the Agent will advance on its objective.

values

Values for loss calculation at the end of the simulation.

Type:list

forward(observation: numpy.ndarray) → <sphinx.ext.autodoc.importer._MockObject object at 0x7fc9de07a2e8>

Applies the forward pass of the inner model.

Outputs the value estimation of being at that state.

Parameters:observation (ndarray) – The observation as a proxy of the environment’s state. Returns:Tensor with the value estimation.

forward_value(observation: numpy.ndarray) → <sphinx.ext.autodoc.importer._MockObject object at 0x7fc9de07a438>

Forward pass of the inner model for learning the value estimation.

When the observation is processed, a scalar is given as a value estimation of being at that state.

Parameters:observation (ndarray) – The observation as a proxy of the environment’s state. Returns:Tensor with the value estimation for the given observation.

loss() → float

Calculates the loss for the value learning agent.

First it transforms the rewards, which result from the agent’s behaviour during the simulation. It calculates the losses for each step and then it sums them.

Returns:Sum of all the value losses, which constitute the agent loss for the run.

save_value(value: <sphinx.ext.autodoc.importer._MockObject object at 0x7fc9de07a4e0>) → None

Saves a value in values.

Parameters:value (Tensor) – Value for the current observation to be stored.

value_loss(rews: list) → list

Calculates the losses based on the values estimation learned.

It is calculated based on Huber loss. It compares how good of an approximation the agent has provided as expected future reward in comparison with the actual reward.

Parameters:rews (list) – List with transformed rewards. Returns:List with the value losses.