In my research on intelligent energy management for building clusters equipped with photovoltaic (PV) systems and battery energy storage system (BESS), I have focused on developing a flexible and robust scheduling framework. The core challenge lies in coordinating multiple BESS units under uncertainty from renewable generation and load variations. Traditional methods often suffer from poor adaptability or high computational complexity. To overcome these issues, I propose a novel approach named Predictive-Guided Attention-based Deep Deterministic Policy Gradient (PGADDPG). This method integrates rolling prediction, attention-based communication, and hierarchical control to enhance the performance of the battery energy storage system in microgrid environments.
System Model and Problem Formulation
I consider a microgrid consisting of multiple residential buildings, each equipped with a PV panel and a battery energy storage system (BESS). The net power exchange with the external grid at time step \( t \) is defined as \( P_t^G = P_t^{G2MG} – P_t^{MG2G} \), where purchasing and selling cannot occur simultaneously. The PV generation \( P_t^{PV} \) first satisfies local load \( P_t^{Load} \), and the remaining power charges or discharges the BESS. The state of charge (SOC) dynamics are governed by:
$$
SOC_t = \frac{E_t}{Q}
$$
$$
E_{t+1} = (1 – \eta_{sd}) E_t + \left( \eta_{c,t} P_{c,t} + \frac{P_{d,t}}{\eta_{d,t}} \right) \Delta t
$$
where \( E_t \) is the stored energy, \( Q \) is the rated capacity, \( \eta_{sd} \) is the self-discharge rate, \( \eta_{c,t} \) and \( \eta_{d,t} \) are dynamic charging and discharging efficiencies that I model using piecewise linear interpolation. The battery energy storage system (BESS) must operate within safe SOC bounds: \( SOC_{\min} \le SOC_t \le SOC_{\max} \). Table 1 summarizes the BESS parameters used throughout my experiments.
| Parameter | Value |
|---|---|
| Capacity (kWh) | 11.0 |
| Max power (kW) | 6 |
| Max SOC | 91% |
| Min SOC | 5% |
| Initial charge/discharge efficiency | 94% |
| Self-discharge rate | 0.001% |
| Efficiency interpolation intervals (power) | [[0, 0.85], [0.6, 0.94], [1, 0.85]] |
| Efficiency interpolation intervals (energy) | [[0.0, 1], [0.7, 1], [1.0, 0.2]] |
The time-of-use (TOU) electricity tariff adopted in my simulation is given in Table 2. I use these price signals to design a reward function that encourages the battery energy storage system (BESS) to charge during low-price periods and discharge during peak price periods.
| Time Slot | Jun–Sep Weekday | Jun–Sep Weekend | Oct–May Weekday | Oct–May Weekend |
|---|---|---|---|---|
| 8 AM – 4 PM | 0.22 | 0.22 | 0.21 | 0.21 |
| 4 PM – 9 PM | 0.54 | 0.40 | 0.50 | 0.50 |
| 9 PM – 8 AM | 0.22 | 0.22 | 0.21 | 0.21 |
Reinforcement Learning Framework
I formulate the BESS scheduling problem as a Markov decision process (MDP). At each time step \( t \), the agent observes the state \( s_t \) (including date, temperature, humidity, irradiance, load, PV output, SOC, net electricity cost, and electricity price) and selects a continuous action \( a_t \in [SOC_{\min}, SOC_{\max}] \) representing the target SOC. The environment transitions to a new state and returns a reward. The goal is to maximize the cumulative discounted reward.
To train multiple agents, each controlling its own battery energy storage system (BESS), I adopt the Deep Deterministic Policy Gradient (DDPG) algorithm. Each agent has its own actor network \( \mu_{\theta} \) and critic network \( Q_{\phi} \). The actor outputs the action, and the critic estimates the Q-value. The policy gradient update for the actor is:
$$
\nabla_{\theta} J(\theta) \approx \mathbb{E}_{s \sim \rho^{\mu}} \left[ \nabla_{\theta} \mu_{\theta}(s) \nabla_{a} Q_{\phi}(s, a) \big|_{a = \mu_{\theta}(s)} \right]
$$
The critic is updated by minimizing the temporal-difference (TD) error:
$$
L(\phi) = \mathbb{E}_{(s,a,r,s’) \sim D} \left[ \left( Q_{\phi}(s, a) – y \right)^2 \right], \quad y = r + \gamma Q_{\phi’}(s’, \mu_{\theta’}(s’))
$$
Attention-Based Multi-Agent Communication
To improve coordination among multiple battery energy storage systems (BESS), I equip each agent with an attention mechanism that selectively shares information from other agents. For agent \( i \), the attention output is computed as:
$$
\text{out}_i = \text{attention}(o_i, \{o_j, a_j\}_{j \neq i})
$$
where \( o_i \) is the local observation of agent \( i \), and \( o_j, a_j \) are the observation and action of other agents. The attention weights are learned jointly with the actor-critic network. The combined input to the actor and critic is \( [o_i, \text{out}_i] \). This allows each agent to consider the collective impact of its decisions on the overall microgrid performance.
Multi-Objective Hybrid Reward Function
I design a hybrid reward function to guide the learning of the battery energy storage system (BESS) agents toward multiple objectives: minimizing electricity cost, carbon emissions, and peak load fluctuations. The base reward for building \( b \) at time \( t \) is:
$$
X_0(b,t) = -1.4 + 1.2 \times \text{sign}\big(C_e(b,t)\big) \times SOC(b,t)
$$
$$
R_c(b,t) = X_0(b,t) \times \left( C_e(b,t) + C_c(b,t) \right) + \sqrt{ \left( \sum_{b=1}^N C_e(b,t) \right)^2 + \left( \sum_{b=1}^N C_c(b,t) \right)^2 }
$$
$$
R(b,t) = R_c(b,t) + \sum_{i=1}^4 w_i X_i
$$
Here, \( C_e(b,t) \) and \( C_c(b,t) \) are the electricity cost and carbon cost, respectively. The additional penalty terms \( X_i \) discourage undesirable behaviors such as insufficient charging before peak hours or excessive discharging during peak hours. The weights \( w_i \) are set to 0.1 for \( i=1,2,3 \) and 0.2 for \( i=4 \).
Rolling Prediction and Hierarchical Control
To provide a strong initial policy and accelerate training, I develop a two-step rolling prediction module. For load forecasting, I use an adaptive cyclic mean model combined with linear regression. The forecast for the next 24 hours is given by:
$$
\hat{y}_t = w_0 \hat{y}_t^0 + w_1 \hat{y}_t^1 + w_2 \hat{y}_t^2 + b
$$
where \( \hat{y}_t^0 \), \( \hat{y}_t^1 \), \( \hat{y}_t^2 \) are the daily, weekly, and monthly cyclic mean predictions, and \( w_0, w_1, w_2, b \) are updated every 72 hours using gradient descent. For PV forecasting, I employ a Self-Attention Bidirectional LSTM (SA-BiLSTM) model that captures temporal dependencies and feature correlations. The attention mechanism computes:
$$
e_{ij} = \frac{q_i^T k_j}{\sqrt{d_k}}, \quad \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^n \exp(e_{ik})}, \quad c_i = \sum_{j=1}^n \alpha_{ij} v_j
$$
where \( q_i, k_j, v_j \) are the query, key, and value from the BiLSTM outputs. The final output is \( \text{out}_i = W_o [q_i, c_i] + b_o \).
Table 3 and Table 4 show the prediction errors for PV and load, respectively. The SA-BiLSTM significantly outperforms standard LSTM, especially for 24-hour PV forecasts.
| Metric | SA-BiLSTM (Building 1) | SA-BiLSTM (Building 2) | LSTM (Building 1) | LSTM (Building 2) |
|---|---|---|---|---|
| 1-hour MSE | 0.034 | 0.018 | 0.217 | 0.270 |
| 1-hour MAE | 0.041 | 0.064 | 0.200 | 0.215 |
| 24-hour MSE | 0.046 | 0.041 | 0.207 | 0.267 |
| 24-hour MAE | 0.072 | 0.049 | 0.193 | 0.212 |
| Metric | Adaptive Cyclic Mean (Building 1) | Adaptive Cyclic Mean (Building 2) | Day-ahead Persistence (Building 1) | Day-ahead Persistence (Building 2) |
|---|---|---|---|---|
| 1-hour MSE | 0.332 | 0.464 | 0.624 | 0.983 |
| 1-hour MAE | 0.449 | 0.583 | 0.529 | 0.619 |
| 24-hour MSE | 0.477 | 0.829 | 0.654 | 1.159 |
| 24-hour MAE | 0.504 | 0.630 | 0.530 | 0.620 |
Based on the predicted PV and load, I design a hierarchical control strategy. The upper layer ensures that the SOC before peak hours meets a minimum requirement \( SOC_{\min}^{\text{peak}} \). If the predicted surplus PV before the peak is insufficient, the battery energy storage system (BESS) is forced to charge. The lower layer optimizes surplus energy usage by adjusting the action \( a_t = R_{t+1}^{pv} / Q \) where \( R_{t+1}^{pv} = E_{t+1}^{pv} – E_{t+1}^{\text{load}} \), with constraints to prioritize the upper layer.
Predictive-Guided Policy and Entropy Regularization
To combine the strengths of the learned DDPG policy and the hierarchical rule-based policy, I introduce a dynamic fusion mechanism. The final action \( a \) is computed as:
$$
a = w_{rl} \cdot a_{rl} + w_{bf} \cdot a_{bf}
$$
where \( w_{bf} = 1 / (1 + e^{-(diff – t) \cdot \lambda(n) }) \), with \( diff = |a_{rl} – a_{bf}| \). The weight \( \lambda(n) = \max(0, 1 – (eps – n) / (total\_eps – n)) \) decreases linearly over the first \( n \) episodes (set to 15). After \( n \) episodes, \( \lambda(n) = 0 \) and \( w_{bf} = 0 \), leaving only the DDPG policy. Additionally, I use importance sampling and entropy regularization to prevent the policy from overfitting to the guiding policy. The actor loss becomes:
$$
J(\theta) \propto -\sum \left( \text{normw} \cdot \text{Critic} \right) + \varepsilon \cdot \text{entropy}
$$
where \( \varepsilon = 0.15 \) is the entropy coefficient.
Simulation Results and Analysis
I evaluate four variants: Independent DDPG (IDDPG), Attention-based IDDPG (AIDDPG), Predictive-Guided DDPG without attention (PGDDPG), and the full PGADDPG. All models are trained for 20 episodes. Figure 5 (not shown here) indicates that PGADDPG achieves the highest final reward (−0.59) compared to IDDPG (−0.76). The guidance from predictions accelerates early learning, while entropy regularization prevents premature convergence.
Table 5 presents the annual performance metrics normalized to the baseline (no BESS control). PGADDPG reduces electricity consumption, cost, and carbon emissions to 0.771, 0.659, and 0.743, respectively, for Building 1. In contrast, IDDPG achieves 0.879, 0.770, and 0.855. The collective improvements over IDDPG are 10.3%, 15.1%, and 15.2% in cost, emissions, and consumption.
| Algorithm | Electricity Consumption (Building 1) | Electricity Consumption (Building 2) | Economic Cost (Building 1) | Economic Cost (Building 2) | Carbon Emissions (Building 1) | Carbon Emissions (Building 2) |
|---|---|---|---|---|---|---|
| PGADDPG | 0.771 | 0.675 | 0.659 | 0.606 | 0.743 | 0.666 |
| PGDDPG | 0.788 | 0.750 | 0.662 | 0.664 | 0.760 | 0.733 |
| AIDPPG | 0.813 | 0.777 | 0.692 | 0.697 | 0.786 | 0.762 |
| IDDPG | 0.879 | 0.870 | 0.770 | 0.784 | 0.855 | 0.854 |
I also test the transferability of the trained policies on two new residential buildings (Buildings 3 and 4). The results in Table 6 show that PGADDPG maintains superior performance, reducing collective cost by 8.8%, emissions by 10.7%, and consumption by 11.2% compared to IDDPG. This demonstrates the strong generalization capability of the proposed method.
| Algorithm | Electricity Consumption (Building 3) | Electricity Consumption (Building 4) | Economic Cost (Building 3) | Economic Cost (Building 4) | Carbon Emissions (Building 3) | Carbon Emissions (Building 4) |
|---|---|---|---|---|---|---|
| PGADDPG | 0.888 | 0.753 | 0.839 | 0.679 | 0.890 | 0.764 |
| PGDDPG | 0.901 | 0.742 | 0.856 | 0.666 | 0.908 | 0.759 |
| AIDPPG | 0.914 | 0.767 | 0.867 | 0.701 | 0.917 | 0.781 |
| IDDPG | 0.963 | 0.881 | 0.920 | 0.817 | 0.955 | 0.869 |
Furthermore, I analyze the impact of the battery energy storage system (BESS) scheduling on grid peak load and load volatility. The PGADDPG strategy yields the lowest daily peak demand and the smallest weekly rolling load fluctuation, as shown conceptually in Figure 7 (not displayed). This confirms that the guidance from predictions combined with attention-based coordination effectively stabilizes the microgrid operation.
Conclusion
In this work, I have presented a predictive-guided multi-agent deep reinforcement learning framework for the optimal scheduling of battery energy storage systems (BESS) in residential microgrids. The proposed PGADDPG method integrates rolling forecasts of PV and load, attention-based inter-agent communication, and a hybrid reward structure. Experimental results based on real-world smart meter data demonstrate that PGADDPG significantly outperforms both independent DDPG and rule-based baselines in terms of cost reduction, emission mitigation, and load smoothing. The policy also exhibits strong transferability to unseen buildings. Future work will explore larger-scale systems with more diverse energy sources and advanced forecasting techniques to further enhance the performance of the battery energy storage system (BESS) scheduling.