In recent years, the integration of distributed energy resources has significantly increased the need for flexible and intelligent energy management. Among these resources, battery energy storage systems play a crucial role in mitigating the intermittency of renewable sources such as photovoltaics. However, the optimal scheduling of battery energy storage systems in a multi-building microgrid remains challenging due to uncertainties in load and generation, as well as the need for coordination among distributed storage units. Traditional rule-based or model predictive control methods often suffer from rigidity, high computational complexity, or poor generalization. To address these issues, we propose a novel approach named Predictive-Guided Attention-based Deep Deterministic Policy Gradient, which leverages rolling forecasts and multi-agent communication to improve the performance of battery energy storage systems scheduling. Our method not only enhances energy efficiency and reduces costs but also provides strong transferability to unseen environments.
The core idea is to train independent DDPG agents for each building, but with the addition of an attention mechanism that enables selective sharing of state and action information among agents. Furthermore, a 24-hour rolling prediction of load and photovoltaic output is performed using a hybrid cyclical averaging model and a self-attention bidirectional LSTM, respectively. The predictions are used to design a hierarchical control strategy that guides the reinforcement learning agent during early training phases, ensuring safe and efficient exploration. After guidance, the agent continues autonomous learning with importance sampling and entropy regularization to prevent overfitting to the guidance policy. We validate the method on real-world smart meter data from five residential buildings, demonstrating significant improvements in collective economic cost, carbon emissions, and power fluctuation mitigation compared to baseline DDPG and independent DDPG approaches.
Environment Modeling and Control Framework
We model a microgrid consisting of multiple residential buildings, each equipped with a battery energy storage system and a photovoltaic array. At each time step t, the net power exchange with the external grid, denoted as PGt, is given by the difference between power purchased from the grid and power sold to the grid. The photovoltaic generation PPVt first satisfies the local load PLoadt, and the remaining energy is used to charge the battery or is fed into the grid. The state of charge of the battery at time t, SOCt=Et/Q, evolves according to the following update equation:
$$
E_{t+1} = (1 – \eta_{sd}) E_t + \left( \eta_{c,t} P_{c,t} + \frac{P_{d,t}}{\eta_{d,t}} \right) \Delta t
$$
where Q is the battery capacity, ηsd is the self-discharge rate, and ηc,t, ηd,t are the charging and discharging efficiencies, which are dynamically adjusted using piecewise linear interpolation based on the operating power level. The control action at directly determines the SOC change, subject to the constraints SOCmin ≤ SOCt ≤ SOCmax. The architecture of our overall control scheme is illustrated in the following figure (please refer to the conceptual diagram of the microgrid environment).
Each DDPG agent observes local state information and receives a reward signal after taking an action. The experience tuples (observation, action, reward, next observation, done) are stored in a replay buffer for batch training. Additionally, an attention mechanism is integrated into the policy network to allow agents to communicate selectively. Specifically, agent i receives the states and actions of all other agents and computes attention weights to construct a weighted summary of the most relevant information. This summary is then concatenated with the local observation before being fed into the actor and critic networks. The overall training process is guided by a hierarchical control strategy derived from rolling predictions, which will be detailed later.
Multi-Objective Hybrid Reward Function
The design of the reward function is critical for the agent to learn an effective policy. We propose a multi-objective hybrid reward that balances individual and collective goals. The base reward for building b at time t is computed as:
$$
R_c(b,t) = X_0(b,t) \times \left[ \left( C_e(b,t) + C_c(b,t) \right) + \sqrt{ \left( \sum_{b=1}^N C_e(b,t) \right)^2 + \left( \sum_{b=1}^N C_c(b,t) \right)^2 } \right]
$$
where Ce and Cc are the electricity cost and carbon emission cost, respectively. X0(b,t) is a penalty coefficient defined as:
$$
X_0(b,t) = -1.4 + 1.2 \times \text{sign}\big(C_e(b,t)\big) \times \text{SOC}(b,t)
$$
The final reward R(b,t) is obtained by adding additional penalty terms Xi (e.g., penalties for not charging before peak hours, or not discharging during peak hours) weighted by coefficients wi:
$$
R(b,t) = R_c(b,t) + \sum_{i=1}^4 w_i X_i
$$
In our experiments, we set w4=0.2 and all other wi=0.1 to ensure the magnitudes are comparable. This reward encourages the agent to use storage effectively while considering the collective benefit of the entire microgrid.
Rolling Prediction for Load and Photovoltaic
Accurate short-term forecasting is essential for the hierarchical guidance strategy. For load prediction, we propose a cyclical averaging hybrid model that captures periodic patterns. At each time step, the model computes three separate forecasts based on the previous day, previous week, and previous month of historical data, weighting more recent data more heavily. Every 72 hours, the combination coefficients (w0, w1, w2) and the intercept b are updated via gradient descent. The final 24-hour load forecast at time t is:
$$
\hat{y}_t = w_0 \hat{y}_t^0 + w_1 \hat{y}_t^1 + w_2 \hat{y}_t^2 + b
$$
where each ^ytk is the cyclical average forecast from model k. After the full 24-hour forecast is produced, the first predicted value is fine-tuned using exponential smoothing to improve short-term accuracy.
For photovoltaic forecasting, we employ a Self-Attention Bidirectional Long Short-Term Memory network. The model is trained offline on data from other buildings (not the target buildings). Input features include date, historical PV power, and solar irradiance. The Bi-LSTM captures temporal dependencies from both forward and backward directions, while the self-attention layer assigns different weights to different time steps, focusing on the most relevant patterns. The attention mechanism computes:
$$
e_{ij} = \frac{q_i^T k_j}{\sqrt{d_k}}, \quad \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{n} \exp(e_{in})}, \quad c_i = \sum_{j} \alpha_{ij} v_j
$$
where qi, kj, vj are derived from the Bi-LSTM outputs, and dk is the dimension. The weighted context ci is then combined with the original output via a residual connection. The prediction accuracy is shown in the following tables, comparing our SA-BiLSTM with a standard LSTM for two representative buildings.
| Metric | SA-BiLSTM (Building 1) | SA-BiLSTM (Building 2) | LSTM (Building 1) | LSTM (Building 2) |
|---|---|---|---|---|
| 1-hour MSE | 0.034 | 0.018 | 0.217 | 0.270 |
| 1-hour MAE | 0.041 | 0.064 | 0.200 | 0.215 |
| 24-hour MSE | 0.046 | 0.041 | 0.207 | 0.267 |
| 24-hour MAE | 0.072 | 0.049 | 0.193 | 0.212 |
| Metric | Cyclical Averaging (Building 1) | Cyclical Averaging (Building 2) | Day-ahead (Building 1) | Day-ahead (Building 2) |
|---|---|---|---|---|
| 1-hour MSE | 0.332 | 0.464 | 0.624 | 0.983 |
| 1-hour MAE | 0.449 | 0.583 | 0.529 | 0.619 |
| 24-hour MSE | 0.477 | 0.829 | 0.654 | 1.159 |
| 24-hour MAE | 0.504 | 0.630 | 0.530 | 0.620 |
Prediction-Driven Hierarchical Control Strategy
Based on the 24-hour rolling forecasts, we design a hierarchical control strategy that operates in two layers. At the upper layer, we compute the minimum required state of charge at peak hours, SOCpeakmin, and the total available surplus PV energy before peak hours, SOCprepeak. If the surplus is insufficient to meet the minimum requirement, the battery is forced to charge. Otherwise, we enter the lower layer, which uses the next-hour forecast to maximize the utilization of remaining capacity:
$$
R_{t+1}^{pv} = E_{t+1}^{pv} – E_{t+1}^{load}, \quad a_t = \frac{R_{t+1}^{pv}}{Q}
$$
with the constraint that if Rt+1pv ≤ 0, the discharge action is limited to at ≤ SOCprepeak – SOCpeakmin to ensure peak-hour reserve. This hierarchical policy serves as a “teacher” for the reinforcement learning agent in the early training episodes, providing safe and reasonable actions.
Deep Reinforcement Learning Framework
The scheduling problem is formulated as a Markov Decision Process. The state space includes date, temperature, humidity, solar irradiance, load, PV output, SOC, net electricity cost, and electricity price. The action space is continuous, ranging between SOCmin and SOCmax. The Q-value function in DDPG is defined as:
$$
Q^\pi(s,a) = R(s,a) + \gamma \sum_{s’ \in S} \mathcal{P}(s’ | s,a) V^\pi(s’)
$$
To learn a deterministic policy μθ(s) in continuous action spaces, we use the deterministic policy gradient theorem. The actor is updated by ascending the gradient:
$$
\nabla_\theta J(\theta) \approx \mathbb{E}_{s \sim \rho^\mu} \left[ \nabla_\theta \mu_\theta(s) \nabla_a Q^\phi(s,a) \big|_{a=\mu_\theta(s)} \right]
$$
The critic is updated by minimizing the TD error:
$$
L(\phi) = \mathbb{E}_{(s,a,r,s’) \sim D} \left[ \left( Q^\phi(s,a) – y \right)^2 \right], \quad y = r + \gamma Q^{\phi’}(s’, \mu_{\theta’}(s’))
$$
Soft updates are used to copy parameters to target networks:
$$
\theta’ \leftarrow \tau \theta + (1-\tau) \theta’, \quad \phi’ \leftarrow \tau \phi + (1-\tau) \phi’
$$
Multi-Agent Communication with Attention and Policy Guidance
To enable coordination among the distributed battery energy storage systems, we equip each agent with an attention module that selectively aggregates information from other agents. For agent i:
$$
\text{out}_i = \text{attention}\left( o_i, \{o_j, a_j\}_{j \neq i} \right)
$$
The attention output is concatenated with the local observation to form the input to both actor and critic. The policy and value networks are still updated independently using the agent’s own replay buffer. During training, the guidance strategy derived from predictions is blended with the DDPG action using a sigmoid-based weighted combination:
$$
a = w_{rl} a_{rl} + w_{bf} a_{bf}, \quad w_{bf} = \frac{1}{1 + e^{-(d – t) \cdot \lambda(n)}}, \quad \lambda(n) = \max\left(0, 1 – \frac{n}{\text{total\_eps}}\right)
$$
where d is the absolute difference between the two actions, and t is an adaptive threshold. The guidance weight wbf decays over training episodes, allowing the DDPG agent to gradually take over. We also incorporate importance sampling and entropy regularization to encourage exploration beyond the guidance policy. The final actor loss is:
$$
J(\theta) \propto -\sum ( \text{norm}_w \cdot \text{Critic} ) + \epsilon \cdot \text{entropy}
$$
where ε is the regularization coefficient (set to 0.15 in our experiments).
Experimental Validation
We evaluate the proposed PGADDPG method on real data from the Pecan Street Dataport, using five residential buildings with one year of hourly load and PV data. The battery parameters are listed below.
| Parameter | Value |
|---|---|
| Capacity (kWh) | 11.0 |
| Power (kW) | 6 |
| Maximum SOC | 91% |
| Minimum SOC | 5% |
| Initial charge/discharge efficiency | 94% |
| Self-discharge rate | 0.001% |
| Power interpolation intervals | [[0,0.85], [0.6,0.94], [1,0.85]] |
| Efficiency interpolation intervals | [[0.0,1], [0.7,1], [1.0,0.2]] |
| Time Period | Jun-Sep (Weekday) | Jun-Sep (Weekend) | Oct-May (Weekday) | Oct-May (Weekend) |
|---|---|---|---|---|
| 8 AM – 4 PM | 0.22 | 0.22 | 0.21 | 0.21 |
| 4 PM – 9 PM | 0.54 | 0.40 | 0.50 | 0.50 |
| 9 PM – 8 AM | 0.22 | 0.22 | 0.21 | 0.21 |
We compare four configurations: independent DDPG (IDDPG), attention-based DDPG (AIDDPG), prediction-guided DDPG without attention (PGDDPG), and our full PGADDPG. All agents are trained for 20 episodes; the first episode uses random exploration, the last episode evaluates the learned policy. Hyperparameters: replay buffer size 200,000, batch size 256, actor and critic learning rates 0.0004, discount factor 0.992, target network soft update rate 0.0003, learning rate and noise decay 0.99. The actor and critic networks have two hidden layers with 300 and 400 units respectively, using ReLU activation. The attention layer has one hidden layer of 300 units with Softmax activation.
During training, the cumulative reward curves show that our PGADDPG achieves faster initial convergence due to guidance, and after the guidance phase ends (around episode 5), the reward dips slightly as the agent explores but then stabilizes at the highest final value. In contrast, IDDPG converges slower and reaches a lower final reward. The evaluation results on buildings 1 and 2 are summarized in the following table (normalized values relative to the baseline of no storage control).
| Algorithm | Electricity Consumption (B1) | Electricity Consumption (B2) | Economic Cost (B1) | Economic Cost (B2) | Carbon Emissions (B1) | Carbon Emissions (B2) |
|---|---|---|---|---|---|---|
| PGADDPG | 0.771 | 0.675 | 0.659 | 0.606 | 0.743 | 0.666 |
| PGDDPG | 0.788 | 0.750 | 0.662 | 0.664 | 0.760 | 0.733 |
| AIDPPG | 0.813 | 0.777 | 0.692 | 0.697 | 0.786 | 0.762 |
| IDDPG | 0.879 | 0.870 | 0.770 | 0.784 | 0.855 | 0.854 |
The collective microgrid metrics (averaged over all five buildings) reveal that PGADDPG reduces overall economic cost by 10.3%, carbon emissions by 15.1%, and electricity consumption by 15.2% compared to IDDPG. Furthermore, the daily peak demand and weekly rolling power fluctuation are significantly lower for PGADDPG, indicating improved stability and safety of the battery energy storage systems operation.
We also test transferability by directly applying the trained policies to two unseen buildings (Buildings 3 and 4). The results in the table below show that all DRL methods outperform the baseline, and PGADDPG again yields the best performance, with reductions of 8.8% in cost, 10.7% in carbon, and 11.2% in consumption relative to IDDPG. This demonstrates the strong generalization capability of our approach.
| Algorithm | Electricity Consumption (B3) | Electricity Consumption (B4) | Economic Cost (B3) | Economic Cost (B4) | Carbon Emissions (B3) | Carbon Emissions (B4) |
|---|---|---|---|---|---|---|
| PGADDPG | 0.888 | 0.753 | 0.839 | 0.679 | 0.890 | 0.764 |
| PGDDPG | 0.901 | 0.742 | 0.856 | 0.666 | 0.908 | 0.759 |
| AIDPPG | 0.914 | 0.767 | 0.867 | 0.701 | 0.917 | 0.781 |
| IDDPG | 0.963 | 0.881 | 0.920 | 0.817 | 0.955 | 0.869 |
Conclusion
In this work, we have presented a novel deep reinforcement learning framework for the optimal scheduling of multiple battery energy storage systems in a residential microgrid. By integrating rolling prediction-based guidance, attention-based multi-agent communication, and a carefully designed hybrid reward function, our PGADDPG method achieves superior performance in terms of economic cost, carbon emissions, and power fluctuation mitigation. The prediction-guided phase, although limited to a small number of initial episodes, provides critical prior knowledge that accelerates learning and prevents unsafe exploration. The attention mechanism enables efficient coordination among distributed battery energy storage systems without compromising user privacy. Experimental results on real-world data confirm that our method outperforms both independent and attention-only DDPG baselines, and exhibits strong transferability to unseen buildings. Future work will extend the framework to larger-scale microgrids with more heterogeneous resources and investigate more efficient guidance strategies.