Mathieu, S., Manuel de Villena Millan, M., Vermeulen, E., & Ernst, D. (2019). Harnessing the flexibility of energy management systems: a retailer perspective.

*Proceedings PowerTech 2019*.Peer reviewed

The business of electricity retailing is changing following the current evolution of the electricity system. An example of such an evolution is the increasing number of clients installing sophisticated energy ...

Radu, D.-C., Berger, M., Fonteneau, R., Hardy, S., Fettweis, X., Le Du, M., Panciatici, P., Balea, L., & Ernst, D. (2019). Complementarity Assessment of South Greenland Katabatic Flows and West Europe Wind Regimes.

*Energy, 175*, 393-401.Peer reviewed

Current global environmental challenges require vigorous and diverse actions in the energy sector. One solution that has recently attracted interest consists in harnessing high-quality variable renewable ...

Berger, M., Radu, D.-C., Fonteneau, R., Deschuyteneer, T., Detienne, G., & Ernst, D. (2019).

*The Role of Power-to-Gas and Carbon Capture Technologies in Cross-Sector Decarbonisation Strategies*. Eprint/Working paper retrieved from http://orbi.ulg.ac.be/handle/2268/235110.Peer reviewed

This paper proposes an optimisation-based framework to tackle long-term centralised planning problems of multi-sector, integrated energy systems including electricity, hydrogen, natural gas, synthetic methane ...

François-Lavet, V., Rabusseau, G., Pineau, J., Ernst, D., & Fonteneau, R. (2019). On overfitting and asymptotic bias in batch reinforcement learning with partial observability.

*Journal of Artificial Intelligence Research, 65*, 1-30.Peer reviewed (verified by ORBi)

This paper provides an analysis of the tradeoff between asymptotic bias (suboptimality with unlimited data) and overfitting (additional suboptimality due to limited data) in the context of reinforcement ...

Vangulick, D., Cornélusse, B., & Ernst, D. (2019). Blockchain: A novel approach for the consensus algorithm using Condorcet Voting procedure.

*Proceedings of the IEEE International Conference on Decentralized Applications and Infrastructures*.Peer reviewed

The blockchain technology allows interested parties to access a common register, the update, and integrity of which are collectively managed in a decentralized manner by a network of actors. It is the ...

Manuel de Villena Millan, M., Fonteneau, R., Gautier, A., & Ernst, D. (2019, March). Evaluating the Evolution of Distribution Networks under Different Regulatory Frameworks with Multi-Agent Modelling.

*Energies*.Peer reviewed (verified by ORBi)

In the context of increasing decentralised electricity generation, this paper evaluates the effect of different regulatory frameworks on the evolution of distribution networks. This problem is addressed by ...

Vecoven, N., Ernst, D., Wehenkel, A., & Drion, G. (2019).

*Introducing neuromodulation in deep neural networks to learn adaptive behaviours*. Eprint/Working paper retrieved from https://arxiv.org/abs/1812.09113.In this paper, we propose a new deep neural network architecture, called NMD net, that has been specifically designed to learn adaptive behaviours. This architecture exploits a biological mechanism called ...

Marulli, D., Mathieu, S., Sutera, A., & Ernst, D. (2019).

*Reconstruction of low-voltage networks with limited observability*. Eprint/Working paper retrieved from http://orbi.ulg.ac.be/handle/2268/238913.This work addresses the problem of reconstructing an electrical model for low voltage networks when no information about topology and line parameters is available. Obtaining an accurate electrical model for ...

Petitet, M., Perrot, M., Mathieu, S., Ernst, D., & Phulpin, Y. (2019). Impact of gate closure time on the efficiency of power systems balancing.

*Energy Policy, 129*, 562-573.Peer reviewed (verified by ORBi)

This paper focuses on market design options for operational balancing management in self-dispatch electric power systems. In particular, it investigates the most relevant timing for the balancing gate closure ...

Godart, M.-F., Lambotte, J.-M., Bianchet, B., Maldague, H., Hendrickx, S., Copée, P., Wilmotte, P.-F., Ruelle, C., Bastin, F., Mercenier, C., Sandu, R., Vasquez-Parras, J., Devillet, G., Sérusiaux, E., Teller, J., Ernst, D., Halleux, J.-M., Bottiau, V., Romain, J., Harou, R., Hanin, Y., Quadu, F., Quériat, S., Castiau, E., Courtois, X., Leclercq, A., Grandjean, M., Coppens, A., D'Andrimont, C., & Verelst, S. (2018, December 08).

*ETAT DU TERRITOIRE WALLON - PARTIE 2 : DYNAMIQUES SECTORIELLES TERRITORIALISÉES*. Paper presented at Colloque 2016 de la CODT : Territoire(s) wallon(s) : tendances et perspectives, Mons, Belgique.Ce support PPT est basé sur les travaux menés sur le volet Analyses sectorielles de l'Etat du territoire wallon réalisé en 2015-2016 par la CPDT. Il a servi de base lors d'un exposé réalisé dans le cadre du ...

Berger, M., Radu, D.-C., Fonteneau, R., Henry, R., Glavic, M., Fettweis, X., Le Du, M., Panciatici, P., Balea, L., & Ernst, D. (2018).

*Critical Time Windows for Renewable Resource Complementarity Assessment*. Eprint/Working paper retrieved from https://arxiv.org/abs/1812.02809.This paper proposes a systematic framework to assess the complementarity of renewable resources over arbitrary geographical scopes and temporal scales which is particularly well-suited to exploit very large ...

Berger, M., Radu, D.-C., Fonteneau, R., Ernst, D., Deschuyteneer, T., & Detienne, G. (2018). Centralised Planning of National Integrated Energy System with Power-to-Gas and Gas Storages.

*Proceedings of the 11th Mediterranean Conference on Power Generation, Transmission, Distribution and Energy Conversion (Medpower2018)*.Peer reviewed

This paper proposes an optimisation-based framework to tackle long-term centralised planning problems of integrated energy systems with bi-directional electricity-gas carriers coupling under various policy ...

Vanrykel, F., Ernst, D., & Bourgeois, M. (2018, November). Fostering Share&Charge through proper regulation.

*Competition and Regulation in Network Industries*.Peer reviewed (verified by ORBi)

This article studies the emergence of Share&Charge, a German platform that organizes the sharing of charging stations for electric vehicles (EVs) and the billing for the energy transactions. Share&Charge ...

Petitet, M., Perrot, M., Mathieu, S., Ernst, D., & Phulpin, Y. (2018).

*On the efficiency of decentralized decision-making in self-dispatch power systems*. Eprint/Working paper retrieved from http://www.ceem-dauphine.org/assets/wp/pdf/CEEM_Working_Paper_33_Marie_Petitet_et_al.pdf.Peer reviewed

This paper focuses on market design options for operational balancing management in self-dispatch electric power systems. In particular, it investigates the most relevant timing for the balancing gate closure ...

Vanrykel, F., Ernst, D., & Bourgeois, M. (2018). Regulatory Challenges for Share&Charge Models.

*Network Industries Quarterly, 20*(3), 17-21.Peer reviewed

The platform Share&Charge provides an innovative solution to the lack of electric vehicles charging infrastructure, lying at the crossroad of the electricity market, the transport market and digitalisation ...

Olivier, F., Fonteneau, R., Mathieu, S., & Ernst, D. (2018). Distributed Control of Photovoltaic Units in unbalanced LV Distribution Networks to Prevent Overvoltages.

*Proc. of the 6th IEEE International Conference on Smart Energy Grid Engineering (SEGE 2018)*.Peer reviewed

As more and more photovoltaic units are being installed, some LV networks have already attained their maximum hosting capacity, i.e. the maximum amount of distributed energy resources that they can accommodate ...

Olivier, F. (2018).

*Solutions for Integrating Photovoltaic Panels Into Low-voltage Distribution Networks*. Unpublished doctoral thesis, Université de Liège, Liège, Belgique.As more PhotoVoltaic (PV) units are being installed, some low-voltage (LV) distribution networks have already at- tained their maximum hosting capacity, i.e. the maximum amount of distributed energy resources ...

Vangulick, D., Cornélusse, B., & Ernst, D. (2018). Blockchain for peer-to-peer energy exchanges: design and recommendations (additional work).

*CIRED Workshop 2018 Ljubljana, Slovania*.Peer reviewed

Energy communities and peer-to-peer energy exchanges are expected to play an important role in the energy transition.
In this context, the blockchain approach can be employed to foster this decentralized ...

Vangulick, D., Cornélusse, B., & Ernst, D. (2018). Blockchain for peer-to-peer energy exchanges: Probabilistic approach of Proof of Stake.

*CIRED WORKSHOP 2018*.Peer reviewed

Energy communities and peer-to-peer energy exchanges are expected to play an important role in the energy transition.
In this context, the blockchain approach can be employed to foster this decentralized ...

Boukas, I., Ernst, D., & Cornélusse, B. (2018). Real-Time Bidding Strategies from Micro-Grids Using Reinforcement Learning.

*Proceedings of CIRED Workshop 2018*.Peer reviewed

We address the problem faced by the operator of a microgrid participating in a continuous real-time market. The microgrid consists of distributed generation, flexible loads and a storage device. The goal of ...

Boukas, I., Ernst, D., Papavasiliou, A., & Cornélusse, B. (2018). Intra-day Bidding Strategies for Storage Devices Using Deep Reinforcement Learning.

*International Conference on the European Energy Market, Łódź 27-29 June 2018*(pp. 6).Peer reviewed

The problem faced by the operator of a storage device participating in a continuous intra-day (CID) market is addressed in this paper. The goal of the storage device operator is the maximization of the ...

Cornélusse, B., Ernst, D., & Lachi, S. (2018). Optimal operation and fair profit allocation in community microgrids.

*Proceedings of CIRED Workshop 2018*.Peer reviewed

This work fits in the context of community migrogrids, where entities of a community can exchange energy and services among them without going through the usual channels of the public grid. We propose and ...

Manuel de Villena Millan, M., Gautier, A., Fonteneau, R., & Ernst, D. (2018). A multi-agent system approach to model the interaction between distributed generation deployment and the grid.

*Proc. of CIRED Workshop 2018*.Peer reviewed

This paper introduces a multi-agent dynamical system of the interaction between electricity consumers, the electricity distribution system operator, and the technological (generation, storage) and regulatory ...

Olivier, F., Fonteneau, R., & Ernst, D. (2018). Modelling of three-phase four-wire low-voltage cables taking into account the neutral connection to the earth.

*Proc. of CIRED Workshop 2018*.Peer reviewed

Local energy communities (LECs) usually occur at the level of low-voltage distribution networks, which are inherently unbalanced due to single-phase household appliances and distributed generation. To simulate ...

Olivier, F., Sutera, A., Geurts, P., Fonteneau, R., & Ernst, D. (2018). Phase Identification of Smart Meters by Clustering Voltage Measurements.

*Proceedings of the XX Power Systems Computation Conference (PSCC 2018)*.Peer reviewed

When a smart meter, be it single-phase or threephase, is connected to a three-phase network, the phase(s) to which it is connected is (are) initially not known. This means that each of its measurements is not ...

Vangulick, D., Cornélusse, B., & Ernst, D. (2018). Blockchain for peer-to-peer energy exchanges: design and recommendations.

*Proceedings of the XX Power Systems Computation Conference (PSCC2018)*.Peer reviewed

Energy communities and peer-to-peer energy exchanges are expected to play an important role in the energy transition. In this context, the blockchain approach can be employed to foster this decentralized ...

Li, J., Xiong, R., Mu, H., Cornélusse, B., Vanderbemden, P., Ernst, D., & Yuan, W. (2018). Design and real-time test of a hybrid energy storage system in the microgrid with the benefit of improving the battery lifetime.

*Applied Energy, 218*, 470-478.Peer reviewed (verified by ORBi)

This study proposes a hybrid energy storage system (HESS) composed of the superconducting energy storage system (SMES) and the battery. The system is designed to compensate power fluctuations within a ...

Manuel de Villena Millan, M., Fonteneau, R., Gautier, A., & Ernst, D. (2018). Exploring Regulation Policies in Distribution Networks through a Multi-Agent Simulator.

*Proc. of YRS2018*.Peer reviewed

This paper presents a multi-agent simulator that describes the interactions between the agents of a distribution network (DN), and an environment. The agents are the users of the DN and the electricity ...

Sootla, A., Mauroy, A., & Ernst, D. (2018). An Optimal Control Formulation of Pulse-Based Control Using Koopman Operator.

*Automatica, 91*, 217-224.Peer reviewed (verified by ORBi)

In many applications, and in systems/synthetic biology in particular, it is desirable to compute control policies that force the trajectory of a bistable system from one equilibrium (the initial point) to ...

Sootla, A., & Ernst, D. (2018). Pulse-Based Control Using Koopman Operator Under Parametric Uncertainty.

*IEEE Transactions on Automatic Control, 63*(3), 791-796.Peer reviewed (verified by ORBi)

In applications, such as biomedicine and systems/synthetic biology, technical limitations in actuation complicate implementation of time-varying control signals. In order to alleviate some of these limitations ...

Mathieu, S., Olivier, F., Ernst, D., & Cornélusse, B. (2018). Effect of Voltage Constraints on the Exchange of Flexibility Services in Distribution Networks.

*Proceedings of the 9th Conference on Innovative Smart Grid Technology North America (ISGT 2018)*.Peer reviewed

Many possibilities exist to organise exchanges of flexibility within a distribution system. In this paper, we call such a possibility an interaction model. The DSIMA (Distribution System Interaction Model ...

Mathieu, S., Petitet, M., Perrot, M., Ernst, D., & Phulpin, Y. (2017).

*SiSTEM, a Model for the Simulation of Short-Term Electricity Markets*. Eprint/Working paper retrieved from http://www.ceem-dauphine.org/working/en/SiSTEM-A-MODEL-FOR-THE-SIMULATION-OF-SHORT-TERM-ELECTRICITY-MARKETS.Peer reviewed

The aim of this document is to present SiSTEM, a multi-level simulation model of European short-term electricity markets, covering day-ahead and intraday exchanges to balancing activations in real-time, and ...

François-Lavet, V. (2017).

*Contributions to deep reinforcement learning and its applications in smartgrids*. Unpublished doctoral thesis, Université de Liège, Liège, Belgique.Reinforcement learning and its extension with deep learning have led to a field of research called deep reinforcement learning.
Applications of that research have recently shown the possibility to solve ...

Olivier, F., Marulli, D., Ernst, D., & Fonteneau, R. (2017). Foreseeing New Control Challenges in Electricity Prosumer Communities.

*Proc. of the 10th Bulk Power Systems Dynamics and Control Symposium – IREP’2017*.Peer reviewed

This paper is dedicated to electricity prosumer communities, which are groups of people producing, sharing and consuming electricity locally. This paper focuses on building a rigorous mathematical framework in ...

Glavic, M., Fonteneau, R., & Ernst, D. (2017). Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives.

*The 20th World Congress of the International Federation of Automatic Control, Toulouse 9-14 July 2017*(pp. 1-10).Peer reviewed

In this paper, we review past (including very recent) research considerations in using reinforcement
learning (RL) to solve electric power system decision and control problems. The
RL considerations are ...

Olivier, F., Ernst, D., & Fonteneau, R. (2017). Automatic phase identification of smart meter measurement data.

*Proceedings of the 24th International Conference and Exhibition on Electricity Distribution, CIRED 2017*.Peer reviewed

This paper highlights the importance of the knowledge of the phase identification for the different measurement points inside a low-voltage distribution network. Besides considering existing solutions, we ...

Sutera, A., Joly, A., François-Lavet, V., Qiu, Z., Ernst, D., & Geurts, P. (2017). Simple connectome inference from partial correlation statistics in calcium imaging. In J., Soriano, D., Battaglia, I., Guyon, V., Lemaire, J., Orlandi, & B., Ray (Eds.),

*Neural Connectomics Challenge*(pp. 23-36). Springer.Peer reviewed

In this work, we propose a simple yet effective solution to the problem of connectome inference in calcium imaging data. The proposed algorithm consists of two steps. First, processing the raw signals to ...

Vangulick, D., Cornélusse, B., Vanherck, T., Devolder, O., & Ernst, D. (2017). E-CLOUD, the open microgrid in existing network infrastructure.

*Proceedings of the 24th International Conference on Electricity Distribution*.Peer reviewed

The main goal of the E-Cloud, as with every microgrid, is to maximize the consumption of energy produced locally. To reach this goal, based on consumption profiles of customers willing to participate in the E ...

Vangulick, D., Ernst, D., & Van Cutsem, T. (2017). Resilience of the DSO network near to 50.2Hz.

*Proceedings of the 24th International Conference on Electricity Distribution*.Peer reviewed

In an electrical system where decentralized and embedded productions are becoming more and more important, it is essential to ensure there is a good understanding of their behaviour at their operating limits ...

Ernst, D. (2017).

*Energy: the clash of nations*. Paper presented at - Namur, Belgique.Le monde de l'énergie est en guerre. C'est une guerre qui est menée sur de nombreux fronts, du Texas à l'Arabie saoudite, des salons feutrés de l'ONU et de l'organisation mondiale du commerce aux conseils d ...

Castronovo, M. (2017).

*Offline Policy-search in Bayesian Reinforcement Learning*. Unpublished doctoral thesis, Université de Liège, Liège, Belgique.This thesis presents research contributions in the study field of Bayesian Reinforcement Learning — a subfield of Reinforcement Learning where, even though the dynamics of the system are un- known, the ...

Castronovo, M., François-Lavet, V., Fonteneau, R., Ernst, D., & Couëtoux, A. (2017). Approximate Bayes Optimal Policy Search using Neural Networks.

*Proceedings of the 9th International Conference on Agents and Artificial Intelligence (ICAART 2017)*.Peer reviewed

Bayesian Reinforcement Learning (BRL) agents aim to maximise the expected collected rewards obtained when interacting with an unknown Markov Decision Process (MDP) while using some prior knowledge. State-of ...

Dubois, A., Wehenkel, A., Fonteneau, R., Olivier, F., & Ernst, D. (2017). An App-based Algorithmic Approach for Harvesting Local and Renewable Energy Using Electric Vehicles.

*Proceedings of the 9th International Conference on Agents and Artificial Intelligence (ICAART 2017)*.Peer reviewed

The emergence of electric vehicles (EVs), combined with the rise of renewable energy production capacities, will strongly impact the way electricity is produced, distributed and consumed in the very near ...

Georges, E., Cornélusse, B., Ernst, D., Lemort, V., & Mathieu, S. (2017). Residential heat pump as flexible load for direct control service with parametrized duration and rebound effect.

*Applied Energy, 187*, 140-153.Peer reviewed (verified by ORBi)

This paper addresses the problem of an aggregator controlling residential heat pumps to offer a direct control flexibility service. The service consists of a power modulation, upward or downward, that is ...

de Faria Jr., H., Resende, M., & Ernst, D. (2017). A biased random key genetic algorithm applied to the electric distribution network reconfiguration problem.

*Journal of Heuristics*.Peer reviewed (verified by ORBi)

This work presents a biased random-key genetic algorithm (BRKGA) to solve the electric distribution network reconfiguration problem (DNR). The DNR is one of the most studied combinatorial optimization problems ...

Fonteneau, R., & Ernst, D. (2017). On the Dynamics of the Deployment of Renewable Energy Production Capacities. In J. N., Furze, K., Swing, A. K., Gupta, R. H., McClatchey, & D. M., Reynolds,

*Mathematical Advances Towards Sustainable Environmental Systems*(pp. 43-60). Springer.Peer reviewed

This chapter falls within the context of modeling the deployment of renewable en-ergy production capacities in the scope of the energy transition. This problem is addressed from an energy point of view, i.e ...

Li, J., Cornélusse, B., Vanderbemden, P., & Ernst, D. (2017). A SC/battery Hybrid Energy Storage System in the Microgrid.

*Energy Procedia, 142*, 3697–3702.Peer reviewed (verified by ORBi)

The major challenges in power systems are driven by the energy shortage and environmental concerns, namely facilitating the penetration of renewable energy and improving the efficiency of the renewable powers ...

Mathieu, S., Ernst, D., & Cornélusse, B. (2017). Agent-based analysis of dynamic access ranges to the distribution network.

*Proceedings of the 6th European Innovative Smart Grid Technologies (ISGT)*.Peer reviewed

There is a need to clearly state an interaction model that formalizes interactions between actors of the distribution system exchanging flexibility. In previous works we quantitatively evaluated the ...

François-Lavet, V., Taralla, D., Ernst, D., & Fonteneau, R. (2016). Deep Reinforcement Learning Solutions for Energy Microgrids Management.

*European Workshop on Reinforcement Learning (EWRL 2016)*.Peer reviewed

This paper addresses the problem of efficiently operating the storage devices in an electricity microgrid featuring photovoltaic (PV) panels with both short- and long-term storage capacities. The problem of ...

Gemine, Q. (2016).

*Active Network Management for Electrical Distribution Systems*. Unpublished doctoral thesis, Université de Liège, Liège, Belgique.With the increasing share of renewable and distributed generation in electrical distribution systems, Active Network Management (ANM) has become a valuable option for a distribution system operator to operate ...

Ernst, D. (2016).

*La transition énergétique, l’affaire de tous*. Paper presented at Soirée de lancement de la saison 2016-2017 de Liège Créative, Liège, Belgique.Conférence d'ouverture donnée par le Prof . Ernst lors de la soirée de lancement de la saison 2016-2017 de Liège Créative. Accéder à la vidéo de la conférence: https://www.youtube.com/watch?v=2SJ14Sj33bI

Gemine, Q., Ernst, D., & Cornélusse, B. (2016). Active network management for electrical distribution systems: problem formulation, benchmark, and approximate solution.

*Optimization and Engineering, 18*(3), 587-629.Peer reviewed

With the increasing share of renewable and distributed generation in electrical distribution systems, active network management (ANM) becomes a valuable option for a distribution system operator to operate his ...

Aittahar, S., Sootla, A., & Ernst, D. (Other coll.). (2016, September 19).

*Policy transfer using Value Function as Prior Information*. Paper presented at Phd Forum, Riva del Garda, Italy.Peer reviewed

This work proposes an approach based on reward shaping
techniques in a reinforcement learning setting to approximate the opti-
mal decision-making process (also called the optimal policy) in a desired
task ...

Castronovo, M., Ernst, D., Couëtoux, A., & Fonteneau, R. (2016, June). Benchmarking for Bayesian Reinforcement Learning.

*PLoS ONE*.Peer reviewed (verified by ORBi)

In the Bayesian Reinforcement Learning (BRL) setting, agents try to maximise the col- lected rewards while interacting with their environment while using some prior knowledge that is accessed beforehand. Many ...

Gemine, Q., Cornélusse, B., Glavic, M., Fonteneau, R., & Ernst, D. (2016). A Gaussian mixture approach to model stochastic processes in power systems.

*Proceedings of the 19th Power Systems Computation Conference (PSCC'16)*.Peer reviewed

Probabilistic methods are emerging for operating electrical networks, driven by the integration of renewable generation. We present an algorithm that models a stochastic process as a Markov process using a ...

Georges, E., Cornélusse, B., Ernst, D., Louveaux, Q., Lemort, V., & Mathieu, S. (2016). Direct control service from residential heat pump aggregation with specified payback.

*Proceedings of the 19th Power Systems Computation Conference (PSCC)*.Peer reviewed

This paper addresses the problem of an aggregator controlling residential heat pumps to offer a direct control flexibility service. The service is defined by a 15 minute power modulation, upward or downward ...

Cornélusse, B., Vangulick, D., Glavic, M., & Ernst, D. (2016). Further validation and extensions of the Global Capacity ANnouncement procedure for distribution systems.

*CIRED Workshop Proceedings, Helsinki 14-15 June 2016*.Peer reviewed

This paper extends the Global Capacity ANnouncement
procedure proposed in [5] along two directions. First,
two new stopping criteria are considered. Second, annual
losses are evaluated using representative ...

López-Erauskin, R., Gyselinck, J., Olivier, F., Ernst, D., Hervás, M. E., & Fabre, A. (2016). Modelling and Emulation of an Unbalanced LV Feeder with Photovoltaic Inverters.

*Proc. of 8th IEEE Benelux Young researchers symposium in Electrical Power Engineering*.Peer reviewed

In this paper, the penetration of grid-connected pho- tovoltaic systems is studied, experimentally tested and compared to simulation results. In particular, how the inverse current flow and unbalance ...

François-Lavet, V., Gemine, Q., Ernst, D., & Fonteneau, R. (2016). Towards the Minimization of the Levelized Energy Costs of Microgrids using both Long-term and Short-term Storage Devices.

*Smart Grid: Networking, Data Management, and Business Models*(pp. 295-319). CRC Press.Peer reviewed

This chapter falls within the context of the optimization of the levelized energy cost (LEC) of microgrids featuring photovoltaic panels (PV) associated with both long-term (hydrogen) and short-term (batteries ...

Mathieu, S. (2016).

*Flexibility services in the electrical system*. Unpublished doctoral thesis, Université de Liège, Liège, Belgique.The work presented in this thesis considers the electrical flexibility from the electric load to its usage as a commodity. The conception of the European electrical system has led to a large amount of actors ...

Olivier, F., Aristidou, P., Ernst, D., & Van Cutsem, T. (2016). Active Management of Low-Voltage Networks for Mitigating Overvoltages due to Photovoltaic Units.

*IEEE Transactions on Smart Grid, 2*(7), 926-936.Peer reviewed (verified by ORBi)

In this paper, the overvoltage problems that might arise from the integration of photovoltaic panels into low-voltage distribution networks is addressed. A distributed scheme is proposed that adjusts the ...

Taralla, D., Qiu, Z., Sutera, A., Fonteneau, R., & Ernst, D. (2016). Decision Making from Confidence Measurement on the Reward Growth using Supervised Learning: A Study Intended for Large-Scale Video Games.

*Proceedings of the 8th International Conference on Agents and Artificial Intelligence (ICAART 2016) - Volume 2*(pp. 264-271).Peer reviewed

Video games have become more and more complex over the past decades. Today, players wander in visually and option- rich environments, and each choice they make, at any given time, can have a combinatorial ...

Georges, E., Cornélusse, B., Ernst, D., Lemort, V., & Mathieu, S. (2016, January 26).

*Residential heat pumps as flexible loads for direct control service with constrained payback*. Paper presented at First Seminar on Demand Response, Brussels, Belgium.This paper addresses the problem of an aggregator controlling residential heat pumps to offer a direct control ﬂexibility service. The service is deﬁned by a 15 minute power modulation, upward or ...

Mathieu, S., Louveaux, Q., Ernst, D., & Cornélusse, B. (2016). DSIMA: A testbed for the quantitative analysis of interaction models within distribution networks.

*Sustainable Energy, Grids and Networks, 5*, 78 - 93.Peer reviewed (verified by ORBi)

This article proposes an open-source testbed to simulate interaction models governing the exchange of flexibility services located within a distribution network. The testbed is an agent-based system in which ...

Aittahar, S., François-Lavet, V., Lodeweyckx, S., Ernst, D., & Fonteneau, R. (2015). Imitative Learning for Online Planning in Microgrids. In W. L., Woon, A., Zeyar, & M., Stuart (Eds.),

*Data Analytics for Renewable Energy Integration*(pp. 1-15). Springer.Peer reviewed

This paper aims to design an algorithm dedicated to operational planning for microgrids in the challenging case where the scenarios of production and consumption are not known in advance. Using expert ...

François-Lavet, V., Fonteneau, R., & Ernst, D. (2015). How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies.

*NIPS 2015 Workshop on Deep Reinforcement Learning*.Peer reviewed

Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a ...

Cornélusse, B., Vangulick, D., Glavic, M., & Ernst, D. (2015). Global capacity announcement of electrical distribution systems: A pragmatic approach.

*Sustainable Energy, Grids and Networks, 4*, 43-53.Peer reviewed (verified by ORBi)

We propose a pragmatic procedure to facilitate the connection process of Distributed Generation (DG) with reference to the European regulatory framework where Distribution System Operators (DSOs) are, except ...

Johnen, A., Ernst, D., & Geuzaine, C. (2015). Sequential decision-making approach for quadrangular mesh generation.

*Engineering with Computers, 31*(4), 729-735.Peer reviewed (verified by ORBi)

A new indirect quadrangular mesh generation algorithm which relies on sequential decision-making techniques to search for optimal triangle recombinations is presented. In contrast to the state-of-art Blossom ...

Vandael, S., Claessens, B., Ernst, D., Holvoet, T., & Deconinck, G. (2015). Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market.

*IEEE Transactions on Smart Grid, 6*(4), 1795 - 1805.Peer reviewed (verified by ORBi)

This paper addresses the problem of defining a day-ahead consumption plan for charging a fleet of electric vehicles (EVs), and following this plan during operation. A challenge herein is the beforehand unknown ...

Taralla, D. (2015).

*Learning Artificial Intelligence in Large-Scale Video Games: A First Case Study with Hearthstone: Heroes of Warcraft*. Unpublished master thesis, Université de Liège, Liège, Belgique.Over the past twenty years, video games have become more and more complex thanks to the emergence of new computing technologies. The challenges players face now involve the simultaneous consideration of many ...

Cornélusse, B., Leroux, A., Glavic, M., & Ernst, D. (2015). Graph matching for reconciling SCADA and GIS of a distribution network.

*Proceedings of the International Conference on Electricity Distribution, CIRED 2015*.Peer reviewed

This article deals with the problem of automatically es- tablishing a correspondence between two databases popu- lated independently over the years by a distribution com- pany, for instance a SCADA system and ...

Cornélusse, B., Vangulick, D., Glavic, M., & Ernst, D. (2015). A process to address electricity distribution sector challenges: the GREDOR project approach.

*Proceedings of the International Conference on Electricity Distribution, CIRED 2015*.Peer reviewed

This paper presents a general process set in the GREDOR (French acronym for “Gestion des Réseaux Electriques de Distribution Ouverts aux Renouvelables”) project to address the challenges in distribution ...

Mathieu, S., Ernst, D., & Cornélusse, B. (2015). Macroscopic analysis of interaction models for the provision of flexibility in distribution systems.

*Proceedings of the International Conference on Electricity Distribution, CIRED 2015*.Peer reviewed

To ease the transition towards the future of distribution grid management, regulators must revise the current interaction model, that is, the set of rules guiding the interactions between all the parties of ...

Safadi, F. (2015).

*Artificial Intelligence in Video Games: Towards a Unified Framework*. Unpublished doctoral thesis, Université de Liège, Liège, Belgique.The work presented in this dissertation revolves around the problem of designing artificial intelligence (AI) for video games. This problem becomes increasingly challenging as video games grow in complexity ...

Safadi, F., Fonteneau, R., & Ernst, D. (2015). Artificial Intelligence in Video Games: Towards a Unified Framework.

*International Journal of Computer Games Technology, 2015*, 30.Peer reviewed

With modern video games frequently featuring sophisticated and realistic environments, the need for smart and comprehensive agents that understand the various aspects of complex environments is pressing. Since ...

Léonard, G., François-Lavet, V., Ernst, D., Meinrenken J., C., & Lackner S., K. (2015). Electricity storage with liquid fuels in a zone powered by 100% variable renewables.

*Proceedings of the 12th International Conference on the European Energy Market - EEM15*.Peer reviewed

In this work, an electricity zone with 100% renewables is simulated to determine the optimal sizing of generation and storage capacities in such a zone. Using actual wind output data, the model evaluates the ...

Merciadri, L., Mathieu, S., Ernst, D., & Louveaux, Q. (2015). Optimal Assignment of Off-Peak Hours to Lower Curtailments in the Distribution Network.

*Proceedings of the 5th European Innovative Smart Grid Technologies (ISGT)*.Peer reviewed

We consider a price signal with two settings: off-peak tariff and on-peak tariff.
Some loads are connected to specific electricity meters which allow the consumption of power only in off-peak periods ...

François-Lavet, V., Fonteneau, R., & Ernst, D. (2014, December). Using approximate dynamic programming for estimating the revenues of a hydrogen-based high-capacity storage device.

*IEEE Symposium Series on Computational Intelligence*.Peer reviewed

This paper proposes a methodology to estimate the maximum revenue that can be generated by a company that operates a high-capacity storage device to buy or sell electricity on the day-ahead electricity market ...

Rivadeneira, P., Moog, C., Stan, G.-B., Brunet, C., Raffi, F., Ferré, V., Costanza, V., Mhawej, M.-J., Biafore, F., Ouattara, D., Ernst, D., Fonteneau, R., & Xia, X. (2014). Mathematical Modeling of HIV Dynamics After Antiretroviral Therapy Initiation: A Review.

*BioResearch Open Access, 3*(5), 233-241.Peer reviewed (verified by ORBi)

This review shows the potential ground-breaking impact that mathematical tools may have in the analysis and the understanding of the HIV dynamics. In the first part, early diagnosis of immunological failure is ...

Moog, C., Rivadeneira, P., Stan, G.-B., Brunet, C., Raffi, F., Ferre, V., Costanza, V., Mhawej, M.-J., Ernst, D., Fonteneau, R., Biafore, F., Ouattara, D., & Xia, X. (2014). Mathematical modeling of HIV dynamics after antiretroviral therapy initiation: A clinical research study.

*AIDS Research and Human Retroviruses, 30*(9), 831-834.Peer reviewed (verified by ORBi)

Immunological failure is identified from the estimation of certain parameters of a mathematical model of the HIV infection dynamics. This identification is supported by clinical research results from an ...

Chatzivasileiadis, S., Ernst, D., & Andersson, G. (2014). Global power grids for harnessing world renewable energy. In L., Jones (Ed.),

*Renewable Energy Integration: Practical Management of Variability, Uncertainty and Flexibility in Power Grids*(pp. 175-188). Academic Press.Peer reviewed

The Global Grid advocates the connection of all regional power systems into one electricity transmission system spanning the whole globe. Power systems are currently forming larger and larger interconnections ...

Gemine, Q., Ernst, D., Louveaux, Q., & Cornélusse, B. (2014). Relaxations for multi-period optimal power flow problems with discrete decision variables.

*Proceedings of the 18th Power Systems Computation Conference (PSCC'14)*.Peer reviewed

We consider a class of optimal power flow (OPF) applications where some loads offer a modulation service in exchange for an activation fee. These applications can be modeled as multi-period formulations of the ...

Mathieu, S., Louveaux, Q., Ernst, D., & Cornélusse, B. (2014). A quantitative analysis of the effect of flexible loads on reserve markets.

*Proceedings of the 18th Power Systems Computation Conference (PSCC)*.Peer reviewed

We propose and analyze a day-ahead reserve market model that handles bids from flexible loads.
This pool market model takes into account the fact that a load modulation in one direction must usually be ...

Lorent, P. (2014).

*Sequential decision making under uncertainty in randomly generated games : A Minecraft intelligent agent for resource gathering*. Unpublished master thesis, Université de Liège, Liège, Belgique.Video games have significantly evolved since the emergence of first arcade games. They have
become more and more complex and now allow for the simulation of advanced environments.
This work focuses on the ...

Aristidou, P., Olivier, F., Hervas, M. E., Ernst, D., & Van Cutsem, T. (2014). Distributed Model-free Control of Photovoltaic Units for Mitigating Overvoltages in Low-Voltage Networks.

*Proc. of CIRED 2014 workshop*(pp. 0087).Peer reviewed

In this paper, a distributed model-free control scheme to mitigate overvoltage problems caused by high photovoltaic generation in low-voltage feeders is proposed. The distributed controllers are implemented on ...

Castronovo, M., Ernst, D., & Fonteneau, R. (2014). Bayes Adaptive Reinforcement Learning versus Off-line Prior-based Policy Search: an Empirical Comparison.

*Proceedings of the 23rd annual machine learning conference of Belgium and the Netherlands (BENELEARN 2014)*.Peer reviewed

This paper addresses the problem of decision making in unknown finite Markov decision processes (MDPs). The uncertainty about the MDPs is modeled using a prior distribution over a set of candidate MDPs. The ...

Sutera, A., Joly, A., François-Lavet, V., Qiu, Z., Louppe, G., Ernst, D., & Geurts, P. (2014). Simple connectome inference from partial correlation statistics in calcium imaging. In J., Soriano, D., Battaglia, I., Guyon, V., Lemaire, J., Orlandi, & B., Ray (Eds.),

*Neural Connectomics Challenge*. Springer.Peer reviewed

In this work, we propose a simple yet effective solution to the problem of connectome inference in calcium imaging data. The proposed algorithm consists of two steps. First, processing the raw signals to ...

Castronovo, M., Ernst, D., & Fonteneau, R. (2014). Apprentissage par renforcement bayésien versus recherche directe de politique hors-ligne en utilisant une distribution a priori: comparaison empirique.

*Proceedings des 9èmes Journée Francophones de Planification, Décision et Apprentissage*.Peer reviewed

Cet article aborde le problème de prise de décision séquentielle dans des processus de déci- sion de Markov (MDPs) finis et inconnus. L’absence de connaissance sur le MDP est modélisée sous la forme d’une ...

François-Lavet, V., Fonteneau, R., & Ernst, D. (2014). Estimating the revenues of a hydrogen-based high-capacity storage device: methodology and results.

*Proceedings des 9èmes Journée Francophones de Planification, Décision et Apprentissage*.Peer reviewed

This paper proposes a methodology to estimate the maximum revenue that can be generated by a company that operates a high-capacity storage device to buy or sell electricity on the day-ahead electricity market ...

Gemine, Q., Ernst, D., & Cornélusse, B. (2014). Gestion active d’un réseau de distribution d’électricité : formulation du problème et benchmark.

*Proceedings des 9èmes Journées Francophones de Planification, Décision et Apprentissage*.Peer reviewed

Afin d’opérer un réseau de distribution d’électricité de manière fiable et efficace, c’est-à-dire de respecter les contraintes physiques tout en évitant des coûts de renforcement prohibitifs, il devient ...

Sootla, A., Strelkowa, N., Ernst, D., Barahona, M., & Guy-Bart, S. (2014). Toggling a genetic switch using reinforcement learning.

*Proceedings of the 9th French Meeting on Planning, Decision Making and Learning*.Peer reviewed

In this paper, we consider the problem of optimal exogenous control of gene regulatory networks. Our approach consists in adapting an established reinforcement learning algorithm called the ﬁtted Q iteration ...

Jung, T., Wehenkel, L., Ernst, D., & Maes, F. (2014). Optimized look-ahead tree policies: a bridge between look-ahead tree policies and direct policy search.

*International Journal of Adaptive Control and Signal Processing, 28*(3-5), 255-289.Peer reviewed (verified by ORBi)

Direct policy search (DPS) and look-ahead tree (LT) policies are two popular techniques for solving difficult sequential decision-making problems. They both are simple to implement, widely applicable without ...

St-Pierre, D. L., Maes, F., Ernst, D., & Louveaux, Q. (2014). A learning procedure for sampling semantically different valid expressions.

*International Journal of Artificial Intelligence, 12*(1), 18-35.Peer reviewed

A large number of problems can be formalized as finding the best symbolic expression to maximize a given numerical objective. Most approaches to approximately solve such problems rely on random exploration of ...

Ernst, D. (2014, February). L'invité - Damien Ernst - "Nous allons vers une globalisation du marché de l'électricité".

*La revue Wallonie*, (120), p. 28-31.En décembre 2013, Damien Ernst, Professeur à l’ULG, a donné une conférence au CESW intitulée : «Vers une globalisation du marché de l’électricité. Quel rôle pour les acteurs du secteur belge de l’électricité?» ...

Fonteneau, R., Ernst, D., Boigelot, B., & Louveaux, Q. (2014). Lipschitz robust control from off-policy trajectories.

*Proceedings of the 53rd IEEE Conference on Decision and Control (IEEE CDC 2014)*.Peer reviewed

We study the minmax optimization problem introduced in [Fonteneau et al. (2011), ``Towards min max reinforcement learning'', Springer CCIS, vol. 129, pp. 61-77] for computing control policies for batch mode ...

Fonteneau, R., Murphy, S. A., Wehenkel, L., & Ernst, D. (2014). Apprentissage par renforcement batch fondé sur la reconstruction de trajectoires artificielles.

*Proceedings of the 9èmes Journées Francophones de Planification, Décision et Apprentissage (JFPDA 2014)*.Peer reviewed

Cet article se situe dans le cadre de l’apprentissage par renforcement en mode batch, dont le problème central est d’apprendre, à partir d’un ensemble de trajectoires, une politique de décision optimisant un ...

Ruiz-Vega, D., Wehenkel, L., Ernst, D., Pizano-Martinez, A., & Fuerte-Esquivel, C. (2014). Power system transient stability preventive and emergency control. In S., Savulescu (Ed.),

*Real-Time Stability in Power Systems 2nd Edition*(pp. 123-158). Springer.A general approach to real-time transient stability control is described, yielding various complementary techniques: pure preventive, open loop emergency, and closed loop emergency controls. Recent progress in ...

Sootla, A., Strelkowa, N., Ernst, D., Barahona, M., & Stan, G.-B. (2013). On periodic reference tracking using batch-mode reinforcement learning with application to gene regulatory network control.

*Proceedings of the 52nd Annual Conference on Decision and Control (CDC 2013)*(pp. 4086-4091).Peer reviewed

In this paper, we consider the periodic reference tracking problem in the framework of batch-mode reinforcement learning, which studies methods for solving optimal control problems from the sole knowledge of a ...

Mathieu, S., Ernst, D., & Louveaux, Q. (2013). An efficient algorithm for the provision of a day-ahead modulation service by a load aggregator.

*Proceedings of the 4th European Innovative Smart Grid Technologies (ISGT)*.Peer reviewed

This article studies a decision making problem faced by an aggregator willing to offer a load modulation service to a Transmission System Operator. This service is contracted one day ahead and consists in a ...

Chatzivasileiadis, S., Ernst, D., & Andersson, G. (2013). The global grid.

*Renewable Energy, 57*, 372-383.Peer reviewed (verified by ORBi)

This paper puts forward the vision that a natural future stage of the electricity network could be a grid spanning the whole planet and connecting most of the large power plants in the world: this is the ...

Maes, F., Lupien St-Pierre, D., & Ernst, D. (2013). Monte Carlo search algorithm discovery for single-player games.

*IEEE Transactions on Computational Intelligence and AI in Games, 5*(3), 201-213.Peer reviewed

Much current research in AI and games is being devoted to Monte Carlo search (MCS) algorithms. While the quest for a single uniﬁed MCS algorithm that would perform well on all problems is of major interest for ...

Lupien St-Pierre, D. (2013).

*Contributions to Monte Carlo Search*. Unpublished doctoral thesis, Université de Liège, Liège, Belgique.This research is motivated by improving decision making under uncertainty and in particular for games and symbolic regression. The present dissertation gathers research contributions in the field of Monte ...

Olivier, F. (2013).

*Integration of Photovoltaic Panels Into Low-voltage Distribution Networks*. Unpublished master thesis, Université de Liège, Liège, Belgique.We address in this thesis the problem of overvoltages caused by photovoltaic (PV) panels in low-voltage networks. As a solution to this problem, we propose new active network management schemes. They work by ...

Bubeck, S., Ernst, D., & Garivier, A. (2013). Optimal discovery with probabilistic expert advice: finite time analysis and macroscopic optimality.

*Journal of Machine Learning Research, 14*, 601-623.Peer reviewed (verified by ORBi)

We consider an original problem that arises from the issue of security analysis of a power system and that we name optimal discovery with probabilistic expert advice. We address it with an algorithm based on ...

Jung, T., Martin, S., Nassar, M., Ernst, D., & Leduc, G. (2013). Outbound SPIT Filter with Optimal Performance Guarantees.

*Computer Networks, 57*(7), 1630–1643.Peer reviewed (verified by ORBi)

This paper presents a formal framework for identifying and filtering SPIT calls (SPam in Internet Telephony) in an outbound scenario with provable optimal performance. In so doing, our work is largely ...

Defourny, B., Ernst, D., & Wehenkel, L. (2013). Scenario Trees and Policy Selection for Multistage Stochastic Programming Using Machine Learning.

*INFORMS Journal on Computing, 25*(3), 488-501.Peer reviewed (verified by ORBi)

In the context of multistage stochastic optimization problems, we propose a hybrid strategy for generalizing to nonlinear decision rules, using machine learning, a finite data set of constrained vector-valued ...

Fonteneau, R., Ernst, D., Boigelot, B., & Louveaux, Q. (2013). Généralisation Min Max pour l'Apprentissage par Renforcement Batch et Déterministe : Relaxations pour le Cas Général T Etapes.

*8èmes Journées Francophones de Planification, Décision et Apprentissage pour la conduite de systèmes (JFPDA'13)*.Peer reviewed

Cet article aborde le problème de généralisation minmax dans le cadre de l'apprentissage par renforcement batch et déterministe. Le problème a été originellement introduit par [Fonteneau, 2011], et il a déjà ...

Fonteneau, R., Ernst, D., Boigelot, B., & Louveaux, Q. (2013). Min max generalization for deterministic batch mode reinforcement learning: relaxation schemes.

*SIAM Journal on Control and Optimization, 51*(5), 3355–3385.Peer reviewed (verified by ORBi)

We study the min max optimization problem introduced in Fonteneau et al. [Towards min max reinforcement learning, ICAART 2010, Springer, Heidelberg, 2011, pp. 61–77] for computing policies for batch mode ...

Fonteneau, R., Murphy, S. A., Wehenkel, L., & Ernst, D. (2013). Stratégies d'échantillonnage pour l'apprentissage par renforcement batch.

*Revue d'Intelligence Artificielle, 27*(2), 171-194.Peer reviewed (verified by ORBi)

We propose two strategies for experiment selection in the context of batch mode reinforcement learning. The ﬁrst strategy is based on the idea that the most interesting experiments to carry out at some stage ...

Gemine, Q., Karangelos, E., Ernst, D., & Cornélusse, B. (2013). Active network management: planning under uncertainty for exploiting load modulation.

*Proceedings of the 2013 IREP Symposium - Bulk Power Systems Dynamics and Control - IX*. IEEE.Peer reviewed

This paper addresses the problem faced by a distribution system operator (DSO) when planning the operation of a network in the short-term. The problem is formulated in the context of high penetration of ...

Jung, T., Ernst, D., & Maes, F. (2013). Optimized Look-Ahead Trees: Extensions to Large and Continuous Action Spaces.

*Proc. of IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL'13)*.Peer reviewed

This paper studies look-ahead tree based control policies from the viewpoint of online decision making with constraints on the computational budget allowed per decision (expressed as number of calls to the ...

Maes, F., Wehenkel, L., & Ernst, D. (2013). Meta-learning of Exploration/Exploitation Strategies: The Multi-Armed Bandit Case. In J., Filipe & A., Fred (Eds.),

*Agents and Artificial Intelligence: 4th International Conference, ICAART 2012, Vilamoura, Portugal, February 6-8, 2012. Revised Selected Papers*(pp. 110-115). Springer.Peer reviewed

The exploration/exploitation (E/E) dilemma arises naturally in many subﬁelds of Science. Multi-armed bandit problems formalize this dilemma in its canonical form. Most current research in this ﬁeld focuses on ...

Jung, T., & Ernst, D. (2012, December 07).

*Biorthogonalization Techniques for Least Squares Temporal Difference Learning*. Poster session presented at Neural Information Processing Systems (NIPS), South Lake Tahoe, NV, USA.Peer reviewed

We consider Markov reward processes and study OLS-LSTD, a framework for selecting basis functions from a set of candidates to obtain a sparse representation of the value function in the context of least ...

Bubeck, S., Ernst, D., & Garivier, A. (2012). Optimal discovery with probabilistic expert advice.

*Proceedings of the 51st IEEE Conference on Decision and Control (CDC 2012)*.Peer reviewed

Motivated by issues of security analysis for power systems, we analyze a new problem, called optimal discovery with probabilistic expert advice. We address it with an algorithm based on the optimistic paradigm ...

Sarlette, A., Dai, J., Phulpin, Y., & Ernst, D. (2012). Cooperative frequency control with a multi-terminal high-voltage DC network.

*Automatica, 48*(12), 3128–3134.Peer reviewed (verified by ORBi)

We consider frequency control in power systems made of several non-synchronous AC areas connected by a multi-terminal high-voltage direct current (HVDC) grid. We propose two HVDC control schemes to make the ...

Mathieu, S., Karangelos, E., Louveaux, Q., & Ernst, D. (2012, October 08).

*A computationally efficient algorithm for the provision of a day-ahead modulation service by a load aggregator*. Poster session presented at DYSCO Study Day : Dynamical systems, control and optimization Kickoff of phase VII.We study a decision making problem faced by an aggregator willing to offer a load modulation service to a Transmission System Operator (TSO). In particular, we concentrate on a day-ahead service consisting of ...

Maes, F., Fonteneau, R., Wehenkel, L., & Ernst, D. (2012). Policy search in a space of simple closed-form formulas: towards interpretability of reinforcement learning.

*Discovery Science 15th International Conference, DS 2012, Lyon, France, October 29-31, 2012. Proceedings*(pp. 37-51). Berlin, Germany: Springer.Peer reviewed

In this paper, we address the problem of computing interpretable solutions to reinforcement learning (RL) problems. To this end, we propose a search algorithm over a space of simple losed-form formulas that ...

Jung, T., Martin, S., Ernst, D., & Leduc, G. (2012).

*Contextual Multi-armed Bandits for the Prevention of Spam in VoIP Networks*. Eprint/Working paper retrieved from http://orbi.ulg.ac.be/handle/2268/115524.In this paper we argue that contextual multi-armed bandit algorithms could open avenues for designing self-learning security modules for computer networks and related tasks. The paper has two contributions: a ...

Castronovo, M. (2012).

*Learning for exploration/exploitation in reinforcement learning*. Unpublished master thesis, Université de Liège, Liège, Belgique.We consider the problem of learning high-performance Exploration/Exploitation (E/E)
strategies for finite Markov Decision Processes (MDPs) when the MDP to be controlled
is supposed to be drawn from a known ...

Gemine, Q. (2012).

*Imitative learning for designing intelligent agents for video games*. Unpublished master thesis, Université de Liège, Liège, Belgique.Over the past decades, video games have become increasingly popular and complex. Virtual worlds have gone a long way since the first arcades and so have the artificial intelligence (AI) techniques used to ...

Fonteneau, R., Ernst, D., Boigelot, B., & Louveaux, Q. (2012). Généralisation min max pour l'apprentissage par renforcement batch et déterministe : schémas de relaxation.

*Septièmes Journées Francophones de Planification, Décision et Apprentissage pour la conduite de systèmes (JFPDA 2012)*.Peer reviewed

On s’intéresse au problème de généralisation min max dans le cadre de l’apprentissage par renforcement batch et déterministe. Le problème a été originellement introduit par Fonteneau et al. (2011). Dans un ...

Fonteneau-Belmudes, F. (2012).

*Identification of dangerous contingencies for large scale power system security assessment*. Unpublished doctoral thesis, Université de Liège, Belgique.This thesis presents an approach for identifying a maximal number of dangerous contingencies in large scale power system security assessment problems with bounded computational resources.
The method ...

Dai, J., Phulpin, Y., Sarlette, A., & Ernst, D. (2012). Coordinated primary frequency control among non-synchronous systems connected by a multi-terminal high-voltage direct current grid.

*IET Generation, Transmission and Distribution, 6*(2), 99-108.Peer reviewed (verified by ORBi)

The authors consider a power system composed of several non-synchronous AC areas connected by a multiterminal high-voltage direct current (HVDC) grid. In this context, the authors propose a distributed control ...

Maes, F., Wehenkel, L., & Ernst, D. (2012). Learning to play K-armed bandit problems.

*Proceedings of the 4th International Conference on Agents and Artificial Intelligence (ICAART 2012)*.Peer reviewed

We propose a learning approach to pre-compute K-armed bandit playing policies by exploiting prior information describing the class of problems targeted by the player. Our algorithm ﬁrst samples a set of K ...

Castronovo, M., Maes, F., Fonteneau, R., & Ernst, D. (2012). Learning exploration/exploitation strategies for single trajectory reinforcement learning.

*Proceedings of the 10th European Workshop on Reinforcement Learning (EWRL 2012)*(pp. 1-9).Peer reviewed

We consider the problem of learning high-performance Exploration/Exploitation (E/E) strategies for finite Markov Decision Processes (MDPs) when the MDP to be controlled is supposed to be drawn from a known ...

Gemine, Q., Safadi, F., Fonteneau, R., & Ernst, D. (2012). Imitative Learning for Real-Time Strategy Games.

*Proceedings of the 2012 IEEE Conference on Computational Intelligence and Games*(pp. 424-429).Peer reviewed

Over the past decades, video games have become increasingly popular and complex. Virtual worlds have gone a long way since the first arcades and so have the artificial intelligence (AI) techniques used to ...

Jung, T., Martin, S., Ernst, D., & Leduc, G. (2012). Contextual Multi-armed Bandits for Web Server Defense. In A., Hussein (Ed.),

*Proceedings of 2012 International Joint Conference on Neural Networks (IJCNN)*(pp. 8). IEEE.Peer reviewed

In this paper we argue that contextual multi-armed bandit algorithms
could open avenues for designing self-learning security modules for
computer networks and related tasks. The paper has two contributions ...

Jung, T., Martin, S., Ernst, D., & Leduc, G. (2012). SPRT for SPIT: Using the Sequential Probability Ratio Test for Spam in VoIP Prevention.

*Proc. of 6th International Conference on Autonomous Infrastructure, Management and Security*. Springer Berlin / Heidelberg.Peer reviewed

This paper presents the first formal framework for identifying and filtering SPIT calls (SPam in Internet Telephony) in an outbound scenario with provable optimal performance. In so doing, our work deviates ...

Perrick, P., Lupien St-Pierre, D., Maes, F., & Ernst, D. (2012). Comparison of Different Selection Strategies in Monte-Carlo Tree Search for the Game of Tron.

*IEEE Conference on Computational and Intelligence in Games 2012*(pp. 242-249).Peer reviewed

Monte-Carlo Tree Search (MCTS) techniques are essentially known for their performance on turn-based games, such as Go, for which players have considerable time for choosing their moves. In this paper, we apply ...

Fonteneau, R., Ernst, D., Boigelot, B., & Louveaux, Q. (2011). Relaxation schemes for min max generalization in deterministic batch mode reinforcement learning.

*4th International NIPS Workshop on Optimization for Machine Learning (OPT 2011)*.Peer reviewed

We study the min max optimization problem introduced in [Fonteneau, 2011] for computing policies for batch mode reinforcement learning in a deterministic setting. This problem is NP-hard. We focus on the two ...

Safadi, F., Fonteneau, R., & Ernst, D. (2011). Artificial intelligence design for real-time strategy games.

*NIPS Workshop on Decision Making with Multiple Imperfect Decision Makers*.Peer reviewed

For now over a decade, real-time strategy (RTS) games have been challenging intelligence, human and artificial (AI) alike, as one of the top genre in terms of overall complexity. RTS is a prime example problem ...

Phulpin, Y., & Ernst, D. (2011). Ancillary services and operation of multi-terminal HVDC grids.

*Proceedings of the International Workshop on Transmission Networks for Offshore Wind Power as well as on Transmission Networks for Offshore Wind Power Farms Plants*.Peer reviewed

This paper addresses the problem of ancillary services in ac systems interconnected by a multi-terminal HVdc system. It presents opportunities for new control schemes and discusses operation strategies for ...

Phulpin, Y., Hazra, J., & Ernst, D. (2011). Model predictive control of HVDC power ﬂow to improve transient stability in power systems.

*Proceedings of the Second IEEE International Conference on Smart Grid Communications (IEEE SmartGridComm)*(pp. 611-616).Peer reviewed

This paper addresses the problem of HVDC control using real-time information to avoid loss of synchronism phenomena in power systems. It proposes a discrete-time control strategy based on model predictive ...

Busoniu, L., Babuska, R., De Schutter, B., & Ernst, D. (2011). Approximate reinforcement learning: an overview.

*Proceedings of the 2011 IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL-11)*.Peer reviewed

Reinforcement learning (RL) allows agents to learn how to optimally interact with complex environments. Fueled by recent advances in approximation-based algorithms, RL has obtained impressive successes in ...

Fonteneau, R., Murphy, S., Wehenkel, L., & Ernst, D. (2011). Active exploration by searching for experiments that falsify the computed control policy.

*Proceedings of the 2011 IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL-11)*.Peer reviewed

We propose a strategy for experiment selection - in the context of reinforcement learning - based on the idea that the most interesting experiments to carry out at some stage are those that are the most liable ...

Fonteneau, R. (2011).

*Contributions to Batch Mode Reinforcement Learning*. Unpublished doctoral thesis, Université de Liège, Liège, Belgium.This dissertation presents various research contributions published during these four years of PhD in the ﬁeld of batch mode reinforcement learning, which studies optimal control problems for which the only ...

Busoniu, L., Ernst, D., Babuska, R., & De Schutter, B. (2011). Cross-entropy optimization of control policies with adaptive basis functions.

*IEEE Transactions on Systems, Man and Cybernetics. Part B, Cybernetics, 41*(1), 196-209.Peer reviewed (verified by ORBi)

This paper introduces an algorithm for direct search of control policies in continuous-state, discrete-action Markov decision processes. The algorithm looks for the best closed-loop policy that can be ...

Defourny, B., Ernst, D., & Wehenkel, L. (2011). Multistage stochastic programming: A scenario tree based approach to planning under uncertainty. In L. E., Sucar, E. F., Morales, & J., Hoey (Eds.),

*Decision Theory Models for Applications in Artificial Intelligence: Concepts and Solutions*. Hershey, PA: Information Science Publishing.Peer reviewed

In this chapter, we present the multistage stochastic programming framework for sequential decision making under uncertainty. We discuss its differences with Markov Decision Processes, from the point of view ...

Fonteneau, R., Murphy, S., Wehenkel, L., & Ernst, D. (2011). Towards min max generalization in reinforcement learning. In J., Filipe, A., Fred, & B., Sharp (Eds.),

*Agents and Artificial Intelligence: International Conference, ICAART 2010, Valencia, Spain, January 2010, Revised Selected Papers*(pp. 61-77). Springer.Peer reviewed

In this paper, we introduce a min max approach for addressing the generalization problem in Reinforcement Learning. The min max approach works by determining a sequence of actions that maximizes the worst ...

Jing, D., Phulpin, Y., Sarlette, A., & Ernst, D. (2011). Voltage control in an HVDC system to share primary frequency reserves between non-synchronous areas.

*Proceedings of the 17th Power Systems Computation Conference (PSCC-11)*.Peer reviewed

This paper addresses the problem of frequency control for non-synchronous AC areas connected by a multi-terminal HVDC grid. It proposes a decentralized control scheme for the DC voltages of the HVDC converters ...

Maes, F., Wehenkel, L., & Ernst, D. (2011). Automatic discovery of ranking formulas for playing with multi-armed bandits.

*Proceedings of the 9th European Workshop on Reinforcement Learning (EWRL 2011)*.Peer reviewed

We propose an approach for discovering in an automatic way formulas for ranking arms while playing with multi-armed bandits. The approach works by de ning a grammar made of basic elements such as for example ...

Maes, F., Wehenkel, L., & Ernst, D. (2011). Optimized look-ahead tree policies.

*Proceedings of the 9th European Workshop on Reinforcement Learning (EWRL 2011)*.Peer reviewed

We consider in this paper look-ahead tree techniques for the discrete-time control of a deterministic dynamical system so as to maximize a sum of discounted rewards over an in finite time horizon. Given the ...

Rachelson, E., Schnitzler, F., Wehenkel, L., & Ernst, D. (2011). Optimal sample selection for batch-mode reinforcement learning.

*Proceedings of the 3rd International Conference on Agents and Artificial Intelligence (ICAART 2011)*.Peer reviewed

We introduce the Optimal Sample Selection (OSS) meta-algorithm for solving discrete-time Optimal Control problems. This meta-algorithm maps the problem of ﬁnding a near-optimal closed-loop policy to the ...

Dai, J., Phulpin, Y., Sarlette, A., & Ernst, D. (2010). Impact of delays on a consensus-based primary frequency control scheme for AC systems connected by a multi-terminal HVDC grid.

*Proceedings of the 2010 IREP Symposium - Bulk Power Systems Dynamics and Control - VIII*.Peer reviewed

This paper addresses the problem of sharing primary frequency control reserves among nonsynchronous AC systems connected by a multi-terminal HVDC grid. We focus on a control scheme that modifies the power ...

Fonteneau, F., Ernst, D., Druet, C., Panciatici, P., & Wehenkel, L. (2010). Consequence driven decomposition of large-scale power system security analysis.

*Proceedings of the 2010 IREP Symposium - Bulk Power Systems Dynamics and Control - VIII*.Peer reviewed

This paper presents an approach for assessing, in operation planning studies, the security of a large-scale power system by decomposing it into elementary subproblems, each one corresponding to a structural ...

Phulpin, Y., Begovic, M., & Ernst, D. (2010). Coordination of voltage control in a power system operated by multiple transmission utilities.

*Proceedings of the 2010 IREP Symposium - Bulk Power Systems Dynamics and Control - VIII*.Peer reviewed

This paper addresses the problem of coordinating voltage control in a large-scale power system partitioned into control areas operated by independent utilities. Two types of coordination modes are considered ...

Busoniu, L., De Schutter, B., Babuska, R., & Ernst, D. (2010). Using prior knowledge to accelerate online least-squares policy iteration.

*Proceedings of the 2010 IEEE International Conference on Automation, Quality and Testing, Robotics*.Peer reviewed

Reinforcement learning (RL) is a promising paradigm for learning optimal control. Although RL is generally envisioned as working without any prior knowledge about the system, such knowledge is often available ...

Busoniu, L., Ernst, D., De Schutter, B., & Robert, B. (2010). Approximate dynamic programming with a fuzzy parameterization.

*Automatica, 46*(5), 804-814.Peer reviewed (verified by ORBi)

Dynamic programming (DP) is a powerful paradigm for general, nonlinear optimal control. Computing exact DP solutions is in general only possible when the process states and the control actions take values in a ...

Fonteneau, R., Murphy, S., Wehenkel, L., & Ernst, D. (2010). Generating informative trajectories by using bounds on the return of control policies.

*Proceedings of the Workshop on Active Learning and Experimental Design 2010 (in conjunction with AISTATS 2010)*.Peer reviewed

We propose new methods for guiding the generation of informative trajectories when solving discrete-time optimal control problems. These methods exploit recently published results that provide ways for ...

Fonteneau, R., Murphy, S., Wehenkel, L., & Ernst, D. (2010). Model-free Monte Carlo-like policy evaluation.

*Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010)*(pp. 217-224).Peer reviewed

We propose an algorithm for estimating the finite-horizon expected return of a closed loop control policy from an a priori given (off-policy) sample of one-step transitions. It averages cumulated rewards along ...

Fonteneau, R., Murphy, S., Wehenkel, L., & Ernst, D. (2010). Model-free Monte Carlo–like policy evaluation.

*Proceedings of Conférence Francophone sur l'Apprentissage Automatique (CAp) 2010*.Peer reviewed

We propose an algorithm for estimating the finite-horizon expected return of a closed loop control policy from an a priori given (off-policy) sample of one-step transitions. It averages cumulated rewards along ...

jouini, W., Ernst, D., Moy, C., & Palicot, J. (2010). Upper confidence bound based decision making strategies and dynamic spectrum access.

*Proceedings of the 2010 IEEE International Conference on Communications*.Peer reviewed

In this paper, we consider the problem of exploiting spectrum resources for a secondary user (SU) of a wireless communication network. We suggest that Upper Confidence Bound (UCB) algorithms could be useful to ...

Fonteneau, R., Murphy, S., Wehenkel, L., & Ernst, D. (2010). A cautious approach to generalization in reinforcement learning.

*Proceedings of the 2nd International Conference on Agents and Artificial Intelligence*(pp. 10).Peer reviewed

In the context of a deterministic Lipschitz continuous environment over continuous state spaces, finite action spaces, and a finite optimization horizon, we propose an algorithm of polynomial complexity which ...

Busoniu, L., Ernst, D., Babusku, R., & De Schutter, B. (2010). Exploiting policy knowledge in online least-squares policy iteration: An empirical study.

*Automation, Computers, Applied Mathematics, 19*(4), 521-529.Peer reviewed (verified by ORBi)

Reinforcement learning (RL) is a promising paradigm for learning optimal control. Traditional RL works for discrete variables only, so to deal with the continuous variables appearing in control problems ...

Busoniu, L., Ernst, D., De Schutter, B., & Babuska, R. (2010). Online least-squares policy iteration for reinforcement learning control.

*Proceedings of the 2010 American Control Conference*(pp. 486-491).Peer reviewed

Reinforcement learning is a promising paradigm for learning optimal control. We consider policy iteration (PI) algorithms for reinforcement learning, which iteratively evaluate and improve control policies ...

Fonteneau, R., & Ernst, D. (2010).

*Voronoi model learning for batch mode reinforcement learning*. University of Liège.We consider deterministic optimal control problems with continuous state spaces where the information on the system dynamics and the reward function is constrained to a set of system transitions. Each system ...

Fonteneau, R., Murphy, S. A., Wehenkel, L., & Ernst, D. (2010).

*Computing bounds for kernel-based policy evaluation in reinforcement learning*. University of Liège.This technical report proposes an approach for computing bounds on the finite-time return of a policy using kernel-based approximators from a sample of trajectories in a continuous state space and deterministic framework.

Jouini, W., Ernst, D., Moy, C., & Palicot, J. (2009). Multi-armed bandit based policies for cognitive radio's decision making issues.

*Proceedings of the 3rd International Conference on Signals, Circuits and Systems (SCS)*.Peer reviewed

We suggest in this paper that many problems related to Cognitive Radio’s (CR) decision making inside CR equipments can be formalized as Multi-Armed Bandit problems and that solving such problems by using Upper ...

Mhawej, M.-J., Brunet-Francois, C., Fonteneau, R., Ernst, D., Ferré, V., Stan, G.-B., Raffi, F., & Moog, C. H. (2009). Apoptosis characterizes immunological failure of HIV infected patients.

*Control Engineering Practice, 17*(7), 798-804.Peer reviewed (verified by ORBi)

This paper studies the influence of apoptosis in the dynamics of the HIV infection. A new modeling of the healthy CD4+ T-cells activation-induced apoptosis is used. The parameters of this model are identified ...

Belmudes, F., Ernst, D., & Wehenkel, L. (2009). Pseudo-geographical representations of power system buses by multidimensional scaling.

*Proceedings of the 15th International Conference on Intelligent System Applications to Power Systems (ISAP 2009)*.Peer reviewed

Graphical representations of power systems are systematically used for planning and operation. The coordinate systems commonly used by Transmission System Operators are static and reflect the geographical ...

Belmudes, F., Ernst, D., & Wehenkel, L. (2009). A rare-event approach to build security analysis tools when N-k (k > 1) analyses are needed (as they are in large-scale power systems).

*Proceedings of the 2009 IEEE Bucharest PowerTech*.Peer reviewed

We consider the problem of performing N − k security analyses in large scale power systems. In such a context, the number of potentially dangerous N − k contingencies may become rapidly very large when k grows ...

Busoniu, L., Ernst, D., De Schutter, B., & Babuska, R. (2009). Policy search with cross-entropy optimization of basis functions.

*Proceedings of the IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL-09)*(pp. 153-160).Peer reviewed

This paper introduces a novel algorithm for approximate policy search in continuous-state, discrete-action Markov decision processes (MDPs). Previous policy search approaches have typically used ad-hoc ...

Dai, J., Phulpin, Y., Vannier, J.-C., & Ernst, D. (2009). Apprentissage par renforcement appliqué à la commande des systèmes électriques.

*Proceedings of "Les Journées Electrotechnique du Futur 2009"*.Peer reviewed

Cet article propose une revue de littérature concernant les applications de l’apprentissage par renforcement à la commande des systèmes électriques. L'apprentissage par renforcement a pour caractéristique ...

Defourny, B., Ernst, D., & Wehenkel, L. (2009). Bounds for Multistage Stochastic Programs using Supervised Learning Strategies. In O., Watanabe & T., Zeugmann (Eds.),

*Stochastic Algorithms: Foundations and Applications*(pp. 61-73). Springer.Peer reviewed

We propose a generic method for obtaining quickly good upper bounds on the minimal value of a multistage stochastic program. The method is based on the simulation of a feasible decision policy, synthesized by ...

Defourny, B., Ernst, D., & Wehenkel, L. (2009). Planning under uncertainty, ensembles of disturbance trees and kernelized discrete action spaces.

*Proceedings of the IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL-09)*(pp. 145-152).Peer reviewed

Optimizing decisions on an ensemble of incomplete disturbance trees and aggregating their first stage decisions has been shown as a promising approach to (model-based) planning under uncertainty in large ...

Ernst, D., Glavic, M., Capitanescu, F., & Wehenkel, L. (2009). Reinforcement learning versus model predictive control: a comparison on a power system problem.

*IEEE Transactions on Systems, Man and Cybernetics. Part B, Cybernetics, 33*(2), 517-519.Peer reviewed (verified by ORBi)

This paper compares reinforcement learning (RL) with model predictive control (MPC) in a unified framework and reports experimental results of their application to the synthesis of a controller for a nonlinear ...

Ernst, D., Wehenkel, L., & Pavella, M. (2009). What is the likely future of real-time transient stability ?

*Proceedings of the 2009 IEEE/PES Power Systems Conference & Exposition (PSCE 2009)*.Peer reviewed

Despite very intensive research efforts in the field of transient stability during the last five decades, the large majority of the derived techniques have hardly moved from the research laboratories to the ...

Fonteneau, R., Murphy, S., Wehenkel, L., & Ernst, D. (2009). Inferring bounds on the performance of a control policy from a sample of trajectories.

*Proceedings of the IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL-09)*(pp. 117-123).Peer reviewed

We propose an approach for inferring bounds on the finite-horizon return of a control policy from an off-policy sample of trajectories collecting state transitions, rewards, and control actions. In this paper ...

Hazra, J., Phulpin, Y., & Ernst, D. (2009). HVDC control strategies to improve transient stability in interconnected power systems.

*Proceedings of the 2009 IEEE Bucharest PowerTech*.Peer reviewed

This paper presents three HVDC modulation strategies to improve transient stability in an interconnected power system. AC variables such as rotor speeds, voltage phasors, and tieline power flows are used as ...

Phulpin, Y., Begovic, M., Petit, M., & Ernst, D. (2009). Decentralized reactive power dispatch for a time-varying multi-TSO system.

*Proceedings of the 42nd Hawaii International Conference on System Sciences (HICSS-42)*.Peer reviewed

This paper addresses the problem of reactive power dispatch in a power system partitioned into several areas controlled by different transmission system operators. Previous research has shown that nearly ...

Phulpin, Y., Begovic, M., Petit, M., & Ernst, D. (2009). A fair method for centralized optimization of multi-TSO power systems.

*International Journal of Electrical Power and Energy Systems, 31*, 482-488.Peer reviewed (verified by ORBi)

This paper addresses the problem of centralized optimization of an interconnected power system partitioned into several regions controlled by different transmission system operators (TSOs). It is assumed ...

Phulpin, Y., Miroslav, B., Petit, M., Heyberger, J.-B., & Ernst, D. (2009). Evaluation of network equivalents for voltage optimization in multi-area power systems.

*IEEE Transactions on Power Systems, 24*(2), 729-743.Peer reviewed (verified by ORBi)

The paper addresses the problem of decentralized optimization for a power system partitioned into several areas controlled by different transmission system operators (TSOs). The optimization variables are the ...

Wehenkel, L., Ernst, D., Rousseaux, P., & Van Cutsem, T. (2008, December). Research and Education Activities in Electric Power Systems at the University of Liège.

*Revue E: Revue d'Electricité et d'Electronique Industrielle*, (4), 54-59.Peer reviewed

This paper presents research and education activities of the power systems group of the Department of Electrical Engineering and Computer Science of the University of Liège. These activities cover power ...

Stan, G.-B., Belmudes, F., Fonteneau, R., Zeggwagh, F., Lefebvre, M.-A., Michelet, C., & Ernst, D. (2008). Modelling the influence of activation-induced apoptosis of CD4+ and CD8+ T-cells on the immune system response of a HIV-infected patient.

*IET Systems Biology, 2*(2), 94-102.Peer reviewed (verified by ORBi)

On the basis of the human immunodeficiency virus (HIV) infection dynamics model proposed by Adams, the authors propose an extended model that aims at incorporating the influence of activation-induced apoptosis ...

Ali, M., Glavic, M., Buisson, J., Wehenkel, L., & Ernst, D. (2008). Analyzing transient instability phenomena beyond the classical stability boundary.

*Proceedings of the 40th North American Power Symposium (NAPS 2008)*.Peer reviewed

We consider power systems for which the amount of power produced by their individual power plants is small with respect to the total generation of the system, and analyze how the transient instability ...

Belmudes, F., Ernst, D., & Wehenkel, L. (2008). Cross-entropy based rare-event simulation for the identification of dangerous events in power systems.

*Proceedings of the 10th International Conference on Probabilistic Methods Applied to Power Systems (PMAPS-08)*.Peer reviewed

We propose in this paper a novel approach for identifying rare events that may endanger power system integrity. This approach is inspired by the rare-event simulation literature and, in particular, by the ...

Busoniu, L., Ernst, D., Babuska, R., & De Schutter, B. (2008). Consistency of fuzzy model-based reinforcement learning.

*Proceedings of the 2008 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE-08)*(pp. 518-524).Peer reviewed

Reinforcement learning (RL) is a widely used paradigm for learning control. Computing exact RL solutions is generally only possible when process states and control actions take values in a small discrete set ...

Busoniu, L., Ernst, D., Babuska, R., & De Schutter, B. (2008). Fuzzy partition optimization for approximate fuzzy Q-iteration.

*Proceedings of the 17th IFAC World Congress (IFAC-08)*.Peer reviewed

Reinforcement learning (RL) is a widely used learning paradigm for adaptive agents. Because exact RL can only be applied to very simple problems, approximate algorithms are usually necessary in practice. Many ...

Busoniu, L., Ernst, D., De Schutter, B., & Babuska, R. (2008). Continuous-state reinforcement learning with fuzzy approximation. In K., Tuyls, A., Nowé, Z., Guessoum, & D., Kudenko (Eds.),

*Adaptive Agents and Multi-Agent Systems III, Adaptation and Multi-Agent Learning*(pp. 27-43).Peer reviewed

Reinforcement learning (RL) is a widely used learning paradigm for adaptive agents. There exist several convergent and consistent RL algorithms which have been intensively studied. In their original form ...

Defourny, B., Ernst, D., & Wehenkel, L. (2008). Lazy planning under uncertainty by optimizing decisions on an ensemble of incomplete disturbance trees. In B., Defourny, D., Ernst, & L., Wehenkel,

*Recent Advances in Reinforcement Learning*(pp. 1-14).Peer reviewed

This paper addresses the problem of solving discrete-time optimal sequential decision making problems having a disturbance space W composed of a finite number of elements. In this context, the problem of ...

Defourny, B., Ernst, D., & Wehenkel, L. (2008).

*Risk-aware decision making and dynamic programming*. Paper presented at NIPS-08 Workshop on Model Uncertainty and Risk in Reinforcement Learning, Whistler, Canada.Peer reviewed

This paper considers sequential decision making problems under uncertainty, the tradeoff between the expected return and the risk of high loss, and methods that use dynamic programming to find optimal policies ...

Fonteneau, R., Wehenkel, L., & Ernst, D. (2008).

*Variable selection for dynamic treatment regimes: a reinforcement learning approach*. Paper presented at European Workshop on Reinforcement Learning 2008 (EWRL'08), Villeneuve d'Ascq, France.Peer reviewed

Dynamic treatment regimes (DTRs) can be inferred from data collected through some randomized clinical trials by using reinforcement learning algorithms. During these clinical trials, a large set of clinical ...

Jing, D., Phulpin, Y., Rious, V., & Ernst, D. (2008). How compatible is perfect competition with transmission loss allocation methods?

*Proceedings of the 5th International Conference on the European Electricity Market (EEM-08)*.Peer reviewed

This paper addresses the problem of transmission loss allocation in a power system where the generators, the demands and the system operator are independent. We suppose that the transmission losses are ...

Phulpin, Y., Begovic, M., Petit, M., & Ernst, D. (2008). On the fairness of centralised decision-making strategies in multi-area power systems.

*Proceedings of the 16th Power Systems Computation Conference (PSCC-08)*.Peer reviewed

In this paper, we consider an interconnected power system, where the different Transmission System Operators (TSOs) have agreed to transferring some of their competences to a Centralised Control Center (CCC ...

Capitanescu, F., Glavic, M., Ernst, D., & Wehenkel, L. (2007). Contingency filtering techniques for preventive security-constrained optimal power flow.

*IEEE Transactions on Power Systems, 22*(4), 1690-1697.Peer reviewed (verified by ORBi)

This paper focuses on contingency filtering to accelerate the iterative solution of preventive security-constrained optimal power flow (PSCOPF) problems. To this end, we propose two novel filtering techniques ...

Del Angel, A., Geurts, P., Ernst, D., Glavic, M., & Wehenkel, L. (2007). Estimation of rotor angles of synchronous machines using artificial neural networks and local PMU-based quantities.

*Neurocomputing, 70*(16-18), 2668-2678.Peer reviewed (verified by ORBi)

This paper investigates a possibility for estimating rotor angles in the time frame of transient (angle) stability of electric power systems, for use in real-time. The proposed dynamic state estimation ...

Capitanescu, F., Glavic, M., Ernst, D., & Wehenkel, L. (2007). Interior-point based algorithms for the solution of optimal power flow problems.

*Electric Power Systems Research, 77*(5-6), 508-517.Peer reviewed (verified by ORBi)

Interior-point method (IPM) is a very appealing approach to the optimal power flow (OPF) problem mainly due to its speed of convergence and ease of handling inequality constraints. This paper analyzes the ...

Beck, E. V., Cherkaoui, R., Minoia, A., & Ernst, D. (2007). Nash equilibrium as the minimum of a function. Application to electricity markets with large number of actors.

*Proceedings of the 2007 Power Tech*(pp. 837-842).Peer reviewed

We introduce in this paper a new approach for efficiently identifying Nash equilibria for games composed of large numbers of players having discrete and not too large strategy spaces. The approach is based on ...

Busoniu, L., Ernst, D., Babuska, R., & De Schutter, B. (2007). Continuous-state reinforcement learning with fuzzy approximation.

*Proceedings of the 7th European Symposium on Adaptive Learning Agents and Multi-Agent Systems (ALAMAS-07)*(pp. 21-35).Peer reviewed

Reinforcement learning (RL) is a widely used learning paradigm for adaptive agents. Well-understood RL algorithms with good convergence and consistency properties exist. In their original form, these ...

Busoniu, L., Ernst, D., Babuska, R., & De Schutter, B. (2007). Fuzzy approximation for convergent model-based reinforcement learning.

*Proceedings of the 2007 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE-07)*.Peer reviewed

Reinforcement learning (RL) is a learning control paradigm that provides well-understood algorithms with good convergence and consistency properties. Unfortunately, these algorithms require that process states ...

Ernst, D., Glavic, M., Capitanescu, F., & Wehenkel, L. (2007). Model predictive control and reinforcement learning as two complementary frameworks.

*International Journal of Tomography and Statistics, 6*, 122-127.Peer reviewed (verified by ORBi)

Model predictive control (MPC) and reinforcement learning (RL) are two popular families of methods to control system dynamics. In their traditional setting, they formulate the control problem as a discrete ...

Ernst, D., Glavic, M., Stan, G.-B., Mannor, S., & Wehenkel, L. (2007). The cross-entropy method for power system combinatorial optimization problems.

*Proceedings of the 2007 Power Tech*(pp. 1290-1295).Peer reviewed

We present an application of a cross-entropy based combinatorial optimization method for solving some unit commitment problems. We report simulation results and analyze, under several perspectives (accuracy ...

Glavic, M., Ernst, D., Ruiz-Vega, D., Wehenkel, L., & Pavella, M. (2007). E-SIME- A method for transient stability closed-loop emergency control: achievements and prospects.

*Proceedings of 2007 IREP Symposium - Bulk Power Systems Dynamics and Control - VII*.Peer reviewed

A general response-based technique is presented for closed-loop transient stability emergency control. It relies on E-SIME, derived from the hybrid transient stability method, SIME. E-SIME uses real-time ...

Wehenkel, L., Glavic, M., & Ernst, D. (2007). A collaborative framework for multi-area dynamic security assessment of large scale systems.

*Proceedings of the 2007 Power Tech*(pp. 261-266).Peer reviewed

In this paper we propose a collaborative framework to carry out multi-area dynamic security assessment over an interconnection operated by a team of TSOs responsible of different areas. In this framework each ...

Krause, T., Beck, E. V., Cherkaoui, R., Germond, A., Andersson, G., & Ernst, D. (2006). A comparison of Nash equilibria analysis and agent-based modelling for power markets.

*International Journal of Electrical Power and Energy Systems, 28*(9), 599-607.Peer reviewed (verified by ORBi)

In this paper we compare Nash equilibria analysis and agent-based modelling for assessing the market dynamics of network-constrained pool markets. Power suppliers submit their bids to the market place in order ...

Geurts, P., Ernst, D., & Wehenkel, L. (2006). Extremely randomized trees.

*Machine Learning, 63*(1), 3-42.Peer reviewed (verified by ORBi)

This paper proposes anew tree-based ensemble method for supervised classification and regression problems. It essentially consists of randomizing strongly both attribute and cut-point choice while splitting a ...

Minoia, A., Ernst, D., Dicorato, M., Trovato, M., & Ilic, M. (2006). Reference transmission network: A game theory approach.

*IEEE Transactions on Power Systems, 21*(1), 249-259.Peer reviewed (verified by ORBi)

The transmission network plays a key role in an oligopolistic electricity market. In fact, the capacity of a transmission network determines the degree to which the generators in different locations compete ...

Capitanescu, F., Glavic, M., Ernst, D., & Wehenkel, L. (2006). Applications of security-constrained optimal power flows.

*In Proceedings of Modern Electric Power Systems Symposium, MEPS06*.Peer reviewed

This paper proposes to formulate security control as a sequential decision making problem and presents new developments in automatic learning of sequential decision making strategies from simulations and/or ...

Ernst, D., Glavic, M., Capitanescu, F., & Wehenkel, L. (2006). Model predictive control and reinforcement learning as two complementary frameworks.

*Proceedings of the 13th IFAC Workshop on Control Applications of Optimisation*.Peer reviewed

Model predictive control (MPC) and reinforcement learning (RL) are two popular families of methods to control system dynamics. In their traditional setting, they formulate the control problem as a discrete ...

Ernst, D., Marée, R., & Wehenkel, L. (2006). Reinforcement learning with raw image pixels as input state.

*Advances in machine vision, image processing & pattern analysis (Lecture notes in computer science, Vol. 4153)*. Berlin, Germany: Springer-Verlag Berlin.Peer reviewed

We report in this paper some positive simulation results obtained when image pixels are directly used as input state of a reinforcement learning algorithm. The reinforcement learning algorithm chosen to carry ...

Ernst, D., Stan, G.-B., Gonçalves, J., & Wehenkel, L. (2006). Clinical data based optimal STI strategies for HIV: a reinforcement learning approach.

*Proceedings of the 15th Annual Machine Learning Conference of Belgium and The Netherlands (Benelearn 2006)*.Peer reviewed

This paper addresses the problem of computing optimal structured treatment interruption strategies for HIV infected patients. We show that reinforcement learning may be useful to extract such strategies ...

Ernst, D., Stan, G.-B., Gonçalves, J., & Wehenkel, L. (2006). Clinical data based optimal STI strategies for HIV: a reinforcement learning approach.

*Proceedings of the 45th IEEE Conference on Decision and Control (CDC 2006)*(pp. 667 - 672).Peer reviewed

This paper addresses the problem of computing optimal structured treatment interruption strategies for HIV infected patients. We show that reinforcement learning may be useful to extract such strategies ...

Glavic, M., Wehenkel, L., & Ernst, D. (2006). Damping control by fusion of reinforcement learning and control Lyapunov functions.

*Proceedings of the 38th North American Power Symposium (NAPS 2006)*(pp. 361-367).Peer reviewed

The main idea behind the concept, proposed in the paper, is the opportunity to make control systems with increased capabilities by synergetic fusion of the domain-specific knowledge and the methodologies from ...

Wehenkel, L., Ernst, D., & Geurts, P. (2006). Ensembles of extremely randomized trees and some generic applications.

*Proceedings of Robust Methods for Power System State Estimation and Load Forecasting*.Peer reviewed

In this paper we present a new tree-based ensemble method called “Extra-Trees”. This algorithm averages predictions of trees obtained by partitioning the inputspace with randomly generated splits, leading to ...

Wehenkel, L., Glavic, M., & Ernst, D. (2006). Multi-area security assessment: results using efficient bounding method.

*Proceedings of the 38th North American Power Symposium (NAPS 2006)*(pp. 511-515).Peer reviewed

We present our recent results on using previously introduced framework for multi-area security assessment in large interconnections. The basic idea of the framework is exchanging just enough information so ...

Wehenkel, L., Glavic, M., & Ernst, D. (2006). On multi-area security assessment of large interconnected power systems.

*Proceedings of the Second Carnegie Mellon Conference in Electric Power Systems: Monitoring, Sensing, Software and its Valuation for the Changing electric Power Industry*.Peer reviewed

The paper introduces a framework for information exchange and coordination of security assessment suitable for distributed multi-area control in large interconnections operated by a team of transmission system ...

Wehenkel, L., Glavic, M., Geurts, P., & Ernst, D. (2006). About automatic learning for advanced sensing, monitoring and control of electric power systems.

*Proceedings of the Second Carnegie Mellon Conference in Electric Power Systems: Monitoring, Sensing, Software and its Valuation for the Changing electric Power Industry*.Peer reviewed

The paper considers the possible uses of automatic learning for improving power system performance by software methodologies. Automatic learning per se is first reviewed and recent developements of the field ...

Wehenkel, L., Glavic, M., Geurts, P., & Ernst, D. (2006). Automatic learning of sequential decision strategies for dynamic security assessment and control.

*Proceedings of the IEEE Power Engineering Society General Meeting 2006*.Peer reviewed

This paper proposes to formulate security control as a sequential decision making problem and presents new developments in automatic learning of sequential decision making strategies from simulations and/or ...

Glavic, M., Ernst, D., & Wehenkel, L. (2005). A reinforcement learning based discrete supplementary control for power system transient stability enhancement.

*Engineering Intelligent Systems for Electrical Engineering and Communications, 13*(2 Sp. Iss. SI), 81-88.Peer reviewed

This paper proposes an application of a Reinforcement Learning (RL) method to the control of a dynamic brake aimed to enhance power system transient stability. The control law of the resistive brake is in the ...

Ernst, D., Geurts, P., & Wehenkel, L. (2005). Tree-based batch mode reinforcement learning.

*Journal of Machine Learning Research, 6*, 503-556.Peer reviewed (verified by ORBi)

Reinforcement learning aims to determine an optimal control policy from interaction with a system or from observations gathered from a system. In batch mode, it can be achieved by approximating the so-called Q ...

Glavic, M., Ernst, D., & Wehenkel, L. (2005). Combining a stability and a performance-oriented control in power systems.

*IEEE Transactions on Power Systems, 20*(1), 525-526.Peer reviewed (verified by ORBi)

This paper suggests that the appropriate combination of a stability-oriented and a performance-oriented control technique is a promising way to implement advanced control schemes in power systems. The ...

Cirio, D., Lucarella, D., Vimercati, G., Massucco, S., Morini, A., Silvestro, F., Ernst, D., Pavella, M., & Wehenkel, L. (2005). Application of an advanced transient stability assessment and control method to a realistic power system.

*Proceedings of the 15th Power System Computation Conference (PSCC 2005)*.Peer reviewed

The paper presents a technical overview of a large research project on Dynamic Security Assessment (DSA) supported by EU. Transient Stability Assessment and Control, which was one of the main goals of the ...

Ernst, D. (2005). Selecting concise sets of samples for a reinforcement learning agent.

*Proceedings of the 3rd International Conference on Computational Intelligence, Robotics and Autonomous Systems (CIRAS 2005)*.Peer reviewed

We derive an algorithm for selecting from the set of samples gathered by a reinforcement learning agent interacting with a deterministic environment, a concise set from which the agent can extract a good ...

Ernst, D., Glavic, M., Geurts, P., & Wehenkel, L. (2005). Approximate value iteration in the reinforcement learning context. Application to electrical power system control.

*International Journal of Emerging Electrical Power Systems, 3*(1).Peer reviewed (verified by ORBi)

In this paper we explain how to design intelligent agents able to process the information acquired from interaction with a system to learn a good control policy and show how the methodology can be applied to ...

Krause, T., Andersson, G., Ernst, D., Vdovina-Beck, E., Cherkaoui, R., & Germond, A. (2005). A comparison of Nash equilibria analysis and agent-based modelling for power markets.

*Proceedings of the 15th Power System Computation Conference (PSCC 2005)*.Peer reviewed

In this paper we compare Nash equilibria analysis and agent-based modelling for assessing the market dynamics of network-constrained pool markets. Power suppliers submit their bids to the market place in order ...

Wehenkel, L., Glavic, M., & Ernst, D. (2005). New developments in the application of automatic learning to power system control.

*Proceedings of the 15th Power System Computation Conference (PSCC 2005)*.Peer reviewed

In this paper we present the basic principles of supervised learning and reinforcement learning as two complementary frameworks to design control laws or decision policies within the context of power system ...

Wehenkel, L., Ruiz-Vega, D., Ernst, D., & Pavella, M. (2005). Preventive and emergency control of power systems.

*Real Time Stability in Power Systems - Techniques for Early Detection of the Risk of Blackout*(pp. 199-232). Springer Berlin / Heidelberg.A general approach to real-time transient stability control is described, yielding various complementary techniques: pure preventive, open loop emergency, and closed loop emergency controls. The organization ...

Zima, M., & Ernst, D. (2005). On multi-area control in electric power systems.

*Proceedings of the 15th Power System Computation Conference (PSCC 2005)*.Peer reviewed

In this paper we study the concept of electric power system control, when the responsibility for controlling the entire system is shared by agents controlling their assigned areas. Within this framework, we ...

Ernst, D., Glavic, M., & Wehenkel, L. (2004). Power systems stability control: Reinforcement learning framework.

*IEEE Transactions on Power Systems, 19*(1), 427-435.Peer reviewed (verified by ORBi)

In this paper, we explore how a computational approach to learning from interactions, called reinforcement learning (RL), can be applied to control power systems. We describe some challenges in power system ...

Ernst, D., Minoia, A., & Marija, I. (2004). Market dynamics driven by the decision-making power producers.

*Proceedings of 2004 IREP Symposium - Bulk Power System Dynamics and Control - VI*.Peer reviewed

In this paper we consider a tool for analyzing the market outcomes when competitive agents (power producers) interact through the market place. The market clearing mechanism is based on the locational marginal ...

Krause, T., Andersson, G., Ernst, D., Vdovina-Beck, E., Cherkaoui, R., & Germond, A. (2004). Nash equilibria and reinforcement learning for active decision maker modelling in power markets.

*Proceedings of the 6th IAEE European Conference: Modelling in Energy Economics and Policy*.Peer reviewed

In this paper, we study the behavior of power suppliers who submit their bids to the market place in order to maximize their payoffs. The market clearing mechanism is based on the locational marginal price ...

Minoia, A., Ernst, D., & Ilic, M. (2004). Market dynamics driven by the decision-making of both power producers and transmission owners.

*Proceedings of the IEEE Power Engineering Society General Meeting 2004*(pp. 255-260).Peer reviewed

In this paper we consider an electricity market in which not only the power producers but also the transmission owners can submit a bid. The market is cleared at each stage by minimizing the sum of the ...

Ernst, D., Geurts, P., & Wehenkel, L. (2003). Iteratively extending time horizon reinforcement learning.

*Machine Learning: ECML 2003, 14th European Conference on Machine Learning*(pp. 96-107). Berlin, Germany: Springer-Verlag Berlin.Peer reviewed

Reinforcement learning aims to determine an (infinite time horizon) optimal control policy from interaction with a system. It can be solved by approximating the so-called Q-function from a sample of four ...

Glavic, M., Ernst, D., & Wehenkel, L. (2003). A reinforcement learning based discrete supplementary control for power system transient stability enhancement.

*Proceedings of the 12th Intelligent Systems Application to Power Systems Conference (ISAP 2003)*.Peer reviewed

This paper proposes an application of a Reinforcement Learning (RL) method to the control of a dynamic brake aimed to enhance power system transient stability. The control law of the resistive brake is in the ...

Ruiz-Vega, D., Glavic, M., & Ernst, D. (2003). Transient stability emergency control combining open-loop and closed-loop technique.

*Proceedings of the IEEE Power Engineering Society General Meeting, 2003*.Peer reviewed

An on-line transient stability emergency control approach is proposed, which couples an open-loop and a closed-loop emergency control technique. The open-loop technique uses on-line transient stability ...

Ernst, D., & Wehenkel, L. (2002). FACTS devices controlled by means of reinforcement learning algorithms.

*Proceedings of the 14th Power Systems Computation Conference (PSCC 2002)*.Peer reviewed

Reinforcement learning consists of a collection of methods for approximating solutions to deterministic and stochastic optimal control problems of unknown dynamics. These methods learn by experience how to ...

Ernst, D., Ruiz-Vega, D., Pavella, M., Hirsch, P., & Sobajic, D. (2001). A unified approach to transient stability contingency filtering, ranking and assessment.

*IEEE Transactions on Power Systems, 16*(3), 435-443.Peer reviewed (verified by ORBi)

This paper proposes a unified approach to contingency filtering, ranking and assessment in power system transient stability studies. The approach consists of two-block techniques in which the first block ...

Ghandhari, M., Andersson, G., Pavella, M., & Ernst, D. (2001). A control strategy for controllable series capacitor in electric power systems.

*Automatica, 37*, 1575-1583.Peer reviewed (verified by ORBi)

It has been veri"ed that a controllable series capacitor with a suitable control scheme can improve transient stability and help to damp electromechanical oscillations. A question of great importance is the ...

Roth, A., Ruiz-Vega, D., Ernst, D., Bulac, C., Pavella, M., & Andersson, G. (2001). An approach to modal analysis of power system angle stability.

*Proceedings of IEEE Powertech 2001*.Peer reviewed

An approach to modal analysis and modal identification is proposed, capable of complementing the panoply of existing methods. It is based on a hybrid time-domain – direct transient stability method called ...

Avilas-Rosales, R., Ruiz-Vega, D., Ernst, D., & Pavella, M. (2000). On-line transient stability constrained ATC calculations.

*Proceedings of the IEEE Power Engineering Society Summer Meeting 2000 - Volume 2*(pp. 1291-1296).Peer reviewed

Transient Stability Assessment, preventive control measures and dynamic A X calculations are addressed for the new deregulated EMS system. The combination of time domain analysis, SIME and Optimal Power Flow ...

Druet, C., Ernst, D., & Wehenkel, L. (2000). Application of reinforcement learning to electrical power system closed-loop emergency control.

*Principles of Data Mining and Knowledge Discovery, 4th European Conference, PKDD 2000*. Springer Berlin / Heidelberg.Peer reviewed

This paper investigates the use of reinforcement learning in electric power system emergency control. The approach consists of using numerical simulations together with on-policy Monte Carlo control to ...

Ernst, D., & Pavella, M. (2000). Closed-loop transient stability emergency control.

*Proceedings of the IEEE Power Engineering Society Winter Meeting 2000*(pp. 58-62).Peer reviewed

The question of transient stability control is revisited, various types of controls are identified, and a general approach to closed-loop emergency control is proposed. The focus is on feasibility aspects ...

Ernst, D., Ruiz-Vega, D., & Pavella, M. (2000). Preventive and emergency transient stability control.

*Proceedings of the VII Symposium of Specialists in Electric Operational and Expansion Planning (SEPOPE 2000)*.Peer reviewed

A unified approach to transient stability closed-loop control is presented. It relies on the general transient stability method called SIME, from which the Preventive and the Emergency SIMEs are derived. The ...

Ruiz-Vega, D., Ernst, D., & Pavella, M. (2000). Preventive countermeasures for transient stability-constrained power systems.

*Proceedings of the VII Symposium of Specialists in Electric Operational and Expansion Planning (SEPOPE 2000)*.Peer reviewed

A general transient stability control technique is applied to the design of preventive countermeasures consisting of rescheduling generators’ active power. The technique relies on SIME, a hybrid direct–time ...

Bettiol, A., Ruiz-Vega, D., Ernst, D., Wehenkel, L., & Pavella, M. (1999). Transient stability-constrained optimal power flow.

*Proceedings of the IEEE Power Tech'99*.Peer reviewed

This paper proposes a new approach able to maximize the interface flow limits in power systems and to find a new operating state that is secure with respect to both, dynamic (transient stability) and static ...

Ernst, D., Bettiol, A., Zhang, Y., Wehenkel, L., & Pavella, M. (1998). Real-time transient stability emergency control of the South-Southeast Brazilian system.

*Proceedings of SEPOPE 1998*.A method is proposed for during transients emergency control which predicts the evolution of a system undergoing a major disturbance and, if loss of synchronism is anticipated decides control actions able to ...