Analog Circuit Design with Dyna-Style Reinforcement Learning

TL;DR

This paper introduces DynaOpt, a reinforcement learning-based method for analog circuit design that efficiently explores the solution space and reduces simulation costs by using a learned surrogate model and stochastic policy generation.

Contribution

It presents a novel Dyna-style framework combining a neural network-based reward model with stochastic exploration for efficient analog circuit optimization.

Findings

DynaOpt outperforms model-free methods in fewer simulations.

Achieves better circuit performance with only 500 simulations.

Demonstrates effectiveness on a two-stage operational amplifier benchmark.

Abstract

In this work, we present a learning based approach to analog circuit design, where the goal is to optimize circuit performance subject to certain design constraints. One of the aspects that makes this problem challenging to optimize, is that measuring the performance of candidate configurations with simulation can be computationally expensive, particularly in the post-layout design. Additionally, the large number of design constraints and the interaction between the relevant quantities makes the problem complex. Therefore, to better facilitate supporting the human designers, it is desirable to gain knowledge about the whole space of feasible solutions. In order to tackle these challenges, we take inspiration from model-based reinforcement learning and propose a method with two key properties. First, it learns a reward model, i.e., surrogate model of the performance approximated by neural networks, to reduce the required number of simulation. Second, it uses a stochastic policy generator to explore the diverse solution space satisfying constraints. Together we combine these in a Dyna-style optimization framework, which we call DynaOpt, and empirically evaluate the performance on a circuit benchmark of a two-stage operational amplifier. The results show that, compared to the model-free method applied with 20,000 circuit simulations to train the policy, DynaOpt achieves even much better performance by learning from scratch with only 500 simulations.