MIT Researchers Develop an Efficient Way to Train more Reliable AI Agents

Fields ranging from robotics to medication to government are attempting to train AI systems to make meaningful choices of all kinds. For instance, using an AI system to smartly control traffic in a congested city might assist drivers reach their destinations quicker, while improving safety or sustainability.

Unfortunately, teaching an AI system to make good decisions is no easy job.

Reinforcement learning models, which underlie these AI decision-making systems, still frequently stop working when faced with even small variations in the jobs they are trained to carry out. In the case of traffic, a model may have a hard time to control a set of intersections with different speed limitations, varieties of lanes, or traffic patterns.

To increase the dependability of support knowing designs for with variability, MIT scientists have presented a more effective algorithm for training them.

The algorithm strategically chooses the finest tasks for training an AI representative so it can effectively carry out all tasks in a collection of related tasks. When it comes to traffic signal control, each task might be one crossway in a task area that includes all crossways in the city.

By focusing on a smaller sized variety of intersections that contribute the most to the algorithm’s general efficiency, this technique takes full advantage of performance while keeping the training cost low.

The researchers discovered that their technique was between five and 50 times more effective than standard approaches on a variety of simulated jobs. This gain in performance assists the algorithm learn a better option in a quicker manner, eventually improving the efficiency of the AI representative.

“We were able to see incredible efficiency improvements, with a very basic algorithm, by thinking outside the box. An algorithm that is not very complex stands a much better chance of being embraced by the neighborhood since it is simpler to execute and simpler for others to comprehend,” says senior author Cathy Wu, the Thomas D. and Virginia W. Cabot Career Development Associate Professor in Civil and Environmental Engineering (CEE) and the Institute for Data, Systems, and Society (IDSS), and a member of the Laboratory for Information and Decision Systems (LIDS).

She is joined on the paper by lead author Jung-Hoon Cho, a CEE graduate student; Vindula Jayawardana, a graduate student in the Department of Electrical Engineering and Computer Science (EECS); and Sirui Li, an IDSS college student. The research study will be presented at the Conference on Neural Information Processing Systems.

Finding a middle ground

To train an algorithm to manage traffic control at lots of crossways in a city, an engineer would normally pick between 2 primary methods. She can train one algorithm for each intersection individually, utilizing only that crossway’s information, or train a bigger algorithm using information from all intersections and after that apply it to each one.

But each approach comes with its share of disadvantages. Training a different algorithm for each task (such as an offered intersection) is a lengthy process that needs an enormous quantity of data and computation, while training one algorithm for all jobs typically results in subpar efficiency.

Wu and her collaborators looked for a sweet area between these 2 techniques.

For their technique, they pick a subset of jobs and train one algorithm for each task individually. Importantly, they strategically choose individual tasks which are most likely to improve the algorithm’s overall efficiency on all jobs.

They utilize a common trick from the reinforcement knowing field called zero-shot transfer learning, in which a currently trained model is applied to a new task without being further trained. With transfer learning, the design often performs extremely well on the brand-new next-door neighbor task.

“We know it would be perfect to train on all the tasks, however we questioned if we might get away with training on a subset of those tasks, use the result to all the tasks, and still see an efficiency boost,” Wu states.

To identify which tasks they need to pick to take full advantage of anticipated performance, the scientists developed an algorithm called Model-Based Transfer Learning (MBTL).

The MBTL algorithm has 2 pieces. For one, it models how well each algorithm would carry out if it were trained separately on one job. Then it designs how much each algorithm’s performance would deteriorate if it were moved to each other job, a principle understood as generalization efficiency.

Explicitly modeling generalization performance allows MBTL to estimate the value of training on a new task.

MBTL does this sequentially, choosing the job which leads to the greatest efficiency gain first, then picking additional jobs that offer the most significant subsequent minimal improvements to general performance.

Since MBTL just focuses on the most appealing tasks, it can considerably improve the efficiency of the training process.

Reducing training expenses

When the researchers tested this strategy on simulated tasks, consisting of controlling traffic signals, handling real-time speed advisories, and performing numerous timeless control jobs, it was five to 50 times more effective than other methods.

This means they might get here at the exact same service by training on far less information. For circumstances, with a 50x effectiveness boost, the MBTL algorithm could train on simply two jobs and accomplish the same performance as a basic method which uses information from 100 tasks.

“From the point of view of the two main approaches, that indicates data from the other 98 jobs was not required or that training on all 100 jobs is confusing to the algorithm, so the efficiency ends up even worse than ours,” Wu states.

With MBTL, adding even a percentage of extra training time might result in better performance.

In the future, the researchers plan to design MBTL algorithms that can encompass more complicated issues, such as high-dimensional job areas. They are also interested in applying their approach to real-world issues, specifically in next-generation movement systems.