Foundations, Challenges, and Future Directions

View the PDF file from the paper entitled “A comprehensive survey of agents to use computer: foundations, challenges, and future trends, by Pascal C. Siger, Benjamin Maeer, Pang Yan, Rebecca von, Corrje-Kotler, Layan Etay, ARF Enayati, Gabriel Nobel, ahmed Abdulkadir, Benjamin F.

PDF view

a summary:Agents for computer use (ACUS) is a category of systems that are able to carry out complex tasks on digital devices – such as desktops, mobile phones and web platforms – instructions in the natural language. These agents can automate tasks by controlling programs through low -level actions such as mouse clicks and touch screen gestures. However, despite rapid progress, the ACUS has not yet matured for daily use.

In this poll, we check the latest gaps, trends and research in the development of the ACUS process. We offer a comprehensive review of the ACU scene, as it provided a unified rating that extends in three dimensions: (1) The field perspective, and the description of the agent operating contexts; (2) A reaction perspective, description of monitoring methods (for example, screenshots, HTML) and procedures (for example, mouse, keyboard, code implementation); And (3) the perspective of the agent, explains in detail how the agents, reason and learning look.

We review 87 ACUS and 33 data sets via methods based on the foundation and classic model through this classification. Our analysis determines six main research gaps: insufficient generalization, ineffective learning, limited planning, the complexity of the low task in standards, non -standard evaluation, and the separation between research and practical conditions.

To address these gaps, we call for the following: (a) Notes based on vision and control of the low level to enhance the circular; (B) Adaptive learning goes beyond fixed induction; (C) Effective methods and models of planning and thinking; (D) The criteria that reflect the complexity of the task in the real world; (E) Unified evaluation based on the success of the mission; (F) Align the design of the agent with publishing restrictions in the real world.

Together, classification and analysis of ACU’s research are established towards agents for general purposes to use a strong and developmental computer.