Integrating Soft Gripper and Gripping Agent for Universal Robotic Grasping

Abstract

It is crucial for successful operation that robots have the ability to flexibly grasp objects and accurately estimate the point of grasp in the field of general-purpose robotic gripping. To address these challenges, we present an Adaptive Rigid-Soft Gripping Agent (GAgent) that combines the adaptability of soft grippers with the cognitive capabilities of Multimodal Language Models (MMLMs) to effectively perform gripping tasks across diverse objects and environments. Our system features a variable-stiffness soft finger that integrates silicone and a Nitinol spring, supported by a rectangular structure and a double-tendon drive mechanism. This design ensures precise and reliable grasping. Additionally, we introduce task-focused prompts and step-level reasoning to fully leverage the generalization and reasoning capabilities of MMLMs. This enables accurate object texture recognition, categorization of objects into multiple hardness levels, and appropriate stiffness modulation to dynamically adjust the gripper’s rigidity, ensuring precise grip point estimation. Moreover, our framework includes Data-Driven Gripper features, integrating a vector database and continuous feedback from past experiences to refine grasping strategies over time. Experimental results validate our system’s effectiveness in grasping a wide range of objects under varying lighting conditions. These advancements collectively enhance the gripper’s adaptability, reliability, and efficiency in handling a broad spectrum of objects, establishing it as a versatile solution for advanced robotic tasks.

Soft gripper, Grasping, Variable Stiffness, MMLM model, Multi Agent, Bionic Robot.

1. Introduction

grasping and manipulation are essential capabilities for robotic systems, enabling them to effectively interact with their environment. These abilities are crucial for a wide range of applications, from industrial automation to service robotics. However, traditional rigid grippers often struggle with handling objects of varying shapes, sizes, and fragility.

To address these limitations, soft grippers inspired by biological systems have been developed. These grippers utilize flexible materials to adapt their shape to the objects they handle, thereby reducing the risk of damage. Various techniques, including fluidic actuation (Z. Li et al., 2024; Sinatra et al., 2019; Zhuang et al., 2023), tendon-driven mechanisms (Y. Chen et al., 2024; Hirose et al., 2019; Lee et al., 2020; Z. Li et al., 2022), and shape memory alloys (Yang et al., 2022), have been explored to enhance their performance. Despite these advancements, achieving an optimal balance between adaptability and robustness remains challenging.

Variable stiffness mechanisms have been introduced to improve the load-bearing capacity of soft grippers while retaining flexibility. Methods such as particle jamming (Y. Li et al., 2017), low melting point alloys (Tonazzini et al., 2016), shape memory alloys (SMA) (Hu et al., 2021; Y.-F. Zhang et al., 2019), electric stimulation (Sheng & Wen, 2012), additional exoskeletons (J. Zhu et al., 2023), and antagonistic actuation mechanisms (Z. Li et al., 2025) enable the gripper to increase its stiffness when needed, providing better performance in diverse scenarios. Among these, antagonistic mechanisms stand out by enhancing stability through balanced force pairs while avoiding significant increases in system complexity or mass. Nevertheless, most other approaches often introduce added weight and mechanical or control complexity.

In parallel, there has been a growing interest in integrating cognitive capabilities into robotic systems, an approach exemplified by initiatives such as Robotgpt (Jin et al., 2024) and Graspgpt (Tang et al., 2023). At the forefront of this integration are Multimodal Models (MMLMs) and Visual-Language Models (VLMs), including MiniGPT-4 (D. Zhu et al., 2023) and LLaVA (Liu, Li, Wu, et al., 2023), which have shown exceptional performance in tasks involving natural instruction-following and visual cognition. These models enhance the interaction between robots and the physical world by providing superior generalization and advanced reasoning abilities. As a result, VLMs are increasingly being integrated into embodied agents to improve their interaction capabilities. Unlike specific-task reinforcement learning (RL) algorithms, VLM-based agents offer significant generalization capabilities (Driess et al., 2023; Liu et al., 2023; J. Wang et al., 2025; Yin et al., 2025; M. Zhang, Yin, et al., 2025), facilitated by sophisticated fine-tuning methodologies such as human demonstrations (G. Chen et al., 2023; Y. Wang et al., 2025), vision-language cross-modal connectors (K. Li et al., 2025; Liu, Li, Li, et al., 2024; M. Zhang, Fang, et al., 2025), and the continuous expansion of skill libraries (G. Wang et al., 2023). However, on-policy RL algorithms can still present challenges, particularly regarding sample efficiency. VLM-based agents, such as those discussed in (Driess et al., 2023), leverage joint training across language, vision, and visual-language domains, enabling effective transfer across diverse tasks and datasets. This capability is crucial as complex tasks often demand cogent reasoning abilities, and VLMs with emergent reasoning capabilities (Liang et al., 2025; Wei et al., 2022) are increasingly employed for complex tasks in few-shot scenarios (Wei et al., 2022; Zeng et al., 2025; M. Zhang, Shen, Yin, et al., 2024; M. Zhang, Shen, Li, et al., 2024a). Despite these advancements, VLM-based agents face environmental challenges, particularly pixel-level noise in complex lighting environments.

This paper introduces a novel soft-rigid gripper system that combines the adaptability of soft grippers with the cognitive capabilities of intelligent agents based on MMLMs. The gripper features adaptive gripping and a wide gripping range, utilizing envelope gripping and pinching. It is mounted on a continuum robotic arm for versatile object interaction. The gripper’s rigidity can self-adjust in real-time based on feedback from an integrated object recognition system, enhancing its adaptability to various grasping scenarios. The key contributions of this article are as follows:

• Variable Stiffness Soft Finger: We developed a variable stiffness soft finger that combines silicone with a rectangular nitinol spring, addressing torsional deformation issues via a rectangular structure and a double-tendon drive mechanism, while achieving both variable stiffness and lightweight characteristics.

• Advanced Object Recognition: The GAgent framework integrates advanced Multimodal Models (MMLMs) with a monocular camera to accurately recognize object textures, thereby facilitating the effective capture of objects in new and unstructured environments. This framework categorizes objects into five distinct hardness levels, each corresponding to specific stiffness settings of the gripper. This precise classification enables the gripper to modulate its stiffness appropriately, ensuring a firm and secure grasp of rigid objects while applying the necessary delicacy to handle fragile items. Such an approach significantly enhances the gripper’s ability to manipulate a diverse array of objects with varying textures and hardness, thereby improving its operational flexibility, adaptability, and reliability in complex and dynamic scenarios.

• Data Driven Gripper Framework: The framework establishes a continuous feedback loop by integrating task-focused prompts, step-level reasoning, and a vector database. This approach enhances the gripper’s generalization and reasoning capabilities, while also allowing it to learn from past experiences and refine its grasping strategies over time.

The remainder of this article is structured as follows: Section II details the design of the variable rigidity soft gripper. Section III presents the static analysis of the finger and performance of variable stiffness gripper. Section IV introduces novel visual language modeling agents for grippers. The performance of the gripper is evaluated in Section V, and the findings are summarized in Section VI.

2. Bionic soft-rigid Gripper Design Concept

The design concept of the bionic soft-rigid gripper is inspired by the human hand, which has evolved to exhibit remarkable versatility. Whether grasping small items or carrying larger ones, the human hand adapts its technique according to the object’s shape and characteristics, employing actions such as pinching, enveloping, hooking, and lifting. Notably, even when confronted with unfamiliar objects, the hand can skillfully grasp them without causing damage, owing to its soft-rigid and responsive structure.

Based on the study of the versatility and complex structure of the human hand (Z. Li et al., 2024), we present a new soft-rigid gripper consisting of silicone, rectangular nitinol springs, pressure sensors, tendons, and support structures, as shown in Fig. 1. The soft silicone mimics the muscle layer of the finger, acting as a cushion and adaptor that absorbs impact when gripping an object, increases friction, and adapts to the object’s surface. Rectangular Nitinol springs mimic the skeleton of a human hand, supporting the hollow silicone and enabling the gripper to change stiffness to increase load carrying capacity without losing flexibility.

To achieve precise and biomimetic finger articulation, each finger of the gripper is actuated by a pair of antagonistic cables that function as artificial tendons. This dual-tendon structure design enables stable and controllable bending. These actuation tendons are anchored at the free ends of the gripper’s rollers, routed through a rectangular spring, and terminate at the fingertip. This routing strategy effectively shifts the primary friction point from the tendon-silicone interface to the more robust tendon-spring interface, thereby minimizing energy dissipation and significantly amplifying the output force of the finger. The return to the neutral (open) position is facilitated by the synergistic elastic recovery of an embedded Nitinol(NiTi) springs and the inherent elasticity of the silicone matrix, eliminating the need for a dedicated return tendon.

The gripper incorporates an independent, dedicated stiffness-tuning tendon for each finger. This separate tendon is specifically designed to modulate the gripper’s rigidity. By applying tension to this tendon, the embedded NiTi spring is further stretched, increasing the overall structural stiffness of the finger. This decoupled design allows for independent control of finger posture (via the antagonistic actuation tendons) and finger stiffness (via the dedicated stiffness-tuning tendon), mimicking the independent control of motion and muscle tone in the human hand. Furthermore, integrated pressure sensors at the fingertips provide real-time tactile feedback, simulating human somatosensation. This sensory input enables the gripper to dynamically adjust its grip force, allowing for the delicate and damage-free handling of fragile objects.

3. Static mechanics and variable stiffness capabilities of gripper

3.1 Static mechanics of a single finger

Since the design is the same for the three fingers, static analysis by Euler-beam model is performed for one finger. The gripper is divided into small units by flexible and rigid modules, as shown in Fig. 2. When a finger grasps an object, the tension transfer relationship of the tendon can be expressed by Eq. (1).

where $\alpha$ is the transmission relation of tension in a single beam model and $\mu$ is the coefficient of friction. The transmission of tendon tension from the proximal to the distal tip of the finger can be expressed by Eq. (2).

When adding the tensions in the 20 flexible and rigid cells, it is clear that the output force of the motor is equal to the tension acting at the first node:

The force on the $i$ elastic beam unit can be decomposed as:

Based on the above analysis, the end coordinates of the flexible beam are characterized by a beam constraint model (Awtar & Slocum, 2007; Wu et al., 2024). According to the dimensionless beam formula for a rectangular cross-section and given any set of $F_{xi}$, $F_{zi}$, $M_{i}$, the beam end coordinates $\overset{\land}{z}(1)$, $\overset{\land}{x}(1)$, $\overset{\land}{\theta}(1)$ can be obtained as:

where $\text{A},\text{B},\text{C},\text{D}$ are dimensionless characteristic coefficients of the beam constraint model, which can be found in (Awtar, 2003). The deflected shape of the beam can be obtained from the following equation:

where $\overset{\land}{s} \in \lbrack 0,1\rbrack$, matrix $\mathbf{C}^{*}$ and $\mathbf{K}^{*}$ can be found in (Ma & Chen, 2016). Quantizing the above dimensionless formula to get:

where $s \in \lbrack 0,l\rbrack$. The coordinates and angles of each unit can be calculated using Eq. (7).

For the curved shape of the soft-rigid gripper, the angles of the 20 flexible beams can be summed up to get the angle of curvature of the whole finger $\theta_{L}$:

The results of static simulations, Abaqus simulations, and physical experiments show similar outcomes (Fig. 4). In the Abaqus simulation, the fingers are set with a preload so that there is a closer resemblance to reality in the statics simulation. To ensure the fingers achieve an enveloping grasp on an object, the contact points of the gripper can be evaluated using Eq. (9)：

where $r$ is the bending radius of the three fingers and $R$ is the radius of the object being grasped.

The stiffness of the entire gripper can be expressed as :

$k_{T}$ is the total stiffness coefficient, $k_{s}$ is the stiffness coefficient of the spring, $k_{t}$ is the stiffness coefficient of the tendon and $k_{m}$ is the stiffness coefficient of the soft material.

The classification of the gripper’s variable stiffness into five categories corresponding to the different stiffnesses of the items is intended to cover a wide range of stiffnesses when applying the variable stiffness grippers. This classification maintains accuracy and practicality, corresponding to super-soft, soft, medium-hard, hard, and super-hard items. A summary of these categories is provided in Table 1.

3.2 Variable stiffness analysis of finger

The ability of the human hand to comfortably cope with everything from fragile objects to heavy grips is due to the variable rigidity of the human hand. In order to evaluate the variable stiffness capability of this design, the gripper design was subjected to finite element analysis and compared with a purely flexible design as well as a rigid design, and a stability analysis of the gripper was carried out.

We performed finite element analysis on a soft silicone finger, a hard nitinol finger, and a rigid-soft finger to assess the gripper’s ability to change stiffness, as shown in Fig.3(b)-(d). It is clear to see that the soft silicone finger bends at the root of the finger, which reduces the gripper’s ability to fit the object, compared to a rigid-soft finger embedded with nitinol, which spreads the stress over the entire finger and bends at a more linear angle. This not only enables more controlled grasping, but also facilitates safer interactions with objects, as shown in Fig. 3(a).

The pure silicone finger stops moving after attaining a vertical position of 45mm. This could be attributed to the fact that the flexible material ($DragonSkin10$) is incompressible, resulting in a significant reverse force on the finger’s substructure at this position, rendering it unable to bend. In contrast, the spring-silicone coupled finger achieves a maximum vertical displacement of 62mm, with the spring bending up to 55mm. Upon comparing the vertical and horizontal displacements of all three, it appears that the bending curvature of the spring-silicone coupled finger exhibits greater linearization.

To corroborate the precision of the simulation, the simulated tension is increased and compared with the actual bending. Concurrently, we record the three coordinate positions of the actual finger trajectory corresponding to $0^{\circ}$, $45^{\circ}$, and $90^{\circ}$ of motor rotation to compare with the simulation curve. The actual trajectories of finger displacement basically coincide with those of the theoretical simulation and the finite element simulation, as shown in Fig. 4.

3.3 Stabilization of variable stiffness finger

The stability of a gripper stands as a critical measure of its performance. Given that grippers are used in industrial settings or implemented on robots, it is unavoidable that they will confront collisions that could impact the equipment. For instance, Vibrations formed by collisions in fast-moving or unstructured environments can lead to a decrease in grasping results. Achieving rapid stabilization after an unexpected collision is one of the most important metrics for evaluating grippers. To broaden the applicability of rigid-soft grippers across various spheres, the authors conducted experimentation on the stability of grippers in response to interference, comparing both the time it takes to regain stability before and after spring pre-tensioning and the angular range of change.

In this experiment, we mounted the MPU6050 at the end of the fingertip, and the rigid-soft finger was mounted in a bench vise, as shown in Fig. 5(a). We collected data from IMU by repeatedly tapping fingers for comparison.

In the experimental crash tests, we perform impact experiments with the same force for the initial state and the variable rigidity state, where the initial and variable rigidity states correspond to levels 0 and 4 in Table 1. As can be clearly seen in Fig. 5(b), under variable stiffness, the initial velocity of the finger after impact is 5.8$dps$ compared to that of the initial state can be reduced by about 20% of the impact velocity. The initial state can bend up to a maximum angle of 25° after impact and shows an unstable state on both sides, while after the variable stiffness, it is clearly seen that the two sides of the shock are stable after impact, as shown in Fig. 5(c). In the variable stiffness state, the finger was able to return to a stable state in roughly 2.5s, which is a 45% improvement in performance compared to the initial state which stabilized after 4.5s. Consequently, rigid-soft fingers return to a steady position more quickly after tightening, making it easier to design control systems and perform faster operations for mobile robotics, industrial production, and domestic applications.

4. An Adaptive Rigid-Soft Gripping Agent with Vision Language Models

Based on the previously designed soft gripper, we also have developed an intelligent grasping framework, as shown in Fig. 6, which incorporates a Visual-language Model equipped with an adaptive rigidity gripper, optimized for precision grasping. The system inputs include captured images and relevant prior information, while the output consists of the optimal grasping point and stiffness score (see Table 1).

In detail, the framework comprises A Multi-Modal Large Language Model (MMLM), a Task-focused prompt, Step-level reason, Task Flow, and a Vector database.

• MMLM has been utilized by GPT-4o, which is attributed to its powerful reasoning capabilities. It converts complex instructions combined with Task-focused prompts and step-level reason into a position of grasp area and stiffness score, which can guide the soft grasp to complete the grasping task.

• The task-focused prompt compiles task descriptions, role descriptions, tool descriptions, and basic operational guidelines into a coherent prompt that the MMLM can then process.

• The step-level reasoning process consists of four iterative stages: detection, grasping, critical thinking, and reflection. The detection stage examines the overall content of the image to identify the object and its features. During the grasping stage, potential grasping areas of the object are inferred using tools from the ToolSet. The critical thinking stage involves determining the optimal grasping point based on the previously gathered information. Finally, in the reflection stage, the optimal grasping area and point are reevaluated. If necessary, a new cycle can be initiated, incorporating updates from Retrieval Augmented Generation (RAG).

• Tool flow for robotic grasping, integrating various tools to enhance precision and effectiveness. The process is broken down into several steps under three main categories: Only Detection, Detection & Grasping, and Only Grasping.

• Create a continuous feedback loop using a vector database where data retrieval, augmentation, generation, and database updates work together to improve the performance of machine learning models.

In summary, the framework’s innovation lies in integrating a large multi-modal model with strong generalization capabilities, step-level reasoning with powerful reasoning ability, and various tools including a low-light enhancement algorithm (M. Zhang, Shen, Li, et al., 2024b), reasoning segmentation algorithm (Lai et al., 2024). This enables efficient operation across a wide range of environments. Simultaneously, the soft gripper ensures precise and flexible object manipulation.

>

5. Experimental Validation of soft-rigid Gripper Performance

5.1 Comprehensive grasping experiments

Capability validation experiments were conducted based on the Amazon Picking Challenge (Hernandez et al., 2017), with 45 items added for grasping experiments. These items were used to verify the gripping range and load capacity of the soft-rigid gripper. Jelly and pudding were the softest items, while potato chips and seaweed were the lightest. The smallest items were small steel balls and coins, while the heaviest and hardest were weights. The largest item was a bucket of water, as well as a spiked durian shell (see Fig. 7).

The soft-rigid gripper was mounted on a continuous robotic arm for comprehensive gripping experiments to evaluate its structural performance. A total of 46 distinct objects, including items of various weights, sizes, and materials (e.g., soft objects, small regular objects, heavy items, long objects, and oversized items), were selected for testing. For each object, 10 repeated grasping trials were conducted, resulting in 460 total attempts. In these experiments, the manipulator operated without a MMLM model, relying on an experience-informed capture strategy while varying the stiffness threshold of the pressure sensor. A grasp was deemed successful if the object remained stable in the air for 5 seconds after lifting and was then smoothly lowered; otherwise, it was counted as a failure.

The results demonstrate that the gripper is capable of handling a wide range of objects, with representative successful grasps shown in Fig. 8(a). Overall, the experience-informed approach achieved a high success rate of 88.5 $\pm$ 4.1% across all trials. The gripper exhibits efficient performance with soft, small, and regular objects, particularly those smaller than itself. For heavy objects, it maintains a maximum load capacity of 600 g while preserving structural rigidity; by adjusting the spring stiffness, this capacity can be increased to 700 g without risk of detachment, as illustrated in Fig. 8(b).

5.2 Qualitative Analysis of GAgent

This study introduces GAgent to enhance the flexibility and versatility of soft-rigid grippers’ grasping capabilities. This section will qualitatively analyze GAgent’s overall workflow. As shown in Fig. 9, six objects with diverse shapes were selected for grasping, including a bowl, scissors, books, and others. Step A involves inputting captured images to reason the basic characteristics of objects Property_{grasping-object} to be grasped, including category, shape, size, and environmental information. In Step B, we use the results from Step A as the input prompt for the reasoning segmentation algorithm (Lai et al., 2024), which generates a segmentation map Image_segmentation of the target object, and subsequently produces Prompt_{optimal-grasping-section}. Afterward, we input Prompt_{optimal-grasping-section} back into the reasoning segmentation algorithm to determine the optimal gripping area Image_{optimal-grasping-section} . For a bowl, the optimal gripping area is located at the edge; for scissors, it is the plastic part; for a pear, it is the lower half; and for a book, it is the edge. Moving to Step 3, we further extract the optimal gripping points Point_{optimal-grasping} within these optimal gripping areas, as indicated by the white dots in the last row.

5.3 Quantitative Analysis of GAgent

This section conducts a Quantitative Analysis of GAgent. Table 2 presents success rates for nine objects using different versions of the GAgent system. The table shows success rates for recognizing and handling different objects, including Bowl, Pear, Book, Cookies, Scissors, Hammer, Towel, Mouse, and Coin. Success rates generally increase as more detection steps are employed, with GAgent_All achieving highest success rates across all objects. For example, the success rate for the Bowl object increases from 20% with GAgent_S1 to 90% with GAgent_All. Similarly, for the object Coin, the success rate increases from 0% with GAgent_S1 to 70% with GAgent_All. This indicates the effectiveness of using multiple detection steps in improving the gripper’s performance across a variety of objects.

5.4 Ablation analysis of GAgent

This gripper is also attached to a section of the continuous robotic arm for gripping experiments. Our goal is to replicate various complex real-world scenarios with spontaneously changing outdoor conditions. Throughout these experiments, our innovative gripper self-adjusts its rigidity. By integrating object recognition feedback from a single-lens camera with the MMLM model, the gripper can adeptly respond to an array of grasping dynamics.

To evaluate the effectiveness of our new model, we conducted an ablation study comparing its performance against GPT-4o and manual operations. The study focused on various metrics, including accuracy and robustness across different tasks.

1. Accuracy: Our GAgent framework demonstrated significantly improved accuracy over pure GPT-4o and manual operations. Specifically, it achieved an average accuracy of 93.7% compared to 88.5% for manual operations and different development stages. Fig. 10 shows specific values, with details of each development stage as follows:

   • Manual Operations: Average success rate of 88.5 $\pm$ 4.1%.

   • S1 (Step A, Pure GPT-4o for Object Recognition): Success rate of 30.0 $\pm$ 3.5%, only capable of recognizing objects without providing positional information.

   • S2 (Step A+B, S1 with Reasoning Segmentation): Success rate improved to 53.9 $\pm$ 2.7%, accurately identifying the boundaries of objects.

   • S3 (Step A+B+C, S2 with Refined Grasp Boundaries): Success rate increased to 85.4 $\pm$ 3.2%, further refining the grasp boundaries.

   • All (All steps, S3 with Precise Finger Grasp Points, RAG, and Rigidity Adjustment): Success rate reached 93.7 $\pm$3.1%, incorporating precise grasp points for each finger along with item and gripper rigidity adjustments, as shown in Fig. 9.

• Robustness: The robustness of GAgent was tested across various adverse light conditions. The new model maintained consistent performance with a 2% drop in accuracy under adverse conditions, whereas GPT-4o showed a 5% drop. This indicates that our agent is more resilient to input variations and can handle real-world data more effectively.

• Error Handling: In failed gripper tasks, GAgent can record each failed interaction and reflective case to form an independent vector database, conferring our framework with a strong capacity for error correction and handling of gripper problems. GAgent_All has risen to 93.7% from 85.4%.

6. Conclusion

In this study, we designed a multi-stage adjustable variable stiffness soft gripper composed of a rectangular nitinol spring, silicone, tendons, and motors. The gripper can perform multi-stage stiffness adjustments by stretching the nitinol spring and addresses the torsion problem of soft materials through double tendon balance control. We employed the Euler-beam model theory for static analysis of the gripper and conducted finite element analysis to examine changes in its stiffness. Our data-driven framework incorporates task-centered cues, step-level reasoning, and a vector database, creating a continuous feedback loop that enhances the accuracy of the gripper’s operations. Additionally, our GAgent classifies objects into five categories based on their softness and hardness, enabling precise adjustment of the gripper’s stiffness for accurate gripping. These advancements collectively enhance the gripper’s adaptability, reliability, and efficiency in handling a wide range of objects, making it a versatile solution for advanced robotic tasks. Our system represents a significant step forward in robotic grasping technology, offering improved performance and adaptability in real-world applications.

Acknowledgment

The work described in this paper is partially supported by Shenzhen Fundamental Research (General Program)(WDZC20231129163533001) and Hunan Provincial Transportation Science and Technology Project (No.202403).

7. Appendix

{Section_{grasping_object}:
  {section1: [Material, Size, Stiffness Levels],
   section2: ...
  }
}