I explore merging art and robotics, creating a system that guides a robotic arm to paint like the masters. It’s about turning sketches into robotic motion, making art with a tech twist​.

Ever imagined a world where robots wield paintbrushes with the grace of Da Vinci and the passion of Van Gogh? Where metal meets canvas and technology romances art? Welcome aboard our imagination train, as we unravel our latest masterpiece: a fusion of artistry and machinery. Dive deep into this innovative realm where the lines between man-made and machine-made blur, introducing a new age of robotic Renaissance!

Peek below and you’ll see our 6-axis robotic manipulator in action, casually sketching out our fave lil’ blue cat - Doraemon. I mean, who knew tech could have such an artsy flair?

Drawing System Platform via Kuka Industrial Robotic Manipulator

Our system is a culmination of rigorous engineering and computational design to realize the potential of robotic drawing:

1. Robotic Manipulator Kinematics and Control:
   • Forward and Inverse Kinematics: The inherent design of the KUKA platform primarily interfaces with the angle data of its six motors. Recognizing this, we’ve pioneered a proprietary endpoint movement control system. With the advent of our robotic manipulator controller, the intricate dance of motion begins with simply providing six parameters: x, y, z coordinates, and the pitch, roll, and yaw orientations. The controller’s sophisticated algorithm seamlessly translates these spatial cues into specific angular positions for each of the six motors, orchestrating the manipulator’s precise movements in space.
   • Communication Protocols: The system utilizes the UDP protocol for communication, interpreting data in an XML format, ensuring efficient and accurate data exchange.
2. Image Processing for Accurate Contour Recognition:
   • Image Pre-processing: The system employs grayscale transformations, filtering techniques, and adaptive thresholding to ensure optimal image quality for contour extraction.
   • Contour Extraction: Our advanced algorithm discerns key pixel points and their inter-relationships to define cohesive line segments integral to the drawing.
3. Path Planning for Drawing:
   • Contour Sequencing and Transition Path Planning: Once the contours are identified, a meticulous strategy is formulated for the drawing sequence and transitioning between distinct contours.
   • Refinement with B-Spline Algorithm: Considering the granular nature of pixel-based representations, we’ve integrated the B-Spline algorithm to ensure smoothness in large-scale translations of the drawing.

In summary, our system proficiently interprets image data, extracts essential artistic contours, refines them using the B-Spline algorithm, and translates this data into precise movements for the KUKA robotic manipulator. This movement data, encapsulated in XML format, is relayed over a network, ensuring that the robotic manipulator executes the drawing task with utmost accuracy and efficiency.

System Architecture

Let’s delve into the technical architecture of the system:

Our framework is bifurcated into two primary modules: Image Processing and Robotic Manipulator Motion Control. Upon the input of an image into the system, the sequence of operations commences. The process initiates with adaptive binarization, followed by a filtering phase. Subsequently, the system captures the contours, plans the movement path, and employs the B-Spline algorithm to ensure the trajectory is smooth.

Transitioning to the robotic manipulator control, it’s noteworthy that the system communicates via the UDP protocol, transmitting data packets in XML format. This intricate communication sequence effectively dictates the end-point movement of the robotic manipulator, facilitating the rendering of a precise illustration.

System Workflow Visualization

Below is a comprehensive flowchart that intricately captures our system’s rigorous processing sequence. Taking a basic ‘Doraemon’ sketch as our point of reference, the image first undergoes a grayscale transformation to achieve tonal uniformity. Subsequently, an Adaptive Histogram Equalization is utilized to augment the image’s contrast nuances. This is pursued by a filtering process to effectively minimize noise interferences. Thereafter, Adaptive Thresholding segments the salient features, paving the way for contour extraction. This extracted framework then enters our sophisticated path planning, harnessing the prowess of the B-spline algorithm. Ultimately, the Robotic Manipulator executes the drawing, meticulously translating the image processing results onto the physical canvas.

Input

Step 1: Gray Scale

Step 2: Adaptive Histogram

Step 3: Filter

Step 4: Adaptive Threshold

Step 5: Contour Extraction

Step 6: Path Planning w/ B-Spline 3D View

Step 6: Path Planning w/ B-Spline Zoom in

Result

Demo Insights

Paper

Presented below is a detailed demonstration categorized into three progressive stages of complexity. Each scenario distinctly showcases the original input image, the meticulously generated 3D movement path post-path-planning, and the final artistry rendered by the robotic manipulator. Starting with fundamental linear patterns in the first, the subsequent layers delve deeper into intricate hand-drawn sketches, culminating in the third, which undertakes the challenge of a detailed landscape photograph. This stratified representation not only highlights the precision of our algorithms but also the versatile capability of our robotic manipulator in handling varying degrees of complexity.

Results

Video Archive: Watch the Robotic Manipulator in Action

Dive deep into our trove of recorded experiments, accessible via a series of YouTube links below. These videos chronicle the robotic manipulator’s journey, capturing its evolutions and refinements in real-time. Witness the transformations, the troubleshooting, and the exhilarating moments when theory turns into reality. From initial stumbles to its current fluid motions, you’ll gain a comprehensive insight into the entire developmental process. Grab some popcorn, and embark on this visual journey of technological evolution with us!

Gallery: A Historical Glimpse into our Technical Evolution

As we guide you through our visual journey, you’ll witness a chronological tapestry of our experiments, each timestamped for reference. It commenced humbly with basic robotic manipulator controls drawing a simple spiral. The evolution is palpable: from elementary line art like ‘Doraemon’, to more intricate sketches like a bear and a shark. An ode to our cultural appreciation, there’s even a rendition of the iconic logo from the beloved Japanese anime, ‘Ghost in the Shell’. At this juncture, it’s evident the jagged edges and less fluid lines hinting at the growing pains of our system.

Yet, as time marched on, our relentless endeavors to refine both the manipulator’s control and image processing bore fruit. Gradually, from rendering a seamless ‘Doraemon’ to masterfully capturing intricate landscape sceneries, our system’s prowess stands testament to our progress and dedication.

Gallery 17 images