AR/Projected Work Instructions - Championing New Technology:
Note: Due to NDAs limited photos, drawings, and models can be shared. The product and technology I showcase in my video is a demo-line. As much as possible has been included, if you'd like to learn more do not hesitate to reach out to me.
Project Overview:
Video here
In the manufacturing space at this employer, many of our processes are still entirely manual and are subject to human error. The obvious solution is to automate these processes, but this is an unrealistic short-term goal as many of our products and facilities are not designed for this. As a result I was tasked with answering the question "how can we bring automation into our manual processes without significant workstation overhaul?". To answer that I turned to a company called LightGuide. LightGuide provides you with their proprietary software and a simple office projector. With this you can project onto (or only illuminate certain areas) of a work surface with instructions, indicators, warnings, and more. The real power of their system is that you can pair it with common automation technologies and tools. In the video below you'll see I've done just that with an Atlas Copco smart torque tool, a Keyence VS-L500MX camera, and an Orbbec Femto Bolt 3D camera.
The entirety of the system in the video (and its supporting automation technology) was programmed, trained, mounted, wired, and debugged solely by myself! This particular system was an exploratory project to see if it answers the original question I was asked.
As you'll find on this page the answer was yes, not only is this a powerful tool, but minimal workstation overhaul is required as the mounting structure (containing all equipment) is built up around the already existing station. I am now working with several of our manufacturing sites to implement this enterprise wide with several more unique use cases. Be sure to turn the volume on for the video as I talk about the process at work!
Supporting Technology:
Orbbec Femto Bolt 3D Camera (Overhead):
-
Used to detect tool and hand location via topography toolset
-
Hand location monitored to confirm correct parts picked
-
Instructions advance only if correct part picked
-
-
Tool location monitored to confirm correct rundown location and order​
-
Tool locked if placed in incorrect rundown location
-
-
-
Used to detect assembly orientation via topography toolset
-
Provide feedback to program on current orientation
-
Program projects instructions to rotate to correct side
-
-
​
Atlas Copco Power Focus 8LV with Pistol Grip Torque Tool:
-
Used to fasten sheet metal paneling
-
Tool is only unlocked if 3D camera confirms correct rundown location
-
Final rundown torque and angle for each fastener recorded
-
Correct torque + angle: Program advances to project next instruction
-
Incorrect torque or angle: Tool locks to only reverse with projected instructions requesting fastener to be removed
-
Once back-out confirmed (back out angle greater than or equal to NOK rundown angle) projected instructions request rerun​
-
Repeat until correct torque and rundown angle recorded​​​
-
-
-
-
​
Keyence VS-L500MX Vision System (Rear of workstation):
-
Used to confirm correct decal placement
-
Within tolerance of projected location
-
Correct language (English vs French) sticker placed
-
- Rework Instructions projected if either incorrect​​​
Process/Program Flow:
The entirety of the program I had written (and demonstrated above) was based on the principles of "no fault forward". Simply put, the program will not have the projector display the next process/step until the supporting technology (3D camera, smart torque tool, and/or the industrial vision system) is able to confirm that the previous step was completed successfully. The flow of the program's logic is highlighted in the image below.
Fastening Sequence:
Once the 3D camera is able to confirm that the product is in the correct orientation, the LightGuide program instructs (and projects onto) the correct panel to retrieve. If the wrong panel or part is retrieved (as viewed by the 3D camera) the HMI and projector indicate this before reverting to the original instruction of which part to retrieve. Once the correct part is retrieved, the LightGuide program advances to the next step, instructing the operator to place the panel into the correct seam. Once completed, the program (using the 3D camera again) follows a similar process to instruct the operator to retrieve the correct fasteners. When the 3D camera is able to verify that the correct fasteners have been retrieved
the projector then highlights the first rundown location, instructing the operator to begin torqueing down. The torque tool is only unlocked once the 3D camera is able to verify the bolt and/or tooling is at the correct rundown location. As mentioned previously, if a NOK final rundown torque or angle is recorded by the Atlas Copco controller, the LightGuide program locks the tool to only reverse, instructing the operator that the rundown has failed and the fastener must be removed. Once the controller records that a torque of zero has been reached with a back out angle greater than or equal to the rundown angle, the operator is instructed to reperform the rundown. Once a good rundown is recorded the process repeats for the next rundown locations of said panel before repeating this process for the next panel.
​
Decal Sequence:
Once the 3D camera is able to confirm that the product is in the correct orientation, the LightGuide program instructs (and projects onto) the correct pick bin of decals to grab from. If the incorrect bin or part is retrieved (as viewed by the 3D camera) the HMI and projector indicate this and revert to the original instruction of which part to retrieve. Once the correct stickers have been picked, LightGuide instructs the projector to display the correct sticker locations on the panel facing the operator. Once the operator has placed all stickers, they are to hit the soft button to advance to the next step. The program instructs the operator to rotate to the next panel repeating the instructions described above. These placement and rotation instructions are repeated until the Keyence camera (located on the opposite side of the operator) has imaged all completed sides of the unit. If the Keyence program determines that a decal is out of tolerance, missing, or the incorrect language sticker is used it informs the LightGuide software which sticker (and corresponding side of the product) is incorrect. The LightGuide program instructs the operator to rotate the product to the panel(s) that require rework, projecting only onto the stickers that require rework. Once the operator has made the required changes and rotated the product for the vision system to see, the process is complete when no more reworks are required.
System Architecture:
The image to the right gives an overview of the system's architecture. The torque tool's tightening and loosening sequence was created by myself, passing information to the LightGuide software via a 5380 Compact Logix as an IO device. The torque controller transmits final rundown torque, rundown angle, and visa versa for loosening. The LightGuide software controls the tooling by passing bits that either lock or unlock the tooling.
​
The Keyence camera functions similarly to the Atlas Controller. The camera was trained using Keyence's VS Creator software to detect sticker position as well as text language. Again, each sticker's vision study is transmitted to the LightGuide program as a pass/fail bit. The LightGuide program controls the Keyence software by passing a bit to commence the vision
​​study. However the Femto Bolt 3D camera does not communicate with the LightGuide system via the PLC. Instead the 3D camera is wired directly to the LightGuide PC, making use of their "Observatory" tool. The observatory tool acts similar to Keyence's VS Creator software in that the camera can be trained and calibrated from the LightGuide program directly. This helps consolidate the number of support software programs needed to operate this system.​​