Previous baggage handler turned software dev here. Looks like you are targeting the bag room right now which is the best spot to start to prove the concept. It was surprising to me how manual the entire loading & sorting process was when I worked for the airlines.
I'm curious if there's any roadmap eventually to get this out to the ramp itself. Most of the back injuries seemed to happen in the bin itself because you have to often hunch/be on your knees in a bin tossing 50+ pound bags. I know airlines would probably be very hesitant to have any new equipment around their planes but just curious if there's any discussion around that.
Other side note, I also used to work in Cargo and always thought there could be way more efficient ways of loading loose packages that are on every flight and this seems to be a great possibility for those as well.
John B was obviously aware from previous experience what a manual and injury prone process this was, but I've also been really surprised as I've dived deeper into airport operations myself.
Bagroom is definitely what we're targeting first - being indoors (usually) is a huge plus, and lets us focus on the manipulation part of the problem without going fully mobile yet.
That said, we're definitely targeting tarmac/ramp operations, particularly between a TUG/PowerStow and narrow-body bag carts. Inside the bin is much trickier but we agree it's the least ergonomic part of the job, you just can't move a massive industrial arm in and out of a plane very easily. We have it on our longer-term roadmap, though, and intend to leverage the baggage dynamics data we collect everywhere else to give us a head start on the packing and manipulation problems there, just with a different mechanism.
Cargo packing is a huge area of interest for us! Particularly around optimizing weight distribution in loaded planes, or just optimizing packing efficiency in general.
This is much like a palletizer. Here's a mixed-case cage palletizer, which fills up wire cages open on one side with various boxes.[1] For rectangular boxes, palletizing is a solved problem with many companies selling systems.
Irregular items are a problem. In the baggage handling video, the last bag, the soft one with straps, is sticking out after being placed in the baggage container. That's the same problem which keeps Amazon from totally automating picking. They keep trying, but nothing works well enough yet.
Yep! We're obviously operating alongside a very well established world of palletization and other order fulfillment type robots.
We think that due to irregularity that it's not an easy tech transfer from the existing logistics world into the aviation world. We're very interesting in looking the other direction, though!
This is very cool, thank you for sharing. I work in automation and SWE for a certain 4-letter organization that delivers your mail. Pick/place is something we've rolled out using articulated and delta robots with vacuum end effectors, and it's an interesting and challenging space to be in. As in your case, bags and other amorphous shapes are always the most difficult. It's always an uphill battle to hit throughput targets due to exception cases that can stop things until a human gets involved. Ultimately, it can be a struggle to avoid overpromising and to generate ROI since automation is so costly, especially when there's no opportunity to bound the problem by influencing the inputs to your system or the output requirements (in your case, the cart being loaded). Best of luck and looking forward to seeing your new end effector.
I'm surprised the end effector works on misshapen fabric bags. Makes me jealous that I can't test it.
Are you considering dynamic trajectory constraints in the planner (e.g. for multiple robots loading simultaneously)? That was a thorny problem back when I worked on arms.
There are a lot of constraints in our planning, including what actions we can do for a particular bag/gripper interaction. Multiple robots working in collision range of each other isn't on the roadmap right now, but it's always a possibility.
Suction works really well but it's not enough, we're rolling out a new gripper soon that covers more cases mechanically (there's a very long tail in the distribution of what comes through airports).
Super cool! You are additionally going to be saving a lot of workers from getting chronic back pains.
I thought maybe y'all are going too slow, but after looking at some baggage handling videos, it seems like you're at a comparable speed already?
In deployment these things are probably going to be on some kind of cart system, I presume all your algorithms can handle small changes in the XY travel plane (i.e. the robots location w.r.t the end of the belt).
Thanks! It's a really destructive job for workers' lower backs and elbow tendons. This actually puts into perspective the blended throughput rate - you can imagine loading a few bags really quickly, but moving 20kg bags for 10 minutes straight will slow you down. That said - we still have a lot of runway on speed for these mechanisms and are still running fairly conservatively as we shake out our software.
There are a few ways we plan to deploy (some fixed rails, some mobile). Since the carts we're loading aren't placed with much precision, even the fixed deployments need to do serious environmental perception / localization.
It's not as though this is a profession which people spend a decade learning to perform. And it's not 2009 any more, there is demand for labor throughout the economy. I think the benefit of getting people out of physically damaging work outweighs the pain of having to find a new job in this case
That's not how this works. Most airports are looking to expand, they can keep the same staffing levels with a rapid expansion in throughput with these machines.
(Assuming the unions allow it).
> Unfortunately for airlines, passengers don’t package their luggage in nicely uniform cardboard boxes. If they did, then the airlines could benefit directly from the recent takeoff in manipulator tech for warehouses. But airline luggage is way more wacky and irregular. If robots are going to handle it, they need to reason about how to grasp each item, handle its deformability, stack it in a stable way, and do all of this quickly, safely, and reliably.
Curious if you guys have put any thought into seeing if there's an operational change you could introduce to airlines, that would result in the tech side being a lot easier?
Palletizing logistics for consumer airtravel would be interesting...
Operational changes for airlines are quite tricky - one of our bets is that most of the value for customers here is in handling "brownfield" deployments where you drop into an existing process, and that intelligence (or at least, good perception and reactive planning) really unlocks this ability from the robotics side.
For widebody planes, bags are already loaded into Unit Load Devices (ULDs), which are large semi-truncated boxes that get loaded directly onto aircraft. Narrow body planes don't use these (apparently) because they impact turn-time and decrease the amount of time a plane can be in the air, and also impacts how quickly bags come out, since it adds an extra step to unloading.
Many airport conveyance systems also load each bag into a bin, but those bins aren't loaded into the airplane because they belong to the airport and waste space/weight.
The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
Was on a plane a few days ago watching someone do this out the window 10 yards away.
- Seemed like the baggage handler was required to do a very fast cycle of <scan, toss, align, repeat next bag>. Automation seems helpful, and certainly it’s hard labor.
- This was also a young woman, in presumably a safe union job, working in a very pricey city (one of the mountain west towns that exploded). Adios union job hello robots.
Tricky ethics! Outside of picking stuff up and putting stuff down, not too many automation union-safe jobs left. Saving them from back pain is also going to be saving them from a job.
We're really focused on health and safety aspects of this job - in a repetitive stress sense, these jobs are much more dangerous than many people imagine they would be and people end up with lifelong injuries.
Generally, regulators seem to be moving in this direction as well. The EU has introduced new regulations on the total amount of weight someone can move in a shift, and the Dutch government has mandated that baggage handling move away from manual processes like this in the near future.
Despite the focus difference, do you think it's unlikely that automating baggage handlers will replace their jobs?
The regulator focus seems like it'd reduce the max allowed weight of a checked bag, not automate the baggage handler handing the checked bag. So, I don't see the similarity between the regulatory push and your product? Edit - to clarify, beyond what Dutch regulators say about Dutch markets, which are a very small subset of "regulator focus" internationally.
The suction "cup" seems to be doing a good job, but I don't think bags are made to be handled that way. Did you alternate test with some kind of a grabber (like the claw machine, but actually effective)? It would make the grasp selection problem much more tractable imo, since all bags have at least one "grasp point" built in.
In teleoperated mode, I'm guessing you're using the captured data to train the autonomous mode?
Final note, the robot looks beefy enough to lift an entire airplane, forget luggage -- is it overengineered on purpose?
The cup has taken us very far, which we're excited about, but it's definitely not enough - we're currently testing a multimodal claw-ish + suction gripper, which we've had good results with so far but aren't ready to unveil.
The teleop data is really useful for training data indeed, and lets us collect data on current failure points (e.g. with suction, just how far can we tilt this fabric bag before it peels away, etc). We're not going full behavior-cloned end-to-end for a lot of reasons (sample complexity, safety, adaptability, etc), but we do a lot of learning in specific parts of the system (particularly around grasping and placement).
The robot is indeed beefy, as many robots rated for 50kg applications are (check them out online). We've accidentally stress tested this unit way beyond 50kg without a hiccup, so we're very interested in figuring out what the right-size unit is for our application. There are a few other great aspects to this unit - it's a 7-DOF arm + 1 more DOF for the linear rail, so we have two extra degrees of freedom to play with for collision avoidance during planning.
As part of one of my classes in engineering school we went and toured the DIA baggage handling system, studied why and how it was such a big failure and what they did to fix it.
There have been some solid attempts in this space before - many projects take on the whole baggage system design and end up very very complex and often over budget. We're focusing on introducing tech that plays well in a larger system, particularly in "brownfield" existing processes - our bet is that recent advances in robot autonomy give us ability to handle items that weren't possible before, and therefore our units can be introduced in a more flexible way.
Using a vacuum to pick up baggage is a very interesting choice. I wonder how it would fare with extremely uneven surfaces or even porous ones.
Also --- I couldn't see any obvious sensors. What sensors are you using to perceive the bag? I am imagining some kind of RGB-D sensor like a Kinect (or its successors like the Orbbec).
Suction has gotten us pretty far at the prototype level but definitely isn't enough - we're testing out some new gripper designs that use suction as a broader part of an overall grasping system.
For these videos we have lidars and two Intel Realsense depth cameras mounted to the safety cage and on a wall. We're working on moving as many sensor on-robot as possible in the near future to aid with deployability.
This is excellent! I live in a small city with an airport so have known many baggage handlers during my years here. I honestly didn’t believe how common back and shoulder injuries were in the industry. This could prevent a whole lot of medical problems.
Solid question and something we think about a lot! Worst case is a weak zipper or similar. We're bringing a new gripper online which is more multimodal - some mechanical grasping, some suction, and the ability to choose what you use. We're moving away from pure suction partly for this reason and partly for textiles.
Suction is great though, and ~75% of bags checked through the US are hardshells, so it's something we're not ready to ignore entirely.
Also good question and something we've thought about. The difficulty there is actually getting the forklift tines out after placement. Actual forklifts in real warehouses rely on pallets as an affordance for manipulation, and we don't have that luxury here.
There have been some neat attempts with short conveyor belts as end effector tools [1]! Generally these systems rely on being able to rearchitect a significant amount of the process (building a controllable conveyor belt or rearchitecting part of the bag-room), and we're focused on dropping into existing processes.
call me disappointed. Not to belittle whatever work was put into it as there were probably a lot, just the one-suction-cup way of doing it seems so 30 years ago. These days i'd have expected [several] hands with fingers.
Great point and we 100% agree! We have a new multi-modal gripper that we're testing now but are keeping hush while we bring it up. (We're actually swapping out a new arm too for a variety of reasons)
These videos are more to set a scene for where we're operating the the general process.
Teleop and monitoring are systems that we've built ourselves and are pretty happy with. Since we use MuJoCo for simulation/visualization and some kinematics subroutines, to visualize, I just keep the MuJoCo GL context open after rendering and then throw all of our sensor data into it - it's very performant and low latency!
We've since introduced a message-bus layer that makes it possible to do it all over the internet etc, but adds the associated serialization and transport latency.
This is awesome! It's amazing how many jobs there are that require heavy manual lifting with repetitive motion that will be up-skilled in the coming years. New roles will be much more monitoring and problem solving which.
Good video! The overall question here is the blended rate of bags placed per minute, rather than how fast each action needs to be.
That said, the arm itself can move 180deg/s in every joint (roughly 5m/s max at the end effector) - these videos are still very much v1 and we're looking forward to leveraging more of the mechanical capability with a better gripper, better perception, and some new planning techniques we're rolling out in the next few months.
Previous baggage handler turned software dev here. Looks like you are targeting the bag room right now which is the best spot to start to prove the concept. It was surprising to me how manual the entire loading & sorting process was when I worked for the airlines.
I'm curious if there's any roadmap eventually to get this out to the ramp itself. Most of the back injuries seemed to happen in the bin itself because you have to often hunch/be on your knees in a bin tossing 50+ pound bags. I know airlines would probably be very hesitant to have any new equipment around their planes but just curious if there's any discussion around that.
Other side note, I also used to work in Cargo and always thought there could be way more efficient ways of loading loose packages that are on every flight and this seems to be a great possibility for those as well.
Awesome work & will keep tabs on it!
Thanks for the feedback!
John B was obviously aware from previous experience what a manual and injury prone process this was, but I've also been really surprised as I've dived deeper into airport operations myself.
Bagroom is definitely what we're targeting first - being indoors (usually) is a huge plus, and lets us focus on the manipulation part of the problem without going fully mobile yet.
That said, we're definitely targeting tarmac/ramp operations, particularly between a TUG/PowerStow and narrow-body bag carts. Inside the bin is much trickier but we agree it's the least ergonomic part of the job, you just can't move a massive industrial arm in and out of a plane very easily. We have it on our longer-term roadmap, though, and intend to leverage the baggage dynamics data we collect everywhere else to give us a head start on the packing and manipulation problems there, just with a different mechanism.
Cargo packing is a huge area of interest for us! Particularly around optimizing weight distribution in loaded planes, or just optimizing packing efficiency in general.
This is much like a palletizer. Here's a mixed-case cage palletizer, which fills up wire cages open on one side with various boxes.[1] For rectangular boxes, palletizing is a solved problem with many companies selling systems.
Irregular items are a problem. In the baggage handling video, the last bag, the soft one with straps, is sticking out after being placed in the baggage container. That's the same problem which keeps Amazon from totally automating picking. They keep trying, but nothing works well enough yet.
[1] https://www.youtube.com/watch?v=TN-6QaLd3VY
Yep! We're obviously operating alongside a very well established world of palletization and other order fulfillment type robots.
We think that due to irregularity that it's not an easy tech transfer from the existing logistics world into the aviation world. We're very interesting in looking the other direction, though!
This is very cool, thank you for sharing. I work in automation and SWE for a certain 4-letter organization that delivers your mail. Pick/place is something we've rolled out using articulated and delta robots with vacuum end effectors, and it's an interesting and challenging space to be in. As in your case, bags and other amorphous shapes are always the most difficult. It's always an uphill battle to hit throughput targets due to exception cases that can stop things until a human gets involved. Ultimately, it can be a struggle to avoid overpromising and to generate ROI since automation is so costly, especially when there's no opportunity to bound the problem by influencing the inputs to your system or the output requirements (in your case, the cart being loaded). Best of luck and looking forward to seeing your new end effector.
I'm surprised the end effector works on misshapen fabric bags. Makes me jealous that I can't test it.
Are you considering dynamic trajectory constraints in the planner (e.g. for multiple robots loading simultaneously)? That was a thorny problem back when I worked on arms.
There are a lot of constraints in our planning, including what actions we can do for a particular bag/gripper interaction. Multiple robots working in collision range of each other isn't on the roadmap right now, but it's always a possibility.
Suction works really well but it's not enough, we're rolling out a new gripper soon that covers more cases mechanically (there's a very long tail in the distribution of what comes through airports).
Super cool! You are additionally going to be saving a lot of workers from getting chronic back pains. I thought maybe y'all are going too slow, but after looking at some baggage handling videos, it seems like you're at a comparable speed already?
In deployment these things are probably going to be on some kind of cart system, I presume all your algorithms can handle small changes in the XY travel plane (i.e. the robots location w.r.t the end of the belt).
Thanks! It's a really destructive job for workers' lower backs and elbow tendons. This actually puts into perspective the blended throughput rate - you can imagine loading a few bags really quickly, but moving 20kg bags for 10 minutes straight will slow you down. That said - we still have a lot of runway on speed for these mechanisms and are still running fairly conservatively as we shake out our software.
There are a few ways we plan to deploy (some fixed rails, some mobile). Since the carts we're loading aren't placed with much precision, even the fixed deployments need to do serious environmental perception / localization.
> You are additionally going to be saving a lot of workers from getting chronic back pains.
That'll probably be a big comfort to those who lose their jobs
It's not as though this is a profession which people spend a decade learning to perform. And it's not 2009 any more, there is demand for labor throughout the economy. I think the benefit of getting people out of physically damaging work outweighs the pain of having to find a new job in this case
That's not how this works. Most airports are looking to expand, they can keep the same staffing levels with a rapid expansion in throughput with these machines. (Assuming the unions allow it).
> Unfortunately for airlines, passengers don’t package their luggage in nicely uniform cardboard boxes. If they did, then the airlines could benefit directly from the recent takeoff in manipulator tech for warehouses. But airline luggage is way more wacky and irregular. If robots are going to handle it, they need to reason about how to grasp each item, handle its deformability, stack it in a stable way, and do all of this quickly, safely, and reliably.
Curious if you guys have put any thought into seeing if there's an operational change you could introduce to airlines, that would result in the tech side being a lot easier?
Palletizing logistics for consumer airtravel would be interesting...
Operational changes for airlines are quite tricky - one of our bets is that most of the value for customers here is in handling "brownfield" deployments where you drop into an existing process, and that intelligence (or at least, good perception and reactive planning) really unlocks this ability from the robotics side.
For widebody planes, bags are already loaded into Unit Load Devices (ULDs), which are large semi-truncated boxes that get loaded directly onto aircraft. Narrow body planes don't use these (apparently) because they impact turn-time and decrease the amount of time a plane can be in the air, and also impacts how quickly bags come out, since it adds an extra step to unloading.
Many airport conveyance systems also load each bag into a bin, but those bins aren't loaded into the airplane because they belong to the airport and waste space/weight.
The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
This looks cool, interested to see where it goes in the future.
Going to be a pest for a minute - have you considered a stack other than ROS2?
Haha I want to work for you guys! This sounds like so much fun, hope you guys become trillionaires!
Was on a plane a few days ago watching someone do this out the window 10 yards away.
- Seemed like the baggage handler was required to do a very fast cycle of <scan, toss, align, repeat next bag>. Automation seems helpful, and certainly it’s hard labor.
- This was also a young woman, in presumably a safe union job, working in a very pricey city (one of the mountain west towns that exploded). Adios union job hello robots.
Tricky ethics! Outside of picking stuff up and putting stuff down, not too many automation union-safe jobs left. Saving them from back pain is also going to be saving them from a job.
We're really focused on health and safety aspects of this job - in a repetitive stress sense, these jobs are much more dangerous than many people imagine they would be and people end up with lifelong injuries.
Generally, regulators seem to be moving in this direction as well. The EU has introduced new regulations on the total amount of weight someone can move in a shift, and the Dutch government has mandated that baggage handling move away from manual processes like this in the near future.
Despite the focus difference, do you think it's unlikely that automating baggage handlers will replace their jobs?
The regulator focus seems like it'd reduce the max allowed weight of a checked bag, not automate the baggage handler handing the checked bag. So, I don't see the similarity between the regulatory push and your product? Edit - to clarify, beyond what Dutch regulators say about Dutch markets, which are a very small subset of "regulator focus" internationally.
The suction "cup" seems to be doing a good job, but I don't think bags are made to be handled that way. Did you alternate test with some kind of a grabber (like the claw machine, but actually effective)? It would make the grasp selection problem much more tractable imo, since all bags have at least one "grasp point" built in.
In teleoperated mode, I'm guessing you're using the captured data to train the autonomous mode?
Final note, the robot looks beefy enough to lift an entire airplane, forget luggage -- is it overengineered on purpose?
The cup has taken us very far, which we're excited about, but it's definitely not enough - we're currently testing a multimodal claw-ish + suction gripper, which we've had good results with so far but aren't ready to unveil.
The teleop data is really useful for training data indeed, and lets us collect data on current failure points (e.g. with suction, just how far can we tilt this fabric bag before it peels away, etc). We're not going full behavior-cloned end-to-end for a lot of reasons (sample complexity, safety, adaptability, etc), but we do a lot of learning in specific parts of the system (particularly around grasping and placement).
The robot is indeed beefy, as many robots rated for 50kg applications are (check them out online). We've accidentally stress tested this unit way beyond 50kg without a hiccup, so we're very interested in figuring out what the right-size unit is for our application. There are a few other great aspects to this unit - it's a 7-DOF arm + 1 more DOF for the linear rail, so we have two extra degrees of freedom to play with for collision avoidance during planning.
As part of one of my classes in engineering school we went and toured the DIA baggage handling system, studied why and how it was such a big failure and what they did to fix it.
There have been some solid attempts in this space before - many projects take on the whole baggage system design and end up very very complex and often over budget. We're focusing on introducing tech that plays well in a larger system, particularly in "brownfield" existing processes - our bet is that recent advances in robot autonomy give us ability to handle items that weren't possible before, and therefore our units can be introduced in a more flexible way.
Using a vacuum to pick up baggage is a very interesting choice. I wonder how it would fare with extremely uneven surfaces or even porous ones.
Also --- I couldn't see any obvious sensors. What sensors are you using to perceive the bag? I am imagining some kind of RGB-D sensor like a Kinect (or its successors like the Orbbec).
Suction has gotten us pretty far at the prototype level but definitely isn't enough - we're testing out some new gripper designs that use suction as a broader part of an overall grasping system.
For these videos we have lidars and two Intel Realsense depth cameras mounted to the safety cage and on a wall. We're working on moving as many sensor on-robot as possible in the near future to aid with deployability.
This is excellent! I live in a small city with an airport so have known many baggage handlers during my years here. I honestly didn’t believe how common back and shoulder injuries were in the industry. This could prevent a whole lot of medical problems.
Are most suitcases designed to bear its fully loaded weight just by sucking on one surface?
Solid question and something we think about a lot! Worst case is a weak zipper or similar. We're bringing a new gripper online which is more multimodal - some mechanical grasping, some suction, and the ability to choose what you use. We're moving away from pure suction partly for this reason and partly for textiles.
Suction is great though, and ~75% of bags checked through the US are hardshells, so it's something we're not ready to ignore entirely.
How about a forklift like handling ?
Also good question and something we've thought about. The difficulty there is actually getting the forklift tines out after placement. Actual forklifts in real warehouses rely on pallets as an affordance for manipulation, and we don't have that luxury here.
There have been some neat attempts with short conveyor belts as end effector tools [1]! Generally these systems rely on being able to rearchitect a significant amount of the process (building a controllable conveyor belt or rearchitecting part of the bag-room), and we're focused on dropping into existing processes.
[1] https://www.youtube.com/watch?v=n2Wy_tduq5k
Another robot to clean up after the zipper fails is wip
call me disappointed. Not to belittle whatever work was put into it as there were probably a lot, just the one-suction-cup way of doing it seems so 30 years ago. These days i'd have expected [several] hands with fingers.
Great point and we 100% agree! We have a new multi-modal gripper that we're testing now but are keeping hush while we bring it up. (We're actually swapping out a new arm too for a variety of reasons)
These videos are more to set a scene for where we're operating the the general process.
I've heard about you guys before. Nice application! What are you using for teleop/remote-monitoring? Did you build that yourself?
Teleop and monitoring are systems that we've built ourselves and are pretty happy with. Since we use MuJoCo for simulation/visualization and some kinematics subroutines, to visualize, I just keep the MuJoCo GL context open after rendering and then throw all of our sensor data into it - it's very performant and low latency!
We've since introduced a message-bus layer that makes it possible to do it all over the internet etc, but adds the associated serialization and transport latency.
> to do it all over the internet, but adds the associated serialization and transport latency.
I wrote this blog post on that topic a while back after having seen various approaches robotics companies take and their shortcomings: https://transitiverobotics.com/blog/streaming-video-from-rob...
Excellent post! Curious if WebRTC can be adapted for 3d sensor data and would love to chat more about it - I'll send an email!
This is awesome! It's amazing how many jobs there are that require heavy manual lifting with repetitive motion that will be up-skilled in the coming years. New roles will be much more monitoring and problem solving which.
How much faster will the robotic arms be able to go? Currently this looks far too slow, though as a v1 it's great.
It's about as fast as human does it now.
Not even close
https://youtube.com/shorts/KMlCCyQPfME?si=BsyZ9QZgZDJWVh4f
Good video! The overall question here is the blended rate of bags placed per minute, rather than how fast each action needs to be.
That said, the arm itself can move 180deg/s in every joint (roughly 5m/s max at the end effector) - these videos are still very much v1 and we're looking forward to leveraging more of the mechanical capability with a better gripper, better perception, and some new planning techniques we're rolling out in the next few months.
I guess, just my opinion (probably worthless), if you can get it to speed up even ~20%, I think that would be enough to make the average person agree.
Well, this didn't go very well in Denver when they tried it. Good luck.
Please tell me the robot's name is not Iggy.
No - people keep asking us to name our robots, but we haven't landed on anything yet!