Previous baggage handler turned software dev here. Looks like you are targeting the bag room right now which is the best spot to start to prove the concept. It was surprising to me how manual the entire loading & sorting process was when I worked for the airlines.
I'm curious if there's any roadmap eventually to get this out to the ramp itself. Most of the back injuries seemed to happen in the bin itself because you have to often hunch/be on your knees in a bin tossing 50+ pound bags. I know airlines would probably be very hesitant to have any new equipment around their planes but just curious if there's any discussion around that.
Other side note, I also used to work in Cargo and always thought there could be way more efficient ways of loading loose packages that are on every flight and this seems to be a great possibility for those as well.
John B was obviously aware from previous experience what a manual and injury prone process this was, but I've also been really surprised as I've dived deeper into airport operations myself.
Bagroom is definitely what we're targeting first - being indoors (usually) is a huge plus, and lets us focus on the manipulation part of the problem without going fully mobile yet.
That said, we're definitely targeting tarmac/ramp operations, particularly between a TUG/PowerStow and narrow-body bag carts. Inside the bin is much trickier but we agree it's the least ergonomic part of the job, you just can't move a massive industrial arm in and out of a plane very easily. We have it on our longer-term roadmap, though, and intend to leverage the baggage dynamics data we collect everywhere else to give us a head start on the packing and manipulation problems there, just with a different mechanism.
Cargo packing is a huge area of interest for us! Particularly around optimizing weight distribution in loaded planes, or just optimizing packing efficiency in general.
Why not rear-load like military aircraft and avoid that 90º turn? Then you could have a hull-shaped hopper to drop the bags in, so they are tetris'd together in the right geometry before you get to the ramp. Roll the hopper in, have a pez-despenser style pusher so as you withdraw the hopper, the bags are left behind in the aircraft?
This is much like a palletizer. Here's a mixed-case cage palletizer, which fills up wire cages open on one side with various boxes.[1] For rectangular boxes, palletizing is a solved problem with many companies selling systems.
Irregular items are a problem. In the baggage handling video, the last bag, the soft one with straps, is sticking out after being placed in the baggage container. That's the same problem which keeps Amazon from totally automating picking. They keep trying, but nothing works well enough yet.
Yep! We're obviously operating alongside a very well established world of palletization and other order fulfillment type robots.
We think that due to irregularity that it's not an easy tech transfer from the existing logistics world into the aviation world. We're very interesting in looking the other direction, though!
This is very cool, thank you for sharing. I work in automation and SWE for a certain 4-letter organization that delivers your mail. Pick/place is something we've rolled out using articulated and delta robots with vacuum end effectors, and it's an interesting and challenging space to be in. As in your case, bags and other amorphous shapes are always the most difficult. It's always an uphill battle to hit throughput targets due to exception cases that can stop things until a human gets involved. Ultimately, it can be a struggle to avoid overpromising and to generate ROI since automation is so costly, especially when there's no opportunity to bound the problem by influencing the inputs to your system or the output requirements (in your case, the cart being loaded). Best of luck and looking forward to seeing your new end effector.
I'm surprised the end effector works on misshapen fabric bags. Makes me jealous that I can't test it.
Are you considering dynamic trajectory constraints in the planner (e.g. for multiple robots loading simultaneously)? That was a thorny problem back when I worked on arms.
There are a lot of constraints in our planning, including what actions we can do for a particular bag/gripper interaction. Multiple robots working in collision range of each other isn't on the roadmap right now, but it's always a possibility.
Suction works really well but it's not enough, we're rolling out a new gripper soon that covers more cases mechanically (there's a very long tail in the distribution of what comes through airports).
Was on a plane a few days ago watching someone do this out the window 10 yards away.
- Seemed like the baggage handler was required to do a very fast cycle of <scan, toss, align, repeat next bag>. Automation seems helpful, and certainly it’s hard labor.
- This was also a young woman, in presumably a safe union job, working in a very pricey city (one of the mountain west towns that exploded). Adios union job hello robots.
Tricky ethics! Outside of picking stuff up and putting stuff down, not too many automation union-safe jobs left. Saving them from back pain is also going to be saving them from a job.
We're really focused on health and safety aspects of this job - in a repetitive stress sense, these jobs are much more dangerous than many people imagine they would be and people end up with lifelong injuries.
Generally, regulators seem to be moving in this direction as well. The EU has introduced new regulations on the total amount of weight someone can move in a shift, and the Dutch government has mandated that baggage handling move away from manual processes like this in the near future.
Despite the focus difference, do you think it's unlikely that automating baggage handlers will replace their jobs?
The regulator focus seems like it'd reduce the max allowed weight of a checked bag, not automate the baggage handler handing the checked bag. So, I don't see the similarity between the regulatory push and your product? Edit - to clarify, beyond what Dutch regulators say about Dutch markets, which are a very small subset of "regulator focus" internationally.
This is excellent! I live in a small city with an airport so have known many baggage handlers during my years here. I honestly didn’t believe how common back and shoulder injuries were in the industry. This could prevent a whole lot of medical problems.
Super cool! You are additionally going to be saving a lot of workers from getting chronic back pains.
I thought maybe y'all are going too slow, but after looking at some baggage handling videos, it seems like you're at a comparable speed already?
In deployment these things are probably going to be on some kind of cart system, I presume all your algorithms can handle small changes in the XY travel plane (i.e. the robots location w.r.t the end of the belt).
Thanks! It's a really destructive job for workers' lower backs and elbow tendons. This actually puts into perspective the blended throughput rate - you can imagine loading a few bags really quickly, but moving 20kg bags for 10 minutes straight will slow you down. That said - we still have a lot of runway on speed for these mechanisms and are still running fairly conservatively as we shake out our software.
There are a few ways we plan to deploy (some fixed rails, some mobile). Since the carts we're loading aren't placed with much precision, even the fixed deployments need to do serious environmental perception / localization.
It's not as though this is a profession which people spend a decade learning to perform. And it's not 2009 any more, there is demand for labor throughout the economy. I think the benefit of getting people out of physically damaging work outweighs the pain of having to find a new job in this case
I don't follow how the existence of these robots is a benefit to the worker. A robot took their job, that does not mean another job was created out of thin air.
I'm not saying this isn't back breaking work, I'm just saying that your logic doesn't make sense.
Many people will be put out of a low-skilled manual job, but many better-paid jobs are created for robot salespeople, robot installers, training specialists, delivery drivers, maintenance technicians etc. Those who adapt to these new jobs will thrive.
That's not how this works. Most airports are looking to expand, they can keep the same staffing levels with a rapid expansion in throughput with these machines.
(Assuming the unions allow it).
Another HN roboticist chiming in; the videos look good! I have many questions, but will keep it to just a couple for brevity's sake.
- How much are you able to use the robot's internals to estimate the gripped bag's inertial properties? If you're trying to put rigid, heavy bags below light, amorphous bags, are you adjusting final placement location on-the-fly?
- How dynamic is your scene beyond what we can observe? If you're using light curtains with a single robot on a track, and you're able to estimate some rough geometry of and track the bags down the conveyance, and you are updating occupancy of the bin as you're going, what else is there?
- Is this just inside for the foreseeable future or are you all going to tackle unpacking outside, as well as all the, ahem, baggage that comes along with operating outside the terminal walls.
Nice and straightforward problem, relatively speaking.
> Unfortunately for airlines, passengers don’t package their luggage in nicely uniform cardboard boxes. If they did, then the airlines could benefit directly from the recent takeoff in manipulator tech for warehouses. But airline luggage is way more wacky and irregular. If robots are going to handle it, they need to reason about how to grasp each item, handle its deformability, stack it in a stable way, and do all of this quickly, safely, and reliably.
Curious if you guys have put any thought into seeing if there's an operational change you could introduce to airlines, that would result in the tech side being a lot easier?
Palletizing logistics for consumer airtravel would be interesting...
Operational changes for airlines are quite tricky - one of our bets is that most of the value for customers here is in handling "brownfield" deployments where you drop into an existing process, and that intelligence (or at least, good perception and reactive planning) really unlocks this ability from the robotics side.
For widebody planes, bags are already loaded into Unit Load Devices (ULDs), which are large semi-truncated boxes that get loaded directly onto aircraft. Narrow body planes don't use these (apparently) because they impact turn-time and decrease the amount of time a plane can be in the air, and also impacts how quickly bags come out, since it adds an extra step to unloading.
Many airport conveyance systems also load each bag into a bin, but those bins aren't loaded into the airplane because they belong to the airport and waste space/weight.
The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
> The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
I could see a budget airline cooperating with a luggage/case manufacturer and offering "if your checked bag is EXACTLY a Pelican 1615TRVL, it flies free/cheap" - and then work with them to design a case that is easily automatable.
The suction "cup" seems to be doing a good job, but I don't think bags are made to be handled that way. Did you alternate test with some kind of a grabber (like the claw machine, but actually effective)? It would make the grasp selection problem much more tractable imo, since all bags have at least one "grasp point" built in.
In teleoperated mode, I'm guessing you're using the captured data to train the autonomous mode?
Final note, the robot looks beefy enough to lift an entire airplane, forget luggage -- is it overengineered on purpose?
The cup has taken us very far, which we're excited about, but it's definitely not enough - we're currently testing a multimodal claw-ish + suction gripper, which we've had good results with so far but aren't ready to unveil.
The teleop data is really useful for training data indeed, and lets us collect data on current failure points (e.g. with suction, just how far can we tilt this fabric bag before it peels away, etc). We're not going full behavior-cloned end-to-end for a lot of reasons (sample complexity, safety, adaptability, etc), but we do a lot of learning in specific parts of the system (particularly around grasping and placement).
The robot is indeed beefy, as many robots rated for 50kg applications are (check them out online). We've accidentally stress tested this unit way beyond 50kg without a hiccup, so we're very interested in figuring out what the right-size unit is for our application. There are a few other great aspects to this unit - it's a 7-DOF arm + 1 more DOF for the linear rail, so we have two extra degrees of freedom to play with for collision avoidance during planning.
Suction cups are way better than multi-finger grippers.
Multi-finger grippers rely on "affordances" i.e. space between the boxes to get the fingers in. If the boxes are pushed up against one another, you've got to do something to make space for the fingers first. So to grab a box with a multi-finger gripper you need space at the top, left and right.
Suction cups, on the other hand? They only need access to the top. No need for space at the left and right, no need to slide things to make space, just apply the suction cup and pull upwards.
Suction cups can also be made of soft, flexible rubber so if the planning is a bit inaccurate? No problem, the suction cup just bends. A multi-finger gripper, though? If the fingers are rigid and strong enough to lift a 30kg bag then they're also probably rigid and strong enough to punch a hole in an adjacent bag.
Suction cups do have a bunch of disadvantages, of course. Porous materials you can't get a seal on. Vacuuming up detritus and messing up your vacuum generator. The payload swinging about on the flexible suction cup. Losing vacuum and dropping the load if there's a power cut.
But they're certainly a great starting point in an application like this one.
As part of one of my classes in engineering school we went and toured the DIA baggage handling system, studied why and how it was such a big failure and what they did to fix it.
There have been some solid attempts in this space before - many projects take on the whole baggage system design and end up very very complex and often over budget. We're focusing on introducing tech that plays well in a larger system, particularly in "brownfield" existing processes - our bet is that recent advances in robot autonomy give us ability to handle items that weren't possible before, and therefore our units can be introduced in a more flexible way.
Using a vacuum to pick up baggage is a very interesting choice. I wonder how it would fare with extremely uneven surfaces or even porous ones.
Also --- I couldn't see any obvious sensors. What sensors are you using to perceive the bag? I am imagining some kind of RGB-D sensor like a Kinect (or its successors like the Orbbec).
Suction has gotten us pretty far at the prototype level but definitely isn't enough - we're testing out some new gripper designs that use suction as a broader part of an overall grasping system.
For these videos we have lidars and two Intel Realsense depth cameras mounted to the safety cage and on a wall. We're working on moving as many sensor on-robot as possible in the near future to aid with deployability.
Solid question and something we think about a lot! Worst case is a weak zipper or similar. We're bringing a new gripper online which is more multimodal - some mechanical grasping, some suction, and the ability to choose what you use. We're moving away from pure suction partly for this reason and partly for textiles.
Suction is great though, and ~75% of bags checked through the US are hardshells, so it's something we're not ready to ignore entirely.
Also good question and something we've thought about. The difficulty there is actually getting the forklift tines out after placement. Actual forklifts in real warehouses rely on pallets as an affordance for manipulation, and we don't have that luxury here.
There have been some neat attempts with short conveyor belts as end effector tools [1]! Generally these systems rely on being able to rearchitect a significant amount of the process (building a controllable conveyor belt or rearchitecting part of the bag-room), and we're focused on dropping into existing processes.
How would a single armed robot handle oversized luggage or luggage with weird centre of mass? Or poorly packed luggage who's centre of mass is shifting?
Say golf club hard shells case where the owner's left a bunch of golf balls in the bottom of one of the pockets. It's difficult to tell that one end's quite a lot heavier then the other until you actually try to pick it up.
At least to me I don't see a good way of handling this edge case. You'd need either a second manipulator, a very fast feedback system, or a way for the weird luggage to be diverted for human intervention.
Great questions and a few answers to various parts of your question that you've mostly identified yourself I think:
- Golf clubs specifically (and most sporting equipment) actually goes down a different pipeline in most airports, since generally this stuff doesn't behave well on conveyor belts.
- Data driven approaches can tell you a lot just from visual information, usually about deformability of objects, but also about expected centers of mass, etc
- Part of the reason we're using single big robots is because you can use heavy duty end-effectors - grasp all the way around the object with a grasp that's predicted to be robust to these kinds of perturbances, and then use quick feedback to safely execute a plan to place it
You're absolutely correct though, that there's a long tail of things coming through and that some objects are very, very difficult. Our problem formulation then becomes identifying confidence in graspability, and deciding explicitly that we shouldn't attempt to grasp some object and should instead flag them for human handling.
Teleop and monitoring are systems that we've built ourselves and are pretty happy with. Since we use MuJoCo for simulation/visualization and some kinematics subroutines, to visualize, I just keep the MuJoCo GL context open after rendering and then throw all of our sensor data into it - it's very performant and low latency!
We've since introduced a message-bus layer that makes it possible to do it all over the internet etc, but adds the associated serialization and transport latency.
For a robotics company, their code is the "secret sauce" that makes their company valuable. It wouldn't make sense to open-source it all and let their competitors do the same thing without having had to spend so much money and time developing it.
Open source works great for shared code that isn't part of the "value added" by a company. So for a modern robotics company, it makes a lot more sense to use Linux for instance rather than rolling their own proprietary OS. And to use an open-source compiler for building the code. They're in the business of providing solutions using robotics, not selling operating systems and compilers, just like countless other companies build their products on top of these infrastructural tools, and sometimes contribute bug fixes and improvements back. But the code that actually makes the robot work (vision, motion planning, etc.) is what they spent most of their funding building, so giving it away makes no business sense.
Basically, you're complaining about all companies having trade secrets, and ultimately, you're complaining that competition exists instead of just having a single company having a monopoly over a whole market.
Just my 2 cents but I would try with a suction gripper on a swiffer-broom-like joint so you can pick vertically (TCP facing +Z) and place horizontally (TCP facing +X): https://img.uline.com/is/image/uline/H-1960
Not a pest at all and I've long been frustrated with ROS - our early demos were actually just a single C++ binary with multiple threads running for perception, control, and visualization, and I byte-packed robot control messages in our own software to avoid using ROS.
Unfortunately this breaks down in a few ways that you're probably familiar with, given that you asked this question:
- A crash in (e.g.) a third party sensor driver can bring down your whole binary, any signal catching here is awkward and you end up wanting process isolation
- Perception is, for better, or worse, easiest to prototype and try off the shelf in Python / Pytorch, so you either end up with pybind11 and driving things in Python, ONNX which is IME brittle for some of the crazier Pytorch modules, or message serialization and process isolation.
ROS/ROS2 does _way_ too much in my opinion - why does it have a build system and a ton of packages? This plus pinning OS versions are huge pain points. Unfortunately I also think many community-contributed ROS/2 packages are fairly low code-quality, with notable exceptions. Overall, I'd prefer to have ROS be a pubsub library with a few extra tools for logging and visualization.
In the end, we're currently using ROS2 for the reasons listed above and for easy prototyping, but I'd like to move to something more like FPrime, Basis, Cerulion, or Copper in the near future. I really want to grow something in-house with Zenoh or IceOryx2, but don't want to waste a lot of time on middleware, since I don't think it's what's kept the problem from being solved.
(At the end of this post I now see you're working on Basis, I apologize that I'm over-explaining to you. I'd love to chat about Basis if you have some time in the next few days!)
I agree. :) We don't specify an OS version, we allow arbitrary process boundaries, we use plain old CMake. Our perception story isn't great, yet. We have some tooling which could make certain GPU workflows easier, but have only prototyped with ONNX w/C++.
Would love to be able to provide your middleware, we've connected on LinkedIn, let's chat.
Good video! The overall question here is the blended rate of bags placed per minute, rather than how fast each action needs to be.
That said, the arm itself can move 180deg/s in every joint (roughly 5m/s max at the end effector) - these videos are still very much v1 and we're looking forward to leveraging more of the mechanical capability with a better gripper, better perception, and some new planning techniques we're rolling out in the next few months.
This is awesome! It's amazing how many jobs there are that require heavy manual lifting with repetitive motion that will be up-skilled in the coming years. New roles will be much more monitoring and problem solving which.
call me disappointed. Not to belittle whatever work was put into it as there were probably a lot, just the one-suction-cup way of doing it seems so 30 years ago. These days i'd have expected [several] hands with fingers.
Great point and we 100% agree! We have a new multi-modal gripper that we're testing now but are keeping hush while we bring it up. (We're actually swapping out a new arm too for a variety of reasons)
These videos are more to set a scene for where we're operating the the general process.
Have you considered using handles on the luggage? Seems like with some sort of hook you would have better hold on the luggage, but i guess it would be challenging to identify and approach each handle.
Thanks, Benjie! Great to see you here. I hope it's OK if I plug your excellent writings on robotics that I think everyone should check out: https://generalrobots.substack.com/
Previous baggage handler turned software dev here. Looks like you are targeting the bag room right now which is the best spot to start to prove the concept. It was surprising to me how manual the entire loading & sorting process was when I worked for the airlines.
I'm curious if there's any roadmap eventually to get this out to the ramp itself. Most of the back injuries seemed to happen in the bin itself because you have to often hunch/be on your knees in a bin tossing 50+ pound bags. I know airlines would probably be very hesitant to have any new equipment around their planes but just curious if there's any discussion around that.
Other side note, I also used to work in Cargo and always thought there could be way more efficient ways of loading loose packages that are on every flight and this seems to be a great possibility for those as well.
Awesome work & will keep tabs on it!
Thanks for the feedback!
John B was obviously aware from previous experience what a manual and injury prone process this was, but I've also been really surprised as I've dived deeper into airport operations myself.
Bagroom is definitely what we're targeting first - being indoors (usually) is a huge plus, and lets us focus on the manipulation part of the problem without going fully mobile yet.
That said, we're definitely targeting tarmac/ramp operations, particularly between a TUG/PowerStow and narrow-body bag carts. Inside the bin is much trickier but we agree it's the least ergonomic part of the job, you just can't move a massive industrial arm in and out of a plane very easily. We have it on our longer-term roadmap, though, and intend to leverage the baggage dynamics data we collect everywhere else to give us a head start on the packing and manipulation problems there, just with a different mechanism.
Cargo packing is a huge area of interest for us! Particularly around optimizing weight distribution in loaded planes, or just optimizing packing efficiency in general.
Why don't they just push the loaded rack into the airplane?
Weight of the rack is "wasted weight" on the flight, and that costs money.
Each plane is different, so the racks either waste space or have to have tons of different ones.
Many planes don't have easy big doors for racks (the ones you've probably seen are for designated cargo jets - like this: https://www.aircargo.ups.com/media/Containers-Pallets/upsair... )
And now even if you solve all the above, you still have to load the bags into the rack.
Why not rear-load like military aircraft and avoid that 90º turn? Then you could have a hull-shaped hopper to drop the bags in, so they are tetris'd together in the right geometry before you get to the ramp. Roll the hopper in, have a pez-despenser style pusher so as you withdraw the hopper, the bags are left behind in the aircraft?
Step one would be "get Boeing or Airbus to design, build, certify, and sell thousands of a significantly different airplane model".
That's a big hurdle for an airline to start climbing now to someday maybe start saving money on bag loading in many years.
surely someone thought of this before though. There must be a "why not" answer already.
It's expensive and difficult to put a large door on the front or back of a plane, resulting in inefficiencies or maintenance headaches.
The planes that DO have them are almost always military planes not used for other things, or very large jumbos (search 747 cargo or C-5 galaxy).
Almost all other planes are optimized for passengers (the cargo space in even a largish jet may be too short to stand up in, for example).
They do for larger planes; but commuter jets and most narrow bodies don’t have enough space in the belly for that.
Seems like someone needs to invent a rack that can easily change size based on the plane being loaded
ah, liquid metal, why didn't I think of that!
This is much like a palletizer. Here's a mixed-case cage palletizer, which fills up wire cages open on one side with various boxes.[1] For rectangular boxes, palletizing is a solved problem with many companies selling systems.
Irregular items are a problem. In the baggage handling video, the last bag, the soft one with straps, is sticking out after being placed in the baggage container. That's the same problem which keeps Amazon from totally automating picking. They keep trying, but nothing works well enough yet.
[1] https://www.youtube.com/watch?v=TN-6QaLd3VY
Yep! We're obviously operating alongside a very well established world of palletization and other order fulfillment type robots.
We think that due to irregularity that it's not an easy tech transfer from the existing logistics world into the aviation world. We're very interesting in looking the other direction, though!
Many airports have solved the irregular items problem by simply stipulating that items must be cubical/cuboidal/close enough in shape.
Miss the days when my parents took a kitchen sink from Dubai to India, or when my uncle took coconut saplings and a literal banana tree to the US lol.
This is very cool, thank you for sharing. I work in automation and SWE for a certain 4-letter organization that delivers your mail. Pick/place is something we've rolled out using articulated and delta robots with vacuum end effectors, and it's an interesting and challenging space to be in. As in your case, bags and other amorphous shapes are always the most difficult. It's always an uphill battle to hit throughput targets due to exception cases that can stop things until a human gets involved. Ultimately, it can be a struggle to avoid overpromising and to generate ROI since automation is so costly, especially when there's no opportunity to bound the problem by influencing the inputs to your system or the output requirements (in your case, the cart being loaded). Best of luck and looking forward to seeing your new end effector.
I'm surprised the end effector works on misshapen fabric bags. Makes me jealous that I can't test it.
Are you considering dynamic trajectory constraints in the planner (e.g. for multiple robots loading simultaneously)? That was a thorny problem back when I worked on arms.
There are a lot of constraints in our planning, including what actions we can do for a particular bag/gripper interaction. Multiple robots working in collision range of each other isn't on the roadmap right now, but it's always a possibility.
Suction works really well but it's not enough, we're rolling out a new gripper soon that covers more cases mechanically (there's a very long tail in the distribution of what comes through airports).
Was on a plane a few days ago watching someone do this out the window 10 yards away.
- Seemed like the baggage handler was required to do a very fast cycle of <scan, toss, align, repeat next bag>. Automation seems helpful, and certainly it’s hard labor.
- This was also a young woman, in presumably a safe union job, working in a very pricey city (one of the mountain west towns that exploded). Adios union job hello robots.
Tricky ethics! Outside of picking stuff up and putting stuff down, not too many automation union-safe jobs left. Saving them from back pain is also going to be saving them from a job.
We're really focused on health and safety aspects of this job - in a repetitive stress sense, these jobs are much more dangerous than many people imagine they would be and people end up with lifelong injuries.
Generally, regulators seem to be moving in this direction as well. The EU has introduced new regulations on the total amount of weight someone can move in a shift, and the Dutch government has mandated that baggage handling move away from manual processes like this in the near future.
Despite the focus difference, do you think it's unlikely that automating baggage handlers will replace their jobs?
The regulator focus seems like it'd reduce the max allowed weight of a checked bag, not automate the baggage handler handing the checked bag. So, I don't see the similarity between the regulatory push and your product? Edit - to clarify, beyond what Dutch regulators say about Dutch markets, which are a very small subset of "regulator focus" internationally.
This is excellent! I live in a small city with an airport so have known many baggage handlers during my years here. I honestly didn’t believe how common back and shoulder injuries were in the industry. This could prevent a whole lot of medical problems.
Super cool! You are additionally going to be saving a lot of workers from getting chronic back pains. I thought maybe y'all are going too slow, but after looking at some baggage handling videos, it seems like you're at a comparable speed already?
In deployment these things are probably going to be on some kind of cart system, I presume all your algorithms can handle small changes in the XY travel plane (i.e. the robots location w.r.t the end of the belt).
Thanks! It's a really destructive job for workers' lower backs and elbow tendons. This actually puts into perspective the blended throughput rate - you can imagine loading a few bags really quickly, but moving 20kg bags for 10 minutes straight will slow you down. That said - we still have a lot of runway on speed for these mechanisms and are still running fairly conservatively as we shake out our software.
There are a few ways we plan to deploy (some fixed rails, some mobile). Since the carts we're loading aren't placed with much precision, even the fixed deployments need to do serious environmental perception / localization.
> You are additionally going to be saving a lot of workers from getting chronic back pains.
That'll probably be a big comfort to those who lose their jobs
It's not as though this is a profession which people spend a decade learning to perform. And it's not 2009 any more, there is demand for labor throughout the economy. I think the benefit of getting people out of physically damaging work outweighs the pain of having to find a new job in this case
I don't follow how the existence of these robots is a benefit to the worker. A robot took their job, that does not mean another job was created out of thin air.
I'm not saying this isn't back breaking work, I'm just saying that your logic doesn't make sense.
Many people will be put out of a low-skilled manual job, but many better-paid jobs are created for robot salespeople, robot installers, training specialists, delivery drivers, maintenance technicians etc. Those who adapt to these new jobs will thrive.
That's not how this works. Most airports are looking to expand, they can keep the same staffing levels with a rapid expansion in throughput with these machines. (Assuming the unions allow it).
Another HN roboticist chiming in; the videos look good! I have many questions, but will keep it to just a couple for brevity's sake.
- How much are you able to use the robot's internals to estimate the gripped bag's inertial properties? If you're trying to put rigid, heavy bags below light, amorphous bags, are you adjusting final placement location on-the-fly?
- How dynamic is your scene beyond what we can observe? If you're using light curtains with a single robot on a track, and you're able to estimate some rough geometry of and track the bags down the conveyance, and you are updating occupancy of the bin as you're going, what else is there?
- Is this just inside for the foreseeable future or are you all going to tackle unpacking outside, as well as all the, ahem, baggage that comes along with operating outside the terminal walls.
Nice and straightforward problem, relatively speaking.
> Unfortunately for airlines, passengers don’t package their luggage in nicely uniform cardboard boxes. If they did, then the airlines could benefit directly from the recent takeoff in manipulator tech for warehouses. But airline luggage is way more wacky and irregular. If robots are going to handle it, they need to reason about how to grasp each item, handle its deformability, stack it in a stable way, and do all of this quickly, safely, and reliably.
Curious if you guys have put any thought into seeing if there's an operational change you could introduce to airlines, that would result in the tech side being a lot easier?
Palletizing logistics for consumer airtravel would be interesting...
Operational changes for airlines are quite tricky - one of our bets is that most of the value for customers here is in handling "brownfield" deployments where you drop into an existing process, and that intelligence (or at least, good perception and reactive planning) really unlocks this ability from the robotics side.
For widebody planes, bags are already loaded into Unit Load Devices (ULDs), which are large semi-truncated boxes that get loaded directly onto aircraft. Narrow body planes don't use these (apparently) because they impact turn-time and decrease the amount of time a plane can be in the air, and also impacts how quickly bags come out, since it adds an extra step to unloading.
Many airport conveyance systems also load each bag into a bin, but those bins aren't loaded into the airplane because they belong to the airport and waste space/weight.
The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
> The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
I could see a budget airline cooperating with a luggage/case manufacturer and offering "if your checked bag is EXACTLY a Pelican 1615TRVL, it flies free/cheap" - and then work with them to design a case that is easily automatable.
Very cool stuff thanks for the commentary and good luck!
The suction "cup" seems to be doing a good job, but I don't think bags are made to be handled that way. Did you alternate test with some kind of a grabber (like the claw machine, but actually effective)? It would make the grasp selection problem much more tractable imo, since all bags have at least one "grasp point" built in.
In teleoperated mode, I'm guessing you're using the captured data to train the autonomous mode?
Final note, the robot looks beefy enough to lift an entire airplane, forget luggage -- is it overengineered on purpose?
The cup has taken us very far, which we're excited about, but it's definitely not enough - we're currently testing a multimodal claw-ish + suction gripper, which we've had good results with so far but aren't ready to unveil.
The teleop data is really useful for training data indeed, and lets us collect data on current failure points (e.g. with suction, just how far can we tilt this fabric bag before it peels away, etc). We're not going full behavior-cloned end-to-end for a lot of reasons (sample complexity, safety, adaptability, etc), but we do a lot of learning in specific parts of the system (particularly around grasping and placement).
The robot is indeed beefy, as many robots rated for 50kg applications are (check them out online). We've accidentally stress tested this unit way beyond 50kg without a hiccup, so we're very interested in figuring out what the right-size unit is for our application. There are a few other great aspects to this unit - it's a 7-DOF arm + 1 more DOF for the linear rail, so we have two extra degrees of freedom to play with for collision avoidance during planning.
Suction cups are way better than multi-finger grippers.
Multi-finger grippers rely on "affordances" i.e. space between the boxes to get the fingers in. If the boxes are pushed up against one another, you've got to do something to make space for the fingers first. So to grab a box with a multi-finger gripper you need space at the top, left and right.
Suction cups, on the other hand? They only need access to the top. No need for space at the left and right, no need to slide things to make space, just apply the suction cup and pull upwards.
Suction cups can also be made of soft, flexible rubber so if the planning is a bit inaccurate? No problem, the suction cup just bends. A multi-finger gripper, though? If the fingers are rigid and strong enough to lift a 30kg bag then they're also probably rigid and strong enough to punch a hole in an adjacent bag.
Suction cups do have a bunch of disadvantages, of course. Porous materials you can't get a seal on. Vacuuming up detritus and messing up your vacuum generator. The payload swinging about on the flexible suction cup. Losing vacuum and dropping the load if there's a power cut.
But they're certainly a great starting point in an application like this one.
@btbuildem isn't arguing any of that; the concern is that if you grab a bag the wrong way it may break open.
As part of one of my classes in engineering school we went and toured the DIA baggage handling system, studied why and how it was such a big failure and what they did to fix it.
There have been some solid attempts in this space before - many projects take on the whole baggage system design and end up very very complex and often over budget. We're focusing on introducing tech that plays well in a larger system, particularly in "brownfield" existing processes - our bet is that recent advances in robot autonomy give us ability to handle items that weren't possible before, and therefore our units can be introduced in a more flexible way.
Very cool, makes sense. Totally, the real challenge in this space is all the weird corner cases.
Using a vacuum to pick up baggage is a very interesting choice. I wonder how it would fare with extremely uneven surfaces or even porous ones.
Also --- I couldn't see any obvious sensors. What sensors are you using to perceive the bag? I am imagining some kind of RGB-D sensor like a Kinect (or its successors like the Orbbec).
Suction has gotten us pretty far at the prototype level but definitely isn't enough - we're testing out some new gripper designs that use suction as a broader part of an overall grasping system.
For these videos we have lidars and two Intel Realsense depth cameras mounted to the safety cage and on a wall. We're working on moving as many sensor on-robot as possible in the near future to aid with deployability.
Are most suitcases designed to bear its fully loaded weight just by sucking on one surface?
Solid question and something we think about a lot! Worst case is a weak zipper or similar. We're bringing a new gripper online which is more multimodal - some mechanical grasping, some suction, and the ability to choose what you use. We're moving away from pure suction partly for this reason and partly for textiles.
Suction is great though, and ~75% of bags checked through the US are hardshells, so it's something we're not ready to ignore entirely.
How about a forklift like handling ?
Also good question and something we've thought about. The difficulty there is actually getting the forklift tines out after placement. Actual forklifts in real warehouses rely on pallets as an affordance for manipulation, and we don't have that luxury here.
There have been some neat attempts with short conveyor belts as end effector tools [1]! Generally these systems rely on being able to rearchitect a significant amount of the process (building a controllable conveyor belt or rearchitecting part of the bag-room), and we're focused on dropping into existing processes.
[1] https://www.youtube.com/watch?v=n2Wy_tduq5k
Another robot to clean up after the zipper fails is wip
How would a single armed robot handle oversized luggage or luggage with weird centre of mass? Or poorly packed luggage who's centre of mass is shifting?
Say golf club hard shells case where the owner's left a bunch of golf balls in the bottom of one of the pockets. It's difficult to tell that one end's quite a lot heavier then the other until you actually try to pick it up.
At least to me I don't see a good way of handling this edge case. You'd need either a second manipulator, a very fast feedback system, or a way for the weird luggage to be diverted for human intervention.
Great questions and a few answers to various parts of your question that you've mostly identified yourself I think:
- Golf clubs specifically (and most sporting equipment) actually goes down a different pipeline in most airports, since generally this stuff doesn't behave well on conveyor belts.
- Data driven approaches can tell you a lot just from visual information, usually about deformability of objects, but also about expected centers of mass, etc
- Part of the reason we're using single big robots is because you can use heavy duty end-effectors - grasp all the way around the object with a grasp that's predicted to be robust to these kinds of perturbances, and then use quick feedback to safely execute a plan to place it
You're absolutely correct though, that there's a long tail of things coming through and that some objects are very, very difficult. Our problem formulation then becomes identifying confidence in graspability, and deciding explicitly that we shouldn't attempt to grasp some object and should instead flag them for human handling.
I've heard about you guys before. Nice application! What are you using for teleop/remote-monitoring? Did you build that yourself?
Teleop and monitoring are systems that we've built ourselves and are pretty happy with. Since we use MuJoCo for simulation/visualization and some kinematics subroutines, to visualize, I just keep the MuJoCo GL context open after rendering and then throw all of our sensor data into it - it's very performant and low latency!
We've since introduced a message-bus layer that makes it possible to do it all over the internet etc, but adds the associated serialization and transport latency.
> to do it all over the internet, but adds the associated serialization and transport latency.
I wrote this blog post on that topic a while back after having seen various approaches robotics companies take and their shortcomings: https://transitiverobotics.com/blog/streaming-video-from-rob...
Excellent post! Curious if WebRTC can be adapted for 3d sensor data and would love to chat more about it - I'll send an email!
There should be a fully fledged, robust and comprehensive open source robotics OS.
I imagine most of this code being reinvented on a daily basis at countless companies around the world, what a waste of human resources.
For a robotics company, their code is the "secret sauce" that makes their company valuable. It wouldn't make sense to open-source it all and let their competitors do the same thing without having had to spend so much money and time developing it.
Open source works great for shared code that isn't part of the "value added" by a company. So for a modern robotics company, it makes a lot more sense to use Linux for instance rather than rolling their own proprietary OS. And to use an open-source compiler for building the code. They're in the business of providing solutions using robotics, not selling operating systems and compilers, just like countless other companies build their products on top of these infrastructural tools, and sometimes contribute bug fixes and improvements back. But the code that actually makes the robot work (vision, motion planning, etc.) is what they spent most of their funding building, so giving it away makes no business sense.
Basically, you're complaining about all companies having trade secrets, and ultimately, you're complaining that competition exists instead of just having a single company having a monopoly over a whole market.
[There is, and by some estimates 1.3M people use it.](https://docs.ros.org/en/rolling/)
Just my 2 cents but I would try with a suction gripper on a swiffer-broom-like joint so you can pick vertically (TCP facing +Z) and place horizontally (TCP facing +X): https://img.uline.com/is/image/uline/H-1960
This looks cool, interested to see where it goes in the future.
Going to be a pest for a minute - have you considered a stack other than ROS2?
Not a pest at all and I've long been frustrated with ROS - our early demos were actually just a single C++ binary with multiple threads running for perception, control, and visualization, and I byte-packed robot control messages in our own software to avoid using ROS.
Unfortunately this breaks down in a few ways that you're probably familiar with, given that you asked this question:
- A crash in (e.g.) a third party sensor driver can bring down your whole binary, any signal catching here is awkward and you end up wanting process isolation
- Perception is, for better, or worse, easiest to prototype and try off the shelf in Python / Pytorch, so you either end up with pybind11 and driving things in Python, ONNX which is IME brittle for some of the crazier Pytorch modules, or message serialization and process isolation.
ROS/ROS2 does _way_ too much in my opinion - why does it have a build system and a ton of packages? This plus pinning OS versions are huge pain points. Unfortunately I also think many community-contributed ROS/2 packages are fairly low code-quality, with notable exceptions. Overall, I'd prefer to have ROS be a pubsub library with a few extra tools for logging and visualization.
In the end, we're currently using ROS2 for the reasons listed above and for easy prototyping, but I'd like to move to something more like FPrime, Basis, Cerulion, or Copper in the near future. I really want to grow something in-house with Zenoh or IceOryx2, but don't want to waste a lot of time on middleware, since I don't think it's what's kept the problem from being solved.
(At the end of this post I now see you're working on Basis, I apologize that I'm over-explaining to you. I'd love to chat about Basis if you have some time in the next few days!)
I agree. :) We don't specify an OS version, we allow arbitrary process boundaries, we use plain old CMake. Our perception story isn't great, yet. We have some tooling which could make certain GPU workflows easier, but have only prototyped with ONNX w/C++.
Would love to be able to provide your middleware, we've connected on LinkedIn, let's chat.
How much faster will the robotic arms be able to go? Currently this looks far too slow, though as a v1 it's great.
It's about as fast as human does it now.
Not even close
https://youtube.com/shorts/KMlCCyQPfME?si=BsyZ9QZgZDJWVh4f
Good video! The overall question here is the blended rate of bags placed per minute, rather than how fast each action needs to be.
That said, the arm itself can move 180deg/s in every joint (roughly 5m/s max at the end effector) - these videos are still very much v1 and we're looking forward to leveraging more of the mechanical capability with a better gripper, better perception, and some new planning techniques we're rolling out in the next few months.
I guess, just my opinion (probably worthless), if you can get it to speed up even ~20%, I think that would be enough to make the average person agree.
This becomes a crowded field. Here is similar robotic arm from Boston Dynamics
https://www.youtube.com/watch?v=S9N2jlie9c8
This is awesome! It's amazing how many jobs there are that require heavy manual lifting with repetitive motion that will be up-skilled in the coming years. New roles will be much more monitoring and problem solving which.
call me disappointed. Not to belittle whatever work was put into it as there were probably a lot, just the one-suction-cup way of doing it seems so 30 years ago. These days i'd have expected [several] hands with fingers.
Great point and we 100% agree! We have a new multi-modal gripper that we're testing now but are keeping hush while we bring it up. (We're actually swapping out a new arm too for a variety of reasons)
These videos are more to set a scene for where we're operating the the general process.
Have you considered using handles on the luggage? Seems like with some sort of hook you would have better hold on the luggage, but i guess it would be challenging to identify and approach each handle.
Why do they load from a ramp to a cart at the same height? Why not ramp UP over a dump-style cart and just have the luggage fall into it?
Haha I want to work for you guys! This sounds like so much fun, hope you guys become trillionaires!
Congrats David! I love your approach and your problem space. I'm rooting for you.
Thanks, Benjie! Great to see you here. I hope it's OK if I plug your excellent writings on robotics that I think everyone should check out: https://generalrobots.substack.com/
Congrats David & John B! Excited to watch you guys grow.
Interesting. Is it a vacuum to attach to the bags?
Well, this didn't go very well in Denver when they tried it. Good luck.
Please tell me the robot's name is not Iggy.
No - people keep asking us to name our robots, but we haven't landed on anything yet!
Bopomofo @ https://en.wikipedia.org/wiki/Bopomofo Bagel Banjo Baxter Bastian Barry Barnaby Babu Barney Basil Brangelina Balzac Baudilaire Batiste Baldric Bacchus Barrington Ballard Barney Bartosh Bartek Bart