This is something I was trying to implement myself. I am surprised it can be done with just an s3 bucket. I was messing with API Gateways, Lambda functions and DynamoDB tables to support the s3 bucket. It didn't occur to me to implement it client side.
I might have stuck a bit too much to the lfs test server implementation. https://github.com/git-lfs/lfs-test-server
Actually, moto is just one bandaid for that problem - there are SO MANY s3 storage implementations, including the pre-license-switch Apache 2 version of minio (one need not use a bleeding edge for something as relatively stable as the S3 Api)
Well, I look forward to their `docker run awslabs/the-real-s3:latest` implementation then. Until such time, monkeypatching api calls to always give the exact answer the consumer is looking for is damn cheating
Agreed, haha. Well, I think it should work with Minio & co. just as well, but be prepared to have your issues closed as unsupported. (Pesonally, I might give it a go with Backblaze B2 just to play around, yeah)
Happy 10,000th Day to you :-D Yes, moto and its friend localstack are just fantastic for being able to play with AWS without spending money, or to reproduce kabooms that only happen once a month with the real API
I believe moto has an "embedded" version such that one need not even have in listen on a network port, but I find it much, much less mental gymnastics to just supersede the "endpoint" address in the actual AWS SDKs to point to 127.0.0.1:4566 and off to the races. The AWS SDKs are even so friendly as to not mandate TLS or have allowlists of endpoint addresses, unlike their misguided Azure colleagues
S3 recently got conditional writes and you can use do locking entirely in S3 - I don't think they are using this though. Must be too recent an addition.
I thank goodness I have access to a non-stupid Terraform state provider[1] so I've never tried that S3+dynamodb setup but, if I understand the situation correctly, introducing Yet Another AWS Service ™ into this mix would mandate that callers also be given a `dynamo:WriteSomething` IAM perm, which is actually different from S3 in that in S3 one can -- at their discretion -- set the policies on the bucket such that it would work without any explicit caller IAM
This is something I was trying to implement myself. I am surprised it can be done with just an s3 bucket. I was messing with API Gateways, Lambda functions and DynamoDB tables to support the s3 bucket. It didn't occur to me to implement it client side. I might have stuck a bit too much to the lfs test server implementation. https://github.com/git-lfs/lfs-test-server
All this mocking when moto exists is just :-( https://github.com/awslabs/git-remote-s3/blob/v0.1.19/test/r...
Actually, moto is just one bandaid for that problem - there are SO MANY s3 storage implementations, including the pre-license-switch Apache 2 version of minio (one need not use a bleeding edge for something as relatively stable as the S3 Api)
> there are SO MANY s3 storage implementations
I suppose given this is under the AWS Labs org, they don’t really care about non-AWS S3 implementations.
Well, I look forward to their `docker run awslabs/the-real-s3:latest` implementation then. Until such time, monkeypatching api calls to always give the exact answer the consumer is looking for is damn cheating
Agreed, haha. Well, I think it should work with Minio & co. just as well, but be prepared to have your issues closed as unsupported. (Pesonally, I might give it a go with Backblaze B2 just to play around, yeah)
it wouldn't be unprecedented. dynamodb-local exists.
Do you mean boto (the python SDK for AWS)?
EDIT: They probably do not, I'm guessing they mean https://docs.getmoto.org/en/latest/index.html ?
moto server for testing S3 is pretty great. It’s about the same experience as using a minio container to run integration tests against.
I use this, and testing.postgresql for unit testing my api servers with barely any mocks used at all.
There is also testcontainers. Supports multiple languages. Uses containers though.
https://testcontainers-python.readthedocs.io/en/latest/
Happy 10,000th Day to you :-D Yes, moto and its friend localstack are just fantastic for being able to play with AWS without spending money, or to reproduce kabooms that only happen once a month with the real API
I believe moto has an "embedded" version such that one need not even have in listen on a network port, but I find it much, much less mental gymnastics to just supersede the "endpoint" address in the actual AWS SDKs to point to 127.0.0.1:4566 and off to the races. The AWS SDKs are even so friendly as to not mandate TLS or have allowlists of endpoint addresses, unlike their misguided Azure colleagues
> Happy 10,000th Day to you :-D
Sorry, not sure what you mean?
https://xkcd.com/1053/
I'm surprised they just punt on concurrent updates [0] instead of locking with something like dynamodb, like terraform does.
[0] https://github.com/awslabs/git-remote-s3?tab=readme-ov-file#...
S3 recently got conditional writes and you can use do locking entirely in S3 - I don't think they are using this though. Must be too recent an addition.
Google Cloud Storage is good enough to implement locks all by itself: https://reddit.com/r/golang/comments/t52d4f/gmutex_a_global_...
Doesn't S3 provide primitives to do the same? At least since moving to strong read-after-write consistency?
I thank goodness I have access to a non-stupid Terraform state provider[1] so I've never tried that S3+dynamodb setup but, if I understand the situation correctly, introducing Yet Another AWS Service ™ into this mix would mandate that callers also be given a `dynamo:WriteSomething` IAM perm, which is actually different from S3 in that in S3 one can -- at their discretion -- set the policies on the bucket such that it would work without any explicit caller IAM
1: https://docs.gitlab.com/ee/user/infrastructure/iac/terraform...
Amazon has deprecated Amazon Code Commit, so this may be an interesting alternative.