SFTP to S3 with S3FS and EC2

I’ve been looking for an SFTP to S3 solution for years now. In an infinite number of universes, I’ve had an infinite number of experiences where team members and/or outsource teams need to upload files to clients’ S3 buckets. If you’re just seeing the AWS console for the first time, it’s overwhelming. Even limiting access (in my experience this is lukewarm at best: functional but not visually reflected; users can see what’s set up even if they can’t access them).

Perhaps you’ve seen AWS now offers an SFTP to S3 endpoint service. What you might not know (yet): you’ll spend $0.30 an hour plus $0.04/GB (up and down). While there are plenty of customers that simply don’t care about the cost, there are enough that do, so I went off in search of what I could do with a simple EC2 instance instead.

First, I used a couple of different tutorials: one from WinSCP, and a more in-depth one from Cloud Academy. For my time and effort, the Cloud Academy one was a little more detailed and, IMO, better for a first-time run. From a security standpoint, I prefer Cloud Academy’s use of IAM roles instead of local aws-cli config files.

The core to making this transition work is utilizing the S3FS-FUSE filesystem, which lets you mount an S3 bucket as a local drive. I won’t go into details here, feel free to read up on it if you so desire.

Improving Security

I suggest making a handful of changes to the tutorials above to further secure your FTP server:

  1. don’t install a local FTP server; use FTP over SSH (SFTP); avoid unnecessary services and running processes whenever possible
  2. if you want to be even more secure, use EC2 Security Groups and firewall the server to only necessary users (you could also change the port SSH runs on your server if you feel it necessary)
  3. use key exchanges over SSH instead of passwords (and disable SSH passwords altogether)
  4. when setting up the IAM Policy, limit the access only to your local AWS VPC

DoCuMenTaTiOn DoEsN’T MatTer

Ho. Lee. Fuck.

Two fucking days. Two mother-fucking shit-eating stupid-waste-of-time days…

Technically, the last couple of hours of one day and all of another.

Lost. Gone to the great timekeeper in the sky.

There’s something about “assume” and “making an ass out of ‘u’ and me…”

Where does this come from? Thanks for asking! Thomas Bradshaw at Atlassian provided the most helpful response I’ve seen in two days’ research (I’m not the OP). A huge thank-you to Thomas for that information. Usually, I find myself learning as I go, often with no internal support for anything new that needs to happen. I’m a lone wolf by circumstance, not by choice per se.

I started writing this as a rant against substandard documentation. The developers or technical writers could have (and probably should have) written clearer documentation. I might not have lost two days otherwise. However, reading through some of the code examples again for this post, I realized I glossed over content. Nothing like a mouthful of crow. Mmmm.

Now I don’t get to complain about the documentation, developers, or technical writers…

Instead, I have to express my frustration at myself. I read through the great walkthrough GitConnected provides. Somehow, I repeatedly miss the reference to executing the zip command from the bitbucket-pipelines.yml file.

As important as writing good documentation is, reading comprehension matters. Take the time, when learning something new, to slow down a bit. Often, you can find the extra information you need in code samples. Take the time to read. You might surprise yourself.

That aside…

Developers and technical writers, do us all (including yourselves) a favor…

Write better documentation.

I know, I know…I don’t like to, either. But together we can make the world a better place…or whatever tripe the tech bros spew these days.

Seriously. Take that extra little bit of time.

Now get out there, and make me fucking proud.

Surprise! Private IP’s Can Move…

The AWS adventures continue…

I’ll keep this short and sweet, because it was almost life-changing and I’m tired of ranting. Something something be more positive something attitude affects everything something something eat a dick.

I noticed, previously, when adding additional private IP’s to an AWS instance, the input field has a placeholder value of Auto-assign. I never gave it much thought until my EC2 instance stopped connecting to the Internet yesterday. Maybe I can write another post on that later. Hours pass. Attempts to fix an issue affecting updates and certbot fail miserably. I rebuild the damn thing this morning; hence the lesson…

I debate and decide to test the merits of a separate standalone Network Interface. I’m certain, out there somewhere, in the deathlike silence and chill of a datacenter, a Sysadmin or DevOps Engineer knows the answer. I’m not that wizard, and the Interface idea fails spectacularly when binding those private IP’s to the EC2 instance. You might think I’d learn to test one or two IP’s instead of rewriting the whole damn netplan file, but apparently you don’t know me. So, the separate Network Interface idea dies in solitary.

At this point, I’m resigned to rewriting the netplan YAML again (that would be the second time in one day). “What the fuck, I might as well try to move these private IP’s and save myself an hour of work after spending 20 hours trying to fix it.”

Lo and behold, it worked. The heavens parted and the networking gods smiled on me. The fact you’re reading this is proof they haven’t shit on me since (at least in this case).

Slaying the Dragon

or “How I Figured the Fucking IP’s Out and Learned to Love netplan

I suppose it’s that time of year again, where I write one of my two annual posts. Some years I only get through one. I actually enjoy writing, I just never seem to find time to do it. Oh, well, as the title implies, and the subject line explicitly states, I finally figured out that pain-in-the-ass (PITA) netplan. If you’re curious where this started, take a look (just don’t forget to come back)!

The problem never changed, I ended up trying to worm myself out of facing this enemy again with a number of t3.nano instances instead of one m5.medium instance. Unfortunately (or so I thoughtlittle did I know…), some of the WordPress and other database-heavy code stubbornly refused to run on a t3.nano. By the time I had eight t3.nanos, four t3.micros, a t2.micro, and a partridge in a pear tree, I opted to switch to a bigger server and put everything on it.

That meant one of two options: learn netplan (shudder) or use Ubuntu 16.04 and go through the joy of Canonical’s lack of support for an old OS (read: having to pull in additional repos to build PHP 7.x). I’ll admit…I started with 16.04. It didn’t take long to piss me off, though, and I “embraced the suck” (as High Overlord Pelosi would like all citizens…er, except Congress…to do).

Much to my surprise, it came way easier this time… (lucky “it”…S/O Rob, Anybody, and Dawn)…

The first thing I had to do (which I could swear I tried last time) was create a separate YAML file for storing the network data…and call it first (the base file is called 50-cloud-init.yaml, I named mine 01-multi-ips.yaml). Hold up a sec there, cowboy…the first thing I did was build an AMI out of my server so I could spin everything back up if I fucked everything to hell again.

So, technically the second was that new file thing.

I kept the names the default file did.

The Original File

# /etc/netplan/50-cloud-init.yaml

# a bunch of stuff Ubuntu adds to remind you this file
# is auto-generated and might wipe your changes if you
# save them here
network:
version: 2
ethernets:
ens9:
dhcp4: true
match:
macaddress: HA:HA:IM:NO:TR:EA:L0
set-name: ens9

Bright and Shiny (the new config file)

I’m only using a couple IP’s for an example, but I ended up binding whatever the maximum is. But before you take a look, keep a few things in mind:

  1. 172.172.172.1 is the primary private IP on this imaginary instance
  2. 172.170.180.3 is the first additional private IP on this instance
  3. ens9 is the name of the network device from the previous file; you might see a different name, and if you do, use the name you see in your cloud config file
  4. the cloud config file might have a different name, but you should only have one or two files max in that directory
  5. back your shit up before you fuck about with IP’s and network interfaces
  6. I’m fairly sure this is spaced correctly, but it’s a good idea to get a YAML Linter or otherwise validate your YAML before running the command to kick off the IP update or restarting your server
# /etc/netplan/01-multi-ips.yaml

# DISCLAIMER: Fuck shit up at your
# own risk. Back your shit up. Wash
# your hands. Don't be a dick.

network:
version: 2
renderer: networkd
ethernets:
# surprise, motherfucker! this is the same as the one above
ens9:
# you need to list every private IP address
# that you want netplan to recognize here
addresses:
- 172.172.172.1/20
- 172.170.180.3/20
# and so on...
dhcp4: no
# now for the real fun...map the routes
routes:
- to: 0.0.0.0/0
via: 172.172.172.1
table: 1000
- to: 172.172.172.1
via: 0.0.0.0
table: 1000
- to: 172.170.180.3
via: 0.0.0.0
table: 1000
routing-policy:
- from: 172.172.172.1
- from: 172.170.180.3

I really wish I could explain what everything here is, but I can’t, but the good news is…Google exists…and it’s your friend for finding answers on StackOverflow.

Just one two more steps…

Run sudo netplan apply –debug

I’m a fan of running –debug because it will warn you if something is awry before you fuck your connection to hell. You’ll still have a chance to delete or comment out or rename your monster before things go south.

Finally, bind your Elastic IP’s to your private IP’s and you’ll see things work. ifconfig will not look right (neither will sudo netplan ip leases ens9)

AWS Private IP’s on Ubuntu 18.04 with Netplan

Ugh. Uggggggggghhhhhhhhhhhhh. Ffffffffffffffffffffffffffffffffffff. Twelve hours to get this damn thing right. OK, deep breath. Use Google, ask sys admin friends, LINT your YAML. Spin up alternate instances because you mis-configured Netplan and it took the server offline. Realize your snapshot was designed too late and you’re totally fucked because of course the dev team doesn’t have backups. Stop what you’re doing because a domain expired on an app that has traction and fuck how did I miss that? Been a ball-busting week. Live and learn. Wear a Thai steel cup next time, discomfort aside, the protection is second to none.

Sigh heavily. Have a Lagavulin (or 3). Smoke half a box of Davidoff cigarillos…get wired for sound…that’s a lot of nicotine for someone who has one cigar a week, two at most. Don’t think about how much was spent between the scotch and cigarillos, or how quickly you consumed either one of them. Deep breath (again). Yoga…just kidding, another scotch and two more cigarillos. Three more and this pack is done…can’t just run down to the nearest convenience store…fuck, the closest tobacconist that will have something worth smoking is in Sacramento and that’s an hour round trip if traffic is light…fuck, it’s Saturday, traffic won’t be light…better manage these last two cigarillos. Hey, here’s an idea, write a blog post, haven’t done that in a while…hmm…but what to write about? Dammit, this server needs to be ready yesterday. System administrators understand the pain but customers don’t, especially when the design team dragged ass and turned in half-assed work. Fuck, cost a design client, but can’t leave them hanging, that’s awful customer service. There’s too little care about customers today as-is, oh no, the customer hurt my feelings or wrote some awful shit about me. Fuck, time to scrub Yelp…I mean, ask a customer politely to change their review. Fuck it, they’ve never been good with password management, won’t remember what they wrote, and don’t have time to go back and look at their past reviews. Sure, some people might, but that customer won’t. Where were we? That’s right, shitty customer service. Won’t provide bad customer service, that’s bad business. ipconfig is deprecated in favor of ip, and that YAML config is a bitch dealing with multiple IP’s…it isn’t supposed to be, but for whatever reason it doesn’t want to take. vi keeps enforcing its own formatting on the YAML, which is pretty awful when you consider the importance of formatting in YAML…like Python. Netplan has extensive documentation, and even some great examples of config files.

Thought about sharing the first iteration of the config file (set routes at your own risk…broke the server twice trying that shit), but that seems pointless. No one wants to see what doesn’t work. You wouldn’t have bothered reading this far if you did. IP’s have been changed to protect the innocent…

They say there’s no atheist in a foxhole, but there’s no atheist when the network connection keeps failing and there’s no backup, either. AWS shows terminal screenshots, if only they had some way to virtually connect locally rather than via SSH… Oh, look, the public IP changed, maybe it will work as soon as I update PuTTY…fuck. Let’s add another private IP and an Elastic IP to that, and maybe that will work…fuck. No connection, the server is fucked. I’m fucked.

Conversation with a friend with AWS Certification. Suggests shutting down the instance, spinning up a new one, and mounting the volume to the new instance. Done this with VM’s dozens of times. Guess sometimes we’re just too close to the problem to see a solution. Fuck, I should probably eat while this volume is detaching from the original and attaching to the new one. Alcohol has calories, that counts, right? Half a bottle. Fuck me. So much for going and grabbing a bite somewhere, not that I really have the time, but shit it would be nice to get out of the house today. Nope, back to the grind…12 hours in and counting. Damn, JoyRun was great, too bad as soon as you move out of a college town it doesn’t work anymore. Maybe Postmates delivers? Ha! Not here. The only pizza place still open is Domino’s…”pizza”

I need to know more local people, convince someone to do a food run for me while I’m grinding through this damn server. OK. Deep breath, the first sign of life. The volume mounted and I was able to edit the YAML file, get rid of the offending code (it was an attempt to separate the addresses into ifconfig-style ethX:format):

# don't do this
# for real
# it's gonna take down your fuckin' server
# some other YAML config
    eth1: # this is fine
        addresses: # still fine
            - 10.0.0.5 # yup, still fine
            - 10.0.0.6 # still
    eth1s0: # woah, nellie
       addresses: # doesn't matter, the previous line already broke shit

This works if the system has an interface eth1s0; if not, the IP configuration gets fucked. Yes, fucked proper. No clue why people keep asking.

Time to knuckle down and learn the ip tool…

ubuntu:~$ sudo ip -c a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:07:aa:ff:f9:36 brd ff:ff:ff:ff:ff:ff
    inet 10.0.16.1/20 brd 10.0.31.255 scope global dynamic ens5
       valid_lft 2195sec preferred_lft 2195sec
    inet6 fe80::7:aaff:feff:f936/64 scope link
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:44:ac:46:58:e0 brd ff:ff:ff:ff:ff:ff

Do you see what I see? Further, why did it take me twelve fucking hours to see this???  Chalk it up to the aging brain not learning new things nearly as quickly as it used to. Look… ens5 shows “UP,LOWER_UP” while ens6 does not. The good news is, Ubuntu recognizes the new interface and wants to work with it. Fresh Ubuntu 18.04 install (on what I think is the seventh instance in the last 90 minutes); I had not touched the network configuration on this system…yet. Fortunately, that disconnected volume saved my bacon…keep backups, kids. Take snapshots before you muck with network configurations. I know better and didn’t until it was too late.

No one ever tells you how much time sysadmins spend waiting on shit. Roughly a third of my day has been waiting at this point, maybe more. Word to the wise, netplan reads config from lowest number to highest; if you set a configuration in one of the lower-numbered files, netplan will not override that value with a higher-number file. I set my extra private IP config file to 01 and nothing would override it at that point.

Configuration is in, and I run netplan apply…

Config File:

# /etc/netplan/60-extra-privates.yaml # yup, I'm 12

network:
    version: 2
    renderer: networkd
    ethernets:
        ens6:
            dhcp4: off
            dhcp6: on
            addresses:
                - 10.31.21.174/32
                - 10.31.21.44/32
                - 10.31.21.184/32
                - 10.31.21.6/32
                - 10.31.21.22/32
                - 10.31.21.150/32
                - 10.31.21.212/32
                - 10.31.21.196/32
                - 10.31.21.99/32
                - 10.31.21.157/32
            match:
                macaddress: 02:44:ac:46:58:e0
            set-name: ens6

Rerun ip show…

ubuntu:~$ sudo ip -c a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:07:aa:ff:f9:36 brd ff:ff:ff:ff:ff:ff
    inet 10.0.16.1/20 brd 10.0.31.255 scope global dynamic ens5
       valid_lft 3596sec preferred_lft 3596sec
    inet6 fe80::7:aaff:feff:f936/64 scope link
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:44:ac:46:58:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.31.21.174/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.44/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.184/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.6/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.22/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.150/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.212/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.196/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.99/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.157/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::44:acff:fe46:58e0/64 scope link
       valid_lft forever preferred_lft forever

Looks a little better, at least ens6 is UP now…

IP addresses are all showing, that’s a good sign. ifconfig gives a different response, but we don’t have to worry about that anymore…we’re ip users now!

But…(there’s always a but)…going to the public IP attached to that private IP still comes up with nothing. Time to run through the ol’ checklist…

  1. Is NGINX running? If so, better catch it…if not, restart it.
  2. Is PHP ok? Is PHP-FPM running and is it the right version? (7.2 as of this writing)
  3. Is NGINX configured properly?
  4. Is each site configured to the appropriate private IP address?

Well, shit. I’ve never hoped I fucked up an NGINX config file so badly before.

Day two; already two hours in. Strong black coffee and an apple…should have eaten more. Should have added Tullamore Dew to the coffee.

Finally came across Thomas Ward’s answer on AskUbuntu which says sub-interfaces aren’t possible (Netplan supports multiple IP’s on a single interface). Has this all been a snipe hunt? In all fairness, Netplan’s documentation says so, had I only read those examples closer…FML (and also, blocking all connections to the server with my virtual interface pretty much taught me as much…but at least now we have different ways of phrasing the question). According to user slangasek on AskUbuntu, multiple IP’s on a single interface is considered best practice (as compared to virtualizing additional interfaces). Clearly, I’m not a networking geek and don’t keep up (I’m much more Dev than Ops). But hey, what more can we ask than constantly improving and learning new things? Sure, ifconfig no longer ships with the most recent Ubuntu, but I could have rolled back to 16.04 or even installed ifconfig and overrode the configuration, telling Ubuntu to return to the Old Ways. No one likes change, but no wise man denies its inevitability (disclaimer: I’m no wise man). I’m trusting slangasek on this; I lack the time to research networking best practices, and Netplan is favored over ifconfig on Ubuntu 18.04, so good enough for me. Forward we march toward insanity greatness!

Tested switching ens6 to dhcp4 = true…bad idea. Detaching the network interface saved my bacon. Beats spawning yet another instance. FWIW, you might see examples setting dhcp4 to either on/off or true/false…both work. Back to the drawing board. Don’t need bridging, the IP’s attach direct to the interface and already show up with the ip -c a command. Sigh.

While I’m thinking about it, here’s a couple aliases I add to make jumping screens easier (screen is a lifesaver, especially when the server disconnects you):

# ~/.bashrc

# ... lots of stuff here...
# somewhere around line 95...
# screen aliases
alias s='screen -x'
alias ss='screen -S'

Hey, maybe it’s a firewall issue…

ubuntu:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

iptables is clear (makes perfect sense, we rely on the AWS firewall instead). AWS firewall looks good, allows connections on ports 80 and 443 (NGINX handles the redirects from 80 to 443; can’t trust users to automatically type in https yet…if ever).

Two hours later and I realize the Elastic IP’s are bound to the VPC, not the EC2 instance itself (the new way on AWS). FML. Time to figure this shit out…and learn the VPC has no default access to the outside world (which seems odd seeing as I can hit the AWS-set public IP, but maybe that’s bound to the EC2 instance and not the VPC; I just don’t know). Elastic IP’s already set up means I don’t have to worry about assigning public IP’s (seems obvious).

Time to enable Flow Logs via CloudWatch. “Good” news…the VPC blocks the connections. OK, time to look at the ACL…everything open on 0.0.0.0\0…sigh. Time to look at the Internet Gateway again.

Checklist time…

  1. Does the VPC have an Internet Gateway? Yuuup.
  2. Does the subnet route table point to the Internet Gateway? Yuuuuuup.
  3. Do the instances in the subnet have globally unique IP addresses? Yuuuuuuuuup.
  4. Do the network access control and security group rules allow relevant traffic to flow to and from the instance? It sure looks like it, but CloudWatch tells a different story…
  5. If I visit one of the Elastic IP’s connected to the interface from my browser, does it work? NOPE. FML.
  6. Maybe it’s just me? Not according to “Down for Everyone or Just Me?”

So. Reviewing CloudWatch shows a request to the Elastic IP pointed to the Private IP pointed to the correct EC2 instance was Rejected. OK, back to the old drawing board, time to figure out where the rejection occurs (and why). For a second there, I thought I was losing my mind and couldn’t read the log, but it turns out I was correct. I’d like to say I’m only wrong when I think I’m wrong, but this post proves otherwise.

OK, the logs show an ACCEPT on 80 and 22, at least for the Network Interface in the current subnet. So the Elastic IP’s pass to the Private IP’s no problem. Good to know. If you ever need a reminder that your servers are under constant attack from everywhere, record REJECT VPC Flow Logs to CloudWatch.

Maybe that’s a breakthrough… 

When looking at the NGINX default configuration page, the page loads just fine. Every Elastic IP on that network interface loads just fine…with the default page. It looks like NGINX doesn’t recognize the local IP…two steps forward and 47 back. Can’t find anything on Google about NGINX seeing (or not seeing) IP’s defined in Netplan. Almost every example shows a config that doesn’t specify an IP. Back to conversing with my AWS Certified friend…suggests I bind the domain to the directory (the server_name directive, which I did) and don’t worry about defining the IP on the NGINX config. Sometimes we’re too close to the problem to see a clear solution.

Well. NGINX is ignoring every private IP I’ve got except for the primary one (they all come up when running ip -c a, so I know Ubuntu sees them).

The most frustrating thing…none of this is uncommon and shouldn’t be difficult. AWS supports multiple IP’s on a Network Interface for easy swapping when EC2 instances have problems.

You came here to understand how to assign multiple private IP’s to a single AWS instance using Netplan, and we all got derailed. However, I think it’s safe to say I made a bunch of mistakes so you don’t have to.

If you’re anything like me, you were really hoping for a happy ending to this story. There is one, in the sense that the server is finally back up and churning out happy little 1’s and 0’s. However, I “fixed” it by rolling back to 16.04 and using ifconfig. I’m unsure NGINX has any way to work with the setup I had. I’ve talked with experts, done countless Google searches, and finally said fuck it…shit has to work and it has to work on a deadline. In a perfect world, I’d have had the time to delve into the NGINX and Netplan documentation headfirst and resolve the issue (assuming it can be resolved within the limitations of both). I might have used Apache instead, or done any number of things.

Unfortunately, that’s not how it went. But from a delivery perspective, the customer is happy with his robust new server and I no longer have to lose a moment of sleep on this roller coaster I spent the last two days on.

Happy travels!