SFTP to S3 with S3FS and EC2

I’ve been looking for an SFTP to S3 solution for years now. In an infinite number of universes, I’ve had an infinite number of experiences where team members and/or outsource teams need to upload files to clients’ S3 buckets. If you’re just seeing the AWS console for the first time, it’s overwhelming. Even limiting access (in my experience this is lukewarm at best: functional but not visually reflected; users can see what’s set up even if they can’t access them).

Perhaps you’ve seen AWS now offers an SFTP to S3 endpoint service. What you might not know (yet): you’ll spend $0.30 an hour plus $0.04/GB (up and down). While there are plenty of customers that simply don’t care about the cost, there are enough that do, so I went off in search of what I could do with a simple EC2 instance instead.

First, I used a couple of different tutorials: one from WinSCP, and a more in-depth one from Cloud Academy. For my time and effort, the Cloud Academy one was a little more detailed and, IMO, better for a first-time run. From a security standpoint, I prefer Cloud Academy’s use of IAM roles instead of local aws-cli config files.

The core to making this transition work is utilizing the S3FS-FUSE filesystem, which lets you mount an S3 bucket as a local drive. I won’t go into details here, feel free to read up on it if you so desire.

Improving Security

I suggest making a handful of changes to the tutorials above to further secure your FTP server:

  1. don’t install a local FTP server; use FTP over SSH (SFTP); avoid unnecessary services and running processes whenever possible
  2. if you want to be even more secure, use EC2 Security Groups and firewall the server to only necessary users (you could also change the port SSH runs on your server if you feel it necessary)
  3. use key exchanges over SSH instead of passwords (and disable SSH passwords altogether)
  4. when setting up the IAM Policy, limit the access only to your local AWS VPC

How Can I Become a Developer and Make Tons of Money Super Fast?

Spoiler alert: you can’t. You won’t become a developer overnight, and it won’t happen fast.

I’ve seen (and feel like I’ve answered) this question so many times on Quora, that I decided to just write an answer here and link it often. Mostly because I don’t like to write things more than once, but also because I like free traffic.

Everyone…seriously now…this isn’t a fast process. You don’t need to go to college or a coding bootcamp, you can do it totally on your own (but you’re nearly guaranteed to miss some of the more theoretical stuff…this won’t make or break you but may make future jobs unavailable).

Louder, for those in the back…


Stop listening to the ads that promise you “become a coder in 6 weeks!” Sure, you’ll have some fundamentals, but you won’t be at a high level. Do you believe ads telling you “you can grow 6 inches overnight!,” “lonely wives want to meet you!,” or “lose 30 lbs of fat…and GAIN 30 lbs of muscle…in just 3 weeks!”

Shocker Incoming…

  1. You’re not gonna gain 6 inches…you aren’t getting a longer, thicker, stronger cock and you’re not getting any taller. The world isn’t a Daft Punk song. You can go have painful and expensive surgery to stretch your legs or inject silicon into your lil’ buddy, but that’s about it.
  2. The only lonely wife who wants to meet you is the one already married to you. If you’re not married, sorry, you’re probably out of luck and will die alone.
  3. Yes, people really do want to believe weight loss and muscle mass gain is this easy (admittedly, I’ve never tried this subject line, but now I lowkey want to…)
  4. You won’t become a developer in six weeks. You’ll get some fundamentals in a bootcamp if you’re self-driven, but most bootcamps are about as educationally rich as Trump University.

If you want to become a developer, you need to put in work.

Before you even start, stop trying to be everything to everyone…it’s a fool’s errand. Feel free to do some research on what you want to do at this point, but you have a few steps before you really get started. I happen to like PHP with Laravel or Symfony for backend work, but you can also use Ruby on Rails, Python, JavaScript via Angular, React, Vue, Svelte/Sapper…or a slew of other options. For web-based development, be sure you understand static content before moving onto dynamic.

A brief note on testing and TDD

You don’t need to test everything, that’s not realistic, but have some test cases for things when you need to confirm they work. Automated testing will save you QA, and if you have to QA your own stuff, you’re in trouble…you know how it’s supposed to work, you know how to use it from your experience, but you are probably not comfortable stress-testing a system.

So how do I start?

  1. Start with a language: HTML (technically a markup language), JavaScript, and CSS if you’re going to do anything web-based; Java if you want to build apps for Android devices (including Chromebooks) and Swift if you want to build for iOS.
  2. Learn how to set up a development environment for your chosen language. Docker isn’t a bad option, but Bitnami or Heroku might be easier. Alternately, HostGator (use code YVOSAVE99 for $9.99 off your first month of hosting and to put money in my coffers) is pretty simple for a webhost and includes SSL/TLS.
  3. Learn the best practices for your chosen language(s)…including the testing framework(s). Learn git (use GitHub, GitLab, Bitbucket, whatever…) and understand why we use version control.
  4. Now that you have some groundwork (and an environment to work in), it’s time to start learning to develop.
  5. Tackle tutorials and learn the fundamentals. Develop some local projects just to learn how to do the work.
  6. This would be a good time to bring a database into the picture: traditional relational, like MariaDB, PostgreSQL, MySQL…or NoSQL like MongoDB. Almost everything today requires a data model and interaction with said model.
  7. Find some popular open source projects you can contribute to, understand what they’re doing, and tackle one or more of the issues. The hope, with more popular projects, is those projects are better moderated and therefore will require code reviews and feedback when you commit a pull request.
  8. Jump into freelancing or go work for someone else to get some team experience and a feel for working with other developers…no one likes a lone wolf (especially other lone wolves).

That’s it. That’s the big secret. Get out there and put in work.

DoCuMenTaTiOn DoEsN’T MatTer

Ho. Lee. Fuck.

Two fucking days. Two mother-fucking shit-eating stupid-waste-of-time days…

Technically, the last couple of hours of one day and all of another.

Lost. Gone to the great timekeeper in the sky.

There’s something about “assume” and “making an ass out of ‘u’ and me…”

Where does this come from? Thanks for asking! Thomas Bradshaw at Atlassian provided the most helpful response I’ve seen in two days’ research (I’m not the OP). A huge thank-you to Thomas for that information. Usually, I find myself learning as I go, often with no internal support for anything new that needs to happen. I’m a lone wolf by circumstance, not by choice per se.

I started writing this as a rant against substandard documentation. The developers or technical writers could have (and probably should have) written clearer documentation. I might not have lost two days otherwise. However, reading through some of the code examples again for this post, I realized I glossed over content. Nothing like a mouthful of crow. Mmmm.

Now I don’t get to complain about the documentation, developers, or technical writers…

Instead, I have to express my frustration at myself. I read through the great walkthrough GitConnected provides. Somehow, I repeatedly miss the reference to executing the zip command from the bitbucket-pipelines.yml file.

As important as writing good documentation is, reading comprehension matters. Take the time, when learning something new, to slow down a bit. Often, you can find the extra information you need in code samples. Take the time to read. You might surprise yourself.

That aside…

Developers and technical writers, do us all (including yourselves) a favor…

Write better documentation.

I know, I know…I don’t like to, either. But together we can make the world a better place…or whatever tripe the tech bros spew these days.

Seriously. Take that extra little bit of time.

Now get out there, and make me fucking proud.

Dear Recruiters…

It’s not you, it’s me. I’m an asshole…I know this. I come from a long line of assholes: my father before me, his before him, and I’m sure all down the line back to my roots in the frozen wasteland of Northern Europe.

Look, I know you have to meet goals. I understand you have to make sure I and my kindred spirits have the skills you need.

But let’s be honest for a second here. This is a terrible sales strategy.

I’m going to let you in on a little secret…

Come closer…


Whoa, whoa, whoa, too close! Personal space is a thing, man!

But I digress. I don’t give a fuck what you need. It doesn’t make any difference to me; either I can do the work (and I’ll work with you to move forward) or I can’t/don’t want to, and I won’t.

Do you know (or want to know) what does matter to me?

Four questions. That’s all I care about:

  1. What are you planning to pay me? Let’s be honest, if we can’t meet here we’re both wasting our time. Help me help you. If you can’t meet my threshold, that’s totally cool, no harm, no foul. If I charge more than you can or want to pay, I get it. I’m not for everyone (just read this blog and you’ll know that). Let’s save time if we’re not gonna meet on this point, letting you get in more calls (and hopefully more commission…I’m all about everyone coming out ahead financially). I’m not stuck on an hourly wage, either…let’s build win-win-win situations, so you, the client, and I all feel like we’re fairly compensated and get what we pay for (I pay, too: in time, not money).

  2. Can I work remote? I’ve spent a lot of time commuting in my life. I’d really rather not.

  3. Can I set my own hours? I’ve spent a lot of time getting up at 5 AM to commute to an 8 AM job. Some days I kill it waking up at 6 AM and some days I need a couple extra hours’ sleep, and kill it til 2 in the morning. I’m inconsistent, and while that’ll probably come home to roost in 20 years, in the meantime, it’s what works for me.

  4. Can we set this up as a corp-to-corp? This makes my accountant and lawyer happier. Corporations carry liability insurance. Everyone is protected, everyone wins.

We all want the same thing…a fleet of exotics in our driveway and lining the street. Let’s get out there and get that horsepower!

Surprise! Private IP’s Can Move…

The AWS adventures continue…

I’ll keep this short and sweet, because it was almost life-changing and I’m tired of ranting. Something something be more positive something attitude affects everything something something eat a dick.

I noticed, previously, when adding additional private IP’s to an AWS instance, the input field has a placeholder value of Auto-assign. I never gave it much thought until my EC2 instance stopped connecting to the Internet yesterday. Maybe I can write another post on that later. Hours pass. Attempts to fix an issue affecting updates and certbot fail miserably. I rebuild the damn thing this morning; hence the lesson…

I debate and decide to test the merits of a separate standalone Network Interface. I’m certain, out there somewhere, in the deathlike silence and chill of a datacenter, a Sysadmin or DevOps Engineer knows the answer. I’m not that wizard, and the Interface idea fails spectacularly when binding those private IP’s to the EC2 instance. You might think I’d learn to test one or two IP’s instead of rewriting the whole damn netplan file, but apparently you don’t know me. So, the separate Network Interface idea dies in solitary.

At this point, I’m resigned to rewriting the netplan YAML again (that would be the second time in one day). “What the fuck, I might as well try to move these private IP’s and save myself an hour of work after spending 20 hours trying to fix it.”

Lo and behold, it worked. The heavens parted and the networking gods smiled on me. The fact you’re reading this is proof they haven’t shit on me since (at least in this case).

Slaying the Dragon

or “How I Figured the Fucking IP’s Out and Learned to Love netplan

I suppose it’s that time of year again, where I write one of my two annual posts. Some years I only get through one. I actually enjoy writing, I just never seem to find time to do it. Oh, well, as the title implies, and the subject line explicitly states, I finally figured out that pain-in-the-ass (PITA) netplan. If you’re curious where this started, take a look (just don’t forget to come back)!

The problem never changed, I ended up trying to worm myself out of facing this enemy again with a number of t3.nano instances instead of one m5.medium instance. Unfortunately (or so I thoughtlittle did I know…), some of the WordPress and other database-heavy code stubbornly refused to run on a t3.nano. By the time I had eight t3.nanos, four t3.micros, a t2.micro, and a partridge in a pear tree, I opted to switch to a bigger server and put everything on it.

That meant one of two options: learn netplan (shudder) or use Ubuntu 16.04 and go through the joy of Canonical’s lack of support for an old OS (read: having to pull in additional repos to build PHP 7.x). I’ll admit…I started with 16.04. It didn’t take long to piss me off, though, and I “embraced the suck” (as High Overlord Pelosi would like all citizens…er, except Congress…to do).

Much to my surprise, it came way easier this time… (lucky “it”…S/O Rob, Anybody, and Dawn)…

The first thing I had to do (which I could swear I tried last time) was create a separate YAML file for storing the network data…and call it first (the base file is called 50-cloud-init.yaml, I named mine 01-multi-ips.yaml). Hold up a sec there, cowboy…the first thing I did was build an AMI out of my server so I could spin everything back up if I fucked everything to hell again.

So, technically the second was that new file thing.

I kept the names the default file did.

The Original File

# /etc/netplan/50-cloud-init.yaml

# a bunch of stuff Ubuntu adds to remind you this file
# is auto-generated and might wipe your changes if you
# save them here
version: 2
dhcp4: true
macaddress: HA:HA:IM:NO:TR:EA:L0
set-name: ens9

Bright and Shiny (the new config file)

I’m only using a couple IP’s for an example, but I ended up binding whatever the maximum is. But before you take a look, keep a few things in mind:

  1. is the primary private IP on this imaginary instance
  2. is the first additional private IP on this instance
  3. ens9 is the name of the network device from the previous file; you might see a different name, and if you do, use the name you see in your cloud config file
  4. the cloud config file might have a different name, but you should only have one or two files max in that directory
  5. back your shit up before you fuck about with IP’s and network interfaces
  6. I’m fairly sure this is spaced correctly, but it’s a good idea to get a YAML Linter or otherwise validate your YAML before running the command to kick off the IP update or restarting your server
# /etc/netplan/01-multi-ips.yaml

# DISCLAIMER: Fuck shit up at your
# own risk. Back your shit up. Wash
# your hands. Don't be a dick.

version: 2
renderer: networkd
# surprise, motherfucker! this is the same as the one above
# you need to list every private IP address
# that you want netplan to recognize here
# and so on...
dhcp4: no
# now for the real fun...map the routes
- to:
table: 1000
- to:
table: 1000
- to:
table: 1000
- from:
- from:

I really wish I could explain what everything here is, but I can’t, but the good news is…Google exists…and it’s your friend for finding answers on StackOverflow.

Just one two more steps…

Run sudo netplan apply –debug

I’m a fan of running –debug because it will warn you if something is awry before you fuck your connection to hell. You’ll still have a chance to delete or comment out or rename your monster before things go south.

Finally, bind your Elastic IP’s to your private IP’s and you’ll see things work. ifconfig will not look right (neither will sudo netplan ip leases ens9)

AWS Private IP’s on Ubuntu 18.04 with Netplan

Ugh. Uggggggggghhhhhhhhhhhhh. Ffffffffffffffffffffffffffffffffffff. Twelve hours to get this damn thing right. OK, deep breath. Use Google, ask sys admin friends, LINT your YAML. Spin up alternate instances because you mis-configured Netplan and it took the server offline. Realize your snapshot was designed too late and you’re totally fucked because of course the dev team doesn’t have backups. Stop what you’re doing because a domain expired on an app that has traction and fuck how did I miss that? Been a ball-busting week. Live and learn. Wear a Thai steel cup next time, discomfort aside, the protection is second to none.

Sigh heavily. Have a Lagavulin (or 3). Smoke half a box of Davidoff cigarillos…get wired for sound…that’s a lot of nicotine for someone who has one cigar a week, two at most. Don’t think about how much was spent between the scotch and cigarillos, or how quickly you consumed either one of them. Deep breath (again). Yoga…just kidding, another scotch and two more cigarillos. Three more and this pack is done…can’t just run down to the nearest convenience store…fuck, the closest tobacconist that will have something worth smoking is in Sacramento and that’s an hour round trip if traffic is light…fuck, it’s Saturday, traffic won’t be light…better manage these last two cigarillos. Hey, here’s an idea, write a blog post, haven’t done that in a while…hmm…but what to write about? Dammit, this server needs to be ready yesterday. System administrators understand the pain but customers don’t, especially when the design team dragged ass and turned in half-assed work. Fuck, cost a design client, but can’t leave them hanging, that’s awful customer service. There’s too little care about customers today as-is, oh no, the customer hurt my feelings or wrote some awful shit about me. Fuck, time to scrub Yelp…I mean, ask a customer politely to change their review. Fuck it, they’ve never been good with password management, won’t remember what they wrote, and don’t have time to go back and look at their past reviews. Sure, some people might, but that customer won’t. Where were we? That’s right, shitty customer service. Won’t provide bad customer service, that’s bad business. ipconfig is deprecated in favor of ip, and that YAML config is a bitch dealing with multiple IP’s…it isn’t supposed to be, but for whatever reason it doesn’t want to take. vi keeps enforcing its own formatting on the YAML, which is pretty awful when you consider the importance of formatting in YAML…like Python. Netplan has extensive documentation, and even some great examples of config files.

Thought about sharing the first iteration of the config file (set routes at your own risk…broke the server twice trying that shit), but that seems pointless. No one wants to see what doesn’t work. You wouldn’t have bothered reading this far if you did. IP’s have been changed to protect the innocent…

They say there’s no atheist in a foxhole, but there’s no atheist when the network connection keeps failing and there’s no backup, either. AWS shows terminal screenshots, if only they had some way to virtually connect locally rather than via SSH… Oh, look, the public IP changed, maybe it will work as soon as I update PuTTY…fuck. Let’s add another private IP and an Elastic IP to that, and maybe that will work…fuck. No connection, the server is fucked. I’m fucked.

Conversation with a friend with AWS Certification. Suggests shutting down the instance, spinning up a new one, and mounting the volume to the new instance. Done this with VM’s dozens of times. Guess sometimes we’re just too close to the problem to see a solution. Fuck, I should probably eat while this volume is detaching from the original and attaching to the new one. Alcohol has calories, that counts, right? Half a bottle. Fuck me. So much for going and grabbing a bite somewhere, not that I really have the time, but shit it would be nice to get out of the house today. Nope, back to the grind…12 hours in and counting. Damn, JoyRun was great, too bad as soon as you move out of a college town it doesn’t work anymore. Maybe Postmates delivers? Ha! Not here. The only pizza place still open is Domino’s…”pizza”

I need to know more local people, convince someone to do a food run for me while I’m grinding through this damn server. OK. Deep breath, the first sign of life. The volume mounted and I was able to edit the YAML file, get rid of the offending code (it was an attempt to separate the addresses into ifconfig-style ethX:format):

# don't do this
# for real
# it's gonna take down your fuckin' server
# some other YAML config
    eth1: # this is fine
        addresses: # still fine
            - # yup, still fine
            - # still
    eth1s0: # woah, nellie
       addresses: # doesn't matter, the previous line already broke shit

This works if the system has an interface eth1s0; if not, the IP configuration gets fucked. Yes, fucked proper. No clue why people keep asking.

Time to knuckle down and learn the ip tool…

ubuntu:~$ sudo ip -c a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:07:aa:ff:f9:36 brd ff:ff:ff:ff:ff:ff
    inet brd scope global dynamic ens5
       valid_lft 2195sec preferred_lft 2195sec
    inet6 fe80::7:aaff:feff:f936/64 scope link
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:44:ac:46:58:e0 brd ff:ff:ff:ff:ff:ff

Do you see what I see? Further, why did it take me twelve fucking hours to see this???  Chalk it up to the aging brain not learning new things nearly as quickly as it used to. Look… ens5 shows “UP,LOWER_UP” while ens6 does not. The good news is, Ubuntu recognizes the new interface and wants to work with it. Fresh Ubuntu 18.04 install (on what I think is the seventh instance in the last 90 minutes); I had not touched the network configuration on this system…yet. Fortunately, that disconnected volume saved my bacon…keep backups, kids. Take snapshots before you muck with network configurations. I know better and didn’t until it was too late.

No one ever tells you how much time sysadmins spend waiting on shit. Roughly a third of my day has been waiting at this point, maybe more. Word to the wise, netplan reads config from lowest number to highest; if you set a configuration in one of the lower-numbered files, netplan will not override that value with a higher-number file. I set my extra private IP config file to 01 and nothing would override it at that point.

Configuration is in, and I run netplan apply…

Config File:

# /etc/netplan/60-extra-privates.yaml # yup, I'm 12

    version: 2
    renderer: networkd
            dhcp4: off
            dhcp6: on
                macaddress: 02:44:ac:46:58:e0
            set-name: ens6

Rerun ip show…

ubuntu:~$ sudo ip -c a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:07:aa:ff:f9:36 brd ff:ff:ff:ff:ff:ff
    inet brd scope global dynamic ens5
       valid_lft 3596sec preferred_lft 3596sec
    inet6 fe80::7:aaff:feff:f936/64 scope link
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:44:ac:46:58:e0 brd ff:ff:ff:ff:ff:ff
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::44:acff:fe46:58e0/64 scope link
       valid_lft forever preferred_lft forever

Looks a little better, at least ens6 is UP now…

IP addresses are all showing, that’s a good sign. ifconfig gives a different response, but we don’t have to worry about that anymore…we’re ip users now!

But…(there’s always a but)…going to the public IP attached to that private IP still comes up with nothing. Time to run through the ol’ checklist…

  1. Is NGINX running? If so, better catch it…if not, restart it.
  2. Is PHP ok? Is PHP-FPM running and is it the right version? (7.2 as of this writing)
  3. Is NGINX configured properly?
  4. Is each site configured to the appropriate private IP address?

Well, shit. I’ve never hoped I fucked up an NGINX config file so badly before.

Day two; already two hours in. Strong black coffee and an apple…should have eaten more. Should have added Tullamore Dew to the coffee.

Finally came across Thomas Ward’s answer on AskUbuntu which says sub-interfaces aren’t possible (Netplan supports multiple IP’s on a single interface). Has this all been a snipe hunt? In all fairness, Netplan’s documentation says so, had I only read those examples closer…FML (and also, blocking all connections to the server with my virtual interface pretty much taught me as much…but at least now we have different ways of phrasing the question). According to user slangasek on AskUbuntu, multiple IP’s on a single interface is considered best practice (as compared to virtualizing additional interfaces). Clearly, I’m not a networking geek and don’t keep up (I’m much more Dev than Ops). But hey, what more can we ask than constantly improving and learning new things? Sure, ifconfig no longer ships with the most recent Ubuntu, but I could have rolled back to 16.04 or even installed ifconfig and overrode the configuration, telling Ubuntu to return to the Old Ways. No one likes change, but no wise man denies its inevitability (disclaimer: I’m no wise man). I’m trusting slangasek on this; I lack the time to research networking best practices, and Netplan is favored over ifconfig on Ubuntu 18.04, so good enough for me. Forward we march toward insanity greatness!

Tested switching ens6 to dhcp4 = true…bad idea. Detaching the network interface saved my bacon. Beats spawning yet another instance. FWIW, you might see examples setting dhcp4 to either on/off or true/false…both work. Back to the drawing board. Don’t need bridging, the IP’s attach direct to the interface and already show up with the ip -c a command. Sigh.

While I’m thinking about it, here’s a couple aliases I add to make jumping screens easier (screen is a lifesaver, especially when the server disconnects you):

# ~/.bashrc

# ... lots of stuff here...
# somewhere around line 95...
# screen aliases
alias s='screen -x'
alias ss='screen -S'

Hey, maybe it’s a firewall issue…

ubuntu:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

iptables is clear (makes perfect sense, we rely on the AWS firewall instead). AWS firewall looks good, allows connections on ports 80 and 443 (NGINX handles the redirects from 80 to 443; can’t trust users to automatically type in https yet…if ever).

Two hours later and I realize the Elastic IP’s are bound to the VPC, not the EC2 instance itself (the new way on AWS). FML. Time to figure this shit out…and learn the VPC has no default access to the outside world (which seems odd seeing as I can hit the AWS-set public IP, but maybe that’s bound to the EC2 instance and not the VPC; I just don’t know). Elastic IP’s already set up means I don’t have to worry about assigning public IP’s (seems obvious).

Time to enable Flow Logs via CloudWatch. “Good” news…the VPC blocks the connections. OK, time to look at the ACL…everything open on\0…sigh. Time to look at the Internet Gateway again.

Checklist time…

  1. Does the VPC have an Internet Gateway? Yuuup.
  2. Does the subnet route table point to the Internet Gateway? Yuuuuuup.
  3. Do the instances in the subnet have globally unique IP addresses? Yuuuuuuuuup.
  4. Do the network access control and security group rules allow relevant traffic to flow to and from the instance? It sure looks like it, but CloudWatch tells a different story…
  5. If I visit one of the Elastic IP’s connected to the interface from my browser, does it work? NOPE. FML.
  6. Maybe it’s just me? Not according to “Down for Everyone or Just Me?”

So. Reviewing CloudWatch shows a request to the Elastic IP pointed to the Private IP pointed to the correct EC2 instance was Rejected. OK, back to the old drawing board, time to figure out where the rejection occurs (and why). For a second there, I thought I was losing my mind and couldn’t read the log, but it turns out I was correct. I’d like to say I’m only wrong when I think I’m wrong, but this post proves otherwise.

OK, the logs show an ACCEPT on 80 and 22, at least for the Network Interface in the current subnet. So the Elastic IP’s pass to the Private IP’s no problem. Good to know. If you ever need a reminder that your servers are under constant attack from everywhere, record REJECT VPC Flow Logs to CloudWatch.

Maybe that’s a breakthrough… 

When looking at the NGINX default configuration page, the page loads just fine. Every Elastic IP on that network interface loads just fine…with the default page. It looks like NGINX doesn’t recognize the local IP…two steps forward and 47 back. Can’t find anything on Google about NGINX seeing (or not seeing) IP’s defined in Netplan. Almost every example shows a config that doesn’t specify an IP. Back to conversing with my AWS Certified friend…suggests I bind the domain to the directory (the server_name directive, which I did) and don’t worry about defining the IP on the NGINX config. Sometimes we’re too close to the problem to see a clear solution.

Well. NGINX is ignoring every private IP I’ve got except for the primary one (they all come up when running ip -c a, so I know Ubuntu sees them).

The most frustrating thing…none of this is uncommon and shouldn’t be difficult. AWS supports multiple IP’s on a Network Interface for easy swapping when EC2 instances have problems.

You came here to understand how to assign multiple private IP’s to a single AWS instance using Netplan, and we all got derailed. However, I think it’s safe to say I made a bunch of mistakes so you don’t have to.

If you’re anything like me, you were really hoping for a happy ending to this story. There is one, in the sense that the server is finally back up and churning out happy little 1’s and 0’s. However, I “fixed” it by rolling back to 16.04 and using ifconfig. I’m unsure NGINX has any way to work with the setup I had. I’ve talked with experts, done countless Google searches, and finally said fuck it…shit has to work and it has to work on a deadline. In a perfect world, I’d have had the time to delve into the NGINX and Netplan documentation headfirst and resolve the issue (assuming it can be resolved within the limitations of both). I might have used Apache instead, or done any number of things.

Unfortunately, that’s not how it went. But from a delivery perspective, the customer is happy with his robust new server and I no longer have to lose a moment of sleep on this roller coaster I spent the last two days on.

Happy travels!

Let’s Talk Freelancing

Freelancing. It’s kind of an ugly word, really, like outsourcing. Of course, managers love outsourcing because they can save money (usually it costs more in the long run, but who has time to notice that?).

Getting Burned

How many times have you done freelance work, only to have your “client” burn you? What options do you have (personally, I like contracts and collection agencies). You know who does not like collection agencies? Florida (and Texas, but much more Florida). Collection agents do not collect from Florida because they have little to no legal weight, and cannot get the money anyway. I also like escrow services, I believe the investment (especially with new clients) pays off.

Freelancing Sites

I have good and bad experiences with some of the freelancing websites currently available. The only invite UpWork recruiters sent me was an Excel file with a Trojan (that practice of not opening random files from the web comes in handy). UpWork sent confirmation of the Trojan a couple days after the invites went out. I have good experiences hiring from eLance (the precursor to UpWork), but not getting work through them.

Outsource offers lots of opportunity, but also lots of people looking for super-cheap work. You do pay for credits, but at least they carry over if you do not find anything. Outsource also does a great job of refunding if the client does not view your offer within a week. I’ve pulled in about $5k across two jobs with Outsource. However, I also have had a couple of nearly-insulting offers to work for anywhere from half to 10% of what I normally charge, with clients expecting me to thank them for it.

I have not had the opportunity to explore Moonlighting yet, but it looks like an interesting opportunity…when I can try it out, I can report back.

Finally, I recently came across Toptal Web Freelancers Network, and honestly, I find this one the most interesting. First, developers must go beyond an application and through a screening process. I hope this translates to clients who have no qualms about spending money. After all, this blog isn’t putting a gated Murci in my driveway… 😉

I’m looking forward to the interview process (I apologize in advance, I cannot share anything that goes on therein).

All the best in your endeavors!

Misdirected Frustrations

Let's Ban...
If you’re familiar with gun laws, this cartoon is laughable. If not, it might really make you frustrated and want to lash out at the GOP. Now, even as a registered Republican, I have to say I’m not re-registering with that party until they get their heads out of their asses. I may never register Republican again (women’s health, women’s rights, xenophobia, and on and on and on).

Let’s dissect this one panel at a time…
“Assault weapons?” Really? Still? You might as well say “climate change doesn’t exist” and “evolution is a theory.” If you think it’s by what the gun looks like (take a look at the Ruger Mini-14 in two different “models,” the tactical and ranch…this is the EXACT same gun except for what it “looks like”…yet the left would call the tactical an “assault weapon”…okay, and evolution is a theory, vaccines cause autism, and climate change is made up). It’s all bullshit and we all know it deep down.
Background checks…there’s the kicker, right? I mean, why not have better background checks; who could argue that? I’d love to see better background checks that cross state and federal agencies. One little problem that everyone overlooks, and that’s HIPAA privacy. Yikes…wait…that’s a HUGE problem. See, we can’t just access medical data anymore, and what’s next, are you going to limit the rest of the Bill of Rights to people you deem medically safe? First they came for the Communists…
The “gun show loophole”…doesn’t even exist. This is either the biggest or second biggest lie with regards to gun sales the media tells you. The way legal gun buying works is you buy through a dealer, they run a background check, and then you wait some period and get your gun (some states don’t have waiting periods on long guns). The US Government leaves it up to the states to decide whether citizens of that state can buy guns from other states (generally, the rule is you can’t). Here’s where the “loophole” comes in (spoiler alert; it’s not a loophole!): some states require you to do a private party gun sale through a Federally licensed firearms dealer (FFL holder). However, some states do not. Those same laws that allow a buyer to make a purchase from any legal individual allow buyers to make a purchase from private sellers at gun shows. For what it’s worth, in California, you must go through a dealer for every transaction, private party or not (it’s been that way for more than 20 years now). That means, that if you go to a gun show in California, you must get a background check. Anyone willing to sell you a gun at a gun show without going through an FFL holder is either stupid or undercover. Neither one will come out in your favor.
Ah, armor-piercing (AP) bullets…why would anyone need those?
Before we get into the “real life” lesson, let’s understand that AP rounds have been illegal for US civilians to use, import, or manufacture since 1986.
Well, let’s talk about ballistic (unfortunately…and incorrectly…often called bulletproof) armor. First, as the proper name states, this is armor to protect you against ballistic projectiles (it does not guarantee you survive or magically block bullets). The most important part of ballistic armor for this conversation is what’s known as the NIJ 0101.05 Standard; this is a rating system from the National Institute of Justice for soft body armor. There are three levels of protection for soft armor (common ballistic vests, ballistic protective clothing, etc.) as opposed to hard plate armor (we’ll get to that in a second). As with anything, there’s a cost/protection tradeoff (there’s a financial portion to the cost, and also a weight portion; adding a pound on Level II-A to get III-A armor might not be a big deal, but adding an additional 8 pounds for hard plate might be!). Here’s a handy table as to what kind of armor is good for protecting against what (this table was taken from BulletProofME.com’s website):
Level Tested For Comments
II-A 9mm FMJ at ~1,090 FPS (332 m/s); .40 S&W FMJ at 1,025 FPS (312 m/s) Absolute minimum recommended armor; not significant protection against the blunt trauma from the impact of the bullet against the armor. Generally a special-order item.
II 9mm FMJ at ~1,175 FPS (~358 m/s); .357 JSP at ~1,400 FPS (~427 m/s) A great balance between blunt trauma protection, cost, comfort, and concealment. Handles the blunt trauma of higher velocity rounds than II-A.
III-A 9mm FMJ at ~1,400 FPS (~427 m/s); .44 Magnum SJHP at ~1,400 FPS (~427 m/s) The highest blunt trauma protection available in soft armor. Minimizes blunt trauma injury, allowing more effective return fire. Recommended for use in high-risk areas.
Well, that’s interesting…do you know what’s missing from that table? Rifle rounds (or, less common, pistol rounds fired from a rifle or carbine). What this table tells us is that a 9mm FMJ (full metal jacket) fired at 1,400 fps will penetrate (defeat) Level II armor. Technically, that makes a 9mm round fired at that velocity armor-piercing for Level II armor (for that matter, a 44 Magnum round will also defeat Level II). So, all of you clamoring for an end to “armor piercing” ammo but “don’t want to take hunters’ guns away” need to get up to speed on physics and limitations of body armor in the real world.
Let’s just look at hard plate armor for shits and giggles:
Level Tested For Comment
III .308 Winchester FMJ/7.62x51mm NATO; 6 rounds at ~2,750 FPS (~838 m/s) 1/4″ (6mm) Ballistic Steel

1/2″ (13mm) Ceramic

1″ (25mm) Polyethylene

IV .30-06 Armor-Piercing/.30 M2 AP; 1 round at ~2,850 FPS (~869 m/s) The highest rating for body armor.

3/4″ (18mm) Ceramic

1/2″ (12mm) Ballistic Steel
(vehicle armor only; too heavy for body armor)

Well now…that’s not what we expected. What happened to bulletproof armor? Ballistic armor isn’t what it’s made out to be in the movies, and maybe we’re a little loco about our obsession with armor-piercing rounds (since they already are illegal, in the strictest form of the term).

You’re probably interested, if you’re still reading, what makes a round “armor piercing.” Generally, these bullets have very dense cores; steel, tungsten-carbide, or depleted uranium. By the way…if your barrel isn’t set up to use AP rounds, you won’t get very far with them.

Restricting power of a political lobby group. Hmm, are you sure you want to really start there? I mean, because there are so many lobby groups out there, and I can almost guarantee one of them will say something you believe in that I disagree with. If you want to decrease the NRA’s power, just be aware that those exact same rules need to apply to everyone…not just people you disagree with.

My First Unity3D App

Wow…so that was exciting.

I just developed and published my first app using Unity 3D. I’ve learned a couple things along the way that might help someone else. For what it’s worth, I monetized with AdMob…not because I think AdMob is the best option (I don’t believe that at all), but because AdMob is a pretty solid mobile ad server. I may use a different solution in the future, but I’m not investing big money until I see how this pans out.

Which reminds me, I’ll also buy a Mac if this pans out. Using a VM was a pain in my ass (the setup went really quick). First, install your Mac OS X virtual machine on an SSD. I guarantee it will make a world of difference. Second, I like to work in big screens. It didn’t help me do that at all. I’m sure there’s a way, but in the last three weeks I’ve learned the basics of Unity 3D, C# (for scripting in Unity), how to basically use XCode (having never done it before). In the event you’re looking to build iOS apps with Unity…you need access to a Mac in some form or fashion. You can’t export iOS code without XCode (and integrating the AdMob SDK requires XCode anyway).

Second, make sure you have your code-signing ducks in a row. In XCode, this means using Apple’s Developer and iTunes Connect sites. There’s a caveat—you’ll need to make sure your account allows for push notifications, even if your app doesn’t push. Whether it was Unity 3D or the AdMob SDK, I’m not 100% sure, but you’ll save a step now (and potentially later) by enabling it (it requires you to generate some keys; Apple does a pretty good job of explaining how). However, Apple doesn’t really tell you how to get your keys, so I’ve included that below. For Android, you have to view the build settings in Unity 3D, create a new keystore (check the box, enter and confirm your password, click “Browse Keystores,” and save the keystore), and then create a new key with Unity (you’ll have the option to do so from the “Alias” drop down). Sign your code before you upload it.

  1. Log into iOS Dev Center
  2. Select “Certificates, Identities, & Profiles” from the “iOS Developer Program” menu on the right-hand side (click here for a screen shot)
  3. Choose “Provisioning Profiles” under the iOS Apps header
  4. Select “App ID’s” under “Identifiers”
  5. Click the “+” at the top
  6. Name the App ID, determine whether you want a specific or wildcard ID, and select the services you want (make sure to select “Push Notifications”)
  7. Confirm the information is correct and submit
  8. You will have to set up SSL certificates to use Push Notifications; Apple will walk you through it
  9. Select “Distribution” under “Provisioning Profiles” in the right-hand navigation
  10. Click the “+” at the top
  11. Select “App Store” under distribution and click “Continue”
  12. Select the App ID you want to create the profile for
  13. Select the associated certificate (may have one or more to select from)
  14. Name and generate the profile
  15. You’ll have to download the profile but everything else is pretty clear-cut

Feel free to post questions here as you have them!