The Problem With “FizzBuzz”

The first time I saw it, I was already rattled. I left from my home in Davis for San Francisco with plenty of time to make it even with traffic (an extra 90 minutes over what even the most pessimistic traffic forecasts gave). I was 10 minutes late, stress levels through the roof. It was unseasonably warm in SF that day and I ended up walking half a mile in the [relative] heat to get from parking to the interview.

I walked in and was grilled. My stress never came down; I don’t think my heart rate did, either. At this time in my career, I’d already provided solutions for Cal-Fire (as a one man scrum team) and the California Community College Tech Center (as lead developer). They gave me FizzBuzz, to write in my notebook. I got Fizz and Buzz without thinking about it…and completely forgot there could be a FizzBuzz result. To no one’s surprise, I didn’t get the job.

I’d already had community college board members bitch and moan and put down my entire dev team on a call. I’d been yelled at by clients; been called names and had my integrity, quality, and knowledge questioned when the client “couldn’t” pay her bill (<$400 while running two storefronts and having just bought a brand new Audi TT); I’d had a hyper-egotistical CEO ask me to lie to investors—I didn’t—and demand impossible deadlines). For whatever reason, I was rattled and missed a branch.

I don’t even recall what that job was anymore, and today I would know ahead of time and refuse the interview based on their reliance of code examples that only test whether you’ve seen the question posed while you’re crotch deep in an already unpleasant situation. IMO we both lost out over this question, and hiring managers continue to.

How Do I Start Getting Clients?

I’ve seen some form of this question a ton, pasted all over the Q-and-A websites. Here’s a couple of ideas to get you started.

Disclaimer: this is pretty much “Internet Marketing 101,” so if you’re here for marketing goodness, this isn’t it. This is specifically targeted toward individuals not sales-and-marketing minded who need to start somewhere.

Lead Funnels

If you can afford it, start with a lead funnel. If you’re a web developer, you can design it yourself.

Set up a lead funnel, with a splash page capturing contact info (at a minimum, first, last, company,  email, and phone). When prospects opt-in, follow up with emails and SMS messages. Use an existing ESP service that also provides SMS messaging. If you’re not good at copy, hire someone who is.

Advertise to your target audience with Facebook and Google (chances are you’re gonna need to get really creative here). As you get clients, ask them for referrals. 

Pounding the Pavement

If you don’t have the money to advertise, it’s time to start pounding the pavement. You’re going to have to start selling your skills. There will be lots of refusals. Ask anyway, and ask for referrals (“Do you know anyone who needs a website, it needs their website modernized?”).

If you’re really cutthroat, go find websites that aren’t ADA Compliant (in the US), are insecure, or are in dire need of SEO help (depending on your specialty). Run a preliminary scan and provide that free-of-charge with an introductory email. 

Referrals

Every time you finish up with one client, ask for a referral. Every time you’re told “Not interested,” ask for a referral.

Keep copious notes on the businesses you contact, and find out what their other needs are. You’re talking to a lot of people, and even if there’s not a fit for you right now, you might be able to bring others together. These are the actions that benefit you in the long run.

Never be afraid to ask for the sale.

SFTP to S3 with S3FS and EC2

I’ve been looking for an SFTP to S3 solution for years now. In an infinite number of universes, I’ve had an infinite number of experiences where team members and/or outsource teams need to upload files to clients’ S3 buckets. If you’re just seeing the AWS console for the first time, it’s overwhelming. Even limiting access (in my experience this is lukewarm at best: functional but not visually reflected; users can see what’s set up even if they can’t access them).

Perhaps you’ve seen AWS now offers an SFTP to S3 endpoint service. What you might not know (yet): you’ll spend $0.30 an hour plus $0.04/GB (up and down). While there are plenty of customers that simply don’t care about the cost, there are enough that do, so I went off in search of what I could do with a simple EC2 instance instead.

First, I used a couple of different tutorials: one from WinSCP, and a more in-depth one from Cloud Academy. For my time and effort, the Cloud Academy one was a little more detailed and, IMO, better for a first-time run. From a security standpoint, I prefer Cloud Academy’s use of IAM roles instead of local aws-cli config files.

The core to making this transition work is utilizing the S3FS-FUSE filesystem, which lets you mount an S3 bucket as a local drive. I won’t go into details here, feel free to read up on it if you so desire.

Improving Security

I suggest making a handful of changes to the tutorials above to further secure your FTP server:

  1. don’t install a local FTP server; use FTP over SSH (SFTP); avoid unnecessary services and running processes whenever possible
  2. if you want to be even more secure, use EC2 Security Groups and firewall the server to only necessary users (you could also change the port SSH runs on your server if you feel it necessary)
  3. use key exchanges over SSH instead of passwords (and disable SSH passwords altogether)
  4. when setting up the IAM Policy, limit the access only to your local AWS VPC

How Can I Become a Developer and Make Tons of Money Super Fast?

Spoiler alert: you can’t. You won’t become a developer overnight, and it won’t happen fast.

I’ve seen (and feel like I’ve answered) this question so many times on Quora, that I decided to just write an answer here and link it often. Mostly because I don’t like to write things more than once, but also because I like free traffic.

Everyone…seriously now…this isn’t a fast process. You don’t need to go to college or a coding bootcamp, you can do it totally on your own (but you’re nearly guaranteed to miss some of the more theoretical stuff…this won’t make or break you but may make future jobs unavailable).

Louder, for those in the back…

BECOMING. A. GOOD. DEVELOPER. TAKES. TIME.

Stop listening to the ads that promise you “become a coder in 6 weeks!” Sure, you’ll have some fundamentals, but you won’t be at a high level. Do you believe ads telling you “you can grow 6 inches overnight!,” “lonely wives want to meet you!,” or “lose 30 lbs of fat…and GAIN 30 lbs of muscle…in just 3 weeks!”

Shocker Incoming…

  1. You’re not gonna gain 6 inches…you aren’t getting a longer, thicker, stronger cock and you’re not getting any taller. The world isn’t a Daft Punk song. You can go have painful and expensive surgery to stretch your legs or inject silicon into your lil’ buddy, but that’s about it.
  2. The only lonely wife who wants to meet you is the one already married to you. If you’re not married, sorry, you’re probably out of luck and will die alone.
  3. Yes, people really do want to believe weight loss and muscle mass gain is this easy (admittedly, I’ve never tried this subject line, but now I lowkey want to…)
  4. You won’t become a developer in six weeks. You’ll get some fundamentals in a bootcamp if you’re self-driven, but most bootcamps are about as educationally rich as Trump University.

If you want to become a developer, you need to put in work.

Before you even start, stop trying to be everything to everyone…it’s a fool’s errand. Feel free to do some research on what you want to do at this point, but you have a few steps before you really get started. I happen to like PHP with Laravel or Symfony for backend work, but you can also use Ruby on Rails, Python, JavaScript via Angular, React, Vue, Svelte/Sapper…or a slew of other options. For web-based development, be sure you understand static content before moving onto dynamic.

A brief note on testing and TDD

You don’t need to test everything, that’s not realistic, but have some test cases for things when you need to confirm they work. Automated testing will save you QA, and if you have to QA your own stuff, you’re in trouble…you know how it’s supposed to work, you know how to use it from your experience, but you are probably not comfortable stress-testing a system.

So how do I start?

  1. Start with a language: HTML (technically a markup language), JavaScript, and CSS if you’re going to do anything web-based; Java if you want to build apps for Android devices (including Chromebooks) and Swift if you want to build for iOS.
  2. Learn how to set up a development environment for your chosen language. Docker isn’t a bad option, but Bitnami or Heroku might be easier. Alternately, HostGator (use code YVOSAVE99 for $9.99 off your first month of hosting and to put money in my coffers) is pretty simple for a webhost and includes SSL/TLS.
  3. Learn the best practices for your chosen language(s)…including the testing framework(s). Learn git (use GitHub, GitLab, Bitbucket, whatever…) and understand why we use version control.
  4. Now that you have some groundwork (and an environment to work in), it’s time to start learning to develop.
  5. Tackle tutorials and learn the fundamentals. Develop some local projects just to learn how to do the work.
  6. This would be a good time to bring a database into the picture: traditional relational, like MariaDB, PostgreSQL, MySQL…or NoSQL like MongoDB. Almost everything today requires a data model and interaction with said model.
  7. Find some popular open source projects you can contribute to, understand what they’re doing, and tackle one or more of the issues. The hope, with more popular projects, is those projects are better moderated and therefore will require code reviews and feedback when you commit a pull request.
  8. Jump into freelancing or go work for someone else to get some team experience and a feel for working with other developers…no one likes a lone wolf (especially other lone wolves).

That’s it. That’s the big secret. Get out there and put in work.

DoCuMenTaTiOn DoEsN’T MatTer

Ho. Lee. Fuck.

Two fucking days. Two mother-fucking shit-eating stupid-waste-of-time days…

Technically, the last couple of hours of one day and all of another.

Lost. Gone to the great timekeeper in the sky.

There’s something about “assume” and “making an ass out of ‘u’ and me…”

Where does this come from? Thanks for asking! Thomas Bradshaw at Atlassian provided the most helpful response I’ve seen in two days’ research (I’m not the OP). A huge thank-you to Thomas for that information. Usually, I find myself learning as I go, often with no internal support for anything new that needs to happen. I’m a lone wolf by circumstance, not by choice per se.

I started writing this as a rant against substandard documentation. The developers or technical writers could have (and probably should have) written clearer documentation. I might not have lost two days otherwise. However, reading through some of the code examples again for this post, I realized I glossed over content. Nothing like a mouthful of crow. Mmmm.

Now I don’t get to complain about the documentation, developers, or technical writers…

Instead, I have to express my frustration at myself. I read through the great walkthrough GitConnected provides. Somehow, I repeatedly miss the reference to executing the zip command from the bitbucket-pipelines.yml file.

As important as writing good documentation is, reading comprehension matters. Take the time, when learning something new, to slow down a bit. Often, you can find the extra information you need in code samples. Take the time to read. You might surprise yourself.

That aside…

Developers and technical writers, do us all (including yourselves) a favor…

Write better documentation.

I know, I know…I don’t like to, either. But together we can make the world a better place…or whatever tripe the tech bros spew these days.

Seriously. Take that extra little bit of time.

Now get out there, and make me fucking proud.

Dear Recruiters…

It’s not you, it’s me. I’m an asshole…I know this. I come from a long line of assholes: my father before me, his before him, and I’m sure all down the line back to my roots in the frozen wasteland of Northern Europe.

Look, I know you have to meet goals. I understand you have to make sure I and my kindred spirits have the skills you need.

But let’s be honest for a second here. This is a terrible sales strategy.

I’m going to let you in on a little secret…

Come closer…

Closer…

Whoa, whoa, whoa, too close! Personal space is a thing, man!

But I digress. I don’t give a fuck what you need. It doesn’t make any difference to me; either I can do the work (and I’ll work with you to move forward) or I can’t/don’t want to, and I won’t.

Do you know (or want to know) what does matter to me?

Four questions. That’s all I care about:

  1. What are you planning to pay me? Let’s be honest, if we can’t meet here we’re both wasting our time. Help me help you. If you can’t meet my threshold, that’s totally cool, no harm, no foul. If I charge more than you can or want to pay, I get it. I’m not for everyone (just read this blog and you’ll know that). Let’s save time if we’re not gonna meet on this point, letting you get in more calls (and hopefully more commission…I’m all about everyone coming out ahead financially). I’m not stuck on an hourly wage, either…let’s build win-win-win situations, so you, the client, and I all feel like we’re fairly compensated and get what we pay for (I pay, too: in time, not money).

  2. Can I work remote? I’ve spent a lot of time commuting in my life. I’d really rather not.

  3. Can I set my own hours? I’ve spent a lot of time getting up at 5 AM to commute to an 8 AM job. Some days I kill it waking up at 6 AM and some days I need a couple extra hours’ sleep, and kill it til 2 in the morning. I’m inconsistent, and while that’ll probably come home to roost in 20 years, in the meantime, it’s what works for me.

  4. Can we set this up as a corp-to-corp? This makes my accountant and lawyer happier. Corporations carry liability insurance. Everyone is protected, everyone wins.

We all want the same thing…a fleet of exotics in our driveway and lining the street. Let’s get out there and get that horsepower!

Surprise! Private IP’s Can Move…

The AWS adventures continue…

I’ll keep this short and sweet, because it was almost life-changing and I’m tired of ranting. Something something be more positive something attitude affects everything something something eat a dick.

I noticed, previously, when adding additional private IP’s to an AWS instance, the input field has a placeholder value of Auto-assign. I never gave it much thought until my EC2 instance stopped connecting to the Internet yesterday. Maybe I can write another post on that later. Hours pass. Attempts to fix an issue affecting updates and certbot fail miserably. I rebuild the damn thing this morning; hence the lesson…

I debate and decide to test the merits of a separate standalone Network Interface. I’m certain, out there somewhere, in the deathlike silence and chill of a datacenter, a Sysadmin or DevOps Engineer knows the answer. I’m not that wizard, and the Interface idea fails spectacularly when binding those private IP’s to the EC2 instance. You might think I’d learn to test one or two IP’s instead of rewriting the whole damn netplan file, but apparently you don’t know me. So, the separate Network Interface idea dies in solitary.

At this point, I’m resigned to rewriting the netplan YAML again (that would be the second time in one day). “What the fuck, I might as well try to move these private IP’s and save myself an hour of work after spending 20 hours trying to fix it.”

Lo and behold, it worked. The heavens parted and the networking gods smiled on me. The fact you’re reading this is proof they haven’t shit on me since (at least in this case).

Slaying the Dragon

or “How I Figured the Fucking IP’s Out and Learned to Love netplan

I suppose it’s that time of year again, where I write one of my two annual posts. Some years I only get through one. I actually enjoy writing, I just never seem to find time to do it. Oh, well, as the title implies, and the subject line explicitly states, I finally figured out that pain-in-the-ass (PITA) netplan. If you’re curious where this started, take a look (just don’t forget to come back)!

The problem never changed, I ended up trying to worm myself out of facing this enemy again with a number of t3.nano instances instead of one m5.medium instance. Unfortunately (or so I thoughtlittle did I know…), some of the WordPress and other database-heavy code stubbornly refused to run on a t3.nano. By the time I had eight t3.nanos, four t3.micros, a t2.micro, and a partridge in a pear tree, I opted to switch to a bigger server and put everything on it.

That meant one of two options: learn netplan (shudder) or use Ubuntu 16.04 and go through the joy of Canonical’s lack of support for an old OS (read: having to pull in additional repos to build PHP 7.x). I’ll admit…I started with 16.04. It didn’t take long to piss me off, though, and I “embraced the suck” (as High Overlord Pelosi would like all citizens…er, except Congress…to do).

Much to my surprise, it came way easier this time… (lucky “it”…S/O Rob, Anybody, and Dawn)…

The first thing I had to do (which I could swear I tried last time) was create a separate YAML file for storing the network data…and call it first (the base file is called 50-cloud-init.yaml, I named mine 01-multi-ips.yaml). Hold up a sec there, cowboy…the first thing I did was build an AMI out of my server so I could spin everything back up if I fucked everything to hell again.

So, technically the second was that new file thing.

I kept the names the default file did.

The Original File

# /etc/netplan/50-cloud-init.yaml

# a bunch of stuff Ubuntu adds to remind you this file
# is auto-generated and might wipe your changes if you
# save them here
network:
version: 2
ethernets:
ens9:
dhcp4: true
match:
macaddress: HA:HA:IM:NO:TR:EA:L0
set-name: ens9

Bright and Shiny (the new config file)

I’m only using a couple IP’s for an example, but I ended up binding whatever the maximum is. But before you take a look, keep a few things in mind:

  1. 172.172.172.1 is the primary private IP on this imaginary instance
  2. 172.170.180.3 is the first additional private IP on this instance
  3. ens9 is the name of the network device from the previous file; you might see a different name, and if you do, use the name you see in your cloud config file
  4. the cloud config file might have a different name, but you should only have one or two files max in that directory
  5. back your shit up before you fuck about with IP’s and network interfaces
  6. I’m fairly sure this is spaced correctly, but it’s a good idea to get a YAML Linter or otherwise validate your YAML before running the command to kick off the IP update or restarting your server
# /etc/netplan/01-multi-ips.yaml

# DISCLAIMER: Fuck shit up at your
# own risk. Back your shit up. Wash
# your hands. Don't be a dick.

network:
version: 2
renderer: networkd
ethernets:
# surprise, motherfucker! this is the same as the one above
ens9:
# you need to list every private IP address
# that you want netplan to recognize here
addresses:
- 172.172.172.1/20
- 172.170.180.3/20
# and so on...
dhcp4: no
# now for the real fun...map the routes
routes:
- to: 0.0.0.0/0
via: 172.172.172.1
table: 1000
- to: 172.172.172.1
via: 0.0.0.0
table: 1000
- to: 172.170.180.3
via: 0.0.0.0
table: 1000
routing-policy:
- from: 172.172.172.1
- from: 172.170.180.3

I really wish I could explain what everything here is, but I can’t, but the good news is…Google exists…and it’s your friend for finding answers on StackOverflow.

Just one two more steps…

Run sudo netplan apply –debug

I’m a fan of running –debug because it will warn you if something is awry before you fuck your connection to hell. You’ll still have a chance to delete or comment out or rename your monster before things go south.

Finally, bind your Elastic IP’s to your private IP’s and you’ll see things work. ifconfig will not look right (neither will sudo netplan ip leases ens9)

AWS Private IP’s on Ubuntu 18.04 with Netplan

Ugh. Uggggggggghhhhhhhhhhhhh. Ffffffffffffffffffffffffffffffffffff. Twelve hours to get this damn thing right. OK, deep breath. Use Google, ask sys admin friends, LINT your YAML. Spin up alternate instances because you mis-configured Netplan and it took the server offline. Realize your snapshot was designed too late and you’re totally fucked because of course the dev team doesn’t have backups. Stop what you’re doing because a domain expired on an app that has traction and fuck how did I miss that? Been a ball-busting week. Live and learn. Wear a Thai steel cup next time, discomfort aside, the protection is second to none.

Sigh heavily. Have a Lagavulin (or 3). Smoke half a box of Davidoff cigarillos…get wired for sound…that’s a lot of nicotine for someone who has one cigar a week, two at most. Don’t think about how much was spent between the scotch and cigarillos, or how quickly you consumed either one of them. Deep breath (again). Yoga…just kidding, another scotch and two more cigarillos. Three more and this pack is done…can’t just run down to the nearest convenience store…fuck, the closest tobacconist that will have something worth smoking is in Sacramento and that’s an hour round trip if traffic is light…fuck, it’s Saturday, traffic won’t be light…better manage these last two cigarillos. Hey, here’s an idea, write a blog post, haven’t done that in a while…hmm…but what to write about? Dammit, this server needs to be ready yesterday. System administrators understand the pain but customers don’t, especially when the design team dragged ass and turned in half-assed work. Fuck, cost a design client, but can’t leave them hanging, that’s awful customer service. There’s too little care about customers today as-is, oh no, the customer hurt my feelings or wrote some awful shit about me. Fuck, time to scrub Yelp…I mean, ask a customer politely to change their review. Fuck it, they’ve never been good with password management, won’t remember what they wrote, and don’t have time to go back and look at their past reviews. Sure, some people might, but that customer won’t. Where were we? That’s right, shitty customer service. Won’t provide bad customer service, that’s bad business. ipconfig is deprecated in favor of ip, and that YAML config is a bitch dealing with multiple IP’s…it isn’t supposed to be, but for whatever reason it doesn’t want to take. vi keeps enforcing its own formatting on the YAML, which is pretty awful when you consider the importance of formatting in YAML…like Python. Netplan has extensive documentation, and even some great examples of config files.

Thought about sharing the first iteration of the config file (set routes at your own risk…broke the server twice trying that shit), but that seems pointless. No one wants to see what doesn’t work. You wouldn’t have bothered reading this far if you did. IP’s have been changed to protect the innocent…

They say there’s no atheist in a foxhole, but there’s no atheist when the network connection keeps failing and there’s no backup, either. AWS shows terminal screenshots, if only they had some way to virtually connect locally rather than via SSH… Oh, look, the public IP changed, maybe it will work as soon as I update PuTTY…fuck. Let’s add another private IP and an Elastic IP to that, and maybe that will work…fuck. No connection, the server is fucked. I’m fucked.

Conversation with a friend with AWS Certification. Suggests shutting down the instance, spinning up a new one, and mounting the volume to the new instance. Done this with VM’s dozens of times. Guess sometimes we’re just too close to the problem to see a solution. Fuck, I should probably eat while this volume is detaching from the original and attaching to the new one. Alcohol has calories, that counts, right? Half a bottle. Fuck me. So much for going and grabbing a bite somewhere, not that I really have the time, but shit it would be nice to get out of the house today. Nope, back to the grind…12 hours in and counting. Damn, JoyRun was great, too bad as soon as you move out of a college town it doesn’t work anymore. Maybe Postmates delivers? Ha! Not here. The only pizza place still open is Domino’s…”pizza”

I need to know more local people, convince someone to do a food run for me while I’m grinding through this damn server. OK. Deep breath, the first sign of life. The volume mounted and I was able to edit the YAML file, get rid of the offending code (it was an attempt to separate the addresses into ifconfig-style ethX:format):

# don't do this
# for real
# it's gonna take down your fuckin' server
# some other YAML config
    eth1: # this is fine
        addresses: # still fine
            - 10.0.0.5 # yup, still fine
            - 10.0.0.6 # still
    eth1s0: # woah, nellie
       addresses: # doesn't matter, the previous line already broke shit

This works if the system has an interface eth1s0; if not, the IP configuration gets fucked. Yes, fucked proper. No clue why people keep asking.

Time to knuckle down and learn the ip tool…

ubuntu:~$ sudo ip -c a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:07:aa:ff:f9:36 brd ff:ff:ff:ff:ff:ff
    inet 10.0.16.1/20 brd 10.0.31.255 scope global dynamic ens5
       valid_lft 2195sec preferred_lft 2195sec
    inet6 fe80::7:aaff:feff:f936/64 scope link
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:44:ac:46:58:e0 brd ff:ff:ff:ff:ff:ff

Do you see what I see? Further, why did it take me twelve fucking hours to see this???  Chalk it up to the aging brain not learning new things nearly as quickly as it used to. Look… ens5 shows “UP,LOWER_UP” while ens6 does not. The good news is, Ubuntu recognizes the new interface and wants to work with it. Fresh Ubuntu 18.04 install (on what I think is the seventh instance in the last 90 minutes); I had not touched the network configuration on this system…yet. Fortunately, that disconnected volume saved my bacon…keep backups, kids. Take snapshots before you muck with network configurations. I know better and didn’t until it was too late.

No one ever tells you how much time sysadmins spend waiting on shit. Roughly a third of my day has been waiting at this point, maybe more. Word to the wise, netplan reads config from lowest number to highest; if you set a configuration in one of the lower-numbered files, netplan will not override that value with a higher-number file. I set my extra private IP config file to 01 and nothing would override it at that point.

Configuration is in, and I run netplan apply…

Config File:

# /etc/netplan/60-extra-privates.yaml # yup, I'm 12

network:
    version: 2
    renderer: networkd
    ethernets:
        ens6:
            dhcp4: off
            dhcp6: on
            addresses:
                - 10.31.21.174/32
                - 10.31.21.44/32
                - 10.31.21.184/32
                - 10.31.21.6/32
                - 10.31.21.22/32
                - 10.31.21.150/32
                - 10.31.21.212/32
                - 10.31.21.196/32
                - 10.31.21.99/32
                - 10.31.21.157/32
            match:
                macaddress: 02:44:ac:46:58:e0
            set-name: ens6

Rerun ip show…

ubuntu:~$ sudo ip -c a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:07:aa:ff:f9:36 brd ff:ff:ff:ff:ff:ff
    inet 10.0.16.1/20 brd 10.0.31.255 scope global dynamic ens5
       valid_lft 3596sec preferred_lft 3596sec
    inet6 fe80::7:aaff:feff:f936/64 scope link
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:44:ac:46:58:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.31.21.174/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.44/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.184/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.6/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.22/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.150/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.212/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.196/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.99/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet 10.31.21.157/32 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::44:acff:fe46:58e0/64 scope link
       valid_lft forever preferred_lft forever

Looks a little better, at least ens6 is UP now…

IP addresses are all showing, that’s a good sign. ifconfig gives a different response, but we don’t have to worry about that anymore…we’re ip users now!

But…(there’s always a but)…going to the public IP attached to that private IP still comes up with nothing. Time to run through the ol’ checklist…

  1. Is NGINX running? If so, better catch it…if not, restart it.
  2. Is PHP ok? Is PHP-FPM running and is it the right version? (7.2 as of this writing)
  3. Is NGINX configured properly?
  4. Is each site configured to the appropriate private IP address?

Well, shit. I’ve never hoped I fucked up an NGINX config file so badly before.

Day two; already two hours in. Strong black coffee and an apple…should have eaten more. Should have added Tullamore Dew to the coffee.

Finally came across Thomas Ward’s answer on AskUbuntu which says sub-interfaces aren’t possible (Netplan supports multiple IP’s on a single interface). Has this all been a snipe hunt? In all fairness, Netplan’s documentation says so, had I only read those examples closer…FML (and also, blocking all connections to the server with my virtual interface pretty much taught me as much…but at least now we have different ways of phrasing the question). According to user slangasek on AskUbuntu, multiple IP’s on a single interface is considered best practice (as compared to virtualizing additional interfaces). Clearly, I’m not a networking geek and don’t keep up (I’m much more Dev than Ops). But hey, what more can we ask than constantly improving and learning new things? Sure, ifconfig no longer ships with the most recent Ubuntu, but I could have rolled back to 16.04 or even installed ifconfig and overrode the configuration, telling Ubuntu to return to the Old Ways. No one likes change, but no wise man denies its inevitability (disclaimer: I’m no wise man). I’m trusting slangasek on this; I lack the time to research networking best practices, and Netplan is favored over ifconfig on Ubuntu 18.04, so good enough for me. Forward we march toward insanity greatness!

Tested switching ens6 to dhcp4 = true…bad idea. Detaching the network interface saved my bacon. Beats spawning yet another instance. FWIW, you might see examples setting dhcp4 to either on/off or true/false…both work. Back to the drawing board. Don’t need bridging, the IP’s attach direct to the interface and already show up with the ip -c a command. Sigh.

While I’m thinking about it, here’s a couple aliases I add to make jumping screens easier (screen is a lifesaver, especially when the server disconnects you):

# ~/.bashrc

# ... lots of stuff here...
# somewhere around line 95...
# screen aliases
alias s='screen -x'
alias ss='screen -S'

Hey, maybe it’s a firewall issue…

ubuntu:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

iptables is clear (makes perfect sense, we rely on the AWS firewall instead). AWS firewall looks good, allows connections on ports 80 and 443 (NGINX handles the redirects from 80 to 443; can’t trust users to automatically type in https yet…if ever).

Two hours later and I realize the Elastic IP’s are bound to the VPC, not the EC2 instance itself (the new way on AWS). FML. Time to figure this shit out…and learn the VPC has no default access to the outside world (which seems odd seeing as I can hit the AWS-set public IP, but maybe that’s bound to the EC2 instance and not the VPC; I just don’t know). Elastic IP’s already set up means I don’t have to worry about assigning public IP’s (seems obvious).

Time to enable Flow Logs via CloudWatch. “Good” news…the VPC blocks the connections. OK, time to look at the ACL…everything open on 0.0.0.0\0…sigh. Time to look at the Internet Gateway again.

Checklist time…

  1. Does the VPC have an Internet Gateway? Yuuup.
  2. Does the subnet route table point to the Internet Gateway? Yuuuuuup.
  3. Do the instances in the subnet have globally unique IP addresses? Yuuuuuuuuup.
  4. Do the network access control and security group rules allow relevant traffic to flow to and from the instance? It sure looks like it, but CloudWatch tells a different story…
  5. If I visit one of the Elastic IP’s connected to the interface from my browser, does it work? NOPE. FML.
  6. Maybe it’s just me? Not according to “Down for Everyone or Just Me?”

So. Reviewing CloudWatch shows a request to the Elastic IP pointed to the Private IP pointed to the correct EC2 instance was Rejected. OK, back to the old drawing board, time to figure out where the rejection occurs (and why). For a second there, I thought I was losing my mind and couldn’t read the log, but it turns out I was correct. I’d like to say I’m only wrong when I think I’m wrong, but this post proves otherwise.

OK, the logs show an ACCEPT on 80 and 22, at least for the Network Interface in the current subnet. So the Elastic IP’s pass to the Private IP’s no problem. Good to know. If you ever need a reminder that your servers are under constant attack from everywhere, record REJECT VPC Flow Logs to CloudWatch.

Maybe that’s a breakthrough… 

When looking at the NGINX default configuration page, the page loads just fine. Every Elastic IP on that network interface loads just fine…with the default page. It looks like NGINX doesn’t recognize the local IP…two steps forward and 47 back. Can’t find anything on Google about NGINX seeing (or not seeing) IP’s defined in Netplan. Almost every example shows a config that doesn’t specify an IP. Back to conversing with my AWS Certified friend…suggests I bind the domain to the directory (the server_name directive, which I did) and don’t worry about defining the IP on the NGINX config. Sometimes we’re too close to the problem to see a clear solution.

Well. NGINX is ignoring every private IP I’ve got except for the primary one (they all come up when running ip -c a, so I know Ubuntu sees them).

The most frustrating thing…none of this is uncommon and shouldn’t be difficult. AWS supports multiple IP’s on a Network Interface for easy swapping when EC2 instances have problems.

You came here to understand how to assign multiple private IP’s to a single AWS instance using Netplan, and we all got derailed. However, I think it’s safe to say I made a bunch of mistakes so you don’t have to.

If you’re anything like me, you were really hoping for a happy ending to this story. There is one, in the sense that the server is finally back up and churning out happy little 1’s and 0’s. However, I “fixed” it by rolling back to 16.04 and using ifconfig. I’m unsure NGINX has any way to work with the setup I had. I’ve talked with experts, done countless Google searches, and finally said fuck it…shit has to work and it has to work on a deadline. In a perfect world, I’d have had the time to delve into the NGINX and Netplan documentation headfirst and resolve the issue (assuming it can be resolved within the limitations of both). I might have used Apache instead, or done any number of things.

Unfortunately, that’s not how it went. But from a delivery perspective, the customer is happy with his robust new server and I no longer have to lose a moment of sleep on this roller coaster I spent the last two days on.

Happy travels!

Let’s Talk Freelancing

Freelancing. It’s kind of an ugly word, really, like outsourcing. Of course, managers love outsourcing because they can save money (usually it costs more in the long run, but who has time to notice that?).

Getting Burned

How many times have you done freelance work, only to have your “client” burn you? What options do you have (personally, I like contracts and collection agencies). You know who does not like collection agencies? Florida (and Texas, but much more Florida). Collection agents do not collect from Florida because they have little to no legal weight, and cannot get the money anyway. I also like escrow services, I believe the investment (especially with new clients) pays off.

Freelancing Sites

I have good and bad experiences with some of the freelancing websites currently available. The only invite UpWork recruiters sent me was an Excel file with a Trojan (that practice of not opening random files from the web comes in handy). UpWork sent confirmation of the Trojan a couple days after the invites went out. I have good experiences hiring from eLance (the precursor to UpWork), but not getting work through them.

Outsource offers lots of opportunity, but also lots of people looking for super-cheap work. You do pay for credits, but at least they carry over if you do not find anything. Outsource also does a great job of refunding if the client does not view your offer within a week. I’ve pulled in about $5k across two jobs with Outsource. However, I also have had a couple of nearly-insulting offers to work for anywhere from half to 10% of what I normally charge, with clients expecting me to thank them for it.

I have not had the opportunity to explore Moonlighting yet, but it looks like an interesting opportunity…when I can try it out, I can report back.

Finally, I recently came across Toptal Web Freelancers Network, and honestly, I find this one the most interesting. First, developers must go beyond an application and through a screening process. I hope this translates to clients who have no qualms about spending money. After all, this blog isn’t putting a gated Murci in my driveway… 😉

I’m looking forward to the interview process (I apologize in advance, I cannot share anything that goes on therein).

All the best in your endeavors!