Route 53: How To Alias Application Load Balancers

Hello!

This is a simple one but I kept getting stuck trying to figure it out. My brain was blocked on it. I’m sharing the pattern here in case you had the same problem.

All I needed was a Route 53 Hosted Zone with an alias record for an Application Load Balancer. I needed these defined in a CloudFormation template. Here’s how to do it:

---
AWSTemplateFormatVersion: '2010-09-09'
 
Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
  Subnets:
    Type: List<AWS::EC2::Subnet::Id>
 
Resources:
  HostedZone:
    Type: AWS::Route53::HostedZone
    Properties:
      Name: demo-zone.internal
      VPCs:
        - VPCId: !Ref VpcId
          VPCRegion: !Ref 'AWS::Region'
 
  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      IpAddressType: ipv4
      Name: demo-lb
      Scheme: internal
      Subnets: !Ref Subnets
 
  LoadBalancerAlias:
    Type: AWS::Route53::RecordSet
    Properties:
      AliasTarget:
        DNSName: !GetAtt LoadBalancer.DNSName
        HostedZoneId: !GetAtt LoadBalancer.CanonicalHostedZoneID
      HostedZoneId: !Ref HostedZone
      Name: friendly-name.demo-zone.internal
      Type: A

These were two details that got me.

First, you need a different HostedZoneId in each of two places:

  • In the AliasTarget of the record. This is not the ID of the zone where you’re creating the record. All ALBs automatically get a DNS name. Like this: internal-demo-lb-XXXXXXXXXX.us-west-2.elb.amazonaws.com. As far as I understand, you need the ID of the zone where that automatic record lives. AWS manages that zone, so it won’t appear anywhere in your infrastructure. You get its ID from a property on the ALB resource: !GetAtt LoadBalancer.CanonicalHostedZoneID.
  • In the root Properties of the record. This is the ID of the zone where you’re creating the record.

Second, you need an A record (type), not a CNAME record.

Route 53 alias records are an AWS-specific technology, but they’re still aliases. CNAMEs are the native DNS aliases, so I expected Route 53 aliases to be an extension of that type. Nope! Aliases of ALBs are A records.

I think the detail is that aliases point directly to the IP addresses of the load balancer, there’s no chained DNS resolution like there is with CNAMEs. That makes them effectively magic A records. The magic is that AWS keeps them up to date with the dynamically changing IPs of load balancers.

Happy automating!

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

Securing AWS Security Groups: Restricting Egress Rules

Today’s article demonstrates a surprisingly easy way to tighten the network-layer permissions in an AWS VPC. If you’re in AWS but you’re not in a VPC, I recommend migrating.

Security Groups have ingress and egress rules (also called inbound and outbound rules). In most SGs, the egress rules allow all traffic to everywhere. You’ve probably seen this:

AllowAllEgress

That’s a problem because, someday, you will get hacked. Breaches are inevitable, perfect security doesn’t exist. Someone or some bot will get access to what that SG was protecting (EC2 instance, Fargate ENI, whatever). When that happens, they can send out anything they want. Instead, you want them to have the most limited capabilities possible. They should find walls everywhere they turn. It’s not just about what’s coming in, it’s also about what’s going out.

I like AWS’s directive from the third bullet of the Security Pillar in their Well-Architected Framework: “Apply security at all layers”. Incoming and outgoing.

Fortunately, SG outbound rules are easy to tighten!

Imagine an Autoscaling Group of Linux instances. They handle background jobs and sometimes engineers SSH into them to run diagnostics (for this example we’re imagining you haven’t set up SSM Session Manager). At boot time they yum install some packages.

As usual, you need an incoming rule to allow SSH:

AllowSSHIncoming

Now, here’s how to determine the outgoing rules you need: if the resource protected by the SG starts the connection, you need an outgoing rule. If it only replies to connections started by someone else, you don’t need an outgoing rule. Details farther down.

For anyone who’s forgotten, yum uses HTTP/S (ports 80/443) and FTP (ports 20/21).

The instance receives and then replies to SSH requests. That means it didn’t start those connections, so we don’t need an outgoing rule. But, it sends HTTP/S and FTP requests because it has to request to download packages from the yum servers. That means it starts those connections, so we need outgoing rules:

AllowYumOutgoing.png

That’s it. If you attach this SG to the instances, engineers will still be able to SSH into it and it will still be able to yum install packages. But, if the instance tries to do something else, like maybe connect to a MySQL database (port 3306), the SG will block that traffic and the connection will time out. When something Evil breaks into this instance, it won’t be able to access that database.

These three rules are enough because Security Groups are stateful. To dramatically simplify statefulness, it means that SGs know whether traffic passing through them is part of a connection the instance has already agreed to. If it is, they pass the traffic whether or not a rule is present.

I’m skipping a ton of details. This is meant to help you quickly do one round of tightening on your network. There are advantages and disadvantages to stateful filtering, and the details can take you deep into the weeds, but most of the time it’s enough to know what rules you need. If you want to go farther with this on your own, check out this project for a demonstration environment where you can experiment with variations.

Two details:

  • We have to allow yum traffic to 0.0.0.0/0 because we don’t know the IP addresses of the upstream yum servers.
  • VPC subnet ACLs are not stateful, so you need different rules for those. I’ll cover that in another article.

In this example we weren’t able to stop whatever Evil Thing had broken into your instance from sending your Super Secret Stuff to some Evil Webserver somewhere, but we did take away some of their other tools. We’ve put up one more wall they might run in to, and every wall they hit might stop them. Walls at every turn.

Stay safe!

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

Beating AWS Security Groups

Today I’ll show you how to pass traffic through an AWS Security Group that’s configured not to allow that traffic.

This isn’t esoteric hacking, it’s a detail in the difference between config and state that’s easy to miss when you’re operating an infrastructure.

Like I showed in a previous post, AWS Security Groups are stateful. They know the difference between the first packet of a new connection and packets that are part of connections that are already established.

This statefulness is why you can let host A SSH to host B just by allowing outgoing SSH on A’s SG and incoming SSH on B’s SG. B doesn’t need to allow outgoing SSH because it knows the return traffic is part of a connection that was already allowed. Similarly for A and incoming SSH.

Here’s the detail of today’s post: if the Security Group sees traffic as part of an established connection, it’ll allow it even if its rules say not to. Ok now let’s break a Security Group.

The Lab

Two hosts, testa and testb. One SG for each, both allowing all outgoing traffic. Testb’s SG allows incoming TCP on port 4321 (a random ephemeral port I’m using for this test):

TrafficAllowed

To test traffic flow, I’m going to use nc. It’s a common Linux utility that sends and receives TCP traffic:

  • Listen: nc -l [port]
  • Send: nc [host] [port]

Test Steps:

(screenshots of shell output below)

  1. Listen on port 4321 on testb.
  2. Start a connection from testa to port 4321 on testb.
  3. Send a message. It’s delivered, as expected.
  4. Remove testb’s SG rule allowing port 4321:TrafficDenied
  5. Send another message through the connection. It will get through! There’s no rule to allow it, but it still gets through.

WAT.

To show nothing else was going on, let’s redo the test with the security group as it is now (no rule allowing 4321).

  1. Quit nc on testa to close the connection. You’ll see it also close on testb.
  2. Listen on port 4321 on testb.
  3. Start a connection from tests a to port 4321 on testb.
  4. Send a message. Not delivered. This time there was no established connection so the traffic was compared to the SGs rules. There was no rule to allow it, so it was denied.

Testb Output

(where we listened)

testb

Only two messages got through.

Testa Output

(where we sent)

testa

We sent three messages. The last two were sent while the SG had the same rules, but the first message was allowed and the second was denied.

Beware!

The rules in Security Groups don’t apply to open (established) TCP connections. If you need to ensure traffic isn’t flowing between two instances you can’t just remove rules from your SGs. You have to close all open connections.

Happy securing,

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

AWS Security Groups: Stateful Statelessness

Hello!

Recently, I rediscovered a fiddly networking detail: although ICMP’s ping is stateless, AWS security groups will pass return ping traffic even when only one direction is defined in their rules. I wanted to see this in action, so I built a lab.

Background

Network hosts using stateful protocols (like TCP) distinguish between packets that are part of an established connection and packets that are new. For example, when I SSH (which runs on TCP) from A to B:

  1. A asks B to start a new connection.
  2. B agrees.
  3. A and B exchange bunches of packets that are part of the connection they agreed to.
  4. A and B agree to close the connection.

There’s a difference between a new packet and a packet that’s part of an ongoing connection. That means the connection, and its packets, have state (e.g. new vs established). Stateful firewalls (vs stateless) are aware of this:

  1. A ask B to start a new connection.
  2. Firewalls in between allow these packets if there is an explicit rule allowing traffic from A to B.
  3. A and B exchange bunches of packets.
  4. Firewalls in between allow the packets from A to B because of the explicit rule above. However, they allow the return traffic from B to A even if there is no explicit rule to allow it. Since B agreed to the connection the firewall assumes that packets in that connection should be allowed.

This is why you only need an outgoing rule on A’s Security Group (SG) and an incoming rule on B’s Security Group to SSH from A to B. AWS SGs are stateful, and allow the return traffic implicitly.

Ok, here’s the gnarly bit. ICMP (the protocol behind ping) is stateless. Hosts don’t have a negotiation phase where the agree to establish a connection. They just send packets and hope. So, doesn’t that mean I need to write explicit firewall rules in the SGs to allow the return traffic? If the firewall can’t see the state of the connection, it won’t be able to implicitly figure out to allow that traffic, right?

Nope, they infer state based on timeouts and packet types. ICMP pings are ECHO requests answered by ECHO replies. If the SG has seen a request within the timeout, it makes the educated guess that replies are essentially part of “established” connections and allows them. This is what I wanted to see in action.

The Lab

I setup a VPC with two hosts, A (10.0.1.70) and B (10.0.2.35). They’re in different subnets but the ACLs allow all traffic so they don’t influence the test. Here are the SG rules for A (10.0.0.0/16 covers the entire VPC):

test_a_inbound
test_a_outbound

And the rules for B:

test_b_inbound
test_b_outbound

A allows outgoing ICMP to B, and B allows incoming ICMP from A. The return traffic is not allowed by any rules.

The Test Script

I didn’t find a way to send just replies without requests in Linux, so I bodged together a Python script:

"""
This is a stripped-down version of ping that allows you to send a reply without responding a request. This was needed
to test the details of how security groups handle state with ICMP traffic. You shouldn't use this for normal
pings.
 
The ping implementation was based on Samuel Stauffer's python-ping: https://github.com/samuel/python-ping (which only
works with Python 2).
 
This must be run as root.
You must tell the Linux kernel to ignore ICMP before you run this or it'll eat some of the traffic:
    echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all
"""
 
import argparse, socket, struct, time
 
def get_arguments():
    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    parser.add_argument('--send-request', metavar='IP_ADDRESS', type=str, help='IP address to send ECHO request.')
    parser.add_argument('--receive', action='store_true', help='Wait for a reply.')
    parser.add_argument('--send-reply', metavar='IP_ADDRESS', type=str, help='IP address to send ECHO reply.')
    return parser.parse_args()
 
def receive(my_socket):
    while True:
        recPacket, addr = my_socket.recvfrom(1024)
        icmpHeader = recPacket[20:28]
        icmp_type, code, checksum, packetID, sequence = struct.unpack("bbHHh", icmpHeader)
        print('Received type {}.'.format(icmp_type))
 
def ping(my_socket, dest_addr, icmp_type):
    dest_addr = socket.gethostbyname(dest_addr)
    bytesInDouble = struct.calcsize("d")
    data = (192 - bytesInDouble) * "Q"
    data = struct.pack("d", time.time()) + data
    dummy_checksum = 1 & 0xffff
    dummy_id = 1 & 0xFFFF
    # Header is type (8), code (8), checksum (16), id (16), sequence (16)
    header = struct.pack("bbHHh", icmp_type, 0, socket.htons(dummy_checksum), dummy_id, 1)
    packet = header + data
    my_socket.sendto(packet, (dest_addr, 1))
 
if __name__ == '__main__':
    args = get_arguments()
    icmp = socket.getprotobyname("icmp")
    my_socket = socket.socket(socket.AF_INET, socket.SOCK_RAW, icmp)
    if args.send_request:
        ping(my_socket, args.send_request, icmp_type=8)  # Type 8 is ECHO request.
    if args.receive:
        receive(my_socket)
    if args.send_reply:
        ping(my_socket, args.send_reply, icmp_type=0)  # Type 0 is ECHO reply.

You can skip reading the code, the important thing is that we can individually choose to listen for packets, send ECHO requests, or send ECHO replies:

python ping.py --help
usage: ping.py [-h] [--send-request IP_ADDRESS] [--receive]
               [--send-reply IP_ADDRESS]
 
optional arguments:
  -h, --help            show this help message and exit
  --send-request IP_ADDRESS
                        IP address to send ECHO request. (default: None)
  --receive             Wait for a reply. (default: False)
  --send-reply IP_ADDRESS
                        IP address to send ECHO reply. (default: None)

The Experiment

SSH to each host and tell Linux to ignore ICMP traffic so I can use the script to capture it (see docstring in the script above):

sudo su -
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all
exit

Normal Ping

I send a request from A to B and expect the reply from B to A to be allowed. Here’s what happened:

Remember A is 10.0.1.70 and B is 10.0.2.35.

normal_ping_test_a
normal_ping_test_b

Ok, nothing surprising. I sent a request from A to B and started listening on A, a little later I send a reply from B to A and it was allowed. You can do this test with the normal Linux ping command (but not until you tell the kernel to stop ignoring ICMP traffic). This test just validates that my bodged Python actually works.

Reply Only

First we wait a bit. The previous test sent a request from A to B, which started a timer in the SG. Until that timer expires, reply traffic will be allowed. We need to wait for that expiration before this next test is valid.

reply_only_ping_test_a
reply_only_ping_test_b

Boom! I start listening on A, without sending a request. On B I send a reply to A but it never arrives. The Security Group didn’t allow it. This demonstrates that Security Groups are inferring the state of ICMP pings by reading their type.

Other Tests

I also tried a couple other things that I’ll leave to you to reproduce in your own lab if you want to see them.

  • Start out like the normal test. Send a request from A to B and start listening on A. Then send several replies from B to A. They’re all allowed. This shows that the SG isn’t counting to ensure it only allows one reply for each request; if it has seen just one request within the timeout it allows replies even if there are multiple.
  • Edit the script above to set the hardcoded ID to be different on A than it is on B. Then nothing works at all. I’m not actually sure what causes this. Could be that the SG is looking at more than just the type, but it could also be in the kernel or the network drivers or somewhere else I haven’t thought of. If you figure it out, message me!

Conclusion

I had free time over the holidays this year! Realistically, understanding this demo isn’t a priority for doing good work on AWS. I just enjoy unwrapping black boxes to see how the parts move.

Be well!

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles: