pcarleton's blog

TIL: The difference between netstat and ss

Thu, 31 May 2018 00:00:00 +0000

Motivation

In this post, I’m going to go over two tools for inspecting the socket states (netstat and ss), and why to choose one over the other (spoiler: you can really choose either). This is not going to be a 12 ways to inspect socket states article, there are lots of those.

What is a socket?

A socket is a Linux file descriptor for communicating with the network. In Linux, they say everything is a file. In this case, you can treat a socket like a file that writes to the network instead of writing to a disk. Sockets come in different flavors for TCP vs. UDP. For more on sockets, checkout these links:

The Linux Programming Interface (this is really great)
Beej’s Network Programming Guide (not quite TLPI, but available for free)

Why would I care about sockets?

Sockets can be in a bunch of different states (listed in this snippet of ss) which can be useful to answer questions like:

Is my server process actually listening on the port I think it is?
Is it listening on the loopback interface (127.0.0.1), or could someone on my network connect to it (* or 0.0.0.0)? (This can be bad if you’re developing something and running a database that anyone in the coffee shop can make requests to)
What processes are listening on what ports in general?

These questions pertain to sockets that are listening on a host, but there are also various connected states a socket can be in. High Performance Browser Networking’s TCP overview if great for some explanation of the other states (or again TLPI is excellent)

How do I inspect the state of sockets on my machine?

The two tools I’ll cover are the command line tools netstat and ss. I’ll start with where they get their information.

Procfs

The Procfs is a file system that Linux exposes that is like a peek into kernel memory. It lives in /proc and it exposes information about TCP and UDP sockets at /proc/net/tcp and /proc/net/udp. If I cat either of those I’ll get some inscrutable output.

Netlink

The other source of information is called the netlink protocol. In this case you open a socket (a SOCK_RAW with AF_NETLINK as seen here). I can then send requests on that socket for information about other sockets (pretty meta). I have a lot to learn still about netlink, but here are some things I found:

libnldocumentation which is a library for interacting with netlink sockets.
RFC 3594 – Linux Netlink as an IP Services Protocol

Netstat vs. ss

netstat gets its information from /proc/net directly. It parses the file and prints out information based on it.

ss was written more recently to use the netlink API (it will fall back to proc/net if netlink is unavailable). The information in both systems is essentially the same (from what I’ve seen), but here are some arguments for why to use ss

It’s faster (I just read that a lot, I don’t find netstat to be noticeably slower)
Netlink exposes more TCP states (again I mostly look for LISTEN so that’s not a huge selling point)
It has better default argument

None of these are a huge homerun, which is why I expect a lot of people still use netstat. It’s also likely that netstat is installed more places. For instance my Macbook has netstat but not ss.

The default arguments is a little more compelling. netstat by default will try to resolve IP addresses through DNS which really slows it down. It also opens a bunch of new UDP sockets, which might clutter the picture if you’re investigating something. netstat -n stops this behavior, but ss has that on by default (you can use ss -r if you do want the resolution).

One other nice thing about ss is that its source code is much nicer to read!

Open Questions

Here are some things I am still wondering

What are the states that netlink supports that netstat won’t show? Are they states I actually care about?
Where does lsof -i fit in with all of this? Why would I choose that over ss?
What other users are there for netlink?
Can I use netlink to poll for any new UDP connection? (I’ve been wanting to do this to figure out what process is sending UDP packets to a particular IP address, which is easy with TCP but hard with UDP)

New Format: Today I Learned

Wed, 30 May 2018 00:00:00 +0000

I am going to try a new format where I write about something I learned. My goals with this format are:

Increase the amount of writing I do
Increase the frequency of writing
Lower the barrier for making a new post
Document things I’ve learned

My rough goal is to spend:

20 minutes researching / outlining
20 minutes writing
20 minutes editting

We’ll see how this time commitment works out, it may be a little extreme for daily writing. I’m going to try it for the next 3 days and then re-evaluate.

TIL: There are 16 T1 Internet Service Providers

Wed, 30 May 2018 00:00:00 +0000

I’ve been collecting information for a “Practical Guide to Linux Networking” by reading material I can find. Two things I’ve been reading recently are:

A Practical Guide to (Correctly) Troubleshooting with Traceroute
- a presentation by Richard A Steenbergen
Compute Networking: A Top Down Approach
- a textbook by Kurose and Ross

In the traceroute guide, Richard mentions that it’s useful to find the boundaries of your network and that it’s also useful to be able to translate DNS names to ISP’s. It seemed highly unlikely that I would ever be able to do this translation given the examples shown, like:

p16-1-0-0.r21.asbnva01.us.bb.verio.net
ldn-bb2-link.telia.net
tbr2.wswdc.ip.att.net
xe-3-0-0.cr1.nyc3.us.nlayer.net
te2-4.ar5.PAO2.gblx.net

Sure, I can see that there are some words just before the .net, but I don’t have any mapping of what that means. I was imagining a list of international Comcasts that I would never be able to remember. However, “Computer Networking”, they mention that the list of T1 ISP’s is actually pretty small!

Here they are:

Provider	Country
AT&T	USA
CenturyLink (formerly Level3, Global Crossing (gblx) and some others)	USA
Deutsche Telekom AG (ICSS)	Germany
GTT (formerly Tinet & nLayer)	USA
KPN International	Netherlands
Liberty Global	UK
NTT Communications (formerly Verio)	Japan
Orange (OpenTransit)	France
PCCW Global	Hong Kong
Sprint	Japan
Tata Commmunications	India
Telecom Italia Sparkle	Italy
Telxius (subsidiary Telefonica)	Spain
Telia Carrier	Sweden
Verizon Enterprise Solutions (UUNET & XO Communications)	USA
Zayo Group (formerly AboveNet)	USA

Granted, that’s still a lot, but at least it’s tractable. And knowing there’s a small list makes it more tractable to look up when I don’t see one I recognize (rather than google “All names of internet providers, ever…”)

I wanted to find what some of their hostnames looked like. So I tried some random tracerouting starting with the company name .net also trying some variations of former names. This worked pretty well! I also tried in TCP mode with sudo traceroute -T which uncovered a few that eluded the first attempts. Here are the non-exhaustive results.

AT&T

$ traceroute att.net
...
 8  cr2.sffca.ip.att.net (12.122.149.134)  75.585 ms  76.402 ms  75.958 ms
...
10  cr2.dlstx.ip.att.net (12.122.2.81)  78.300 ms  77.370 ms  78.694 ms

We can see we jumped from SF over to Dallas Texas!

CenturyLink

I tried centurylink.com but that didn’t give me anything, so I tried:

$ traceroute level3.net
...
 6  lag-14.ear2.SanJose1.Level3.net (4.68.72.101)  18.185 ms  16.268 ms  17.309 ms
 7  * * *
 8  4.68.88.74 (4.68.88.74)  39.986 ms  41.772 ms  41.428 ms
 9  * * *
10  Level3IsNowCenturylink.com (4.68.80.110)  41.310 ms  41.700 ms  36.501 ms

# Similarly for globalcrossing.com
10  thenewcenturylink.com (4.68.80.110)  43.422 ms  37.603 ms  37.615 ms

Deutsche Telekom AG (ICSS)

$ sudo traceroute -T telekom.com
...
9  m-eb7-i.M.DE.NET.DTAG.DE (217.5.69.14)  180.473 ms  175.974 ms  176.948 ms

GTT Communications

$ traceroute tinet.net
...
 8  xe-7-3-0.cr0-trn3.ip4.gtt.net (89.149.184.162)  180.859 ms  179.917 ms  181.247 ms
 9  it-farm-gw2.ip4.gtt.net (77.67.94.202)  203.231 ms  198.183 ms  203.110 ms

KPN International

$ traceroute kpn.com 
lag-14.ear2.SanJose1.Level3.net (4.68.72.101)  25.916 ms  17.694 ms  18.140 ms
ae-237-3613.edge6.Amsterdam1.Level3.net (4.69.162.242)  158.544 ms  151.896 ms  153.006 ms
IPTRIPLEPLA.edge6.Amsterdam1.Level3.net (212.72.47.250)  157.888 ms  159.041 ms  158.746 ms
* * *
cca-iaas-cr01.net.kpnvdc.nl (145.128.23.193)  159.382 ms  159.334 ms  159.067 ms
145.128.23.242 (145.128.23.242)  159.278 ms  159.481 ms  159.424 ms
apd-iaas-cr01.net.kpnvdc.nl (145.128.13.146)  171.001 ms  165.615 ms  165.130 ms
* * *
static.kpnvdc.nl (145.128.64.11)  160.604 ms  161.201 ms  161.399 ms

Sprint

$ sudo traceroute -T sprint.net
...
18  sl-sprin881-320471-0.sprintlink.net (144.223.33.34)  85.353 ms  84.287 ms  97.713 ms

Open Questions

After digging in to this stuff, I still had some questions.

What is an AS number? And how does it translate to IP CIDR ranges?
Is there a map of T1 endpoints somewhere?
How much does someone pay to send traffic over a T1?

Other resources:

Wikipedia: Tier 1 Network
mtris like traceroute but continuous (I wanted easy copy-pasteable output, so I stuck with traceroute.)
traceroute bad.horse (Just try it)
- Also openssl s_client -connect signed.bad.horse:443 -servername signed.bad.horse < /dev/null

Dumb dig clone in Rust

Mon, 19 Feb 2018 00:00:00 +0000

Why write a DNS client

When I started digging in to DNS, I thought it would be interesting to try implementing a very simple DNS client similar to dig, but using Rust. My goal was to better understand DNS at the protocol level. I wanted to know what bytes were being sent and received to drive the system.

The final result is in this repo). In the rest of the post, I’ll detail what I found out along the way.

The Protocol

The DNS protocol is detailed in Domain Names - Implementation and Specification RFC1035. Section “4.1 MESSAGES - Format” details the bytes of the protocol. Here are the ascii diagrams I found useful:

Message

    +---------------------+
    |        Header       |
    +---------------------+
    |       Question      | the question for the name server
    +---------------------+
    |        Answer       | RRs answering the question
    +---------------------+
    |      Authority      | RRs pointing toward an authority
    +---------------------+
    |      Additional     | RRs holding additional information
    +---------------------+

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      ID                       |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    QDCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ANCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    NSCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ARCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Question Section

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                     QNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QTYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QCLASS                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Resource Record (answer)

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                                               /
    /                      NAME                     /
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     CLASS                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TTL                      |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                   RDLENGTH                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--|
    /                     RDATA                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Implementation

I started out by “hand” building a DNS query for google.com with hard-coded values to ensure my tool was working. Once I had this working, I started taking input from the command line.

I am relatively new to Rust, so I was initially trying to put everying on the stack and have nothign on the heap. While a nice idea in theory, accepting input from the command line, or parsing dynamically sized elements from a server response make this pretty impractical. I considered having a really large array and saying anything above that size wasn’t supported, but ergonomically this didn’t make the code any easier to work with.

Around this time, I came across TRust DNS, “A Rust based DNS client and server, built to be safe and secure from the ground up.” I took some inspiration from the message parsing code there, specically the bindecoder idea.

The end result was a client that could make a very simple DNS query and spit back the response.

Weird and Interesting Things I came across

Compression. The protocol uses a bespoke method of compression where it will refer to resource names that have previously been seen. It starts with 11 in the leading bits which distinguishes it from an ordinary label because an ordinary label starts with a length byte, and 11 as the leading bytes would make the label too long (they’re limited to 63 bytes).
Bash has a special builtin tool for opening UDP and TCP sockets – I used this to sanity check my tool very early on (see hello_udp.sh in the repo). See this post for more details on how it works. (Notably, these builtins are not present in Zsh which threw me off).
Unified Domain-name Dispute Resolution Policy
Postel DNS Root incident
Is ICANN legal?
TLD Change email template

Unanswered Questions and future work

Who assigns new public IP addresses? And who maintains the existing mappings?
Is there a super nice CLI library like Cobra Commander for Go in Rust?
Recently, I wanted to know which process was issuing a DNS query against a DNS host I needed to shut-down, but I couldn’t seem to do it. It would be nice if there were a tool to trace particular traffic to an individual process.
Writing this super simple version of a common command line tool taught me a lot about how the system works on a lower level, so I’d like to try it with some other ones.

Starting to figure out SSL

Wed, 14 Feb 2018 00:00:00 +0000

Introduction

I’ve been slowly working towards a personal finance app, but before I can actually put it on the open web, I want to be reasonably confident that I am not opening myself up to being hacked.

One of the pieces to that puzzle is having secure connections in the browser for accessing my site. That involves HTTPS.

Progress

Here’s what I’ve got so far. There’s symmetric and asymmetric encryption. Symmetric is when both parties share the same secret. They use the same secret to encode and decode messages. In Caesar’s cipher, you shift all the letters in your message by some number N, so the secret would be N. When somebody knows N, they can encode your message, and they can eaisly re-encode another message.

Asymmetric encryption involves 2 keys: public, and private. The public key is for encryption and the private key is for decryption. The public key is not sensitive and can be shared. The private key is kept secret because it is the only thing that can decrypt something that has been encrypted with the public key. This means posting your public key somewhere is like saying here is a secure way to send me information at any point in time. It also provides a way for you to prove your identity. For instance, if you get a public key from a source you trust, you can verify that someone holds the private key by encrypting something, sending it to them, and having them send your message back to you..

Remaining questions

How does signing of a certificate work? Like you create your public key, and then a CA signs it, but only your private key can still decrypt it?

( Maybe there’s a portion of it that indicates that it’s signed, and then there’s your public key in there.)

How does the math work in symmetric and asymmetric encryption?
What is X.509?
How do I use a public key to verify something without consulting the private key? I want to verify the CA signed this thing, but that doesn’t fit in my pub-encrypt, priv-decrypt model.

To be continued!

Meandering History of DNS Authority

Wed, 14 Feb 2018 00:00:00 +0000

As a follow up to my DNS post, I started trying to answer “Who’s in charge of all this?”.

Basically, I wanted to know what keeps me from claiming a different domain name or for that matter for claiming a different IP address.

I went down a rabbit hole of RFC’s and never quite pulled out the answer to my question, but I wanted to post what I did find out since it’s been languishing in my drafts for too long.

Summary

Systems were each keeping track of the list of “names” of hosts and their “addresses” (which seem to just be decimal numbers, it’s unclear what protocol these addresses were for as this was before the IP protocol was developed.).

In 1973, the NIC put out an official list of host names and addresses, afterwhich somebody suggested that the NIC maintain a file that other hosts can pull down. The NIC agreed, and started serving a “HOSTS.TXT” file in ASCII format with the host name and addresses available via FTP.

In 1981, Internet Protocol (IP) was introduced, and J. Postel put out a list of IP ranges assigned to existing networks. These networks were free to assign IP addresses in that range as they saw fit.

In 1982, Feinler updates the HOSTS.TXT format to use IP addresses and to specify networks, gateways and hosts separately.

1969

First computers linked

Engelbart volunteers Stanford Research Institute (SRI) to be the Network Information Center (NIC) since he has NLS (oN-Line System) which could help manage it.

1972

Jake Feinler transfers into ARC (Augmentation Research Center) to work on NIC.

Socket catalog request - RFC 322

RFC 349 - Proposed Socket Numbers Postel suggests there be a czar for socket numbers for standard protocols and that it be him

Socket Number List - RFC 433 Postel specifies the socket number assignent

1973

RFC 597 - List of hosts on ARPANET interestingly the address numbers are between 0 and 255

OFFICE-1 the NIC host is on a PDP-10 along with NLS
Also offers "TENEX"

RFC 606 - Host Names On-Line L. Peter Deutsch recommends a centralized list of host names

1974

RFC 608 - Host Names On-Line Jake commits to updating a text file File lives at HOSTS.TXT Host "OFFICE-1" Host Address = 43 decimal

^^ so Host Addresses were just numbers

1977

NLS sold to Tymshare, with it the machine NIC ran on, so NIC switches to DEC-10

RFC 739 - Assigned Numbers Postel lists the assigned “link” numbers and socket numbers for protocols. Not clear to me exactly what a “link” is. I’m not sure how packets were routed, this was pre-IP though, so maybe it was in a really weird way

Refers to the “internetwork protocols”

1980

RFC 760 - Internet Protocol

1981

Convert from NCP to TCP/IP (NCP was original Arpanet protocol) TCP/IP created?

DDN created

RFC 790 - Assigned Numbers Class A,B,C networks described. IP ranges assigned to different networks

RFC 796 - Address Mappings Class A,B,C networks, describes how network specific addresses correspond to IP addresses.

RFC 791 - Internet Protocol (IP) This is an update

1982

RFC 810 - Host Table Specification

It can be obtained by connecting to host SRI-NIC (10.0.0.73) from your local FTP server, logging in as user=ANONYMOUS, password=GUEST, and doing a ‘get’ on HOSTS.TXT.

RFC 811 - Hostname Server SRI-NIC is mentioned to be running on a Foonly Describes command interface for looking up the host addresses for machines by name

1983

Deadline for military to adopt TCP/IP

1984

Domain Name requirements RFC 920

The administrator of a level N domain must register with the registrar (or responsible person) of the level N-1 domain. This upper level authority must be satisfied that the requirements are met before authorization for the domain is granted.

Lays out the information necessary to register a domain. Essentially: email hostmaster@sri-nic.arpa with your information including the IP addresses of the name servers that will host the DNS information.

The IP address listed is of the typical x.x.x.x format.

1985

RFC 995: Hostname server (Feinler update to RFC 811)

1987

NIC took over maintaining assigned numbers and become Arpanet/DDN’s naming auhtority

RFC 1035 - DNS protocol specification

1991

NIC contract awarded to Network Solutions Inc (NSI).

1993

NSI adds root server

1994

RFC 1591 (Postel): internic.net - registers second level domains under the TLD’s (com, edu, org, gov, net)

now is US Department of Commerce

.mil - DDN .int - PVM at ISI.edu (USC)

1998

ICANN established

Unanswered Questions

If NIC did the host names, what did IANA do?
What is TIP? and what is a TIP phone number?
What was Namedroppers?
What format were addresses before IP? How did things get routed?

Wifi Woes resolved with Ubiquiti

Tue, 16 Jan 2018 00:00:00 +0000

Fast Wifi is possible!

I have declared my Ubiquiti experiment a success! I am consistently measuring internet speeds over 100 MB/s with the lowest I’ve seen around 40 MB/s. This is a huge upgrade from seeing 5-8 MB/s regularly with the highest being around 15 MB/s (see my previous post for more details on the diagnosis of my old system).

In this post, I want to document my installatin process because it was a bit of a doozy. I also want to offer some recommendations in case anyone else is considering investing in a Ubiquiti wifi set up.

The Installation Process (A Network Bootstrapping Wrestling Match)

First, here is the hardware I ordered ¹:

The first step after plugging everything in and powering it on was “adopting” the devices to a controller. I started out putting the controller software on my laptop for expediency with the eventual goal of running it on a RaspberryPi.

Laptop Controller

My laptop does not have an ethernet port, so I connected to the AP using the QR code on bottom side of the AP. I had to use the iOS App to do this scanning because the Android app wasn’t working for me. This gave me the SSID for a hidden network (the MAC address of the AP w/o the colons) and a password. I was able to then see the USG in the controller interface, but I could not see the AP. I assumed this was because I was connected through the AP and that in order for the controller to adopt the AP it had to be running independently of it².

Raspberry Pi Controller over Ethernet

This was when I decided to plug the Raspberry Pi directly in to the USG andstart using it as the controller. I followed a guide to get the software installed and after a couple of speedbumps³, got the controller up and running. I was SSH’ing to the machine via my laptop which was still connected to the AP’s MAC address network, and I assumed I needed to not be connected that way in order for the AP to be adoptable (an assumption that may have been flawed). In order to work around that, I set up the RaspberryPi as a wireless access point and then connected and SSH’d that way.

Raspberry Pi Controller over WiFi

I then was able to adopt the AP and the USG, and things were looking good. However, I needed to free up an ethernet port in order to plug in the brigde for my Hue lights, so I wanted to get my Raspberry Pi acting as the controller over Wifi. I figured this would be a simple exercise of connecting the Raspberry Pi to Wifi and updating some configuration settings. Once I did this, and unplugged the ethernet cable, it started reporting the USG as not connected. Apparently, the USG wants the controller to be on the same subnet as it, and in this case the AP (and therefore the controller) was on the LAN2 subnet⁴. I tried naively swapping the AP to the LAN1 port, but this really messed evertyhing up, so I factory reset everything and started from the beginning.

On my next attempt, after I got the Raspberry Pi up and running as the controller. I unplugged it, and plugged the AP into the same port. This way the AP was running on the LAN1 subnet. After reconnecting the Raspberry pi, everything worked, and I recorded much faster speeds than I had previously read.

In the future, I want to look into running a Controller from an EC2 or GCE instance so I don’t have to keep my RaspberryPi plugged in to power all the time.

Reccomendations

Be generous with your ethernet ports - investing in an ethernet switch will save you heartache even if you only use it for the initial set up.
Buying a “Cloud Key” will also make your setup life a lot easier, some of the “Discovery” and “Set Up” tools have special features that only work with a Cloud Key.
Be sure your Controller is on the same subnet as the USG.
If your modem doesn’t immediately work with the USG, unplug everything, wait a few minutes and try again. (This may seem like the technology equivalent of a ceremonial sacrifice, but it worked for me, and I’m willing to chalk it up to something in the modem needing to be cleared out from the previous router)

Take-away

Hopefully this saves someone from the mostly self-inflicted torment I experienced. Overall, I am very happy with the Unifi set up and would recommend it to anyone who’s willing to invest a little bit extra in the set up.

If you’ve read up on the Unifi line of products, you may notice this is missing a “Controller”. I have RaspberryPi laying around which I decided I was going to use at the Controller, but more on that later. ↩
Looking back, there was probably a way to manually adopt the AP by SSH’ing to it as described in this forum post, but I did not know enough at the time to try that. ↩
First, I had some weird version of Raspian installed which made it impossible to install the right packages. I had to flash a new image and remember to touch ssh in the boot directory in order to enable SSH access on a fresh install because I didn’t have a monitor or keyboard with me. ↩
I gathered this from an article about how to adopt a USG into an existing network. It says “If the controller is on a subnet other than the USG’s default 192.168.1.0/24, it is necessary to change the USG’s LAN IP so the controller and USG can communicate”. ↩

Debugging Slow Wifi

Thu, 04 Jan 2018 00:00:00 +0000

Introduction

I ran into the ever common problem of slow WiFi recently. In this post I want to detail the experiments I ran and what I gleaned from the results.

Setting the Stage

First I’ll describe the setup I have for my home wifi network and the problems I was experiencing.

Symptoms:

No problem streaming Netflix
Trouble loading web pages occasionally

I had a weird set of problems in that I typically could stream episodes of shows on Netflix without any huge problems, but sometimes loading sites would be unbearably slow. It was common enough to be a huge pain, but the times when it worked well confused me. Why was it only sometimes slow?

Hardware:

Modem: Arris Surfboard SB6141
Router/AP: Google OnHub by TP-Link

Investigation

I wanted to check if my internet was actually slow or whether I was just impatient. I ran speedtests from several different providers, and it was in fact slow. (billed speed was 100 Mb/s, observed was between 3-8 Mb/s).

At first, I was convinced that Comcast was to blame for my slow wifi speed. However, I ruled this out by checking the internet speed over ethernet. Comcast was delivering speeds to my router that were much higher than I was observing at my laptop. (observed ethernet speeds between 56-75 Mb/s)

Next, I wanted to establish a baseline for what the point to point speed on my network was. This came out around 5 Mb/s which was really slow!

Next, I thought maybe there was a lot of WiFi congestion in my area. Maybe my router was picking a congested channel, or there was some kind of crazy interference in my apartment. To test this, I ran my raspberry pi right next to my router in AP mode. I was very surprised when the Raspberry Pi was able to get 33 Mb/s running on the same channel as my router.

Experiments

Speed Tests

Ran speed tests on 3 different sites: fast.com (Netflix’s Speed test site), speedof.me (Javascript based graphs and reports), and speedtest.net.

Results:

Fast.com: 8 Mbps
speedtest.net: 4.6 Mbps
speedof.me: 3.15 Mbps

Raspberry Pi on Ethernet

Setup:

I plugged my Raspberry Pi into the ethernet port of my router. I then downloaded speedtest-cli (a CLI for running speed tests against speedtest.net). I also ran a test with wget¹

Results:

speedtest-cli: 75 Mb/s
wget: 6.9 MB/s => 56 Mb/s

Laptop to Laptop communication

Setup: I installed iperf on 2 laptops. Laptop 1 is running Linux on a 2011 Macbook Air and was running iperf -s (the server). Laptop 2 is a 2017 Macbook Pro running iperf -c. I started out running with the default parameters which runs a test for 10 seconds, then switched to longer times using iperf -c -t 60 to run the test for 60 seconds.

Results:

Average for 10s (6 trials): 17.7 Mb/s
Average for 60s (3 trials): 5 Mb/s

Raspberry Pi as AP

Setup: I put my raspberry pi in AP mode² and had my laptop connect to it directly.

Results:

iperf -c -t 60 : 33.4 Mb/s

Conclusions

My conclusion is that my router is not performing. I tried power cycling it to see if that would help, but the measurements were the same after. I am not certain why this is the case, especially since it is supposed to have 13 antennae which I would expect to lead to high LAN bandwidth. My next experiment is to order some UniFi hardware and see if was that improves the situation³.

I never got a good answer as to why I was able to stream Netflix but regular websites would have trouble loading.

Hopefully my new hardware does the trick. I plan on writing a follow up post when I have it all set up to see if it makes a difference.

The exact command I ran: wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip NB: wget reports speeds in megaBYTES not megaBITS (like most wifi tools). Multiply by 8 to get a comparable number. ↩
To setup a Raspberry Pi as an AP, I used hostapd and udhcpd which you can find instructions for on raspberrypi.org ↩
I am going with a USG and a UniFi AP AC Lite. I went with Ubiquiti based on recommendations from some coworkers and a blog post by Troy Hunt that describes similar frustrations as mine. I debated whether to get an EdgeMax router over the USG after seeing some discussion in Troy’s gist. I ended up going with the USG since I don’t believe I’ll benefit from the more advanced configuration opions of the EdgeMax, and I like the idea of all of it working with the UniFi portal. ↩

DNS End to End

Sat, 25 Nov 2017 00:00:00 +0000

Domain Name System: End to End

We all interact with the Domain Name System (DNS) every day. Every time we load a web page, or click a link, our software relies on DNS to figure out what address to send requests to. In this post, I will dig into how DNS works by tracing through the machinery that happens behind the scenes.

What is the purpose of DNS?

It is first worth calling out what DNS is and why anybody should care about it. A good place to start is Domain Names - Implementation and Specification RFC1035. In the introduction, it says:

The goal of domain names is to provide a mechanism for naming resources in such a way that the names are usable in different hosts, networks, protocol families, internets, and administrative organizations.

Domain names are supposed to be a convenient, general purpose way to refer to resources. Convenient in this case most likely means relative to referring things to an actual host address. Practically, this means DNS turns “google.com” into an IP address like “216.58.192.14” ¹. This provides a number of benefits. “google.com” is much easier to remember than a series of 4 numbers. Also, a web service can change which host address it wants to use without end users even noticing².

DNS Resolution “by hand”

Now that we know the goal of DNS, I want to demonstrate how DNS works by crafting DNS queries using dig. I will ignore a lot of pieces of the system for this illustration (like caching and recursive resolution). I will try to resolve “pcarleton.com”

Root Domain

In order to figure out what IP address is serving “pcarleton.com.”, I can work my way down the hierachy. There are servers at each level of the hierarchy that can tell me where to find information for the level below. The domain name “pcarleton.com” becomes ["", "com", "pcarleton"] in the hierarchy where the empty string is the implied “root” domain³.

A name server at the root level can tell me about what nameservers are responsible for each Top Level Domain (TLD) like “.com”, “.net” etc. In order to query a root server, I need to know one of its IP addresses. I can pick one from the IANA website, so I’ll pick a.root-servers.net with IP address 198.41.0.4⁴

I can then query this root server for what name servers are responsible for pcarleton.com via dig -t ns pcarleton.com @198.41.0.4. This gives me a list of servers which are responsible for the .com TLD.

.Com Domain

The root server gave a list of .com nameservers (and IP addresses) that look like a.gtld-servers.net with the letters A through M. For instance, a.gtld-servers.net has address: 192.5.6.30. I can then query this address to see which .com name server is responsible for the pcarleton domain name with dig -t ns pcarleton.com @192.5.6.30.

Namecheap DNS servers

This query reveals that dns1.registrar-servers.com has information for pcarleton.com (and has IP 216.87.155.33). This name server is the one run by Namecheap which is where I registered pcarleton.com⁵. If we query this one with dig pcarleton.com @216.87.155.33, we get the IP address of this site which we can then use.

Making Changes

This example demonstrated how to interact with the DNS system, but it did not show how changes to that information would propagate. To understand that, let’s look back at what data was required:

Changes to the list of root IP’s should never happen. If it did, it would require pushing a file listing the new IP’s to all running DNS servers by system admins all over the world.

Changes of the mapping of TLD’s to authoritative name servers does not happen frequently, but is administered by ICANN. It will be communicated to all the root servers which will update their records and serve them to requests.

Changes to the TLD server’s mapping (the Registry) changes more frequently. Every time a new domain name is registered, the TLD’d servers need to know which domain name servers have the required IP addresses. A Registrar makes a request to the “Registry” to update these records.

Changes to the final DNS servers mapping of domain name to IP address can happen much more frequently since it only needs to be updated in the two DNS servers listed. I can change this by making a request to Namecheap.

Reality Check

This example showed how some of the pieces of the DNS system work, but it is not typically how a DNS request goes. In reality, the client usually makes a request to a DNS server⁶ which has a lot of information cached (like the .com TLD servers) and will make requests to the appropriate servers rather than telling the client which nameservers to look at⁷.

Further Information

Here’s a list of resources to look for further information about DNS:

Internet Numbers Registry System - RFC 7010 List of Local Internet Registries (This who Namecheap could contract with to update the nameserver mapping)

IANA’s role - RFC 1591

Jon Postel (the original IANA)’s RFC Eulogy

Jon Postel root swapping incident

Argument that ICANN’s authority is illegal

Unified Domain-name Dispute Resolution Policy

Email Template to update IP for a TLD name server

ARIN

RIR

You can test this out locally by running dig google.com from the command line. If you put this IP address into your browser, it will load google.com. ↩
It is not difficult to imagine how a bad actor could cause trouble with this too. If they were to compromise DNS information, they could cause clients to send information to a location where they can intercept it. For this post, I won’t get into that, and I will pretend like everybody on the internet are behaving nicely. ↩
pcarleton.com is implicitly re-written as “pcarleton.com.” (note the trailing period) where the root domain is the empty string at the end. ↩
Ordinary websites do not load “iana.org” in order to figure out root servers. These 13 name servers are like internet constants in that their IP addresses never change. Domain name servers usually have these IP addreses in a file somewhere. ↩
I verified this was Namecheap’s by going to whois.com. This info can also be obtaind with the whois CLI with whois dns1.registrar-servers.com --host whois.enom.com (I got the whois.enom.com part from first issuing the command without the --host argument). Enom must be the registrar that Namecheap used to register its domain with. That leads me to wonder if there are any circular registrar dependencies, but is looks like Enom used itself to register “enom.com”. ↩
One such server is Google Public DNS which resides at IP address 8.8.8.8. ↩
This is assuming that the request is “recursive”. A recursive request asks that the server make requests to other DNS servers if it doesn’t have the answer. An “iterative” request by contrast asks that the server tell the client what server will have the answer if it does not have the answer. A DNS server gets to decide whether it supports “recursive” requests. ↩

What does mount do?

Mon, 10 Jul 2017 00:00:00 +0000

Background

As I mentioned in my post about containers, I want to write a container tool in rust to learn more about rust and containers. I started doing that with my friend Kevin, and our (humble) progress so far is on Github in a project we called bucket (like a container, that’s sometimes rusty… or something).

We’re following a long a talk given by Eric Chiang at CoreOS Fest called Containers From Scratch (slides). In the talk, Eric walks through the linux command line utilities that you can string together yourself to make a “container”. We got as far as creating a “root filesystem”, chrooting into it, and then making a separate PID namespace by calling unshare. It was during this unshare step that I ran into an error message that led me to learn more about mount that I want to document here.

What does `mount` do?

mount is a command line utility which wraps the mount system call. (Another fun fact I learned: the #’s after a name in a man page indicates what section that man page is in. So mount(2) is the system call because it is in the “System calls” section of the manual and mount(8) is the command line utility because it is in the “System Administation Commands and Daemons” section. More info in this SO Answer).

If you’re like me, you have had to call mount some times when you’ve plugged in a USB drive and it hasn’t worked correctly. Usually, the OS handles the mounting for you. For instance, when I plug in a USB stick to my computer, it pops up a Files window with the contents.

First, it’s worth knowing how to get information about current mounts. The place for that is mtab. mtab is a read-only file that lives at /etc/mtab (and in my case is a symlink to /proc/self/mounts) which lists the current active mounts. When I plugged in my USB drive, I saw the following line at the bottom of mtab:

/dev/sdc2 /media/paul/P16G hfsplus ro,nosuid,nodev,relatime,umask=22,uid=0,gid=0,nls=utf8 0

If we break that down, we have a device on the far left called /dev/sdc2. (Quick side note: to break down sdc2, first ignore the sd, then you have c which indicates it’s the third drive that the OS has seen. sda is my computer’s hard drive, I’m not sure what sdb is. Then the 2 means what partition on that drive, so this is the 2nd partition. (more info in this Superuser answer))

After that, we have the mount point. This is directory where the files on the device are accessible. In this case, I can go to /media/paul/P16G to access the files on the USB stick.

Next is hfsplus indicating the filesystem type. Then there are several options and then 2 0’s indicating the dump/pass options. 0’s mean we don’t back up this mount (dump) and we don’t run fsck on it to detect errors (pass). The fstab wikipedia page has more info on this file’s format.

If I want to unmount this, I can run umount /media/paul/P16G. I can then remount it with sudo mount -t hfsplus /dev/sdc2 /media/paul/P16G. (Interestingly, I had to create the P16G directory since when I unmounted it, the directory went away. I’m not sure why that is.)

Another interesting place related to mount is fstab. It’s the same format as mtab, but instead of reflecting the current mounts, it represents what mounts should be created at boot time. If you change that file, you will have new mounts when you reboot.

That gives us enough information to move on to unshare.

What does `unshare` do?

The particular unshare command we were looking at was:

sudo unshare -p -f --mount-proc=$PWD/rootfs/proc chroot rootfs /bin/bash

This creates a new PID namespace, mounts a proc filesystem under the rootfs/proc directory and then executes chroot and finally bash.

The tricky part I was running in to was that I was getting a “Invalid argument” error message when I tried to run this command.

First, I’ll look at mounting a proc filesystem. It turns out I can do this anywhere and I don’t need a device like I did for mounting a USB drive. Instead, whatever I pass in as the device becomes a kind of dummy label. For instance, I can say mount -t proc dummy ~/fs1 and I’ll get a proc filesystem in the fs1 directory. If I do ls ~/fs1, it will look identical to doing ls /proc. Additionally, I get the following line in mtab:

dummy /home/paul/fs1 proc rw,relatime 0 0

Now, I want to look at what the unshare command does when it has the --mount-proc command because it starts to demonstrate why I ran in to my error. Here are some lines from the unshare source code:

if (procmnt &&
    (mount("none", procmnt, NULL, MS_PRIVATE|MS_REC, NULL) != 0 ||
     mount("proc", procmnt, "proc", MS_NOSUID|MS_NOEXEC|MS_NODEV, NULL) != 0))
          err(EXIT_FAILURE, _("mount %s failed"), procmnt);

I can see there are 2 calls to mount(2) (the system call). The first isn’t creating a mount, but it’s taking the mount point and making it private recursively. This is the same as saying mount --make-private $procmnt. Interestingly, if I unmount fs1, and then try to make it private with mount --make-private fs1, it fails, saying that fs1 is not a mountpoint. This is not the exact error I was getting from unshare, but I assume it has the same source. If I run mount --bind fs1 fs1, and then run the make private command it works (and similarly with the unshare command). The problem was that the unshare command expects the target for mounting the new proc filesystem to already be a mount point.

What does `--make-private` do?

It’s nice to know why the command was failing and how to fix it, but what might be more important to understand is why unshare would be trying to make the proc filesystem private to begin with.

The mount(8) manpage explains the different options for the sharing status of a file system so I won’t repeat them here. However, it’s useful to know how to tell what the sharing bits are for a particular filesystem. To get that, I can look at /proc/self/mountinfo and I can see for my procfs a line like this with the word shared in it:

170 24 0:4 / /home/paul/fs1 rw,relatime shared:146 - proc dummy rw

If I call mount --make-private fs1, the shared:146 portion of the line goes away.

How does a shared mount differ from a private one? A shared mount means that subsequent calls to umount and mount will propagate to other --bind mounted file systems. In a private mount, those calls will not propagate. To illustrate this, here is an example:

$ mkdir fs1 fs2 subd1
$ sudo mount --bind fs1 fs1
$ sudo mount --bind fs1 fs2
$ sudo mount --make-private fs1
$ mkdir fs1/sub_mount
$ ls fs2
sub_mount
# ^ Files are propagated between the mounts

$ sudo mount --bind subd1 fs1/sub_mount
$ touch subd1/hello
$ ls fs1/sub_mount
hello
$ ls fs2/sub_mount
$ # nothing

It turns out that unshare mounts all its new mounts as private by default (see unshare(1)). Since the purpose of unshare is to isolate it from other things, this makes sense conceptually, but practically, I am having a hard time thinking of when a proc file system being shared would be a thing that mattered, but then again I do not know if there is more mounting that happens during the creation of a proc fs.

Conclusion

This post was a bit of a random walk around mount and unshare, but I learned something during it, so I wanted to record it in case I find myself wondering about it later.

pcarleton's blog

TIL: The difference between netstat and ss

Motivation

What is a socket?

Why would I care about sockets?

How do I inspect the state of sockets on my machine?

Procfs

Netlink

Netstat vs. ss

Open Questions

New Format: Today I Learned

TIL: There are 16 T1 Internet Service Providers

AT&T

CenturyLink

Deutsche Telekom AG (ICSS)

GTT Communications

KPN International

Sprint

Open Questions

Other resources:

Dumb dig clone in Rust

Why write a DNS client

The Protocol

Message

Header

Question Section

Resource Record (answer)

Implementation

Weird and Interesting Things I came across

Unanswered Questions and future work

Starting to figure out SSL

Introduction

Progress

Remaining questions

Meandering History of DNS Authority

Summary

1969

1972

1973

1974

1977

1981

1982

1983

1984

1985

1987

1991

1993

1994

1998

Unanswered Questions

Further Reading:

Wifi Woes resolved with Ubiquiti

Fast Wifi is possible!

The Installation Process (A Network Bootstrapping Wrestling Match)

Laptop Controller

Raspberry Pi Controller over Ethernet

Raspberry Pi Controller over WiFi

Reccomendations

Take-away

Debugging Slow Wifi

Introduction

Setting the Stage

Investigation

Experiments

Speed Tests

Raspberry Pi on Ethernet

Laptop to Laptop communication

Raspberry Pi as AP

Conclusions

DNS End to End

Domain Name System: End to End

What is the purpose of DNS?

DNS Resolution “by hand”

Root Domain

.Com Domain

Namecheap DNS servers

Making Changes

Reality Check

What does `mount` do?

What does `unshare` do?

What does `--make-private` do?