Windows Firefox installer

Thanks to Arthur Casals, we have a Windows installer! See the Firefox page for the link.

This version only supports TLS transport however, in order to keep the build issues sane! The TLS only Firefox patch is quite small and I'll try to push it upstream at some point. However, Chrome is keeping me busy at the moment.

We hit Slashdot

We hit Slashdot on Tuesday, the craze is dying down now.

I think I learn two things:

1. The introduction video was a bit crappy and several people, rightly, said so. However, the general level of misunderstanding was substantially reduced, so it worked I guess
2. I should be provided a transcript of the video or, at least, a non-video replacement for it. I've now done this.

TLS transport support

TLS transport support is here, yay! Well, it's just hitting the site now. In short, you can configure the DNS advert to say "Actually, connect with HTTPS on port x". Firefox will see this and, even when the user entered "http://", will use HTTPS and ignore any certificate problems.

This makes server support easy, it's just HTTPS. Performance isn't so great, but, well, there you go. Port 443 also works better with firewalls.

The documentation and git trees are up to date. I'll be uploading the Firefox build (64-bit Linux) in just a second.

ABI break, wire break

I've switched the default cipher from Salsa20 to Salsa20/8. I had always intended to do this, it was just a question of getting a good implementation of Salsa20/8 first.

However, this entails an ABI and wire protocol break. Hopefully there won't be many more of these. So I've uploaded new tarballs for Firefox and libobstcp. Also, there's now a tarball for Firefox built on IA32 Linux (currently untested).

Testing needed

Ok, all the documentation on the site has been updated. There is now new code (using DNS adverts), a new video introduction, new Firefox patches etc.

Please, if you have any time, goto the code site, act dumb, watch the video and follow the instructions, either as just a user, or as a webmaster too.

Tell me what doesn't work and what doesn't make sense.

Cheers

DNS support

It's been a couple of weeks since I worked on ObsTCP. There has been some public key signature work and family matters in between.

However, I did have a stonking good idea today (with partial credit to djb).

Previously I intended to encode adverts in a DNS TXT record under a special name. This would mean writing a custom resolver to handle this and doubling the number of DNS lookups for each host.

However, if I encode the advert in a CNAME, I don't need either.

Consider www.example.com, a CNAME for opa12354zbcdefghijkl12345.example.com. When one performs a lookup for an A record for www.example.com, the server will return the A record opa...example.com and the CNAME record for www. By encoding the advert in the CNAME, we don't need an additional request.

Also, the CNAME chain is returned by gethostbyname (but not getaddrinfo), thus we can use the standard libc resolver too.

I'm very happy with this. I'm patching up Firefox now to use gethostbyname and to parse adverts from DNS.

New release: 0.2

Another week, another release. This week brings lots of rewriting of the core to be cleaner and Apache support.

See the project page for details. Remember that, if you're installing 0.2 you should download a new Firefox, repatch lighttpd etc. Also, when you make install, if you have processes linked against a shared library that you override, they'll probably crash.

Obfuscated TCP (take 3) ready for alpha testing

Well, after rewriting it again, the new design is ready for testing!

Not all of the old pages have been updated, but most have. Start at http://code.google.com/p/obstcp/wiki/Testing.

This time it doesn't involve any kernel patching. You just install the core library, download a patched Firefox binary, install a Firefox extension and you're done!

I'll be working on some more documentation tonight / tomorrow.

If you try it (sucessfully or otherwise) please tell me. Comments on the blog, comments on the site (http://code.google.com/p/obstcp), mailing list posts (http://groups.google.com/group/obstcp) or email (agl AT imperialviolet DOT org) all welcome.

And, of course, you should thank Google, who are still employing me as I write this. Although no support of the idea by Google, as a company, is implied.

The firefox modifications to support Obfuscated TCP appear to be working. There's still a fair bit of work until they exist as an extension rather than a source patch however.

I've written the new libobstcp, it's not in a final state yet (of course), but it's functional. The header file is included below, I'll release the source once there's a client side implementation (maybe as a Firefox extension).

The design is thus: I started writing the HTTP-in-HTTP encapsulation and discovered that it's really hideous. I don't think anything that feels so wrong can be a good design decision, so, here's the main point: it's using a different port number for obfuscation now.

The client finds out the alternative port number via an "advert" for the server. The advert can either be in the DNS, or can be remembered from previous (non-obfuscated) HTTP connections where the advert is returned by the server in an X-ObsTCP header. The advert is a small, base64 encoded string that contains the server's public key, obfuscated port number, and (optionally) the TLS port number.

This isn't specific to HTTP, the general idea is that the advert is a way of saying "This service is alternatively available over ObsTCP/TLS on these ports...". The advert is a tagged format, so there's room to extend it in the future. Planned extensions currently include integrety protection for the data stream using either packet signing in the kernel or MACs at the application layer (or both).

So, the major work at the moment is writing a Firefox extension to do this. Firefox is a big piece of code.

#ifndef LIBOBSTCP_H
#define LIBOBSTCP_H

#include <stdint.h>
#include <sys/uio.h>

// -----------------------------------------------------------------------------
// struct obstcp_keypair - a public/private key pair
// keyid: the 32-bit xor folding of the public key
// -----------------------------------------------------------------------------
struct obstcp_keypair {
  uint8_t private_key[32];
  uint8_t public_key[32];
  uint32_t keyid;
  struct obstcp_keypair *next;
};

struct obstcp_keys {
  struct obstcp_keypair *keys;
};

// -----------------------------------------------------------------------------
// Setup a keyset structure. This must be called before any other operations
// on the keyset.
// -----------------------------------------------------------------------------
void obstcp_keys_init(struct obstcp_keys *keys);
// -----------------------------------------------------------------------------
// Calculate the public key from a private key and prepend the key pair to the
// list of keys installed in the keyset. This becomes the default key
//
// returns: 1 on success, 0 on error, in which case errno is set:
//   ENOMEM: out of memory
//   ENOSPC: a key with the same keyid is already in the keyset
// -----------------------------------------------------------------------------
int obstcp_keys_key_add(struct obstcp_keys *keys, const uint8_t *private_key);
// -----------------------------------------------------------------------------
// Free all keys contained in the keyset
// -----------------------------------------------------------------------------
void obstcp_keys_free(struct obstcp_keys *keys);

// -----------------------------------------------------------------------------
// Keys for the variable arguments part of obstcp_advert_create and
// obstcp_advert_parse
// -----------------------------------------------------------------------------
enum {
  OBSTCP_ADVERT_END = 0,      // no argument
  OBSTCP_ADVERT_OBSPORT,      // takes an int giving the port
  OBSTCP_ADVERT_TLSPORT,      // takes an int giving the port
};

// -----------------------------------------------------------------------------
// Generate a base64 encoded obstcp "advert", this is the data that can be
// included in a DNS TXT record of via other side channels. It takes a variable
// argument list. The variable part of the argument list are a series of pairs
// where the first element of the pair is one of OBSTCP_ADVERT_* and the second
// depends on the value of the first, and may not even exist. The default key
// from the keyset is used for the public value.
//
// The variable list must be terminated with OBSTCP_ADVERT_END.
//
// output: a buffer to which the base64 encoded data is written
// length: number of bytes of space in @output
// returns: on success a value <= to @length is returned. This is the number of
//   bytes written. If @output was too small, a value > @length is returned.
//   This is the number of bytes that is required. Otherwise -1 is returned and
//   errno is set:
//
//   E2BIG: too many options were selected
//   EINVAL: an unknown or invalid pair was found in the arguments
//   ENOKEY: no default key was found in the keyset
// -----------------------------------------------------------------------------
int obstcp_advert_create(char *output, unsigned length,
                         const struct obstcp_keys *keys, ...);

// -----------------------------------------------------------------------------
// Parse a base64 encoded obstcp "advert". This can be used to extract the
// advertised obfuscated and TLS port numbers.
//
// The variable arguments come in pairs. The first of each pair is one of
// OBSTCP_ADVERT_* which are shared with obstcp_advert_create, above. However,
// since this is a parsing function, where, say, OBSTCP_ADVERT_OBSPORT is
// documented as having an int as its second element, for this function is
// would be a pointer to an int.
//
// The variable list must be terminated with OBSTCP_ADVERT_END.
//
// For elements which aren't found a special value is used to denote this. For
// port numbers, this value is 0.
//
// input: the base64 encoded banner
// length: the number of bytes in @input (not inc \n etc)
// returns: 1 on success, 0 on parse error.
// -----------------------------------------------------------------------------
int obstcp_advert_parse(const char *input, unsigned length, ...);

// -----------------------------------------------------------------------------
// This is the crypto context for a given direction of data
// -----------------------------------------------------------------------------
struct obstcp_half_connection {
  uint8_t  keystream[64];  // keystream bytes
  unsigned used;  // number of bytes of @keystream used
  uint32_t input[16];  // salsa20 context
};

// -----------------------------------------------------------------------------
// Server interfaces...
//
// These functions are designed to be used by the 'server' side of the
// connection. It's up to the user of this library to decide which side is the
// server, it doesn't have to correspond to the sockets API notion of a server,
// although it's expected that it usually will.

struct obstcp_server_ctx {
  const struct obstcp_keys *keys;
  union {
    struct {
      uint8_t buffer[386];
      unsigned read;
    } a;
    struct {
      struct obstcp_half_connection in;
      struct obstcp_half_connection out;
    } b;
  } u;

  int state;
  char frame_open, frame_valid;
};

// -----------------------------------------------------------------------------
// Setup a context structure. This must be called before any other operations
// on the context struture.
// -----------------------------------------------------------------------------
void obstcp_server_ctx_init(struct obstcp_server_ctx *ctx,
                            const struct obstcp_keys *keys);

// -----------------------------------------------------------------------------
// Read from a socket
// fd: the file descriptor to read from
// buffer: buffer to write data to
// len: number of bytes in @buffer
// ready: (output) on success, this is true if a MAC was successfully
//   calculated.
//
// Reading from a socket has two phases:
//   1) setup phase, before key agreement has completed. In this phase this
//      function with return -1 with errno set to EAGAIN until it has enough
//      data to complete key agreement.
//
//      If key agreement is successful, we move to phase two. Otherwise, we
//      return -1 and set errno to EPROTO.
//
//   2) In this phase we are reading application data from the socket. It may
//      be that the data is MAC protected. In this case, the read calls will
//      return positive byte counts, but *ready will be false on return. Once
//      the MAC has been read and checked *ready will be true. This means that
//      all the data since the last time ready returned true is valid and can
//      be processed. There cannot be > 16K of unready data.
//
//      Do not use this for application level framing since MAC may not be in
//      operation - in this case *ready is always set to true.
//
// Otherwise, this call returns like read(2) - i.e. 0 return means EOF etc. One
// exception is that this call will never return -1 with an errno of EINTR. The
// socket can be blocking or non-blocking.
//
// If you wish to know if key agreement has completed after calling this
// function, use obstcp_server_ready.
// -----------------------------------------------------------------------------
ssize_t obstcp_server_read(int fd, struct obstcp_server_ctx *ctx,
                           uint8_t *buffer, size_t len, char *ready);

// -----------------------------------------------------------------------------
// Returns true iff key agreement has completed.
// -----------------------------------------------------------------------------
int obstcp_server_ready(const struct obstcp_server_ctx *ctx);

// -----------------------------------------------------------------------------
// Write examples:
//
// Simple case: in this case, the application is writing blocks of data to a
// socket. It would normally use a series of write() calls. The socket is
// blocking.
//
// ssize_t write_data(int fd, uint8_t *data, size_t len) {
//   struct iovec[3] iov;
//
//   const unsigned n = obstcp_server_encrypt(ctx, data, data, len, 0);
//
//   switch (obstcp_server_ends(ctx, &iovec[0], &iovec[2])) {
//   case -1:
//     abort();
//   case 0:
//     return write(fd, data, n);
//   default:
//     iovec[1].iov_base = data;
//     iovec[1].iov_len = n;
//     return writev(fd, iovec, 3);
//   }
// }
//
// Note that, for non-blocking sockets it's up to the application to deal with
// feeding the data to the socket. However, obstcp_server_ends may be called
// repeatedly to get the same data.
// -----------------------------------------------------------------------------

// -----------------------------------------------------------------------------
// This library doesn't perform writing as such, instead call this function to
// encrypt the data in preparation for writing.
//
// output: encrypted data is written here (maybe equal to @buffer)
// buffer: data to be encrypted
// len: the length, in bytes, of @buffer and @output.
// frame_accum: if true, more data is next so don't close out the frame.
// returns: the number of bytes encrypted
//
// A frame can cover, at most, 16384 bytes, so if you repeatedly call this
// function with frame_accum true it may return less than @len bytes to denote
// that the frame is full.
//
// NB that the peer cannot process anything until a frame's worth of data has
// been processed. Thus, if this is the end of a message, and you expect the
// client to answer you, you must close out the frame.
//
// Before writing anything to a socket you must call obstcp_server_ends to get
// the prepended and appended data for the frame.
// -----------------------------------------------------------------------------
ssize_t obstcp_server_encrypt(struct obstcp_server_ctx *ctx,
                              uint8_t *output, const uint8_t *buffer, size_t len,
                              char frame_accum);

// -----------------------------------------------------------------------------
// The protocol may need to prepend and append data to a frame. Call this
// function to get that data. You pass it two iovec's: one to be sent at the
// beginning of the frame and one to be sent at the end.
//
// returns: -1 on error, or the number of iovecs with data.
//
// If this function returns -1 it means there has been a programming error on
// the part of the library user. Either you haven't closed out a frame or you
// haven't opened one. Aborting the process is reasonable in this case.
//
// Note that some connections may not having framing, thus this function may
// return 0. It's worth detecting this and falling back to a simple write()
// rather than a writev() in this case.
// -----------------------------------------------------------------------------
int obstcp_server_ends(struct obstcp_server_ctx *ctx, struct iovec *start,
                       struct iovec *end);

// -----------------------------------------------------------------------------



// -----------------------------------------------------------------------------
// Client interfaces...

struct obstcp_client_ctx {
  uint8_t buffer[386];
  unsigned n;

  struct obstcp_half_connection in;
  struct obstcp_half_connection out;

  int state;
  ssize_t frame_size;
  char frame_open;
};

// -----------------------------------------------------------------------------
// keys: a keyset for the client. Only the default key will be uesd.
// advert: a base64-encoded advert for the server. This can be obtained via TXT
//   records in DNS or any other side channel.
// len: length of @advert, in bytes.
// random: 16 random bytes, never to be reused
// returns: 1 on success, 0 on failure (in which case errno is set)
//
// Obviously, the advert can fail to parse, in which case errno is set to
// EINVAL.
// -----------------------------------------------------------------------------
int obstcp_client_ctx_init(struct obstcp_client_ctx *ctx, struct obstcp_keys *keys,
                           const char *advert, unsigned len,
                           const uint8_t *random);

// -----------------------------------------------------------------------------
// After connecting the socket you must call this function to get the banner
// that is to be sent to the server. This function must be called exactly once.
// To do otherwise will trigger an assertion failure. The banner returned in
// the iovec must be completed enqueued to the kernel's socket buffer before
// doing anything else with this context object.
// -----------------------------------------------------------------------------
void obstcp_client_banner(struct obstcp_client_ctx *ctx,
                          struct iovec *out);

// -----------------------------------------------------------------------------
// Same as the server version
// -----------------------------------------------------------------------------
ssize_t obstcp_client_read(int fd, struct obstcp_client_ctx *ctx,
                           uint8_t *buffer, size_t len, char *ready);

// -----------------------------------------------------------------------------
// Same as the server version
// -----------------------------------------------------------------------------
ssize_t obstcp_client_encrypt(struct obstcp_client_ctx *ctx,
                              uint8_t *output, const uint8_t *buffer, size_t len,
                              char frame_accum);

// -----------------------------------------------------------------------------
// Same as the server version
// -----------------------------------------------------------------------------
int obstcp_client_ends(struct obstcp_client_ctx *ctx, struct iovec *start,
                       struct iovec *end);

// -----------------------------------------------------------------------------

#endif  // LIBOBSTCP_H

The current state of play:

1. I've made curve25519-donna constant time, and, at the request of DJB, I'll be making it public domain too. I've been told that a group has it working in 165us and I just can't even get close to that. I spent a day rewriting the field multiplication using SSE2, but that didn't help at all. Nevermind, that group aren't releasing their source it seems so they don't count :)
2. I'm working on getting djbdns's client library in a useful state in order to integrate into Firefox. (did you know that libresolv doesn't parse DNS packets for you? How useless!) So I spent today reviewing 1000s of lines of DJB's code and adding comments.

The point of (2) is that my current plans are revolving around these ideas:

Keeping the server's public key in a TXT record for _obs_80.example.com. Why TXT? Because it's better supported. Thus clients will need a real resolver library, hence my work on DJB's code.

However, this presents some problems, mostly in the face of transproxies: If we were to connect and write a NUL (to escape HTTP) and then start with encrypting the usual HTTP stream, a transproxy will probably barf. But users can't get around them! (I need to test this with Squid tomorrow).

So one solution is for the TXT record to be able to indicate a separate port number. However, a separate port is a real pain for some sites to setup. So we might need something different. I think that might be encapsulation like:

HEAD / HTTP/1.1
Host: example.com
X-ObsReq: a4bfdz33456sdfFdsFA...
X-TransProxyDetect: flowers

The X-ObsReq carries the encrypted request, so that we don't burn a RTT on the probe. The hope is that transproxies will remove the header that they don't understand (again, have to test with Squid). The server recognises the headers and replies with an encrypted reply in the expected case. However, if a transproxy removed the header, it sends a normal reply. Thus the client can detect the proxy and back off (sadly, probably forever).

Also, is the TXT requirement too much for some? Do we need cross session state as well to get the server keys to the client? (The idea here being that a server can advertise it's keys in an unencrypted reply and the client caches the key, on disk, for some time.)

Again, it's much more messy furthur up the stack, although we don't need kernel changes.

Comments welcome. Email or comments on the post.

(updated: before I go to sleep, it strikes me that a series of POSTs with uncachable replies may work better)