Intelligence Chameleon

The IC cluster is a network designed for mass obfuscation of high profile traffic. It takes massive amounts of traffic from various customers, proxies, then obfuscates it using a device we call 'the exploder'. Each proxy listed below (onslaught and broadside) has apache2 running, with a handful of name based virtual hosts. One on each proxy, for each customer: equating to two proxies per customer. The billing for this network is done by analyzing apache logs, and writing to the database.

This is a basic layout of the entire network and cluster is as follows

 
C C Internet C C                             _____________
 C            C       ____________      ->--|             |---->->---->------- 
C C  Cloud   C C      |            |   |    |__Onslaught__|      |            |  <- Billing database
 U u u u u u  U   ->- | ServerIron |->-|      _____________      |            |        SQL inserts
 |                    |____________|   |     |             |-->--|---->-------|
 |   |                                  -->--|__Broadside__|     |            |
/|\ /|\                                                         \|/           |
 |   |                     ____________                          |            |
 |   |                    |   Ro-Tor   |  http proxy traffic ->  |           \|/
 |    -----<------<------ | (Exploder) |---------<--------------<             | 
 |                        |____________|----                                  |
 |                                          |                                 |
 |             ______________               |                                 |
/|\           |   Ip Tester  |             /|\                                |
 |            |______________|--->-->       |                                 |
 |                                   |      |                                \|/
 |             ______________       \|/    /|\                                |
 |            | Datbase      |       |      |                                 |
/|\    --<--<-|    Server    |-----<-<      |                                 |
 |    |       |______________|--->---->-----                                  |
 |   \|/               |                                                       |
 |____|______          <-------<-----------<------------<------------<---------
|           |
| Breakdown |
|___________|
 
 

This network works on the premonition that if IPS change quickly enough, it is impossible to block the service. The exploder is a BSD machine with one ip on its inside interface, and ~250 on its outside interface. The bsd machine uses a combination of NATD and IPFW to randomly choose an outbound interface for each packet. Onslaught and broadside use Ro-Tor as a gateway.

This system rotates the outside 250 ip addresses every hour on the hour, and experiences approximately 30 seconds of downtime during rotation.

IP Testing

The other peice to this puzzle is the IP testing database. This database is populated by a sentry I have written called 'Camshaft' (yes, ive named most of the components on the backend after engine parts:). Camshaft runs 24/7, grabbing ips out of the database, and testing them against the sites our customers have reported as being problematic, then reports its findings back to the database. The exploder clients (Ro-tor, for example) Read from the database and query for IP addresses that are good for the destination sites it frequently is used to grab.

IF YOU NEED TO STOP THE IP TESTER FOR ANY REASON
touch /tmp/ip_tester.kill
 
it will gracefully die. 
IF YOU DO NOT STOP IT GRACEFULLY THINGS WILL NOT WORK RIGHT LATER
ips will be marked in use, when they arent, and other things we dont want to happen.

Billing

The tertiary portion of this document describes how we actually make money off of this product.

In the diagram you will note 'sql insert statements' listed. These occur every night at logrotate time (Which right now on broadside and onslaught I think is around 6am.) Right after the rotate (postrotate) a script is run. This script is /usr/local/bin/billing-local2.

This script takes the last rotation file (the access_<customername>.log.1 file) and reads in every date, httpcode, and bytecount. It then compiles data for each day stating how many of each type of http code there was, and the total bytes sent/received for that count. It then inserts this data into a database called "intel_cham_billing" on our main database server. The finance guys and the sales guys use this data to bill the customers directly. The customers are billed on number of requests, exlcuding the 502 code.

Adding new Customers, Chapter 1 - the proxies

With the addition of CFENGINE we have made creating new inteligence chameleon customers very very simple. Just dns, cfengine and then firewall and loadbalancer configs.

Step 1. Adding the proxies to DNS

   Go add the new proxies to dns. You'lll need to add "proxy-<customer>1" and "proxy-<customer>2" in the predesignated
   format. Go into the zone files and look for "proxy-pisa", you should be able to follow from there (If you still
   dont understand, please ask another admin.
 
   In addition to the proxies themselves, You'll need to add dns for the real outside ip and the VIP.. Go look in 
   db.anonymizer.com, and db.verio and add another entry in the same format for an ip that is available below the 
   existing customers.

Step 2. Adding the proxies in the host files.

   You'll need to add entries into onslaught and broadsides host files. CFEngine creates the interfaces and proxies
   from the entries it finds in the host files on each respective machine. "Make sure you follow the existing format"
   and please double (triple?) check your work for typos and errors. 

Step 3. Run CFEngine

   SSH to onslaught and run:
     "sudo cfengine -q -v"
   Make sure you read through the log to  make sure there are no glaring errors.
   SSH to broadside and run:
     "sudo cfengine -q -v"
   Make sure you read through the log, lather rinse repeat etc.. 

=== QUICK SELF TEST! ===

   From  your workstation:
     telnet proxy-<customer>1 80
              GET http://www.google.com
              (you should see about a page of html, then it should disconnect you)
 
     telnet proxy-<customer>2 80
              GET http://www.google.com
              (you should see about a page of html, then it should disconnect you)

At this stage, we should have two proxies all setup in dns and ready to go. If these tests fail, theres something wrong. Please go back and follow the steps again, check over your work and make sure you didnt miss steps like: not incrementing the dns serial number, not adding a reverse zone, not running cfengine on one of the machines.. etc.

Adding New Customers, Chapter 2 -Firewall and Loadbalancer Configuration

First we must create the farm on the serveriron. Once you've selected a vip to use for the farm, there are entries that need to be made to SI1 and SI2. You'll need to cut/paste the following section into notepad and make the proper changes, then cut/paste to the serveriron terminal window. (note: you'll need to do this from the 'enabled' mode, in the 'configure terminal' section,(conf t))

• **** REMEMBER TO CHANGE THE SYM-PRIORITY WHEN PASTING TO SERVER IRON *****

  Serveriron1 gets sym priority 200, Serveriron2 gets sym priority 100. (higher number = higher priority)

   conf t
   server real proxy-<customer>1 10.9.60.xx
   source-nat
   port http
   port http url "HEAD /"
   server real proxy-<customer>2 10.9.60.xx
   source-nat
   port http
   port http url "HEAD /"
   server virtual <customer> 168.143.120.xx
   sym-priority 200
   sym-active
   port http
   bind http proxy-<customer>1 http proxy-<customer>2 http
 

Once you have that configured, make sure you do 'write mem' to each of the serverirons.

Now for the firewall.

 
 
  Log into galvatron (http://galvatron.verio.anonymizer.com)
  Open: Objects -> Addresses -> List
      Add the customers source IP address (should be in the RT ticket, or provided by Bruce)
      Describe it as the customers codename(name you gave the proxies)
      Add the customers real outside ip you gave them (customer.anonymizer.com, or 168.143.120.xx)
  
  Open: Objects -> Addresses -> Groups
      Create a new group - MAKE SURE that you have the proper zone selected, in this case 'untrust', in the top left.
      Describe it as The customers codename addressgroup.
 
  Open: Policies
      Select:  From: Untrust
          Create a new policy FROM (customers addressgroup) To (the customers real outside ip)

Now to test, telnet to (customers real outside ip) on port 80, and type "GET http://www.google.com"

If you get a page of HTML returned, and it appears as though it was google, all is well. If Not, Go over this documentation again, and try to work backwards. If all else fails consult another administrator.