DokuWiki

Try#2 : Dec 17, 2013.
Result: Success.

Unable to deploy new server on the ExpertusCB network on OpSource Cloud.

We have a Linux Box a.k.a Frontend server - Apache (mod_proxy). As of Dec 4 2013, the team has assigned a set (10) of private IP addresses directly as pseudo interfaces on the box.

So the box's network conf looks something likes this: eth0: ⇒ (from VMware / OpSource when provisioned) eth0:1, eth0:2, eth0:3, eth0:5, eth0:6, eth0:7, eth0:8, eth0:9, eth0:10, eth0:11 ⇒ Assigned by team directly within the same range. (ifconfig output at the bottom of this message).

This configuration works fine; until we deploy a new box on this network. As soon as that is done, the new box gets stuck on Step 11: Waiting for Guest IP. I believe VMWare or the provisioning software (on OpSource) does not have any knowledge that the IP is being used and resulting in an IP conflict ( Vmware is trying to assign .18 and its already used on the linux box ).

These are some possible solutions:

1) Reserve (mark as used in vmware / provisioning tool) certain range of IP address from our allotted private ip address pool (The first 40 or so).

2) Reconfigure the box with ip address in the far range of our private pool.

3) Use multiple public address to private address mapping, possibly using the VIP function.
== Notes on solutions / observations ==

#1 is pretty self explanatory. No reason to hope that OpSource will make an exception and custom configure their VmWare physical host for us.


#2 The cloudUI allows us to specify the private IP but does not allow us to enter the corresponding public IP address. Furthermore, we need to know how long will it take for the an “unNAT” followed by a NAT operation for a public IP.
Generally, the latter should not be an issue but we would like a confirmation on that regardless.

Update: 7:20 PM. Opsource confirmed that it should be almost immediate, as expected.

It has been observed that OpSource consistently re-assigns the recently released IP. This could be used to our advantage by crafting a timed unNAT and NAT operation sequentially.

Update: 7:20 PM.

Spoke with amazing Shruti Mahadevaiah @ Opsource.
She understood the problem and agreed with my observation above.
She, however, could not guarantee that the unNAT/NAT procedure will result in allocation of the public IP address that we desire. She also mentioned that there is no way for her to do it from the backend “it will result in the frontend not reflecting the backend”.
She also stated that we have two more public IP's available.

The IP allocation process:
A block of 8 IPs is assigned to each network (upto a total of 3). With each NAT request, the next available public IP is assigned. No over allocation is possible and when all IP's are assigned, the system alerts the user that all public IP's are exhausted. At this point, if you unNAT/NAT a public IP, the system is guaranteed to allocate the just released IP.

Suggested workaround:
–> NAT 10.166.152.155 and 10.166.152.156 (unused private IP's) At this point, our network public address pool is exhausted.
–> Start with the checklist.


Implementation Checklist:

• *You might need IP ADDRESS ASSIGNMENT OPSOURCE
  Start loop :
  
  1) Pick the next unmodified interface and note its IP address … IP_A.
  2) Pick the highest “farthest” IP from the “APP NETWORK POOL RANGE” … IP_B.
  KEEP THIS OPEN –> One time: Login to CloudUI, US EAST, Gear symbol on EXPERTUSCB network, NAT tab (bottom). 3) NAT → Search carefully a find the IP_A NAT entry. Make a note of the public IP … IP_P
  downtime imminent 4) Delete the NAT entry found in #3 and immediately enter IP_B and submit.
  5) Once the request goes through, confirm IP_P matches with #3.
  6) Replace IP_A with IP_B on the Front end Server.
  7) Verify connectivity from at least 3 different networks.
  8) Sync IP with the /etc/sysconfig/network-scripts/<INTERFACE_NAME_PICKED> file.
  9) Replace in Apache conf.
  End Loop when all interfaces modified:
  
  
  Testing Procedure
  Each private IP ↔ public IP pair is tested at step #7 to ensure successful translation.
  Additionally, once the procedure is completed i.e. all IP's are implemented, all the public IPs assigned to clients so far must be tested by making a HTTP request to their domain via a browser.
  Similar to step #7, this test must be done from at least 3 different networks.
  
  
  Rollback Procedure **
  

The checklist ensures that if for some reason, we are unable to allot the exact same public IP to the same client (i.e. we are unable to assign the same public IP to a different private IP), it will be known on step # 7 and the procedure should be deemed as failed. In such an event, the first and only IP that has been assigned should be rolled back in the following manner:

1) Replace IP_B with IP_A on the front end server.
2) Revert the IP_B NAT entry to IP_A in Opsource CloudUI.
3) Test #1 and #2 in TCP and then HTTP Mode (By pinging and by requesting a document by using clients' domain)

#3 Theoretically, since traffic is terminating at the Frontend server, we should be able to utilize your VIP functionality to MAP all the public IP address to a single private IP address. I need to know if there is an option to support traffic originating from the private IP. Case opened with OpSource. Update: Opsource confirmed that this is possible and supported.

== Post Solution Steps ==

Dec 18, 2013.
1) Document the new IP mappings in Wiki → Refer table below:

2) Update the IP allocations table. Updated → [https://wiki.exphosted.com/doku.php/ip_assignment?s[]=ip&s[]=allocation]
3) Update Zabbix Server IP address in monitored systems: [https://tracker.exphosted.com/view.php?id=4401]

== Additional Info ==

Two cases have been opened with OpSource so far: 17855 and 17860. Any other possible solutions are also welcome from either the team or OpSource.


[root@10-166-152-14 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.18 Bcast:10.166.152.255 Mask:255.255.255.0

eth0:1 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.17 Bcast:10.166.152.255 Mask:255.255.255.0

eth0:2 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.14 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:3 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94 inet addr:10.166.152.19 Bcast:10.166.152.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:5 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.21 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:6 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.22 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:7 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.23 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:8 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.24 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:9 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.244 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:10 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.245 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0:11 Link encap:Ethernet HWaddr 00:50:56:9B:1A:94
inet addr:10.166.152.242 Bcast:10.166.152.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

===Status History===
Try#1 : Dec 14, 2013.
Result: Failed.
MethodYoga brought up Apache default page even after IP change.
Page to be updated with additional information.
End Try#1 Info.
Updated with Testing and RollBack Procedures @ 9th Dec
Postponed due to deployment issues @ 4th Dec
Received information from OpSource @ 4th Dec
Waiting for info from OpSource Support @ 4th Dec
Created. @ 4th Dec