We connect branches in one network. Reduce Internet Costs

I greet you habrazhitel, not so long ago I faced the task of connecting the branches of one large company scattered across Siberia into a single network. The main problem was that OpenVPN had to be forced to work on top of unstable PPPoE while simultaneously sending all traffic through OpenVPN. The


initial goal was to save money on Internet traffic in branches, as in remote areas, the price of unlimited ADSL with a width of 256kb / s cost about 7-10t.r. per month, and the Internet was vital.
All the joy was that almost all branches had connections of one provider, in which there was a concept of local and peer-to-peer traffic, and the Head Office had a dedicated broad Internet (another provider, but by chance he was loyal to the branch provider and he had “ peer-to-peer traffic ”with a price of about 6 kopecks per megabyte).

1. proxy

The fastest solution was the usual cascade of proxy servers, and this was done because Before all the branches handed out the Internet directly to their modem, it was necessary for everyone to allocate 1 system unit that would act as a gateway, the system engineers were not gifts, who would give the 800th stump, who would give 233, in general, who had what ... Although today it’s 4- 7 tr You can collect a decent gateway, but the master is a master, I want to say without costs!

Ubuntu 8.04 was installed on these gateways. LTS is configured as a gateway so that it plugs into the local network, into the modem and into the outlet, and everything worked right away. in many branches, admins could only press “Any key” on the user's keyboard, but it doesn’t matter, things went, gradually 7 branches reconfigured their modems, and stuck gateways :)

immediately raised the proxy in a cascade, taxied the http traffic there, but as we all know, https traffic is only a certain% of the total traffic, switching to simpler tariffs was a saving, but conditional, because the negligent admin or user could, for example, pull through torrent that it’s significant, which promised a branch getting money ...

Along the way, other tasks appeared at the central office - transferring the comparative postman, gateway, portal configured in 2002 and not touched since then, but it deserves a separate article ...
But for now we are just interested in the network ...

2. OpenVPN

I saw this thing for the first time, there was a certain fear of the first acquaintance, then reading manuals and the Internet, I rolled my sleeves up and put it :)

2.1 The server
has 2 network adapters eth1 (192.168.5.x) - Local network and eth0 (real ip 111.111.111.111) Internet with a wide channel.

apt-get install openvpn

Next, we create a server configuration file
touch /etc/openvpn/server.conf

when the system boots up, all VPN connections are automatically raised, for which there are corresponding files with the extension .conf in the / etc / openvpn folder

, I got it like this. We create a directory in which individual client settings will be stored: Now we need to create keys and certificates for encryption and authorization Now we will create a certificate and a private key for the server:

port 1194 #Порт
proto udp #Протокол
dev tun #Название виртуального устройства
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key # This file should be kept secret
dh /etc/openvpn/dh1024.pem
server 10.10.10.0 255.255.255.0 # vpn subnet
ifconfig-pool-persist ipp.txt # Тут будут храниться ip адреса клиентов
push "route 192.168.5.0 255.255.255.0" # home
keepalive 10 120
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
log-append openvpn.log
verb 4
mute 20
client-to-client
client-config-dir /etc/openvpn/ccd # Тут будут настройки для каждого филиала
route 192.168.0.0 255.255.255.0 # Маршрут от сервера до филиала 1
route 192.168.1.0 255.255.255.0 # Маршрут от сервера до филиала 2




mkdir /etc/openvpn/ccd



cd /usr/share/doc/openvpn/examples/easy-rsa/2.0
source ./vars
./clean-all
./build-ca



./build-key-server server

We create a key for the client (if there are several clients, the procedure will have to be repeated):

./build-key client1

each client must have its own unique name (in this case client1).

if a new client is created after some time, the procedure will look as follows: Generate Diffie-Hellman parameters: Put the following files in the directory / etc / openvpn / Create the file /etc/openvpn/ipp.txt The client machine configuration file / etc / openvpn / client .conf I got something like this Now you need to copy the generated client keys and server authoritarian certificate from the server to the / etc / openvpn / folder: If the client is hiding the network 192.168.1.x, so that the server sees it, you need to add a route to it on the server .

cd /usr/share/doc/openvpn/examples/easy-rsa/2.0
source ./vars
./build-key client2




./build-dh



* ca.crt
* server.crt
* dh1024.pem
* server.key





remote 111.111.111.111 1194
client
dev tun
proto udp
resolv-retry infinite # this is necessary for DynDNS
nobind
user nobody
group nogroup
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client1.crt
key /etc/openvpn/client1.key
comp-lzo
verb 4
mute 20
redirect-gateway
#show-net-up
verb 4


* ca.crt
* client1.crt
* client1.key



On the server, we create the file / etc / openvpn / ccd / client1 with the following content: Here, in fact, the most evil problem happened to me. OpenVPN received a directive (if there is a 'pull' in its configuration), the client does not delete the old route, but adds entries to the routing table: and if openvpn goes over ethernet then everything works and pleases the admin and users, but the great ppp likes to pick up such route. And OpenVPN swears like this. The solution to this problem was searched for a long and tedious, although it is on the surface. if in this "curved" ppp route you specify a gateway instead of 0.0.0.0 a real gateway, then OpenVPN sees this route and adds its own without problems. So I created a file

iroute 192.168.1.0 255.255.255.0
# роутинг на сеть филиала2, чтоб 2 филиала знали друг друга
#push "route 192.168.100.0 255.255.255.0"
#Заворачиваем весь трафик в OpenVPN
push "redirect-gateway def1"




push "redirect-gateway def1"

0.0.0.0/1 via 192.168.231.5 dev tun0
128.0.0.0/1 via 192.168.231.5 dev tun0

0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 ppp0


Jul 2 19:28:53 ino ovpn-client[14465]: NOTE: unable to redirect default gateway -- Cannot read current default gateway from system




/etc/ppp/ip-up.d/routing
In which I registered a small script. (I ask you not to kick with your feet, I am very lazy, and did not write a full-fledged script for determining the gateway and editing routes, but made it as a simple crutch)
I would be very happy if someone offers a more logical, reliable, and universal method for editing routes on the fly.
So far, there is no certainty that everything will work 100% with ppp breaks, But life will show, if anything, I’ll fix the topic. We make it executable. After that, reboot, after some time, the server stopped pinging on the external ip, but began to respond on the internal one - 10.10.10.26 PS I ask you to take into account that the firewall of the gateway and server and client need to be fixed so that users have vital important internet. For example, I did it like this:

#! /bin/sh
#Определяем выданный шлюз по умолчанию у меня он всегда разный но в сети 222.х.х.х
gw1=`ip route show | grep 222 | awk '{print $1}'`
# Удаляем 0.0.0.0 0.0.0.0
route del default
# Добавляем маршрут с верным шлюзом
route add -net default gw ${gw1} dev ppp0


chmod ug+x /etc/ppp/ip-up.d/routing





-A POSTROUTING -s 192.168.0.0/255.255.0.0 -j MASQUERADE
There is no link to either the external interface or ip :),
when for various reasons we don’t have openvpn, then users will have a direct Internet, and when it appears, then all traffic will fly through it.

Conclusion

In life, this system is assembled in stages, first the VPN server and VPN client are launched, ping each other at the addresses 10.10.10.x then routes to the networks that are behind the server and the client are added, pings are checked when everything is stable and reliable we add the
push "redirect-gateway def1"
AND directive we get work and life again, everything was done without leaving the office, on the branch gateways, network interfaces were signed, so that the administrator simply plugged in the network cables and power, called me, and then I configured ssh to work.

Oh yes, I almost forgot about the most important thing - profit

In addition to the fact that now the entire network is accessing the servers and services of the branches and the center by internal IP addresses, there is also financial savings.

Previously, each branch spent on the Internet an average of 7,000 rubles. per month, now per month each of them pays 550 rubles. for access to peering, the Internet is not consumed (except for the central one), for starters, 7 branches were launched, then there will be more.
it turns out that for a year under the old scheme, the company would spend 588,000 rubles on the Internet . , and with the current scheme, 46,200 rubles will be spent per year .

What's next?

And now, on this whole heh ... we’ll try to take off! I will try to deploy IP Telephony in order to minimize the cost of long-distance telephone calls between branches, to connect softATS with hardware in branches, which I will write about. Good luck

update1 Many questions have arisen for the term “Large company” I will try to clarify.

It is large in Siberia, the center has a staff of 200 employees, branches from 20 to 60, 10-20 objects are still attached to each branch. 5-15 people each. branches less than 30.

The company is not very agile, the main control comes from the state, equipment, the company takes some steps towards IT development :)

Update2
And so, after testing it turned out that if you lower the administration server for a long time, then clients are not willing to openvpn and traffic can go directly to an expensive Internet.
Also, if the ADSL channel is forcibly torn, then the open server seems to be trying to start again, but something it doesn’t work for, it goes around in circles.
I tried all sorts of options and keepalive and ping-restart and so on ... did not help ...

Therefore, we write a small script that will check the state of affairs.
touch /usr/bin/vpn_keepalive.sh

With content. We make it executable. The script pings the host, and if 0 packets are returned, it will execute the restart command. after which everything is guaranteed to rise, the correct routes flow in, and the traffic goes through VPN traffic. We throw this script into crowns for example every 2 minutes.

#! /bin/sh
# Нужна ли нам отладка, рас комментировать нужное
#debug_out=/dev/null
debug_out=/dev/stdout
#NEXTHOP Это хост внутри ВПН сети
# я взял внутренний ип OpenVPN сервера
NEXTHOP=192.168.5.1
# Комманда перезапуска Опенвпн
OPEN_VPN_CMD="sudo /etc/init.d/openvpn restart"
PING=/bin/ping

logger_opts="-t $0"
if [ "$debug_out" = "/dev/stdout" ]
then
logger_opts="$logger_opts -s"
fi
pckts_rcvd=`$PING -c 8 -q -W 2 $NEXTHOP | grep transm | awk '{print $4}'`
echo "host: $NEXTHOP, pckts_rcvd: $pckts_rcvd" >$debug_out
if [ $pckts_rcvd -eq 0 ]
then
echo "Connection with $NEXTHOP lost, resetting" | logger $logopts
$OPEN_VPN_CMD > $debug_out
else
echo "Connection with $NEXTHOP up, no action" | logger $logopts
fi


chmod ug+x /usr/bin/vpn_keepalive.sh





crontab -e


0-59/2 * * * * /usr/bin/vpn_keepalive.sh

And all if debug is enabled, then in the logs (syslog) it will be marked like this.
logger: Connection with 192.168.5.1 up, no action

Good luck