Managed Services

   

q!Bang was recently awarded the opportunity to implement a FOSS VoIP solution for Firefly Energy as they expanded into a new facility. Stephen Hultquist has written an excellent blog at InfoWorld about managed IT services. Why not leave your IT to the experts while you concentrate on your core. After all isn’t that why you are in business?

802.11b/g and Bluetooth

   

I knew for a long time that they overlap in frequency, but never knew the exact range. I recently bought a bluetooth headset for my phone, and some people have complained about hearing statics when I talk to them. I was curious if it’s because my 802.11b/g AP is getting in the way, so I did some look up:

Bluetooth: 2402MHz - 2480MHz
802.11b/g Channel 1: 2401MHz - 2423MHz
802.11b/g Channel 2: 2405MHz - 2428MHz
802.11b/g Channel 3: 2411MHz -2433MHz
…
802.11b/g Channel 11: 2451MHz - 2473MHz

(See the spectrum chart here for a graphical view.)

Well well, so it looks like there is no escape from Bluetooth if you’re using 802.11b/g… If you set your AP to channel 1, at least you avoid 1MHz of the spectrum

High Availability with Open Source

   

(Article originally posted at InfoWorld Magazine)

When I build a network or a system, I try my best to make sure that everything is as redundant as possible: redundant power supplies, RAID for the drives in case of a hard drive failure, backup routes in OSPF in case someone trips over the network cable… you get the idea. But what happens if the CPU overheats in one of the web servers and causes it to crash? Or what if someone yanks the network cable from your LDAP server? Or if someone flips the switch and accidentally turns off the accounting database server? If you incorporate High Availability (HA) into your system design, the answer is “nothing”. Your web site will still be running, your network users can still login, and the accounting department won’t notice any glitch. You don’t even have to leave your desk.

Open Source makes HA easy to implement, with offerings like heartbeat, keepalived, and CARP.

Let’s say you have mostly static web content for your web server and you can fit everything on a 2GB compact flash card. Then you can build two solid state machines using the Debian Router Project. Using simple solid state hardware means less moving parts and less likelihood of a hardware failure. Then you can use heartbeat to create your HA web server cluster. If you have content which changes more frequently, like the leases file for a DHCP server, a database, or a file server, then you should look into using DRBD to synchronize the two file systems.

Heartbeat requires you to setup a private link for the two machines (nodes) to communicate, so they know the other node is still alive. While you can just use a crossover cable to connect the two nodes I would strongly recommend that you install two network cards in each node and setup a private VLAN or network just for the heartbeat communication. This will give you a little more flexibility later. You will need 5 IP addresses total, two for the private heartbeat link, two public ips - one for each of the nodes (if you wish to manage them remotely), and one more public ip for a “virtual” IP address that is held up by the heartbeat software. This virtual ip address is the IP address your users visit. (By the way, heartbeat supports IPv6)

Once you have heartbeat configured on both nodes and have designated one of nodes to be the master, the two will start “pinging” each other over the private link. Now to see it in action: Start a ping to the “virtual” IP address, and unplug the network cable for the master node or just shut it down to simulate a disaster. You should lose a few pings, but in just a few seconds, the backup node will realize that the master is no longer responding, and will take over the virtual ip address and reply to your pings. This means, if one of the nodes failed in production environment, users will only experience seconds of outage, instead of minutes, or dare I say, hours.

Now, if you have followed my advice about putting the heartbeat link on its own VLAN instead of just hooking it up with a crossover cable, you have the flexibility to move the backup server to a different location in the building (or however far you VLAN will reach). Why? This protects you from a bigger scale of disaster, say, a power outage for the entire room, fire, or flood (hey, I’ve seen it happen). If you have the two heartbeat hosts separated physically, you stand a better chance of surviving the disaster. Plugging both machines into the same network switch creates another single point of failure, so it is highly recommended that your backup machine be connected to a different network switch, and preferably a different power grid.

keepalived uses VRRP (Virtual Router Redundancy Protocol), a widely supported protocol amongst routers. This means it can be integrated nicely into your existing network infrastructure. keepalived was originally designed to work for multiple routers, and it works pretty much the same way heartbeat does, except keepalived does not need a dedicated private link, and it is easier to setup more than two nodes. (It is unclear whether or not keepalived currently supports IPv6)

So far you’ve achieved automatic fail-over. But don’t you feel that all these back up nodes sitting around is a bit of a waste? Can you leverage all those idle computing power? You mean you want load balancing on top of your HA functionality? Open Source answers with CARP (Common Address Redundancy Protocol). The OpenBSD team released CARP in 2003 as a replacement and enhancement to VRRP, it features:

• Secure “heartbeat” communication
• No need for dedicated, private link for “heartbeat” communication
• Basic load balancing functionality
• Supports IPv6
• Available for OpenBSD, NetBSD, FreeBSD, and Linux (implemented as ucarp)

You can also combine CARP with pfsync (OpenBSD’s packet filter), and now you can build a cluster of firewalls/routers that are always online, load balances amongst each node, and in case of a failure, users do not lose any sessions or states.

In conclusion, heartbeat (along with DRBD) is the easiest to setup for a 2-node cluster, Keepalived integrates well into your VRRP environment, and CARP brings security and load balancing to the table. In case you are wondering how mature this technology is, heartbeat has been around for years, and has a list of success stories.

Josh Kuo
Co-Owner of q!Bang Solutions

Beef Up Your Wireless Router

   

(Article originally posted at InfoWorld Magazine)

Sure you have one. Everyone nowadays has at least one wireless router at home, be it Linksys, NetGear, D-Link, or Buffalo. With new wireless products being released nearly every month, I am willing to bet that some of you even have a couple of the older wireless routers collecting dust in your closet. Well, it’s time to take them out and put them to good use.

Check out the OpenWRT project. OpenWRT is a Linux distribution for embedded devices, and it brings a lot of exciting possibilities to your humble wireless router. Although still in its release candidate stage (currently at RC6), OpenWRT is very usable and feature-rich right out of the box. Be warned, you could void your manufacturer warranty by installing OpenWRT on your wireless routers.

So what can you do with an embedded Linux device running on limited RAM and very small storage? As it turns out, quite a lot actually. You can install asterisk, and have your personal, customizable PBX (private branch exchange). If you already have a SIP phone or some kind of VoIP phone interface (such as the Cisco ATA 186 adapter), you can have your very own VoIP system at home, all running out of your low power-consumption embedded hardware.

Put your router/firewall on steroids by installing packages like nmap (network security scanner), snort (intrusion detection), and tcpdump (packet sniffer). Together with iptables (which comes with the Linux kernel), you can turn your OpenWRT box into a powerful security tool. Install openvpn, and you have a very affordable VPN device. And if it strikes your fancy, you can install quagga and turn your dusty little Linksys into an OSPF and BGP-capable router.

Want to provide your own wireless hotspot? No problem. Install chillispot, and you are ready to go. You can even install FreeRADIUS on the OpenWRT for the authentication back-end, and WPA (wifi protected access) for the added security.

You can turn it into an all purpose office server by installing DHCP, cups (print server), lighthttpd (web server), NTP (time server) and OpenSSH or dropbear (secure remote administration). If your router has a USB port, you can also turn it into a file server by hooking it up with a USB hard drive and installing NFS.

And don’t forget that this is a wireless router. It has a wireless card, so take advantage of it! Install kismet on it, and you have a wireless sniffer. This can prove to be invaluable if you ever need to analyze the airwaves at a remote location, but don’t want to leave your expensive laptop on-site. Drop in place a $50 OpenWRT box loaded with kismet instead.

Here is one way to use your old wireless router: In the past, I had setup a few cheap Linksys WRT54g boxes with OpenWRT and vtun, and dropped one at each of our remote locations. This gave me the ability to have layer 2 tunnels to each of the remote sites. I kept one in my house, and if I ever needed to troubleshoot a remote network problem, I just setup the tunnel between the two OpenWRT boxes, connected my laptop or testing equipment to the OpenWRT sitting on my desk, and it was like being on the remote physical network! This saved me a number of times, being able to perform packet capturing on the remote network, observing the network traffic in real-time, requesting and obtaining DHCP addresses… essentially, I could experience exactly what the remote user was experiencing, all from the comfort of my own home.
This is just the beginning of what embedded Linux can do for you. To find out more what embedded Linux can do fo r your enterprise, check out Secure Linux Appliances in Your Enterprise. So dig up your old wireless router, check it against the hardware compatibility list, and see if your router is OpenWRT compatible, and open yourself up to a wrt of possibilities!

Josh Kuo
Co-Owner of q!Bang Solutions

Secure Linux Appliances in Your Enterprise

   

(Article originally posted at InfoWorld Magazine)

By now you’ve either seen them or read about them. Companies are selling all kinds of useful appliances based on embedded Linux. Some are for small tasks like wireless APs, mobile devices, or cell phones. Others are geared towards enterprise needs like load balancers, routers, and NAS (network attached storage) and SANs (storage attached network). They all run some version of Linux or BSD. You know you have a couple of Linux geeks working for you in the IT department. Why aren’t they coming up with some of these cool Linux appliances for your own company to use? The excellent Debian Router project by Vadim Berkgaut is the help that your Linux admins need to develop their very own Linux appliances.

At my company, q!Bang Solutions, we provide all types of IT solutions, but our strong suit is our solutions built upon Open Source software. Our employees have used the Debian Router Project (which we refer to as “DebRouter”) to build numerous solutions, including firewalls, OSPF and BGP routers, DNS servers, and even VoIP servers. DebRouter is a cornerstone of our technology solutions.

What’s great about DebRouter is that you get a fully functional Debian Linux installation. So you can add whatever software packages you want to extend the functionality of the DebRouter. This is implemented through the usual Debian package management utilities, which means that you can change a DebRouter’s functionality on the fly and in the field after it’s been deployed.

Another important feature of DebRouter is that it boots from a flash device like a compact flash card (via an IDE adapter) or a USB flash drive. So if there are any problems with changes you’ve made, a reboot takes you back to the previous known-good version of your running system. Does this mean that you lose changes you’ve made when power to the DebRouter goes out? No. DebRouter implements a “write to flash” function much like a hardware router or manageable switch. So you can install and configure new packages, test them out, and write your changes to the flash-based boot media if everything went well in testing. If your tests revealed there was a problem, then just reboot without writing the changes to flash and you will roll back to the same state of the filesystem that you had before your changes. This makes it extremely easy to test potentially unstable software and configuration changes. If things don’t work, just reboot, and voila! Your working system is back within seconds.

This also means that the machines are harder for crackers to abuse if they succeed in infiltrating the DebRouter. If you discover that your DebRouter has been compromised, you can reboot and be rid of the cracker. Then you check for security updates from Debian, install them, write your changes, and you’re back up and running. I can tell you from experience that eradicating a cracker’s presence from a normal machine with hard drives whose data persists across reboots is not this easy!

The boot process of the DebRouter provides another nice benefit. DebRouter boots from flash media, creates a RAM disk, copies the flash media’s filesystem to the RAM disk and then unmounts the flash media filesystem and runs from the RAM disk. RAM is fast - lot faster than any hard drive. So now your filesystem I/O speed is absurdly fast. So if you install the Apache web server and put up some HTML and image files, you now have one of the fastest web servers available - without the hassle of a special configuration to load your pages into a ramdisk. It can also run web scripts (such as PHP, Perl, Python, Ruby, etc.) as fast as your normal hard drive based servers do.

What can you build with a DebRouter? Here are a few ideas to get you started:

• Add the Quagga routing software package to make an OSPF/RIP/BGP router
• Install the Apache web server with Perl/PHP/Python/etc scripting environments
• Use the Asterisk software for a cheap VoIP server for a remote office
• NAT/Firewall
• Web content filtering via the Squid proxy package
• Make a captive portal system for wireless networks in cafes or other public access areas
• DNS server using the venerable and always popular BIND software
• Create a network sniffer with the tcpdump utility which writes data to a remote NAS or other storage device
• Combined with a NAS (Network Attached Storage) or an NFS server, a DebRouter can do most anything.

Since most enterprises will try to install all machines in racks, I checked a couple of online vendors to see how much it would cost to build a good 1RU DebRouter machine. I found that a 1RU machine far above the minimum specs can be had for $500, including shipping. This includes a 1RU case, motherboard with all essential functionality on board, a P4 2.8GHz CPU, 1GB ram, and a 512MB CF card and IDE-based CF reader.

So how about a $500 router that can do RIP/OSPF/BGP? Consider both the business and technology reasons that your company might want to use a DebRouter instead of a router from Cisco or one of the other routing big boys. The business side is easy. The hardware is cheap, even for a system with generous amounts of RAM and CPU. For the price of a typical router support contract, you can buy a couple of extra DebRouters to have sitting around as spares ready to jump into action if you have a hardware failure on your primary DebRouter. Subsequent years of support contracts you don’t need to buy equal money that remains in your coffers helping to fatten up your Christmas bonus next year. Of course, let’s not forget that most router vendors charge extra for the advanced software like OSPF or BGP routing, or encryption software so that you can use the more secure SSH instead of the gaping security hole called Telnet to remotely connect to your router. DebRouter has all that (and so much more) for free!

On the technology side, with the screaming fast processors available today, a DebRouter can pretty well hold its own against most of the major router vendors’ offerings. And it’s the versatility of the DebRouter that will likely interest your techies. Did I mention that Linux does 802.1q VLANs? How about an OSPF router that does double duty as a slave DNS server? Or perhaps an edge router that also acts as a VPN concentrator with strong encryption for hundreds of tunnels?

So walk on down to IT and find those two Linux guys tucked away in their cubicles and let them loose on a Debian Router project. They should be glad to have an interesting project to work on instead of trying to recover emails that Marge from Accounting accidentally deleted the other day, and you just might get some nifty devices from them that save you some cash on your bottom line. Your Linux admins are welcome to reach out to me if they need some help or just want to share their ideas on a new use for a Debian Router.

In the future, I’ll touch on embedded Linux in extremely cheap devices that are excellent for smaller tasks.
[My q!Bang Solutions co-owner Josh Kuo beat me to the punch. Read his article “Beef Up Your Wireless Router”.]

High Mobley
Co-Owner of q!Bang Solutions

Computer security explained for the masses

   

(Originally posted on InfoWorld Magazine)

It is often cited that the biggest issue in the fight against worms and viruses and other such malware is uneducated users. If a person doesn’t understand why it’s a bad thing to open email attachments from people that he doesn’t know, then you can bet that he will open every attachment which comes to him. Several email clients (not just MS Outlook!) will happily open and execute any Visual Basic or batch file that a user clicks on. Then wham! - You’ve got an infected machine that’s probably already calling home to the nasty individual who wrote the malware and now “owns” the user’s computer - which you as the IT department have to go and fix…

Of course the various network security and bug tracking sites are great about announcing the security flaws and exploits that are found, but arguably their audience is only people who are already pretty savvy about security issues. So I was pleased to see an article written more for public consumption at howstuffworks.com today, entitled “What’s the problem with Microsoft Word?”. The author, Julia Layton, does an excellent job of explaining some computer security jargon and bringing the layman up to speed with the MS Word zero-day flaws which were recently announced. I hope that this is a sign of a new trend of educating the end user in a comprehensible language.

When I was a full time sysadmin and helpdesk tech responsible for a few hundred users and 50 servers, I struggled to explain the same topics to the many end users individually. So instead, I sent out ocassional messages via email with some helpful tip on how to use their computer or a link to a web article that contained some useful information on a subject that I knew would tweak their interest. So I always had these sorts of articles bookmarked to send out to my users. They appreciated that I was trying to educate them and I appreciated that I had fewer infected machines to reformat and reinstall.

High Mobley
Co-Owner of q!Bang Solutions

Open Source Network Monitoring at SCALE 5x article posted at InfoWorld Magazine

   

High Mobley’s article Open Source Network Monitoring at SCALE 5x has been posted to the InfoWorld Magazine web site. In the article he talks about three major network montioring products that were being exhibited at the SCALE 5x event in Los Angeles last week. Comments are welcomed and appreciated.

High Availability with Open Source article posted to InfoWorld Magazine

   

Josh Kuo’s article High Availability with Open Source has been posted to the InfoWorld web site. Josh gives the low down on three of the more popular Open Source high availability (HA) apps out there, offering pros and cons and explaining the optimal situation in which each one should be deployed. Comments are welcomed and appreciated.

Beef Up Your Wireless Router article posted at InfoWorld Magazine

   

Josh Kuo has posted his article Beef Up Your Wireless Router on the InfoWorld magazine web site. This is a good overview on the opportunities presented by instaling the Linux based OpenWRT OS on your wireless router device. As always, your comments are appreciated.

Secure Linux Appliances article posted to InfoWorld Magazine

   

High Mobley’s article Secure Linux Appliances in Your Enterprise has been posted to the InfoWolrd magazine web site. This article is an informative overview of the Debian Router Project, and the myriad possibiltiies that it presents for Linux appliances that you can easily make yourself. Your comments are appreciated.

Computer Security Explained Article Posted to InfoWorld Magazine

   

The article Computer Security Explained for the Masses by q!Bang co-owner High Mobley has been published on the InfoWorld magazine blog site. Please give it a read and post your comments.

Cisco owners be very afraid

   

Cisco Systems Inc. security has announced 3 remotely exploitable flaws for the Cisco IOS software:

• Crafted TCP Packet Can Cause Denial of Service
• Crafted IP Option Vulnerability
• IPv6 Routing Header Vulnerability

Usually these security notifications are released to large customers before the general public, so large customers have time to update or protect their equipment. However, it’s the smaller networks that are at the greatest risk. Many don’t have Cisco support contracts (or can’t afford them), or don’t have an individual on staff to upgrade their equipment.

When will the first exploit code be released? Will anybody admit to being compromised by the exploit? How will a common user realize they have a problem? The small business customers who think owning Cisco is the way to go need to address the total cost of keeping those systems up to date. Many times a customer won’t upgrade a core router or switch because they don’t know how or don’t know they have a problem. How many service providers will contact their customers warning them about these flaws? Most small businesses don’t have a clue if they are vulnerable or not. How does Cisco fix this issue? What means does a small company have to keep all their systems up to date? Most end users barely can keep up with Windows, virus, adware and spyware updates little alone keeping up with all their network equipment. How many people have updated the software on your home router?

I believe the next great worm will be targeted towards networking equipment. How about taking over all the Linksys routers/access points and making them spam bots or open relays? What about using a Cisco vulnerability to create tunnels to specific locations to monitor all traffic through a router. More to come …