A good solution applied with vigor now is better than a perfect solution applied ten minutes later.
- General George Smith Patton III (source)

Most readers are probably familiar with X.509 certificates as used with TLS/SSL to secure websites with strong encryption. For commercial websites, this usually means presenting a digital certificate which has been verified and signed by a certificate authority , once the secure connection is established, user credentials are passed, e.g. username/password; however, X.509 certificates may also be used on the client side to supplement or even eliminate the need for users to enter passwords.

Intranets and temporary web applications are two examples wherein developers may choose to forgo the use of a widely accepted certificate authority in favor of a self-signed certificate. Naturally, this is a compromise which may or not impinge overall security depending on factors which will not be considered here.Strictly considered, this text has little information on security at all, rather, it is simply outlines an expedient way to set up X.509 client certificate authentication.

1. The Server Certificate

The following command will generate a self-signed web certificate and unencrypted key (no password required).

openssl req -new -x509 -nodes -out server.crt -keyout server.key -days 1825 \
 -subj "/C=US/ST=NY/O=Example Inc/CN=example.com/emailAddress=info@example.com/"

2. Configuring Apache

This example illustrates enabling SSL in the apache config, note that because this is a self signed certificate, the same file is used for the Certificate and CA Certificate. The <Location> directive specifies where client certificates will be required; visit the Apache site for additional relevant directives.

<VirtualHost 192.168.1.1:443>
  DocumentRoot /var/www/example.com
  ServerName example.com
  ServerAlias *.example.com
 
  SSLEngine on
  SSLCertificateFile conf/ssl/example.crt
  SSLCertificateKeyFile conf/ssl/example.key
  SSLCACertificateFile conf/ssl/example.crt
 
  <Location /clients>
    SSLRequireSSL
    SSLVerifyClient require
    SSLVerifyDepth 10
  </Location>
</VirtualHost>

3. Client Certificates

The server certificate is used to generate a client certificate using the PKCS#12 standard

openssl pkcs12 -export -out example.pfx -in example.crt -inkey example.key \
-name "Example Client Certificate"

If a password was entered during the previous command, users will need to enter the same password when installing the certificate. Firefox users can import example.pfx by navigating through:

Prefs ⇨ Encryption ⇨ View Certs ⇨ Your Certs ⇨ Import

Upon visiting https://example.com/clients users will be prompted to present the client certificate.

I generally favor ‘net neutrality, and I certainly don’t take a kindly view of the arbitrary packet discrimination employed by unscrupulous companies; left unchecked, such practices easily (perhaps inevitably) lead to “the pseudo service scenario of bribery … extortion“, but the same slippery slope analogy could slide the other way. Had the appellate court ruled in favor of the F.C.C. it would have set a precedent for allowing a regulatory authority to essentially invent new powers not specifically delegated to it by any act of Congress. If you would prefer that Congress pass such a law, you may wish to ask your representatives to support H.R. 3458.

Now some may ask, “Why automount removable media at all?” It is unwise to remove an active device, such as unplugging a USB drive without first unmounting it, and automounting may encourage this sort of recklessness. I don’t contend this, but if one runs a server using an external USB drive, there are two words which should spark an immediate interest in automatic mounts: power failure.

To manually mount an external USB drive, a user might enter a command such as: mount /dev/sdd1 /mnt/usbdrive. Naturally this assumes that sdd1 is the correct device node, and that’s the thing about mounting things like USB drives– there is no guarantee which device node it will use.

Fortunately UDEV, which is what handles all those devices in the /dev directory, has a rules syntax to control how those device nodes are named and what happens next. I’ll explain briefly how to automount a USB drive, but readers looking for detailed information about UDEV rules should consult Writing udev rules by Daniel Drake as well as the udev man page (man udev).

Start by plugging in the device you want to mount, for these examples I have used a USB thumbdrive.  Run dmesg and look for something like:

[6388458.851691]  sdb: sdb1
[6388458.852594] sd 15:0:0:0: [sdb] Attached SCSI removable disk
[6388458.852635] sd 15:0:0:0: Attached scsi generic sg2 type 0

This tells me that the assigned device node is /dev/sdb1. To find out more about that device, run: udevinfo -a -p /sys/block/sdb and look for a single block of text that has a useful attribute– a single rule may only match elements from a single block. In the example below, I’ve highlighted two lines I can use in my rule:

looking at parent device '/devices/pci0000:00/0000:00:02.1/usb2/2-5/2-5.1':
KERNELS=="2-5.1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{dev}=="189:152"
... lots more ATTRS ...
ATTRS{manufacturer}=="Prolific Technology Inc."
ATTRS{product}=="USB Mass Storage Device"


I now have four bits of information I can use to write a useful rule:

• The device node will be similar to /dev/sdb1, which I can match with sd?1
• SUBSYSTEMS==”usb”
• ATTRS{product}==”USB Mass Storage Device”
• I want to run /bin/mount and mount the device at /mnt/usb

After looking up the correct syntax in the man page, I wrote my rule like so:

KERNEL=="sd?1", SUBSYSTEMS=="usb", ATTRS{product}=="USB Mass Storage Device",
↪ RUN+="/bin/mount /dev/%k /mnt/usb"

This I saved in a file at /etc/udev/rules.d/10-codefix.rules. The rule should be picked up automatically, but one can always run sudo udevcontrol reload_rules or sudo /etc/init.d/udev reload.

With the device attached, this rule can be tested with udevtest /sys/block/sdb/sdb1 usb. If the rule is correct, the output of udevtest will include the mount command.

1. Defective hardware.

If there is a working volume control in Applications  Sound & Video ⇨ Volume Control or Applications ⇨ Sound & Video ⇨ AlsamixerGui, or on the desktop toolbar, then defective hardware is less likely. On new installations, the case may be simply that the Master or PCM channel is muted. If no sound card is detected these controls should be “grayed out” and unusable. If the volume controls are missing or disabled (i.e. you cannot move the slider control), you can confirm the diagnosis by following the instructions in the next section, then take the machine back to the shop that sold it to you or seek assistance from a professional.

2. Incorrectly detected hardware.

Whether or not volume controls work, it is worthwhile to check whether and what devices have been detected by Linux.  Most Linux distributions should have the command line tool lspci or lshw available, however hardinfo provides a nice graphic interface & report generation. Ubuntu users can install it via the package manager or at the command line with: sudo aptitude install hardinfo

The menu icon should appear in Applications ⇨ System Tools ⇨ System Profiler & Benchmark but may also be launched with at a command line: sudo hardinfo

In any case, the audio device will be listed under PCI devices. The reported device should be compared with the actual installed hardware to determine if it was correctly detected. If no audio device is listed, then this is a clear indication of missing or defective hardware.

3. Application Issues

If the hardware appears to be correctly identified and in good working order, the Master and PCM channels are enabled and volumes are set sufficiently high, speakers are plugged in and their volume is also turned up, and you are still unable to produce any sound in any application, professional assistance is probably in order. Those who are brave, foolish, lucky, or eager to learn can poke around online for reports of similar issues, in particular, users of Ubuntu Karmic (or any derived distributions) should check Ubuntu’s Karmic Caveats as well as the Ubuntu Multimedia & Video Forums as many users have reported issues related to the Pulse Audio sound server.

If you’re still stuck after all the above, your best bet is to seek out your local Linux User’s Group where you are sure to find someone willing to help.

I don’t often have the opportunity to experiment on computers running Windows, but every once in a long while it simply cannot be avoided. I recently found myself wanting to look up a password in Revelation, a password manager for the Gnome Desktop on Linux; I have previously written about using OpenSSH’s ProxyCommand directive to tunnel through a firewall and forward X11 (GUI) applications remotely from a an isolated workstation on a private LAN, the difference here was that I needed to forward that application to a Windows workstation.

I haven’t used Windows enough to do this sort of thing for about ten years, so it took a bit of fiddling, but I eventually worked out the following methodology. Like the other posts in this series, it is assumed that the reader is familiar with the basics, specifically the use of PuTTY and Pageant to log in to Linux hosts using ssh keys; a windows installer is available to install all needed utilites, but the only others used in this experiment were PuTTYgen (to generate an SSH key) and plink which I simply tossed in the Windows directory so I wouldn’t need a full path in the local proxy command (below). Some additional software is needed to run X Windows applications on Windows, and in this case I used Xming. I had never used Xming before, and may never need it again, but I was impressed that it was as easy as point, click, run– as long as it’s running in the background, it will do what’s needed. All the configuration is done in PuTTY.

Session Settings
A proxy connection has only a few specific settings, all others can be left at default values or the user’s preference; this screen shot is only included to emphasize that the Session Host is the box on the private LAN running the application we want, not the proxy host which has the public Internet connection we will be using.

Proxy Settings
The proxy hostname is the box with the public connection; it will use the local proxy command to connect our SSH client to the session host specified on the previous screen. Note that SSH will only use the Auto-login username (Connection=>Data=>Login Details) with the session host, which is why I have specified a username here.

X11 Forwarding
There is nothing complicated about the X forwarding settings, this must be enabled in PuTTy, as well as on the remote Linux box, and on the proxy. In my case, the proxy was an Ubuntu server not running X Windows, so I first had to install xauth (sudo aptitude install xauth).

This post is as much about customizing the root shell as it is about SSH environment variables, but I’m adding this to my OpenSSH collection because it’s applicable to any user.

I occasionally work on servers where, for a variety of reasons, I share an account with one or more other users; this is almost always suboptimal, but it does happen nonetheless. Over time I’ve grown somewhat partial to zShell, so one method I’ve used is to log in to a default shell, usually bash, then run zsh.

Even on server’s where I am the sole administrator, I usually don’t change the default shell– not so much because of days gone by when doing such a thing would break boot scripts & such, but because I try to practice the good habit of logging in as a normal user and using sudo for escalated privileges.

Eventually I struck upon the idea to have some code in the shell init script (e.g. $HOME/.bash_profile) switch to the shell of my choosing automatically as I log in. What I came up with looks something like this:

if [ -n "$CDFX_SHELL" ]; then

    tty -s && exec $CDFX_SHELL

fi

Briefly this code says, “if $CDFX_SHELL isn’t empty and the tty program says we’re connected to a terminal (on STDIN), then replace this shell by running the command in $CDFX_SHELL without creating a new process.”

Readers familiar with shell initialization may recognize potentially unnecessary checks in this example but this avoids having to delve into the differences between shell sessions which are interactive, login, both, or neither, as well as how this relates to scp and rsync. Also noteworthy are the checks which should be in the code before it’s used on a production server, such as verifying that $CDFX_SHELL specifies a valid shell. This isn’t intended to be cut-n-paste code.

Two steps are necessary for this to work: obviously $CDFX_SHELL must be set in the local environment for SSH to have anything to pass to the server, but less obviously the server must be configured to allow this variable to be set. This can be configured in the sshd config file (e.g. /etc/ssh/sshd_config) by adding this line:

AcceptEnv CDFX_SHELL

I prefer this method over those which require enabling PermitUserEnvironment because it’s less prone to unintended side-effects, as noted in the man page. In addition to (or in lieu of) using exec to switch shells, this method could also be used to set custom environments in the same shell, for different users or the same user, anytime it’s useful to have login behavior change based upon variables set in the SSH client.

Network address translation (NAT) is a very common method of providing secure access to hosts on a private network. Given the limited amount of IPv4 addresses, computer networks with relatively few, very few, and even a single public IP address are common. A typical small business customer of my consulting practice has one or more Linux servers on an office network protected by a firewall. The following is a close look at Example Industries, the theoretical owners of example.com; this customer receives support for two Linux servers, a mail server and a PBX, but only one public IP address between them. Through NAT, public services (namely mail and VoIP) on both servers are accessible via example.com. This works well for inbound mail and phone calls, which only need to access one or the other host, but SSH access is the lifeblood of remote system administration, and there’s the rub– when I enter ssh example.com I land at the mail server. SSH access to the PBX would seemingly threaten to litter my command line with unsightly extra characters, if not subsequent commands outright.

My carpals are tunneled enough, I don’t want to type more than ssh mail and ssh pbx to access these servers, and while I’m at it I want to have scripted log-ins as well– securely, not those namby-pamby no-password keys. In fact, I don’t even want to have private keys on either server.

Fear not! With the power of OpenSSH, I can fix this.

Recipes for SSH proxies are like homespun cure-alls: few do much good and some are actually harmful. As indicated in my introductory rant, I have a few criteria for this sort of thing:

• No interactive passwords.
• No password-less keys.
• No private keys on servers.
• Avoid command line options.
• Demur scripts.

Previous installments in this series have intentionally focused on declarations which will help define my Example Industries SSH client configuration; here’s the file so far:

# /home/garrison/.ssh/config
 
# Global Options
Host *
ForwardAgent yes
 
Host mail
HostName example.com
 
Host pbx
HostName example.com
LocalForward 8080 localhost:80
LocalForward 3306 localhost:3306

At this point ssh mail and ssh pbx are functionally equivalent in that both land me on the mail server, and I want forwarded access to pbx’s web configuration and database but I end up with webmail and mail user database (which I can access without port forwarding). While I can certainly connect to pbx once I’m in mail, I must specify it’s internal address (ssh 192.168.1.20) and the services I require will not be forwarded to my workstation. What I need is a way to tell SSH to bypass mail and connect me directly to pbx.

Rubbish. In life, particularly in technology, we often confound our challenges by mischaracterizing the requirements of a solution as I just have. It is no coincidence that my most successful customers are also the ones who make use of my ability to solve problems, rather than simply implementing solutions. Often the most challenging part of a solution is correctly stating the problem. What I really need is just for ssh pbx to connect me to pbx and forward my ports; I don’t really care whether mail is bypassed or not, so long as it stays out of my way.

As it turns out, the solution I favor does not bypass the gateway host (mail) at all, but uses it as a proxy for my connection to pbx. With the addition of a ProxyCommand directive to my SSH config, I can achieve all my objectives.

The ProxyCommand directive is a subtle beast, and my early attempts to use it were unsuccessful. At the time I was doing something similar on the command line: ssh -t example.com ssh 192.168.1.20 Because I initially hoped to “do the same thing in the config file” I mistakenly assumed that ProxyCommand would allow my to connect to mail and immediately fire off a connection to pbx; LocalCommand behaves this way but doesn’t allow me to accomplish what I can with ProxyCommand.
After a few unsuccessful syntax variations, I began to suspect that I might have the wrong idea about this directive.

When I eventually sorted out the correct syntax, I knew I had the wrong notion because I had no clue why one version worked and the others did not. Richard Silverman, one of the authors of the snail book was kind enough to set me straight. He explained:

ProxyCommand specifies a program which the SSH client will use to contact the remote SSH server. Instead of opening a TCP connection, it runs this program and uses its stdin/stdout as the communications channel.

I then understood that with ProxyCommand in play, SSH expects the command it executes to provide the TCP connection between mail and pbx; netcat, a phenomenally useful tool, was designed for just this sort of task:

ProxyCommand ssh example.com nc -v %h %p

Adding this directive to the Host pbx section of my config gets the whole proxy business out of my way and I can connect with just ssh pbx; tho forwarded HTTP and MySQL connections are just the beginning. I can use scp, sftp, FuseSSH, sshfs or anything built on SSH just as if pbx had a public IP. One more example:

rsync -Hav pbx:/usr/stuff backup:/archive

I often use such a command to transfer data from a machine with no public IP address to a backup server which also has no public IP and lives on another private network in a different town, state, or country. This is all done with ProxyCommand directives, over secure SSH connections, and most importantly, with no special command line syntax. What could be easier?

Port forwarding is a versatile feature which informs several popular concepts, including X Forwarding and tunneling which are briefly explained below; more advanced port magic will be addressed elsewhere.

X Forwarding
At the end of the previous installment of this series is an example SSH client configuration file, usually located at ~/.ssh/conf; a more complete description of the global declarations shown was deferred until this section, where they are more relevant.
# global declarations
This line is a comment and while it is ignored by SSH, it is very helpful to use comments and white space to maximize readability, and maintainability.
Host *As previously stated, Host declarations define the context of all following lines until the next Host declaration. Because the ‘*’ will match any host, these declarations will be applied to all hosts– unless overridden by a later declaration or a command line option.
ForwardAgent yesThe key agent allows a user to store one or more authenticated keys in memory, enabling dual key authentication; this is sometimes (incorrectly) referred to as passwordless authentication, but a password check is still required to load the key into the agent. Agent forwarding allows one not only to rely on keys instead of passwords to connect to a remote server, but to use the same agent to connect to a third host and so on as long as the keys are recognized and ForwardAgent is enabled on each host.
ForwardX11 yesThis declaration turns on port forwarding for X Windows authentication; briefly put, this allows an X session (i.e. the active GUI environment) on a Linux workstation to interact with X Windows on a remote host using an encrypted tunnel. The practical benefit is that programs on a remote host needing or allowing a graphic interface may use one provided by the client. The canonical example is that I run xclock on a remote server and the clock appears on my screen.

Tunneling
X forwarding may be fun, but port forwarding has many more practical uses. Suppose you need to connect to a web application on, perhaps FreePBX to configure a Linux PBX, but you cannot connect directly because port 80 is blocked– there are a few reasons why this might occur but port forwarding can work equally well in all likely scenarios.

If you log in to the remote host with ssh -L 8080:localhost:80 joe.telco@pbx.example.com SSH will create an encrypted tunnel between port 8080 on your workstation and port 80 on the server, ergo you can now get to the web app by pointing your browser to http://localhost:8080. Because the connection is tunneled through SSH, it works even if port 80 is restricted by IP address or an interceding router, firewall, or cable modem; because of SSH’s strong encryption, this technique also provides a secure connection to servers which do not offer SSL.

Many users mistakenly assume that the localhost in the preceding command is the one referenced in the subsequent URL; in fact, the given example is connecting (client) localhost:8080 to (server) localhost:80 and could be entered as:
ssh -L localhost:8080:pbx.example.com:80 joe.telco@pb»
The notable concept is that the latter host:port pair is evaluated on the remote host, meaning it’s a bit like saying:

“connect my self:8080 to your self:80“

ssh -L localhost:8080:localhost:80 joe.telco@pbx.exam»
ssh -L 192.168.1.7:8080:localhost:80 joe.telco@pbx.ex»
ssh -L *:8080:localhost:80 joe.telco@pbx.example.com
The explicit use of localhost in the first of the preceding examples restricts listening port 8080 for local use only; in contrast, the second example binds the port to a specific network interface, and other users on the network may use the forwarded port at the specified IP address; the last example avails the port on all interfaces. If the bind address is not specified, the port is bound to the loopback address unless GatewayPorts is enabled, wherein the wildcard address issued.

The config file is the place for complexity, and the following example specifies that connections to pbx.example.com should be made as user joe.telco, and forwarded HTTP and MySQL connections should be available on all interfaces using the specified ports.
Host pbx
HostName pbx.example.com
GatewayPorts yes
User joe.telco
LocalForward 8080 localhost:80
LocalForward 3306 localhost:3306
As previously illustrated, this simplifies the command line syntax such that in lieu of:
ssh -g -u joe.telco -L 8080:localhost:80 \
-L 3306:localhost:3306 pbx.example.comone need only enter:
ssh pbxMuch more information on the port forwarding capabilities of SSH are available in the man pages as well as previously cited sources; however, the examples here lay the foundation for the next installment of this OpenSSH series: Proxy Connections.

Though less obvious, another great feature of á la carte or “on demand” plans is scalability, if I suddenly find myself needing to host frequent call-in conference calls between a customer, their overseas manufacturing division, regional sales reps, and myself, the only change I’ll see on my invoices will be in usage. I am not aware of any “unlimited” residential plans which offer unlimited channels (simultaneous callers). With currently just three phone numbers, my setup is small enough, and with just enough complexity to provide a good example.

I use one number for my consulting, which has separate extensions, voice mail, etc.; I have a fax number for the luddite crowd, and a home number associated with a family voice mail, options for the caller to forward the call to my wife’s or my mobile phone, and a ring group which includes a line in my office. I’ll use an even usage split for comparison; for although Codefix Consulting has its own phone number, those who know me well tend to call my home number rather than risk my having a life outside of work.

My primary VoIP provider is VoiceMeUp.com and I have two DIDs (phone numbers) ($4.95 ea) and a prepaid, on-demand plan which bills 30/6 at $0.0185/min. My backup provider is CallWithUs.org who bill $0.0125 in whole minutes; while I hadn’t originally intended to acquire a DID through CallWithUs.org, I found one for $6/mo which includes 3000 free inbound minutes and couldn’t pass it up. My base VoIP price is therefore 4.95 * 2 + 6 = $15.90 plus usage, or $7.95 on an even split. Theoretically this leaves me with just over 650 minutes before exceeding my $20 target, but this doesn’t account for incremental billing, free VoIP to VoIP calls, and other variables which impinge cost.

It’s now more than a month since I dumped Broadvoice, ergo September’s charges and complete usage data are available for a real world comparison against a $40 average, a $35 example, and a $20 target. As it turned out we made no outbound calls on the (secondary) CallWithUs.com trunk and didn’t exceed the 3000 inbound minutes, so all billable usage was on the VoiceMeUp.com trunk which makes accounting easier. September’s total was 9.95 + 6 + 23.38 = $39.28 or $19.64 per split which helps validate my “less than $20 phone bill” theory. Our total usage was 36h 7m 33s (2167.55 min) or nearly 1,100 “home” minutes and more than 2,000 unused inbound minutes– how much do you talk?

If you’re not already using a config file (~/.ssh/config) you should peruse the documentation to see what it offers; an ongoing benefit I enjoy is that it allows me to accomplish more while typing less. Suppose, for example, you need to access two mail servers which are both behind a firewall and sharing a single public IP address. One server uses NAT (port forwarding) to provide user access via IMAP-SSL, POP3-SSL, and perhaps even webmail, all on default ports; similarly SSH can be accessed on port 22. The other server happens to be a mail relay, which handles all of the spam and virus scanning for inbound and outbound mail; while the SMTP, SMTPS, and submission services all enjoy a NAT configuration using default ports, SSH access is on port 23 because port 22 already forwards to the IMAP server and the sysadmin hasn’t read this series of articles.

As an added bonus, your accounts have usernames which differ from each other (let’s use “fred” and “barney”) as well as from your workstation. To log in to these machines using the command line, you would type:

ssh fred@example.com
ssh -p 23 barney@example.com

This isn’t a great deal of typing but already one can see that differentiating more complex connections may be confusing when distinguished only by the port used. We can clarify things a bit with a config file like this:

Host imap
HostName example.com
User fred
Host smtp
HostName example.com
Port 23
User barney

Now our SSH commands look nicer:

ssh imap
ssh smtp

The config file can always be overridden with command line options, so ssh admin@smtp will log in as admin rather than barney, but will still use port 23 and any other options set in ~/.ssh/config. Once you start using LocalForward and ProxyCommand command line options quickly become tedious and unwieldy, even if you can remember all options for every host you attend.

One final note, in addition to acting as a convenient alias, the host keywords may also be used to make declarations for groups of servers, or all servers, by using wildcards and pattern-lists. The ssh_config man page (or your preferred documentation) has a detailed PATTERNS section, but a below is a brief example to whet your appetite:

# global declarations
  Host *
  ForwardAgent yes
  ForwardX11 yes

# just for example.com servers
  Host *.example.com
  ServerAliveInterval 60
  StrictHostKeyChecking no