• Timeout in a Shell Script

    Although GNU coreutils includes a `timeout` command, sometimes that's not available. There are a *[lot](https://www.google.com/search?&q=shell%20script%20timeout)* of ham fisted approaches by very intelligent people. The "right" way to do this is with the `ALRM` signal. That's what it's for. So rather than reinvent the wheel, here's a correctly working timeout function. This works in at least `bash` and `zsh`. cleanup () { [[ -z $! ]] && kill -s TERM $! sleep 1 [[ -z $! ]] && kill -s KILL $! } timeout () { ( sleep $1 ; kill -s ALRM $$ ) & shift "$@" & wait $! } trap cleanup ALRM timeout 5 sleep 7 In this case, `timeout 5` executes with timeout of 5 seconds and `sleep 7` is the command to execute. This example will timeout. The timeout function will return with `142` if the process timed out.

  • Statement by Edward Snowden to Human Rights Groups at Moscow's Sheremetyevo Airport

    Republished from [WikiLeaks][origin]. ---- ### Friday July 12, 15:00 UTC Edward Joseph Snowden delivered a statement to human rights organizations and individuals at Sheremetyevo airport at 5pm Moscow time today, Friday 12th July. The meeting lasted 45 minutes. The human rights organizations included Amnesty International and Human Rights Watch and were given the opportunity afterwards to ask Mr Snowden questions. The Human Rights Watch representative used this opportunity to tell Mr Snowden that on her way to the airport she had received a call from the US Ambassador to Russia, who asked her to relay to Mr Snowden that the US Government does not categorise Mr Snowden as a whistleblower and that he has broken United States law. This further proves the United States Government's persecution of Mr Snowden and therefore that his right to seek and accept asylum should be upheld. Seated to the left of Mr. Snowden was Sarah Harrison, a legal advisor in this matter from WikiLeaks and to Mr. Snowden's right, a translator. Transcript of Edward Joseph Snowden statement, given at 5pm Moscow time on Friday 12th July 2013. (Transcript corrected to delivery) ---- Hello. My name is Ed Snowden. A little over one month ago, I had family, a home in paradise, and I lived in great comfort. I also had the capability without any warrant to search for, seize, and read your communications. Anyone's communications at any time. That is the power to change people's fates. It is also a serious violation of the law. The 4th and 5th Amendments to the Constitution of my country, Article 12 of the Universal Declaration of Human Rights, and numerous statutes and treaties forbid such systems of massive, pervasive surveillance. While the US Constitution marks these programs as illegal, my government argues that secret court rulings, which the world is not permitted to see, somehow legitimize an illegal affair. These rulings simply corrupt the most basic notion of justice &emdash; that it must be seen to be done. The immoral cannot be made moral through the use of secret law. I believe in the principle declared at Nuremberg in 1945: "Individuals have international duties which transcend the national obligations of obedience. Therefore individual citizens have the duty to violate domestic laws to prevent crimes against peace and humanity from occurring." Accordingly, I did what I believed right and began a campaign to correct this wrongdoing. I did not seek to enrich myself. I did not seek to sell US secrets. I did not partner with any foreign government to guarantee my safety. Instead, I took what I knew to the public, so what affects all of us can be discussed by all of us in the light of day, and I asked the world for justice. That moral decision to tell the public about spying that affects all of us has been costly, but it was the right thing to do and I have no regrets. Since that time, the government and intelligence services of the United States of America have attempted to make an example of me, a warning to all others who might speak out as I have. I have been made stateless and hounded for my act of political expression. The United States Government has placed me on no-fly lists. It demanded Hong Kong return me outside of the framework of its laws, in direct violation of the principle of non-refoulement &emdash; the Law of Nations. It has threatened with sanctions countries who would stand up for my human rights and the UN asylum system. It has even taken the unprecedented step of ordering military allies to ground a Latin American president's plane in search for a political refugee. These dangerous escalations represent a threat not just to the dignity of Latin America, but to the basic rights shared by every person, every nation, to live free from persecution, and to seek and enjoy asylum. Yet even in the face of this historically disproportionate aggression, countries around the world have offered support and asylum. These nations, including Russia, Venezuela, Bolivia, Nicaragua, and Ecuador have my gratitude and respect for being the first to stand against human rights violations carried out by the powerful rather than the powerless. By refusing to compromise their principles in the face of intimidation, they have earned the respect of the world. It is my intention to travel to each of these countries to extend my personal thanks to their people and leaders. I announce today my formal acceptance of all offers of support or asylum I have been extended and all others that may be offered in the future. With, for example, the grant of asylum provided by Venezuela's President Maduro, my asylee status is now formal, and no state has a basis by which to limit or interfere with my right to enjoy that asylum. As we have seen, however, some governments in Western European and North American states have demonstrated a willingness to act outside the law, and this behavior persists today. This unlawful threat makes it impossible for me to travel to Latin America and enjoy the asylum granted there in accordance with our shared rights. This willingness by powerful states to act extra-legally represents a threat to all of us, and must not be allowed to succeed. Accordingly, I ask for your assistance in requesting guarantees of safe passage from the relevant nations in securing my travel to Latin America, as well as requesting asylum in Russia until such time as these states accede to law and my legal travel is permitted. I will be submitting my request to Russia today, and hope it will be accepted favorably. If you have any questions, I will answer what I can. Thank you. ---- For further information, see: [http://wikileaks.org/Statement-from-Edward-Snowden-in.html][foot1] [http://wikileaks.org/Statement-by-Julian-Assange-after,249.html][foot2] [origin]: http://wikileaks.org/Statement-by-Edward-Snowden-to.html). [foot1]: http://wikileaks.org/Statement-from-Edward-Snowden-in.html [foot2]: http://wikileaks.org/Statement-by-Julian-Assange-after,249.html

  • Raspbery Pi and EW-7811Un

    I'm setting up Raspberry Pi's using the [Edimax EW-7811Us][wifi] wifi module available on Amazon for a mere $11. Following the [Debian WiFi][deb_wifi] wiki page initially didn't work. The EW-7811Us uses an RTL8188CUS chipset which requires the rtl8192 kernel driver. There's no `firmware-realtek` package on Raspbian, and the best answer I found was to download some dude's hacked kernel module. No thanks. Instead, install the `rpi-update` package then run `rpi-update`. The firmware will be updated in a way officially supported by raspbian (if there is such a thing). Then reboot. [wifi]: http://www.amazon.com/dp/B003MTTJOY/?tag=digitalelfnet-20 [deb_wifi]: http://wiki.debian.org/WiFi/HowToUse

  • A Case Study in CFEngine Layout

    I've been working a lot with CFEngine newbies. CFEngine has been described as flour, eggs, milk and butter. All the ingredients needed to make a cake. Getting the new CFEngine user to recognize, then become excited about the possibilities that CFEngine provides they are now faced with the question of "What next?" Indeed, anybody can throw some flour, eggs, milk and butter into a bowl, mix and bake it. But will it taste good? This is an exposé of how I have managed my CFEngine repository for more than eight years. This design was used to manage over 1,000 host instances. This works best if you have an agile infrastructure. Use [SmartOS](http://smartos.org/), [OpenStack](http://openstack.org/), [Amazon EC2](https://aws.amazon.com/ec2), [CloudStack](http://cloudstack.apache.org/) or similar. ## The repository, and version control Firstly, place your cfengine repository in some revision control. I am highly partial to `git`. Get the [Pro Git][1] book (or [download][2] it). Read chapters 1, 2, 3. This will make you a git power user. After you're comfortable using git read chapters 6 and 7. When you're hungry for more, read the rest. I symlink `/var/cfengine/masterfiles` to `/cfengine/inputs`. This contains all of my policy files. I also create `/cfengine/files` for files that get copied to remote systems. This mostly contains my configuration files. `/cfengine/` is initialized as a git repository. Changes made to either `inputs` or `files` should be atomic. Adding something new for Apache? Any `inputs` and `files` involved should be checked in as a single commit. This makes reverting a change easier. ### Environments I use four environments. * Alpha * Beta * pre-Production * Production I also lied about initializing `/cfengine` as a git repository. I use a central repository server that contains only a bare git repository. The central repository has four branches. * master * beta * preprod * prod Astute readers will notice there's no alpha branch. I'll get to that later. **beta** is a full integration environment. Everything in beta is expected to work, yet not to be relied upon. That is to say, nothing should move to beta that is known broken. Beta will break. But don't do that intentionally. If it's half finished keep it out of beta. **prod** is the full production environment. Breaking this means losing money. Don't break this. Prod is tagged daily. Rolling back is done by checking out the appropriate tag. **preprod** is for final quality assurance testing. Preprod should be identical to prod except for changes to be imminently released to prod. Preprod can also be used for offline testing of the production environment without affecting capacity or availability. Preprod should be in your production network fabric. **master** is the trunk. All code is initially merged here, then merged to the appropriate branches. No one should be allowed to merge directly to any branch other than master. The repositry czar merges commits to other branches. ## A DevOps Workflow This is why there's no alpha branch. Let's assume that you're going to be making a change to the configuration of Tomcat. 1. Spawn a new cfengine instance. 2. `git clone` the cfengine master branch and bootstrap the server to itself. 3. Spawn as many instances as necessary for your application to work. This will be at least Tomcat instances, possibly including Apache and Postgres instances and bootstrap all of them to your new cfengine server instance. 4. By editing only the cfengine files out of the cloned repository make all of your updates. 5. Code review 6. When that feature is ready push a single commit to master and merge to beta 7. Integration testing in Beta 8. Changes that need to be made are done in the private instance set. When they're ready proceed from step 5. 9. When that feature is ready merge from beta to preprod. 10. Final QA testing. 11. Changes that need to be made are done in the private instance set. When they're ready proceed from step 5. (Yes, that means it goes through integration again). 12. When that feature is ready merge from preprod to prod. ## Managing The CFEngine Repository: Layers I use a layered approach. Each layer is contained within a single bundle. * Meta -- These are things that affect every host that runs cfengine. Layers that are based on intrinsic characteristics: * Operating system families -- `windows`, `unix` (anything Unix like) * Operating System -- `linux`, `solaris`, `bsd`, `darwin` * Distribution -- `debian`, `redhat`, `solaris11`, `omnios`, `freebsd`, `openbsd` * Distro Version sub-layer -- `debian_6`, `redhat_6`, `centos_6` Layers that are based on the role * Application -- `apache`, `postgresql`, `mysql`, `bind`, `tomcat` (These are often named after packages) * Application sub-layer -- `apache1_3`, `apache2`, `tomcat6`, `tomcat7` * Role -- `external_web`, `internal_web`, `proxy`, `smarthost` * Hostname -- `web_f7f274`, `web_4d06a8` ### Bundle <--> Layer Mapping I generally contain one bundle per file, per layer. The default policy files that come with cfengine are in what I consider the meta layer. This is a subset of my policy files to give you an idea of the organization. * `unix.cf` -- bundle agent unix * `linux.cf` -- bundle agent linux * `debian.cf` -- bundle agent debian * `redhat.cf` -- bundle agent redhat * `solaris.cf` -- bundle agent solaris * `apache2.cf` -- bundle agent apache2 * `bind9.cf` -- bundle agent bind9 * `web_ext.cf` -- bundle agent web_ext (policy for public facing web servers) * `dpkg.cf` -- bundle agent dpkg (Package management common to Debian) * `rpm.cf` -- bundle agent rpm (Package management common to RedHat) * `ips.cf` -- bundle agent ips (Package management common to the Image Package System, used by Solaris) * `digitalelf_stdlib.cf` -- Private library of bundles and bodies. This is similar in nature to `cfengine_stdlib.cf`, but I never change `cfengine_stdlib.cf`. I put things into my private library. When they are well tested I open a pull request with [cfengine/core](https://github.com/cfengine/core) to contribute it. All promises are added to the lowest layer bundle (with global being the lowest and hostname behing the highest). Thus, changes to `/etc/resolv.conf`, because all Unix like systems treat `/etc/resolv.conf` alike goes into the `unix` layer. The `sysctl` handling is different per operating system so they go into `linux` and `bsd` bundles at the OS layer. An external facing web server, by nature of being a web server must include apache as does an internal facng web server, so each automatically pulls in `apache2`. Likewise canonical DNS servers and caching DNS servers alike pull in `bind9`. ## Dynamic bundlesequence Because of the layered approach, which inputs and bundles need to be run are dynamically generated. Public web servers running on Debian Linux will be able to select the `ext_web`, `apache2`, `debian`, and `linux` bundles automatically. I can have the same web content on Solaris 11 and it will instead choose `ext_web`, `apache2`, and `solaris` bundles. I have a very large header to `promises.cf` to facilitate this. Here is an excerpt, along with additional commentary of my `promises.cf` to show how the `bundlesequence` is dynamically generated. bundle common classify { # This section classifies hosts instances into roles based on the hostname # I use a completely virtualized infrastructure with hostnames determined by # on a role specific prefix and a hex string separated by an underscore. # The hex string is the last 3 bytes of the MAC address of the lowest # numbered interface (e.g., eth0). Instances are created this way by my # provisioning system. classes: "dns_ns" or => { classmatch("ns[0-9]*") }; "dns_forwarder" or => { classmatch("dns_[0-9a-f]*") }; "db_server" or => { classmatch("db_[0-9a-f]*") }; "gitlab" or => { classmatch("gitlab_[0-9a-f]*") }; "web_ext" or => { classmatch("www_[0-9a-f]*") }; "web_int" or => { classmatch("web_[0-9a-f]*") }; "xwiki" or => { classmatch("xwiki_[0-9a-f]*") }; # Roles choose application bundles "apache" expression => "dpkg_repo|web_ext|web_int"; "bind" expression => "dns_ns|dns_forwarder"; "postgresql" expression => "db_server"; "tomcat" expression => "xwiki|jira"; "rails" expression => "gitlab" # Roles and/or applications can be grouped "app_server" expression => "rails|tomcat" # Applications may also depend on other applications "sql_client" expression => "app_server"; "ssl" expression => "apache|tomcat|rails"; "stunnel" expression => "mysql"; } bundle common g { # This section assigns bundles to application/role/grouping classes. # An array is created, named **bundles**. Each *key* is named after # a *bundle*. The *value* of each key is the input file where that # bundle can be found. vars: # These classes were defined by me in the classify bundle apache:: "bundles[apache]" string => "apache.cf"; bind:: "bundles[bind]" string => "bind.cf"; postgresql:: "bundles[postgresql]" string => "postgresql.cf"; ssl:: "bundles[ssl]" string => "ssl.cf"; stunnel:: "bundles[stunnel]" string => "stunnel.cf"; # Thse are hard classes determined by cfengine. I don't need to explicitly # classify them. debian:: "bundles[dpkg]" string => "dpkg.cf"; "bundles[debian]" string => "debian.cf"; centos:: "bundles[rpm]" string => "rpm.cf"; "bundles[centos]" string => "centos.cf" sunos_5_11:: "bundles[ips]" string => "ips.cf"; "bundles[solaris]" string => "solaris.cf"; xen_dom0:: "bundles[xen_dom0]", string => "xen_dom.cf0"; # Now the magic. # I create two slists. One named "sequence" and one named "inputs". # The "sequence" slist contains a list of bundle names. # The "inputs" slist contains a list of input files. any:: "sequence" slist => getindices("bundles"); "inputs" slist => getvalues("bundles"); } body common control { # The bundlesequence now includes those things which are common to all, plus # the contents of the slist "sequence" (which has ben dynamically generated), # plus the unqualified hostname. bundlesequence => { "global", "main", "@{g.sequence}", "${sys.uqhost}"}; # The inputs now includes common libraries and main.cf which will be run by # all systems, plus the contents of the slist "inputs" (which has been # dynamically) generated, plus an input based on the unqualified hostname. inputs => { "cfengine_stdlib.cf", "digitalelf_stdlib.cf", "main.cf", "@{g.inputs}", "${sys.uqhost}.cf" }; # Sometimes I need to have any specific configuration for a single host (e.g., # one of dns_ns will be the master and the rest will be slaves so the master # needs special configuration). The following options will allow cfengine to # skip the hostname bundle/input if one does not exist (which it usually # doesn't). ignore_missing_bundles => "true"; ignore_missing_inputs => "true"; version => "Community Promises.cf 1.0.0"; } Notice that instances are automatically classified by their hostname. So if I need a new external web server I provision a new instance with the name prefix `www_` (I can also choose the OS at provisioning time). My provisioning system automatically assigns them a unique ID, creates the instance, installs the OS, installs cfengine, bootstraps it to the cfengine master server, runs cfengine to apply the final configuration and finally adds the instance's services to the appropriate load balancer entries. I have repository mirrors of all platforms I run so a newly provisioned host can be in production with a perfect configuration in as little as five minutes. ---- [1]: http://www.amazon.com/gp/product/1430218339/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1430218339&linkCode=as2&tag=digitalelfnet-20 [2]: http://www.git-scm.com/book

  • Mute Means STFU

    For those not keeping track, the iPhone's mute switch has a design flaw[1][f1]. Allow me to summarize: - The New York Times -- [An iPhone alarm disrupts the NY Philharmonic] [nytimes] - John Gruber -- [Defends the iPhone][gruber] - Andy Ihnatko -- [Gruber/iPhone are wrong][andy] - Ben Brooks -- [The picture says it all][brooks] I can't imagine this debate having *ever* started if mute always meant mute. ---- 1. Yes, it's a design flaw.[⇑][fr1] [f1]: #f1 "Yes, it's a design flaw." [nytimes]: http://www.nytimes.com/2012/01/13/nyregion/ringing-finally-stopped-but-concertgoers-alarm-persists.html [gruber]: http://daringfireball.net/2012/01/iphone_mute_switch_design [andy]: http://ihnatko.com/2012/01/14/daring-fireball-on-the-behavior-of-the-iphone-mute-switch/ [brooks]: http://brooksreview.net/2012/01/the-picture-says-it-all/ [fr1]: #fr1 "Jump Back"

  • Anonymizer Universal on Android

    This is a repost of an article written by [Jason Mansfield][1] on his blog. He's changed providers several times and now it can't be read on his site. I've reproduced it here for posterity. ---- Before you begin you'll need to have a subscription to Anonymizer Universal. It's a pay service but it's pretty cheap. With your subscription you should have received an email with a link to the Windows and Mac clients and to the iPhone configuration profile. Download the Mac configuration profile file on your desktop machine. Open it with your text editor and look for the SharedSecret section near the top. In there you'll find a Base64-encoded key that's essential for the setup. Run the value through a Base64 decoder. If you're on a unix type system (including Mac OS X) you can run either of the following: perl -MMIME::Base64 -e 'print decode_base64("_BASE64_ENCODED_SECRET_STRING_GOES_HERE_") . "\n";' or echo '_BASE64_ENCODED_SECRET_STRING_GOES_HERE_' | openssl enc -d -base64 The result will be another Base64 encoded value. If you aren't on a unix system there are web-based Base64 decoders available. On the home screen of your Android device, press **Menu** -> **Settings** -> **Wireless & networks** -> **VPN Settings** -> **Add VPN** -> **Add L2TP/IPSec PSK VPN**. * Set the **name** to `Anonymizer Universal`. * Set the **VPN** server to `universal.anonymizer.com`. * Set the **IPSec pre-shared key** to the value you recovered with the Base64 decoder. Note that this is an error-prone process and is made much easier by having someone read the key to you. * Ensure **Enable L2TP secret** is *not* checked. * Hit your back button, you may be asked for your credentials storage password. * Select Anonymizer Universal to connect. * It should ask for your Anonymizer account credentials. Win! [1]: http://www.clinicallyawesome.com/

  • Good Bye, Steve

  • Philosophy of System Administration

    System administration is not something that monkeys can do. While places like [ITT Tech](http://www.itt-tech.com/) or [Coleman University](http://www.coleman.edu/) do teach the basics of using a computer they don't teach how to be a truly great system administrator. ## Learn why things work on a starship ## In [Star Trek II: The Wrath of Kahn][1] during the first encounter between Kirk and Kahn, Kahn has crippled the Enterprise and is demanding Kirk's surrender. Kirk buys time promising surrender and uses that time to hack into the opposing ship's computer system and lower its shields, thereby leaving Kahn defenseless for Kirk's counter strike. Kirk and Saavik have this exchange: > **Kirk:** Saavik, punch up the data charts of Reliant's command console -- hurry... > > **Saavik:** Reliant's command...I don't understand -- > > **Kirk:** You've got to learn **why** things work on a Starship. Each ship has its own prefix combination code to prevent an enemy from doing what we're attempting; using our console to order Reliant to lower her shields... > > **Star Trek II: The Wrath of Kahn** Computers can only do two things. Add and compare. Everything else is an abstraction. At the highest level you have widgets that look like physical objects that can be manipulated. They can't be, at least not really. Peal off the widgets layer and you have a canvas with areas that trigger actions. Even a shape definition is an abstraction for multiple non-contiguous memory areas. Every swipe, click or keypress is simply sending a combination of instructions which is to add or compare. There's a great article by Joel Spolsky on [Leaky Abstractions][2] that explores this in depth. Learn why the abstractions work, what abstractions they are built on top of, and as Joel explains, what's leaking through. ## Don't cut the ends off the roast ## There's an anecdote about a newly married woman who cooks a roast for dinner for her husband. In preparing it she cuts the ends off of the roast. At dinner time he asks her where the rest of the roast is. Perhaps it's his favorite part. Perhaps he's thrifty and doesn't want good meat to go to waste. She replies "This is how a roast is made. That's how my mother taught me do it." She later calls her mother to ask why the ends of the roast need to be cut off. The mother says that she doesn't know, but that's how gradmother always prepared it. The mother later calls grandma and asks *her* why the ends of the roast must be cut off. Grandma's response is "Because I didn't have a pot large enough to fit the whole roast." Tradition is a terrible reason to do anything. Without understanding *why* decisions were made you have no idea wether or not it's time to change it. You're left with voodoo and superstition, jumping at shadows blaming anything and everything. ## KISS ## > Everything should be as simple as it can be, but not simpler. > -- [Albert Einstein (kind of)](http://en.wikiquote.org/wiki/Albert_Einstein#1930s) > Anyone can create a cryptography product that he himself cannot break. > -- [Bruce Schneier](http://www.schneier.com/crypto-gram-9902.html) > There's an old saying in programming, "Don't make your code as clever as possible, because debugging is harder than programming." > -- [John Siracusa](http://www.siracusasaidso.com/#ep23min61 The hallmark of good design is in reducing complexity. Anything can be made needlessly complex, which is usually an indicator that you don't understand well enough what you're doing or why you're doing it. Any idiot can design a system. But it takes a smarter idiot to figure out what went wrong. Don't design a system beyond your troubleshooting skill. [1]: http://www.imdb.com/title/tt0084726/ [2]: http://www.joelonsoftware.com/articles/LeakyAbstractions.html

  • Jumping on the Bandwagon

    That enormous sucking sound on the Internet is thousands of geeks suddenly switching to [Octopress][1]. This has been mainly prompted by [Matt Gemmell][2]. Octopress, the blogging framework for hackers, surprisingly was easy to set up, and easy to hack. I've been using MovableType for almost two years and I never did get things *quite* like I wanted. I've spent about eight hours now setting up Octopress, including writing a MovableType import script. Why Octopress? * [Blogging with Octopress](http://mattgemmell.com/2011/09/12/blogging-with-octopress/) * [Octopress Documentation](http://octopress.org/docs/) [1]: http://www.octopress.org/ [2]: http://www.mattgemell.com/

  • Slow Cacti Graph Rendering

    I've been running [Cacti](http://www.cacti.net/) at home to keep track of a number of things. It's wonderful, and I highly recommend it if you understand the phrase "turn on SNMP". But ever since upgrading to Solaris 11 Express, Cacti graph rendering has been extremely slow. The PHP renders fine, but the graphs would take several seconds per graph and with some pages having upwards of 20 graphs it was becoming quite aggravating to me. I've been searching off and on for months about why, and most things I can find about `rrdtool` rendering being slow are completely irrelevant to Solaris, but I finally got a break. I found [this Debian bug report](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612713). The problem apparently has to do with the new [FreeDesktop](http://www.freedesktop.org) architecture. `Rrdtool` now uses `fontconfig` caching. Both Solaris 11 and Debian's default permissions on `/var/cache/fontconfig` are 755. Setting the permissions world writable and reloading the page did the trick. `Rrdtool` apparently doesn't need the permissions world readable forever. Just that first time. After `rrdtool` writes the cache once you can set the permissions back to 755. Enjoy your speedy Cacti.