Blog Auto-Publish with Dropbox

This weekend I moved my blog over to Pelican from Octopress. I've been very happy with this decision.

But I decided I wanted to go a step further, and set up autopublish. I knew it could be easily done with a git hook, but I wanted to be able to easily blog from my iOS devices as well, so I decided to go with Dropbox instead.

Getting Dropbox

First, I needed to install Dropbox on my machine. Dropbox provides an easy one-liner:

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -

Now, when I first tried to start up dropbox, it gave me a cryptic error, which I traced back to not having X11 libraries installed. Turns out I just needed to unset my DISPLAY variable:

unset DISPLAY

Now Dropbox will start properly:

~/.dropbox-dist/dropboxd

Follow the instructions to get Dropbox authorized on your server, then once you see the success message, exit using ctrl-c.

Configure Dropbox as System Service

Next we need to create an init script for Dropbox. I found an example here. Here's the Ubuntu/Debian script:

#!/bin/sh
#dropbox service
DROPBOX_USERS="user1 user2"

DAEMON=.dropbox-dist/dropboxd

start() {
   echo "Starting dropbox..."
   for dbuser in $DROPBOX_USERS; do
       HOMEDIR=`getent passwd $dbuser | cut -d: -f6`
       if [ -x $HOMEDIR/$DAEMON ]; then
           HOME="$HOMEDIR" start-stop-daemon -b -o -c $dbuser -S -u $dbuser -x $HOMEDIR/$DAEMON
       fi
   done
}

stop() {
   echo "Stopping dropbox..."
   for dbuser in $DROPBOX_USERS; do
       HOMEDIR=`getent passwd $dbuser | cut -d: -f6`
       if [ -x $HOMEDIR/$DAEMON ]; then
           start-stop-daemon -o -c $dbuser -K -u $dbuser -x $HOMEDIR/$DAEMON
       fi
   done
}

status() {
   for dbuser in $DROPBOX_USERS; do
       dbpid=`pgrep -u $dbuser dropbox`
       if [ -z $dbpid ] ; then
           echo "dropboxd for USER $dbuser: not running."
       else
           echo "dropboxd for USER $dbuser: running (pid $dbpid)"
       fi
   done
}

case "$1" in

   start)
       start
       ;;
   stop)
       stop
       ;;
   restart|reload|force-reload)
       stop
       start
       ;;
   status)
       status
       ;;
   *)
       echo "Usage: /etc/init.d/dropbox {start|stop|reload|force-reload|restart|status}"
       exit 1

esac

exit 0

Note that when I pulled down that script originally, it had a typo. The DAEMON line near the top of the file should be .dropbox-dist/dropboxd -- they left off the trailing d in the version I downloaded.

Place this script in /etc/init.d/dropbox. Replace the DROPBOX_USERS setting with a space-separate list of users for which you want Dropbox running. Then, make the file executable and set it to execute on startup:

sudo chmod +x /etc/init.d/dropbox
sudo update-rc.d dropbox defaults

Now, start Dropbox again, as a service:

sudo service dropbox start

Selective Sync (Optional)

I have a very large Dropbox folder, and so only want to sync the Blogs folder from my Dropbox account.

I downloaded the CLI script mentioned on that installation page:

wget https://linux.dropbox.com/packages/dropbox.py

Then, with Dropbox running, I excluded all the directories within the Dropbox folder, then removed the Blogs folder from the resulting exclusion list:

./dropbox.py exclude add ~/Dropbox/*
./dropbox.py exclude remove ~/Dropbox/blogs

Perfect, now I have only my blogs syncing to my server's dropbox folder.

Autopublish: The Watcher Script

The automagic regeneration of my blog I modified from this post.

I'm going to assume you have a working Pelican setup. Assuming you used the Pelican quickstart, you should have both a pelicanconf.py and a publishconf.py. The publishconf.py is going to be what we use.

First, you must modify your publishconf.py file such that it can operate from any directory, rather than relying on relative paths. Thus, you should add your blog's absolute directory to the Python path inside of that script. Here is my publishconf.py:

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

# This file is only used if you use `make publish` or
# explicitly specify it as your config file.

import os
import sys

#sys.path.append(os.curdir)
# This is the directory where your pelican's
# configuration files reside
pelicanpath = '/home/basepi/Dropbox/Blogs/blog.basepi.net'
sys.path.append(pelicanpath)

sys.path.append(os.curdir)
from pelicanconf import *

SITEURL = 'http://blog.basepi.net'
RELATIVE_URLS = False

FEED_ALL_ATOM = None
CATEGORY_FEED_ATOM = None

DELETE_OUTPUT_DIRECTORY = True

These are the key lines:

pelicanpath = '/home/basepi/Dropbox/Blogs/blog.basepi.net'
sys.path.append(pelicanpath)

Now no matter what my current working directory is, the publishconf.py will be able to find the files it needs to operate.

Next, I created a publish script, publish.sh:

#!/bin/bash

PELICAN=/usr/bin/pelican
CONTENT=/home/basepi/Dropbox/Blogs/blog.basepi.net/content
OUTPUT=/home/basepi/Dropbox/Blogs/blog.basepi.net/output
SETTINGS=/home/basepi/Dropbox/Blogs/blog.basepi.net/publishconf.py

rm -rf /home/basepi/Dropbox/Blogs/blog.basepi.net/output/*
$PELICAN $CONTENT -o $OUTPUT -s $SETTINGS || exit $?
rsync -r --delete /home/basepi/Dropbox/Blogs/blog.basepi.net/output/ /var/www/blog.basepi.net

Note the use of || exit $?. This will cause the script to exit if the retcode of the pelican command is non-zero. I don't want to publish my blog if there are problems.

Make this script executable, and then run it to test that things are working.

The final step is to set up the watcher script which will run our publish script on changes. For this purpose we will be using this watcher script.

This script, conveniently, is ready to be used as an init script. Take the script and put it at /etc/init.d/watcher. Make it executable:

sudo chmod +x /etc/init.d/watcher

Now, we need to create the configuration file for the watcher. This file is at /etc/watcher.ini:

[DEFAULT]

; where to store output
logfile=/var/log/watcher.log

; where to save the PID file
pidfile=/var/run/watcher.pid

[job1]
watch=/home/basepi/Dropbox/Blogs/blog.basepi.net/content
events=modify,create,delete,move
excluded=
recursive=true
autoadd=true
command=/home/basepi/Dropbox/Blogs/blog.basepi.net/publish.sh

Now we just need to start the watcher:

sudo service watcher start

You can tail the log to see if it's working (change a file to trigger it):

tail -f /var/log/watcher.log

If everything is in order, just have the watcher start on boot:

sudo update-rc.d watcher defaults

And you're done! Now, any time you change the content of your blog, it will be automatically re-built and published for you!

Let me know what you think in the comments. Or hit me up on Twitter!

Tagged , , , ,

Low Friction

It's been a long time.

...again.

I thought that I had a pretty low-friction setup with my new blog on Octopress. However, that was not the case. This morning I decided I needed to get my blog back up. There have been a few times in the last couple of years that I wished I had it available. However, I didn't have any of the dependencies set up on my new laptop. So I set out to get ruby installed and ready to generate my blog again.

After an hour of fighting rbenv and bundle, I decided it was finally time to take the plunge, and switch to a Python-based tool.

I installed Pelican, and had it generating my new site from the default skeleton in a few minutes. Granted, I've had to take some more time choosing a theme and porting my content over. But grand total it's only taken me an hour or two of tinkering, and should reduce my blogging friction substantially, both on new machines and on existing setups.

I did port over most of the content from my old blog. I may delete some of it going forward, mostly out of embarassment.

I lost my old disqus comments -- since there was only one comment on my old blog, I decided it wasn't worth the potential hassle of making sure it translated cleanly.

Additionally, the RSS feed is new, so you'll have to resubscribe.

So here we are, on the newest edition of blog.basepi.net. Hope you like it.

I'm Back!

Well, at long last I got around to re-installing Ruby on my machine, updated Octopress, and got my blogs settled. I'm back!

So what's been going on in the interim? Let's hit the top two:

  • I graduated Magna Cum Laude from the University of Utah with a BS in Computer Science.
  • I switched jobs -- I now work for Salt Stack, Inc, making awesome open source software.

A lot has changed, and I hope to have much more to blog about now that I'm more involved in the open source community. Feel free to throw me feedback! I don't pretend to be an awesome blogger. =)

Why I switched to Octopress

Update, Nov 2014: Since I wrote this blog post, I have ditched Jekyll and Octopress (mostly because of the pain of setting up Ruby, not because of those pieces of software themselves). This blog is still static, but is now generated using Pelican, a Python project.

The Problem

Last week I received an e-mail from the RSA Anti-Fraud Command Center. It related to my personal blog:

It appears that your website cmyers.net has been hacked by a fraudster. It is now hosting a phishing attack against RBC. Please remove the fraudulent folders/files as soon as possible and secure your website as it has been compromised. Please note that it is possible that the fraudulent content is embedded in your website's legitimate files.

Great.

It turns out the claim was in fact true. There were new files in my wp-admin folder for my wordpress install. When I loaded these files in a browser, it looked like the home page for a bank.

I had been hacked.

Turns out it's actually quite a common thing. WordPress is a common (and apparently quite easy) target for these types of hacks. It comes down to a combination of it's popularity, lack of really security-centric design, and the fact that it's written in PHP.

I considered a few options. Do I take the time to properly harden my PHP5 install and reinstall wordpress? Or do I change systems?

Enter Jekyll

My friend Andrew generates his blog using jekyll. Jekyll is a "simple, blog aware, static site generator." It's super-cool, check it out. However, unlike WordPress, it doesn't have a bunch of built-in themes, or even a theme directory that I could find. You have to write your own CSS and HTML templates, which it uses to generate the site. But it generates an almost completely static site -- the exception would be if you build in functionality for Disqus comments or put your Twitter roll in the sidebar.

If I was going to switch systems, static is what I was looking for. The resulting site is much faster than a WordPress site, and much more secure. But I didn't have the time, knowledge, or creativity to create a theme from scratch, and porting my theme from WordPress would be a many hour project.

Enter Octopress

Then I received a tweet from my friend Julio that said to check out Octopress. It's built on top of Jekyll with built-in support for great syntax highlighting, Disqus comments, social integration -- everything I was looking for. And a great default theme, which I could use as a jumping point for my own. I was sold.

The rest is history. I'll spell out the details in my next post.

Minecraft!!!

Update, November 2014: These days, I'm hosting my own vanilla minecraft server on Linode, using MCMyAdmin for a web interface.

I finally did it. I broke down and rented a Minecraft server.

Previously, I had hosted a Minecraft server pretty successfully on my personal web server. The problem became two-fold: it was a huge time-sink, as updating all the plugins that I was running took a long time at each release. Also, while hosting the Minecraft server, my server would often become unresponsive, requiring a restart. This was unacceptable, as I also host a Teamspeak 3 server, an IRC bouncer, and multiple websites from my server. I needed maximum uptime! So I finally shut down my Minecraft server for good.

But now it's back! OK, it's not the exact same server -- I didn't bring the old world over. I wanted a fresh start. But it's a Minecraft server, and it's all mine for the building! OK, not all mine: I want you to come too! If you already have a Minecraft account, post it in the comments so I can add you to the user list. (Otherwise you won't be able to build) If you've never played Minecraft, head on over to minecraft.net and buy the game. Just do it, it's worth it.

My server is at mc.basepi.net. Come and join us in building something awesome! =)

Slide Design for Developers

This is a great article by a guy over at Github about slide design. He makes some really good points, and his slides really are beautiful. I want to see more presentations with these design pointers in mind.

More on SOPA and Protect-IP (Don't break the internet!)

More on SOPA and Protect-IP (Don't break the internet!)

As I said in my previous post, SOPA (the Stop Online Piracy Act) and Protect-IP are both very bad pieces of legislation. They are not only technically damning for the internet, but they are unconstitutional. When I started doing my due diligence the other day with regards to these bills, I didn't know much about them. After some research, though, I have discovered just how bad these bills are. Here's some more information, in a post by Adam Savage (from MythBusters).

That article is not that technical and doesn't really quote the bills much. For the full story on the bills and just how technically bad they are, as well as how unconstitutional they are, read the Standford Law Review's take on the bills.

These bills are being pushed by the entertainment industry pretty hard, and not necessarily for bad reason -- piracy is an issue. But these bills are technically stupid, won't actually solve anything (the second link addresses this -- it would just force piraters to use alternate services from DNS, which is just bad for the internet at a whole and won't stop their activities), and puts the US on a slippery slope towards censorship like Iran or China. The bills are being fought by every single major tech company. (And probably all the non-major tech companies as well -- they're just not as visible)

Leave your thoughts in the comments. And speak up to stop SOPA and Protect-IP!

Stop SOPA (Stop Online Privacy Act)

I've heard a lot of chatter about SOPA (the Stop Online Privacy Act), which is currently going through congress. However, today I finally decided to look into it. And I definitely didn't like what I found. Here's a video summing up the problems (the video is for the Protect IP act, which was introduced earlier this year, but most of the same problems apply to both bills). (Found via Matt Cutt's Blog

PROTECT IP / SOPA Breaks The Internet from Fight for the Future on Vimeo.

Watch the video. Then Google "SOPA" for more information. You can also sign a petition against the act here. Visit Matt Cutt's Blog Post for some more links to information and ways you can get involved. Join with me and so many others in speaking out against SOPA!

Android vs. iOS

My iPhone 4S came this last week. Previously, I was a proud owner of the HTC Evo 4G, a great phone. But still I switched. Since I've had a few people ask me why, I decided I'd better just blog it. Sorry for my rambling.

Both platforms have their advantages. I suppose I'll go platform by platform and examine some pros and cons:

iOS

With iOS 5, an iPhone was finally a valid alternative to my Android phone. Previously, I never would have switched, because of the lack of a good notifications system. If a push notification came in, it would interrupt whatever you were doing, sometimes to the effect of restarting your progress in a game or something. This in and of itself was an annoyance, but could be overlooked. The main problem was that once a notification popped, if didn't immediately go to that app and take care of it, it was gone. Poof. You couldn't decide to leave this or that notification for later, because they weren't being stored anyplace. That all changed with Notification Center in iOS 5.

This was basically just a modified copy of Android's version of notifications, which they got right from the beginning. Drag from the top of the screen and you have a list of all the notifications which you have not yet acted upon. In my opinion, Apple did it even better than Android -- you can define the order in which notifications are shown manually, or you can have it based on the time of the notification. You can also act on the notification straight from the lock screen by sliding-to-unlock using the notification icon. (Hard to explain, ask someone with an iPhone to show you if you don't know what I mean) It's a pretty schweet feature.

Other things iOS has going for it:

  • Very stable (rarely crashes, almost never have to reboot, etc)
  • Very clean (everything is hardware accelerated, so every action is smooth out of the box)
  • Higher resolution screen than most Android phones, even the ones with bigger screens. People complain about the screen size on iPhones, but you don't even notice because of how clear the retina display is.
  • More secure (all apps are sandboxed, making malware pretty much nonexistent)
  • More polished apps (I don't know why this is the way it is -- perhaps because of Apple's app store policies, but apps are so much polished on average. It's true of both OSX and iOS) This is a huge one. The general level of quality, both of games and of daily-use apps, is much higher on iOS than on Android.
  • Better hardware, with very little fragmentation. You have great battery life, on every iPhone. It's very compact for its power. You almost never have to worry about your phone not meeting "minimum requirements." This is probably a large part of what makes the previous statement (polished apps) true -- developers can focus on the quality of the app, rather than focusing on making the app work for hundreds of different phones. You're also guaranteed to get all the iOS updates for at least a couple of years, something which 99% of Android users do NOT get.
  • Great camera. Yes, some Android phones also have great cameras, but many of them are pretty much junk.

The iPhone is a pretty great phone, but it does have some cons:

  • Limited customization. Can't replace the stock keyboard, very few homescreen customization options, etc.
  • Sandboxed apps. This is both a pro and a con. Apps can't really interact because they're sandboxed. You also don't have a file system in the classic sense, which makes it more limited for a computer replacement. Most people don't care about this con, but it's still there.

Those are the only cons I can think of right now, leave a comment if you find one I missed.

Android

Android is also a great platform. I love Google, and love their products. Things Android has going for it:

  • The same type of notification system, tried and true.
  • Tons of customization options. There are many great options for homescreen apps which can change the look and functionality of your homescreen and app drawer drastically
  • Separation between homescreen and general apps. This means you can use your homescreen more like the desktop on your computer, with only apps you use often showing there, and the rest hidden in the "app drawer," which you can open at will from your homescreen.
  • True multitasking. This means that apps can run in the background and perform tasks without being open, which is very uncommon in iOS, which utilizes push notifications. However, it can have performance and battery ramifications, which we will explore in the cons section.
  • Apps have more freedom. This means that you can have Google Voice, for example, seamlessly integrate with the phone app, as opposed to being separate. It also allows problems like malware.
  • Dedicated menu and back buttons. This is both a pro and a con, but I find myself missing a dedicated back button on iOS, and having these buttons as hardware buttons saves space on the screen which would be taken up by these buttons.
  • Much better integration with the Google-sphere. Since everything I use is Google, (Gmail, Calendar, Contacts, Voice, Talk), having this integration is really nice. You can get pretty decent integration in iOS, but it takes more setup and is not as seamless.

Here are some cons:

  • Instability. I had to restart my phone every few days because it would become unresponsive or strange bugs would rear their heads. A pain when your phone takes a few minutes to restart.
  • Reliance on phone manufacturer for operating system updates. This is a big one -- unless you've rooted your phone and are using custom roms, most OS updates either never come, or are 6 months to a year late, and by then a new version has been released. This also causes problems with app requirements.
  • Inconsistent hardware quality. Another big one. Most Android touch screens are not as precise or as quick to respond as the iPhone screens. In addition, some of the Android phones are cheaply made, and/or have really bad battery life.
  • True multitasking. Like I said, this is a pro and a con. The con is that a frozen or buggy app can rampage in the background, sucking battery power and processing power. Even the ones that don't act up can continually such battery life in the background. In addition, apps are much harder to kill on Android, since you have to go into system settings and wade through the list of apps to do it.
  • Malware. Without Apple-esque restrictions on apps, there is much more danger of malware. This is becoming an increasing issue as Android becomes more popular.
  • Lower app quality standard. We've visited this already.

Summary and Other Resources

I decided to switch from one of the better Android phones to an iPhone 4S. And unless something big changes in the future, I'll never go back. The stability and polish is important to me, as is battery life. I also will get the newest versions of iOS right as they are released for at least a few years, where Android users are often left out in the cold when new versions of Android come out. What you get will depend on what's most important to you. But even as a power user, I chose iPhone.

Here's a recent article on the subject: Link

Sorry for my rambling, be sure to leave a comment below with your opinions.

More on Python

My good friend Chad recently started learning about Python. After getting a fair bit into a book on the subject, he posted on Google+:

Language seems cool, though I haven't found a compelling reason to need it. I think Django is the main reason I want to use it. Calling upon the powers of +Julio Carlos Menendez and +Colton Myers to give me some examples where using Python is WAY better than another language.

I decided to try to collect some of the research that I found while selecting Python as my latest language-of-choice, and post it here. Hopefully it will be helpful to anyone who's trying to decide if Python is worth picking up. Be aware, though, this is going to be more a random collection of thoughts and links rather than a linear blog post on the awesomeness of Python.


Community

I suppose one of the first things I should point out, before we get into any language specifics, is the community. I love the Python community. Whether it's the mailing list(s), or the #python channel on Freenode, the community is very active and very helpful. In fact, the #python channel is one of the most active on my IRC client. With all that help available, solving problems becomes much less daunting than in other languages.


Indentation as Syntax

One of the most immediately obvious things people notice about Python is the significance of whitespace. This is in contrast to most every other modern programming language, and often throws people when they first discover it. I think of this part of Python as a very positive feature of the language: having indentation as part of the actual syntax of the language makes for very consistent code -- across almost every Python program, control structures look the same, because the indentation delimits code blocks, rather than braces. If this feature seems weird or unnatural to you, I suggest you try writing in Python for your next project -- the initial "weirdness" of this feature rapidly fades away, and if you're like me, you find yourself appreciating the innate readability that it gives Python code.


Intuitive Language Design

For this next section, I'll quote an article by Eric Raymond, where he details his early experience with Python:

My second [surprise] came a couple of hours into the project, when I noticed (allowing for pauses needed to look up new features in Programming Python) I was generating working code nearly as fast as I could type. When I realized this, I was quite startled. An important measure of effort in coding is the frequency with which you write something that doesn't actually match your mental representation of the problem, and have to backtrack on realizing that what you just typed won't actually tell the language to do what you're thinking. An important measure of good language design is how rapidly the percentage of missteps of this kind falls as you gain experience with the language.

When you're writing working code nearly as fast as you can type and your misstep rate is near zero, it generally means you've achieved mastery of the language. But that didn't make sense, because it was still day one and I was regularly pausing to look up new language and library features!

This was my first clue that, in Python, I was actually dealing with an exceptionally good design. Most languages have so much friction and awkwardness built into their design that you learn most of their feature set long before your misstep rate drops anywhere near zero. Python was the first general-purpose language I'd ever used that reversed this process.

I've experienced this myself -- Python's design is such that it works much more fluidly with the solutions as they live in my head. I can just start coding and the solution flows easily from my brainwaves to working Python code. Try it, I think you'll be surprised.

While on the topic of Python's intuitiveness, I think we should talk about IDEs. C# is one of my favorite languages. Using Visual Studio, you can create very advanced and full-featured GUI applications on Windows with relative ease. However, they key part of that last statement is "Using Visual Studio". I find when I'm writing in Java, C#, Objective-C, etc, I end up relying heavily on intellisense to help me recall syntax and method names. In contrast, Python is designed so intuitively that I find that I am able to write full-featured programs with only Vim and a few trips to the Python documentation to refresh the name of a certain function within a certain module. I think that's another testament to the great design of Python, and just how intuitive it is.

In addition, though C# can be used to create great Windows GUIs, Visual Studio is again used to abstract away thousands of lines of GUI code which is generated by Visual Studio as you code the GUI. Have you ever tried to write a C#, Java, or Objective-C GUI in a non-IDE text editor such as Vim? It's nigh unto impossible, because the syntax is so verbose and not intuitive enough to easily remember. Contrast this with Python's Tkinter toolkit, which allows one to create GUIs with relative ease, and with no reliance on an expensive IDE. And the Python GUIs look native on each platform.


Java vs. Python

Might as well throw in a side-by-side comparison of Java vs. Python. Shows some of the differences in verbosity and complexity between the two languages, even if only in small examples.

Java vs. Python


Conclusion

Obviously Python is not perfect. There are definitely downsides to having a completely dynamically-typed language like Python -- if you're not careful, you can have difficult-to-find bugs crop up. And the task at hand can sometimes require the speed of C, for example, or features from other languages -- Python is not a cure-all, and I don't pretend that it is. But for me, it's pretty close.

What it comes down to is that I've come to really enjoy programming in Python, finding it intuitive, straightforward, and full of features that make my job easier as a programmer. And I hope that it treats you as well. =)