New domain and migrating data between Google Apps

Short version


  • If you can't reach me at my @cgoncalves.info, use @cgoncalves.pt from now on.
  • Google Apps doesn't allow one to switch a primary domain with an alias. Had to migrate all data by myself...


A not so short version

Holding the cgoncalves.info domain since 2007, I had it associated to a free Google Apps account ever since.
Recently FCCN (Fundação para a Computação Científica Nacional) decided well it was time to liberalize the .PT TLD to everyone and not just to companies or trademarked brands. Now for ~17€/year any individual can get a .PT domain, no bureaucracy in the way. Hurray!

I did take the opportunity and acquired the cgoncalves.pt domain. It's simple, elegant and somewhat represents pride of being a Portuguese citizen. This domain's just what was pending to fulfill, as of now, the digital image of myself.

With this new domain I could add it to my existing Google Apps account and make it an alias of cgoncalves.info. Although as I'm so existed and convinced I'll keep this new domain till my death, I gave a thought (2 seconds, if much) and concluded I should settle with the domain as the primary one instead of the rather deprecated cgoncalves.info. "So what?" you ask. Well... Google doesn't allow their Google Apps users to switch a primary domain with an alias! Best chance to accomplish this is to create a new Google Apps account associated with the .PT domain and migrate all data from one account to the other, including emails, filters, contacts, calendars, feeds, etc. Moving contacts, calendars and feeds is a trivial action but hundred of thousands of emails from years?! No fun...

Once migration completed, delete the cgoncalves.info Google Apps account and add that domain to the new account as alias was the goal to achieve.

Migrating emails

I started migrating emails between the two Google Apps accounts using the POP3 fetcher GMail provides. Yes, it take ages and only fetches up to 200 emails at a time, but that was the faster way I know. The fetcher automatically retrieves for messages at different rates from accounts and depends on previous mail fetch attempts (the less mail retrieved, the less frequency it will check for new mail). Frustratedly even with thousands of messages yet to get, the intervals were set longer and longer (sometimes to two hours!)... Enabling "Refresh POP accounts" lab ("Fetch messages from your POP accounts on demand by using the refresh link on to of the inbox") would cut the time between fetches but was no usable solution as it required human interaction by clicking on the gear icon now and then.

I also tried this script and threw one dollar (minimum donation amount required) to let me unlock the fixed 8 minutes interval and empower me to set it to a higher rate (down to 3 minutes). It was working like a charm and the migration process was flying fairly decently. BUT NO! OH NO! GMail stopped successfully retrieving messages and started emailing me an unnumbered "Message left on server" messages. Perhaps the daily quota has been reached, or not as it kept that way the day after. This POP3-me thing was running for now a full week and I still had many messages left to migrate. I was doomed!

With the POP3 process discarded there was just one option left and that I was battling to avoid: transfer from IMAP. Don't get me wrong, I love and use IMAP in all my accounts. The side of the coin is I'd have to perform the migration on my end and reached the first 500 messages transferred, only 1 message per second is synced. Imapsync was just the application suited to do the job. It detects duplicated emails and all, great! I installed it on my VPS (so I can't put my laptop to sleep at will) and ran it as follows:

#!/bin/bash
imapsync \
--host1 imap.gmail.com \
--user1 account@old-domain.com \
--passfile1 ./passfile1 \
--ssl1 \
--host2 imap.gmail.com \
--user2 account@new-domain.com \
--passfile2 ./passfile2 \
--ssl2 \
--authmech1 LOGIN \
--authmech2 LOGIN \
--useheader 'Message-Id' \
--useheader 'X-Gmail-Received'

File passfile1 and passfile2 contains the respective email accounts passwords (plain text). And voilá! It might take several hours/days but I didn't know (and still don't) no better solution. And remember, POP3 mail fetcher doesn't copy sent mail, but imapsync does! Win!

Back to Google Apps drama

At last with data migrated I let things stabilize for a couple of days before taking any further action. Next step was ensuring I did not depend anymore on my @cgoncalves.info Google Apps account: check! Logged into the Google Apps administration panel of the new domain and tried to add cgoncalves.info as an alias. It failed with something like "The domain is currently associated to another account". That was right, I still had to purge the former account before proceeding: done! Retried once again hoping I got lucky but no... "This domain name has already been used as an alias or domain". It seems I will have to wait up to seven days before I can add the .INFO domain as an alias.

Conclusion

Update my contact record on your address book and replace everything ending with cgoncalves.info to cgoncalves.pt. I promise I'll keep using this domain for decades, unless someone steps in with a bag full of money and buys me off to concede the ownership of the domain. Starting bid is 1000€. I mean it! :-D

SOCKS proxy over SSH

Here goes a quick and valuable tip people don't usually know, or at least a couple of friends of mine were not aware of. The two keywords will be SOCKS and SSH.

Connecting over to a remote server, one can bypass unauthorized access blocked by usually our network firewall. The most common examples given are restricted access to Facebook, MSN or torrents. We can also have unlimited access to all kinds of contents thanks to the nice sysadmin managing the network who puts no barriers whatsoever, but we may not trust him or the network users to permit unencrypted data flowing openly. Having some way to establish an encrypted connection to a known network who we trust, or distrust less, and tunneling over it would be safer. This is where SOCKS and SSH can be much of helpful!
SOCKS is an Internet protocol that routes network packets between a client and server through a proxy server. -- via Wikipedia
I'm certain as most, if not all, of our daily applications implement themselves some way of proxying (mostly HTTP, HTTPS and SOCKS) or use the system-wide configured network proxy server. Now all we need is a remote and secure server to proxy and tunnel. This is the part where SSH enters:
Secure Shell (SSH) is a network protocol for secure data communication, remote shell services or command execution and other secure network services between two networked computers that it connects via a secure channel over an insecure network: a server and a client (running SSH server and SSH client programs, respectively). -- via Wikipedia
What this means is if we have a SSH-enabled server we can take advantage of it because typically no further configurations or tweaks are needed. Let's take a look at the SSH client manual:

-D [bind_address:]port
Specifies a local ``dynamic'' application-level port forwarding. This works by allocat-
ing a socket to listen to port on the local side, optionally bound to the specified
bind_address. Whenever a connection is made to this port, the connection is forwarded
over the secure channel, and the application protocol is then used to determine where to
connect to from the remote machine. Currently the SOCKS4 and SOCKS5 protocols are sup-
ported, and ssh will act as a SOCKS server. Only root can forward privileged ports.
Dynamic port forwardings can also be specified in the configuration file.
Got it? Great! Let's combine SSH+SOCKS:
ssh -C2qTnN -D <PORT> <USER>@<IP>
I will not provide the details of what each option features - run 'man ssh' and find by yourself. The result is a SOCKS proxy over SSH. Now let's wrap it in a bash script:
#!/bin/bash

if [[ `uname` == 'Darwin' ]]; then
trap " {
echo \"Setting SOCKS proxy down...\" ; \
networksetup -setsocksfirewallproxystate ethernet off ;
networksetup -setsocksfirewallproxystate wi-fi off;
exit 1;
}" ERR INT TERM EXIT

networksetup -setsocksfirewallproxystate ethernet on
networksetup -setsocksfirewallproxystate wi-fi on
fi

ssh -C2qTnN -D 9999 <USER>@<IP>
Replace <USER>@<IP> by your username and IP address. This bash script, in case you are a Mac user, will automatically turn SOCKS on upon execution and turn it off when shutting down (Mac users using an OS X version prior to Lion should replace "wi-fi" by "airport"). Other Unix users (Linux, BSD, etc) should set SOCKS host as "localhost" and SOCKS port "9999" in either your system network configurations or in each application you want to tunnel over.

Apple MacBook Pro 13" battery history


One year has passed since I bought my Apple MacBook Pro 13" (mid 2010) laptop, and at that time I blogged about it. One feature I demanded was good battery capacity - the MBP 13" seemed like a great choice and I did go for it.

By middle of August I discovered coconutBattery, an application that shows the current battery capacity, its designed capacity, and current and maximum charge, as well as age of the laptop, battery load-cycles, temperature and power usage. One additional feature that popped-out right away was the ability to save the maximum battery capacity and as so since August 12, 2010 to today I've recording these statistics with a fully charged battery to later analyze how my laptop's battery health changed over time. That time has just ran out so let's take a quick look over it!

First, data extraction. A config.xsl file was created with the following content:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- Edited by XMLSpy® -->
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">
<html>
<body>
<h2>My Apple MacBook Pro 13" (mid 2010) battery history</h2>
<table border="1">
<tr bgcolor="#9acd32">
<th>Date</th>
<th>Current Capacity</th>
</tr>
<xsl:for-each select="database/object[@type='SAVEDDATA']">
<tr>
<td><xsl:value-of select="attribute[@name='date']" /></td>
<td><xsl:value-of select="attribute[@name='capacity']" /></td>
</tr>
</xsl:for-each>
</table>
</body>
</html>
</xsl:template>
</xsl:stylesheet>
Then, I fired up my terminal and ran:
$ xsltproc --nonet config.xsl ~/Library/Application\ Support/coconutBattery/\
coconutBattery.xml | sed 's|\(.*\)% (\(.*\) mAh)|\2|' > battery.html

This generated a battery.html file with 146 records dumped to a HTML table:

(table dump sniped)

And here is a chart of it:



And finally a screenshot of my coconutBattery:

Battery load-cycles is at 58 after one year and one month and a half. Note that the age of my Mac that reads above is 15 months but the accurate age is 13 months (the former is time since manufactured).

Do you also log your battery health over time? Have these kind of data? Please share it with us!

SSH connection automation

Part of my daily routine involves accessing and managing a considered amount of remote servers through the SSH protocol. I use screen a lot to ease the job by having a couple of windows opened on each server, so avoiding multiple connections to the same server. By using screen not only helps me grouping server windows altogether in one console window/tab but also is a time saver in those days when network connection is not in its glory days allowing me to reattach it and carrying on being productive (*sighs*).

But, before connecting to all those machines, I used to repeat over and over the same setup steps:
  1. send my public key to server
  2. upload my screen configuration file
  3. create an alias (because ssh'ing manually is a truly hassle each time I want to connect!) to connect to the server and reattach or create a new screen session
  4. Change its permissions to 700 (security freak? Oh well...)

Four repetitive and forgettable operations that could all be avoided, sparing me some minutes of my precious time. But no more!

Here is a handy Bash script I wrote the other day (at last!):
#!/bin/bash
# $1 = [remote_user@]remote_host
# $2 = [alias name]
BIN=$HOME/bin
if [ "$#" -lt 1 ] || [ "$#" -gt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
echo "Usage: $0 [user@]machine [alias]" >&2
exit 1
fi
if [ "$#" -eq 2 ]; then
(
cat <<EOF
#!/bin/bash
ssh $1 -t screen -x -R
EOF
) > $BIN/$2
chmod u=rwx,go= $BIN/$2
fi
ssh-copy-id -i ~/.ssh/id_rsa.pub $1 > /dev/null 2>&1
scp ~/.screenrc $1:~/ > /dev/null 2>&1
exit 0

Copy and paste this code to a file in your $PATH (mine is $HOME/bin/setup-machine) and give it execution permission (chmod u=rwx,go= ).
Obvious to some, though may not be to others, here is an usage example:
Macnux:~ carlos$ setup-machine -h
Usage: /Users/carlos/bin/setup-machine [user@]machine [alias]
Macnux:~ carlos$ setup-machine cgoncalves@cgoncalves.info cgoncalves
cgoncalves@cgoncalves.info’s password: *****************
Macnux:~ carlos$ cat bin/cgoncalves
#!/bin/bash
ssh cgoncalves@cgoncalves.info -t screen -x -R
Macnux:~ carlos$ cgoncalves
[SSH connection established and attached to screen]

Ended up buying an Apple...

I have an Acer laptop for about 5 years. It includes a generously 512MB of RAM, Intel Centrino 1.6Ghz (M730 processor to be more precise), ATI Mobility Radeon x600 64MB VRAM, 80GB HDD, yada yada yada. The battery is in an awesome shape if we consider the age and the usage I put on it during this time - previously 3 hours fully charged and now around 2 hours, so... I couldn't have asked for better. Nevertheless, the laptop has an 15.4" display and weighs 3Kg with battery plus the charger which I have to take always with me otherwise the battery would ran out quickly. All this factors led me to buy a new laptop with the main features to consider and give more priority on the weight, battery and size. I didn't want to buy again a new 15.4", up to 4 or 5 hours of battery, and weighed laptop. Basically the requirements were: 12" or 13" screen, up to 2Kg, battery that lasted at least 7 or 8 hours (wireless and reasonable screen brightness), 4GB of RAM or more, and obviously a "good" brand (Sony, Lenovo, Apple, Dell, and HP were on top of the list), not to mention a decent price I could afford. A few hours/days later I decided to pick an Apple Macbook Pro 13" 2.4GHz, not because of the Mac OS X as some might be imagining by now but because of the hardware itself.

The Apple Macbook Pro 13" I bought Tuesday includes an Intel Core 2 Duo 2.4Ghz, 4GB of 1066MHz DDR3, battery up to 10 hours (wireless and 50% screen brightness), and weighs 2.04KG. It cost me 1.149€ but had an (indirect) discount of 5% plus 10€ on the FNAC client card (costs 5€ per year but worths it since it also gives you other discounts and benefits). Here is a short list of highlights regarding the hardware itself and the Mac OS X operating system which, by the way, I have to mention I have never used it until now:

Pros:
  • Laptop is solid rock and its aluminum unibody makes it truly fresh (I don't fell any heat, except in the fan area where the air flows out obviously, while the Acer laptop is more like a heater than a laptop (I bet I could fry an egg on it));
  • Touchpad pretty flexible e practical;
  • Battery lasts up to 10 hours, meaning I don't have to carry the charger wherever I go;
  • The charger has two cables: the charger with the cable to plug in the laptop and another one to extend the length of the cable to the wall socket. This means for, most of the cases, I can leave that extra cable aside or at home;
  • Auto brightness (think a little and you will find how useful it can be);
  • OS X is functional, intuitive, and has a clean UI overall.
Coins:
  • I'm used to have the Ctrl key switched with the Fn key (something I want to get used);
  • At least for the Portuguese keyboard, the square brackets as well as the curly brackets aren't shown in the keyboard and the key combination for the curly brackets isn't easy and convenient: alt+shift+8 for { and alt+shift+9 for } (keys 8 and 9 are where the parentheses are located at). Imagine how great will it to code, not!
  • No "Cut" on files?!
  • "exit" in the Terminal doesn't close the tab, but logs out and stays there opened;
  • If running the OS in Portuguese, cmd+w doesn't close the tab as expected since the shortcut isn't associated and seems there is no way to do so. If running in English, the shortcut is there and do the job;
  • Expected iChat to support the MSN protocol. Using Adium, which is way better;
  • The file (un-)compressor included by default lacks lots of features such has the capability to uncompress split files. Using BetterZip, but still missing Ark from KDE!
  • People advised me to use VLC instead of QuickTime for watching videos, specially those in HD since it seems the codecs used by QuickTime consumes more CPU than it should and that VLC consumes. I would have installed VLC anyway since I'm already used to it being the best video player out there in my opinion.

For those concerned about my devotion to the FOSS world and specially my openSUSE and KDE eccentricity, than there is nothing to you worry about! I'm still the very same guy you used to know. I just needed a laptop with the features I have stated above and that ended up to be an Apple Macbook. That's it, folks!

Technical Preview of PySide-Shiboken


Marcelo Lira (PySide developer) announced moments ago on the PySide mailing list the technical preview of PySide-Shiboken:
Hello folks,

today we got the Shiboken generated PySide bindings in a good enough
shape to do a proper release
with tarballs and all the required niceties, instead of rough git
urls. Keep in mind that this is an alfa release,
or a "technical preview" as the kids in my lawn are used to say, some
modules are missing and bugs are not
hard to find.

This PySide release contains bindings for the following modules:

* QtCore
* QtGui
* QtNetwork
* QtWebKit
* QtSvg
* QtXml
* QtTest
* QtOpenGL
* QtSql

Feel free to try your Python code with this version of PySide. You
know how it works: more users == less bugs.
As long as you guys tell us about the bugs: http://bugs.openbossa.org

A noteworthy fact is the first release of the Shiboken C++ binding
generator. Yay!
We strongly encourage it's use to produce non-Qt C++ bindings for
Python, it will make your life easier.
If it turns out to be not so easy, come and talk with us on #pyside
channel (Freenode).

Here follow everything needed for this release, in order of compilation.

API Extractor, version 0.4.0
http://www.pyside.org/files/apiextractor-0.4.0.tar.bz2

Generator Runner, version 0.4.0
http://www.pyside.org/files/generatorrunner-0.4.0.tar.bz2

Shiboken, version 0.2.0
http://www.pyside.org/files/shiboken-0.2.0.tar.bz2

PySide, version 0.3.0
http://www.pyside.org/files/pyside-qt4.6+0.3.0.tar.bz2

You will notice that the size of the generated bindings for the Qt
modules in no way resemble the figures from the Boost.Python version.
And talking about Boost: packagers, be aware that no component depends
on Boost::Graph anymore. Yay^2!

P.S.: cgoncalves, thanks for all the code. And to everyone else that
provided bug reports, patches, beers, etc.


And I followed by adding the usual pos-release-announcement announcement:

Don't want to get your hands dirty and have headaches due to compilation
issues? That's bloody easy! Get this very same release version already
packaged for your distribution from the openSUSE Build Service! Packages
(32bit and 64bit) available for the following distributions:

- openSUSE 11.0, 11.1, 11.2 (for both Qt 4.5 and Qt 4.6 version available),
and Factory
- Fedora 11 and 12
- Mandriva 2009.1 and 2010

You can install PySide package by adding the proper repository from
http://download.opensuse.org/repositories/home:/cgoncalves:/pyside:/shiboken/
to your package manager (zypper, yum, urpi, smart or any other rpm-md
compatible) followed by the installation of package 'python-pyside'.

But (!) in case you are all a bleeding-eye person, or simply if you want to
take an extremely important role on the development by testing it and
reporting back to developers, you can even get the regularly snapshots taken
from PySide's Git mainline by adding the devel repository[1].

Or if you happen to be from the old-school and love PySide Boost based you can
still install and use it[2].

I would like to take the opportunity to congratulate and thanks the PySide
team for their magnificent effort given to the project. You guys rock!


[1] http://download.opensuse.org/repositories/home:/cgoncalves:/pyside:/devel/
[2] http://download.opensuse.org/repositories/home:/cgoncalves:/pyside:/boost/

Stable versions of PySide packaged


Heads up! I'm glad to inform you all that now stable/released versions of PySide are also available for Mandriva and Fedora besides for openSUSE and snapshots as previously announced. I've updated the download page to reflect this change.

Fedora and Mandriva users: please give me feedback about these packages.
top