An Arduino timer using old HP 5082-7302 displays

My latest little project was to build something out of a handful of old HP 5082-7302 numeric indicators and my home-made breadboarded (almost) Arduino. These displays are very easy to use once you get past the errors in (some versions of) the datasheet. That is fortunate because they stopped making them a long time ago and there isn’t much information or libraries available for controlling them.

The result was a timer that counts up from zero and with a button for resetting it, and a pair of Arduino libraries. The first library, HPNumberDisplay, can control an array of HP5082-7300/7302 (and probably 7340) displays. The second library, TimeCounter, can count (increment/decrement) time by arbitrary amounts. All source code for the libraries are available here.

A timer build from an Atmega328p and five HP5082-7302 displays.
A timer build from an Atmega328p and five HP5082-7302 displays.

I also recorded a short demo using my old camera. The quality could be better. Some day I will buy a proper camera.

 

Some notes on how to use FreeRTOS with a FRDM-KL25Z (almost) without Processor Expert

After a pause while I waited for new hardware to arrive and enough free time to play with it I returned to experimenting with my FRDM-KL25Z. After having tried both the Mbed library and Processor Expert I thought it was time to remove the training wheels and learn to use this thing properly. Both Mbed and PE have their uses, but they hide many important details about how the device functions that I need to know. After figuring out how to configure GPIO pins manually (I recommend this series of articles if you want a gentle introduction to how to use clocks and GPIO pins on the KL25Z) and writing a simple program in C++ (C is a good language, but C++ is so much easier to work with) the only thing I was missing was FreeRTOS. This will not be a complete guide to installing FreeRTOS. I will refer you to other guides that can take you through that part.  These are only a few notes and suggestions that should help you avoid some of my mistakes.

As you may already know, there is no official port of FreeRTOS for the Kinetis L series of ARM Cortex M0+ MCUs. Fortunately there is an unofficial port by Erich Styger. The bad news is that the port is built around PE, and your will need to set it up together with Eclipse if you want to use FreeRTOS. The good news is that you only need to use it once and export the necessary code. Once you are done you can use whatever development tools you want.

Before you go any further you must follow Styger’s instructions for how to set up PE with Eclipse, and while it isn’t necessary you should also read his instructions for setting up and configuring FreeRTOS with PE. Play around with it. Make sure that it is working. Then read his instructions for generating static code from the FreeRTOS PE component that can be extracted and used on its own.

If you follow the instructions you will be left with a folder with all FreeRTOS code. Now comes the tricky part: using it in a project without PE. The exact details will depend on the tools and libraries you are using. In my case I wanted to create a C++ project in Eclipse using the GCC-ARM Eclipse plugin. I will go through some of the steps I had to take to get it to work and point out a few things that gave me trouble.

Once you have created a static copy of the FreeRTOS code,  move it into your new project.folders1

I put all FreeRTOS files in one folder to simplify things.

freertos_folder1

Open the project properties and make the FreeRTOS folder a source folder.

include1

Go to the build settings and also add the folder to the include path.

source_folder1

This should be enough for Eclipse to find all the new files, but if you now try to compile it you will get a long list of errors. Most of these are simple things that are easily fixed.

Getting interrupts to work was the hardest part. It is quite simple once you know what the problem is, but it was not obvious. The FreeRTOS code you have is set up to use the interrupt handlers and interrupt vectors created by other PE components. Since you do not have those components you must modify some parts of the code.

First, copy the ‘Events.h’ and ‘Events.c’ files from your PE project, remove all interrupt handlers not starting with FRTOS_ (assuming there are any), and then remove the FRTOS_ part of all function names (e.g. vApplicationIdleHook instead of FRTOS_vApplicationIdleHook). Do a search through all your other FreeRTOS code and do the same there (or wait for the compiler to complain). Some function names will be incorrect. If you are using C++ also make sure that the event files are enclosed within extern blocks. They will be used both by FreeRTOS’s C code and your C++ code.

#ifdef __cplusplus
extern "C" {
#endif

...

#ifdef __cplusplus
}
#endif

Next, if you are not using the interrupt vector you got from PE and instead uses the one from the GCC-ARM plugin (‘vectors_MKL25Z.c’), then you will have to change some function names. FreeRTOS uses a couple of interrupt handlers.

  • SysTick_Handler
  • SVC_Handler
  • PendSV_Handler

Your FreeRTOS code and ‘vectors_MKL25Z.c’ will use different names for these interrupt handlers. This must be fixed.

In ‘portmacro.h’, replace the lines 291-293

void vPortSVCHandler(void); /* SVC interrupt handler */
void vPortPendSVHandler(void); /* PendSV interrupt handler */
void vPortTickHandler(void); /* Systick interrupt handler */

with

void SVC_Handler(void); /* SVC interrupt handler */
void PendSV_Handler(void); /* PendSV interrupt handler */
void SysTick_Handler(void); /* Systick interrupt handler */

In ‘port.h’, replace line 727

void vPortTickHandler(void) {

with

void SysTick_Handler(void) {

The line 848

__attribute__ ((naked)) void vPortSVCHandler(void) {

with

__attribute__ ((naked)) void SVC_Handler(void) {

And the line 999

__attribute__ ((naked)) void vPortPendSVHandler(void) {

with

__attribute__ ((naked)) void PendSV_Handler(void) {

The rest should be easy. There will be some inclusions of header files to remove and some that must be added, and a few function and variable names will be wrong. You also have to configure FreeRTOS (unless you did so before exporting the code). This is done in the ‘FreeRTOSConfig.h file’. Refer to FreeRTOS’s documentation for more information about that part.

Backing up OpenVZ ploop snapshots

As I mentioned in an earlier post I have been using snapshots to back up my OpenVZ servers. At first I used a slightly modified version of the backup script found on the OpenVZ wiki to create backups, and that worked fine except for one missing feature: logging.

This seemed as good an opportunity as any to start learning some Python (which to my great embarrassment I have not used much before). The result is a small Python script OVZ-Backup that backs up all OpenVZ containers (or a subset of them) using ploop snapshots and rsync. If there are errors it logs them with syslog and (optionally) sends out error messages to a list of email addresses/users.

The script sends out email using the ‘mail’ command and can only send them to local users as is. My last HowTo explains how you can use postfix to forward user email to an external address and relay emails sent from local email clients without SMTP support through your email provider’s SMTP server. This would let you send error messages to any email address you want.

Encrypting and forwarding local email to an external email address

Last week I set out to find a reliable and permanent solution to my problems with unread system mail and undelivered cron error messages, and I can now see that sending, delivering, and receiving email is far more complicated than I had thought. I have had to learn more about SMTP, SSL/TLS, and email delivery than I ever wanted to know, and I still only know the bare basics.

When I started I had two requirements:

  1. All local email sent to the root user (and preferably everyone else) must be forwarded to an external SMTP server and email account
  2. All email that leave my network must be encrypted

After experimenting with number of different applications (GNU Anubis, Nullmailer, ESMTP, MSMTP, etc) I finally found something that worked; Postfix with GPG-Mailgate. If all you need is the ability to send email to an external account then there are other applications you could use instead (I had some success with MSMTP and Nullmailer). The advantage of using Postfix is its flexibility and maturity. All other applications I tried had some small thing they could not do reliably or would fail in some edge cases, such as only forwarding some of the emails sent to root, but not quite all of them.

This guide will show each step needed to set this up, and a few mistakes to look out for. The first part will cover Postfix and how to configure it to forward all emails sent to a local user to an external email account. The second part will show how to set up GPG-Mailgate so that all emails that are sent to certain accounts are encrypted with GPG before they leave the server. If all you need is mail forwarding then you can stop after the first part, and if you already have Postfix configured you can jump straight to the second part.

A few assumptions will be made in this guide.

  1. You are using the root account. If you are not then you will have to prepend ‘sudo’ to some commands.
  2. You are using Debian. Everything should still work if you are using a different Linux distribution, but you may have to make some minor changes.
Names and servers

I will be using the following user names, email accounts, and servers in this guide.

User names

root : The local root user

gpgmap : The user account used for GPG-Mailgate

Email accounts

root@server.localdomain.com : The fully qualified email address of the root user

user@emailprovider.com : An external email account that we want to send email through

admin@anotherdomain.com : An external email account that we want to forward all local emails to

Domains

yourdomain.com : Your local domain

emailprovider.com : The domain of your email provider

anotherdomain.com : The domain of the email provider for the account you want to forward all local email to

Servers

smtp.emailprovider.com : The SMTP server of your email provider

server.yourdomain.com : The computer you want to forward local emails from

Part 1 – Forwarding local email to an external SMTP server

The first step is to install and configure Postfix to forward all emails sent to the root user (or any user you want) to an external SMTP server and email account. This is not difficult, but it does require a few steps.

First, if you do not already have Postfix install it and some dependencies.

apt-get install postfix libsasl2-modules

Fill in any domain names the installer asks for and choose the “satellite” option.

If you are not already using Postfix then you are probably running Sendmail. Before starting Postfix you will have to turn it off. To avoid future error messages and warnings I also recommend you completely uninstall it.

service sendmail stop

apt-get remove sendmail-base sendmail-cf sendmail-doc

Next, start postfix.

service postfix start

Make sure that it started correctly.

service postfix status

Next comes the hard part (not really); configuring Postfix. The exact details depends on your email provider. Everything I describe here will be completely compatible with Gmail.

Most of the settings you need should have been configured for you by the installer.

Open ‘/etc/postfix/main.cf’ and make sure that the relay host is set to your email provider’s SMTP server.

relayhost = smtp.emailprovider.com:587

And add these lines:

smtp_use_tls=yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

Next, create a password file ‘/etc/postfix/sasl_passwd’ and add the following to it:

smtp.emailprovider.com:587 user@emailprovider.com:yourpassword

The SMTP hostname and port must match relayhost in the main.cf file.

Update Postfix’s password database and settings.

postmap /etc/postfix/sasl_passwd
service postfix reload

Everything you need to send emails through Postfix using your email provider’s SMTP server should now be in place. Before proceeding you should test that it is working.

echo "Testing Postfix email delivery" | mail -s "Test email" admin@anotherdomain.com

If the email doesn’t arrive check in ‘/var/log/sysIog’ for errors and make sure that you can successfully send emails using the same settings from an email client (such as Thunderbird).

Now that that is working we only need to tell Postfix to forward all emails sent to the root user to an external email account. Open ‘/etc/aliases’ and add the following line:

root: admin@anotherdomain.com

If you want to forward emails for other users than root you simply add them to the list as well. When you are done run:

newaliases

That’s it. If everything is configured properly all emails sent to the root user will now be forwarded to admin@anotherdomain.com.

echo "Testing Postfix forwarding" | mail -s "Test email" root

If that was all you needed then you can stop here, but you are now sending unencrypted emails through the internet containing potentially sensitive information. If you think that sounds dangerous then continue to the second part were we will set up some automated GPG encryption.

A warning about restrictive SMTP servers and the From header

It is possible that these settings won’t work with your particular email provider. I’m using a Gmail account to forward email. It isn’t my primary email provider, but there are practical reasons for using it in this case. Unlike some other email providers Gmail does not care about what you specify in the ‘From’ header. It will overwrite whatever is in it with the Gmail user you used. This simplifies things greatly when forwarding local emails since you would otherwise have to change the contents of the header before sending it out. Postfix has the ability to replace addresses in outbound email (just google smtp_generic_maps or sender_canonical_maps and you should find instructions for how to configure Postfix), however, depending on how an application fills out the headers in emails they send out this may not work reliably in all situations. I tried to set this up with a more restrictive Zoho.com email account, but I could not get it to forward cron error messages.

If you need this for your email provider, and if you manage to set it up successfully, please leave a comment explaining how you did it. For everyone else; just use a Gmail account. It’s easier.

Part 2 – Encrypting outgoing email with GPG

It is possible that some of the applications sending email to your local accounts might send data that you don’t want any stranger on the internet to see. In my paranoid mind that just isn’t acceptable, and so we will be adding a layer of encryption to Postfix using GPG-Mailgate. GPG-Mailgate is a content filter script for Postfix that will encrypt a received email if there is a public GPG key available for its recipient, and if the email is not already encrypted. It is relatively easy to install and configure, but be warned that if you do not configure it correctly it will probably fail silently and send out empty emails.

First, either download the source code from Github or clone it.

git clone https://github.com/ajgon/gpg-mailgate.git

Next, manually put everything where it needs to be. Replace python2.6 with your python version.

cd gpg-mailgate
cp gpg-mailgate.py /usr/local/bin/gpg-mailgate.py
cp -r GnuPG /usr/lib/python2.6/

Make sure that all permissions are correct.

chown root:root /usr/local/bin/gpg-mailgate.py
chmod 755 /usr/local/bin/gpg-mailgate.py
chown -R root:root /usr/lib/python2.6/GnuPG
chmod 755 /usr/lib/python2.6/GnuPG
chmod 644 /usr/lib/python2.6/GnuPG/__init__.py

Create a user to run the GPG-Mailgate script as and import the public key you want to encrypt forwarded email with.

useradd -s /bin/false -d /var/gpg -M gpgmap
mkdir -p /var/gpg/.gnupg
chown -R gpgmap:gpgmap /var/gpg/.gnupg
chmod 700 /var/gpg/.gnupg
sudo -u gpgmap /usr/bin/gpg --import yourpublic.key

yourpublic.key is the public part of your GPG key pair. If you don’t have one already then your will need to create one. How to do that is beyond the scope of this guide, but it isn’t difficult.

Check that everything worked and that the key is in place.

sudo -u gpgmap /usr/bin/gpg --list-keys  --keyid-format long

This should give you something like this.

/var/gpg/.gnupg/pubring.gpg
 ---------------------------
pub 4096R/0123456789ABCDEF 2014-09-17
uid                        Your Name <admin@anotherdomain.com>
sub 4096R/FEDCBA9876543210 2014-09-17

Save whatever you have instead of ‘0123456789ABCDEF’. This is the identifier of your public key and you will need it later.

Add the following to the end of ‘/etc/postfix/master.cf’:

#GPG-Mailgate
gpg-mailgate unix - n n - - pipe
flags= user=gpgmap argv=/usr/local/bin/gpg-mailgate.py ${recipient}

127.0.0.1:10028 inet n - n - 10 smtpd
  -o content_filter=
  -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
  -o smtpd_helo_restrictions=
  -o smtpd_client_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8
  -o smtpd_authorized_xforward_hosts=127.0.0.0/8

Make certain that the user name and script location matches what you used above.

Add the following to ‘/etc/postfix/main.cf’:

content_filter = gpg-mailgate

And reload Postfix’s settings.

service postfix reload

Create a configuration file for GPG-Mailgate at ‘/etc/gpg-mailgate.conf’ and add the following to it:

[default]
# whether gpg-mailgate should add a header after it has processed an email
# this may be useful for debugging purposes
add_header = yes

# whether we should only sign emails if they are explicitly defined in
# the key mappings below ([keymap] section)
# this means gpg-mailgate won't automatically detect PGP recipients
keymap_only = yes

[gpg]
# the directory where gpg-mailgate public keys are stored
# (see INSTALL for details)
keyhome = /var/gpg/.gnupg

[logging]
# For logging to syslog. 'file = syslog', otherwise use path to the file.
file = syslog
verbose = no

[relay]
# the relay settings to use for Postfix
# gpg-mailgate will submit email to this relay after it is done processing
# unless you alter the default Postfix configuration, you won't have to modify this
host = 127.0.0.1
port = 10028

[keymap]
# You can find these by running the following command:
# gpg --list-keys --keyid-format long user@example.com
# Which will return output similar to:
# pub 1024D/AAAAAAAAAAAAAAAA 2007-10-22
# uid Joe User <user@example.com>
# sub 2048g/BBBBBBBBBBBBBBBB 2007-10-22
# You want the AAAAAAAAAAAAAAAA not BBBBBBBBBBBBBBBB.
#user@example.com = <gpg key id>
admin@anotherdomain.com = 0123456789ABCDEF
root@server.yourdomain.com = 0123456789ABCDEF

Replace the addresses and keys on the last two lines with the email address you want to forward emails to and the identifier of the public key you imported earlier.

The final line is a bit of a hack. GPG-Mailgate uses the ‘To’ header to find the GPG key to use when encrypting email. If it can’t find a matching entry in the configuration file it will not encrypt it. This creates a minor complication since Postfix doesn’t rewrite the ‘To’ header when forwarding local email. Any email sent to the root user will therefore have a ‘To’ address of ‘root@server.yourdomain.com’; which GPG-Mailgate won’t recognise. To fix this we add the local email address to the configuration file as well.

This should be all. Now test that everything is working.

echo "Testing GPG encryption" | mail -s "Test GPG" admin@anotherdomain.com
echo "Testing GPG encryption to root user" | mail -s "Test GPG root" root

These commands should both send an encrypted email to admin@anotherdomain.com. If it doesn’t work check ‘/var/log/syslog’ for errors.

If GPG-Mailgate did not encrypt your emails then it is likely that it did not find a matching public key. Make sure that the keys and addresses in ‘/etc/gpg-mailgate.conf’ are correct.

If you get empty messages then it is likely that GPG returned an error to GPG-Mailgate. This will cause GPG-Mailgate to fail silently. Make sure that the gpgmap user has the permissions needed to use the GPG keys in ‘/var/gpg/.gnupg’ and that gpgmap is used by Postfix when running GPG-Mailgate.

If you get an error about a missing GnuPG module then you either set the wrong permissions for the GnuPG folder and its contents, or you placed it in the wrong python folder.

If everything is working than this should be it.

  • You can now send email through an external SMTP server from the command line
  • All emails sent to the root user are forwarded to an external email address
  • All emails sent to admin@anotherdomain.com are automatically encrypted before they leave the server

Maintenance

I’m in the process of moving all of my servers to a small OpenVZ cluster. I had hoped that this would be painless, but some of my servers are not behaving well. The web server appears to working fine, but If I can’t figure out what the problem is I may have to take everything down for maintenance. Don’t expect the blog to be available for the next couple of days.

Update 1:

Turns out that most of my problems didn’t have anything to do with the move. It was only some old (and in one case very old) misconfigurations that didn’t take effect until I rebooted one of my servers; that, and I had forgotten just how much memory Gitlab uses.

Update 2:

The cluster is almost complete. I only need to configure Heartbeat and it should be done. Unfortunately, moving my servers revealed some rather alarming deficiencies in how some of the older servers have been configured, how backups are handled, and especially in how they are monitored. One server (albeit not a very important one) had not been backed up for months, and I had not been alerted of this. This is not acceptable!

The cluster will have to wait until I have dealt with this.

Update 3:

I have learnt a few things.

First, backup systems will (in this case at least) only work if all clocks are in sync, and if one of the servers doesn’t have a CMOS battery then any NTP failures will break the backup system. If the server in question also doesn’t have a proper hard drive to store logs on, can’t run OSSEC, can’t store logs remotely using rsyslog, and doesn’t send system mail to an external mail server, then the first you will hear of this is when you need those backups.

Second, a DRBD cluster is not worth the effort of managing properly for a couple of private servers. It was very easy to set up, but robust it was not. My first attempt to try taking a node down ended with the whole cluster going down and refusing to synchronize. It was probably my fault (it was in the middle of the night, so I’m pretty sure it must have been my fault in some way), but when the DRBD service started giving error messages and crashing, I decided it was time to try something else while I knew the data on one node was still good.

I did not want to be forced to restore my servers from backups. The reason I did this in the first place was because I’m unhappy with my backup system and I needed a stop-gap I could use until I can set up something more robust. Perhaps I will try this again in the future, but for now I will fall back on simple snapshots.

So ends my latest adventure with needlessly complex solutions to simple problems; in disgraceful failure and the use of a more appropriate tool. Next on the list; soldering and configuration management systems.

writetoserial – A small program that sends binary data to a serial port

While waiting for the hardware I need to get any further on my little programming project I have been experimenting with sending and receiving data from the FRDM-KL25Z through a serial port. Mostly to prepare the tools I will need to debug the system I’m planning on building, and in particular the code and algorithms I will use for file encryption. Unfortunately, sending and receiving files and binary data over a serial port proved to be a little harder to do than I had anticipated. It was possible to do this to some extent by using simple tools like ‘cat’ to write data to the serial port’s device file, but I could not get it to work reliably for binary data or files over a few kilobytes in size. It would sometimes drop data or stop halfway for no apparent reason.

Because of this I wrote a simple little program ‘writetoserial’ that reads the contents of file and writes it to a serial port as binary data. This works far more reliably and I’m now able to send both large text files and binary files to the FRDM-KL25Z.

USB key codes for a Swedish keyboard layout

In preparation for a larger project I have been planning for some time now (involving my FRDM-KL25Z, an emulated USB keyboard, and a bit of cryptography) I needed a list of all USB key codes for keys in a Swedish keyboard layout that produces printable glyphs, and their Unicode/ASCII values. To both my surprise and irritation I could not find any existing lists of key codes for anything except the US keyboard layout. I’m sure they are out there somewhere, but it was faster to simply write a small program for the FRDM-KL25Z that let my try different key codes to see what comes out. Most keys are the same in an SE and US layout so there were only a handful of keys I had to find.

I have uploaded the resulting list to GitHub. Since I had to do part of the work by hand there could be mistakes. I will be updating the list as soon as the program that will be using it is complete and I can verify that the values are correct.